2024-04-24T07:46:29.871 INFO:root:teuthology version: 0.0.1.dev269+g9481b1d 2024-04-24T07:46:29.872 DEBUG:teuthology.run:Teuthology command: teuthology --name rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi --owner scheduled_rishabh@teuthology --verbose --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} --archive /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204 -- /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204/orig.config.yaml 2024-04-24T07:46:29.932 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T07:46:30.022 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204 branch: rishabh-main-apr17-a654945 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} email: null first_in_suite: false job_id: '7671204' kernel: client: branch: testing flavor: default kdb: true sha1: 806ec36b21129f113ce356c007dd59694d340b37 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true name: rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: rishabh-main-apr17-a654945 ceph: cephfs: max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 filestore flush min: 0 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd heartbeat grace: 60 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10000 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request - SLOW_OPS - slow request sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 subvols: create: 2 subvol_options: --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=secure - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-main-apr17-a654945 sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 owner: scheduled_rishabh@teuthology priority: 75 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 7954 sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 sleep_before_teardown: 0 subset: 364/512 suite: fs suite_branch: rishabh-main-apr17-a654945 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi028.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ23IGa87DVqE2toJsvUYoiEyF3yd2TwVJzcc7VwcWrA5K90KJYbl9LUcnJ2MWJyHTiNpXwFyTJCjg+OOU9F7tU= smithi117.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLdWI3E0mxIOK39/rI/CQ3u+lUiKGS+RjCJ1l3jgc0xvGxn8Fto+KTFYA1RzUMOA1faH9bpZXiBz4ztcy65WQhk= smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNxqTNGooqW5Con49ykJPPFp8+6bTVVnxDBRVkmJdgBxNiLAlD7MYelsNSzzGjmMfTh8wk8ql9yQ9MLteaMjq+I= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - workunit: clients: all: - suites/ffsb.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17 timestamp: 2024-04-24_07:32:23 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-04-24T07:46:30.022 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa; will attempt to use it 2024-04-24T07:46:30.023 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa/tasks 2024-04-24T07:46:30.023 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-04-24T07:46:30.025 INFO:teuthology.task.internal:Checking packages... 2024-04-24T07:46:30.048 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '1f81fb397ae98da7563d451a78c61574c8f4e6e0' 2024-04-24T07:46:30.049 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-04-24T07:46:30.049 INFO:teuthology.packaging:ref: None 2024-04-24T07:46:30.049 INFO:teuthology.packaging:tag: None 2024-04-24T07:46:30.049 INFO:teuthology.packaging:branch: rishabh-main-apr17-a654945 2024-04-24T07:46:30.049 INFO:teuthology.packaging:sha1: 1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T07:46:30.049 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=rishabh-main-apr17-a654945 2024-04-24T07:46:30.188 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-3116.g1f81fb39 2024-04-24T07:46:30.190 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-04-24T07:46:30.196 INFO:teuthology.task.internal:no buildpackages task found 2024-04-24T07:46:30.196 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-04-24T07:46:30.262 INFO:teuthology.task.internal:Saving configuration 2024-04-24T07:46:30.284 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-04-24T07:46:30.290 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T07:46:30.313 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 07:41:23.360035', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ23IGa87DVqE2toJsvUYoiEyF3yd2TwVJzcc7VwcWrA5K90KJYbl9LUcnJ2MWJyHTiNpXwFyTJCjg+OOU9F7tU='} 2024-04-24T07:46:30.337 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 07:41:23.361045', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLdWI3E0mxIOK39/rI/CQ3u+lUiKGS+RjCJ1l3jgc0xvGxn8Fto+KTFYA1RzUMOA1faH9bpZXiBz4ztcy65WQhk='} 2024-04-24T07:46:30.361 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 07:41:23.361851', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNxqTNGooqW5Con49ykJPPFp8+6bTVVnxDBRVkmJdgBxNiLAlD7MYelsNSzzGjmMfTh8wk8ql9yQ9MLteaMjq+I='} 2024-04-24T07:46:30.361 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-04-24T07:46:30.367 INFO:teuthology.task.internal:roles: ubuntu@smithi028.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-04-24T07:46:30.368 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-04-24T07:46:30.368 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-04-24T07:46:30.368 INFO:teuthology.run_tasks:Running task console_log... 2024-04-24T07:46:30.484 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f74a4430160>, signals=[15]) 2024-04-24T07:46:30.485 INFO:teuthology.run_tasks:Running task internal.connect... 2024-04-24T07:46:30.491 INFO:teuthology.task.internal:Opening connections... 2024-04-24T07:46:30.491 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi028.front.sepia.ceph.com 2024-04-24T07:46:30.493 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:46:30.569 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2024-04-24T07:46:30.569 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:46:30.650 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-04-24T07:46:30.651 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:46:30.724 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-04-24T07:46:30.732 DEBUG:teuthology.orchestra.run.smithi028:> uname -m 2024-04-24T07:46:30.751 INFO:teuthology.orchestra.run.smithi028.stdout:x86_64 2024-04-24T07:46:30.751 DEBUG:teuthology.orchestra.run.smithi028:> cat /etc/os-release 2024-04-24T07:46:30.810 INFO:teuthology.orchestra.run.smithi028.stdout:NAME="CentOS Stream" 2024-04-24T07:46:30.810 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION="9" 2024-04-24T07:46:30.811 INFO:teuthology.orchestra.run.smithi028.stdout:ID="centos" 2024-04-24T07:46:30.811 INFO:teuthology.orchestra.run.smithi028.stdout:ID_LIKE="rhel fedora" 2024-04-24T07:46:30.811 INFO:teuthology.orchestra.run.smithi028.stdout:VERSION_ID="9" 2024-04-24T07:46:30.811 INFO:teuthology.orchestra.run.smithi028.stdout:PLATFORM_ID="platform:el9" 2024-04-24T07:46:30.811 INFO:teuthology.orchestra.run.smithi028.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T07:46:30.811 INFO:teuthology.orchestra.run.smithi028.stdout:ANSI_COLOR="0;31" 2024-04-24T07:46:30.811 INFO:teuthology.orchestra.run.smithi028.stdout:LOGO="fedora-logo-icon" 2024-04-24T07:46:30.811 INFO:teuthology.orchestra.run.smithi028.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T07:46:30.811 INFO:teuthology.orchestra.run.smithi028.stdout:HOME_URL="https://centos.org/" 2024-04-24T07:46:30.811 INFO:teuthology.orchestra.run.smithi028.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T07:46:30.811 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T07:46:30.811 INFO:teuthology.orchestra.run.smithi028.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T07:46:30.812 INFO:teuthology.lock.ops:Updating smithi028.front.sepia.ceph.com on lock server 2024-04-24T07:46:30.835 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2024-04-24T07:46:30.852 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2024-04-24T07:46:30.852 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2024-04-24T07:46:30.909 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="CentOS Stream" 2024-04-24T07:46:30.909 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="9" 2024-04-24T07:46:30.909 INFO:teuthology.orchestra.run.smithi117.stdout:ID="centos" 2024-04-24T07:46:30.909 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE="rhel fedora" 2024-04-24T07:46:30.909 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="9" 2024-04-24T07:46:30.909 INFO:teuthology.orchestra.run.smithi117.stdout:PLATFORM_ID="platform:el9" 2024-04-24T07:46:30.909 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T07:46:30.909 INFO:teuthology.orchestra.run.smithi117.stdout:ANSI_COLOR="0;31" 2024-04-24T07:46:30.910 INFO:teuthology.orchestra.run.smithi117.stdout:LOGO="fedora-logo-icon" 2024-04-24T07:46:30.910 INFO:teuthology.orchestra.run.smithi117.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T07:46:30.910 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://centos.org/" 2024-04-24T07:46:30.910 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T07:46:30.910 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T07:46:30.910 INFO:teuthology.orchestra.run.smithi117.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T07:46:30.910 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2024-04-24T07:46:30.934 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-04-24T07:46:30.952 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-04-24T07:46:30.953 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-04-24T07:46:31.010 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Stream" 2024-04-24T07:46:31.010 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="9" 2024-04-24T07:46:31.010 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos" 2024-04-24T07:46:31.011 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora" 2024-04-24T07:46:31.011 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="9" 2024-04-24T07:46:31.011 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el9" 2024-04-24T07:46:31.011 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Stream 9" 2024-04-24T07:46:31.011 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2024-04-24T07:46:31.011 INFO:teuthology.orchestra.run.smithi194.stdout:LOGO="fedora-logo-icon" 2024-04-24T07:46:31.011 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-04-24T07:46:31.011 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/" 2024-04-24T07:46:31.011 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-04-24T07:46:31.011 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-04-24T07:46:31.011 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-04-24T07:46:31.011 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-04-24T07:46:31.034 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-04-24T07:46:31.046 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-04-24T07:46:31.073 INFO:teuthology.task.internal:Checking for old test directory... 2024-04-24T07:46:31.073 DEBUG:teuthology.orchestra.run.smithi028:> test '!' -e /home/ubuntu/cephtest 2024-04-24T07:46:31.076 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2024-04-24T07:46:31.079 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-04-24T07:46:31.096 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-04-24T07:46:31.105 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-04-24T07:46:31.105 DEBUG:teuthology.orchestra.run.smithi028:> test -z $(ls -A /var/lib/ceph) 2024-04-24T07:46:31.135 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2024-04-24T07:46:31.138 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-04-24T07:46:31.176 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-04-24T07:46:31.295 INFO:teuthology.run_tasks:Running task kernel... 2024-04-24T07:46:31.446 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '806ec36b21129f113ce356c007dd59694d340b37'} 2024-04-24T07:46:31.447 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-04-24T07:46:31.447 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-04-24T07:46:31.447 DEBUG:teuthology.orchestra.run.smithi028:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T07:46:31.465 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T07:46:31.466 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-04-24T07:46:31.649 DEBUG:teuthology.task.kernel:sha1 for client.0 is 806ec36b21129f113ce356c007dd59694d340b37 2024-04-24T07:46:31.649 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "806ec36b21129f113ce356c007dd59694d340b37"... 2024-04-24T07:46:31.649 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-04-24T07:46:31.671 INFO:teuthology.orchestra.run.smithi028.stdout:5.14.0-437.el9.x86_64 2024-04-24T07:46:31.671 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-437.el9.x86_64 vs 806ec36b21129f113ce356c007dd59694d340b37 2024-04-24T07:46:31.672 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-04-24T07:46:31.672 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum upgrade -y linux-firmware 2024-04-24T07:46:33.796 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.0 MB 00:00 2024-04-24T07:46:35.931 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 19 MB/s | 19 MB 00:01 2024-04-24T07:46:40.498 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 7.0 MB/s | 6.2 MB 00:00 2024-04-24T07:46:41.991 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 58 kB/s | 16 kB 00:00 2024-04-24T07:46:43.644 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-24T07:46:48.900 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-04-24T07:46:49.090 INFO:teuthology.orchestra.run.smithi028.stdout:Last metadata expiration check: 0:00:01 ago on Wed 24 Apr 2024 07:46:48 AM UTC. 2024-04-24T07:46:50.921 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-24T07:46:50.922 INFO:teuthology.orchestra.run.smithi028.stdout:Nothing to do. 2024-04-24T07:46:50.922 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-24T07:46:51.017 INFO:teuthology.task.kernel:Downloading kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-24T07:46:51.018 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=806ec36b21129f113ce356c007dd59694d340b37 2024-04-24T07:46:51.163 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ 2024-04-24T07:46:51.163 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/ --input-file=- 2024-04-24T07:46:51.166 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi028.front.sepia.ceph.com to complete... 2024-04-24T07:47:03.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24 07:47:03 URL:https://4.chacra.ceph.com/r/kernel/testing/806ec36b21129f113ce356c007dd59694d340b37/centos/9/flavors/default/x86_64/kernel-6.9.0_rc3_g806ec36b2112-1.x86_64.rpm [357367792/357367792] -> "/tmp/kernel.x86_64.rpm" [1] 2024-04-24T07:47:03.300 INFO:teuthology.orchestra.run.smithi028.stderr:FINISHED --2024-04-24 07:47:03-- 2024-04-24T07:47:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:Total wall clock time: 12s 2024-04-24T07:47:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:Downloaded: 1 files, 341M in 12s (28.8 MB/s) 2024-04-24T07:47:03.304 INFO:teuthology.task.kernel:Installing kernel 806ec36b21129f113ce356c007dd59694d340b37 on client.0... 2024-04-24T07:47:03.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-04-24T07:47:05.490 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying... ######################################## 2024-04-24T07:47:05.621 INFO:teuthology.orchestra.run.smithi028.stdout:Preparing... ######################################## 2024-04-24T07:47:05.744 INFO:teuthology.orchestra.run.smithi028.stdout:Updating / installing... 2024-04-24T07:47:14.101 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-6.9.0_rc3_g806ec36b2112-1 ######################################## 2024-04-24T07:47:53.757 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi028.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-04-24T07:47:53.757 DEBUG:teuthology.orchestra.run.smithi028:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-04-24T07:47:53.844 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/System.map-6.9.0-rc3-g806ec36b2112 2024-04-24T07:47:53.845 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/config-6.9.0-rc3-g806ec36b2112 2024-04-24T07:47:53.845 INFO:teuthology.orchestra.run.smithi028.stdout:/boot/vmlinuz-6.9.0-rc3-g806ec36b2112 2024-04-24T07:47:53.845 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/System.map 2024-04-24T07:47:53.845 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/config 2024-04-24T07:47:53.845 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel 2024-04-24T07:47:53.845 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch 2024-04-24T07:47:53.845 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86 2024-04-24T07:47:53.845 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto 2024-04-24T07:47:53.845 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-04-24T07:47:53.845 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-04-24T07:47:53.845 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-04-24T07:47:53.845 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-04-24T07:47:53.845 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-04-24T07:47:53.846 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-04-24T07:47:53.846 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-04-24T07:47:53.846 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crc32c-intel.ko 2024-04-24T07:47:53.846 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-04-24T07:47:53.846 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-04-24T07:47:53.846 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-04-24T07:47:53.846 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-04-24T07:47:53.846 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-avx2.ko 2024-04-24T07:47:53.846 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-04-24T07:47:53.846 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-04-24T07:47:53.846 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-04-24T07:47:53.846 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-04-24T07:47:53.846 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-04-24T07:47:53.847 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-04-24T07:47:53.847 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-04-24T07:47:53.847 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel 2024-04-24T07:47:53.847 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu 2024-04-24T07:47:53.847 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce 2024-04-24T07:47:53.847 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-04-24T07:47:53.847 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm 2024-04-24T07:47:53.847 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-amd.ko 2024-04-24T07:47:53.847 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm-intel.ko 2024-04-24T07:47:53.847 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/arch/x86/kvm/kvm.ko 2024-04-24T07:47:53.847 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto 2024-04-24T07:47:53.847 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ansi_cprng.ko 2024-04-24T07:47:53.847 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/anubis.ko 2024-04-24T07:47:53.847 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/arc4.ko 2024-04-24T07:47:53.847 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx 2024-04-24T07:47:53.848 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_memcpy.ko 2024-04-24T07:47:53.848 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_pq.ko 2024-04-24T07:47:53.848 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_raid6_recov.ko 2024-04-24T07:47:53.848 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_tx.ko 2024-04-24T07:47:53.848 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/async_xor.ko 2024-04-24T07:47:53.848 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/async_tx/raid6test.ko 2024-04-24T07:47:53.848 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authenc.ko 2024-04-24T07:47:53.848 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/authencesn.ko 2024-04-24T07:47:53.848 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blake2b_generic.ko 2024-04-24T07:47:53.848 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_common.ko 2024-04-24T07:47:53.848 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/blowfish_generic.ko 2024-04-24T07:47:53.848 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/camellia_generic.ko 2024-04-24T07:47:53.848 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast5_generic.ko 2024-04-24T07:47:53.848 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast6_generic.ko 2024-04-24T07:47:53.849 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cast_common.ko 2024-04-24T07:47:53.849 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/ccm.ko 2024-04-24T07:47:53.849 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cmac.ko 2024-04-24T07:47:53.849 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crc32_generic.ko 2024-04-24T07:47:53.849 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_engine.ko 2024-04-24T07:47:53.849 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/crypto_user.ko 2024-04-24T07:47:53.849 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/cts.ko 2024-04-24T07:47:53.849 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/des_generic.ko 2024-04-24T07:47:53.849 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/dh_generic.ko 2024-04-24T07:47:53.849 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/echainiv.ko 2024-04-24T07:47:53.849 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/essiv.ko 2024-04-24T07:47:53.849 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/fcrypt.ko 2024-04-24T07:47:53.849 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/khazad.ko 2024-04-24T07:47:53.849 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4.ko 2024-04-24T07:47:53.849 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/lz4hc.ko 2024-04-24T07:47:53.850 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/md4.ko 2024-04-24T07:47:53.850 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/michael_mic.ko 2024-04-24T07:47:53.850 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcbc.ko 2024-04-24T07:47:53.850 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/pcrypt.ko 2024-04-24T07:47:53.850 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/rmd160.ko 2024-04-24T07:47:53.850 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/seed.ko 2024-04-24T07:47:53.850 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/serpent_generic.ko 2024-04-24T07:47:53.850 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tcrypt.ko 2024-04-24T07:47:53.850 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/tea.ko 2024-04-24T07:47:53.850 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_common.ko 2024-04-24T07:47:53.850 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/twofish_generic.ko 2024-04-24T07:47:53.850 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/vmac.ko 2024-04-24T07:47:53.850 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/wp512.ko 2024-04-24T07:47:53.850 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xcbc.ko 2024-04-24T07:47:53.851 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xor.ko 2024-04-24T07:47:53.851 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/crypto/xxhash_generic.ko 2024-04-24T07:47:53.851 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers 2024-04-24T07:47:53.851 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi 2024-04-24T07:47:53.851 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_ipmi.ko 2024-04-24T07:47:53.851 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/acpi_pad.ko 2024-04-24T07:47:53.851 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/ec_sys.ko 2024-04-24T07:47:53.851 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/platform_profile.ko 2024-04-24T07:47:53.851 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbs.ko 2024-04-24T07:47:53.851 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/sbshc.ko 2024-04-24T07:47:53.851 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/acpi/video.ko 2024-04-24T07:47:53.851 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata 2024-04-24T07:47:53.851 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/acard-ahci.ko 2024-04-24T07:47:53.851 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ahci_platform.ko 2024-04-24T07:47:53.852 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/ata_generic.ko 2024-04-24T07:47:53.852 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/libahci_platform.ko 2024-04-24T07:47:53.852 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_acpi.ko 2024-04-24T07:47:53.852 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ali.ko 2024-04-24T07:47:53.852 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_amd.ko 2024-04-24T07:47:53.852 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_artop.ko 2024-04-24T07:47:53.852 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atiixp.ko 2024-04-24T07:47:53.852 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_atp867x.ko 2024-04-24T07:47:53.852 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd640.ko 2024-04-24T07:47:53.852 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cmd64x.ko 2024-04-24T07:47:53.852 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_cypress.ko 2024-04-24T07:47:53.852 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_efar.ko 2024-04-24T07:47:53.852 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt366.ko 2024-04-24T07:47:53.852 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt37x.ko 2024-04-24T07:47:53.853 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x2n.ko 2024-04-24T07:47:53.853 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_hpt3x3.ko 2024-04-24T07:47:53.853 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it8213.ko 2024-04-24T07:47:53.853 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_it821x.ko 2024-04-24T07:47:53.853 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_jmicron.ko 2024-04-24T07:47:53.853 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_marvell.ko 2024-04-24T07:47:53.853 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_mpiix.ko 2024-04-24T07:47:53.853 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_netcell.ko 2024-04-24T07:47:53.853 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ninja32.ko 2024-04-24T07:47:53.853 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87410.ko 2024-04-24T07:47:53.853 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_ns87415.ko 2024-04-24T07:47:53.853 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_oldpiix.ko 2024-04-24T07:47:53.853 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_opti.ko 2024-04-24T07:47:53.854 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_optidma.ko 2024-04-24T07:47:53.854 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pcmcia.ko 2024-04-24T07:47:53.854 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc2027x.ko 2024-04-24T07:47:53.854 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-04-24T07:47:53.854 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_piccolo.ko 2024-04-24T07:47:53.854 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_rdc.ko 2024-04-24T07:47:53.854 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sch.ko 2024-04-24T07:47:53.854 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_serverworks.ko 2024-04-24T07:47:53.854 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sil680.ko 2024-04-24T07:47:53.854 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sis.ko 2024-04-24T07:47:53.854 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_sl82c105.ko 2024-04-24T07:47:53.854 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_triflex.ko 2024-04-24T07:47:53.854 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pata_via.ko 2024-04-24T07:47:53.854 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/pdc_adma.ko 2024-04-24T07:47:53.855 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_inic162x.ko 2024-04-24T07:47:53.855 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_mv.ko 2024-04-24T07:47:53.855 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_nv.ko 2024-04-24T07:47:53.855 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_promise.ko 2024-04-24T07:47:53.855 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_qstor.ko 2024-04-24T07:47:53.855 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil.ko 2024-04-24T07:47:53.855 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sil24.ko 2024-04-24T07:47:53.855 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sis.ko 2024-04-24T07:47:53.855 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_svw.ko 2024-04-24T07:47:53.855 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_sx4.ko 2024-04-24T07:47:53.855 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_uli.ko 2024-04-24T07:47:53.855 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_via.ko 2024-04-24T07:47:53.855 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ata/sata_vsc.ko 2024-04-24T07:47:53.855 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay 2024-04-24T07:47:53.856 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/auxdisplay/ks0108.ko 2024-04-24T07:47:53.856 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base 2024-04-24T07:47:53.856 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap 2024-04-24T07:47:53.856 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-i2c.ko 2024-04-24T07:47:53.856 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/base/regmap/regmap-mmio.ko 2024-04-24T07:47:53.856 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma 2024-04-24T07:47:53.856 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/bcma/bcma.ko 2024-04-24T07:47:53.856 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block 2024-04-24T07:47:53.856 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe 2024-04-24T07:47:53.856 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/aoe/aoe.ko 2024-04-24T07:47:53.856 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/brd.ko 2024-04-24T07:47:53.856 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd 2024-04-24T07:47:53.856 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/drbd/drbd.ko 2024-04-24T07:47:53.856 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/floppy.ko 2024-04-24T07:47:53.857 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/loop.ko 2024-04-24T07:47:53.857 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx 2024-04-24T07:47:53.857 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-04-24T07:47:53.857 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/nbd.ko 2024-04-24T07:47:53.857 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk 2024-04-24T07:47:53.857 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/null_blk/null_blk.ko 2024-04-24T07:47:53.857 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/pktcdvd.ko 2024-04-24T07:47:53.857 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/rbd.ko 2024-04-24T07:47:53.857 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/virtio_blk.ko 2024-04-24T07:47:53.857 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback 2024-04-24T07:47:53.857 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-04-24T07:47:53.857 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/xen-blkfront.ko 2024-04-24T07:47:53.857 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram 2024-04-24T07:47:53.858 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/block/zram/zram.ko 2024-04-24T07:47:53.858 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char 2024-04-24T07:47:53.858 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hangcheck-timer.ko 2024-04-24T07:47:53.858 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random 2024-04-24T07:47:53.858 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/amd-rng.ko 2024-04-24T07:47:53.858 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/intel-rng.ko 2024-04-24T07:47:53.858 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-04-24T07:47:53.858 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/via-rng.ko 2024-04-24T07:47:53.858 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/hw_random/virtio-rng.ko 2024-04-24T07:47:53.858 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi 2024-04-24T07:47:53.858 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-04-24T07:47:53.858 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-04-24T07:47:53.858 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-04-24T07:47:53.858 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_si.ko 2024-04-24T07:47:53.859 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-04-24T07:47:53.859 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-04-24T07:47:53.859 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/lp.ko 2024-04-24T07:47:53.859 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave 2024-04-24T07:47:53.859 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/mwave/mwave.ko 2024-04-24T07:47:53.859 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/ppdev.ko 2024-04-24T07:47:53.859 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tlclk.ko 2024-04-24T07:47:53.859 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm 2024-04-24T07:47:53.859 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm.ko 2024-04-24T07:47:53.859 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_atmel.ko 2024-04-24T07:47:53.859 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_infineon.ko 2024-04-24T07:47:53.859 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_nsc.ko 2024-04-24T07:47:53.859 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis.ko 2024-04-24T07:47:53.859 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-04-24T07:47:53.860 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/uv_mmtimer.ko 2024-04-24T07:47:53.860 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/char/virtio_console.ko 2024-04-24T07:47:53.860 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq 2024-04-24T07:47:53.860 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-04-24T07:47:53.860 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-04-24T07:47:53.860 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/p4-clockmod.ko 2024-04-24T07:47:53.860 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-04-24T07:47:53.860 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/powernow-k8.ko 2024-04-24T07:47:53.860 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/cpufreq/speedstep-lib.ko 2024-04-24T07:47:53.860 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto 2024-04-24T07:47:53.860 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp 2024-04-24T07:47:53.860 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-04-24T07:47:53.860 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/ccp/ccp.ko 2024-04-24T07:47:53.860 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel 2024-04-24T07:47:53.860 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat 2024-04-24T07:47:53.861 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common 2024-04-24T07:47:53.861 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-04-24T07:47:53.861 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-04-24T07:47:53.861 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-04-24T07:47:53.861 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-aes.ko 2024-04-24T07:47:53.861 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/padlock-sha.ko 2024-04-24T07:47:53.861 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio 2024-04-24T07:47:53.861 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-04-24T07:47:53.861 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca 2024-04-24T07:47:53.861 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dca/dca.ko 2024-04-24T07:47:53.861 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq 2024-04-24T07:47:53.861 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-04-24T07:47:53.861 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma 2024-04-24T07:47:53.861 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw 2024-04-24T07:47:53.862 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/dw/dw_dmac.ko 2024-04-24T07:47:53.862 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat 2024-04-24T07:47:53.862 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/dma/ioat/ioatdma.ko 2024-04-24T07:47:53.862 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac 2024-04-24T07:47:53.862 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/amd64_edac.ko 2024-04-24T07:47:53.862 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/e752x_edac.ko 2024-04-24T07:47:53.862 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/edac_mce_amd.ko 2024-04-24T07:47:53.862 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3000_edac.ko 2024-04-24T07:47:53.862 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i3200_edac.ko 2024-04-24T07:47:53.862 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5100_edac.ko 2024-04-24T07:47:53.862 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i5400_edac.ko 2024-04-24T07:47:53.862 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7300_edac.ko 2024-04-24T07:47:53.862 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i7core_edac.ko 2024-04-24T07:47:53.862 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/i82975x_edac.ko 2024-04-24T07:47:53.862 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/ie31200_edac.ko 2024-04-24T07:47:53.863 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/sb_edac.ko 2024-04-24T07:47:53.863 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/edac/x38_edac.ko 2024-04-24T07:47:53.863 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware 2024-04-24T07:47:53.863 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/edd.ko 2024-04-24T07:47:53.863 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/firmware/iscsi_ibft.ko 2024-04-24T07:47:53.863 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio 2024-04-24T07:47:53.863 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-ich.ko 2024-04-24T07:47:53.863 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/gpio/gpio-viperboard.ko 2024-04-24T07:47:53.863 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid 2024-04-24T07:47:53.863 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-appleir.ko 2024-04-24T07:47:53.863 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-aureal.ko 2024-04-24T07:47:53.863 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-axff.ko 2024-04-24T07:47:53.863 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-dr.ko 2024-04-24T07:47:53.864 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elecom.ko 2024-04-24T07:47:53.864 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-elo.ko 2024-04-24T07:47:53.864 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-emsff.ko 2024-04-24T07:47:53.864 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gaff.ko 2024-04-24T07:47:53.864 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gt683r.ko 2024-04-24T07:47:53.864 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-gyration.ko 2024-04-24T07:47:53.864 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-kbd.ko 2024-04-24T07:47:53.864 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtek-mouse.ko 2024-04-24T07:47:53.864 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-holtekff.ko 2024-04-24T07:47:53.864 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-hyperv.ko 2024-04-24T07:47:53.864 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-icade.ko 2024-04-24T07:47:53.864 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-keytouch.ko 2024-04-24T07:47:53.864 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-kye.ko 2024-04-24T07:47:53.864 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lcpower.ko 2024-04-24T07:47:53.865 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-led.ko 2024-04-24T07:47:53.865 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-lenovo.ko 2024-04-24T07:47:53.865 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-dj.ko 2024-04-24T07:47:53.865 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-04-24T07:47:53.865 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-multitouch.ko 2024-04-24T07:47:53.865 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-ortek.ko 2024-04-24T07:47:53.865 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-penmount.ko 2024-04-24T07:47:53.865 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-petalynx.ko 2024-04-24T07:47:53.865 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-picolcd.ko 2024-04-24T07:47:53.865 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-pl.ko 2024-04-24T07:47:53.865 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-plantronics.ko 2024-04-24T07:47:53.865 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-primax.ko 2024-04-24T07:47:53.865 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-rmi.ko 2024-04-24T07:47:53.865 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-arvo.ko 2024-04-24T07:47:53.865 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-common.ko 2024-04-24T07:47:53.866 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-isku.ko 2024-04-24T07:47:53.866 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kone.ko 2024-04-24T07:47:53.866 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-04-24T07:47:53.866 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-konepure.ko 2024-04-24T07:47:53.866 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-04-24T07:47:53.866 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-lua.ko 2024-04-24T07:47:53.866 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-pyra.ko 2024-04-24T07:47:53.866 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-ryos.ko 2024-04-24T07:47:53.866 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat-savu.ko 2024-04-24T07:47:53.866 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-roccat.ko 2024-04-24T07:47:53.866 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-saitek.ko 2024-04-24T07:47:53.866 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-samsung.ko 2024-04-24T07:47:53.866 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sensor-hub.ko 2024-04-24T07:47:53.867 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sjoy.ko 2024-04-24T07:47:53.867 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sony.ko 2024-04-24T07:47:53.867 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-speedlink.ko 2024-04-24T07:47:53.867 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-steelseries.ko 2024-04-24T07:47:53.867 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-sunplus.ko 2024-04-24T07:47:53.867 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-thrustmaster.ko 2024-04-24T07:47:53.867 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tivo.ko 2024-04-24T07:47:53.867 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-tmff.ko 2024-04-24T07:47:53.867 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-topseed.ko 2024-04-24T07:47:53.867 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-twinhan.ko 2024-04-24T07:47:53.867 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-uclogic.ko 2024-04-24T07:47:53.867 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-waltop.ko 2024-04-24T07:47:53.867 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-wiimote.ko 2024-04-24T07:47:53.867 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-xinmo.ko 2024-04-24T07:47:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zpff.ko 2024-04-24T07:47:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/hid-zydacron.ko 2024-04-24T07:47:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/uhid.ko 2024-04-24T07:47:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hid/wacom.ko 2024-04-24T07:47:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv 2024-04-24T07:47:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_balloon.ko 2024-04-24T07:47:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_utils.ko 2024-04-24T07:47:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hv/hv_vmbus.ko 2024-04-24T07:47:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon 2024-04-24T07:47:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru.ko 2024-04-24T07:47:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/abituguru3.ko 2024-04-24T07:47:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/acpi_power_meter.ko 2024-04-24T07:47:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7414.ko 2024-04-24T07:47:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ad7418.ko 2024-04-24T07:47:53.868 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adc128d818.ko 2024-04-24T07:47:53.869 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1025.ko 2024-04-24T07:47:53.869 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1026.ko 2024-04-24T07:47:53.869 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1029.ko 2024-04-24T07:47:53.869 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm1031.ko 2024-04-24T07:47:53.869 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adm9240.ko 2024-04-24T07:47:53.869 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ads7828.ko 2024-04-24T07:47:53.869 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7410.ko 2024-04-24T07:47:53.869 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7411.ko 2024-04-24T07:47:53.869 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7462.ko 2024-04-24T07:47:53.869 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7470.ko 2024-04-24T07:47:53.869 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7475.ko 2024-04-24T07:47:53.869 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/adt7x10.ko 2024-04-24T07:47:53.869 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/amc6821.ko 2024-04-24T07:47:53.869 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/applesmc.ko 2024-04-24T07:47:53.869 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asb100.ko 2024-04-24T07:47:53.870 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asc7621.ko 2024-04-24T07:47:53.870 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/asus_atk0110.ko 2024-04-24T07:47:53.870 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/atxp1.ko 2024-04-24T07:47:53.870 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/coretemp.ko 2024-04-24T07:47:53.870 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-04-24T07:47:53.870 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/dme1737.ko 2024-04-24T07:47:53.870 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds1621.ko 2024-04-24T07:47:53.870 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ds620.ko 2024-04-24T07:47:53.870 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc1403.ko 2024-04-24T07:47:53.870 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/emc6w201.ko 2024-04-24T07:47:53.870 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71805f.ko 2024-04-24T07:47:53.870 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f71882fg.ko 2024-04-24T07:47:53.870 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/f75375s.ko 2024-04-24T07:47:53.870 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fam15h_power.ko 2024-04-24T07:47:53.871 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/fschmd.ko 2024-04-24T07:47:53.871 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g760a.ko 2024-04-24T07:47:53.871 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/g762.ko 2024-04-24T07:47:53.871 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl518sm.ko 2024-04-24T07:47:53.871 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/gl520sm.ko 2024-04-24T07:47:53.871 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/hwmon-vid.ko 2024-04-24T07:47:53.871 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5500_temp.ko 2024-04-24T07:47:53.871 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/i5k_amb.ko 2024-04-24T07:47:53.871 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmaem.ko 2024-04-24T07:47:53.871 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ibmpex.ko 2024-04-24T07:47:53.871 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina209.ko 2024-04-24T07:47:53.871 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ina2xx.ko 2024-04-24T07:47:53.871 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/it87.ko 2024-04-24T07:47:53.871 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k10temp.ko 2024-04-24T07:47:53.872 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/k8temp.ko 2024-04-24T07:47:53.872 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lineage-pem.ko 2024-04-24T07:47:53.872 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm63.ko 2024-04-24T07:47:53.872 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm73.ko 2024-04-24T07:47:53.872 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm75.ko 2024-04-24T07:47:53.872 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm77.ko 2024-04-24T07:47:53.872 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm78.ko 2024-04-24T07:47:53.872 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm80.ko 2024-04-24T07:47:53.872 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm83.ko 2024-04-24T07:47:53.872 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm85.ko 2024-04-24T07:47:53.872 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm87.ko 2024-04-24T07:47:53.872 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm90.ko 2024-04-24T07:47:53.872 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm92.ko 2024-04-24T07:47:53.872 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm93.ko 2024-04-24T07:47:53.872 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95234.ko 2024-04-24T07:47:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95241.ko 2024-04-24T07:47:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/lm95245.ko 2024-04-24T07:47:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc2945.ko 2024-04-24T07:47:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4151.ko 2024-04-24T07:47:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4215.ko 2024-04-24T07:47:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4222.ko 2024-04-24T07:47:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4245.ko 2024-04-24T07:47:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4260.ko 2024-04-24T07:47:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/ltc4261.ko 2024-04-24T07:47:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max16065.ko 2024-04-24T07:47:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1619.ko 2024-04-24T07:47:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max1668.ko 2024-04-24T07:47:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max197.ko 2024-04-24T07:47:53.873 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6639.ko 2024-04-24T07:47:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6650.ko 2024-04-24T07:47:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/max6697.ko 2024-04-24T07:47:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/mcp3021.ko 2024-04-24T07:47:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6683.ko 2024-04-24T07:47:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775-core.ko 2024-04-24T07:47:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct6775.ko 2024-04-24T07:47:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/nct7802.ko 2024-04-24T07:47:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87360.ko 2024-04-24T07:47:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pc87427.ko 2024-04-24T07:47:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pcf8591.ko 2024-04-24T07:47:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus 2024-04-24T07:47:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-04-24T07:47:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-04-24T07:47:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-04-24T07:47:53.874 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max16064.ko 2024-04-24T07:47:53.875 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max34440.ko 2024-04-24T07:47:53.875 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/max8688.ko 2024-04-24T07:47:53.875 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-04-24T07:47:53.875 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-04-24T07:47:53.875 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-04-24T07:47:53.875 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-04-24T07:47:53.875 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-04-24T07:47:53.875 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-04-24T07:47:53.875 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/powr1220.ko 2024-04-24T07:47:53.875 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5627.ko 2024-04-24T07:47:53.875 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch5636.ko 2024-04-24T07:47:53.875 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sch56xx-common.ko 2024-04-24T07:47:53.875 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht15.ko 2024-04-24T07:47:53.875 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sht21.ko 2024-04-24T07:47:53.876 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/shtc1.ko 2024-04-24T07:47:53.876 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/sis5595.ko 2024-04-24T07:47:53.876 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47b397.ko 2024-04-24T07:47:53.876 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m1.ko 2024-04-24T07:47:53.876 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/smsc47m192.ko 2024-04-24T07:47:53.876 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/thmc50.ko 2024-04-24T07:47:53.876 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp102.ko 2024-04-24T07:47:53.876 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp103.ko 2024-04-24T07:47:53.876 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp401.ko 2024-04-24T07:47:53.876 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/tmp421.ko 2024-04-24T07:47:53.876 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via-cputemp.ko 2024-04-24T07:47:53.876 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/via686a.ko 2024-04-24T07:47:53.876 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt1211.ko 2024-04-24T07:47:53.876 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/vt8231.ko 2024-04-24T07:47:53.876 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627ehf.ko 2024-04-24T07:47:53.877 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83627hf.ko 2024-04-24T07:47:53.877 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83781d.ko 2024-04-24T07:47:53.877 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83791d.ko 2024-04-24T07:47:53.877 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83792d.ko 2024-04-24T07:47:53.877 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83793.ko 2024-04-24T07:47:53.877 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83795.ko 2024-04-24T07:47:53.877 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l785ts.ko 2024-04-24T07:47:53.877 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/hwmon/w83l786ng.ko 2024-04-24T07:47:53.877 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c 2024-04-24T07:47:53.877 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos 2024-04-24T07:47:53.877 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-04-24T07:47:53.877 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-04-24T07:47:53.877 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses 2024-04-24T07:47:53.877 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-04-24T07:47:53.877 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-04-24T07:47:53.878 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-04-24T07:47:53.878 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-04-24T07:47:53.878 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-04-24T07:47:53.878 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-04-24T07:47:53.878 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-04-24T07:47:53.878 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-04-24T07:47:53.878 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-i801.ko 2024-04-24T07:47:53.878 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-isch.ko 2024-04-24T07:47:53.878 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-04-24T07:47:53.878 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-04-24T07:47:53.878 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-04-24T07:47:53.878 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-parport.ko 2024-04-24T07:47:53.878 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-04-24T07:47:53.879 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-04-24T07:47:53.879 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-04-24T07:47:53.879 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-04-24T07:47:53.879 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-04-24T07:47:53.879 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-04-24T07:47:53.879 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-via.ko 2024-04-24T07:47:53.879 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-04-24T07:47:53.879 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-04-24T07:47:53.879 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-dev.ko 2024-04-24T07:47:53.879 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-mux.ko 2024-04-24T07:47:53.879 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-04-24T07:47:53.879 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-smbus.ko 2024-04-24T07:47:53.879 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/i2c/i2c-stub.ko 2024-04-24T07:47:53.879 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband 2024-04-24T07:47:53.879 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core 2024-04-24T07:47:53.880 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_cm.ko 2024-04-24T07:47:53.880 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_core.ko 2024-04-24T07:47:53.880 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_umad.ko 2024-04-24T07:47:53.880 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-04-24T07:47:53.880 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/iw_cm.ko 2024-04-24T07:47:53.880 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_cm.ko 2024-04-24T07:47:53.880 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-04-24T07:47:53.880 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw 2024-04-24T07:47:53.880 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4 2024-04-24T07:47:53.880 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-04-24T07:47:53.880 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4 2024-04-24T07:47:53.880 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-04-24T07:47:53.880 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5 2024-04-24T07:47:53.880 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-04-24T07:47:53.880 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca 2024-04-24T07:47:53.881 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-04-24T07:47:53.881 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp 2024-04-24T07:47:53.881 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib 2024-04-24T07:47:53.881 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-04-24T07:47:53.881 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser 2024-04-24T07:47:53.881 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-04-24T07:47:53.881 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert 2024-04-24T07:47:53.881 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-04-24T07:47:53.881 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp 2024-04-24T07:47:53.881 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-04-24T07:47:53.881 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt 2024-04-24T07:47:53.881 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-04-24T07:47:53.881 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input 2024-04-24T07:47:53.881 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport 2024-04-24T07:47:53.882 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-04-24T07:47:53.882 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/fm801-gp.ko 2024-04-24T07:47:53.882 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/gameport/gameport.ko 2024-04-24T07:47:53.882 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/joydev.ko 2024-04-24T07:47:53.882 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard 2024-04-24T07:47:53.882 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/keyboard/gpio_keys.ko 2024-04-24T07:47:53.882 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse 2024-04-24T07:47:53.882 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/appletouch.ko 2024-04-24T07:47:53.882 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/bcm5974.ko 2024-04-24T07:47:53.882 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/cyapatp.ko 2024-04-24T07:47:53.882 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/elan_i2c.ko 2024-04-24T07:47:53.882 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/sermouse.ko 2024-04-24T07:47:53.882 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-04-24T07:47:53.882 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/synaptics_usb.ko 2024-04-24T07:47:53.883 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/mouse/vsxxxaa.ko 2024-04-24T07:47:53.883 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4 2024-04-24T07:47:53.883 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/rmi4/rmi_core.ko 2024-04-24T07:47:53.883 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio 2024-04-24T07:47:53.883 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/altera_ps2.ko 2024-04-24T07:47:53.883 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/arc_ps2.ko 2024-04-24T07:47:53.883 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-04-24T07:47:53.883 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/serio/serio_raw.ko 2024-04-24T07:47:53.883 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/input/sparse-keymap.ko 2024-04-24T07:47:53.883 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds 2024-04-24T07:47:53.883 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-blinkm.ko 2024-04-24T07:47:53.883 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lm3530.ko 2024-04-24T07:47:53.883 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-lp3944.ko 2024-04-24T07:47:53.883 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/leds-ss4200.ko 2024-04-24T07:47:53.884 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger 2024-04-24T07:47:53.884 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-04-24T07:47:53.884 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-04-24T07:47:53.884 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-04-24T07:47:53.884 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-04-24T07:47:53.884 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-04-24T07:47:53.884 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-04-24T07:47:53.884 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-04-24T07:47:53.884 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-04-24T07:47:53.884 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-04-24T07:47:53.884 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md 2024-04-24T07:47:53.884 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache 2024-04-24T07:47:53.884 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/bcache/bcache.ko 2024-04-24T07:47:53.884 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-bio-prison.ko 2024-04-24T07:47:53.884 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache-smq.ko 2024-04-24T07:47:53.885 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-cache.ko 2024-04-24T07:47:53.885 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-crypt.ko 2024-04-24T07:47:53.885 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-delay.ko 2024-04-24T07:47:53.885 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-flakey.ko 2024-04-24T07:47:53.885 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-log-userspace.ko 2024-04-24T07:47:53.885 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-multipath.ko 2024-04-24T07:47:53.885 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-queue-length.ko 2024-04-24T07:47:53.885 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-raid.ko 2024-04-24T07:47:53.885 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-round-robin.ko 2024-04-24T07:47:53.885 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-service-time.ko 2024-04-24T07:47:53.885 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-switch.ko 2024-04-24T07:47:53.885 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-thin-pool.ko 2024-04-24T07:47:53.885 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/dm-verity.ko 2024-04-24T07:47:53.885 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data 2024-04-24T07:47:53.886 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-04-24T07:47:53.886 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid0.ko 2024-04-24T07:47:53.886 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid1.ko 2024-04-24T07:47:53.886 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid10.ko 2024-04-24T07:47:53.886 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/md/raid456.ko 2024-04-24T07:47:53.886 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message 2024-04-24T07:47:53.886 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion 2024-04-24T07:47:53.886 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptbase.ko 2024-04-24T07:47:53.886 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptctl.ko 2024-04-24T07:47:53.886 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptfc.ko 2024-04-24T07:47:53.886 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptlan.ko 2024-04-24T07:47:53.886 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptsas.ko 2024-04-24T07:47:53.886 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptscsih.ko 2024-04-24T07:47:53.886 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/message/fusion/mptspi.ko 2024-04-24T07:47:53.887 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd 2024-04-24T07:47:53.887 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_ich.ko 2024-04-24T07:47:53.887 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/lpc_sch.ko 2024-04-24T07:47:53.887 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/mfd-core.ko 2024-04-24T07:47:53.887 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/sm501.ko 2024-04-24T07:47:53.887 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/viperboard.ko 2024-04-24T07:47:53.887 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/vx855.ko 2024-04-24T07:47:53.887 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/mfd/wl1273-core.ko 2024-04-24T07:47:53.887 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc 2024-04-24T07:47:53.887 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl 2024-04-24T07:47:53.887 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-04-24T07:47:53.887 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds9802als.ko 2024-04-24T07:47:53.887 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/apds990x.ko 2024-04-24T07:47:53.887 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/bh1770glc.ko 2024-04-24T07:47:53.887 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710 2024-04-24T07:47:53.888 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/cb710/cb710.ko 2024-04-24T07:47:53.888 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom 2024-04-24T07:47:53.888 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/at24.ko 2024-04-24T07:47:53.888 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-04-24T07:47:53.888 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/eeprom/max6875.ko 2024-04-24T07:47:53.888 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/enclosure.ko 2024-04-24T07:47:53.888 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/hpilo.ko 2024-04-24T07:47:53.888 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29003.ko 2024-04-24T07:47:53.888 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/isl29020.ko 2024-04-24T07:47:53.888 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d 2024-04-24T07:47:53.888 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-04-24T07:47:53.888 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-04-24T07:47:53.888 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei 2024-04-24T07:47:53.888 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-me.ko 2024-04-24T07:47:53.888 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei-txe.ko 2024-04-24T07:47:53.889 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/mei/mei.ko 2024-04-24T07:47:53.889 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru 2024-04-24T07:47:53.889 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-gru/gru.ko 2024-04-24T07:47:53.889 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp 2024-04-24T07:47:53.889 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xp.ko 2024-04-24T07:47:53.889 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpc.ko 2024-04-24T07:47:53.889 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-04-24T07:47:53.889 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_7xx1.ko 2024-04-24T07:47:53.889 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tifm_core.ko 2024-04-24T07:47:53.889 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/tsl2550.ko 2024-04-24T07:47:53.889 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_balloon.ko 2024-04-24T07:47:53.889 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci 2024-04-24T07:47:53.889 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-04-24T07:47:53.889 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net 2024-04-24T07:47:53.890 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding 2024-04-24T07:47:53.890 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/bonding/bonding.ko 2024-04-24T07:47:53.890 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa 2024-04-24T07:47:53.890 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53 2024-04-24T07:47:53.890 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/b53/b53_common.ko 2024-04-24T07:47:53.890 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/bcm-sf2.ko 2024-04-24T07:47:53.890 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6060.ko 2024-04-24T07:47:53.890 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx 2024-04-24T07:47:53.890 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-04-24T07:47:53.890 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/dummy.ko 2024-04-24T07:47:53.890 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/eql.ko 2024-04-24T07:47:53.890 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet 2024-04-24T07:47:53.890 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com 2024-04-24T07:47:53.890 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-04-24T07:47:53.890 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-04-24T07:47:53.891 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-04-24T07:47:53.891 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-04-24T07:47:53.891 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390 2024-04-24T07:47:53.891 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/8390.ko 2024-04-24T07:47:53.891 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-04-24T07:47:53.891 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-04-24T07:47:53.891 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-04-24T07:47:53.891 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec 2024-04-24T07:47:53.891 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-04-24T07:47:53.891 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere 2024-04-24T07:47:53.891 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/agere/et131x.ko 2024-04-24T07:47:53.891 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon 2024-04-24T07:47:53.891 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-04-24T07:47:53.892 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera 2024-04-24T07:47:53.892 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-04-24T07:47:53.892 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd 2024-04-24T07:47:53.892 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-04-24T07:47:53.892 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-04-24T07:47:53.892 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-04-24T07:47:53.892 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros 2024-04-24T07:47:53.892 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx 2024-04-24T07:47:53.892 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-04-24T07:47:53.892 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c 2024-04-24T07:47:53.892 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-04-24T07:47:53.892 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e 2024-04-24T07:47:53.892 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-04-24T07:47:53.892 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx 2024-04-24T07:47:53.893 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-04-24T07:47:53.893 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-04-24T07:47:53.893 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom 2024-04-24T07:47:53.893 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-04-24T07:47:53.893 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-04-24T07:47:53.893 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-04-24T07:47:53.893 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-04-24T07:47:53.893 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-04-24T07:47:53.893 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet 2024-04-24T07:47:53.893 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-04-24T07:47:53.893 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-04-24T07:47:53.893 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade 2024-04-24T07:47:53.893 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna 2024-04-24T07:47:53.893 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-04-24T07:47:53.894 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio 2024-04-24T07:47:53.894 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb 2024-04-24T07:47:53.894 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-04-24T07:47:53.894 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-04-24T07:47:53.894 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-04-24T07:47:53.894 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-04-24T07:47:53.894 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-04-24T07:47:53.894 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-04-24T07:47:53.894 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-04-24T07:47:53.894 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-04-24T07:47:53.894 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-04-24T07:47:53.894 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco 2024-04-24T07:47:53.894 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic 2024-04-24T07:47:53.894 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-04-24T07:47:53.895 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec 2024-04-24T07:47:53.895 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip 2024-04-24T07:47:53.895 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-04-24T07:47:53.895 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-04-24T07:47:53.895 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-04-24T07:47:53.895 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-04-24T07:47:53.895 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-04-24T07:47:53.895 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-04-24T07:47:53.895 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink 2024-04-24T07:47:53.895 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-04-24T07:47:53.895 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-04-24T07:47:53.895 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/dnet.ko 2024-04-24T07:47:53.895 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex 2024-04-24T07:47:53.895 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet 2024-04-24T07:47:53.895 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-04-24T07:47:53.896 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ethoc.ko 2024-04-24T07:47:53.896 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/fealnx.ko 2024-04-24T07:47:53.896 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel 2024-04-24T07:47:53.896 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e100.ko 2024-04-24T07:47:53.896 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000 2024-04-24T07:47:53.896 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-04-24T07:47:53.896 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e 2024-04-24T07:47:53.896 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-04-24T07:47:53.896 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k 2024-04-24T07:47:53.896 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-04-24T07:47:53.896 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e 2024-04-24T07:47:53.896 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-04-24T07:47:53.896 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf 2024-04-24T07:47:53.896 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-04-24T07:47:53.897 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb 2024-04-24T07:47:53.897 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-04-24T07:47:53.897 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf 2024-04-24T07:47:53.897 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-04-24T07:47:53.897 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe 2024-04-24T07:47:53.897 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-04-24T07:47:53.897 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf 2024-04-24T07:47:53.897 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-04-24T07:47:53.897 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/jme.ko 2024-04-24T07:47:53.897 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell 2024-04-24T07:47:53.897 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-04-24T07:47:53.897 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/skge.ko 2024-04-24T07:47:53.897 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-04-24T07:47:53.897 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox 2024-04-24T07:47:53.898 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4 2024-04-24T07:47:53.898 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-04-24T07:47:53.898 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-04-24T07:47:53.898 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5 2024-04-24T07:47:53.898 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-04-24T07:47:53.898 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-04-24T07:47:53.898 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel 2024-04-24T07:47:53.898 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-04-24T07:47:53.898 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom 2024-04-24T07:47:53.898 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge 2024-04-24T07:47:53.898 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-04-24T07:47:53.898 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi 2024-04-24T07:47:53.898 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-04-24T07:47:53.898 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-04-24T07:47:53.899 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion 2024-04-24T07:47:53.899 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-04-24T07:47:53.899 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia 2024-04-24T07:47:53.899 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-04-24T07:47:53.899 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines 2024-04-24T07:47:53.899 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-04-24T07:47:53.899 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-04-24T07:47:53.899 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic 2024-04-24T07:47:53.899 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen 2024-04-24T07:47:53.899 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-04-24T07:47:53.899 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-04-24T07:47:53.899 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-04-24T07:47:53.899 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-04-24T07:47:53.899 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc 2024-04-24T07:47:53.899 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-04-24T07:47:53.900 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek 2024-04-24T07:47:53.900 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-04-24T07:47:53.900 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-04-24T07:47:53.900 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/atp.ko 2024-04-24T07:47:53.900 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-04-24T07:47:53.900 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker 2024-04-24T07:47:53.900 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-04-24T07:47:53.900 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc 2024-04-24T07:47:53.900 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-04-24T07:47:53.900 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan 2024-04-24T07:47:53.900 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-04-24T07:47:53.900 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis 2024-04-24T07:47:53.900 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis190.ko 2024-04-24T07:47:53.900 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sis/sis900.ko 2024-04-24T07:47:53.901 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc 2024-04-24T07:47:53.901 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-04-24T07:47:53.901 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-04-24T07:47:53.901 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-04-24T07:47:53.901 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-04-24T07:47:53.901 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro 2024-04-24T07:47:53.901 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac 2024-04-24T07:47:53.901 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-04-24T07:47:53.901 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-04-24T07:47:53.901 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun 2024-04-24T07:47:53.901 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/cassini.ko 2024-04-24T07:47:53.901 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/niu.ko 2024-04-24T07:47:53.901 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sungem.ko 2024-04-24T07:47:53.901 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-04-24T07:47:53.901 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti 2024-04-24T07:47:53.902 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-04-24T07:47:53.902 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti 2024-04-24T07:47:53.902 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/ti/tlan.ko 2024-04-24T07:47:53.902 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via 2024-04-24T07:47:53.902 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-04-24T07:47:53.902 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-04-24T07:47:53.902 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet 2024-04-24T07:47:53.902 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-04-24T07:47:53.902 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-04-24T07:47:53.902 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom 2024-04-24T07:47:53.902 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-04-24T07:47:53.902 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/geneve.ko 2024-04-24T07:47:53.902 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv 2024-04-24T07:47:53.902 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-04-24T07:47:53.903 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154 2024-04-24T07:47:53.903 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ieee802154/fakelb.ko 2024-04-24T07:47:53.903 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ifb.ko 2024-04-24T07:47:53.903 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan 2024-04-24T07:47:53.903 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ipvlan/ipvlan.ko 2024-04-24T07:47:53.903 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvlan.ko 2024-04-24T07:47:53.903 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/macvtap.ko 2024-04-24T07:47:53.903 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio 2024-04-24T07:47:53.903 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio.ko 2024-04-24T07:47:53.904 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-04-24T07:47:53.904 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-04-24T07:47:53.904 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mdio/mdio-regmap.ko 2024-04-24T07:47:53.904 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/mii.ko 2024-04-24T07:47:53.904 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/net_failover.ko 2024-04-24T07:47:53.904 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/netconsole.ko 2024-04-24T07:47:53.904 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/nlmon.ko 2024-04-24T07:47:53.904 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs 2024-04-24T07:47:53.904 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs-lynx.ko 2024-04-24T07:47:53.904 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-04-24T07:47:53.904 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy 2024-04-24T07:47:53.904 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/amd.ko 2024-04-24T07:47:53.904 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-04-24T07:47:53.904 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-04-24T07:47:53.904 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm7xxx.ko 2024-04-24T07:47:53.905 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/bcm87xx.ko 2024-04-24T07:47:53.905 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/broadcom.ko 2024-04-24T07:47:53.905 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/cicada.ko 2024-04-24T07:47:53.905 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/davicom.ko 2024-04-24T07:47:53.905 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/dp83640.ko 2024-04-24T07:47:53.905 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/et1011c.ko 2024-04-24T07:47:53.905 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/icplus.ko 2024-04-24T07:47:53.905 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/lxt.ko 2024-04-24T07:47:53.905 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/marvell.ko 2024-04-24T07:47:53.905 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/micrel.ko 2024-04-24T07:47:53.905 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/national.ko 2024-04-24T07:47:53.905 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/phylink.ko 2024-04-24T07:47:53.905 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/qsemi.ko 2024-04-24T07:47:53.905 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/realtek.ko 2024-04-24T07:47:53.906 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/smsc.ko 2024-04-24T07:47:53.906 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/ste10Xp.ko 2024-04-24T07:47:53.906 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/phy/vitesse.ko 2024-04-24T07:47:53.906 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp 2024-04-24T07:47:53.906 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/bsd_comp.ko 2024-04-24T07:47:53.906 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_async.ko 2024-04-24T07:47:53.906 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_deflate.ko 2024-04-24T07:47:53.906 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_generic.ko 2024-04-24T07:47:53.906 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_mppe.ko 2024-04-24T07:47:53.906 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/ppp_synctty.ko 2024-04-24T07:47:53.906 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppoe.ko 2024-04-24T07:47:53.906 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pppox.ko 2024-04-24T07:47:53.906 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/ppp/pptp.ko 2024-04-24T07:47:53.906 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip 2024-04-24T07:47:53.907 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slhc.ko 2024-04-24T07:47:53.907 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/slip/slip.ko 2024-04-24T07:47:53.907 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/sungem_phy.ko 2024-04-24T07:47:53.907 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tap.ko 2024-04-24T07:47:53.907 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team 2024-04-24T07:47:53.907 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team.ko 2024-04-24T07:47:53.907 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_activebackup.ko 2024-04-24T07:47:53.907 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_broadcast.ko 2024-04-24T07:47:53.907 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-04-24T07:47:53.907 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_random.ko 2024-04-24T07:47:53.907 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-04-24T07:47:53.907 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/tun.ko 2024-04-24T07:47:53.907 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/veth.ko 2024-04-24T07:47:53.907 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/virtio_net.ko 2024-04-24T07:47:53.908 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3 2024-04-24T07:47:53.908 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-04-24T07:47:53.908 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan 2024-04-24T07:47:53.908 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/vxlan/vxlan.ko 2024-04-24T07:47:53.908 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback 2024-04-24T07:47:53.908 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netback/xen-netback.ko 2024-04-24T07:47:53.908 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/net/xen-netfront.ko 2024-04-24T07:47:53.908 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb 2024-04-24T07:47:53.908 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ntb/ntb.ko 2024-04-24T07:47:53.908 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme 2024-04-24T07:47:53.908 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host 2024-04-24T07:47:53.908 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme-core.ko 2024-04-24T07:47:53.908 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/nvme/host/nvme.ko 2024-04-24T07:47:53.908 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport 2024-04-24T07:47:53.909 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport.ko 2024-04-24T07:47:53.909 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_cs.ko 2024-04-24T07:47:53.909 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_pc.ko 2024-04-24T07:47:53.909 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/parport/parport_serial.ko 2024-04-24T07:47:53.909 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci 2024-04-24T07:47:53.909 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug 2024-04-24T07:47:53.909 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-04-24T07:47:53.909 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie 2024-04-24T07:47:53.909 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/pcie/aer_inject.ko 2024-04-24T07:47:53.909 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pci/xen-pcifront.ko 2024-04-24T07:47:53.909 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia 2024-04-24T07:47:53.909 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/i82092.ko 2024-04-24T07:47:53.909 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/pd6729.ko 2024-04-24T07:47:53.909 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pcmcia/yenta_socket.ko 2024-04-24T07:47:53.910 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl 2024-04-24T07:47:53.910 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel 2024-04-24T07:47:53.910 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-04-24T07:47:53.910 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform 2024-04-24T07:47:53.910 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86 2024-04-24T07:47:53.910 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acer-wmi.ko 2024-04-24T07:47:53.910 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/acerhdf.ko 2024-04-24T07:47:53.910 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/apple-gmux.ko 2024-04-24T07:47:53.910 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-laptop.ko 2024-04-24T07:47:53.910 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-04-24T07:47:53.910 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/asus-wmi.ko 2024-04-24T07:47:53.910 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/classmate-laptop.ko 2024-04-24T07:47:53.910 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-04-24T07:47:53.910 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-04-24T07:47:53.911 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-04-24T07:47:53.911 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-04-24T07:47:53.911 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/hdaps.ko 2024-04-24T07:47:53.911 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel 2024-04-24T07:47:53.911 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-04-24T07:47:53.911 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/intel_ips.ko 2024-04-24T07:47:53.911 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/msi-wmi.ko 2024-04-24T07:47:53.911 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/mxm-wmi.ko 2024-04-24T07:47:53.911 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-04-24T07:47:53.911 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-laptop.ko 2024-04-24T07:47:53.911 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/samsung-q10.ko 2024-04-24T07:47:53.911 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/topstar-laptop.ko 2024-04-24T07:47:53.911 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-04-24T07:47:53.911 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/toshiba_haps.ko 2024-04-24T07:47:53.911 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi-bmof.ko 2024-04-24T07:47:53.912 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/platform/x86/wmi.ko 2024-04-24T07:47:53.912 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap 2024-04-24T07:47:53.912 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_common.ko 2024-04-24T07:47:53.912 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/powercap/intel_rapl_msr.ko 2024-04-24T07:47:53.912 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps 2024-04-24T07:47:53.912 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients 2024-04-24T07:47:53.912 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-gpio.ko 2024-04-24T07:47:53.912 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps-ldisc.ko 2024-04-24T07:47:53.912 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/clients/pps_parport.ko 2024-04-24T07:47:53.912 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/pps/pps_core.ko 2024-04-24T07:47:53.912 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp 2024-04-24T07:47:53.912 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp.ko 2024-04-24T07:47:53.912 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ptp/ptp_kvm.ko 2024-04-24T07:47:53.912 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras 2024-04-24T07:47:53.913 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd 2024-04-24T07:47:53.913 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl 2024-04-24T07:47:53.913 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/atl/amd_atl.ko 2024-04-24T07:47:53.913 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ras/amd/fmpm.ko 2024-04-24T07:47:53.913 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc 2024-04-24T07:47:53.913 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-bq32k.ko 2024-04-24T07:47:53.913 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1286.ko 2024-04-24T07:47:53.913 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1307.ko 2024-04-24T07:47:53.913 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1374.ko 2024-04-24T07:47:53.913 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1511.ko 2024-04-24T07:47:53.913 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1553.ko 2024-04-24T07:47:53.913 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1672.ko 2024-04-24T07:47:53.913 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds1742.ko 2024-04-24T07:47:53.913 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds2404.ko 2024-04-24T07:47:53.913 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-ds3232.ko 2024-04-24T07:47:53.914 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-em3027.ko 2024-04-24T07:47:53.914 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-fm3130.ko 2024-04-24T07:47:53.914 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl12022.ko 2024-04-24T07:47:53.914 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-isl1208.ko 2024-04-24T07:47:53.914 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m41t80.ko 2024-04-24T07:47:53.914 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t35.ko 2024-04-24T07:47:53.914 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-m48t59.ko 2024-04-24T07:47:53.914 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-max6900.ko 2024-04-24T07:47:53.914 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-msm6242.ko 2024-04-24T07:47:53.914 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf2127.ko 2024-04-24T07:47:53.914 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf85063.ko 2024-04-24T07:47:53.914 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8523.ko 2024-04-24T07:47:53.914 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8563.ko 2024-04-24T07:47:53.914 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-pcf8583.ko 2024-04-24T07:47:53.915 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rp5c01.ko 2024-04-24T07:47:53.915 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rs5c372.ko 2024-04-24T07:47:53.915 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-04-24T07:47:53.915 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8025.ko 2024-04-24T07:47:53.915 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-rx8581.ko 2024-04-24T07:47:53.915 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-04-24T07:47:53.915 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/rtc/rtc-x1205.ko 2024-04-24T07:47:53.915 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi 2024-04-24T07:47:53.915 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-9xxx.ko 2024-04-24T07:47:53.915 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-sas.ko 2024-04-24T07:47:53.915 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/3w-xxxx.ko 2024-04-24T07:47:53.915 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/BusLogic.ko 2024-04-24T07:47:53.915 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/a100u2w.ko 2024-04-24T07:47:53.915 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid 2024-04-24T07:47:53.915 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aacraid/aacraid.ko 2024-04-24T07:47:53.916 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/advansys.ko 2024-04-24T07:47:53.916 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx 2024-04-24T07:47:53.916 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-04-24T07:47:53.916 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-04-24T07:47:53.916 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/am53c974.ko 2024-04-24T07:47:53.916 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr 2024-04-24T07:47:53.916 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-04-24T07:47:53.916 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/atp870u.ko 2024-04-24T07:47:53.916 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi 2024-04-24T07:47:53.916 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-04-24T07:47:53.916 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa 2024-04-24T07:47:53.916 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bfa/bfa.ko 2024-04-24T07:47:53.916 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc 2024-04-24T07:47:53.916 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-04-24T07:47:53.916 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i 2024-04-24T07:47:53.917 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-04-24T07:47:53.917 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ch.ko 2024-04-24T07:47:53.917 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor 2024-04-24T07:47:53.917 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/csiostor/csiostor.ko 2024-04-24T07:47:53.917 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi 2024-04-24T07:47:53.917 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i 2024-04-24T07:47:53.917 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-04-24T07:47:53.917 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i 2024-04-24T07:47:53.917 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-04-24T07:47:53.917 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-04-24T07:47:53.917 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dc395x.ko 2024-04-24T07:47:53.917 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler 2024-04-24T07:47:53.917 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-04-24T07:47:53.918 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-04-24T07:47:53.918 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-04-24T07:47:53.918 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-04-24T07:47:53.918 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/dmx3191d.ko 2024-04-24T07:47:53.918 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r 2024-04-24T07:47:53.918 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esas2r/esas2r.ko 2024-04-24T07:47:53.918 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/esp_scsi.ko 2024-04-24T07:47:53.918 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe 2024-04-24T07:47:53.918 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/fcoe.ko 2024-04-24T07:47:53.918 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-04-24T07:47:53.918 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic 2024-04-24T07:47:53.918 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/fnic/fnic.ko 2024-04-24T07:47:53.918 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hpsa.ko 2024-04-24T07:47:53.918 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hptiop.ko 2024-04-24T07:47:53.918 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/hv_storvsc.ko 2024-04-24T07:47:53.919 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/initio.ko 2024-04-24T07:47:53.919 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ipr.ko 2024-04-24T07:47:53.919 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ips.ko 2024-04-24T07:47:53.919 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci 2024-04-24T07:47:53.919 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/isci/isci.ko 2024-04-24T07:47:53.919 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-04-24T07:47:53.919 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/iscsi_tcp.ko 2024-04-24T07:47:53.919 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc 2024-04-24T07:47:53.919 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libfc/libfc.ko 2024-04-24T07:47:53.919 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi.ko 2024-04-24T07:47:53.919 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libiscsi_tcp.ko 2024-04-24T07:47:53.919 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas 2024-04-24T07:47:53.919 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/libsas/libsas.ko 2024-04-24T07:47:53.919 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc 2024-04-24T07:47:53.919 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/lpfc/lpfc.ko 2024-04-24T07:47:53.920 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid 2024-04-24T07:47:53.920 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid.ko 2024-04-24T07:47:53.920 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-04-24T07:47:53.920 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-04-24T07:47:53.920 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-04-24T07:47:53.920 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas 2024-04-24T07:47:53.920 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-04-24T07:47:53.920 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas 2024-04-24T07:47:53.920 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvsas/mvsas.ko 2024-04-24T07:47:53.920 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/mvumi.ko 2024-04-24T07:47:53.920 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001 2024-04-24T07:47:53.920 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-04-24T07:47:53.920 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/pmcraid.ko 2024-04-24T07:47:53.920 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla1280.ko 2024-04-24T07:47:53.921 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx 2024-04-24T07:47:53.921 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-04-24T07:47:53.921 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-04-24T07:47:53.921 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx 2024-04-24T07:47:53.921 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-04-24T07:47:53.921 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/raid_class.ko 2024-04-24T07:47:53.921 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_debug.ko 2024-04-24T07:47:53.921 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_fc.ko 2024-04-24T07:47:53.921 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-04-24T07:47:53.921 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_sas.ko 2024-04-24T07:47:53.921 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_spi.ko 2024-04-24T07:47:53.921 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/scsi_transport_srp.ko 2024-04-24T07:47:53.921 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/ses.ko 2024-04-24T07:47:53.921 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/st.ko 2024-04-24T07:47:53.921 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/stex.ko 2024-04-24T07:47:53.922 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2 2024-04-24T07:47:53.922 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-04-24T07:47:53.922 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/virtio_scsi.ko 2024-04-24T07:47:53.922 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/vmw_pvscsi.ko 2024-04-24T07:47:53.922 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/scsi/wd719x.ko 2024-04-24T07:47:53.922 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb 2024-04-24T07:47:53.922 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ssb/ssb.ko 2024-04-24T07:47:53.922 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target 2024-04-24T07:47:53.922 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi 2024-04-24T07:47:53.922 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-04-24T07:47:53.922 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback 2024-04-24T07:47:53.922 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/loopback/tcm_loop.ko 2024-04-24T07:47:53.922 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_file.ko 2024-04-24T07:47:53.922 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_iblock.ko 2024-04-24T07:47:53.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_mod.ko 2024-04-24T07:47:53.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_pscsi.ko 2024-04-24T07:47:53.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/target_core_user.ko 2024-04-24T07:47:53.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc 2024-04-24T07:47:53.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-04-24T07:47:53.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal 2024-04-24T07:47:53.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel 2024-04-24T07:47:53.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal 2024-04-24T07:47:53.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-04-24T07:47:53.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-04-24T07:47:53.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-04-24T07:47:53.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-04-24T07:47:53.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-04-24T07:47:53.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-04-24T07:47:53.923 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-04-24T07:47:53.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-04-24T07:47:53.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-04-24T07:47:53.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-04-24T07:47:53.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-04-24T07:47:53.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-04-24T07:47:53.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-04-24T07:47:53.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-04-24T07:47:53.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-04-24T07:47:53.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-04-24T07:47:53.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-04-24T07:47:53.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-04-24T07:47:53.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty 2024-04-24T07:47:53.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless 2024-04-24T07:47:53.924 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-04-24T07:47:53.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_gsm.ko 2024-04-24T07:47:53.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/n_hdlc.ko 2024-04-24T07:47:53.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/nozomi.ko 2024-04-24T07:47:53.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial 2024-04-24T07:47:53.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250 2024-04-24T07:47:53.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-04-24T07:47:53.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/arc_uart.ko 2024-04-24T07:47:53.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm 2024-04-24T07:47:53.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/tty/serial/jsm/jsm.ko 2024-04-24T07:47:53.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs 2024-04-24T07:47:53.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core 2024-04-24T07:47:53.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/core/ufshcd-core.ko 2024-04-24T07:47:53.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host 2024-04-24T07:47:53.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-04-24T07:47:53.925 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio 2024-04-24T07:47:53.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio.ko 2024-04-24T07:47:53.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_aec.ko 2024-04-24T07:47:53.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_cif.ko 2024-04-24T07:47:53.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_pci_generic.ko 2024-04-24T07:47:53.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/uio/uio_sercos3.ko 2024-04-24T07:47:53.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb 2024-04-24T07:47:53.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class 2024-04-24T07:47:53.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-acm.ko 2024-04-24T07:47:53.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/cdc-wdm.ko 2024-04-24T07:47:53.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usblp.ko 2024-04-24T07:47:53.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/class/usbtmc.ko 2024-04-24T07:47:53.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host 2024-04-24T07:47:53.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/host/sl811-hcd.ko 2024-04-24T07:47:53.926 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image 2024-04-24T07:47:53.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/mdc800.ko 2024-04-24T07:47:53.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/image/microtek.ko 2024-04-24T07:47:53.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc 2024-04-24T07:47:53.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/adutux.ko 2024-04-24T07:47:53.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/appledisplay.ko 2024-04-24T07:47:53.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi26.ko 2024-04-24T07:47:53.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/emi62.ko 2024-04-24T07:47:53.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ezusb.ko 2024-04-24T07:47:53.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/idmouse.ko 2024-04-24T07:47:53.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/iowarrior.ko 2024-04-24T07:47:53.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/isight_firmware.ko 2024-04-24T07:47:53.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/ldusb.ko 2024-04-24T07:47:53.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/legousbtower.ko 2024-04-24T07:47:53.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga 2024-04-24T07:47:53.927 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-04-24T07:47:53.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/trancevibrator.ko 2024-04-24T07:47:53.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usb3503.ko 2024-04-24T07:47:53.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usblcd.ko 2024-04-24T07:47:53.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/usbsevseg.ko 2024-04-24T07:47:53.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/uss720.ko 2024-04-24T07:47:53.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/misc/yurex.ko 2024-04-24T07:47:53.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy 2024-04-24T07:47:53.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/phy/phy-generic.ko 2024-04-24T07:47:53.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial 2024-04-24T07:47:53.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/aircable.ko 2024-04-24T07:47:53.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ark3116.ko 2024-04-24T07:47:53.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/belkin_sa.ko 2024-04-24T07:47:53.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ch341.ko 2024-04-24T07:47:53.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cp210x.ko 2024-04-24T07:47:53.928 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cyberjack.ko 2024-04-24T07:47:53.929 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/cypress_m8.ko 2024-04-24T07:47:53.929 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/digi_acceleport.ko 2024-04-24T07:47:53.929 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/empeg.ko 2024-04-24T07:47:53.929 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ftdi_sio.ko 2024-04-24T07:47:53.929 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/garmin_gps.ko 2024-04-24T07:47:53.929 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_edgeport.ko 2024-04-24T07:47:53.929 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/io_ti.ko 2024-04-24T07:47:53.929 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipaq.ko 2024-04-24T07:47:53.929 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ipw.ko 2024-04-24T07:47:53.929 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ir-usb.ko 2024-04-24T07:47:53.929 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-04-24T07:47:53.929 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan.ko 2024-04-24T07:47:53.929 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/keyspan_pda.ko 2024-04-24T07:47:53.929 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kl5kusb105.ko 2024-04-24T07:47:53.929 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/kobil_sct.ko 2024-04-24T07:47:53.930 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mct_u232.ko 2024-04-24T07:47:53.930 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7720.ko 2024-04-24T07:47:53.930 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/mos7840.ko 2024-04-24T07:47:53.930 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/navman.ko 2024-04-24T07:47:53.930 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/omninet.ko 2024-04-24T07:47:53.930 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/opticon.ko 2024-04-24T07:47:53.930 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/option.ko 2024-04-24T07:47:53.930 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/oti6858.ko 2024-04-24T07:47:53.930 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/pl2303.ko 2024-04-24T07:47:53.930 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcaux.ko 2024-04-24T07:47:53.930 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/qcserial.ko 2024-04-24T07:47:53.930 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/quatech2.ko 2024-04-24T07:47:53.930 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/safe_serial.ko 2024-04-24T07:47:53.930 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/sierra.ko 2024-04-24T07:47:53.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/spcp8x5.ko 2024-04-24T07:47:53.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ssu100.ko 2024-04-24T07:47:53.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/symbolserial.ko 2024-04-24T07:47:53.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-04-24T07:47:53.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-04-24T07:47:53.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_debug.ko 2024-04-24T07:47:53.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/usb_wwan.ko 2024-04-24T07:47:53.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/visor.ko 2024-04-24T07:47:53.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/whiteheat.ko 2024-04-24T07:47:53.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/serial/xsens_mt.ko 2024-04-24T07:47:53.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage 2024-04-24T07:47:53.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/uas.ko 2024-04-24T07:47:53.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-alauda.ko 2024-04-24T07:47:53.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-cypress.ko 2024-04-24T07:47:53.931 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-datafab.ko 2024-04-24T07:47:53.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-04-24T07:47:53.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-freecom.ko 2024-04-24T07:47:53.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-isd200.ko 2024-04-24T07:47:53.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-04-24T07:47:53.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-karma.ko 2024-04-24T07:47:53.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-onetouch.ko 2024-04-24T07:47:53.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-realtek.ko 2024-04-24T07:47:53.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr09.ko 2024-04-24T07:47:53.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-sddr55.ko 2024-04-24T07:47:53.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/ums-usbat.ko 2024-04-24T07:47:53.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/storage/usb-storage.ko 2024-04-24T07:47:53.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip 2024-04-24T07:47:53.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-core.ko 2024-04-24T07:47:53.932 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/usbip-host.ko 2024-04-24T07:47:53.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-04-24T07:47:53.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio 2024-04-24T07:47:53.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci 2024-04-24T07:47:53.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-04-24T07:47:53.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/pci/vfio-pci.ko 2024-04-24T07:47:53.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio.ko 2024-04-24T07:47:53.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-04-24T07:47:53.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost 2024-04-24T07:47:53.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost.ko 2024-04-24T07:47:53.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_iotlb.ko 2024-04-24T07:47:53.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_net.ko 2024-04-24T07:47:53.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/vhost/vhost_scsi.ko 2024-04-24T07:47:53.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video 2024-04-24T07:47:53.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight 2024-04-24T07:47:53.933 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/apple_bl.ko 2024-04-24T07:47:53.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/lcd.ko 2024-04-24T07:47:53.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/video/backlight/platform_lcd.ko 2024-04-24T07:47:53.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio 2024-04-24T07:47:53.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio.ko 2024-04-24T07:47:53.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_balloon.ko 2024-04-24T07:47:53.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_mmio.ko 2024-04-24T07:47:53.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci.ko 2024-04-24T07:47:53.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-04-24T07:47:53.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-04-24T07:47:53.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/virtio/virtio_ring.ko 2024-04-24T07:47:53.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1 2024-04-24T07:47:53.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters 2024-04-24T07:47:53.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2482.ko 2024-04-24T07:47:53.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/masters/ds2490.ko 2024-04-24T07:47:53.934 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves 2024-04-24T07:47:53.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-04-24T07:47:53.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-04-24T07:47:53.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-04-24T07:47:53.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-04-24T07:47:53.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-04-24T07:47:53.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-04-24T07:47:53.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-04-24T07:47:53.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-04-24T07:47:53.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-04-24T07:47:53.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_smem.ko 2024-04-24T07:47:53.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/slaves/w1_therm.ko 2024-04-24T07:47:53.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/w1/wire.ko 2024-04-24T07:47:53.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog 2024-04-24T07:47:53.935 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim1535_wdt.ko 2024-04-24T07:47:53.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/alim7101_wdt.ko 2024-04-24T07:47:53.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/f71808e_wdt.ko 2024-04-24T07:47:53.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/hpwdt.ko 2024-04-24T07:47:53.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/i6300esb.ko 2024-04-24T07:47:53.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-04-24T07:47:53.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/iTCO_wdt.ko 2024-04-24T07:47:53.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ib700wdt.ko 2024-04-24T07:47:53.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ibmasr.ko 2024-04-24T07:47:53.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-04-24T07:47:53.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it8712f_wdt.ko 2024-04-24T07:47:53.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/it87_wdt.ko 2024-04-24T07:47:53.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/machzwd.ko 2024-04-24T07:47:53.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/nv_tco.ko 2024-04-24T07:47:53.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_pci.ko 2024-04-24T07:47:53.936 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/pcwd_usb.ko 2024-04-24T07:47:53.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-04-24T07:47:53.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sch311x_wdt.ko 2024-04-24T07:47:53.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/softdog.ko 2024-04-24T07:47:53.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/sp5100_tco.ko 2024-04-24T07:47:53.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/via_wdt.ko 2024-04-24T07:47:53.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-04-24T07:47:53.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83877f_wdt.ko 2024-04-24T07:47:53.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/w83977f_wdt.ko 2024-04-24T07:47:53.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/wdt_pci.ko 2024-04-24T07:47:53.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/watchdog/xen_wdt.ko 2024-04-24T07:47:53.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen 2024-04-24T07:47:53.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-acpi-processor.ko 2024-04-24T07:47:53.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-evtchn.ko 2024-04-24T07:47:53.937 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntalloc.ko 2024-04-24T07:47:53.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-gntdev.ko 2024-04-24T07:47:53.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback 2024-04-24T07:47:53.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-04-24T07:47:53.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xen-privcmd.ko 2024-04-24T07:47:53.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs 2024-04-24T07:47:53.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/drivers/xen/xenfs/xenfs.ko 2024-04-24T07:47:53.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs 2024-04-24T07:47:53.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p 2024-04-24T07:47:53.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/9p/9p.ko 2024-04-24T07:47:53.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs 2024-04-24T07:47:53.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/affs/affs.ko 2024-04-24T07:47:53.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs 2024-04-24T07:47:53.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/befs/befs.ko 2024-04-24T07:47:53.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/binfmt_misc.ko 2024-04-24T07:47:53.938 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs 2024-04-24T07:47:53.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/btrfs/btrfs.ko 2024-04-24T07:47:53.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles 2024-04-24T07:47:53.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cachefiles/cachefiles.ko 2024-04-24T07:47:53.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph 2024-04-24T07:47:53.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ceph/ceph.ko 2024-04-24T07:47:53.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda 2024-04-24T07:47:53.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/coda/coda.ko 2024-04-24T07:47:53.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs 2024-04-24T07:47:53.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/cramfs/cramfs.ko 2024-04-24T07:47:53.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm 2024-04-24T07:47:53.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/dlm/dlm.ko 2024-04-24T07:47:53.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs 2024-04-24T07:47:53.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ecryptfs/ecryptfs.ko 2024-04-24T07:47:53.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs 2024-04-24T07:47:53.939 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/f2fs/f2fs.ko 2024-04-24T07:47:53.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat 2024-04-24T07:47:53.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/fat.ko 2024-04-24T07:47:53.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/msdos.ko 2024-04-24T07:47:53.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fat/vfat.ko 2024-04-24T07:47:53.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse 2024-04-24T07:47:53.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/cuse.ko 2024-04-24T07:47:53.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/fuse/fuse.ko 2024-04-24T07:47:53.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2 2024-04-24T07:47:53.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/gfs2/gfs2.ko 2024-04-24T07:47:53.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs 2024-04-24T07:47:53.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfs/hfs.ko 2024-04-24T07:47:53.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus 2024-04-24T07:47:53.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/hfsplus/hfsplus.ko 2024-04-24T07:47:53.940 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs 2024-04-24T07:47:53.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/isofs/isofs.ko 2024-04-24T07:47:53.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs 2024-04-24T07:47:53.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/jfs/jfs.ko 2024-04-24T07:47:53.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd 2024-04-24T07:47:53.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/lockd/lockd.ko 2024-04-24T07:47:53.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix 2024-04-24T07:47:53.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/minix/minix.ko 2024-04-24T07:47:53.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs 2024-04-24T07:47:53.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/netfs/netfs.ko 2024-04-24T07:47:53.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs 2024-04-24T07:47:53.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout 2024-04-24T07:47:53.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-04-24T07:47:53.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout 2024-04-24T07:47:53.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-04-24T07:47:53.941 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout 2024-04-24T07:47:53.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-04-24T07:47:53.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfs.ko 2024-04-24T07:47:53.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv3.ko 2024-04-24T07:47:53.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs/nfsv4.ko 2024-04-24T07:47:53.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common 2024-04-24T07:47:53.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/grace.ko 2024-04-24T07:47:53.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfs_common/nfs_acl.ko 2024-04-24T07:47:53.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd 2024-04-24T07:47:53.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nfsd/nfsd.ko 2024-04-24T07:47:53.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2 2024-04-24T07:47:53.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nilfs2/nilfs2.ko 2024-04-24T07:47:53.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls 2024-04-24T07:47:53.942 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-celtic.ko 2024-04-24T07:47:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-centeuro.ko 2024-04-24T07:47:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-croatian.ko 2024-04-24T07:47:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-cyrillic.ko 2024-04-24T07:47:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-gaelic.ko 2024-04-24T07:47:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-greek.ko 2024-04-24T07:47:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-iceland.ko 2024-04-24T07:47:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-inuit.ko 2024-04-24T07:47:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-roman.ko 2024-04-24T07:47:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-romanian.ko 2024-04-24T07:47:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/mac-turkish.ko 2024-04-24T07:47:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1250.ko 2024-04-24T07:47:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1251.ko 2024-04-24T07:47:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp1255.ko 2024-04-24T07:47:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp737.ko 2024-04-24T07:47:53.943 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp775.ko 2024-04-24T07:47:53.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp850.ko 2024-04-24T07:47:53.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp852.ko 2024-04-24T07:47:53.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp855.ko 2024-04-24T07:47:53.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp857.ko 2024-04-24T07:47:53.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp860.ko 2024-04-24T07:47:53.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp861.ko 2024-04-24T07:47:53.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp862.ko 2024-04-24T07:47:53.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp863.ko 2024-04-24T07:47:53.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp864.ko 2024-04-24T07:47:53.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp865.ko 2024-04-24T07:47:53.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp866.ko 2024-04-24T07:47:53.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp869.ko 2024-04-24T07:47:53.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp874.ko 2024-04-24T07:47:53.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp932.ko 2024-04-24T07:47:53.944 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp936.ko 2024-04-24T07:47:53.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp949.ko 2024-04-24T07:47:53.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_cp950.ko 2024-04-24T07:47:53.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_euc-jp.ko 2024-04-24T07:47:53.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-1.ko 2024-04-24T07:47:53.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-13.ko 2024-04-24T07:47:53.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-14.ko 2024-04-24T07:47:53.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-15.ko 2024-04-24T07:47:53.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-2.ko 2024-04-24T07:47:53.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-3.ko 2024-04-24T07:47:53.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-4.ko 2024-04-24T07:47:53.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-5.ko 2024-04-24T07:47:53.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-6.ko 2024-04-24T07:47:53.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-7.ko 2024-04-24T07:47:53.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_iso8859-9.ko 2024-04-24T07:47:53.945 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-r.ko 2024-04-24T07:47:53.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-ru.ko 2024-04-24T07:47:53.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_koi8-u.ko 2024-04-24T07:47:53.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_ucs2_utils.ko 2024-04-24T07:47:53.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/nls/nls_utf8.ko 2024-04-24T07:47:53.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2 2024-04-24T07:47:53.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster 2024-04-24T07:47:53.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-04-24T07:47:53.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm 2024-04-24T07:47:53.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-04-24T07:47:53.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs 2024-04-24T07:47:53.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-04-24T07:47:53.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2.ko 2024-04-24T07:47:53.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-04-24T07:47:53.946 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-04-24T07:47:53.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-04-24T07:47:53.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs 2024-04-24T07:47:53.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/overlayfs/overlay.ko 2024-04-24T07:47:53.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore 2024-04-24T07:47:53.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/pstore/ramoops.ko 2024-04-24T07:47:53.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs 2024-04-24T07:47:53.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/reiserfs/reiserfs.ko 2024-04-24T07:47:53.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs 2024-04-24T07:47:53.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/romfs/romfs.ko 2024-04-24T07:47:53.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb 2024-04-24T07:47:53.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client 2024-04-24T07:47:53.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/client/cifs.ko 2024-04-24T07:47:53.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common 2024-04-24T07:47:53.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_arc4.ko 2024-04-24T07:47:53.947 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/smb/common/cifs_md4.ko 2024-04-24T07:47:53.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs 2024-04-24T07:47:53.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/squashfs/squashfs.ko 2024-04-24T07:47:53.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv 2024-04-24T07:47:53.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/sysv/sysv.ko 2024-04-24T07:47:53.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf 2024-04-24T07:47:53.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/udf/udf.ko 2024-04-24T07:47:53.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs 2024-04-24T07:47:53.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/ufs/ufs.ko 2024-04-24T07:47:53.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs 2024-04-24T07:47:53.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/fs/xfs/xfs.ko 2024-04-24T07:47:53.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel 2024-04-24T07:47:53.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu 2024-04-24T07:47:53.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/rcu/rcutorture.ko 2024-04-24T07:47:53.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/torture.ko 2024-04-24T07:47:53.948 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace 2024-04-24T07:47:53.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-04-24T07:47:53.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib 2024-04-24T07:47:53.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/asn1_encoder.ko 2024-04-24T07:47:53.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc-itu-t.ko 2024-04-24T07:47:53.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crc8.ko 2024-04-24T07:47:53.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto 2024-04-24T07:47:53.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libarc4.ko 2024-04-24T07:47:53.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/crypto/libdes.ko 2024-04-24T07:47:53.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/libcrc32c.ko 2024-04-24T07:47:53.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lru_cache.ko 2024-04-24T07:47:53.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4 2024-04-24T07:47:53.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4_compress.ko 2024-04-24T07:47:53.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/lz4/lz4hc_compress.ko 2024-04-24T07:47:53.949 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math 2024-04-24T07:47:53.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/math/cordic.ko 2024-04-24T07:47:53.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6 2024-04-24T07:47:53.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/raid6/raid6_pq.ko 2024-04-24T07:47:53.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon 2024-04-24T07:47:53.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/reed_solomon/reed_solomon.ko 2024-04-24T07:47:53.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_bm.ko 2024-04-24T07:47:53.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_fsm.ko 2024-04-24T07:47:53.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/ts_kmp.ko 2024-04-24T07:47:53.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd 2024-04-24T07:47:53.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/lib/zstd/zstd_compress.ko 2024-04-24T07:47:53.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm 2024-04-24T07:47:53.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/mm/hwpoison-inject.ko 2024-04-24T07:47:53.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net 2024-04-24T07:47:53.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan 2024-04-24T07:47:53.950 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/6lowpan.ko 2024-04-24T07:47:53.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_dest.ko 2024-04-24T07:47:53.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_fragment.ko 2024-04-24T07:47:53.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_hop.ko 2024-04-24T07:47:53.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_ipv6.ko 2024-04-24T07:47:53.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_mobility.ko 2024-04-24T07:47:53.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_routing.ko 2024-04-24T07:47:53.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/6lowpan/nhc_udp.ko 2024-04-24T07:47:53.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802 2024-04-24T07:47:53.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/garp.ko 2024-04-24T07:47:53.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/mrp.ko 2024-04-24T07:47:53.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/p8022.ko 2024-04-24T07:47:53.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/psnap.ko 2024-04-24T07:47:53.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/802/stp.ko 2024-04-24T07:47:53.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q 2024-04-24T07:47:53.951 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/8021q/8021q.ko 2024-04-24T07:47:53.952 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p 2024-04-24T07:47:53.952 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet.ko 2024-04-24T07:47:53.952 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_fd.ko 2024-04-24T07:47:53.952 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_rdma.ko 2024-04-24T07:47:53.952 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/9p/9pnet_virtio.ko 2024-04-24T07:47:53.952 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv 2024-04-24T07:47:53.952 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/batman-adv/batman-adv.ko 2024-04-24T07:47:53.952 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge 2024-04-24T07:47:53.952 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/br_netfilter.ko 2024-04-24T07:47:53.952 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/bridge.ko 2024-04-24T07:47:53.952 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter 2024-04-24T07:47:53.952 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-04-24T07:47:53.952 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_among.ko 2024-04-24T07:47:53.952 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arp.ko 2024-04-24T07:47:53.952 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-04-24T07:47:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-04-24T07:47:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip.ko 2024-04-24T07:47:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-04-24T07:47:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_limit.ko 2024-04-24T07:47:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_log.ko 2024-04-24T07:47:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark.ko 2024-04-24T07:47:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-04-24T07:47:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-04-24T07:47:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-04-24T07:47:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-04-24T07:47:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_snat.ko 2024-04-24T07:47:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_stp.ko 2024-04-24T07:47:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-04-24T07:47:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-04-24T07:47:53.953 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-04-24T07:47:53.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-04-24T07:47:53.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/ebtables.ko 2024-04-24T07:47:53.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-04-24T07:47:53.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-04-24T07:47:53.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-04-24T07:47:53.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph 2024-04-24T07:47:53.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ceph/libceph.ko 2024-04-24T07:47:53.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core 2024-04-24T07:47:53.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/failover.ko 2024-04-24T07:47:53.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/core/pktgen.ko 2024-04-24T07:47:53.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp 2024-04-24T07:47:53.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp.ko 2024-04-24T07:47:53.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_diag.ko 2024-04-24T07:47:53.954 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv4.ko 2024-04-24T07:47:53.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dccp/dccp_ipv6.ko 2024-04-24T07:47:53.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver 2024-04-24T07:47:53.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dns_resolver/dns_resolver.ko 2024-04-24T07:47:53.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa 2024-04-24T07:47:53.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/dsa_core.ko 2024-04-24T07:47:53.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_brcm.ko 2024-04-24T07:47:53.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_dsa.ko 2024-04-24T07:47:53.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_none.ko 2024-04-24T07:47:53.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/dsa/tag_trailer.ko 2024-04-24T07:47:53.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154 2024-04-24T07:47:53.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan 2024-04-24T07:47:53.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-04-24T07:47:53.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154.ko 2024-04-24T07:47:53.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ieee802154/ieee802154_socket.ko 2024-04-24T07:47:53.955 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4 2024-04-24T07:47:53.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ah4.ko 2024-04-24T07:47:53.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/esp4.ko 2024-04-24T07:47:53.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/fou.ko 2024-04-24T07:47:53.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/gre.ko 2024-04-24T07:47:53.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/inet_diag.ko 2024-04-24T07:47:53.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_gre.ko 2024-04-24T07:47:53.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_tunnel.ko 2024-04-24T07:47:53.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ip_vti.ko 2024-04-24T07:47:53.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipcomp.ko 2024-04-24T07:47:53.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/ipip.ko 2024-04-24T07:47:53.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter 2024-04-24T07:47:53.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arp_tables.ko 2024-04-24T07:47:53.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-04-24T07:47:53.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-04-24T07:47:53.956 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-04-24T07:47:53.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-04-24T07:47:53.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-04-24T07:47:53.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-04-24T07:47:53.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-04-24T07:47:53.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-04-24T07:47:53.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-04-24T07:47:53.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/iptable_security.ko 2024-04-24T07:47:53.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-04-24T07:47:53.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-04-24T07:47:53.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-04-24T07:47:53.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-04-24T07:47:53.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-04-24T07:47:53.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-04-24T07:47:53.957 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-04-24T07:47:53.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-04-24T07:47:53.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-04-24T07:47:53.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-04-24T07:47:53.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_bic.ko 2024-04-24T07:47:53.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_dctcp.ko 2024-04-24T07:47:53.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_diag.ko 2024-04-24T07:47:53.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_highspeed.ko 2024-04-24T07:47:53.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_htcp.ko 2024-04-24T07:47:53.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_hybla.ko 2024-04-24T07:47:53.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_illinois.ko 2024-04-24T07:47:53.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_lp.ko 2024-04-24T07:47:53.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_scalable.ko 2024-04-24T07:47:53.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_vegas.ko 2024-04-24T07:47:53.958 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_veno.ko 2024-04-24T07:47:53.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_westwood.ko 2024-04-24T07:47:53.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tcp_yeah.ko 2024-04-24T07:47:53.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/tunnel4.ko 2024-04-24T07:47:53.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_diag.ko 2024-04-24T07:47:53.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/udp_tunnel.ko 2024-04-24T07:47:53.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv4/xfrm4_tunnel.ko 2024-04-24T07:47:53.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6 2024-04-24T07:47:53.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ah6.ko 2024-04-24T07:47:53.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/esp6.ko 2024-04-24T07:47:53.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/fou6.ko 2024-04-24T07:47:53.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_tunnel.ko 2024-04-24T07:47:53.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-04-24T07:47:53.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ip6_vti.ko 2024-04-24T07:47:53.959 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/ipcomp6.ko 2024-04-24T07:47:53.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter 2024-04-24T07:47:53.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-04-24T07:47:53.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-04-24T07:47:53.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-04-24T07:47:53.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-04-24T07:47:53.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-04-24T07:47:53.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-04-24T07:47:53.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-04-24T07:47:53.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-04-24T07:47:53.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-04-24T07:47:53.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-04-24T07:47:53.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-04-24T07:47:53.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-04-24T07:47:53.960 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-04-24T07:47:53.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-04-24T07:47:53.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-04-24T07:47:53.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-04-24T07:47:53.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-04-24T07:47:53.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-04-24T07:47:53.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-04-24T07:47:53.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-04-24T07:47:53.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-04-24T07:47:53.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-04-24T07:47:53.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-04-24T07:47:53.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-04-24T07:47:53.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/sit.ko 2024-04-24T07:47:53.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/tunnel6.ko 2024-04-24T07:47:53.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/ipv6/xfrm6_tunnel.ko 2024-04-24T07:47:53.961 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key 2024-04-24T07:47:53.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/key/af_key.ko 2024-04-24T07:47:53.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp 2024-04-24T07:47:53.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_core.ko 2024-04-24T07:47:53.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_debugfs.ko 2024-04-24T07:47:53.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_eth.ko 2024-04-24T07:47:53.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip.ko 2024-04-24T07:47:53.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ip6.ko 2024-04-24T07:47:53.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_netlink.ko 2024-04-24T07:47:53.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/l2tp/l2tp_ppp.ko 2024-04-24T07:47:53.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc 2024-04-24T07:47:53.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/llc/llc.ko 2024-04-24T07:47:53.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154 2024-04-24T07:47:53.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mac802154/mac802154.ko 2024-04-24T07:47:53.962 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls 2024-04-24T07:47:53.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/mpls/mpls_gso.ko 2024-04-24T07:47:53.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter 2024-04-24T07:47:53.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset 2024-04-24T07:47:53.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set.ko 2024-04-24T07:47:53.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-04-24T07:47:53.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-04-24T07:47:53.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-04-24T07:47:53.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-04-24T07:47:53.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-04-24T07:47:53.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-04-24T07:47:53.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-04-24T07:47:53.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-04-24T07:47:53.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-04-24T07:47:53.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-04-24T07:47:53.963 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-04-24T07:47:53.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-04-24T07:47:53.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-04-24T07:47:53.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-04-24T07:47:53.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-04-24T07:47:53.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs 2024-04-24T07:47:53.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs.ko 2024-04-24T07:47:53.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-04-24T07:47:53.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-04-24T07:47:53.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-04-24T07:47:53.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-04-24T07:47:53.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-04-24T07:47:53.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-04-24T07:47:53.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-04-24T07:47:53.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-04-24T07:47:53.964 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-04-24T07:47:53.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-04-24T07:47:53.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-04-24T07:47:53.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-04-24T07:47:53.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-04-24T07:47:53.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conncount.ko 2024-04-24T07:47:53.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack.ko 2024-04-24T07:47:53.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-04-24T07:47:53.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-04-24T07:47:53.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-04-24T07:47:53.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_h323.ko 2024-04-24T07:47:53.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_irc.ko 2024-04-24T07:47:53.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-04-24T07:47:53.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-04-24T07:47:53.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-04-24T07:47:53.965 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sane.ko 2024-04-24T07:47:53.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_sip.ko 2024-04-24T07:47:53.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-04-24T07:47:53.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-04-24T07:47:53.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_dup_netdev.ko 2024-04-24T07:47:53.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table.ko 2024-04-24T07:47:53.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_flow_table_inet.ko 2024-04-24T07:47:53.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_log_syslog.ko 2024-04-24T07:47:53.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat.ko 2024-04-24T07:47:53.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_amanda.ko 2024-04-24T07:47:53.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_ftp.ko 2024-04-24T07:47:53.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_irc.ko 2024-04-24T07:47:53.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_sip.ko 2024-04-24T07:47:53.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_nat_tftp.ko 2024-04-24T07:47:53.966 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_synproxy_core.ko 2024-04-24T07:47:53.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nf_tables.ko 2024-04-24T07:47:53.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink.ko 2024-04-24T07:47:53.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_acct.ko 2024-04-24T07:47:53.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_log.ko 2024-04-24T07:47:53.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_osf.ko 2024-04-24T07:47:53.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nfnetlink_queue.ko 2024-04-24T07:47:53.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_chain_nat.ko 2024-04-24T07:47:53.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_compat.ko 2024-04-24T07:47:53.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_ct.ko 2024-04-24T07:47:53.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_dup_netdev.ko 2024-04-24T07:47:53.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fib.ko 2024-04-24T07:47:53.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_fwd_netdev.ko 2024-04-24T07:47:53.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_hash.ko 2024-04-24T07:47:53.967 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_limit.ko 2024-04-24T07:47:53.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_log.ko 2024-04-24T07:47:53.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_masq.ko 2024-04-24T07:47:53.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_nat.ko 2024-04-24T07:47:53.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_numgen.ko 2024-04-24T07:47:53.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_queue.ko 2024-04-24T07:47:53.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_quota.ko 2024-04-24T07:47:53.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_redir.ko 2024-04-24T07:47:53.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject.ko 2024-04-24T07:47:53.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_reject_inet.ko 2024-04-24T07:47:53.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_socket.ko 2024-04-24T07:47:53.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/nft_tunnel.ko 2024-04-24T07:47:53.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_AUDIT.ko 2024-04-24T07:47:53.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CHECKSUM.ko 2024-04-24T07:47:53.968 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CLASSIFY.ko 2024-04-24T07:47:53.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-04-24T07:47:53.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_CT.ko 2024-04-24T07:47:53.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_DSCP.ko 2024-04-24T07:47:53.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HL.ko 2024-04-24T07:47:53.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_HMARK.ko 2024-04-24T07:47:53.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_IDLETIMER.ko 2024-04-24T07:47:53.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LED.ko 2024-04-24T07:47:53.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_LOG.ko 2024-04-24T07:47:53.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_MASQUERADE.ko 2024-04-24T07:47:53.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NETMAP.ko 2024-04-24T07:47:53.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFLOG.ko 2024-04-24T07:47:53.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_NFQUEUE.ko 2024-04-24T07:47:53.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_RATEEST.ko 2024-04-24T07:47:53.969 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_REDIRECT.ko 2024-04-24T07:47:53.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_SECMARK.ko 2024-04-24T07:47:53.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPMSS.ko 2024-04-24T07:47:53.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-04-24T07:47:53.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TEE.ko 2024-04-24T07:47:53.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TPROXY.ko 2024-04-24T07:47:53.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_TRACE.ko 2024-04-24T07:47:53.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_addrtype.ko 2024-04-24T07:47:53.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_bpf.ko 2024-04-24T07:47:53.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cgroup.ko 2024-04-24T07:47:53.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cluster.ko 2024-04-24T07:47:53.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_comment.ko 2024-04-24T07:47:53.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connbytes.ko 2024-04-24T07:47:53.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlabel.ko 2024-04-24T07:47:53.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connlimit.ko 2024-04-24T07:47:53.970 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_connmark.ko 2024-04-24T07:47:53.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_conntrack.ko 2024-04-24T07:47:53.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_cpu.ko 2024-04-24T07:47:53.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dccp.ko 2024-04-24T07:47:53.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_devgroup.ko 2024-04-24T07:47:53.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_dscp.ko 2024-04-24T07:47:53.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ecn.ko 2024-04-24T07:47:53.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_esp.ko 2024-04-24T07:47:53.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hashlimit.ko 2024-04-24T07:47:53.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_helper.ko 2024-04-24T07:47:53.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_hl.ko 2024-04-24T07:47:53.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipcomp.ko 2024-04-24T07:47:53.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_iprange.ko 2024-04-24T07:47:53.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_ipvs.ko 2024-04-24T07:47:53.971 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_l2tp.ko 2024-04-24T07:47:53.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_length.ko 2024-04-24T07:47:53.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_limit.ko 2024-04-24T07:47:53.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mac.ko 2024-04-24T07:47:53.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_mark.ko 2024-04-24T07:47:53.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_multiport.ko 2024-04-24T07:47:53.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nat.ko 2024-04-24T07:47:53.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_nfacct.ko 2024-04-24T07:47:53.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_osf.ko 2024-04-24T07:47:53.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_owner.ko 2024-04-24T07:47:53.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_physdev.ko 2024-04-24T07:47:53.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_pkttype.ko 2024-04-24T07:47:53.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_policy.ko 2024-04-24T07:47:53.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_quota.ko 2024-04-24T07:47:53.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_rateest.ko 2024-04-24T07:47:53.972 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_realm.ko 2024-04-24T07:47:53.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_recent.ko 2024-04-24T07:47:53.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_sctp.ko 2024-04-24T07:47:53.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_set.ko 2024-04-24T07:47:53.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_socket.ko 2024-04-24T07:47:53.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_state.ko 2024-04-24T07:47:53.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_statistic.ko 2024-04-24T07:47:53.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_string.ko 2024-04-24T07:47:53.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_tcpmss.ko 2024-04-24T07:47:53.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_time.ko 2024-04-24T07:47:53.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netfilter/xt_u32.ko 2024-04-24T07:47:53.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink 2024-04-24T07:47:53.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/netlink/netlink_diag.ko 2024-04-24T07:47:53.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh 2024-04-24T07:47:53.973 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/nsh/nsh.ko 2024-04-24T07:47:53.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch 2024-04-24T07:47:53.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/openvswitch.ko 2024-04-24T07:47:53.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-geneve.ko 2024-04-24T07:47:53.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-gre.ko 2024-04-24T07:47:53.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/openvswitch/vport-vxlan.ko 2024-04-24T07:47:53.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet 2024-04-24T07:47:53.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/packet/af_packet_diag.ko 2024-04-24T07:47:53.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds 2024-04-24T07:47:53.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds.ko 2024-04-24T07:47:53.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_rdma.ko 2024-04-24T07:47:53.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/rds/rds_tcp.ko 2024-04-24T07:47:53.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched 2024-04-24T07:47:53.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_csum.ko 2024-04-24T07:47:53.974 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_gact.ko 2024-04-24T07:47:53.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_mirred.ko 2024-04-24T07:47:53.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_nat.ko 2024-04-24T07:47:53.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_pedit.ko 2024-04-24T07:47:53.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_police.ko 2024-04-24T07:47:53.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_simple.ko 2024-04-24T07:47:53.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_skbedit.ko 2024-04-24T07:47:53.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/act_vlan.ko 2024-04-24T07:47:53.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_basic.ko 2024-04-24T07:47:53.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_bpf.ko 2024-04-24T07:47:53.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_flow.ko 2024-04-24T07:47:53.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_fw.ko 2024-04-24T07:47:53.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_route.ko 2024-04-24T07:47:53.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/cls_u32.ko 2024-04-24T07:47:53.975 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_cmp.ko 2024-04-24T07:47:53.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_ipset.ko 2024-04-24T07:47:53.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_meta.ko 2024-04-24T07:47:53.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_nbyte.ko 2024-04-24T07:47:53.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_text.ko 2024-04-24T07:47:53.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/em_u32.ko 2024-04-24T07:47:53.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_choke.ko 2024-04-24T07:47:53.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_codel.ko 2024-04-24T07:47:53.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_drr.ko 2024-04-24T07:47:53.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_fq.ko 2024-04-24T07:47:53.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_gred.ko 2024-04-24T07:47:53.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hfsc.ko 2024-04-24T07:47:53.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_hhf.ko 2024-04-24T07:47:53.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_htb.ko 2024-04-24T07:47:53.976 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_ingress.ko 2024-04-24T07:47:53.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio.ko 2024-04-24T07:47:53.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_mqprio_lib.ko 2024-04-24T07:47:53.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_multiq.ko 2024-04-24T07:47:53.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_netem.ko 2024-04-24T07:47:53.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_pie.ko 2024-04-24T07:47:53.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_plug.ko 2024-04-24T07:47:53.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_prio.ko 2024-04-24T07:47:53.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_qfq.ko 2024-04-24T07:47:53.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_red.ko 2024-04-24T07:47:53.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfb.ko 2024-04-24T07:47:53.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_sfq.ko 2024-04-24T07:47:53.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_tbf.ko 2024-04-24T07:47:53.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sched/sch_teql.ko 2024-04-24T07:47:53.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp 2024-04-24T07:47:53.977 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp.ko 2024-04-24T07:47:53.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sctp/sctp_diag.ko 2024-04-24T07:47:53.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc 2024-04-24T07:47:53.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss 2024-04-24T07:47:53.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-04-24T07:47:53.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-04-24T07:47:53.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/sunrpc.ko 2024-04-24T07:47:53.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma 2024-04-24T07:47:53.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-04-24T07:47:53.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc 2024-04-24T07:47:53.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc.ko 2024-04-24T07:47:53.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/tipc/tipc_diag.ko 2024-04-24T07:47:53.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix 2024-04-24T07:47:53.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/unix/unix_diag.ko 2024-04-24T07:47:53.978 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock 2024-04-24T07:47:53.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-04-24T07:47:53.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-04-24T07:47:53.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock.ko 2024-04-24T07:47:53.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_diag.ko 2024-04-24T07:47:53.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/vmw_vsock/vsock_loopback.ko 2024-04-24T07:47:53.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm 2024-04-24T07:47:53.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/net/xfrm/xfrm_ipcomp.ko 2024-04-24T07:47:53.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security 2024-04-24T07:47:53.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys 2024-04-24T07:47:53.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys 2024-04-24T07:47:53.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-04-24T07:47:53.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys 2024-04-24T07:47:53.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/kernel/security/keys/trusted-keys/trusted.ko 2024-04-24T07:47:53.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias 2024-04-24T07:47:53.979 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.alias.bin 2024-04-24T07:47:53.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin 2024-04-24T07:47:53.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.alias.bin 2024-04-24T07:47:53.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.bin 2024-04-24T07:47:53.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.builtin.modinfo 2024-04-24T07:47:53.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep 2024-04-24T07:47:53.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep.bin 2024-04-24T07:47:53.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.devname 2024-04-24T07:47:53.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.order 2024-04-24T07:47:53.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.softdep 2024-04-24T07:47:53.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols 2024-04-24T07:47:53.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/modules.symbols.bin 2024-04-24T07:47:53.980 INFO:teuthology.orchestra.run.smithi028.stdout:/lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-24T07:47:53.984 DEBUG:teuthology.task.kernel:get_image_version: 6.9.0-rc3-g806ec36b2112 2024-04-24T07:47:53.985 DEBUG:teuthology.orchestra.run.smithi028:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-04-24T07:47:54.008 INFO:teuthology.orchestra.run.smithi028.stdout:postinstall scriptlet (using /bin/sh): 2024-04-24T07:47:54.008 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-24T07:47:54.009 INFO:teuthology.orchestra.run.smithi028.stdout: /usr/bin/kernel-install add 6.9.0-rc3-g806ec36b2112 /lib/modules/6.9.0-rc3-g806ec36b2112/vmlinuz 2024-04-24T07:47:54.009 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-04-24T07:47:54.009 INFO:teuthology.orchestra.run.smithi028.stdout:for file in vmlinuz System.map config; do 2024-04-24T07:47:54.009 INFO:teuthology.orchestra.run.smithi028.stdout: if ! cmp --silent "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112"; then 2024-04-24T07:47:54.009 INFO:teuthology.orchestra.run.smithi028.stdout: cp "/lib/modules/6.9.0-rc3-g806ec36b2112/${file}" "/boot/${file}-6.9.0-rc3-g806ec36b2112" 2024-04-24T07:47:54.009 INFO:teuthology.orchestra.run.smithi028.stdout: fi 2024-04-24T07:47:54.009 INFO:teuthology.orchestra.run.smithi028.stdout:done 2024-04-24T07:47:54.009 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -d "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" ] && \ 2024-04-24T07:47:54.009 INFO:teuthology.orchestra.run.smithi028.stdout: ! diff -rq "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" >/dev/null 2>&1; then 2024-04-24T07:47:54.009 INFO:teuthology.orchestra.run.smithi028.stdout: rm -rf "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-24T07:47:54.009 INFO:teuthology.orchestra.run.smithi028.stdout: cp -r "/lib/modules/6.9.0-rc3-g806ec36b2112/dtb" "/boot/dtb-6.9.0-rc3-g806ec36b2112" 2024-04-24T07:47:54.009 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-04-24T07:47:54.009 INFO:teuthology.orchestra.run.smithi028.stdout:if [ ! -e "/lib/modules/6.9.0-rc3-g806ec36b2112/modules.dep" ]; then 2024-04-24T07:47:54.009 INFO:teuthology.orchestra.run.smithi028.stdout: /usr/sbin/depmod 6.9.0-rc3-g806ec36b2112 2024-04-24T07:47:54.009 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-04-24T07:47:54.010 INFO:teuthology.orchestra.run.smithi028.stdout:preuninstall scriptlet (using /bin/sh): 2024-04-24T07:47:54.010 INFO:teuthology.orchestra.run.smithi028.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-04-24T07:47:54.010 INFO:teuthology.orchestra.run.smithi028.stdout:kernel-install remove 6.9.0-rc3-g806ec36b2112 2024-04-24T07:47:54.010 INFO:teuthology.orchestra.run.smithi028.stdout:fi 2024-04-24T07:47:54.011 DEBUG:teuthology.orchestra.run.smithi028:> sudo rpm -qi grub2-tools 2024-04-24T07:47:54.092 INFO:teuthology.orchestra.run.smithi028.stdout:Name : grub2-tools 2024-04-24T07:47:54.092 INFO:teuthology.orchestra.run.smithi028.stdout:Epoch : 1 2024-04-24T07:47:54.092 INFO:teuthology.orchestra.run.smithi028.stdout:Version : 2.06 2024-04-24T07:47:54.092 INFO:teuthology.orchestra.run.smithi028.stdout:Release : 68.el9 2024-04-24T07:47:54.092 INFO:teuthology.orchestra.run.smithi028.stdout:Architecture: x86_64 2024-04-24T07:47:54.093 INFO:teuthology.orchestra.run.smithi028.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-04-24T07:47:54.093 INFO:teuthology.orchestra.run.smithi028.stdout:Group : Unspecified 2024-04-24T07:47:54.093 INFO:teuthology.orchestra.run.smithi028.stdout:Size : 8264254 2024-04-24T07:47:54.093 INFO:teuthology.orchestra.run.smithi028.stdout:License : GPLv3+ 2024-04-24T07:47:54.093 INFO:teuthology.orchestra.run.smithi028.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-04-24T07:47:54.093 INFO:teuthology.orchestra.run.smithi028.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-04-24T07:47:54.093 INFO:teuthology.orchestra.run.smithi028.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-04-24T07:47:54.093 INFO:teuthology.orchestra.run.smithi028.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-04-24T07:47:54.093 INFO:teuthology.orchestra.run.smithi028.stdout:Packager : builder@centos.org 2024-04-24T07:47:54.093 INFO:teuthology.orchestra.run.smithi028.stdout:Vendor : CentOS 2024-04-24T07:47:54.093 INFO:teuthology.orchestra.run.smithi028.stdout:URL : http://www.gnu.org/software/grub/ 2024-04-24T07:47:54.094 INFO:teuthology.orchestra.run.smithi028.stdout:Summary : Support tools for GRUB. 2024-04-24T07:47:54.094 INFO:teuthology.orchestra.run.smithi028.stdout:Description : 2024-04-24T07:47:54.094 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:47:54.094 INFO:teuthology.orchestra.run.smithi028.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-04-24T07:47:54.094 INFO:teuthology.orchestra.run.smithi028.stdout:customizable bootloader with modular architecture. It supports a rich 2024-04-24T07:47:54.094 INFO:teuthology.orchestra.run.smithi028.stdout:variety of kernel formats, file systems, computer architectures and 2024-04-24T07:47:54.094 INFO:teuthology.orchestra.run.smithi028.stdout:hardware devices. 2024-04-24T07:47:54.094 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:47:54.094 INFO:teuthology.orchestra.run.smithi028.stdout:This subpackage provides tools for support of all platforms. 2024-04-24T07:47:54.095 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-04-24T07:47:54.095 INFO:teuthology.task.kernel:Updating grub on smithi028 to boot 6.9.0-rc3-g806ec36b2112 2024-04-24T07:47:54.095 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-04-24T07:47:54.696 INFO:teuthology.orchestra.run.smithi028.stderr:Generating grub configuration file ... 2024-04-24T07:47:56.102 INFO:teuthology.orchestra.run.smithi028.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-04-24T07:47:56.131 INFO:teuthology.orchestra.run.smithi028.stderr:done 2024-04-24T07:47:56.134 DEBUG:teuthology.orchestra.run.smithi028:> mktemp 2024-04-24T07:47:56.180 INFO:teuthology.orchestra.run.smithi028.stdout:/tmp/tmp.CQuzV1WoQm 2024-04-24T07:47:56.181 DEBUG:teuthology.orchestra.run.smithi028:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.CQuzV1WoQm 2024-04-24T07:47:56.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0666 /tmp/tmp.CQuzV1WoQm 2024-04-24T07:47:56.345 DEBUG:teuthology.orchestra.remote:smithi028:/tmp/tmp.CQuzV1WoQm is 6KB 2024-04-24T07:47:56.393 DEBUG:teuthology.orchestra.run.smithi028:> rm -fr /tmp/tmp.CQuzV1WoQm 2024-04-24T07:47:56.408 DEBUG:teuthology.orchestra.run.smithi028:> sudo /bin/ls /boot/loader/entries || true 2024-04-24T07:47:56.470 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-04-24T07:47:56.470 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-04-24T07:47:56.470 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-04-24T07:47:56.470 INFO:teuthology.orchestra.run.smithi028.stdout:7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112.conf 2024-04-24T07:47:56.471 DEBUG:teuthology.orchestra.run.smithi028:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-6.9.0-rc3-g806ec36b2112 2024-04-24T07:47:56.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo shutdown -r now 2024-04-24T07:48:26.618 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-04-24T07:48:26.619 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-24T07:48:26.620 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:48:44.958 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-04-24T07:48:53.960 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-24T07:48:53.961 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:49:12.350 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.28 2024-04-24T07:49:24.352 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-24T07:49:24.353 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:49:27.527 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-04-24T07:49:28.230 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-24T07:49:28.230 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.9.0-rc3-g806ec36b2112"... 2024-04-24T07:49:28.230 DEBUG:teuthology.orchestra.run.smithi028:> uname -r 2024-04-24T07:49:28.289 INFO:teuthology.orchestra.run.smithi028.stdout:6.9.0-rc3-g806ec36b2112 2024-04-24T07:49:28.289 DEBUG:teuthology.task.kernel:current kernel version is 6.9.0-rc3-g806ec36b2112 vs 6.9.0-rc3-g806ec36b2112 2024-04-24T07:49:28.289 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-04-24T07:49:28.290 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-04-24T07:49:29.291 DEBUG:teuthology.parallel:result is None 2024-04-24T07:49:29.291 INFO:teuthology.run_tasks:Running task internal.base... 2024-04-24T07:49:29.300 INFO:teuthology.task.internal:Creating test directory... 2024-04-24T07:49:29.300 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T07:49:29.306 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T07:49:29.309 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-04-24T07:49:29.346 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-04-24T07:49:29.353 INFO:teuthology.run_tasks:Running task internal.archive... 2024-04-24T07:49:29.378 INFO:teuthology.task.internal:Creating archive directory... 2024-04-24T07:49:29.378 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T07:49:29.391 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T07:49:29.393 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-04-24T07:49:29.550 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-04-24T07:49:29.557 INFO:teuthology.task.internal:Enabling coredump saving... 2024-04-24T07:49:29.557 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T07:49:29.574 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T07:49:29.574 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-04-24T07:49:29.591 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T07:49:29.591 DEBUG:teuthology.orchestra.run.smithi028:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T07:49:29.600 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T07:49:29.617 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-04-24T07:49:29.653 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T07:49:29.668 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T07:49:29.685 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T07:49:29.701 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T07:49:29.722 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T07:49:29.775 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-04-24T07:49:29.779 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-04-24T07:49:29.790 INFO:teuthology.task.internal:Configuring sudo... 2024-04-24T07:49:29.790 DEBUG:teuthology.orchestra.run.smithi028:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T07:49:29.795 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T07:49:29.797 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-04-24T07:49:29.861 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-04-24T07:49:29.871 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-04-24T07:49:29.871 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T07:49:29.879 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T07:49:29.882 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-04-24T07:49:29.918 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T07:49:30.031 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T07:49:30.099 DEBUG:teuthology.orchestra.run.smithi028:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T07:49:30.167 DEBUG:teuthology.orchestra.run.smithi028:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T07:49:30.273 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T07:49:30.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T07:49:30.353 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T07:49:30.399 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T07:49:30.448 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T07:49:30.531 DEBUG:teuthology.orchestra.run.smithi117:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T07:49:30.576 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-04-24T07:49:30.576 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T07:49:30.641 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T07:49:30.685 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-04-24T07:49:30.736 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T07:49:30.819 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-04-24T07:49:30.863 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-24T07:49:30.863 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-04-24T07:49:30.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo service rsyslog restart 2024-04-24T07:49:30.933 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2024-04-24T07:49:30.935 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-04-24T07:49:30.968 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T07:49:31.010 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T07:49:31.018 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T07:49:31.472 INFO:teuthology.run_tasks:Running task internal.timer... 2024-04-24T07:49:31.481 INFO:teuthology.task.internal:Starting timer... 2024-04-24T07:49:31.481 INFO:teuthology.run_tasks:Running task pcp... 2024-04-24T07:49:31.526 INFO:teuthology.run_tasks:Running task selinux... 2024-04-24T07:49:31.571 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-04-24T07:49:31.572 DEBUG:teuthology.orchestra.run.smithi028:> sudo service auditd rotate 2024-04-24T07:49:31.683 INFO:teuthology.orchestra.run.smithi028.stdout:Rotating logs: 2024-04-24T07:49:31.687 DEBUG:teuthology.orchestra.run.smithi117:> sudo service auditd rotate 2024-04-24T07:49:31.774 INFO:teuthology.orchestra.run.smithi117.stdout:Rotating logs: 2024-04-24T07:49:31.775 DEBUG:teuthology.orchestra.run.smithi194:> sudo service auditd rotate 2024-04-24T07:49:31.863 INFO:teuthology.orchestra.run.smithi194.stdout:Rotating logs: 2024-04-24T07:49:31.865 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-04-24T07:49:31.865 DEBUG:teuthology.orchestra.run.smithi028:> /usr/sbin/getenforce 2024-04-24T07:49:31.916 INFO:teuthology.orchestra.run.smithi028.stdout:Permissive 2024-04-24T07:49:31.916 DEBUG:teuthology.orchestra.run.smithi117:> /usr/sbin/getenforce 2024-04-24T07:49:31.937 INFO:teuthology.orchestra.run.smithi117.stdout:Permissive 2024-04-24T07:49:31.937 DEBUG:teuthology.orchestra.run.smithi194:> /usr/sbin/getenforce 2024-04-24T07:49:31.965 INFO:teuthology.orchestra.run.smithi194.stdout:Permissive 2024-04-24T07:49:31.965 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi028.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi117.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi194.front.sepia.ceph.com': 'permissive'} 2024-04-24T07:49:31.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T07:49:32.034 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T07:49:32.035 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T07:49:32.062 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T07:49:32.062 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T07:49:32.089 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T07:49:32.089 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-04-24T07:49:32.089 DEBUG:teuthology.orchestra.run.smithi028:> sudo /usr/sbin/setenforce permissive 2024-04-24T07:49:32.211 DEBUG:teuthology.orchestra.run.smithi117:> sudo /usr/sbin/setenforce permissive 2024-04-24T07:49:32.237 DEBUG:teuthology.orchestra.run.smithi194:> sudo /usr/sbin/setenforce permissive 2024-04-24T07:49:32.263 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-04-24T07:49:32.273 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-04-24T07:49:32.291 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-04-24T07:49:32.297 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi028.front.sepia.ceph.com,smithi117.front.sepia.ceph.com,smithi194.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-04-24T07:54:46.259 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi028.front.sepia.ceph.com'), Remote(name='ubuntu@smithi117.front.sepia.ceph.com'), Remote(name='ubuntu@smithi194.front.sepia.ceph.com')] 2024-04-24T07:54:46.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-24T07:54:46.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi028.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:54:46.370 DEBUG:teuthology.orchestra.run.smithi028:> true 2024-04-24T07:54:46.521 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi028.front.sepia.ceph.com' 2024-04-24T07:54:46.521 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-04-24T07:54:46.522 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:54:46.636 DEBUG:teuthology.orchestra.run.smithi117:> true 2024-04-24T07:54:46.710 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi117.front.sepia.ceph.com' 2024-04-24T07:54:46.710 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-24T07:54:46.711 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-04-24T07:54:46.784 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-04-24T07:54:46.867 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-04-24T07:54:46.867 INFO:teuthology.run_tasks:Running task clock... 2024-04-24T07:54:46.880 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-04-24T07:54:46.880 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T07:54:46.880 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T07:54:46.888 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T07:54:46.888 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T07:54:46.890 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-04-24T07:54:46.890 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T07:54:46.919 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T07:54:46.934 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T07:54:46.938 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T07:54:46.953 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T07:54:46.959 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: ntpd: command not found 2024-04-24T07:54:46.970 INFO:teuthology.orchestra.run.smithi117.stdout:506 Cannot talk to daemon 2024-04-24T07:54:46.977 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: ntpd: command not found 2024-04-24T07:54:46.982 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-04-24T07:54:46.983 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T07:54:46.988 INFO:teuthology.orchestra.run.smithi194.stdout:506 Cannot talk to daemon 2024-04-24T07:54:46.997 INFO:teuthology.orchestra.run.smithi117.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T07:54:47.002 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T07:54:47.015 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T07:54:47.029 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-04-24T07:54:47.057 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-04-24T07:54:47.060 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T07:54:47.060 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-04-24T07:54:47.060 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T07:54:47.060 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T07:54:47.060 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T07:54:47.060 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T07:54:47.072 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-04-24T07:54:47.076 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T07:54:47.076 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-04-24T07:54:47.076 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T07:54:47.076 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T07:54:47.076 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T07:54:47.076 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T07:54:47.113 INFO:teuthology.orchestra.run.smithi028.stderr:sudo: ntpd: command not found 2024-04-24T07:54:47.148 INFO:teuthology.orchestra.run.smithi028.stdout:506 Cannot talk to daemon 2024-04-24T07:54:47.191 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-04-24T07:54:47.236 INFO:teuthology.orchestra.run.smithi028.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-04-24T07:54:47.381 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-04-24T07:54:47.391 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T07:54:47.391 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-04-24T07:54:47.391 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T07:54:47.392 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T07:54:47.392 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T07:54:47.392 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T07:54:47.393 INFO:teuthology.run_tasks:Running task install... 2024-04-24T07:54:47.401 DEBUG:teuthology.task.install:project ceph 2024-04-24T07:54:47.401 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0'}} 2024-04-24T07:54:47.401 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0'} 2024-04-24T07:54:47.402 INFO:teuthology.task.install:Using flavor: default 2024-04-24T07:54:47.407 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-04-24T07:54:47.407 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-04-24T07:54:47.408 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0', 'tag': None, 'wait_for_package': False} 2024-04-24T07:54:47.408 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T07:54:47.410 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0', 'tag': None, 'wait_for_package': False} 2024-04-24T07:54:47.410 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T07:54:47.411 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0', 'tag': None, 'wait_for_package': False} 2024-04-24T07:54:47.411 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T07:54:47.554 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/ 2024-04-24T07:54:47.554 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3116.g1f81fb39 2024-04-24T07:54:47.569 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/ 2024-04-24T07:54:47.569 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3116.g1f81fb39 2024-04-24T07:54:47.675 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/ 2024-04-24T07:54:47.675 INFO:teuthology.task.install.rpm:Package version is 19.0.0-3116.g1f81fb39 2024-04-24T07:54:47.763 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T07:54:47.763 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-04-24T07:54:47.763 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T07:54:47.784 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T07:54:47.784 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T07:54:47.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T07:54:47.789 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-24T07:54:47.789 DEBUG:teuthology.orchestra.run.smithi117:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1f81fb397ae98da7563d451a78c61574c8f4e6e0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T07:54:47.854 DEBUG:teuthology.orchestra.run.smithi117:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T07:54:47.870 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-24T07:54:47.870 DEBUG:teuthology.orchestra.run.smithi028:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1f81fb397ae98da7563d451a78c61574c8f4e6e0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T07:54:47.928 DEBUG:teuthology.orchestra.run.smithi117:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T07:54:47.991 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-04-24T07:54:47.991 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-24T07:54:47.991 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-04-24T07:54:47.992 INFO:teuthology.orchestra.run.smithi117.stdout:check_obsoletes = 1 2024-04-24T07:54:47.993 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum clean all 2024-04-24T07:54:48.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T07:54:48.023 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-04-24T07:54:48.023 DEBUG:teuthology.orchestra.run.smithi194:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/1f81fb397ae98da7563d451a78c61574c8f4e6e0/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-04-24T07:54:48.093 DEBUG:teuthology.orchestra.run.smithi194:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-04-24T07:54:48.119 DEBUG:teuthology.orchestra.run.smithi028:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T07:54:48.171 DEBUG:teuthology.orchestra.run.smithi194:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-04-24T07:54:48.202 INFO:teuthology.orchestra.run.smithi028.stdout:check_obsoletes = 1 2024-04-24T07:54:48.206 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum clean all 2024-04-24T07:54:48.215 INFO:teuthology.orchestra.run.smithi117.stdout:48 files removed 2024-04-24T07:54:48.236 INFO:teuthology.orchestra.run.smithi194.stdout:check_obsoletes = 1 2024-04-24T07:54:48.238 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum clean all 2024-04-24T07:54:48.242 DEBUG:teuthology.orchestra.run.smithi117:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-24T07:54:48.468 INFO:teuthology.orchestra.run.smithi194.stdout:48 files removed 2024-04-24T07:54:48.496 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-24T07:54:48.542 INFO:teuthology.orchestra.run.smithi028.stdout:48 files removed 2024-04-24T07:54:48.581 DEBUG:teuthology.orchestra.run.smithi028:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-04-24T07:54:49.203 INFO:teuthology.orchestra.run.smithi117.stdout:ceph packages for x86_64 119 kB/s | 83 kB 00:00 2024-04-24T07:54:49.272 INFO:teuthology.orchestra.run.smithi194.stdout:ceph packages for x86_64 161 kB/s | 83 kB 00:00 2024-04-24T07:54:49.619 INFO:teuthology.orchestra.run.smithi028.stdout:ceph packages for x86_64 138 kB/s | 83 kB 00:00 2024-04-24T07:54:49.864 INFO:teuthology.orchestra.run.smithi194.stdout:ceph noarch packages 21 kB/s | 12 kB 00:00 2024-04-24T07:54:50.212 INFO:teuthology.orchestra.run.smithi028.stdout:ceph noarch packages 24 kB/s | 12 kB 00:00 2024-04-24T07:54:50.273 INFO:teuthology.orchestra.run.smithi194.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-04-24T07:54:50.746 INFO:teuthology.orchestra.run.smithi117.stdout:ceph noarch packages 7.7 kB/s | 12 kB 00:01 2024-04-24T07:54:50.971 INFO:teuthology.orchestra.run.smithi028.stdout:ceph source packages 2.8 kB/s | 1.9 kB 00:00 2024-04-24T07:54:51.073 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - BaseOS 10 MB/s | 8.0 MB 00:00 2024-04-24T07:54:51.113 INFO:teuthology.orchestra.run.smithi117.stdout:ceph source packages 5.8 kB/s | 1.9 kB 00:00 2024-04-24T07:54:52.496 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - BaseOS 6.0 MB/s | 8.0 MB 00:01 2024-04-24T07:54:54.088 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - AppStream 34 MB/s | 19 MB 00:00 2024-04-24T07:54:55.016 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - AppStream 6.6 MB/s | 19 MB 00:02 2024-04-24T07:54:58.023 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.2 MB 00:00 2024-04-24T07:54:59.183 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.2 MB 00:00 2024-04-24T07:54:59.623 INFO:teuthology.orchestra.run.smithi117.stdout:CentOS Stream 9 - Extras packages 35 kB/s | 16 kB 00:00 2024-04-24T07:55:00.677 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - BaseOS 857 kB/s | 8.0 MB 00:09 2024-04-24T07:55:00.809 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 16 kB 00:00 2024-04-24T07:55:01.185 INFO:teuthology.orchestra.run.smithi117.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-04-24T07:55:02.685 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 12 MB/s | 21 MB 00:01 2024-04-24T07:55:06.483 INFO:teuthology.orchestra.run.smithi117.stdout:lab-extras 5.7 kB/s | 1.7 kB 00:00 2024-04-24T07:55:07.693 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-04-24T07:55:08.583 INFO:teuthology.orchestra.run.smithi117.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T07:55:08.583 INFO:teuthology.orchestra.run.smithi117.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T07:55:08.594 INFO:teuthology.orchestra.run.smithi117.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-24T07:55:08.595 INFO:teuthology.orchestra.run.smithi117.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-24T07:55:08.595 INFO:teuthology.orchestra.run.smithi117.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-24T07:55:08.596 INFO:teuthology.orchestra.run.smithi117.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-24T07:55:08.598 INFO:teuthology.orchestra.run.smithi117.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-24T07:55:08.600 INFO:teuthology.orchestra.run.smithi117.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-24T07:55:08.601 INFO:teuthology.orchestra.run.smithi117.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-24T07:55:08.661 INFO:teuthology.orchestra.run.smithi117.stdout:Dependencies resolved. 2024-04-24T07:55:08.683 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================= 2024-04-24T07:55:08.683 INFO:teuthology.orchestra.run.smithi117.stdout: Package Arch Version Repository Size 2024-04-24T07:55:08.683 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================= 2024-04-24T07:55:08.683 INFO:teuthology.orchestra.run.smithi117.stdout:Installing: 2024-04-24T07:55:08.683 INFO:teuthology.orchestra.run.smithi117.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-24T07:55:08.683 INFO:teuthology.orchestra.run.smithi117.stdout: ceph x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 6.6 k 2024-04-24T07:55:08.683 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 5.4 M 2024-04-24T07:55:08.683 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 890 k 2024-04-24T07:55:08.683 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 145 k 2024-04-24T07:55:08.683 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 1.5 M 2024-04-24T07:55:08.683 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 142 k 2024-04-24T07:55:08.684 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 3.5 M 2024-04-24T07:55:08.684 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 7.4 M 2024-04-24T07:55:08.684 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 49 k 2024-04-24T07:55:08.684 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 11 M 2024-04-24T07:55:08.684 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 49 M 2024-04-24T07:55:08.684 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 273 k 2024-04-24T07:55:08.684 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 762 k 2024-04-24T07:55:08.684 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-mirror x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 225 k 2024-04-24T07:55:08.684 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-shell x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 27 k 2024-04-24T07:55:08.684 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-top noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 18 k 2024-04-24T07:55:08.684 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-24T07:55:08.684 INFO:teuthology.orchestra.run.smithi117.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-24T07:55:08.684 INFO:teuthology.orchestra.run.smithi117.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-24T07:55:08.685 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 32 k 2024-04-24T07:55:08.685 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 746 k 2024-04-24T07:55:08.685 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 127 k 2024-04-24T07:55:08.685 INFO:teuthology.orchestra.run.smithi117.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-24T07:55:08.685 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-24T07:55:08.685 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-24T07:55:08.685 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-24T07:55:08.685 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-24T07:55:08.685 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-24T07:55:08.685 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 161 k 2024-04-24T07:55:08.685 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 320 k 2024-04-24T07:55:08.685 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 298 k 2024-04-24T07:55:08.686 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 99 k 2024-04-24T07:55:08.686 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 85 k 2024-04-24T07:55:08.686 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.1 M 2024-04-24T07:55:08.686 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 170 k 2024-04-24T07:55:08.686 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-24T07:55:08.686 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrading: 2024-04-24T07:55:08.686 INFO:teuthology.orchestra.run.smithi117.stdout: librados2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.4 M 2024-04-24T07:55:08.686 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.1 M 2024-04-24T07:55:08.686 INFO:teuthology.orchestra.run.smithi117.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-24T07:55:08.686 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-24T07:55:08.686 INFO:teuthology.orchestra.run.smithi117.stdout:Installing dependencies: 2024-04-24T07:55:08.686 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-24T07:55:08.686 INFO:teuthology.orchestra.run.smithi117.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-24T07:55:08.687 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-24T07:55:08.687 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 21 M 2024-04-24T07:55:08.687 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 34 k 2024-04-24T07:55:08.687 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 2.4 M 2024-04-24T07:55:08.687 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 249 k 2024-04-24T07:55:08.687 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 4.7 M 2024-04-24T07:55:08.687 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 17 M 2024-04-24T07:55:08.687 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 16 k 2024-04-24T07:55:08.687 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 25 k 2024-04-24T07:55:08.687 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-24T07:55:08.687 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-24T07:55:08.687 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-24T07:55:08.687 INFO:teuthology.orchestra.run.smithi117.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-24T07:55:08.687 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-24T07:55:08.687 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-24T07:55:08.688 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-24T07:55:08.688 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-24T07:55:08.688 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 163 k 2024-04-24T07:55:08.688 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-24T07:55:08.688 INFO:teuthology.orchestra.run.smithi117.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-24T07:55:08.688 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-24T07:55:08.688 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-24T07:55:08.688 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-24T07:55:08.688 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 501 k 2024-04-24T07:55:08.688 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-24T07:55:08.688 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 5.4 M 2024-04-24T07:55:08.688 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-24T07:55:08.689 INFO:teuthology.orchestra.run.smithi117.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-24T07:55:08.689 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-24T07:55:08.689 INFO:teuthology.orchestra.run.smithi117.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-24T07:55:08.689 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-24T07:55:08.689 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-24T07:55:08.689 INFO:teuthology.orchestra.run.smithi117.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-24T07:55:08.689 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-24T07:55:08.689 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-24T07:55:08.689 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-24T07:55:08.689 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-24T07:55:08.689 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-24T07:55:08.689 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-24T07:55:08.689 INFO:teuthology.orchestra.run.smithi117.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-24T07:55:08.689 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-24T07:55:08.690 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-24T07:55:08.690 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-24T07:55:08.690 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 45 k 2024-04-24T07:55:08.690 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 135 k 2024-04-24T07:55:08.690 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-24T07:55:08.690 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-24T07:55:08.690 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-24T07:55:08.690 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-24T07:55:08.690 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-24T07:55:08.690 INFO:teuthology.orchestra.run.smithi117.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-24T07:55:08.690 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-24T07:55:08.691 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-24T07:55:08.691 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-24T07:55:08.691 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-24T07:55:08.691 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-24T07:55:08.691 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-24T07:55:08.691 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-24T07:55:08.691 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-24T07:55:08.691 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-24T07:55:08.691 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-24T07:55:08.691 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-24T07:55:08.691 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-24T07:55:08.691 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-24T07:55:08.691 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-24T07:55:08.692 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-24T07:55:08.692 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-24T07:55:08.692 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-24T07:55:08.692 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-24T07:55:08.692 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-24T07:55:08.692 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-24T07:55:08.692 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-24T07:55:08.692 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-24T07:55:08.692 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-24T07:55:08.692 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-24T07:55:08.692 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-24T07:55:08.692 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-24T07:55:08.692 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-24T07:55:08.692 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-24T07:55:08.693 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-24T07:55:08.693 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-24T07:55:08.693 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-24T07:55:08.693 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-24T07:55:08.693 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-24T07:55:08.693 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-24T07:55:08.693 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-24T07:55:08.693 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-24T07:55:08.693 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-24T07:55:08.693 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-24T07:55:08.693 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-24T07:55:08.693 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-24T07:55:08.693 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-24T07:55:08.694 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-24T07:55:08.694 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-24T07:55:08.694 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-24T07:55:08.694 INFO:teuthology.orchestra.run.smithi117.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-24T07:55:08.694 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-24T07:55:08.694 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-24T07:55:08.694 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-24T07:55:08.694 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-24T07:55:08.694 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-24T07:55:08.694 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-24T07:55:08.694 INFO:teuthology.orchestra.run.smithi117.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-24T07:55:08.694 INFO:teuthology.orchestra.run.smithi117.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-24T07:55:08.694 INFO:teuthology.orchestra.run.smithi117.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-24T07:55:08.695 INFO:teuthology.orchestra.run.smithi117.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-24T07:55:08.695 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-24T07:55:08.695 INFO:teuthology.orchestra.run.smithi117.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-24T07:55:08.695 INFO:teuthology.orchestra.run.smithi117.stdout:Installing weak dependencies: 2024-04-24T07:55:08.695 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-24T07:55:08.695 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-24T07:55:08.695 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction Summary 2024-04-24T07:55:08.695 INFO:teuthology.orchestra.run.smithi117.stdout:======================================================================================= 2024-04-24T07:55:08.695 INFO:teuthology.orchestra.run.smithi117.stdout:Install 148 Packages 2024-04-24T07:55:08.695 INFO:teuthology.orchestra.run.smithi117.stdout:Upgrade 4 Packages 2024-04-24T07:55:08.695 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-24T07:55:08.696 INFO:teuthology.orchestra.run.smithi117.stdout:Total download size: 222 M 2024-04-24T07:55:08.696 INFO:teuthology.orchestra.run.smithi117.stdout:Downloading Packages: 2024-04-24T07:55:09.811 INFO:teuthology.orchestra.run.smithi194.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T07:55:09.812 INFO:teuthology.orchestra.run.smithi194.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T07:55:09.822 INFO:teuthology.orchestra.run.smithi194.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-24T07:55:09.823 INFO:teuthology.orchestra.run.smithi194.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-24T07:55:09.824 INFO:teuthology.orchestra.run.smithi194.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-24T07:55:09.824 INFO:teuthology.orchestra.run.smithi194.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-24T07:55:09.826 INFO:teuthology.orchestra.run.smithi194.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-24T07:55:09.828 INFO:teuthology.orchestra.run.smithi194.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-24T07:55:09.829 INFO:teuthology.orchestra.run.smithi194.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-24T07:55:09.888 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-04-24T07:55:09.905 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-04-24T07:55:09.905 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-04-24T07:55:09.905 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-04-24T07:55:09.905 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-04-24T07:55:09.906 INFO:teuthology.orchestra.run.smithi194.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-24T07:55:09.906 INFO:teuthology.orchestra.run.smithi194.stdout: ceph x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 6.6 k 2024-04-24T07:55:09.906 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 5.4 M 2024-04-24T07:55:09.906 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 890 k 2024-04-24T07:55:09.906 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 145 k 2024-04-24T07:55:09.906 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 1.5 M 2024-04-24T07:55:09.906 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 142 k 2024-04-24T07:55:09.906 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 3.5 M 2024-04-24T07:55:09.906 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 7.4 M 2024-04-24T07:55:09.906 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 49 k 2024-04-24T07:55:09.906 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 11 M 2024-04-24T07:55:09.906 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 49 M 2024-04-24T07:55:09.906 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 273 k 2024-04-24T07:55:09.907 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 762 k 2024-04-24T07:55:09.907 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-mirror x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 225 k 2024-04-24T07:55:09.907 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-shell x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 27 k 2024-04-24T07:55:09.907 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-top noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 18 k 2024-04-24T07:55:09.907 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-24T07:55:09.907 INFO:teuthology.orchestra.run.smithi194.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-24T07:55:09.907 INFO:teuthology.orchestra.run.smithi194.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-24T07:55:09.907 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 32 k 2024-04-24T07:55:09.907 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 746 k 2024-04-24T07:55:09.907 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 127 k 2024-04-24T07:55:09.908 INFO:teuthology.orchestra.run.smithi194.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-24T07:55:09.908 INFO:teuthology.orchestra.run.smithi194.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-24T07:55:09.908 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-24T07:55:09.908 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-24T07:55:09.908 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-24T07:55:09.908 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-24T07:55:09.908 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 161 k 2024-04-24T07:55:09.908 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 320 k 2024-04-24T07:55:09.908 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 298 k 2024-04-24T07:55:09.908 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 99 k 2024-04-24T07:55:09.908 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 85 k 2024-04-24T07:55:09.908 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.1 M 2024-04-24T07:55:09.908 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 170 k 2024-04-24T07:55:09.909 INFO:teuthology.orchestra.run.smithi194.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-24T07:55:09.909 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrading: 2024-04-24T07:55:09.909 INFO:teuthology.orchestra.run.smithi194.stdout: librados2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.4 M 2024-04-24T07:55:09.909 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.1 M 2024-04-24T07:55:09.909 INFO:teuthology.orchestra.run.smithi194.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-24T07:55:09.909 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-24T07:55:09.909 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-04-24T07:55:09.909 INFO:teuthology.orchestra.run.smithi194.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-24T07:55:09.909 INFO:teuthology.orchestra.run.smithi194.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-24T07:55:09.909 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-24T07:55:09.909 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 21 M 2024-04-24T07:55:09.909 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 34 k 2024-04-24T07:55:09.909 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 2.4 M 2024-04-24T07:55:09.909 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 249 k 2024-04-24T07:55:09.910 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 4.7 M 2024-04-24T07:55:09.910 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 17 M 2024-04-24T07:55:09.910 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 16 k 2024-04-24T07:55:09.910 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 25 k 2024-04-24T07:55:09.910 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-24T07:55:09.910 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-24T07:55:09.910 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-24T07:55:09.910 INFO:teuthology.orchestra.run.smithi194.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-24T07:55:09.910 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-24T07:55:09.910 INFO:teuthology.orchestra.run.smithi194.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-24T07:55:09.910 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-24T07:55:09.910 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-24T07:55:09.910 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 163 k 2024-04-24T07:55:09.911 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-24T07:55:09.911 INFO:teuthology.orchestra.run.smithi194.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-24T07:55:09.911 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-24T07:55:09.911 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-24T07:55:09.911 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-24T07:55:09.911 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 501 k 2024-04-24T07:55:09.911 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-24T07:55:09.911 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 5.4 M 2024-04-24T07:55:09.911 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-24T07:55:09.911 INFO:teuthology.orchestra.run.smithi194.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-24T07:55:09.911 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-24T07:55:09.911 INFO:teuthology.orchestra.run.smithi194.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-24T07:55:09.911 INFO:teuthology.orchestra.run.smithi194.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-24T07:55:09.911 INFO:teuthology.orchestra.run.smithi194.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-24T07:55:09.912 INFO:teuthology.orchestra.run.smithi194.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-24T07:55:09.912 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-24T07:55:09.912 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-24T07:55:09.912 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-24T07:55:09.912 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-24T07:55:09.912 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-24T07:55:09.912 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-24T07:55:09.912 INFO:teuthology.orchestra.run.smithi194.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-24T07:55:09.912 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-24T07:55:09.912 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-24T07:55:09.912 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-24T07:55:09.912 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 45 k 2024-04-24T07:55:09.912 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 135 k 2024-04-24T07:55:09.912 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-24T07:55:09.913 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-24T07:55:09.913 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-24T07:55:09.913 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-24T07:55:09.913 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-24T07:55:09.913 INFO:teuthology.orchestra.run.smithi194.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-24T07:55:09.913 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-24T07:55:09.913 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-24T07:55:09.913 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-24T07:55:09.913 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-24T07:55:09.913 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-24T07:55:09.913 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-24T07:55:09.913 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-24T07:55:09.914 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-24T07:55:09.914 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-24T07:55:09.914 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-24T07:55:09.914 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-24T07:55:09.914 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-24T07:55:09.914 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-24T07:55:09.914 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-24T07:55:09.914 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-24T07:55:09.914 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-24T07:55:09.914 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-24T07:55:09.914 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-24T07:55:09.914 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-24T07:55:09.914 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-24T07:55:09.914 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-24T07:55:09.914 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-24T07:55:09.915 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-24T07:55:09.915 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-24T07:55:09.915 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-24T07:55:09.915 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-24T07:55:09.915 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-24T07:55:09.915 INFO:teuthology.orchestra.run.smithi194.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-24T07:55:09.915 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-24T07:55:09.915 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-24T07:55:09.915 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-24T07:55:09.915 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-24T07:55:09.915 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-24T07:55:09.915 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-24T07:55:09.915 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-24T07:55:09.915 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-24T07:55:09.915 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-24T07:55:09.915 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-24T07:55:09.916 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-24T07:55:09.916 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-24T07:55:09.916 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-24T07:55:09.916 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-24T07:55:09.916 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-24T07:55:09.916 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-24T07:55:09.916 INFO:teuthology.orchestra.run.smithi194.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-24T07:55:09.916 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-24T07:55:09.916 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-24T07:55:09.916 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-24T07:55:09.916 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-24T07:55:09.916 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-24T07:55:09.916 INFO:teuthology.orchestra.run.smithi194.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-24T07:55:09.916 INFO:teuthology.orchestra.run.smithi194.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-24T07:55:09.916 INFO:teuthology.orchestra.run.smithi194.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-24T07:55:09.916 INFO:teuthology.orchestra.run.smithi194.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-24T07:55:09.916 INFO:teuthology.orchestra.run.smithi194.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-24T07:55:09.917 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-24T07:55:09.917 INFO:teuthology.orchestra.run.smithi194.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-24T07:55:09.917 INFO:teuthology.orchestra.run.smithi194.stdout:Installing weak dependencies: 2024-04-24T07:55:09.917 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-24T07:55:09.917 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T07:55:09.917 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-04-24T07:55:09.917 INFO:teuthology.orchestra.run.smithi194.stdout:======================================================================================= 2024-04-24T07:55:09.917 INFO:teuthology.orchestra.run.smithi194.stdout:Install 148 Packages 2024-04-24T07:55:09.917 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrade 4 Packages 2024-04-24T07:55:09.917 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T07:55:09.918 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 222 M 2024-04-24T07:55:09.918 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-04-24T07:55:10.267 INFO:teuthology.orchestra.run.smithi117.stdout:(1/152): ceph-19.0.0-3116.g1f81fb39.el9.x86_64. 25 kB/s | 6.6 kB 00:00 2024-04-24T07:55:10.442 INFO:teuthology.orchestra.run.smithi117.stdout:(2/152): ceph-fuse-19.0.0-3116.g1f81fb39.el9.x8 5.0 MB/s | 890 kB 00:00 2024-04-24T07:55:10.492 INFO:teuthology.orchestra.run.smithi117.stdout:(3/152): ceph-immutable-object-cache-19.0.0-311 2.8 MB/s | 145 kB 00:00 2024-04-24T07:55:10.792 INFO:teuthology.orchestra.run.smithi117.stdout:(4/152): ceph-mds-19.0.0-3116.g1f81fb39.el9.x86 7.9 MB/s | 2.4 MB 00:00 2024-04-24T07:55:11.034 INFO:teuthology.orchestra.run.smithi117.stdout:(5/152): ceph-base-19.0.0-3116.g1f81fb39.el9.x8 5.2 MB/s | 5.4 MB 00:01 2024-04-24T07:55:11.076 INFO:teuthology.orchestra.run.smithi117.stdout:(6/152): ceph-mgr-19.0.0-3116.g1f81fb39.el9.x86 5.3 MB/s | 1.5 MB 00:00 2024-04-24T07:55:11.352 INFO:teuthology.orchestra.run.smithi194.stdout:(1/152): ceph-19.0.0-3116.g1f81fb39.el9.x86_64. 18 kB/s | 6.6 kB 00:00 2024-04-24T07:55:11.559 INFO:teuthology.orchestra.run.smithi117.stdout:(7/152): ceph-mon-19.0.0-3116.g1f81fb39.el9.x86 8.9 MB/s | 4.7 MB 00:00 2024-04-24T07:55:12.578 INFO:teuthology.orchestra.run.smithi194.stdout:(2/152): ceph-fuse-19.0.0-3116.g1f81fb39.el9.x8 727 kB/s | 890 kB 00:01 2024-04-24T07:55:12.753 INFO:teuthology.orchestra.run.smithi194.stdout:(3/152): ceph-immutable-object-cache-19.0.0-311 828 kB/s | 145 kB 00:00 2024-04-24T07:55:12.818 INFO:teuthology.orchestra.run.smithi117.stdout:(8/152): ceph-radosgw-19.0.0-3116.g1f81fb39.el9 8.4 MB/s | 11 MB 00:01 2024-04-24T07:55:12.901 INFO:teuthology.orchestra.run.smithi117.stdout:(9/152): ceph-selinux-19.0.0-3116.g1f81fb39.el9 302 kB/s | 25 kB 00:00 2024-04-24T07:55:13.320 INFO:teuthology.orchestra.run.smithi117.stdout:(10/152): ceph-osd-19.0.0-3116.g1f81fb39.el9.x8 7.5 MB/s | 17 MB 00:02 2024-04-24T07:55:13.695 INFO:teuthology.orchestra.run.smithi117.stdout:(11/152): cephfs-mirror-19.0.0-3116.g1f81fb39.e 600 kB/s | 225 kB 00:00 2024-04-24T07:55:13.779 INFO:teuthology.orchestra.run.smithi117.stdout:(12/152): cephfs-shell-19.0.0-3116.g1f81fb39.el 324 kB/s | 27 kB 00:00 2024-04-24T07:55:13.863 INFO:teuthology.orchestra.run.smithi117.stdout:(13/152): libcephfs-devel-19.0.0-3116.g1f81fb39 381 kB/s | 32 kB 00:00 2024-04-24T07:55:14.128 INFO:teuthology.orchestra.run.smithi194.stdout:(4/152): ceph-base-19.0.0-3116.g1f81fb39.el9.x8 1.7 MB/s | 5.4 MB 00:03 2024-04-24T07:55:14.171 INFO:teuthology.orchestra.run.smithi117.stdout:(14/152): ceph-common-19.0.0-3116.g1f81fb39.el9 5.1 MB/s | 21 MB 00:04 2024-04-24T07:55:14.196 INFO:teuthology.orchestra.run.smithi117.stdout:(15/152): libcephfs2-19.0.0-3116.g1f81fb39.el9. 2.2 MB/s | 746 kB 00:00 2024-04-24T07:55:14.228 INFO:teuthology.orchestra.run.smithi194.stdout:(5/152): ceph-mds-19.0.0-3116.g1f81fb39.el9.x86 1.6 MB/s | 2.4 MB 00:01 2024-04-24T07:55:14.255 INFO:teuthology.orchestra.run.smithi117.stdout:(16/152): librados-devel-19.0.0-3116.g1f81fb39. 2.1 MB/s | 127 kB 00:00 2024-04-24T07:55:14.281 INFO:teuthology.orchestra.run.smithi117.stdout:(17/152): libcephsqlite-19.0.0-3116.g1f81fb39.e 1.5 MB/s | 163 kB 00:00 2024-04-24T07:55:14.406 INFO:teuthology.orchestra.run.smithi117.stdout:(18/152): libradosstriper1-19.0.0-3116.g1f81fb3 3.3 MB/s | 501 kB 00:00 2024-04-24T07:55:14.498 INFO:teuthology.orchestra.run.smithi117.stdout:(19/152): python3-ceph-argparse-19.0.0-3116.g1f 493 kB/s | 45 kB 00:00 2024-04-24T07:55:14.582 INFO:teuthology.orchestra.run.smithi117.stdout:(20/152): python3-ceph-common-19.0.0-3116.g1f81 1.6 MB/s | 135 kB 00:00 2024-04-24T07:55:14.674 INFO:teuthology.orchestra.run.smithi117.stdout:(21/152): python3-cephfs-19.0.0-3116.g1f81fb39. 1.7 MB/s | 161 kB 00:00 2024-04-24T07:55:14.774 INFO:teuthology.orchestra.run.smithi117.stdout:(22/152): python3-rados-19.0.0-3116.g1f81fb39.e 3.1 MB/s | 320 kB 00:00 2024-04-24T07:55:14.845 INFO:teuthology.orchestra.run.smithi194.stdout:(6/152): ceph-mgr-19.0.0-3116.g1f81fb39.el9.x86 2.1 MB/s | 1.5 MB 00:00 2024-04-24T07:55:14.874 INFO:teuthology.orchestra.run.smithi117.stdout:(23/152): python3-rbd-19.0.0-3116.g1f81fb39.el9 2.9 MB/s | 298 kB 00:00 2024-04-24T07:55:14.966 INFO:teuthology.orchestra.run.smithi117.stdout:(24/152): python3-rgw-19.0.0-3116.g1f81fb39.el9 1.1 MB/s | 99 kB 00:00 2024-04-24T07:55:15.058 INFO:teuthology.orchestra.run.smithi117.stdout:(25/152): rbd-fuse-19.0.0-3116.g1f81fb39.el9.x8 932 kB/s | 85 kB 00:00 2024-04-24T07:55:15.441 INFO:teuthology.orchestra.run.smithi117.stdout:(26/152): librgw2-19.0.0-3116.g1f81fb39.el9.x86 4.6 MB/s | 5.4 MB 00:01 2024-04-24T07:55:15.550 INFO:teuthology.orchestra.run.smithi117.stdout:(27/152): rbd-nbd-19.0.0-3116.g1f81fb39.el9.x86 1.5 MB/s | 170 kB 00:00 2024-04-24T07:55:15.642 INFO:teuthology.orchestra.run.smithi117.stdout:(28/152): rbd-mirror-19.0.0-3116.g1f81fb39.el9. 5.3 MB/s | 3.1 MB 00:00 2024-04-24T07:55:15.659 INFO:teuthology.orchestra.run.smithi117.stdout:(29/152): ceph-grafana-dashboards-19.0.0-3116.g 314 kB/s | 34 kB 00:00 2024-04-24T07:55:15.717 INFO:teuthology.orchestra.run.smithi117.stdout:(30/152): ceph-mgr-cephadm-19.0.0-3116.g1f81fb3 1.9 MB/s | 142 kB 00:00 2024-04-24T07:55:17.070 INFO:teuthology.orchestra.run.smithi194.stdout:(7/152): ceph-mon-19.0.0-3116.g1f81fb39.el9.x86 1.7 MB/s | 4.7 MB 00:02 2024-04-24T07:55:17.126 INFO:teuthology.orchestra.run.smithi117.stdout:(31/152): ceph-mgr-diskprediction-local-19.0.0- 5.3 MB/s | 7.4 MB 00:01 2024-04-24T07:55:17.810 INFO:teuthology.orchestra.run.smithi117.stdout:(32/152): ceph-test-19.0.0-3116.g1f81fb39.el9.x 10 MB/s | 49 MB 00:04 2024-04-24T07:55:17.868 INFO:teuthology.orchestra.run.smithi117.stdout:(33/152): ceph-mgr-dashboard-19.0.0-3116.g1f81f 1.6 MB/s | 3.5 MB 00:02 2024-04-24T07:55:17.893 INFO:teuthology.orchestra.run.smithi117.stdout:(34/152): ceph-mgr-rook-19.0.0-3116.g1f81fb39.e 594 kB/s | 49 kB 00:00 2024-04-24T07:55:17.918 INFO:teuthology.orchestra.run.smithi117.stdout:(35/152): ceph-mgr-modules-core-19.0.0-3116.g1f 314 kB/s | 249 kB 00:00 2024-04-24T07:55:17.952 INFO:teuthology.orchestra.run.smithi117.stdout:(36/152): ceph-volume-19.0.0-3116.g1f81fb39.el9 4.6 MB/s | 273 kB 00:00 2024-04-24T07:55:18.002 INFO:teuthology.orchestra.run.smithi117.stdout:(37/152): cephfs-top-19.0.0-3116.g1f81fb39.el9. 365 kB/s | 18 kB 00:00 2024-04-24T07:55:18.053 INFO:teuthology.orchestra.run.smithi117.stdout:(38/152): cephadm-19.0.0-3116.g1f81fb39.el9.noa 5.5 MB/s | 762 kB 00:00 2024-04-24T07:55:18.137 INFO:teuthology.orchestra.run.smithi194.stdout:(8/152): ceph-common-19.0.0-3116.g1f81fb39.el9. 3.0 MB/s | 21 MB 00:07 2024-04-24T07:55:18.170 INFO:teuthology.orchestra.run.smithi117.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 481 kB/s | 80 kB 00:00 2024-04-24T07:55:18.178 INFO:teuthology.orchestra.run.smithi194.stdout:(9/152): ceph-selinux-19.0.0-3116.g1f81fb39.el9 608 kB/s | 25 kB 00:00 2024-04-24T07:55:18.195 INFO:teuthology.orchestra.run.smithi117.stdout:(40/152): ceph-prometheus-alerts-19.0.0-3116.g1 50 kB/s | 16 kB 00:00 2024-04-24T07:55:18.262 INFO:teuthology.orchestra.run.smithi117.stdout:(41/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.1 MB/s | 195 kB 00:00 2024-04-24T07:55:18.412 INFO:teuthology.orchestra.run.smithi117.stdout:(42/152): libgfortran-11.4.1-3.el9.x86_64.rpm 2.2 MB/s | 803 kB 00:00 2024-04-24T07:55:18.437 INFO:teuthology.orchestra.run.smithi117.stdout:(43/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.0 MB/s | 253 kB 00:00 2024-04-24T07:55:18.479 INFO:teuthology.orchestra.run.smithi117.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 5.7 MB/s | 1.2 MB 00:00 2024-04-24T07:55:18.504 INFO:teuthology.orchestra.run.smithi117.stdout:(45/152): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-04-24T07:55:18.530 INFO:teuthology.orchestra.run.smithi117.stdout:(46/152): python3-pycparser-2.20-6.el9.noarch.r 1.4 MB/s | 135 kB 00:00 2024-04-24T07:55:18.563 INFO:teuthology.orchestra.run.smithi117.stdout:(47/152): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-04-24T07:55:18.597 INFO:teuthology.orchestra.run.smithi117.stdout:(48/152): python3-urllib3-1.26.5-5.el9.noarch.r 2.3 MB/s | 215 kB 00:00 2024-04-24T07:55:18.722 INFO:teuthology.orchestra.run.smithi117.stdout:(49/152): automake-1.16.2-8.el9.noarch.rpm 3.5 MB/s | 689 kB 00:00 2024-04-24T07:55:18.747 INFO:teuthology.orchestra.run.smithi117.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 712 kB/s | 107 kB 00:00 2024-04-24T07:55:18.770 INFO:teuthology.orchestra.run.smithi194.stdout:(10/152): ceph-osd-19.0.0-3116.g1f81fb39.el9.x8 4.3 MB/s | 17 MB 00:03 2024-04-24T07:55:18.773 INFO:teuthology.orchestra.run.smithi117.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 429 kB/s | 21 kB 00:00 2024-04-24T07:55:18.831 INFO:teuthology.orchestra.run.smithi117.stdout:(52/152): bison-3.7.4-5.el9.x86_64.rpm 3.4 MB/s | 943 kB 00:00 2024-04-24T07:55:18.848 INFO:teuthology.orchestra.run.smithi117.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 434 kB/s | 33 kB 00:00 2024-04-24T07:55:18.854 INFO:teuthology.orchestra.run.smithi194.stdout:(11/152): cephfs-mirror-19.0.0-3116.g1f81fb39.e 2.6 MB/s | 225 kB 00:00 2024-04-24T07:55:18.882 INFO:teuthology.orchestra.run.smithi117.stdout:(54/152): flex-2.6.4-9.el9.x86_64.rpm 2.3 MB/s | 313 kB 00:00 2024-04-24T07:55:18.905 INFO:teuthology.orchestra.run.smithi194.stdout:(12/152): cephfs-shell-19.0.0-3116.g1f81fb39.el 534 kB/s | 27 kB 00:00 2024-04-24T07:55:18.907 INFO:teuthology.orchestra.run.smithi117.stdout:(55/152): flexiblas-openblas-openmp-3.0.4-8.el9 311 kB/s | 18 kB 00:00 2024-04-24T07:55:18.932 INFO:teuthology.orchestra.run.smithi117.stdout:(56/152): libaio-devel-0.3.111-13.el9.x86_64.rp 222 kB/s | 11 kB 00:00 2024-04-24T07:55:18.963 INFO:teuthology.orchestra.run.smithi194.stdout:(13/152): libcephfs-devel-19.0.0-3116.g1f81fb39 543 kB/s | 32 kB 00:00 2024-04-24T07:55:18.966 INFO:teuthology.orchestra.run.smithi117.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.7 MB/s | 160 kB 00:00 2024-04-24T07:55:18.991 INFO:teuthology.orchestra.run.smithi117.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 772 kB/s | 45 kB 00:00 2024-04-24T07:55:19.025 INFO:teuthology.orchestra.run.smithi117.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 662 kB 00:00 2024-04-24T07:55:19.083 INFO:teuthology.orchestra.run.smithi117.stdout:(60/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 12 MB/s | 3.0 MB 00:00 2024-04-24T07:55:19.109 INFO:teuthology.orchestra.run.smithi117.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 305 kB/s | 25 kB 00:00 2024-04-24T07:55:19.114 INFO:teuthology.orchestra.run.smithi194.stdout:(14/152): libcephfs2-19.0.0-3116.g1f81fb39.el9. 4.9 MB/s | 746 kB 00:00 2024-04-24T07:55:19.134 INFO:teuthology.orchestra.run.smithi117.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 948 kB/s | 47 kB 00:00 2024-04-24T07:55:19.159 INFO:teuthology.orchestra.run.smithi117.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-24T07:55:19.172 INFO:teuthology.orchestra.run.smithi194.stdout:(15/152): libcephsqlite-19.0.0-3116.g1f81fb39.e 2.7 MB/s | 163 kB 00:00 2024-04-24T07:55:19.184 INFO:teuthology.orchestra.run.smithi117.stdout:(64/152): libtool-2.4.6-45.el9.x86_64.rpm 3.0 MB/s | 582 kB 00:00 2024-04-24T07:55:19.210 INFO:teuthology.orchestra.run.smithi117.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 701 kB/s | 35 kB 00:00 2024-04-24T07:55:19.222 INFO:teuthology.orchestra.run.smithi194.stdout:(16/152): librados-devel-19.0.0-3116.g1f81fb39. 2.5 MB/s | 127 kB 00:00 2024-04-24T07:55:19.235 INFO:teuthology.orchestra.run.smithi117.stdout:(66/152): lua-5.4.4-4.el9.x86_64.rpm 1.8 MB/s | 188 kB 00:00 2024-04-24T07:55:19.348 INFO:teuthology.orchestra.run.smithi194.stdout:(17/152): libradosstriper1-19.0.0-3116.g1f81fb3 3.9 MB/s | 501 kB 00:00 2024-04-24T07:55:19.360 INFO:teuthology.orchestra.run.smithi117.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 13 MB/s | 1.6 MB 00:00 2024-04-24T07:55:19.444 INFO:teuthology.orchestra.run.smithi117.stdout:(68/152): openblas-openmp-0.3.21-2.el9.x86_64.r 18 MB/s | 4.7 MB 00:00 2024-04-24T07:55:19.478 INFO:teuthology.orchestra.run.smithi117.stdout:(69/152): postgresql-contrib-13.14-1.el9.x86_64 7.6 MB/s | 911 kB 00:00 2024-04-24T07:55:19.503 INFO:teuthology.orchestra.run.smithi117.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 2.2 MB/s | 133 kB 00:00 2024-04-24T07:55:19.548 INFO:teuthology.orchestra.run.smithi194.stdout:(18/152): ceph-radosgw-19.0.0-3116.g1f81fb39.el 4.3 MB/s | 11 MB 00:02 2024-04-24T07:55:19.570 INFO:teuthology.orchestra.run.smithi117.stdout:(71/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 12 MB/s | 4.4 MB 00:00 2024-04-24T07:55:19.598 INFO:teuthology.orchestra.run.smithi194.stdout:(19/152): python3-ceph-argparse-19.0.0-3116.g1f 911 kB/s | 45 kB 00:00 2024-04-24T07:55:19.648 INFO:teuthology.orchestra.run.smithi194.stdout:(20/152): python3-ceph-common-19.0.0-3116.g1f81 2.6 MB/s | 135 kB 00:00 2024-04-24T07:55:19.699 INFO:teuthology.orchestra.run.smithi194.stdout:(21/152): python3-cephfs-19.0.0-3116.g1f81fb39. 3.2 MB/s | 161 kB 00:00 2024-04-24T07:55:19.795 INFO:teuthology.orchestra.run.smithi117.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 389 kB/s | 87 kB 00:00 2024-04-24T07:55:19.807 INFO:teuthology.orchestra.run.smithi194.stdout:(22/152): python3-rados-19.0.0-3116.g1f81fb39.e 2.9 MB/s | 320 kB 00:00 2024-04-24T07:55:19.891 INFO:teuthology.orchestra.run.smithi194.stdout:(23/152): python3-rbd-19.0.0-3116.g1f81fb39.el9 3.5 MB/s | 298 kB 00:00 2024-04-24T07:55:19.950 INFO:teuthology.orchestra.run.smithi194.stdout:(24/152): python3-rgw-19.0.0-3116.g1f81fb39.el9 1.7 MB/s | 99 kB 00:00 2024-04-24T07:55:19.991 INFO:teuthology.orchestra.run.smithi194.stdout:(25/152): rbd-fuse-19.0.0-3116.g1f81fb39.el9.x8 2.0 MB/s | 85 kB 00:00 2024-04-24T07:55:19.995 INFO:teuthology.orchestra.run.smithi117.stdout:(73/152): protobuf-3.14.0-13.el9.x86_64.rpm 2.0 MB/s | 1.0 MB 00:00 2024-04-24T07:55:20.196 INFO:teuthology.orchestra.run.smithi117.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 1.2 MB/s | 244 kB 00:00 2024-04-24T07:55:20.225 INFO:teuthology.orchestra.run.smithi194.stdout:(26/152): librgw2-19.0.0-3116.g1f81fb39.el9.x86 6.1 MB/s | 5.4 MB 00:00 2024-04-24T07:55:20.254 INFO:teuthology.orchestra.run.smithi117.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.2 MB/s | 248 kB 00:00 2024-04-24T07:55:20.275 INFO:teuthology.orchestra.run.smithi194.stdout:(27/152): rbd-nbd-19.0.0-3116.g1f81fb39.el9.x86 3.4 MB/s | 170 kB 00:00 2024-04-24T07:55:20.304 INFO:teuthology.orchestra.run.smithi117.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 513 kB/s | 26 kB 00:00 2024-04-24T07:55:20.317 INFO:teuthology.orchestra.run.smithi194.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3116.g 824 kB/s | 34 kB 00:00 2024-04-24T07:55:20.371 INFO:teuthology.orchestra.run.smithi117.stdout:(77/152): postgresql-server-13.14-1.el9.x86_64. 6.5 MB/s | 5.8 MB 00:00 2024-04-24T07:55:20.376 INFO:teuthology.orchestra.run.smithi194.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3116.g1f81fb3 2.4 MB/s | 142 kB 00:00 2024-04-24T07:55:20.397 INFO:teuthology.orchestra.run.smithi117.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 209 kB/s | 19 kB 00:00 2024-04-24T07:55:20.422 INFO:teuthology.orchestra.run.smithi117.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 172 kB 00:00 2024-04-24T07:55:20.447 INFO:teuthology.orchestra.run.smithi117.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 694 kB/s | 35 kB 00:00 2024-04-24T07:55:20.531 INFO:teuthology.orchestra.run.smithi117.stdout:(81/152): python3-babel-2.9.1-2.el9.noarch.rpm 8.1 MB/s | 6.0 MB 00:00 2024-04-24T07:55:20.556 INFO:teuthology.orchestra.run.smithi117.stdout:(82/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.3 MB/s | 250 kB 00:00 2024-04-24T07:55:20.589 INFO:teuthology.orchestra.run.smithi117.stdout:(83/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-04-24T07:55:20.615 INFO:teuthology.orchestra.run.smithi117.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 77 kB 00:00 2024-04-24T07:55:20.634 INFO:teuthology.orchestra.run.smithi194.stdout:(30/152): rbd-mirror-19.0.0-3116.g1f81fb39.el9. 4.8 MB/s | 3.1 MB 00:00 2024-04-24T07:55:20.640 INFO:teuthology.orchestra.run.smithi117.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 841 kB/s | 42 kB 00:00 2024-04-24T07:55:20.665 INFO:teuthology.orchestra.run.smithi117.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.3 MB/s | 269 kB 00:00 2024-04-24T07:55:20.691 INFO:teuthology.orchestra.run.smithi117.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-24T07:55:20.716 INFO:teuthology.orchestra.run.smithi117.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-24T07:55:20.733 INFO:teuthology.orchestra.run.smithi117.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.2 MB/s | 51 kB 00:00 2024-04-24T07:55:20.801 INFO:teuthology.orchestra.run.smithi117.stdout:(90/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 14 MB/s | 5.1 MB 00:00 2024-04-24T07:55:20.816 INFO:teuthology.orchestra.run.smithi117.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 535 kB/s | 54 kB 00:00 2024-04-24T07:55:20.842 INFO:teuthology.orchestra.run.smithi117.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 1.0 MB/s | 42 kB 00:00 2024-04-24T07:55:20.859 INFO:teuthology.orchestra.run.smithi117.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.0 MB/s | 44 kB 00:00 2024-04-24T07:55:20.943 INFO:teuthology.orchestra.run.smithi117.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 430 kB/s | 36 kB 00:00 2024-04-24T07:55:20.968 INFO:teuthology.orchestra.run.smithi117.stdout:(95/152): qatlib-24.02.0-1.el9.x86_64.rpm 1.7 MB/s | 221 kB 00:00 2024-04-24T07:55:20.985 INFO:teuthology.orchestra.run.smithi194.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3116.g1f81f 5.8 MB/s | 3.5 MB 00:00 2024-04-24T07:55:20.985 INFO:teuthology.orchestra.run.smithi117.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 1.1 MB/s | 47 kB 00:00 2024-04-24T07:55:21.018 INFO:teuthology.orchestra.run.smithi117.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 305 kB 00:00 2024-04-24T07:55:21.035 INFO:teuthology.orchestra.run.smithi194.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3116.g1f 4.9 MB/s | 249 kB 00:00 2024-04-24T07:55:21.035 INFO:teuthology.orchestra.run.smithi117.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 1.1 MB/s | 58 kB 00:00 2024-04-24T07:55:21.069 INFO:teuthology.orchestra.run.smithi117.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.8 MB/s | 94 kB 00:00 2024-04-24T07:55:21.077 INFO:teuthology.orchestra.run.smithi194.stdout:(33/152): ceph-mgr-rook-19.0.0-3116.g1f81fb39.e 1.2 MB/s | 49 kB 00:00 2024-04-24T07:55:21.103 INFO:teuthology.orchestra.run.smithi117.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 947 kB/s | 64 kB 00:00 2024-04-24T07:55:21.127 INFO:teuthology.orchestra.run.smithi194.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3116.g1 329 kB/s | 16 kB 00:00 2024-04-24T07:55:21.128 INFO:teuthology.orchestra.run.smithi117.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 765 kB/s | 45 kB 00:00 2024-04-24T07:55:21.227 INFO:teuthology.orchestra.run.smithi194.stdout:(35/152): ceph-volume-19.0.0-3116.g1f81fb39.el9 2.7 MB/s | 273 kB 00:00 2024-04-24T07:55:21.287 INFO:teuthology.orchestra.run.smithi117.stdout:(102/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 29 MB/s | 16 MB 00:00 2024-04-24T07:55:21.362 INFO:teuthology.orchestra.run.smithi117.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 86 kB/s | 22 kB 00:00 2024-04-24T07:55:21.411 INFO:teuthology.orchestra.run.smithi194.stdout:(36/152): cephadm-19.0.0-3116.g1f81fb39.el9.noa 4.1 MB/s | 762 kB 00:00 2024-04-24T07:55:21.461 INFO:teuthology.orchestra.run.smithi194.stdout:(37/152): cephfs-top-19.0.0-3116.g1f81fb39.el9. 366 kB/s | 18 kB 00:00 2024-04-24T07:55:21.595 INFO:teuthology.orchestra.run.smithi117.stdout:(104/152): gperftools-libs-2.9.1-2.el9.x86_64.r 1.3 MB/s | 309 kB 00:00 2024-04-24T07:55:21.611 INFO:teuthology.orchestra.run.smithi194.stdout:(38/152): fuse-2.9.9-15.el9.x86_64.rpm 535 kB/s | 80 kB 00:00 2024-04-24T07:55:21.620 INFO:teuthology.orchestra.run.smithi117.stdout:(105/152): protobuf-compiler-3.14.0-13.el9.x86_ 1.7 MB/s | 863 kB 00:00 2024-04-24T07:55:21.646 INFO:teuthology.orchestra.run.smithi117.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 389 kB/s | 19 kB 00:00 2024-04-24T07:55:21.696 INFO:teuthology.orchestra.run.smithi117.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.3 MB/s | 548 kB 00:00 2024-04-24T07:55:21.721 INFO:teuthology.orchestra.run.smithi117.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 527 kB/s | 26 kB 00:00 2024-04-24T07:55:21.745 INFO:teuthology.orchestra.run.smithi194.stdout:(39/152): libgfortran-11.4.1-3.el9.x86_64.rpm 5.9 MB/s | 803 kB 00:00 2024-04-24T07:55:21.747 INFO:teuthology.orchestra.run.smithi117.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 987 kB/s | 49 kB 00:00 2024-04-24T07:55:21.772 INFO:teuthology.orchestra.run.smithi117.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.3 MB/s | 67 kB 00:00 2024-04-24T07:55:21.795 INFO:teuthology.orchestra.run.smithi194.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 3.8 MB/s | 195 kB 00:00 2024-04-24T07:55:21.814 INFO:teuthology.orchestra.run.smithi117.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-04-24T07:55:21.848 INFO:teuthology.orchestra.run.smithi194.stdout:(41/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.8 MB/s | 253 kB 00:00 2024-04-24T07:55:21.873 INFO:teuthology.orchestra.run.smithi117.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 8.2 MB/s | 838 kB 00:00 2024-04-24T07:55:21.923 INFO:teuthology.orchestra.run.smithi117.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 872 kB/s | 43 kB 00:00 2024-04-24T07:55:21.973 INFO:teuthology.orchestra.run.smithi117.stdout:(114/152): python3-cachetools-4.2.4-1.el9.noarc 645 kB/s | 32 kB 00:00 2024-04-24T07:55:21.981 INFO:teuthology.orchestra.run.smithi194.stdout:(42/152): python3-cryptography-36.0.1-4.el9.x86 9.2 MB/s | 1.2 MB 00:00 2024-04-24T07:55:22.023 INFO:teuthology.orchestra.run.smithi117.stdout:(115/152): python3-certifi-2021.10.8-2.el9.noar 299 kB/s | 15 kB 00:00 2024-04-24T07:55:22.040 INFO:teuthology.orchestra.run.smithi194.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 106 kB 00:00 2024-04-24T07:55:22.082 INFO:teuthology.orchestra.run.smithi117.stdout:(116/152): libarrow-9.0.0-10.el9.x86_64.rpm 9.6 MB/s | 4.4 MB 00:00 2024-04-24T07:55:22.090 INFO:teuthology.orchestra.run.smithi194.stdout:(44/152): python3-pycparser-2.20-6.el9.noarch.r 2.6 MB/s | 135 kB 00:00 2024-04-24T07:55:22.107 INFO:teuthology.orchestra.run.smithi117.stdout:(117/152): python3-asyncssh-2.13.2-5.el9.noarch 1.8 MB/s | 548 kB 00:00 2024-04-24T07:55:22.132 INFO:teuthology.orchestra.run.smithi117.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-04-24T07:55:22.158 INFO:teuthology.orchestra.run.smithi117.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-04-24T07:55:22.174 INFO:teuthology.orchestra.run.smithi194.stdout:(45/152): ceph-mgr-diskprediction-local-19.0.0- 4.8 MB/s | 7.4 MB 00:01 2024-04-24T07:55:22.183 INFO:teuthology.orchestra.run.smithi117.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 701 kB/s | 35 kB 00:00 2024-04-24T07:55:22.199 INFO:teuthology.orchestra.run.smithi194.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-04-24T07:55:22.208 INFO:teuthology.orchestra.run.smithi117.stdout:(121/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.9 MB/s | 303 kB 00:00 2024-04-24T07:55:22.233 INFO:teuthology.orchestra.run.smithi117.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 2.7 MB/s | 206 kB 00:00 2024-04-24T07:55:22.291 INFO:teuthology.orchestra.run.smithi194.stdout:(47/152): automake-1.16.2-8.el9.noarch.rpm 7.4 MB/s | 689 kB 00:00 2024-04-24T07:55:22.292 INFO:teuthology.orchestra.run.smithi117.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 183 kB/s | 11 kB 00:00 2024-04-24T07:55:22.317 INFO:teuthology.orchestra.run.smithi117.stdout:(124/152): python3-grpcio-tools-1.46.7-10.el9.x 1.3 MB/s | 144 kB 00:00 2024-04-24T07:55:22.342 INFO:teuthology.orchestra.run.smithi117.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-04-24T07:55:22.366 INFO:teuthology.orchestra.run.smithi194.stdout:(48/152): bison-3.7.4-5.el9.x86_64.rpm 12 MB/s | 943 kB 00:00 2024-04-24T07:55:22.367 INFO:teuthology.orchestra.run.smithi117.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 462 kB/s | 23 kB 00:00 2024-04-24T07:55:22.391 INFO:teuthology.orchestra.run.smithi194.stdout:(49/152): python3-urllib3-1.26.5-5.el9.noarch.r 988 kB/s | 215 kB 00:00 2024-04-24T07:55:22.417 INFO:teuthology.orchestra.run.smithi194.stdout:(50/152): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-04-24T07:55:22.418 INFO:teuthology.orchestra.run.smithi117.stdout:(127/152): python3-grpcio-1.46.7-10.el9.x86_64. 8.7 MB/s | 2.0 MB 00:00 2024-04-24T07:55:22.434 INFO:teuthology.orchestra.run.smithi194.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 516 kB/s | 21 kB 00:00 2024-04-24T07:55:22.443 INFO:teuthology.orchestra.run.smithi117.stdout:(128/152): python3-jaraco-functools-3.5.0-2.el9 193 kB/s | 19 kB 00:00 2024-04-24T07:55:22.460 INFO:teuthology.orchestra.run.smithi117.stdout:(129/152): python3-jaraco-text-3.2.0-6.el9.noar 213 kB/s | 20 kB 00:00 2024-04-24T07:55:22.475 INFO:teuthology.orchestra.run.smithi194.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 313 kB 00:00 2024-04-24T07:55:22.492 INFO:teuthology.orchestra.run.smithi194.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 560 kB/s | 33 kB 00:00 2024-04-24T07:55:22.519 INFO:teuthology.orchestra.run.smithi117.stdout:(130/152): python3-kubernetes-26.1.0-2.el9.noar 10 MB/s | 1.0 MB 00:00 2024-04-24T07:55:22.534 INFO:teuthology.orchestra.run.smithi194.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 438 kB/s | 18 kB 00:00 2024-04-24T07:55:22.544 INFO:teuthology.orchestra.run.smithi117.stdout:(131/152): python3-logutils-0.3.5-21.el9.noarch 554 kB/s | 46 kB 00:00 2024-04-24T07:55:22.569 INFO:teuthology.orchestra.run.smithi117.stdout:(132/152): python3-more-itertools-8.12.0-2.el9. 725 kB/s | 79 kB 00:00 2024-04-24T07:55:22.576 INFO:teuthology.orchestra.run.smithi194.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 267 kB/s | 11 kB 00:00 2024-04-24T07:55:22.595 INFO:teuthology.orchestra.run.smithi117.stdout:(133/152): python3-natsort-7.1.1-5.el9.noarch.r 764 kB/s | 58 kB 00:00 2024-04-24T07:55:22.620 INFO:teuthology.orchestra.run.smithi117.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 326 kB/s | 16 kB 00:00 2024-04-24T07:55:22.653 INFO:teuthology.orchestra.run.smithi117.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-04-24T07:55:22.679 INFO:teuthology.orchestra.run.smithi117.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 483 kB/s | 28 kB 00:00 2024-04-24T07:55:22.685 INFO:teuthology.orchestra.run.smithi194.stdout:(56/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 14 MB/s | 3.0 MB 00:00 2024-04-24T07:55:22.704 INFO:teuthology.orchestra.run.smithi117.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-04-24T07:55:22.710 INFO:teuthology.orchestra.run.smithi194.stdout:(57/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.2 MB/s | 160 kB 00:00 2024-04-24T07:55:22.729 INFO:teuthology.orchestra.run.smithi117.stdout:(138/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 669 kB/s | 90 kB 00:00 2024-04-24T07:55:22.735 INFO:teuthology.orchestra.run.smithi194.stdout:(58/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 905 kB/s | 45 kB 00:00 2024-04-24T07:55:22.754 INFO:teuthology.orchestra.run.smithi117.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.2 MB/s | 59 kB 00:00 2024-04-24T07:55:22.780 INFO:teuthology.orchestra.run.smithi117.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 713 kB/s | 36 kB 00:00 2024-04-24T07:55:22.794 INFO:teuthology.orchestra.run.smithi194.stdout:(59/152): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 582 kB 00:00 2024-04-24T07:55:22.805 INFO:teuthology.orchestra.run.smithi117.stdout:(141/152): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-04-24T07:55:22.830 INFO:teuthology.orchestra.run.smithi117.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 675 kB/s | 51 kB 00:00 2024-04-24T07:55:22.844 INFO:teuthology.orchestra.run.smithi194.stdout:(60/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 509 kB/s | 25 kB 00:00 2024-04-24T07:55:22.847 INFO:teuthology.orchestra.run.smithi117.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-04-24T07:55:22.873 INFO:teuthology.orchestra.run.smithi117.stdout:(144/152): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-04-24T07:55:22.894 INFO:teuthology.orchestra.run.smithi194.stdout:(61/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 953 kB/s | 47 kB 00:00 2024-04-24T07:55:22.898 INFO:teuthology.orchestra.run.smithi117.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-04-24T07:55:22.923 INFO:teuthology.orchestra.run.smithi117.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-04-24T07:55:22.945 INFO:teuthology.orchestra.run.smithi194.stdout:(62/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 5.7 MB/s | 292 kB 00:00 2024-04-24T07:55:22.948 INFO:teuthology.orchestra.run.smithi117.stdout:(147/152): python3-werkzeug-2.0.3-3.el9.1.noarc 3.6 MB/s | 427 kB 00:00 2024-04-24T07:55:22.995 INFO:teuthology.orchestra.run.smithi194.stdout:(63/152): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 188 kB 00:00 2024-04-24T07:55:23.007 INFO:teuthology.orchestra.run.smithi117.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-04-24T07:55:23.021 INFO:teuthology.orchestra.run.smithi194.stdout:(64/152): librdkafka-1.6.1-102.el9.x86_64.rpm 2.1 MB/s | 662 kB 00:00 2024-04-24T07:55:23.095 INFO:teuthology.orchestra.run.smithi194.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 353 kB/s | 35 kB 00:00 2024-04-24T07:55:23.296 INFO:teuthology.orchestra.run.smithi194.stdout:(66/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 22 MB/s | 4.4 MB 00:00 2024-04-24T07:55:23.357 INFO:teuthology.orchestra.run.smithi117.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 4.2 MB/s | 1.3 MB 00:00 2024-04-24T07:55:23.388 INFO:teuthology.orchestra.run.smithi194.stdout:(67/152): postgresql-13.14-1.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-04-24T07:55:23.455 INFO:teuthology.orchestra.run.smithi194.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 13 MB/s | 911 kB 00:00 2024-04-24T07:55:23.505 INFO:teuthology.orchestra.run.smithi194.stdout:(69/152): postgresql-private-libs-13.14-1.el9.x 2.6 MB/s | 133 kB 00:00 2024-04-24T07:55:23.649 INFO:teuthology.orchestra.run.smithi117.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 8.2 MB/s | 2.4 MB 00:00 2024-04-24T07:55:23.714 INFO:teuthology.orchestra.run.smithi194.stdout:(70/152): postgresql-server-13.14-1.el9.x86_64. 28 MB/s | 5.8 MB 00:00 2024-04-24T07:55:23.781 INFO:teuthology.orchestra.run.smithi194.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 15 MB/s | 1.0 MB 00:00 2024-04-24T07:55:23.831 INFO:teuthology.orchestra.run.smithi194.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 1.7 MB/s | 87 kB 00:00 2024-04-24T07:55:23.882 INFO:teuthology.orchestra.run.smithi117.stdout:(151/152): librados2-19.0.0-3116.g1f81fb39.el9. 3.5 MB/s | 3.4 MB 00:00 2024-04-24T07:55:23.923 INFO:teuthology.orchestra.run.smithi194.stdout:(73/152): openblas-openmp-0.3.21-2.el9.x86_64.r 5.3 MB/s | 4.7 MB 00:00 2024-04-24T07:55:23.982 INFO:teuthology.orchestra.run.smithi194.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.1 MB/s | 244 kB 00:00 2024-04-24T07:55:24.090 INFO:teuthology.orchestra.run.smithi194.stdout:(75/152): python3-babel-2.9.1-2.el9.noarch.rpm 23 MB/s | 6.0 MB 00:00 2024-04-24T07:55:24.115 INFO:teuthology.orchestra.run.smithi194.stdout:(76/152): python3-jinja2-2.11.3-5.el9.noarch.rp 1.8 MB/s | 248 kB 00:00 2024-04-24T07:55:24.141 INFO:teuthology.orchestra.run.smithi194.stdout:(77/152): python3-jsonpatch-1.21-16.el9.noarch. 511 kB/s | 26 kB 00:00 2024-04-24T07:55:24.158 INFO:teuthology.orchestra.run.smithi194.stdout:(78/152): python3-jsonpointer-2.0-4.el9.noarch. 457 kB/s | 19 kB 00:00 2024-04-24T07:55:24.183 INFO:teuthology.orchestra.run.smithi194.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 4.0 MB/s | 172 kB 00:00 2024-04-24T07:55:24.200 INFO:teuthology.orchestra.run.smithi194.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 828 kB/s | 35 kB 00:00 2024-04-24T07:55:24.259 INFO:teuthology.orchestra.run.smithi194.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 4.2 MB/s | 250 kB 00:00 2024-04-24T07:55:24.317 INFO:teuthology.orchestra.run.smithi194.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 3.7 MB/s | 222 kB 00:00 2024-04-24T07:55:24.376 INFO:teuthology.orchestra.run.smithi194.stdout:(83/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 5.1 MB 00:00 2024-04-24T07:55:24.401 INFO:teuthology.orchestra.run.smithi194.stdout:(84/152): python3-packaging-20.9-5.el9.noarch.r 926 kB/s | 77 kB 00:00 2024-04-24T07:55:24.426 INFO:teuthology.orchestra.run.smithi194.stdout:(85/152): python3-prettytable-0.7.2-27.el9.noar 844 kB/s | 42 kB 00:00 2024-04-24T07:55:24.451 INFO:teuthology.orchestra.run.smithi194.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 5.2 MB/s | 269 kB 00:00 2024-04-24T07:55:24.477 INFO:teuthology.orchestra.run.smithi194.stdout:(87/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 3.1 MB/s | 159 kB 00:00 2024-04-24T07:55:24.502 INFO:teuthology.orchestra.run.smithi194.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 5.4 MB/s | 279 kB 00:00 2024-04-24T07:55:24.527 INFO:teuthology.orchestra.run.smithi194.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-04-24T07:55:24.553 INFO:teuthology.orchestra.run.smithi194.stdout:(90/152): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 54 kB 00:00 2024-04-24T07:55:24.645 INFO:teuthology.orchestra.run.smithi194.stdout:(91/152): python3-toml-0.10.2-6.el9.noarch.rpm 454 kB/s | 42 kB 00:00 2024-04-24T07:55:25.070 INFO:teuthology.orchestra.run.smithi194.stdout:(92/152): ceph-test-19.0.0-3116.g1f81fb39.el9.x 7.1 MB/s | 49 MB 00:06 2024-04-24T07:55:25.087 INFO:teuthology.orchestra.run.smithi194.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 100 kB/s | 44 kB 00:00 2024-04-24T07:55:25.129 INFO:teuthology.orchestra.run.smithi194.stdout:(94/152): qatlib-service-24.02.0-1.el9.x86_64.r 863 kB/s | 36 kB 00:00 2024-04-24T07:55:25.188 INFO:teuthology.orchestra.run.smithi194.stdout:(95/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 814 kB/s | 47 kB 00:00 2024-04-24T07:55:25.321 INFO:teuthology.orchestra.run.smithi194.stdout:(96/152): socat-1.7.4.1-5.el9.x86_64.rpm 2.2 MB/s | 305 kB 00:00 2024-04-24T07:55:25.388 INFO:teuthology.orchestra.run.smithi194.stdout:(97/152): uuid-1.6.2-55.el9.x86_64.rpm 868 kB/s | 58 kB 00:00 2024-04-24T07:55:25.413 INFO:teuthology.orchestra.run.smithi194.stdout:(98/152): qatlib-24.02.0-1.el9.x86_64.rpm 644 kB/s | 221 kB 00:00 2024-04-24T07:55:25.572 INFO:teuthology.orchestra.run.smithi194.stdout:(99/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 16 MB/s | 16 MB 00:01 2024-04-24T07:55:25.589 INFO:teuthology.orchestra.run.smithi194.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 363 kB/s | 64 kB 00:00 2024-04-24T07:55:25.607 INFO:teuthology.orchestra.run.smithi117.stdout:(152/152): librbd1-19.0.0-3116.g1f81fb39.el9.x8 1.2 MB/s | 3.1 MB 00:02 2024-04-24T07:55:25.613 INFO:teuthology.orchestra.run.smithi117.stdout:-------------------------------------------------------------------------------- 2024-04-24T07:55:25.614 INFO:teuthology.orchestra.run.smithi117.stdout:Total 13 MB/s | 222 MB 00:16 2024-04-24T07:55:25.622 INFO:teuthology.orchestra.run.smithi194.stdout:(101/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 400 kB/s | 94 kB 00:00 2024-04-24T07:55:25.648 INFO:teuthology.orchestra.run.smithi194.stdout:(102/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 595 kB/s | 45 kB 00:00 2024-04-24T07:55:25.732 INFO:teuthology.orchestra.run.smithi194.stdout:(103/152): lua-devel-5.4.4-4.el9.x86_64.rpm 156 kB/s | 22 kB 00:00 2024-04-24T07:55:25.982 INFO:teuthology.orchestra.run.smithi194.stdout:(104/152): abseil-cpp-20211102.0-3.el9.x86_64.r 1.6 MB/s | 548 kB 00:00 2024-04-24T07:55:26.023 INFO:teuthology.orchestra.run.smithi194.stdout:(105/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-04-24T07:55:26.049 INFO:teuthology.orchestra.run.smithi194.stdout:(106/152): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-04-24T07:55:26.074 INFO:teuthology.orchestra.run.smithi194.stdout:(107/152): gperftools-libs-2.9.1-2.el9.x86_64.r 902 kB/s | 309 kB 00:00 2024-04-24T07:55:26.124 INFO:teuthology.orchestra.run.smithi194.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 345 kB/s | 26 kB 00:00 2024-04-24T07:55:26.166 INFO:teuthology.orchestra.run.smithi194.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 749 kB/s | 49 kB 00:00 2024-04-24T07:55:26.191 INFO:teuthology.orchestra.run.smithi194.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-04-24T07:55:26.233 INFO:teuthology.orchestra.run.smithi194.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-04-24T07:55:26.309 INFO:teuthology.orchestra.run.smithi194.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 7.0 MB/s | 838 kB 00:00 2024-04-24T07:55:26.367 INFO:teuthology.orchestra.run.smithi194.stdout:(113/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 747 kB/s | 43 kB 00:00 2024-04-24T07:55:26.409 INFO:teuthology.orchestra.run.smithi194.stdout:(114/152): python3-asyncssh-2.13.2-5.el9.noarch 3.1 MB/s | 548 kB 00:00 2024-04-24T07:55:26.434 INFO:teuthology.orchestra.run.smithi194.stdout:(115/152): python3-cachetools-4.2.4-1.el9.noarc 483 kB/s | 32 kB 00:00 2024-04-24T07:55:26.476 INFO:teuthology.orchestra.run.smithi194.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 224 kB/s | 15 kB 00:00 2024-04-24T07:55:26.502 INFO:teuthology.orchestra.run.smithi194.stdout:(117/152): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-04-24T07:55:26.577 INFO:teuthology.orchestra.run.smithi194.stdout:(118/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.9 MB/s | 303 kB 00:00 2024-04-24T07:55:26.602 INFO:teuthology.orchestra.run.smithi194.stdout:(119/152): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-04-24T07:55:26.644 INFO:teuthology.orchestra.run.smithi194.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 528 kB/s | 35 kB 00:00 2024-04-24T07:55:26.669 INFO:teuthology.orchestra.run.smithi194.stdout:(121/152): python3-google-auth-2.29.0-1.el9.noa 3.0 MB/s | 206 kB 00:00 2024-04-24T07:55:26.733 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction check 2024-04-24T07:55:26.744 INFO:teuthology.orchestra.run.smithi194.stdout:(122/152): python3-grpcio-tools-1.46.7-10.el9.x 1.9 MB/s | 144 kB 00:00 2024-04-24T07:55:26.820 INFO:teuthology.orchestra.run.smithi194.stdout:(123/152): python3-jaraco-8.2.1-3.el9.noarch.rp 143 kB/s | 11 kB 00:00 2024-04-24T07:55:26.877 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction check succeeded. 2024-04-24T07:55:26.877 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction test 2024-04-24T07:55:26.887 INFO:teuthology.orchestra.run.smithi194.stdout:(124/152): python3-grpcio-1.46.7-10.el9.x86_64. 8.4 MB/s | 2.0 MB 00:00 2024-04-24T07:55:26.912 INFO:teuthology.orchestra.run.smithi194.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-04-24T07:55:26.963 INFO:teuthology.orchestra.run.smithi194.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 307 kB/s | 23 kB 00:00 2024-04-24T07:55:26.988 INFO:teuthology.orchestra.run.smithi194.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-04-24T07:55:27.030 INFO:teuthology.orchestra.run.smithi194.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 295 kB/s | 20 kB 00:00 2024-04-24T07:55:27.088 INFO:teuthology.orchestra.run.smithi194.stdout:(129/152): python3-logutils-0.3.5-21.el9.noarch 795 kB/s | 46 kB 00:00 2024-04-24T07:55:27.147 INFO:teuthology.orchestra.run.smithi194.stdout:(130/152): python3-more-itertools-8.12.0-2.el9. 1.3 MB/s | 79 kB 00:00 2024-04-24T07:55:27.189 INFO:teuthology.orchestra.run.smithi194.stdout:(131/152): python3-kubernetes-26.1.0-2.el9.noar 5.1 MB/s | 1.0 MB 00:00 2024-04-24T07:55:27.223 INFO:teuthology.orchestra.run.smithi194.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 764 kB/s | 58 kB 00:00 2024-04-24T07:55:27.248 INFO:teuthology.orchestra.run.smithi194.stdout:(133/152): python3-pecan-1.4.2-3.el9.noarch.rpm 4.5 MB/s | 272 kB 00:00 2024-04-24T07:55:27.273 INFO:teuthology.orchestra.run.smithi194.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 328 kB/s | 16 kB 00:00 2024-04-24T07:55:27.307 INFO:teuthology.orchestra.run.smithi194.stdout:(135/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.5 MB/s | 90 kB 00:00 2024-04-24T07:55:27.324 INFO:teuthology.orchestra.run.smithi194.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 562 kB/s | 28 kB 00:00 2024-04-24T07:55:27.357 INFO:teuthology.orchestra.run.smithi194.stdout:(137/152): python3-repoze-lru-0.7-16.el9.noarch 612 kB/s | 31 kB 00:00 2024-04-24T07:55:27.382 INFO:teuthology.orchestra.run.smithi194.stdout:(138/152): python3-routes-2.5.1-5.el9.noarch.rp 3.2 MB/s | 188 kB 00:00 2024-04-24T07:55:27.408 INFO:teuthology.orchestra.run.smithi194.stdout:(139/152): python3-rsa-4.9-2.el9.noarch.rpm 1.1 MB/s | 59 kB 00:00 2024-04-24T07:55:27.425 INFO:teuthology.orchestra.run.smithi194.stdout:(140/152): python3-tempora-5.0.0-2.el9.noarch.r 852 kB/s | 36 kB 00:00 2024-04-24T07:55:27.450 INFO:teuthology.orchestra.run.smithi194.stdout:(141/152): python3-typing-extensions-4.4.0-2.el 1.2 MB/s | 51 kB 00:00 2024-04-24T07:55:27.475 INFO:teuthology.orchestra.run.smithi194.stdout:(142/152): python3-webob-1.8.7-6.el9.noarch.rpm 4.5 MB/s | 230 kB 00:00 2024-04-24T07:55:27.492 INFO:teuthology.orchestra.run.smithi194.stdout:(143/152): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-04-24T07:55:27.534 INFO:teuthology.orchestra.run.smithi194.stdout:(144/152): python3-werkzeug-2.0.3-3.el9.1.noarc 7.1 MB/s | 427 kB 00:00 2024-04-24T07:55:27.559 INFO:teuthology.orchestra.run.smithi194.stdout:(145/152): python3-zc-lockfile-2.0-10.el9.noarc 299 kB/s | 20 kB 00:00 2024-04-24T07:55:27.593 INFO:teuthology.orchestra.run.smithi194.stdout:(146/152): re2-20211101-3.el9.x86_64.rpm 3.2 MB/s | 192 kB 00:00 2024-04-24T07:55:27.760 INFO:teuthology.orchestra.run.smithi194.stdout:(147/152): libarrow-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 4.4 MB 00:01 2024-04-24T07:55:27.802 INFO:teuthology.orchestra.run.smithi194.stdout:(148/152): thrift-0.15.0-2.el9.x86_64.rpm 6.5 MB/s | 1.6 MB 00:00 2024-04-24T07:55:28.010 INFO:teuthology.orchestra.run.smithi194.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 6.5 MB/s | 1.3 MB 00:00 2024-04-24T07:55:28.119 INFO:teuthology.orchestra.run.smithi194.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 22 MB/s | 2.4 MB 00:00 2024-04-24T07:55:28.649 INFO:teuthology.orchestra.run.smithi117.stdout:Transaction test succeeded. 2024-04-24T07:55:28.649 INFO:teuthology.orchestra.run.smithi117.stdout:Running transaction 2024-04-24T07:55:28.720 INFO:teuthology.orchestra.run.smithi194.stdout:(151/152): librados2-19.0.0-3116.g1f81fb39.el9. 3.0 MB/s | 3.4 MB 00:01 2024-04-24T07:55:28.811 INFO:teuthology.orchestra.run.smithi194.stdout:(152/152): librbd1-19.0.0-3116.g1f81fb39.el9.x8 3.0 MB/s | 3.1 MB 00:01 2024-04-24T07:55:28.818 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-04-24T07:55:28.818 INFO:teuthology.orchestra.run.smithi194.stdout:Total 12 MB/s | 222 MB 00:18 2024-04-24T07:55:29.584 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - AppStream 704 kB/s | 19 MB 00:27 2024-04-24T07:55:29.946 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-04-24T07:55:30.088 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-04-24T07:55:30.089 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-04-24T07:55:31.852 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-04-24T07:55:31.853 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-04-24T07:55:34.751 INFO:teuthology.orchestra.run.smithi117.stdout: Preparing : 1/1 2024-04-24T07:55:34.852 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-24T07:55:34.890 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-24T07:55:34.934 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - CRB 4.6 MB/s | 6.2 MB 00:01 2024-04-24T07:55:34.944 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-24T07:55:34.978 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-24T07:55:35.020 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-24T07:55:35.062 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-24T07:55:35.390 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-24T07:55:35.496 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T07:55:35.617 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T07:55:35.647 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T07:55:35.738 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T07:55:35.776 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 10/156 2024-04-24T07:55:35.925 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-24T07:55:35.973 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-24T07:55:36.001 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T07:55:36.088 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T07:55:36.110 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 14/156 2024-04-24T07:55:36.156 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 14/156 2024-04-24T07:55:36.178 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T07:55:36.204 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T07:55:36.394 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T07:55:36.427 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T07:55:36.540 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-24T07:55:36.631 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-24T07:55:36.668 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-24T07:55:36.703 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-24T07:55:36.735 INFO:teuthology.orchestra.run.smithi028.stdout:CentOS Stream 9 - Extras packages 37 kB/s | 16 kB 00:00 2024-04-24T07:55:36.786 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-24T07:55:36.843 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-24T07:55:36.889 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-24T07:55:36.922 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-24T07:55:36.948 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-24T07:55:37.007 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-24T07:55:37.061 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-24T07:55:37.092 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-24T07:55:37.137 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-24T07:55:37.212 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-24T07:55:37.246 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9. 31/156 2024-04-24T07:55:37.289 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el 32/156 2024-04-24T07:55:37.318 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_6 33/156 2024-04-24T07:55:37.388 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-24T07:55:37.441 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-24T07:55:37.515 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-24T07:55:37.648 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-24T07:55:37.714 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-24T07:55:37.892 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-04-24T07:55:37.981 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-24T07:55:38.019 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-24T07:55:38.073 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-24T07:55:38.115 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-24T07:55:38.141 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-24T07:55:38.184 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-24T07:55:38.229 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-24T07:55:38.314 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-24T07:55:38.343 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-24T07:55:38.411 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-24T07:55:38.450 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_6 43/156 2024-04-24T07:55:38.491 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-24T07:55:38.512 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-24T07:55:38.567 INFO:teuthology.orchestra.run.smithi028.stdout:Extra Packages for Enterprise Linux 13 MB/s | 21 MB 00:01 2024-04-24T07:55:38.625 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T07:55:38.658 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-24T07:55:38.735 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-24T07:55:38.746 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T07:55:38.776 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T07:55:38.790 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-24T07:55:38.849 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T07:55:38.862 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-24T07:55:38.880 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 10/156 2024-04-24T07:55:38.894 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-24T07:55:38.930 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-24T07:55:38.977 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-24T07:55:39.013 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-24T07:55:39.032 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-24T07:55:39.056 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-24T07:55:39.077 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-24T07:55:39.098 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-24T07:55:39.105 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T07:55:39.207 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T07:55:39.231 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 14/156 2024-04-24T07:55:39.234 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-24T07:55:39.293 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 14/156 2024-04-24T07:55:39.295 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-24T07:55:39.315 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T07:55:39.317 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-24T07:55:39.337 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T07:55:39.358 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-24T07:55:39.452 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-24T07:55:39.484 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-24T07:55:39.524 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-24T07:55:39.531 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T07:55:39.549 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-24T07:55:39.564 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T07:55:39.573 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-24T07:55:39.648 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-24T07:55:39.672 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-24T07:55:39.760 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-24T07:55:39.774 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-24T07:55:39.797 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-24T07:55:39.806 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-24T07:55:39.825 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-24T07:55:39.832 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-24T07:55:39.860 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-24T07:55:39.896 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-24T07:55:39.930 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-24T07:55:39.984 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-24T07:55:39.991 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-24T07:55:40.015 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-24T07:55:40.043 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-24T07:55:40.113 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-24T07:55:40.156 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-24T07:55:40.175 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-24T07:55:40.186 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-24T07:55:40.205 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-24T07:55:40.220 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-24T07:55:40.248 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-24T07:55:40.253 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-24T07:55:40.296 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-24T07:55:40.329 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-24T07:55:40.343 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-24T07:55:40.376 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9. 31/156 2024-04-24T07:55:40.410 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-24T07:55:40.419 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el 32/156 2024-04-24T07:55:40.448 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_6 33/156 2024-04-24T07:55:40.460 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-24T07:55:40.491 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-24T07:55:40.519 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-24T07:55:40.538 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-24T07:55:40.563 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-24T07:55:40.636 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-24T07:55:40.637 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-24T07:55:40.684 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-24T07:55:40.783 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-24T07:55:40.840 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-24T07:55:41.378 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-24T07:55:41.413 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-24T07:55:41.456 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-24T07:55:41.460 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-24T07:55:41.489 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-24T07:55:41.490 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-24T07:55:41.531 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-24T07:55:41.570 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_6 43/156 2024-04-24T07:55:41.588 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-24T07:55:41.611 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-24T07:55:41.639 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-24T07:55:41.726 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-24T07:55:41.767 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-24T07:55:41.775 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-24T07:55:41.831 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-24T07:55:41.885 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-24T07:55:41.954 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-24T07:55:41.959 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-24T07:55:41.981 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-24T07:55:41.989 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-24T07:55:42.026 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-24T07:55:42.059 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-24T07:55:42.073 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-24T07:55:42.109 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-24T07:55:42.152 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-24T07:55:42.194 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-24T07:55:42.331 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-24T07:55:42.382 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-24T07:55:42.413 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-24T07:55:42.454 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-24T07:55:42.549 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-24T07:55:42.588 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-24T07:55:42.627 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-24T07:55:42.653 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-24T07:55:42.678 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-24T07:55:42.699 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-24T07:55:42.753 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-24T07:55:42.820 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-24T07:55:42.879 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-24T07:55:42.918 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-24T07:55:42.946 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-24T07:55:42.980 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-24T07:55:43.019 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-24T07:55:43.051 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-24T07:55:43.081 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-24T07:55:43.115 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-24T07:55:43.149 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-24T07:55:43.190 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-24T07:55:43.225 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-24T07:55:43.312 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-24T07:55:43.363 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-24T07:55:43.394 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-24T07:55:43.442 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-24T07:55:43.540 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-24T07:55:43.747 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-24T07:55:44.036 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-24T07:55:44.127 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-24T07:55:44.155 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-24T07:55:44.188 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-24T07:55:44.481 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-24T07:55:44.500 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-24T07:55:44.526 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-24T07:55:44.542 INFO:teuthology.orchestra.run.smithi117.stdout: Upgrading : librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 99/156 2024-04-24T07:55:44.559 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-24T07:55:44.616 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 99/156 2024-04-24T07:55:44.647 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 100/156 2024-04-24T07:55:44.660 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-24T07:55:44.682 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-24T07:55:44.709 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-24T07:55:44.782 INFO:teuthology.orchestra.run.smithi028.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-04-24T07:55:44.806 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-24T07:55:44.845 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-24T07:55:45.022 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-24T07:55:45.043 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-24T07:55:45.128 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-24T07:55:45.161 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-24T07:55:45.195 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 103/156 2024-04-24T07:55:45.266 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 103/156 2024-04-24T07:55:45.288 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 104/156 2024-04-24T07:55:45.784 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-24T07:55:45.908 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-24T07:55:47.149 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-24T07:55:47.220 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-24T07:55:47.249 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-24T07:55:47.280 INFO:teuthology.orchestra.run.smithi028.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T07:55:47.281 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-24T07:55:47.282 INFO:teuthology.orchestra.run.smithi028.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-04-24T07:55:47.303 INFO:teuthology.orchestra.run.smithi028.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-04-24T07:55:47.304 INFO:teuthology.orchestra.run.smithi028.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-04-24T07:55:47.305 INFO:teuthology.orchestra.run.smithi028.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-04-24T07:55:47.307 INFO:teuthology.orchestra.run.smithi028.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-04-24T07:55:47.309 INFO:teuthology.orchestra.run.smithi028.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-04-24T07:55:47.312 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-04-24T07:55:47.314 INFO:teuthology.orchestra.run.smithi028.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-04-24T07:55:47.388 INFO:teuthology.orchestra.run.smithi028.stdout:Dependencies resolved. 2024-04-24T07:55:48.127 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T07:55:48.127 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T07:55:48.128 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-24T07:55:48.128 INFO:teuthology.orchestra.run.smithi028.stdout: Package Arch Version Repository Size 2024-04-24T07:55:48.128 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-24T07:55:48.128 INFO:teuthology.orchestra.run.smithi028.stdout:Installing: 2024-04-24T07:55:48.128 INFO:teuthology.orchestra.run.smithi028.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-04-24T07:55:48.128 INFO:teuthology.orchestra.run.smithi028.stdout: ceph x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 6.6 k 2024-04-24T07:55:48.129 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 5.4 M 2024-04-24T07:55:48.129 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 890 k 2024-04-24T07:55:48.129 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 145 k 2024-04-24T07:55:48.129 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 1.5 M 2024-04-24T07:55:48.129 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 142 k 2024-04-24T07:55:48.129 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 3.5 M 2024-04-24T07:55:48.129 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 7.4 M 2024-04-24T07:55:48.129 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 49 k 2024-04-24T07:55:48.129 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 11 M 2024-04-24T07:55:48.129 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 49 M 2024-04-24T07:55:48.130 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 273 k 2024-04-24T07:55:48.130 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 762 k 2024-04-24T07:55:48.130 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 225 k 2024-04-24T07:55:48.130 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 27 k 2024-04-24T07:55:48.130 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 18 k 2024-04-24T07:55:48.130 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-04-24T07:55:48.130 INFO:teuthology.orchestra.run.smithi028.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-04-24T07:55:48.130 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-04-24T07:55:48.130 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 32 k 2024-04-24T07:55:48.130 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 746 k 2024-04-24T07:55:48.130 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 127 k 2024-04-24T07:55:48.131 INFO:teuthology.orchestra.run.smithi028.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-04-24T07:55:48.131 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-04-24T07:55:48.131 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-04-24T07:55:48.131 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-04-24T07:55:48.131 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-04-24T07:55:48.131 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-04-24T07:55:48.131 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 161 k 2024-04-24T07:55:48.131 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 320 k 2024-04-24T07:55:48.131 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 298 k 2024-04-24T07:55:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 99 k 2024-04-24T07:55:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 85 k 2024-04-24T07:55:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.1 M 2024-04-24T07:55:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 170 k 2024-04-24T07:55:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-04-24T07:55:48.132 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrading: 2024-04-24T07:55:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: librados2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.4 M 2024-04-24T07:55:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 3.1 M 2024-04-24T07:55:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-04-24T07:55:48.132 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-04-24T07:55:48.133 INFO:teuthology.orchestra.run.smithi028.stdout:Installing dependencies: 2024-04-24T07:55:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-04-24T07:55:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-04-24T07:55:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-04-24T07:55:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 21 M 2024-04-24T07:55:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 34 k 2024-04-24T07:55:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 2.4 M 2024-04-24T07:55:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 249 k 2024-04-24T07:55:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 4.7 M 2024-04-24T07:55:48.133 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 17 M 2024-04-24T07:55:48.134 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts noarch 2:19.0.0-3116.g1f81fb39.el9 ceph-noarch 16 k 2024-04-24T07:55:48.134 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 25 k 2024-04-24T07:55:48.134 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-04-24T07:55:48.134 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-04-24T07:55:48.134 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-04-24T07:55:48.134 INFO:teuthology.orchestra.run.smithi028.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-04-24T07:55:48.134 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-04-24T07:55:48.134 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-04-24T07:55:48.134 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-04-24T07:55:48.134 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-04-24T07:55:48.134 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 163 k 2024-04-24T07:55:48.134 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-04-24T07:55:48.135 INFO:teuthology.orchestra.run.smithi028.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-04-24T07:55:48.135 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-04-24T07:55:48.135 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-04-24T07:55:48.135 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-04-24T07:55:48.135 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 501 k 2024-04-24T07:55:48.135 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-04-24T07:55:48.135 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2 x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 5.4 M 2024-04-24T07:55:48.135 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-04-24T07:55:48.135 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-04-24T07:55:48.136 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-04-24T07:55:48.136 INFO:teuthology.orchestra.run.smithi028.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-04-24T07:55:48.136 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-04-24T07:55:48.136 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-04-24T07:55:48.136 INFO:teuthology.orchestra.run.smithi028.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-04-24T07:55:48.136 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-04-24T07:55:48.136 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-04-24T07:55:48.136 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-04-24T07:55:48.136 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-04-24T07:55:48.136 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-04-24T07:55:48.136 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-04-24T07:55:48.137 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-04-24T07:55:48.137 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-04-24T07:55:48.137 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-04-24T07:55:48.137 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-04-24T07:55:48.137 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 45 k 2024-04-24T07:55:48.137 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common x86_64 2:19.0.0-3116.g1f81fb39.el9 ceph 135 k 2024-04-24T07:55:48.137 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-04-24T07:55:48.137 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-04-24T07:55:48.137 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-04-24T07:55:48.137 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-04-24T07:55:48.137 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-04-24T07:55:48.138 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-04-24T07:55:48.138 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-04-24T07:55:48.138 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-04-24T07:55:48.138 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-04-24T07:55:48.138 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-04-24T07:55:48.138 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-04-24T07:55:48.138 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-04-24T07:55:48.138 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-04-24T07:55:48.138 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-04-24T07:55:48.138 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-04-24T07:55:48.138 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-04-24T07:55:48.139 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-04-24T07:55:48.139 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-04-24T07:55:48.139 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-04-24T07:55:48.139 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-04-24T07:55:48.139 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-04-24T07:55:48.139 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-04-24T07:55:48.139 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-04-24T07:55:48.139 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-04-24T07:55:48.139 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-04-24T07:55:48.139 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-04-24T07:55:48.139 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-04-24T07:55:48.139 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-04-24T07:55:48.140 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-04-24T07:55:48.140 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-04-24T07:55:48.140 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-04-24T07:55:48.140 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-04-24T07:55:48.140 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-04-24T07:55:48.140 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-04-24T07:55:48.140 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-04-24T07:55:48.140 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-04-24T07:55:48.140 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-04-24T07:55:48.140 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-04-24T07:55:48.140 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-04-24T07:55:48.140 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-04-24T07:55:48.140 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-04-24T07:55:48.140 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-04-24T07:55:48.141 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-04-24T07:55:48.141 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-04-24T07:55:48.141 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-04-24T07:55:48.141 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-04-24T07:55:48.141 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-04-24T07:55:48.141 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-04-24T07:55:48.141 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-04-24T07:55:48.141 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-04-24T07:55:48.141 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-04-24T07:55:48.141 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-04-24T07:55:48.141 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-04-24T07:55:48.141 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-04-24T07:55:48.141 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-04-24T07:55:48.141 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-04-24T07:55:48.142 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-04-24T07:55:48.142 INFO:teuthology.orchestra.run.smithi028.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-04-24T07:55:48.142 INFO:teuthology.orchestra.run.smithi028.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-04-24T07:55:48.142 INFO:teuthology.orchestra.run.smithi028.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-04-24T07:55:48.142 INFO:teuthology.orchestra.run.smithi028.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-04-24T07:55:48.142 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-04-24T07:55:48.142 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-04-24T07:55:48.142 INFO:teuthology.orchestra.run.smithi028.stdout:Installing weak dependencies: 2024-04-24T07:55:48.142 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-04-24T07:55:48.142 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:55:48.142 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction Summary 2024-04-24T07:55:48.142 INFO:teuthology.orchestra.run.smithi028.stdout:======================================================================================= 2024-04-24T07:55:48.142 INFO:teuthology.orchestra.run.smithi028.stdout:Install 148 Packages 2024-04-24T07:55:48.143 INFO:teuthology.orchestra.run.smithi028.stdout:Upgrade 4 Packages 2024-04-24T07:55:48.143 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:55:48.143 INFO:teuthology.orchestra.run.smithi028.stdout:Total download size: 222 M 2024-04-24T07:55:48.143 INFO:teuthology.orchestra.run.smithi028.stdout:Downloading Packages: 2024-04-24T07:55:48.143 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-24T07:55:48.143 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 99/156 2024-04-24T07:55:48.143 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 99/156 2024-04-24T07:55:48.143 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 100/156 2024-04-24T07:55:48.143 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-24T07:55:48.175 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T07:55:48.213 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 106/156 2024-04-24T07:55:48.263 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-24T07:55:48.289 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 103/156 2024-04-24T07:55:48.305 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 106/156 2024-04-24T07:55:48.305 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-24T07:55:48.305 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-24T07:55:48.305 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-24T07:55:48.339 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 107/156 2024-04-24T07:55:48.360 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 103/156 2024-04-24T07:55:48.390 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 104/156 2024-04-24T07:55:48.906 INFO:teuthology.orchestra.run.smithi028.stdout:(1/152): ceph-19.0.0-3116.g1f81fb39.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-04-24T07:55:49.131 INFO:teuthology.orchestra.run.smithi028.stdout:(2/152): ceph-fuse-19.0.0-3116.g1f81fb39.el9.x8 3.9 MB/s | 890 kB 00:00 2024-04-24T07:55:49.189 INFO:teuthology.orchestra.run.smithi028.stdout:(3/152): ceph-immutable-object-cache-19.0.0-311 2.5 MB/s | 145 kB 00:00 2024-04-24T07:55:49.831 INFO:teuthology.orchestra.run.smithi028.stdout:(4/152): ceph-mds-19.0.0-3116.g1f81fb39.el9.x86 3.7 MB/s | 2.4 MB 00:00 2024-04-24T07:55:50.343 INFO:teuthology.orchestra.run.smithi028.stdout:(5/152): ceph-common-19.0.0-3116.g1f81fb39.el9. 12 MB/s | 21 MB 00:01 2024-04-24T07:55:50.440 INFO:teuthology.orchestra.run.smithi028.stdout:(6/152): ceph-mgr-19.0.0-3116.g1f81fb39.el9.x86 2.5 MB/s | 1.5 MB 00:00 2024-04-24T07:55:50.691 INFO:teuthology.orchestra.run.smithi028.stdout:(7/152): ceph-mon-19.0.0-3116.g1f81fb39.el9.x86 14 MB/s | 4.7 MB 00:00 2024-04-24T07:55:50.706 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T07:55:50.745 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T07:55:51.292 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T07:55:51.331 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 106/156 2024-04-24T07:55:51.422 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 106/156 2024-04-24T07:55:51.422 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-24T07:55:51.423 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-24T07:55:51.423 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T07:55:51.458 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 107/156 2024-04-24T07:55:51.507 INFO:teuthology.orchestra.run.smithi028.stdout:(8/152): ceph-radosgw-19.0.0-3116.g1f81fb39.el9 13 MB/s | 11 MB 00:00 2024-04-24T07:55:51.625 INFO:teuthology.orchestra.run.smithi028.stdout:(9/152): ceph-base-19.0.0-3116.g1f81fb39.el9.x8 1.8 MB/s | 5.4 MB 00:03 2024-04-24T07:55:51.651 INFO:teuthology.orchestra.run.smithi028.stdout:(10/152): ceph-selinux-19.0.0-3116.g1f81fb39.el 178 kB/s | 25 kB 00:00 2024-04-24T07:55:51.725 INFO:teuthology.orchestra.run.smithi028.stdout:(11/152): cephfs-mirror-19.0.0-3116.g1f81fb39.e 3.0 MB/s | 225 kB 00:00 2024-04-24T07:55:51.783 INFO:teuthology.orchestra.run.smithi028.stdout:(12/152): cephfs-shell-19.0.0-3116.g1f81fb39.el 473 kB/s | 27 kB 00:00 2024-04-24T07:55:51.842 INFO:teuthology.orchestra.run.smithi028.stdout:(13/152): libcephfs-devel-19.0.0-3116.g1f81fb39 554 kB/s | 32 kB 00:00 2024-04-24T07:55:51.942 INFO:teuthology.orchestra.run.smithi028.stdout:(14/152): libcephfs2-19.0.0-3116.g1f81fb39.el9. 7.4 MB/s | 746 kB 00:00 2024-04-24T07:55:52.009 INFO:teuthology.orchestra.run.smithi028.stdout:(15/152): libcephsqlite-19.0.0-3116.g1f81fb39.e 2.4 MB/s | 163 kB 00:00 2024-04-24T07:55:52.076 INFO:teuthology.orchestra.run.smithi028.stdout:(16/152): librados-devel-19.0.0-3116.g1f81fb39. 1.9 MB/s | 127 kB 00:00 2024-04-24T07:55:52.168 INFO:teuthology.orchestra.run.smithi028.stdout:(17/152): libradosstriper1-19.0.0-3116.g1f81fb3 5.4 MB/s | 501 kB 00:00 2024-04-24T07:55:52.747 INFO:teuthology.orchestra.run.smithi028.stdout:(18/152): ceph-osd-19.0.0-3116.g1f81fb39.el9.x8 7.3 MB/s | 17 MB 00:02 2024-04-24T07:55:52.861 INFO:teuthology.orchestra.run.smithi028.stdout:(19/152): librgw2-19.0.0-3116.g1f81fb39.el9.x86 7.8 MB/s | 5.4 MB 00:00 2024-04-24T07:55:52.886 INFO:teuthology.orchestra.run.smithi028.stdout:(20/152): python3-ceph-argparse-19.0.0-3116.g1f 319 kB/s | 45 kB 00:00 2024-04-24T07:55:52.919 INFO:teuthology.orchestra.run.smithi028.stdout:(21/152): python3-ceph-common-19.0.0-3116.g1f81 2.3 MB/s | 135 kB 00:00 2024-04-24T07:55:52.953 INFO:teuthology.orchestra.run.smithi028.stdout:(22/152): python3-cephfs-19.0.0-3116.g1f81fb39. 2.4 MB/s | 161 kB 00:00 2024-04-24T07:55:52.995 INFO:teuthology.orchestra.run.smithi028.stdout:(23/152): python3-rados-19.0.0-3116.g1f81fb39.e 4.2 MB/s | 320 kB 00:00 2024-04-24T07:55:53.054 INFO:teuthology.orchestra.run.smithi028.stdout:(24/152): python3-rbd-19.0.0-3116.g1f81fb39.el9 2.9 MB/s | 298 kB 00:00 2024-04-24T07:55:53.096 INFO:teuthology.orchestra.run.smithi028.stdout:(25/152): python3-rgw-19.0.0-3116.g1f81fb39.el9 1.0 MB/s | 99 kB 00:00 2024-04-24T07:55:53.129 INFO:teuthology.orchestra.run.smithi028.stdout:(26/152): rbd-fuse-19.0.0-3116.g1f81fb39.el9.x8 1.1 MB/s | 85 kB 00:00 2024-04-24T07:55:53.179 INFO:teuthology.orchestra.run.smithi028.stdout:(27/152): rbd-nbd-19.0.0-3116.g1f81fb39.el9.x86 3.4 MB/s | 170 kB 00:00 2024-04-24T07:55:53.221 INFO:teuthology.orchestra.run.smithi028.stdout:(28/152): ceph-grafana-dashboards-19.0.0-3116.g 833 kB/s | 34 kB 00:00 2024-04-24T07:55:53.272 INFO:teuthology.orchestra.run.smithi028.stdout:(29/152): ceph-mgr-cephadm-19.0.0-3116.g1f81fb3 2.8 MB/s | 142 kB 00:00 2024-04-24T07:55:53.698 INFO:teuthology.orchestra.run.smithi028.stdout:(30/152): rbd-mirror-19.0.0-3116.g1f81fb39.el9. 5.1 MB/s | 3.1 MB 00:00 2024-04-24T07:55:53.789 INFO:teuthology.orchestra.run.smithi028.stdout:(31/152): ceph-mgr-dashboard-19.0.0-3116.g1f81f 6.9 MB/s | 3.5 MB 00:00 2024-04-24T07:55:53.889 INFO:teuthology.orchestra.run.smithi028.stdout:(32/152): ceph-mgr-modules-core-19.0.0-3116.g1f 2.5 MB/s | 249 kB 00:00 2024-04-24T07:55:54.036 INFO:teuthology.orchestra.run.smithi028.stdout:(33/152): ceph-mgr-rook-19.0.0-3116.g1f81fb39.e 340 kB/s | 49 kB 00:00 2024-04-24T07:55:54.115 INFO:teuthology.orchestra.run.smithi028.stdout:(34/152): ceph-prometheus-alerts-19.0.0-3116.g1 211 kB/s | 16 kB 00:00 2024-04-24T07:55:54.207 INFO:teuthology.orchestra.run.smithi028.stdout:(35/152): ceph-volume-19.0.0-3116.g1f81fb39.el9 3.0 MB/s | 273 kB 00:00 2024-04-24T07:55:54.349 INFO:teuthology.orchestra.run.smithi028.stdout:(36/152): ceph-mgr-diskprediction-local-19.0.0- 11 MB/s | 7.4 MB 00:00 2024-04-24T07:55:54.407 INFO:teuthology.orchestra.run.smithi028.stdout:(37/152): cephadm-19.0.0-3116.g1f81fb39.el9.noa 3.7 MB/s | 762 kB 00:00 2024-04-24T07:55:54.432 INFO:teuthology.orchestra.run.smithi028.stdout:(38/152): cephfs-top-19.0.0-3116.g1f81fb39.el9. 221 kB/s | 18 kB 00:00 2024-04-24T07:55:54.541 INFO:teuthology.orchestra.run.smithi028.stdout:(39/152): fuse-2.9.9-15.el9.x86_64.rpm 607 kB/s | 80 kB 00:00 2024-04-24T07:55:54.608 INFO:teuthology.orchestra.run.smithi028.stdout:(40/152): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-04-24T07:55:54.650 INFO:teuthology.orchestra.run.smithi028.stdout:(41/152): libgfortran-11.4.1-3.el9.x86_64.rpm 3.6 MB/s | 803 kB 00:00 2024-04-24T07:55:54.684 INFO:teuthology.orchestra.run.smithi028.stdout:(42/152): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 253 kB 00:00 2024-04-24T07:55:54.734 INFO:teuthology.orchestra.run.smithi028.stdout:(43/152): python3-ply-3.11-14.el9.noarch.rpm 2.1 MB/s | 106 kB 00:00 2024-04-24T07:55:54.822 INFO:teuthology.orchestra.run.smithi028.stdout:(44/152): python3-cryptography-36.0.1-4.el9.x86 7.2 MB/s | 1.2 MB 00:00 2024-04-24T07:55:54.860 INFO:teuthology.orchestra.run.smithi028.stdout:(45/152): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 135 kB 00:00 2024-04-24T07:55:54.893 INFO:teuthology.orchestra.run.smithi028.stdout:(46/152): python3-requests-2.25.1-8.el9.noarch. 1.7 MB/s | 125 kB 00:00 2024-04-24T07:55:54.927 INFO:teuthology.orchestra.run.smithi028.stdout:(47/152): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 215 kB 00:00 2024-04-24T07:55:54.986 INFO:teuthology.orchestra.run.smithi028.stdout:(48/152): automake-1.16.2-8.el9.noarch.rpm 7.4 MB/s | 689 kB 00:00 2024-04-24T07:55:55.036 INFO:teuthology.orchestra.run.smithi028.stdout:(49/152): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 107 kB 00:00 2024-04-24T07:55:55.078 INFO:teuthology.orchestra.run.smithi028.stdout:(50/152): bison-3.7.4-5.el9.x86_64.rpm 6.2 MB/s | 943 kB 00:00 2024-04-24T07:55:55.120 INFO:teuthology.orchestra.run.smithi028.stdout:(51/152): elfutils-libelf-devel-0.190-2.el9.x86 260 kB/s | 21 kB 00:00 2024-04-24T07:55:55.162 INFO:teuthology.orchestra.run.smithi028.stdout:(52/152): flex-2.6.4-9.el9.x86_64.rpm 3.7 MB/s | 313 kB 00:00 2024-04-24T07:55:55.212 INFO:teuthology.orchestra.run.smithi028.stdout:(53/152): flexiblas-3.0.4-8.el9.x86_64.rpm 359 kB/s | 33 kB 00:00 2024-04-24T07:55:55.279 INFO:teuthology.orchestra.run.smithi028.stdout:(54/152): flexiblas-openblas-openmp-3.0.4-8.el9 277 kB/s | 18 kB 00:00 2024-04-24T07:55:55.346 INFO:teuthology.orchestra.run.smithi028.stdout:(55/152): libaio-devel-0.3.111-13.el9.x86_64.rp 169 kB/s | 11 kB 00:00 2024-04-24T07:55:55.396 INFO:teuthology.orchestra.run.smithi028.stdout:(56/152): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.2 MB/s | 160 kB 00:00 2024-04-24T07:55:55.463 INFO:teuthology.orchestra.run.smithi028.stdout:(57/152): librabbitmq-0.11.0-7.el9.x86_64.rpm 688 kB/s | 45 kB 00:00 2024-04-24T07:55:55.547 INFO:teuthology.orchestra.run.smithi028.stdout:(58/152): flexiblas-netlib-3.0.4-8.el9.x86_64.r 7.8 MB/s | 3.0 MB 00:00 2024-04-24T07:55:55.597 INFO:teuthology.orchestra.run.smithi028.stdout:(59/152): librdkafka-1.6.1-102.el9.x86_64.rpm 4.9 MB/s | 662 kB 00:00 2024-04-24T07:55:55.647 INFO:teuthology.orchestra.run.smithi028.stdout:(60/152): libtool-2.4.6-45.el9.x86_64.rpm 5.7 MB/s | 582 kB 00:00 2024-04-24T07:55:55.698 INFO:teuthology.orchestra.run.smithi028.stdout:(61/152): libuuid-devel-2.37.4-18.el9.x86_64.rp 256 kB/s | 25 kB 00:00 2024-04-24T07:55:55.765 INFO:teuthology.orchestra.run.smithi028.stdout:(62/152): libzstd-devel-1.5.1-2.el9.x86_64.rpm 409 kB/s | 47 kB 00:00 2024-04-24T07:55:55.798 INFO:teuthology.orchestra.run.smithi028.stdout:(63/152): lttng-ust-2.12.0-6.el9.x86_64.rpm 2.9 MB/s | 292 kB 00:00 2024-04-24T07:55:55.832 INFO:teuthology.orchestra.run.smithi028.stdout:(64/152): lua-5.4.4-4.el9.x86_64.rpm 2.8 MB/s | 188 kB 00:00 2024-04-24T07:55:55.882 INFO:teuthology.orchestra.run.smithi028.stdout:(65/152): openblas-0.3.21-2.el9.x86_64.rpm 423 kB/s | 35 kB 00:00 2024-04-24T07:55:56.066 INFO:teuthology.orchestra.run.smithi028.stdout:(66/152): openblas-openmp-0.3.21-2.el9.x86_64.r 20 MB/s | 4.7 MB 00:00 2024-04-24T07:55:56.149 INFO:teuthology.orchestra.run.smithi028.stdout:(67/152): openssl-devel-3.2.1-1.el9.x86_64.rpm 17 MB/s | 4.4 MB 00:00 2024-04-24T07:55:56.233 INFO:teuthology.orchestra.run.smithi028.stdout:(68/152): postgresql-contrib-13.14-1.el9.x86_64 11 MB/s | 911 kB 00:00 2024-04-24T07:55:56.283 INFO:teuthology.orchestra.run.smithi028.stdout:(69/152): postgresql-13.14-1.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-04-24T07:55:56.317 INFO:teuthology.orchestra.run.smithi028.stdout:(70/152): postgresql-private-libs-13.14-1.el9.x 1.6 MB/s | 133 kB 00:00 2024-04-24T07:55:56.409 INFO:teuthology.orchestra.run.smithi028.stdout:(71/152): protobuf-3.14.0-13.el9.x86_64.rpm 11 MB/s | 1.0 MB 00:00 2024-04-24T07:55:56.451 INFO:teuthology.orchestra.run.smithi028.stdout:(72/152): python3-attrs-20.3.0-7.el9.noarch.rpm 2.1 MB/s | 87 kB 00:00 2024-04-24T07:55:56.635 INFO:teuthology.orchestra.run.smithi028.stdout:(73/152): postgresql-server-13.14-1.el9.x86_64. 17 MB/s | 5.8 MB 00:00 2024-04-24T07:55:56.685 INFO:teuthology.orchestra.run.smithi028.stdout:(74/152): python3-devel-3.9.18-3.el9.x86_64.rpm 4.9 MB/s | 244 kB 00:00 2024-04-24T07:55:56.735 INFO:teuthology.orchestra.run.smithi028.stdout:(75/152): python3-jinja2-2.11.3-5.el9.noarch.rp 4.9 MB/s | 248 kB 00:00 2024-04-24T07:55:56.777 INFO:teuthology.orchestra.run.smithi028.stdout:(76/152): python3-jsonpatch-1.21-16.el9.noarch. 629 kB/s | 26 kB 00:00 2024-04-24T07:55:56.819 INFO:teuthology.orchestra.run.smithi028.stdout:(77/152): python3-jsonpointer-2.0-4.el9.noarch. 470 kB/s | 19 kB 00:00 2024-04-24T07:55:56.936 INFO:teuthology.orchestra.run.smithi028.stdout:(78/152): python3-babel-2.9.1-2.el9.noarch.rpm 12 MB/s | 6.0 MB 00:00 2024-04-24T07:55:56.970 INFO:teuthology.orchestra.run.smithi028.stdout:(79/152): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 172 kB 00:00 2024-04-24T07:55:56.995 INFO:teuthology.orchestra.run.smithi028.stdout:(80/152): python3-markupsafe-1.1.1-12.el9.x86_6 603 kB/s | 35 kB 00:00 2024-04-24T07:55:57.046 INFO:teuthology.orchestra.run.smithi028.stdout:(81/152): python3-numpy-f2py-1.20.1-5.el9.x86_6 5.0 MB/s | 250 kB 00:00 2024-04-24T07:55:57.096 INFO:teuthology.orchestra.run.smithi028.stdout:(82/152): python3-oauthlib-3.1.1-5.el9.noarch.r 4.4 MB/s | 222 kB 00:00 2024-04-24T07:55:57.129 INFO:teuthology.orchestra.run.smithi028.stdout:(83/152): python3-packaging-20.9-5.el9.noarch.r 2.3 MB/s | 77 kB 00:00 2024-04-24T07:55:57.171 INFO:teuthology.orchestra.run.smithi028.stdout:(84/152): python3-prettytable-0.7.2-27.el9.noar 1.0 MB/s | 42 kB 00:00 2024-04-24T07:55:57.847 INFO:teuthology.orchestra.run.smithi028.stdout:(85/152): ceph-test-19.0.0-3116.g1f81fb39.el9.x 7.9 MB/s | 49 MB 00:06 2024-04-24T07:55:57.880 INFO:teuthology.orchestra.run.smithi028.stdout:(86/152): python3-protobuf-3.14.0-13.el9.noarch 380 kB/s | 269 kB 00:00 2024-04-24T07:55:57.997 INFO:teuthology.orchestra.run.smithi028.stdout:(87/152): python3-numpy-1.20.1-5.el9.x86_64.rpm 5.0 MB/s | 5.1 MB 00:01 2024-04-24T07:55:58.031 INFO:teuthology.orchestra.run.smithi028.stdout:(88/152): python3-pyasn1-modules-0.4.8-6.el9.no 1.8 MB/s | 279 kB 00:00 2024-04-24T07:55:58.056 INFO:teuthology.orchestra.run.smithi028.stdout:(89/152): python3-pytz-2021.1-5.el9.noarch.rpm 885 kB/s | 51 kB 00:00 2024-04-24T07:55:58.081 INFO:teuthology.orchestra.run.smithi028.stdout:(90/152): python3-pyasn1-0.4.8-6.el9.noarch.rpm 681 kB/s | 159 kB 00:00 2024-04-24T07:55:58.107 INFO:teuthology.orchestra.run.smithi028.stdout:(91/152): python3-requests-oauthlib-1.3.0-12.el 718 kB/s | 54 kB 00:00 2024-04-24T07:55:58.132 INFO:teuthology.orchestra.run.smithi028.stdout:(92/152): python3-toml-0.10.2-6.el9.noarch.rpm 843 kB/s | 42 kB 00:00 2024-04-24T07:55:58.157 INFO:teuthology.orchestra.run.smithi028.stdout:(93/152): python3-wcwidth-0.2.5-8.el9.noarch.rp 897 kB/s | 44 kB 00:00 2024-04-24T07:55:58.199 INFO:teuthology.orchestra.run.smithi028.stdout:(94/152): qatlib-24.02.0-1.el9.x86_64.rpm 3.3 MB/s | 221 kB 00:00 2024-04-24T07:55:58.224 INFO:teuthology.orchestra.run.smithi028.stdout:(95/152): qatlib-service-24.02.0-1.el9.x86_64.r 543 kB/s | 36 kB 00:00 2024-04-24T07:55:58.249 INFO:teuthology.orchestra.run.smithi028.stdout:(96/152): qatzip-libs-1.2.0-1.el9.x86_64.rpm 960 kB/s | 47 kB 00:00 2024-04-24T07:55:58.300 INFO:teuthology.orchestra.run.smithi028.stdout:(97/152): socat-1.7.4.1-5.el9.x86_64.rpm 4.0 MB/s | 305 kB 00:00 2024-04-24T07:55:58.325 INFO:teuthology.orchestra.run.smithi028.stdout:(98/152): uuid-1.6.2-55.el9.x86_64.rpm 778 kB/s | 58 kB 00:00 2024-04-24T07:55:58.342 INFO:teuthology.orchestra.run.smithi028.stdout:(99/152): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 2.2 MB/s | 94 kB 00:00 2024-04-24T07:55:58.368 INFO:teuthology.orchestra.run.smithi028.stdout:(100/152): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.5 MB/s | 64 kB 00:00 2024-04-24T07:55:58.409 INFO:teuthology.orchestra.run.smithi028.stdout:(101/152): zlib-devel-1.2.11-41.el9.x86_64.rpm 672 kB/s | 45 kB 00:00 2024-04-24T07:55:58.476 INFO:teuthology.orchestra.run.smithi028.stdout:(102/152): lua-devel-5.4.4-4.el9.x86_64.rpm 207 kB/s | 22 kB 00:00 2024-04-24T07:55:58.777 INFO:teuthology.orchestra.run.smithi028.stdout:(103/152): python3-scipy-1.6.2-8.el9.x86_64.rpm 23 MB/s | 16 MB 00:00 2024-04-24T07:55:58.819 INFO:teuthology.orchestra.run.smithi028.stdout:(104/152): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-04-24T07:55:58.994 INFO:teuthology.orchestra.run.smithi028.stdout:(105/152): grpc-data-1.46.7-10.el9.noarch.rpm 112 kB/s | 19 kB 00:00 2024-04-24T07:55:59.160 INFO:teuthology.orchestra.run.smithi028.stdout:(106/152): gperftools-libs-2.9.1-2.el9.x86_64.r 809 kB/s | 309 kB 00:00 2024-04-24T07:55:59.301 INFO:teuthology.orchestra.run.smithi028.stdout:(107/152): abseil-cpp-20211102.0-3.el9.x86_64.r 666 kB/s | 548 kB 00:00 2024-04-24T07:55:59.337 INFO:teuthology.orchestra.run.smithi028.stdout:(108/152): libarrow-doc-9.0.0-10.el9.noarch.rpm 148 kB/s | 26 kB 00:00 2024-04-24T07:55:59.537 INFO:teuthology.orchestra.run.smithi028.stdout:(109/152): liboath-2.6.7-2.el9.x86_64.rpm 207 kB/s | 49 kB 00:00 2024-04-24T07:55:59.579 INFO:teuthology.orchestra.run.smithi028.stdout:(110/152): libunwind-1.6.2-1.el9.x86_64.rpm 279 kB/s | 67 kB 00:00 2024-04-24T07:55:59.719 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 107/156 2024-04-24T07:55:59.720 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /sys 2024-04-24T07:55:59.720 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /proc 2024-04-24T07:55:59.720 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /mnt 2024-04-24T07:55:59.720 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /var/tmp 2024-04-24T07:55:59.720 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /home 2024-04-24T07:55:59.720 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /root 2024-04-24T07:55:59.720 INFO:teuthology.orchestra.run.smithi117.stdout:skipping the directory /tmp 2024-04-24T07:55:59.720 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-24T07:55:59.846 INFO:teuthology.orchestra.run.smithi028.stdout:(111/152): luarocks-3.9.2-1.el9.noarch.rpm 491 kB/s | 151 kB 00:00 2024-04-24T07:55:59.891 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 108/156 2024-04-24T07:55:59.907 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 108/156 2024-04-24T07:55:59.907 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:55:59.907 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-24T07:55:59.907 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-24T07:56:00.246 INFO:teuthology.orchestra.run.smithi028.stdout:(112/152): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-04-24T07:56:00.455 INFO:teuthology.orchestra.run.smithi028.stdout:(113/152): libarrow-9.0.0-10.el9.x86_64.rpm 3.0 MB/s | 4.4 MB 00:01 2024-04-24T07:56:00.480 INFO:teuthology.orchestra.run.smithi028.stdout:(114/152): python3-bcrypt-3.2.2-1.el9.x86_64.rp 187 kB/s | 43 kB 00:00 2024-04-24T07:56:00.513 INFO:teuthology.orchestra.run.smithi028.stdout:(115/152): python3-asyncssh-2.13.2-5.el9.noarch 823 kB/s | 548 kB 00:00 2024-04-24T07:56:00.639 INFO:teuthology.orchestra.run.smithi028.stdout:(116/152): python3-certifi-2021.10.8-2.el9.noar 95 kB/s | 15 kB 00:00 2024-04-24T07:56:00.664 INFO:teuthology.orchestra.run.smithi028.stdout:(117/152): python3-cachetools-4.2.4-1.el9.noarc 155 kB/s | 32 kB 00:00 2024-04-24T07:56:00.798 INFO:teuthology.orchestra.run.smithi028.stdout:(118/152): python3-cheroot-8.6.0-4.el9.noarch.r 606 kB/s | 172 kB 00:00 2024-04-24T07:56:00.964 INFO:teuthology.orchestra.run.smithi028.stdout:(119/152): python3-cmd2-2.4.2-2.el9.noarch.rpm 1.0 MB/s | 303 kB 00:00 2024-04-24T07:56:00.990 INFO:teuthology.orchestra.run.smithi028.stdout:(120/152): python3-colorama-0.4.4-12.el9.noarch 184 kB/s | 35 kB 00:00 2024-04-24T07:56:01.023 INFO:teuthology.orchestra.run.smithi028.stdout:(121/152): python3-cherrypy-18.6.1-2.el9.noarch 935 kB/s | 358 kB 00:00 2024-04-24T07:56:01.265 INFO:teuthology.orchestra.run.smithi028.stdout:(122/152): python3-google-auth-2.29.0-1.el9.noa 686 kB/s | 206 kB 00:00 2024-04-24T07:56:01.316 INFO:teuthology.orchestra.run.smithi028.stdout:(123/152): python3-grpcio-tools-1.46.7-10.el9.x 496 kB/s | 144 kB 00:00 2024-04-24T07:56:01.361 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 109/156 2024-04-24T07:56:01.382 INFO:teuthology.orchestra.run.smithi028.stdout:(124/152): python3-jaraco-8.2.1-3.el9.noarch.rp 92 kB/s | 11 kB 00:00 2024-04-24T07:56:01.387 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 109/156 2024-04-24T07:56:01.387 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:01.387 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-24T07:56:01.387 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T07:56:01.387 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T07:56:01.387 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-24T07:56:01.466 INFO:teuthology.orchestra.run.smithi028.stdout:(125/152): python3-jaraco-classes-3.2.1-5.el9.n 118 kB/s | 18 kB 00:00 2024-04-24T07:56:01.575 INFO:teuthology.orchestra.run.smithi028.stdout:(126/152): python3-jaraco-collections-3.0.0-8.e 121 kB/s | 23 kB 00:00 2024-04-24T07:56:01.608 INFO:teuthology.orchestra.run.smithi028.stdout:(127/152): python3-jaraco-functools-3.5.0-2.el9 138 kB/s | 19 kB 00:00 2024-04-24T07:56:01.620 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 110/156 2024-04-24T07:56:01.642 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 110/156 2024-04-24T07:56:01.643 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:01.643 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-24T07:56:01.643 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T07:56:01.643 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T07:56:01.643 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-24T07:56:01.850 INFO:teuthology.orchestra.run.smithi028.stdout:(128/152): python3-jaraco-text-3.2.0-6.el9.noar 72 kB/s | 20 kB 00:00 2024-04-24T07:56:02.017 INFO:teuthology.orchestra.run.smithi028.stdout:(129/152): python3-grpcio-1.46.7-10.el9.x86_64. 2.0 MB/s | 2.0 MB 00:01 2024-04-24T07:56:02.087 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 111/156 2024-04-24T07:56:02.110 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 111/156 2024-04-24T07:56:02.110 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:02.110 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-24T07:56:02.110 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T07:56:02.110 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T07:56:02.110 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-24T07:56:02.184 INFO:teuthology.orchestra.run.smithi028.stdout:(130/152): python3-logutils-0.3.5-21.el9.noarch 139 kB/s | 46 kB 00:00 2024-04-24T07:56:02.217 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-24T07:56:02.266 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-24T07:56:02.321 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-24T07:56:02.409 INFO:teuthology.orchestra.run.smithi028.stdout:(131/152): python3-more-itertools-8.12.0-2.el9. 201 kB/s | 79 kB 00:00 2024-04-24T07:56:02.452 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 107/156 2024-04-24T07:56:02.452 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /sys 2024-04-24T07:56:02.452 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /proc 2024-04-24T07:56:02.452 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /mnt 2024-04-24T07:56:02.452 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /var/tmp 2024-04-24T07:56:02.452 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /home 2024-04-24T07:56:02.452 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /root 2024-04-24T07:56:02.452 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /tmp 2024-04-24T07:56:02.452 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T07:56:02.485 INFO:teuthology.orchestra.run.smithi028.stdout:(132/152): python3-natsort-7.1.1-5.el9.noarch.r 193 kB/s | 58 kB 00:00 2024-04-24T07:56:02.506 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-24T07:56:02.563 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-24T07:56:02.626 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 108/156 2024-04-24T07:56:02.640 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-24T07:56:02.642 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 108/156 2024-04-24T07:56:02.642 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:02.642 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-24T07:56:02.642 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T07:56:02.677 INFO:teuthology.orchestra.run.smithi028.stdout:(133/152): python3-kubernetes-26.1.0-2.el9.noar 979 kB/s | 1.0 MB 00:01 2024-04-24T07:56:02.710 INFO:teuthology.orchestra.run.smithi028.stdout:(134/152): python3-portend-3.1.0-2.el9.noarch.r 73 kB/s | 16 kB 00:00 2024-04-24T07:56:02.729 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T07:56:02.869 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T07:56:02.912 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-24T07:56:02.946 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-24T07:56:02.961 INFO:teuthology.orchestra.run.smithi028.stdout:(135/152): python3-pecan-1.4.2-3.el9.noarch.rpm 495 kB/s | 272 kB 00:00 2024-04-24T07:56:02.972 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-24T07:56:02.986 INFO:teuthology.orchestra.run.smithi028.stdout:(136/152): python3-pyperclip-1.8.2-2.el9.noarch 103 kB/s | 28 kB 00:00 2024-04-24T07:56:03.045 INFO:teuthology.orchestra.run.smithi028.stdout:(137/152): python3-pyOpenSSL-21.0.0-1.el9.noarc 245 kB/s | 90 kB 00:00 2024-04-24T07:56:03.137 INFO:teuthology.orchestra.run.smithi028.stdout:(138/152): python3-repoze-lru-0.7-16.el9.noarch 176 kB/s | 31 kB 00:00 2024-04-24T07:56:03.270 INFO:teuthology.orchestra.run.smithi028.stdout:(139/152): python3-routes-2.5.1-5.el9.noarch.rp 666 kB/s | 188 kB 00:00 2024-04-24T07:56:03.295 INFO:teuthology.orchestra.run.smithi028.stdout:(140/152): python3-rsa-4.9-2.el9.noarch.rpm 237 kB/s | 59 kB 00:00 2024-04-24T07:56:03.329 INFO:teuthology.orchestra.run.smithi028.stdout:(141/152): python3-tempora-5.0.0-2.el9.noarch.r 187 kB/s | 36 kB 00:00 2024-04-24T07:56:03.425 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 123/156 2024-04-24T07:56:03.454 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 123/156 2024-04-24T07:56:03.486 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.e 124/156 2024-04-24T07:56:03.512 INFO:teuthology.orchestra.run.smithi028.stdout:(142/152): python3-typing-extensions-4.4.0-2.el 211 kB/s | 51 kB 00:00 2024-04-24T07:56:03.556 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39. 125/156 2024-04-24T07:56:03.585 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 126/156 2024-04-24T07:56:03.629 INFO:teuthology.orchestra.run.smithi028.stdout:(143/152): python3-webob-1.8.7-6.el9.noarch.rpm 689 kB/s | 230 kB 00:00 2024-04-24T07:56:03.805 INFO:teuthology.orchestra.run.smithi028.stdout:(144/152): python3-zc-lockfile-2.0-10.el9.noarc 114 kB/s | 20 kB 00:00 2024-04-24T07:56:03.847 INFO:teuthology.orchestra.run.smithi028.stdout:(145/152): python3-websocket-client-1.2.3-2.el9 174 kB/s | 90 kB 00:00 2024-04-24T07:56:03.947 INFO:teuthology.orchestra.run.smithi028.stdout:(146/152): python3-werkzeug-2.0.3-3.el9.1.noarc 985 kB/s | 427 kB 00:00 2024-04-24T07:56:04.122 INFO:teuthology.orchestra.run.smithi028.stdout:(147/152): re2-20211101-3.el9.x86_64.rpm 606 kB/s | 192 kB 00:00 2024-04-24T07:56:04.146 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 109/156 2024-04-24T07:56:04.172 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 109/156 2024-04-24T07:56:04.172 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:04.172 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-24T07:56:04.172 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T07:56:04.172 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T07:56:04.172 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T07:56:04.280 INFO:teuthology.orchestra.run.smithi028.stdout:(148/152): librados2-19.0.0-3116.g1f81fb39.el9. 10 MB/s | 3.4 MB 00:00 2024-04-24T07:56:04.405 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 110/156 2024-04-24T07:56:04.427 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 110/156 2024-04-24T07:56:04.427 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:04.427 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-24T07:56:04.427 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T07:56:04.428 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T07:56:04.428 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T07:56:04.448 INFO:teuthology.orchestra.run.smithi028.stdout:(149/152): openssl-3.2.1-1.el9.x86_64.rpm 8.1 MB/s | 1.3 MB 00:00 2024-04-24T07:56:04.540 INFO:teuthology.orchestra.run.smithi028.stdout:(150/152): openssl-libs-3.2.1-1.el9.x86_64.rpm 26 MB/s | 2.4 MB 00:00 2024-04-24T07:56:04.630 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 126/156 2024-04-24T07:56:04.662 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 127/156 2024-04-24T07:56:04.723 INFO:teuthology.orchestra.run.smithi028.stdout:(151/152): librbd1-19.0.0-3116.g1f81fb39.el9.x8 5.2 MB/s | 3.1 MB 00:00 2024-04-24T07:56:04.872 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 111/156 2024-04-24T07:56:04.894 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 111/156 2024-04-24T07:56:04.894 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:04.895 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-24T07:56:04.895 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T07:56:04.895 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T07:56:04.895 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T07:56:04.957 INFO:teuthology.orchestra.run.smithi028.stdout:(152/152): thrift-0.15.0-2.el9.x86_64.rpm 1.4 MB/s | 1.6 MB 00:01 2024-04-24T07:56:04.965 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-24T07:56:04.965 INFO:teuthology.orchestra.run.smithi028.stdout:Total 13 MB/s | 222 MB 00:17 2024-04-24T07:56:05.002 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-24T07:56:05.051 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-24T07:56:05.106 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-24T07:56:05.302 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-24T07:56:05.347 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-24T07:56:05.426 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-24T07:56:05.515 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T07:56:05.629 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 127/156 2024-04-24T07:56:05.654 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 128/156 2024-04-24T07:56:05.663 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T07:56:05.706 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-24T07:56:05.738 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-24T07:56:05.755 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 128/156 2024-04-24T07:56:05.765 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-24T07:56:05.925 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el 129/156 2024-04-24T07:56:05.956 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 130/156 2024-04-24T07:56:05.979 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 130/156 2024-04-24T07:56:05.979 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:05.979 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-24T07:56:05.979 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T07:56:05.979 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T07:56:05.980 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-24T07:56:06.040 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 131/156 2024-04-24T07:56:06.051 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 131/156 2024-04-24T07:56:06.151 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 132/156 2024-04-24T07:56:06.197 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 133/156 2024-04-24T07:56:06.201 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 123/156 2024-04-24T07:56:06.230 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 123/156 2024-04-24T07:56:06.262 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.e 124/156 2024-04-24T07:56:06.332 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39. 125/156 2024-04-24T07:56:06.361 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 126/156 2024-04-24T07:56:07.121 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-24T07:56:07.151 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 135/156 2024-04-24T07:56:07.174 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 135/156 2024-04-24T07:56:07.174 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:07.174 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-24T07:56:07.174 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T07:56:07.174 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T07:56:07.174 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-24T07:56:07.218 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 136/156 2024-04-24T07:56:07.240 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 136/156 2024-04-24T07:56:07.240 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:07.240 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-24T07:56:07.241 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-24T07:56:07.285 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 137/156 2024-04-24T07:56:07.308 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 137/156 2024-04-24T07:56:07.308 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:07.308 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-24T07:56:07.308 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T07:56:07.308 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T07:56:07.308 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-24T07:56:07.407 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 126/156 2024-04-24T07:56:07.447 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 127/156 2024-04-24T07:56:07.611 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 138/156 2024-04-24T07:56:07.637 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 138/156 2024-04-24T07:56:07.637 INFO:teuthology.orchestra.run.smithi117.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:07.637 INFO:teuthology.orchestra.run.smithi117.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-24T07:56:07.637 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T07:56:07.637 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T07:56:07.637 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-24T07:56:07.951 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction check 2024-04-24T07:56:08.206 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction check succeeded. 2024-04-24T07:56:08.206 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction test 2024-04-24T07:56:08.411 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 127/156 2024-04-24T07:56:08.432 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 128/156 2024-04-24T07:56:08.535 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 128/156 2024-04-24T07:56:08.702 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el 129/156 2024-04-24T07:56:08.732 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 130/156 2024-04-24T07:56:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 130/156 2024-04-24T07:56:08.755 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:08.755 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-24T07:56:08.755 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T07:56:08.755 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T07:56:08.755 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T07:56:08.799 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 131/156 2024-04-24T07:56:08.811 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 131/156 2024-04-24T07:56:08.909 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 132/156 2024-04-24T07:56:08.956 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 133/156 2024-04-24T07:56:09.899 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-24T07:56:09.927 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 135/156 2024-04-24T07:56:09.950 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 135/156 2024-04-24T07:56:09.950 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:09.950 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-24T07:56:09.950 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T07:56:09.950 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T07:56:09.950 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T07:56:09.994 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 136/156 2024-04-24T07:56:10.017 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 136/156 2024-04-24T07:56:10.017 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:10.017 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-24T07:56:10.017 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T07:56:10.061 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 137/156 2024-04-24T07:56:10.084 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 137/156 2024-04-24T07:56:10.084 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:10.084 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-24T07:56:10.084 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T07:56:10.084 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T07:56:10.084 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T07:56:10.378 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 138/156 2024-04-24T07:56:10.402 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 138/156 2024-04-24T07:56:10.402 INFO:teuthology.orchestra.run.smithi194.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:10.402 INFO:teuthology.orchestra.run.smithi194.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-24T07:56:10.402 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T07:56:10.402 INFO:teuthology.orchestra.run.smithi194.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T07:56:10.402 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T07:56:10.981 INFO:teuthology.orchestra.run.smithi028.stdout:Transaction test succeeded. 2024-04-24T07:56:10.981 INFO:teuthology.orchestra.run.smithi028.stdout:Running transaction 2024-04-24T07:56:12.206 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 139/156 2024-04-24T07:56:12.247 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 140/156 2024-04-24T07:56:12.288 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 141/156 2024-04-24T07:56:12.320 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-24T07:56:12.347 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-24T07:56:12.531 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 144/156 2024-04-24T07:56:12.576 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-24T07:56:12.606 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_ 146/156 2024-04-24T07:56:13.229 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T07:56:13.271 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T07:56:13.291 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T07:56:14.119 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 148/156 2024-04-24T07:56:14.209 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-24T07:56:14.250 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-24T07:56:14.330 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-24T07:56:14.363 INFO:teuthology.orchestra.run.smithi117.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-24T07:56:14.363 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T07:56:14.446 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T07:56:14.480 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T07:56:14.480 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T07:56:14.569 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T07:56:14.635 INFO:teuthology.orchestra.run.smithi117.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T07:56:14.990 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 139/156 2024-04-24T07:56:15.031 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 140/156 2024-04-24T07:56:15.073 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 141/156 2024-04-24T07:56:15.104 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-24T07:56:15.131 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-24T07:56:15.322 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 144/156 2024-04-24T07:56:15.369 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-24T07:56:15.402 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_ 146/156 2024-04-24T07:56:16.025 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T07:56:16.071 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T07:56:16.092 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T07:56:16.930 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 148/156 2024-04-24T07:56:17.017 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-24T07:56:17.058 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-24T07:56:17.142 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-24T07:56:17.171 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-24T07:56:17.171 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T07:56:17.263 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T07:56:17.305 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T07:56:17.305 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T07:56:17.340 INFO:teuthology.orchestra.run.smithi028.stdout: Preparing : 1/1 2024-04-24T07:56:17.392 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T07:56:17.441 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/156 2024-04-24T07:56:17.460 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T07:56:17.586 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/156 2024-04-24T07:56:17.810 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/156 2024-04-24T07:56:17.877 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/156 2024-04-24T07:56:17.916 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/156 2024-04-24T07:56:18.062 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/156 2024-04-24T07:56:18.524 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/156 2024-04-24T07:56:18.621 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T07:56:18.812 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T07:56:18.839 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T07:56:18.971 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T07:56:19.030 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 10/156 2024-04-24T07:56:19.391 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/156 2024-04-24T07:56:19.462 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/156 2024-04-24T07:56:19.493 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T07:56:19.662 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T07:56:19.694 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 14/156 2024-04-24T07:56:19.802 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 14/156 2024-04-24T07:56:19.837 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T07:56:19.920 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/156 2024-04-24T07:56:20.230 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T07:56:20.266 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/156 2024-04-24T07:56:20.454 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : re2-1:20211101-3.el9.x86_64 17/156 2024-04-24T07:56:20.544 INFO:teuthology.orchestra.run.smithi117.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T07:56:20.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 1/156 2024-04-24T07:56:20.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 2/156 2024-04-24T07:56:20.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 3/156 2024-04-24T07:56:20.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 4/156 2024-04-24T07:56:20.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 5/156 2024-04-24T07:56:20.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 6/156 2024-04-24T07:56:20.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 7/156 2024-04-24T07:56:20.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T07:56:20.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T07:56:20.544 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 10/156 2024-04-24T07:56:20.545 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 11/156 2024-04-24T07:56:20.545 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 12/156 2024-04-24T07:56:20.545 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T07:56:20.545 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 14/156 2024-04-24T07:56:20.545 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_ 15/156 2024-04-24T07:56:20.545 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 16/156 2024-04-24T07:56:20.545 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 17/156 2024-04-24T07:56:20.545 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_6 18/156 2024-04-24T07:56:20.545 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 19/156 2024-04-24T07:56:20.545 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 20/156 2024-04-24T07:56:20.545 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el 21/156 2024-04-24T07:56:20.545 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9. 22/156 2024-04-24T07:56:20.545 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_6 23/156 2024-04-24T07:56:20.545 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 24/156 2024-04-24T07:56:20.546 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 25/156 2024-04-24T07:56:20.546 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 26/156 2024-04-24T07:56:20.546 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 27/156 2024-04-24T07:56:20.546 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 28/156 2024-04-24T07:56:20.546 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 29/156 2024-04-24T07:56:20.546 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39. 30/156 2024-04-24T07:56:20.546 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 31/156 2024-04-24T07:56:20.546 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 32/156 2024-04-24T07:56:20.546 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 33/156 2024-04-24T07:56:20.546 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el 34/156 2024-04-24T07:56:20.546 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 35/156 2024-04-24T07:56:20.546 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.e 36/156 2024-04-24T07:56:20.547 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 37/156 2024-04-24T07:56:20.547 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 38/156 2024-04-24T07:56:20.547 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 39/156 2024-04-24T07:56:20.547 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-24T07:56:20.547 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-24T07:56:20.547 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-24T07:56:20.547 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-24T07:56:20.547 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-24T07:56:20.547 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-24T07:56:20.547 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-24T07:56:20.547 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-24T07:56:20.547 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-24T07:56:20.547 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-24T07:56:20.548 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-24T07:56:20.548 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-24T07:56:20.548 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-24T07:56:20.548 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-24T07:56:20.548 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-24T07:56:20.548 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-24T07:56:20.548 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-24T07:56:20.548 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-24T07:56:20.548 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-24T07:56:20.548 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-24T07:56:20.548 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-24T07:56:20.548 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-24T07:56:20.548 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-24T07:56:20.548 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-24T07:56:20.549 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-24T07:56:20.549 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-24T07:56:20.549 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-24T07:56:20.549 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-24T07:56:20.549 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-24T07:56:20.549 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-24T07:56:20.549 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-24T07:56:20.549 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-24T07:56:20.549 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-24T07:56:20.549 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-24T07:56:20.549 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-24T07:56:20.549 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-24T07:56:20.549 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-24T07:56:20.549 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-24T07:56:20.550 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-24T07:56:20.550 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-24T07:56:20.550 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-24T07:56:20.550 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-24T07:56:20.550 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-24T07:56:20.550 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-24T07:56:20.550 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-24T07:56:20.550 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-24T07:56:20.550 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-24T07:56:20.550 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-24T07:56:20.550 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-24T07:56:20.550 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-24T07:56:20.550 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-24T07:56:20.551 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-24T07:56:20.551 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-24T07:56:20.551 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-24T07:56:20.551 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-24T07:56:20.551 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-24T07:56:20.551 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-24T07:56:20.551 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-24T07:56:20.551 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-24T07:56:20.551 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-24T07:56:20.551 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-24T07:56:20.551 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-24T07:56:20.551 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-24T07:56:20.551 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-24T07:56:20.551 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-24T07:56:20.551 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-24T07:56:20.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-24T07:56:20.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-24T07:56:20.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-24T07:56:20.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-24T07:56:20.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-24T07:56:20.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-24T07:56:20.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-24T07:56:20.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-24T07:56:20.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-24T07:56:20.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-24T07:56:20.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-24T07:56:20.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-24T07:56:20.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T07:56:20.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T07:56:20.552 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-24T07:56:20.553 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-24T07:56:20.553 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-24T07:56:20.553 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-24T07:56:20.553 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-24T07:56:20.553 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-24T07:56:20.553 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-24T07:56:20.553 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-24T07:56:20.553 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-24T07:56:20.553 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-24T07:56:20.553 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-24T07:56:20.553 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-24T07:56:20.553 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-24T07:56:20.553 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-24T07:56:20.553 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-24T07:56:20.553 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-24T07:56:20.553 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-24T07:56:20.554 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-24T07:56:20.554 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-24T07:56:20.554 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-24T07:56:20.554 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-24T07:56:20.554 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-24T07:56:20.554 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-24T07:56:20.554 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-24T07:56:20.554 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-24T07:56:20.554 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-24T07:56:20.554 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-24T07:56:20.554 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-24T07:56:20.554 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-24T07:56:20.554 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 149/156 2024-04-24T07:56:20.554 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-24T07:56:20.555 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 151/156 2024-04-24T07:56:20.555 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-24T07:56:20.555 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-24T07:56:20.555 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T07:56:20.555 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-24T07:56:20.576 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 18/156 2024-04-24T07:56:20.621 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 19/156 2024-04-24T07:56:20.670 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : liboath-2.6.7-2.el9.x86_64 20/156 2024-04-24T07:56:20.774 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/156 2024-04-24T07:56:20.842 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 22/156 2024-04-24T07:56:20.909 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 23/156 2024-04-24T07:56:20.957 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 24/156 2024-04-24T07:56:20.995 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-5.4.4-4.el9.x86_64 25/156 2024-04-24T07:56:21.117 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 26/156 2024-04-24T07:56:21.206 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 27/156 2024-04-24T07:56:21.247 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 28/156 2024-04-24T07:56:21.309 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/156 2024-04-24T07:56:21.428 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/156 2024-04-24T07:56:21.466 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9. 31/156 2024-04-24T07:56:21.523 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el 32/156 2024-04-24T07:56:21.559 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_6 33/156 2024-04-24T07:56:21.683 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 34/156 2024-04-24T07:56:21.748 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : luarocks-3.9.2-1.el9.noarch 35/156 2024-04-24T07:56:21.862 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 36/156 2024-04-24T07:56:22.049 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 37/156 2024-04-24T07:56:22.116 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 38/156 2024-04-24T07:56:22.611 INFO:teuthology.orchestra.run.smithi117.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T07:56:22.611 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-24T07:56:22.611 INFO:teuthology.orchestra.run.smithi117.stdout:Upgraded: 2024-04-24T07:56:22.611 INFO:teuthology.orchestra.run.smithi117.stdout: librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.611 INFO:teuthology.orchestra.run.smithi117.stdout: librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.611 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-24T07:56:22.611 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-24T07:56:22.612 INFO:teuthology.orchestra.run.smithi117.stdout:Installed: 2024-04-24T07:56:22.612 INFO:teuthology.orchestra.run.smithi117.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-24T07:56:22.612 INFO:teuthology.orchestra.run.smithi117.stdout: automake-1.16.2-8.el9.noarch 2024-04-24T07:56:22.612 INFO:teuthology.orchestra.run.smithi117.stdout: bison-3.7.4-5.el9.x86_64 2024-04-24T07:56:22.612 INFO:teuthology.orchestra.run.smithi117.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-24T07:56:22.612 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.612 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.612 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.612 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.612 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:22.612 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-immutable-object-cache-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.613 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.613 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.613 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:22.613 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:22.613 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:22.613 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:22.613 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:22.613 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.613 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.613 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:22.613 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.613 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.613 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.613 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:22.614 INFO:teuthology.orchestra.run.smithi117.stdout: cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:22.614 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.614 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.614 INFO:teuthology.orchestra.run.smithi117.stdout: cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:22.614 INFO:teuthology.orchestra.run.smithi117.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-24T07:56:22.614 INFO:teuthology.orchestra.run.smithi117.stdout: flex-2.6.4-9.el9.x86_64 2024-04-24T07:56:22.614 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-24T07:56:22.614 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-24T07:56:22.614 INFO:teuthology.orchestra.run.smithi117.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-24T07:56:22.614 INFO:teuthology.orchestra.run.smithi117.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-24T07:56:22.614 INFO:teuthology.orchestra.run.smithi117.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-24T07:56:22.614 INFO:teuthology.orchestra.run.smithi117.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-24T07:56:22.614 INFO:teuthology.orchestra.run.smithi117.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-24T07:56:22.615 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-24T07:56:22.615 INFO:teuthology.orchestra.run.smithi117.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-24T07:56:22.615 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.615 INFO:teuthology.orchestra.run.smithi117.stdout: libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.615 INFO:teuthology.orchestra.run.smithi117.stdout: libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.615 INFO:teuthology.orchestra.run.smithi117.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-24T07:56:22.615 INFO:teuthology.orchestra.run.smithi117.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-24T07:56:22.615 INFO:teuthology.orchestra.run.smithi117.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-24T07:56:22.615 INFO:teuthology.orchestra.run.smithi117.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-24T07:56:22.615 INFO:teuthology.orchestra.run.smithi117.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-24T07:56:22.615 INFO:teuthology.orchestra.run.smithi117.stdout: librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.615 INFO:teuthology.orchestra.run.smithi117.stdout: libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.616 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-24T07:56:22.616 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.616 INFO:teuthology.orchestra.run.smithi117.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-24T07:56:22.616 INFO:teuthology.orchestra.run.smithi117.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-24T07:56:22.616 INFO:teuthology.orchestra.run.smithi117.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-24T07:56:22.616 INFO:teuthology.orchestra.run.smithi117.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-24T07:56:22.616 INFO:teuthology.orchestra.run.smithi117.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-24T07:56:22.616 INFO:teuthology.orchestra.run.smithi117.stdout: lua-5.4.4-4.el9.x86_64 2024-04-24T07:56:22.616 INFO:teuthology.orchestra.run.smithi117.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-24T07:56:22.616 INFO:teuthology.orchestra.run.smithi117.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-24T07:56:22.616 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-24T07:56:22.617 INFO:teuthology.orchestra.run.smithi117.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-24T07:56:22.617 INFO:teuthology.orchestra.run.smithi117.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-24T07:56:22.617 INFO:teuthology.orchestra.run.smithi117.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-24T07:56:22.617 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-13.14-1.el9.x86_64 2024-04-24T07:56:22.617 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-24T07:56:22.617 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-24T07:56:22.617 INFO:teuthology.orchestra.run.smithi117.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-24T07:56:22.617 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-24T07:56:22.617 INFO:teuthology.orchestra.run.smithi117.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-24T07:56:22.617 INFO:teuthology.orchestra.run.smithi117.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-24T07:56:22.617 INFO:teuthology.orchestra.run.smithi117.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-24T07:56:22.617 INFO:teuthology.orchestra.run.smithi117.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-24T07:56:22.618 INFO:teuthology.orchestra.run.smithi117.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-24T07:56:22.618 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-24T07:56:22.618 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.618 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.618 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.618 INFO:teuthology.orchestra.run.smithi117.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-24T07:56:22.618 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-24T07:56:22.618 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-24T07:56:22.618 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-24T07:56:22.618 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-24T07:56:22.618 INFO:teuthology.orchestra.run.smithi117.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-24T07:56:22.618 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-24T07:56:22.618 INFO:teuthology.orchestra.run.smithi117.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-24T07:56:22.618 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-24T07:56:22.619 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-24T07:56:22.619 INFO:teuthology.orchestra.run.smithi117.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-24T07:56:22.619 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-24T07:56:22.619 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-24T07:56:22.619 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-24T07:56:22.619 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-24T07:56:22.619 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-24T07:56:22.619 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-24T07:56:22.619 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-24T07:56:22.619 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-24T07:56:22.619 INFO:teuthology.orchestra.run.smithi117.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-24T07:56:22.619 INFO:teuthology.orchestra.run.smithi117.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-24T07:56:22.619 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-24T07:56:22.620 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-24T07:56:22.620 INFO:teuthology.orchestra.run.smithi117.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-24T07:56:22.620 INFO:teuthology.orchestra.run.smithi117.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-24T07:56:22.620 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-24T07:56:22.620 INFO:teuthology.orchestra.run.smithi117.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-24T07:56:22.620 INFO:teuthology.orchestra.run.smithi117.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-24T07:56:22.620 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-24T07:56:22.620 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-24T07:56:22.620 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ply-3.11-14.el9.noarch 2024-04-24T07:56:22.620 INFO:teuthology.orchestra.run.smithi117.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-24T07:56:22.620 INFO:teuthology.orchestra.run.smithi117.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-24T07:56:22.620 INFO:teuthology.orchestra.run.smithi117.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-24T07:56:22.621 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-24T07:56:22.621 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-24T07:56:22.621 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-24T07:56:22.621 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-24T07:56:22.621 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-24T07:56:22.621 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-24T07:56:22.621 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.621 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.621 INFO:teuthology.orchestra.run.smithi117.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-24T07:56:22.621 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-24T07:56:22.621 INFO:teuthology.orchestra.run.smithi117.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-24T07:56:22.621 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.621 INFO:teuthology.orchestra.run.smithi117.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-24T07:56:22.621 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-24T07:56:22.621 INFO:teuthology.orchestra.run.smithi117.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-24T07:56:22.621 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-24T07:56:22.622 INFO:teuthology.orchestra.run.smithi117.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-24T07:56:22.622 INFO:teuthology.orchestra.run.smithi117.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-24T07:56:22.622 INFO:teuthology.orchestra.run.smithi117.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-24T07:56:22.622 INFO:teuthology.orchestra.run.smithi117.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-24T07:56:22.622 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-24T07:56:22.622 INFO:teuthology.orchestra.run.smithi117.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-24T07:56:22.622 INFO:teuthology.orchestra.run.smithi117.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-24T07:56:22.622 INFO:teuthology.orchestra.run.smithi117.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-24T07:56:22.622 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-24T07:56:22.622 INFO:teuthology.orchestra.run.smithi117.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-24T07:56:22.622 INFO:teuthology.orchestra.run.smithi117.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-24T07:56:22.622 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.622 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.622 INFO:teuthology.orchestra.run.smithi117.stdout: rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:22.622 INFO:teuthology.orchestra.run.smithi117.stdout: re2-1:20211101-3.el9.x86_64 2024-04-24T07:56:22.622 INFO:teuthology.orchestra.run.smithi117.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-24T07:56:22.623 INFO:teuthology.orchestra.run.smithi117.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-24T07:56:22.623 INFO:teuthology.orchestra.run.smithi117.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-24T07:56:22.623 INFO:teuthology.orchestra.run.smithi117.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-24T07:56:22.623 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-24T07:56:22.623 INFO:teuthology.orchestra.run.smithi117.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-24T07:56:22.623 INFO:teuthology.orchestra.run.smithi117.stdout: 2024-04-24T07:56:22.623 INFO:teuthology.orchestra.run.smithi117.stdout:Complete! 2024-04-24T07:56:22.854 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rsa-4.9-2.el9.noarch 39/156 2024-04-24T07:56:22.966 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 40/156 2024-04-24T07:56:22.973 DEBUG:teuthology.parallel:result is None 2024-04-24T07:56:23.006 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 41/156 2024-04-24T07:56:23.064 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T07:56:23.064 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 1/156 2024-04-24T07:56:23.064 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 2/156 2024-04-24T07:56:23.064 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 3/156 2024-04-24T07:56:23.064 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 4/156 2024-04-24T07:56:23.064 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 5/156 2024-04-24T07:56:23.064 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 6/156 2024-04-24T07:56:23.065 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 7/156 2024-04-24T07:56:23.065 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T07:56:23.065 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T07:56:23.065 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 10/156 2024-04-24T07:56:23.065 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 11/156 2024-04-24T07:56:23.065 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 12/156 2024-04-24T07:56:23.065 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T07:56:23.065 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 14/156 2024-04-24T07:56:23.065 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_ 15/156 2024-04-24T07:56:23.065 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 16/156 2024-04-24T07:56:23.065 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 17/156 2024-04-24T07:56:23.065 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_6 18/156 2024-04-24T07:56:23.066 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 19/156 2024-04-24T07:56:23.066 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 20/156 2024-04-24T07:56:23.066 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el 21/156 2024-04-24T07:56:23.066 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9. 22/156 2024-04-24T07:56:23.066 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_6 23/156 2024-04-24T07:56:23.066 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 24/156 2024-04-24T07:56:23.066 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 25/156 2024-04-24T07:56:23.066 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 26/156 2024-04-24T07:56:23.066 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 27/156 2024-04-24T07:56:23.066 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 28/156 2024-04-24T07:56:23.066 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 29/156 2024-04-24T07:56:23.067 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39. 30/156 2024-04-24T07:56:23.067 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 31/156 2024-04-24T07:56:23.067 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 32/156 2024-04-24T07:56:23.067 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 33/156 2024-04-24T07:56:23.067 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el 34/156 2024-04-24T07:56:23.067 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 35/156 2024-04-24T07:56:23.067 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.e 36/156 2024-04-24T07:56:23.067 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 37/156 2024-04-24T07:56:23.067 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 38/156 2024-04-24T07:56:23.067 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 39/156 2024-04-24T07:56:23.067 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-24T07:56:23.067 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-24T07:56:23.068 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-24T07:56:23.068 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-24T07:56:23.068 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-24T07:56:23.068 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-24T07:56:23.068 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-24T07:56:23.068 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-24T07:56:23.068 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-24T07:56:23.068 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-24T07:56:23.068 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-24T07:56:23.068 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-24T07:56:23.068 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-24T07:56:23.068 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-24T07:56:23.068 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-24T07:56:23.069 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-24T07:56:23.069 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-24T07:56:23.069 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-24T07:56:23.069 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-24T07:56:23.069 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-24T07:56:23.069 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-24T07:56:23.069 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-24T07:56:23.069 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-24T07:56:23.069 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-24T07:56:23.069 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-24T07:56:23.069 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-24T07:56:23.069 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-24T07:56:23.070 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-24T07:56:23.070 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-24T07:56:23.070 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-24T07:56:23.070 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-24T07:56:23.070 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-24T07:56:23.070 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-24T07:56:23.070 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-24T07:56:23.070 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-24T07:56:23.070 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-24T07:56:23.070 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-24T07:56:23.070 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-24T07:56:23.070 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-24T07:56:23.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-24T07:56:23.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-24T07:56:23.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-24T07:56:23.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-24T07:56:23.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-24T07:56:23.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-24T07:56:23.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-24T07:56:23.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-24T07:56:23.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-24T07:56:23.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-24T07:56:23.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-24T07:56:23.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-24T07:56:23.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-24T07:56:23.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-24T07:56:23.071 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-24T07:56:23.072 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-24T07:56:23.072 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-24T07:56:23.072 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-24T07:56:23.072 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-24T07:56:23.072 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-24T07:56:23.072 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-24T07:56:23.072 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-24T07:56:23.072 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-24T07:56:23.072 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-24T07:56:23.072 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-24T07:56:23.072 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-24T07:56:23.073 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-24T07:56:23.073 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-24T07:56:23.073 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-24T07:56:23.073 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-24T07:56:23.073 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-24T07:56:23.073 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-24T07:56:23.073 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-24T07:56:23.073 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-24T07:56:23.073 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-24T07:56:23.073 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-24T07:56:23.073 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-24T07:56:23.074 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-24T07:56:23.074 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-24T07:56:23.074 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T07:56:23.074 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T07:56:23.074 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-24T07:56:23.074 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-24T07:56:23.074 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-24T07:56:23.074 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-24T07:56:23.074 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-24T07:56:23.074 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-24T07:56:23.074 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-24T07:56:23.075 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-24T07:56:23.075 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-24T07:56:23.075 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-24T07:56:23.075 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-24T07:56:23.075 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-24T07:56:23.075 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-24T07:56:23.075 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-24T07:56:23.075 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-24T07:56:23.075 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-24T07:56:23.075 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-24T07:56:23.075 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-24T07:56:23.075 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-24T07:56:23.075 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-24T07:56:23.076 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-24T07:56:23.076 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-24T07:56:23.076 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-24T07:56:23.076 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-24T07:56:23.076 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-24T07:56:23.076 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-24T07:56:23.076 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-24T07:56:23.076 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-24T07:56:23.076 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-24T07:56:23.076 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 149/156 2024-04-24T07:56:23.076 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-24T07:56:23.076 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 151/156 2024-04-24T07:56:23.076 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-24T07:56:23.076 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-24T07:56:23.077 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T07:56:23.077 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-24T07:56:23.077 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 42/156 2024-04-24T07:56:23.112 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_6 43/156 2024-04-24T07:56:23.186 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 44/156 2024-04-24T07:56:23.492 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 45/156 2024-04-24T07:56:23.578 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 46/156 2024-04-24T07:56:23.663 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 47/156 2024-04-24T07:56:23.762 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 48/156 2024-04-24T07:56:23.800 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 49/156 2024-04-24T07:56:23.839 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 50/156 2024-04-24T07:56:23.911 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 51/156 2024-04-24T07:56:23.957 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 52/156 2024-04-24T07:56:24.022 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 53/156 2024-04-24T07:56:24.086 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 54/156 2024-04-24T07:56:24.311 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 55/156 2024-04-24T07:56:24.371 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 56/156 2024-04-24T07:56:24.426 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 57/156 2024-04-24T07:56:24.485 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 58/156 2024-04-24T07:56:24.631 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 59/156 2024-04-24T07:56:24.674 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 60/156 2024-04-24T07:56:24.732 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 61/156 2024-04-24T07:56:24.773 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 62/156 2024-04-24T07:56:24.807 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 63/156 2024-04-24T07:56:25.009 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 64/156 2024-04-24T07:56:25.139 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T07:56:25.140 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T07:56:25.140 INFO:teuthology.orchestra.run.smithi194.stdout:Upgraded: 2024-04-24T07:56:25.140 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.140 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.140 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-24T07:56:25.140 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-24T07:56:25.140 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-04-24T07:56:25.140 INFO:teuthology.orchestra.run.smithi194.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-24T07:56:25.140 INFO:teuthology.orchestra.run.smithi194.stdout: automake-1.16.2-8.el9.noarch 2024-04-24T07:56:25.140 INFO:teuthology.orchestra.run.smithi194.stdout: bison-3.7.4-5.el9.x86_64 2024-04-24T07:56:25.141 INFO:teuthology.orchestra.run.smithi194.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-24T07:56:25.141 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.141 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.141 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.141 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.141 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:25.141 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.141 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.141 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.141 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:25.141 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:25.141 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:25.142 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:25.142 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:25.142 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.142 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.142 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:25.142 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.142 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.142 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.142 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:25.142 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:25.142 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.143 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.143 INFO:teuthology.orchestra.run.smithi194.stdout: cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:56:25.143 INFO:teuthology.orchestra.run.smithi194.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-24T07:56:25.143 INFO:teuthology.orchestra.run.smithi194.stdout: flex-2.6.4-9.el9.x86_64 2024-04-24T07:56:25.143 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-24T07:56:25.143 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-24T07:56:25.143 INFO:teuthology.orchestra.run.smithi194.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-24T07:56:25.143 INFO:teuthology.orchestra.run.smithi194.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-24T07:56:25.143 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-24T07:56:25.143 INFO:teuthology.orchestra.run.smithi194.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-24T07:56:25.143 INFO:teuthology.orchestra.run.smithi194.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-24T07:56:25.143 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-24T07:56:25.143 INFO:teuthology.orchestra.run.smithi194.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-24T07:56:25.144 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.144 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.144 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.144 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-24T07:56:25.144 INFO:teuthology.orchestra.run.smithi194.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-24T07:56:25.144 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-24T07:56:25.144 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-24T07:56:25.144 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-24T07:56:25.144 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.144 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.144 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-24T07:56:25.144 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.144 INFO:teuthology.orchestra.run.smithi194.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-24T07:56:25.144 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-24T07:56:25.145 INFO:teuthology.orchestra.run.smithi194.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-24T07:56:25.145 INFO:teuthology.orchestra.run.smithi194.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-24T07:56:25.145 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-24T07:56:25.145 INFO:teuthology.orchestra.run.smithi194.stdout: lua-5.4.4-4.el9.x86_64 2024-04-24T07:56:25.145 INFO:teuthology.orchestra.run.smithi194.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-24T07:56:25.145 INFO:teuthology.orchestra.run.smithi194.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-24T07:56:25.145 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-24T07:56:25.145 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-24T07:56:25.145 INFO:teuthology.orchestra.run.smithi194.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-24T07:56:25.145 INFO:teuthology.orchestra.run.smithi194.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-24T07:56:25.145 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-13.14-1.el9.x86_64 2024-04-24T07:56:25.145 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-24T07:56:25.145 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-24T07:56:25.146 INFO:teuthology.orchestra.run.smithi194.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-24T07:56:25.146 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-24T07:56:25.146 INFO:teuthology.orchestra.run.smithi194.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-24T07:56:25.146 INFO:teuthology.orchestra.run.smithi194.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-24T07:56:25.146 INFO:teuthology.orchestra.run.smithi194.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-24T07:56:25.146 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-24T07:56:25.146 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-24T07:56:25.146 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-24T07:56:25.146 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.146 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.146 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.146 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-24T07:56:25.146 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-24T07:56:25.147 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-24T07:56:25.147 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-24T07:56:25.147 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-24T07:56:25.147 INFO:teuthology.orchestra.run.smithi194.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-24T07:56:25.147 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-24T07:56:25.147 INFO:teuthology.orchestra.run.smithi194.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-24T07:56:25.147 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-24T07:56:25.147 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-24T07:56:25.147 INFO:teuthology.orchestra.run.smithi194.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-24T07:56:25.147 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-24T07:56:25.147 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-24T07:56:25.147 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-24T07:56:25.147 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-24T07:56:25.147 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-24T07:56:25.148 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-24T07:56:25.148 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-24T07:56:25.148 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-24T07:56:25.148 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-24T07:56:25.148 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-24T07:56:25.148 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-24T07:56:25.148 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-24T07:56:25.148 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-24T07:56:25.148 INFO:teuthology.orchestra.run.smithi194.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-24T07:56:25.148 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-24T07:56:25.148 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-24T07:56:25.148 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-24T07:56:25.148 INFO:teuthology.orchestra.run.smithi194.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-24T07:56:25.148 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-24T07:56:25.149 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ply-3.11-14.el9.noarch 2024-04-24T07:56:25.149 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-24T07:56:25.149 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-24T07:56:25.149 INFO:teuthology.orchestra.run.smithi194.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-24T07:56:25.149 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-24T07:56:25.149 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-24T07:56:25.149 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-24T07:56:25.149 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-24T07:56:25.149 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-24T07:56:25.149 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-24T07:56:25.149 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.149 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.149 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-24T07:56:25.149 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-24T07:56:25.150 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-24T07:56:25.150 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.150 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-24T07:56:25.150 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-24T07:56:25.150 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-24T07:56:25.150 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-24T07:56:25.150 INFO:teuthology.orchestra.run.smithi194.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-24T07:56:25.150 INFO:teuthology.orchestra.run.smithi194.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-24T07:56:25.150 INFO:teuthology.orchestra.run.smithi194.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-24T07:56:25.150 INFO:teuthology.orchestra.run.smithi194.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-24T07:56:25.150 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-24T07:56:25.150 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-24T07:56:25.150 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-24T07:56:25.151 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-24T07:56:25.151 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-24T07:56:25.151 INFO:teuthology.orchestra.run.smithi194.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-24T07:56:25.151 INFO:teuthology.orchestra.run.smithi194.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-24T07:56:25.151 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.151 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.151 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:56:25.151 INFO:teuthology.orchestra.run.smithi194.stdout: re2-1:20211101-3.el9.x86_64 2024-04-24T07:56:25.151 INFO:teuthology.orchestra.run.smithi194.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-24T07:56:25.151 INFO:teuthology.orchestra.run.smithi194.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-24T07:56:25.151 INFO:teuthology.orchestra.run.smithi194.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-24T07:56:25.151 INFO:teuthology.orchestra.run.smithi194.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-24T07:56:25.151 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-24T07:56:25.152 INFO:teuthology.orchestra.run.smithi194.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-24T07:56:25.152 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T07:56:25.152 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-04-24T07:56:25.193 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 65/156 2024-04-24T07:56:25.237 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 66/156 2024-04-24T07:56:25.301 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 67/156 2024-04-24T07:56:25.359 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 68/156 2024-04-24T07:56:25.429 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : uuid-1.6.2-55.el9.x86_64 69/156 2024-04-24T07:56:25.485 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 70/156 2024-04-24T07:56:25.513 DEBUG:teuthology.parallel:result is None 2024-04-24T07:56:25.535 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 71/156 2024-04-24T07:56:25.590 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 72/156 2024-04-24T07:56:25.648 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 73/156 2024-04-24T07:56:25.726 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 74/156 2024-04-24T07:56:25.777 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 75/156 2024-04-24T07:56:25.928 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 76/156 2024-04-24T07:56:25.994 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 77/156 2024-04-24T07:56:26.035 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 78/156 2024-04-24T07:56:26.107 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 79/156 2024-04-24T07:56:26.266 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-packaging-20.9-5.el9.noarch 80/156 2024-04-24T07:56:26.340 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 81/156 2024-04-24T07:56:27.348 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 82/156 2024-04-24T07:56:27.405 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 83/156 2024-04-24T07:56:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 84/156 2024-04-24T07:56:27.568 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 85/156 2024-04-24T07:56:27.635 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 86/156 2024-04-24T07:56:27.751 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 87/156 2024-04-24T07:56:27.789 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 88/156 2024-04-24T07:56:28.034 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-0.3.21-2.el9.x86_64 89/156 2024-04-24T07:56:28.069 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 90/156 2024-04-24T07:56:28.178 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 91/156 2024-04-24T07:56:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 92/156 2024-04-24T07:56:29.224 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 93/156 2024-04-24T07:56:30.927 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 94/156 2024-04-24T07:56:31.018 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 95/156 2024-04-24T07:56:31.065 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 96/156 2024-04-24T07:56:31.111 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 97/156 2024-04-24T07:56:31.522 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 98/156 2024-04-24T07:56:31.580 INFO:teuthology.orchestra.run.smithi028.stdout: Upgrading : librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 99/156 2024-04-24T07:56:31.715 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 99/156 2024-04-24T07:56:31.746 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 100/156 2024-04-24T07:56:31.805 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 101/156 2024-04-24T07:56:32.394 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 102/156 2024-04-24T07:56:32.440 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 103/156 2024-04-24T07:56:32.593 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 103/156 2024-04-24T07:56:32.618 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 104/156 2024-04-24T07:56:35.662 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T07:56:35.706 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T07:56:36.382 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 105/156 2024-04-24T07:56:36.426 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 106/156 2024-04-24T07:56:36.605 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 106/156 2024-04-24T07:56:36.606 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-04-24T07:56:36.606 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-04-24T07:56:36.606 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:56:36.652 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 107/156 2024-04-24T07:56:49.937 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 107/156 2024-04-24T07:56:49.937 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /sys 2024-04-24T07:56:49.937 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /proc 2024-04-24T07:56:49.937 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /mnt 2024-04-24T07:56:49.937 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /var/tmp 2024-04-24T07:56:49.937 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /home 2024-04-24T07:56:49.937 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /root 2024-04-24T07:56:49.937 INFO:teuthology.orchestra.run.smithi028.stdout:skipping the directory /tmp 2024-04-24T07:56:49.937 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:56:50.184 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 108/156 2024-04-24T07:56:50.234 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 108/156 2024-04-24T07:56:50.234 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:50.234 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-04-24T07:56:50.234 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:56:51.945 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 109/156 2024-04-24T07:56:52.027 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 109/156 2024-04-24T07:56:52.027 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:52.027 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-04-24T07:56:52.027 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T07:56:52.027 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-04-24T07:56:52.028 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:56:52.319 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 110/156 2024-04-24T07:56:52.388 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 110/156 2024-04-24T07:56:52.388 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:52.388 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-04-24T07:56:52.388 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T07:56:52.388 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-04-24T07:56:52.388 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:56:52.912 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 111/156 2024-04-24T07:56:52.985 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 111/156 2024-04-24T07:56:52.985 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:52.986 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-04-24T07:56:52.986 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T07:56:52.986 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-04-24T07:56:52.986 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:56:53.121 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : automake-1.16.2-8.el9.noarch 112/156 2024-04-24T07:56:53.185 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-ply-3.11-14.el9.noarch 113/156 2024-04-24T07:56:53.261 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 114/156 2024-04-24T07:56:53.521 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 115/156 2024-04-24T07:56:53.585 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 116/156 2024-04-24T07:56:53.687 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 117/156 2024-04-24T07:56:53.807 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T07:56:53.990 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T07:56:54.043 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 120/156 2024-04-24T07:56:54.084 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 121/156 2024-04-24T07:56:54.115 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : fuse-2.9.9-15.el9.x86_64 122/156 2024-04-24T07:56:55.079 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 123/156 2024-04-24T07:56:55.118 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 123/156 2024-04-24T07:56:55.166 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.e 124/156 2024-04-24T07:56:55.274 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39. 125/156 2024-04-24T07:56:55.311 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 126/156 2024-04-24T07:56:56.674 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 126/156 2024-04-24T07:56:56.721 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 127/156 2024-04-24T07:56:57.887 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 127/156 2024-04-24T07:56:57.938 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 128/156 2024-04-24T07:56:58.100 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 128/156 2024-04-24T07:56:58.287 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el 129/156 2024-04-24T07:56:58.315 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 130/156 2024-04-24T07:56:58.398 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 130/156 2024-04-24T07:56:58.398 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:58.398 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-04-24T07:56:58.398 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T07:56:58.398 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-04-24T07:56:58.398 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:56:58.458 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 131/156 2024-04-24T07:56:58.497 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 131/156 2024-04-24T07:56:58.613 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 132/156 2024-04-24T07:56:58.684 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 133/156 2024-04-24T07:56:59.749 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libtool-2.4.6-45.el9.x86_64 134/156 2024-04-24T07:56:59.794 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 135/156 2024-04-24T07:56:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 135/156 2024-04-24T07:56:59.865 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:56:59.865 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-04-24T07:56:59.865 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T07:56:59.865 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-04-24T07:56:59.865 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:57:00.011 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 136/156 2024-04-24T07:57:00.084 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 136/156 2024-04-24T07:57:00.084 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:57:00.084 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-04-24T07:57:00.084 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:57:00.170 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 137/156 2024-04-24T07:57:00.255 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 137/156 2024-04-24T07:57:00.255 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:57:00.255 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-04-24T07:57:00.255 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T07:57:00.255 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-04-24T07:57:00.255 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:57:00.671 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 138/156 2024-04-24T07:57:00.754 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 138/156 2024-04-24T07:57:00.755 INFO:teuthology.orchestra.run.smithi028.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-04-24T07:57:00.755 INFO:teuthology.orchestra.run.smithi028.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-04-24T07:57:00.755 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T07:57:00.755 INFO:teuthology.orchestra.run.smithi028.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-04-24T07:57:00.755 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:57:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 139/156 2024-04-24T07:57:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 140/156 2024-04-24T07:57:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 141/156 2024-04-24T07:57:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 142/156 2024-04-24T07:57:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 143/156 2024-04-24T07:57:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 144/156 2024-04-24T07:57:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 145/156 2024-04-24T07:57:06.742 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_ 146/156 2024-04-24T07:57:07.647 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T07:57:07.699 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T07:57:07.758 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 147/156 2024-04-24T07:57:09.240 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 148/156 2024-04-24T07:57:09.356 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 149/156 2024-04-24T07:57:09.455 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 150/156 2024-04-24T07:57:09.652 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : flex-2.6.4-9.el9.x86_64 151/156 2024-04-24T07:57:09.694 INFO:teuthology.orchestra.run.smithi028.stdout: Installing : bison-3.7.4-5.el9.x86_64 152/156 2024-04-24T07:57:09.694 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T07:57:09.857 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 153/156 2024-04-24T07:57:09.911 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T07:57:09.911 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T07:57:10.085 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 155/156 2024-04-24T07:57:10.221 INFO:teuthology.orchestra.run.smithi028.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T07:57:10.509 INFO:teuthology.orchestra.run.smithi028.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T07:57:10.509 INFO:teuthology.orchestra.run.smithi028.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-04-24T07:57:10.509 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:57:16.129 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 1/156 2024-04-24T07:57:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 2/156 2024-04-24T07:57:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 3/156 2024-04-24T07:57:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 4/156 2024-04-24T07:57:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-3116.g1f81f 5/156 2024-04-24T07:57:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 6/156 2024-04-24T07:57:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 7/156 2024-04-24T07:57:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 8/156 2024-04-24T07:57:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 9/156 2024-04-24T07:57:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 10/156 2024-04-24T07:57:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 11/156 2024-04-24T07:57:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 12/156 2024-04-24T07:57:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 13/156 2024-04-24T07:57:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 14/156 2024-04-24T07:57:16.130 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_ 15/156 2024-04-24T07:57:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 16/156 2024-04-24T07:57:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 17/156 2024-04-24T07:57:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_6 18/156 2024-04-24T07:57:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86 19/156 2024-04-24T07:57:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 20/156 2024-04-24T07:57:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el 21/156 2024-04-24T07:57:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9. 22/156 2024-04-24T07:57:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_6 23/156 2024-04-24T07:57:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 24/156 2024-04-24T07:57:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 25/156 2024-04-24T07:57:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 26/156 2024-04-24T07:57:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 27/156 2024-04-24T07:57:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 28/156 2024-04-24T07:57:16.131 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 29/156 2024-04-24T07:57:16.143 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39. 30/156 2024-04-24T07:57:16.143 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noa 31/156 2024-04-24T07:57:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.n 32/156 2024-04-24T07:57:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f8 33/156 2024-04-24T07:57:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el 34/156 2024-04-24T07:57:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 35/156 2024-04-24T07:57:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.e 36/156 2024-04-24T07:57:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 37/156 2024-04-24T07:57:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 38/156 2024-04-24T07:57:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 39/156 2024-04-24T07:57:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/156 2024-04-24T07:57:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/156 2024-04-24T07:57:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/156 2024-04-24T07:57:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/156 2024-04-24T07:57:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/156 2024-04-24T07:57:16.144 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/156 2024-04-24T07:57:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/156 2024-04-24T07:57:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/156 2024-04-24T07:57:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/156 2024-04-24T07:57:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/156 2024-04-24T07:57:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/156 2024-04-24T07:57:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/156 2024-04-24T07:57:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/156 2024-04-24T07:57:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/156 2024-04-24T07:57:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/156 2024-04-24T07:57:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/156 2024-04-24T07:57:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/156 2024-04-24T07:57:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/156 2024-04-24T07:57:16.145 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/156 2024-04-24T07:57:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/156 2024-04-24T07:57:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/156 2024-04-24T07:57:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/156 2024-04-24T07:57:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/156 2024-04-24T07:57:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/156 2024-04-24T07:57:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/156 2024-04-24T07:57:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/156 2024-04-24T07:57:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/156 2024-04-24T07:57:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/156 2024-04-24T07:57:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/156 2024-04-24T07:57:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/156 2024-04-24T07:57:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/156 2024-04-24T07:57:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/156 2024-04-24T07:57:16.146 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/156 2024-04-24T07:57:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/156 2024-04-24T07:57:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/156 2024-04-24T07:57:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/156 2024-04-24T07:57:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/156 2024-04-24T07:57:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/156 2024-04-24T07:57:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/156 2024-04-24T07:57:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/156 2024-04-24T07:57:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/156 2024-04-24T07:57:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/156 2024-04-24T07:57:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/156 2024-04-24T07:57:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/156 2024-04-24T07:57:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/156 2024-04-24T07:57:16.147 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/156 2024-04-24T07:57:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/156 2024-04-24T07:57:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/156 2024-04-24T07:57:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/156 2024-04-24T07:57:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/156 2024-04-24T07:57:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/156 2024-04-24T07:57:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/156 2024-04-24T07:57:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/156 2024-04-24T07:57:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/156 2024-04-24T07:57:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/156 2024-04-24T07:57:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/156 2024-04-24T07:57:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/156 2024-04-24T07:57:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/156 2024-04-24T07:57:16.148 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/156 2024-04-24T07:57:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/156 2024-04-24T07:57:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/156 2024-04-24T07:57:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/156 2024-04-24T07:57:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/156 2024-04-24T07:57:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/156 2024-04-24T07:57:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/156 2024-04-24T07:57:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/156 2024-04-24T07:57:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/156 2024-04-24T07:57:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/156 2024-04-24T07:57:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 108/156 2024-04-24T07:57:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 109/156 2024-04-24T07:57:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/156 2024-04-24T07:57:16.149 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/156 2024-04-24T07:57:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/156 2024-04-24T07:57:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 113/156 2024-04-24T07:57:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/156 2024-04-24T07:57:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/156 2024-04-24T07:57:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/156 2024-04-24T07:57:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/156 2024-04-24T07:57:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/156 2024-04-24T07:57:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/156 2024-04-24T07:57:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/156 2024-04-24T07:57:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/156 2024-04-24T07:57:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/156 2024-04-24T07:57:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/156 2024-04-24T07:57:16.150 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/156 2024-04-24T07:57:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/156 2024-04-24T07:57:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/156 2024-04-24T07:57:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/156 2024-04-24T07:57:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/156 2024-04-24T07:57:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/156 2024-04-24T07:57:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/156 2024-04-24T07:57:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/156 2024-04-24T07:57:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/156 2024-04-24T07:57:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/156 2024-04-24T07:57:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/156 2024-04-24T07:57:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/156 2024-04-24T07:57:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/156 2024-04-24T07:57:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/156 2024-04-24T07:57:16.151 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/156 2024-04-24T07:57:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/156 2024-04-24T07:57:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/156 2024-04-24T07:57:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/156 2024-04-24T07:57:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/156 2024-04-24T07:57:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/156 2024-04-24T07:57:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/156 2024-04-24T07:57:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/156 2024-04-24T07:57:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/156 2024-04-24T07:57:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/156 2024-04-24T07:57:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/156 2024-04-24T07:57:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 149/156 2024-04-24T07:57:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/156 2024-04-24T07:57:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 151/156 2024-04-24T07:57:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/156 2024-04-24T07:57:16.152 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 153/156 2024-04-24T07:57:16.153 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 154/156 2024-04-24T07:57:16.153 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 155/156 2024-04-24T07:57:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 156/156 2024-04-24T07:57:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:57:18.490 INFO:teuthology.orchestra.run.smithi028.stdout:Upgraded: 2024-04-24T07:57:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: librados2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: librbd1-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-04-24T07:57:18.490 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-04-24T07:57:18.491 INFO:teuthology.orchestra.run.smithi028.stdout:Installed: 2024-04-24T07:57:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-04-24T07:57:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: automake-1.16.2-8.el9.noarch 2024-04-24T07:57:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: bison-3.7.4-5.el9.x86_64 2024-04-24T07:57:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-04-24T07:57:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-base-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-grafana-dashboards-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:57:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-immutable-object-cache-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mds-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.491 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:57:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-dashboard-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:57:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-diskprediction-local-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:57:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-modules-core-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:57:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mgr-rook-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:57:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-mon-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-osd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-prometheus-alerts-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:57:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-radosgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-selinux-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-test-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: ceph-volume-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:57:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: cephadm-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:57:18.492 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-shell-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: cephfs-top-2:19.0.0-3116.g1f81fb39.el9.noarch 2024-04-24T07:57:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-04-24T07:57:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: flex-2.6.4-9.el9.x86_64 2024-04-24T07:57:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-04-24T07:57:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-04-24T07:57:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-04-24T07:57:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: fuse-2.9.9-15.el9.x86_64 2024-04-24T07:57:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-04-24T07:57:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: grpc-data-1.46.7-10.el9.noarch 2024-04-24T07:57:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-04-24T07:57:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-9.0.0-10.el9.x86_64 2024-04-24T07:57:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-04-24T07:57:18.493 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs-devel-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.494 INFO:teuthology.orchestra.run.smithi028.stdout: libcephfs2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.494 INFO:teuthology.orchestra.run.smithi028.stdout: libcephsqlite-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.494 INFO:teuthology.orchestra.run.smithi028.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-04-24T07:57:18.494 INFO:teuthology.orchestra.run.smithi028.stdout: liboath-2.6.7-2.el9.x86_64 2024-04-24T07:57:18.494 INFO:teuthology.orchestra.run.smithi028.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-04-24T07:57:18.494 INFO:teuthology.orchestra.run.smithi028.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-04-24T07:57:18.494 INFO:teuthology.orchestra.run.smithi028.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-04-24T07:57:18.494 INFO:teuthology.orchestra.run.smithi028.stdout: librados-devel-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.494 INFO:teuthology.orchestra.run.smithi028.stdout: libradosstriper1-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.494 INFO:teuthology.orchestra.run.smithi028.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-04-24T07:57:18.494 INFO:teuthology.orchestra.run.smithi028.stdout: librgw2-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.494 INFO:teuthology.orchestra.run.smithi028.stdout: libtool-2.4.6-45.el9.x86_64 2024-04-24T07:57:18.494 INFO:teuthology.orchestra.run.smithi028.stdout: libunwind-1.6.2-1.el9.x86_64 2024-04-24T07:57:18.494 INFO:teuthology.orchestra.run.smithi028.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-04-24T07:57:18.494 INFO:teuthology.orchestra.run.smithi028.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-04-24T07:57:18.495 INFO:teuthology.orchestra.run.smithi028.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-04-24T07:57:18.495 INFO:teuthology.orchestra.run.smithi028.stdout: lua-5.4.4-4.el9.x86_64 2024-04-24T07:57:18.495 INFO:teuthology.orchestra.run.smithi028.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-04-24T07:57:18.495 INFO:teuthology.orchestra.run.smithi028.stdout: luarocks-3.9.2-1.el9.noarch 2024-04-24T07:57:18.495 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-0.3.21-2.el9.x86_64 2024-04-24T07:57:18.495 INFO:teuthology.orchestra.run.smithi028.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-04-24T07:57:18.495 INFO:teuthology.orchestra.run.smithi028.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-04-24T07:57:18.495 INFO:teuthology.orchestra.run.smithi028.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-04-24T07:57:18.495 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-13.14-1.el9.x86_64 2024-04-24T07:57:18.495 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-04-24T07:57:18.495 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-04-24T07:57:18.495 INFO:teuthology.orchestra.run.smithi028.stdout: postgresql-server-13.14-1.el9.x86_64 2024-04-24T07:57:18.495 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-3.14.0-13.el9.x86_64 2024-04-24T07:57:18.495 INFO:teuthology.orchestra.run.smithi028.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-04-24T07:57:18.495 INFO:teuthology.orchestra.run.smithi028.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-04-24T07:57:18.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-04-24T07:57:18.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-babel-2.9.1-2.el9.noarch 2024-04-24T07:57:18.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-04-24T07:57:18.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-04-24T07:57:18.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-argparse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ceph-common-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cephfs-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-04-24T07:57:18.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-04-24T07:57:18.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-04-24T07:57:18.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-04-24T07:57:18.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-04-24T07:57:18.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-04-24T07:57:18.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-04-24T07:57:18.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-04-24T07:57:18.496 INFO:teuthology.orchestra.run.smithi028.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-04-24T07:57:18.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-04-24T07:57:18.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-04-24T07:57:18.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-04-24T07:57:18.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-04-24T07:57:18.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-04-24T07:57:18.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-04-24T07:57:18.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-04-24T07:57:18.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-04-24T07:57:18.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-04-24T07:57:18.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-04-24T07:57:18.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-04-24T07:57:18.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-04-24T07:57:18.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-mako-1.1.4-6.el9.noarch 2024-04-24T07:57:18.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-04-24T07:57:18.497 INFO:teuthology.orchestra.run.smithi028.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-04-24T07:57:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-04-24T07:57:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-04-24T07:57:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-04-24T07:57:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-04-24T07:57:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-packaging-20.9-5.el9.noarch 2024-04-24T07:57:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-04-24T07:57:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-ply-3.11-14.el9.noarch 2024-04-24T07:57:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-portend-3.1.0-2.el9.noarch 2024-04-24T07:57:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-04-24T07:57:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-04-24T07:57:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-04-24T07:57:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-04-24T07:57:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-04-24T07:57:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pycparser-2.20-6.el9.noarch 2024-04-24T07:57:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-04-24T07:57:18.498 INFO:teuthology.orchestra.run.smithi028.stdout: python3-pytz-2021.1-5.el9.noarch 2024-04-24T07:57:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rados-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-04-24T07:57:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-2.25.1-8.el9.noarch 2024-04-24T07:57:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-04-24T07:57:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rgw-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: python3-routes-2.5.1-5.el9.noarch 2024-04-24T07:57:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: python3-rsa-4.9-2.el9.noarch 2024-04-24T07:57:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-04-24T07:57:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-04-24T07:57:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: python3-toml-0.10.2-6.el9.noarch 2024-04-24T07:57:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-04-24T07:57:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-04-24T07:57:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-04-24T07:57:18.499 INFO:teuthology.orchestra.run.smithi028.stdout: python3-webob-1.8.7-6.el9.noarch 2024-04-24T07:57:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-04-24T07:57:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-04-24T07:57:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-04-24T07:57:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-24.02.0-1.el9.x86_64 2024-04-24T07:57:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-04-24T07:57:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-04-24T07:57:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-fuse-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-mirror-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: rbd-nbd-2:19.0.0-3116.g1f81fb39.el9.x86_64 2024-04-24T07:57:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: re2-1:20211101-3.el9.x86_64 2024-04-24T07:57:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: socat-1.7.4.1-5.el9.x86_64 2024-04-24T07:57:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: thrift-0.15.0-2.el9.x86_64 2024-04-24T07:57:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: uuid-1.6.2-55.el9.x86_64 2024-04-24T07:57:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-04-24T07:57:18.500 INFO:teuthology.orchestra.run.smithi028.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-04-24T07:57:18.501 INFO:teuthology.orchestra.run.smithi028.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-04-24T07:57:18.501 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:57:18.501 INFO:teuthology.orchestra.run.smithi028.stdout:Complete! 2024-04-24T07:57:19.068 DEBUG:teuthology.parallel:result is None 2024-04-24T07:57:19.068 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T07:57:19.211 DEBUG:teuthology.orchestra.run.smithi028:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T07:57:19.268 INFO:teuthology.orchestra.run.smithi028.stdout:19.0.0-3116.g1f81fb39.el9 2024-04-24T07:57:19.269 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3116.g1f81fb39.el9 2024-04-24T07:57:19.269 INFO:teuthology.task.install:The correct ceph version 19.0.0-3116.g1f81fb39 is installed. 2024-04-24T07:57:19.271 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T07:57:19.413 DEBUG:teuthology.orchestra.run.smithi117:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T07:57:19.439 INFO:teuthology.orchestra.run.smithi117.stdout:19.0.0-3116.g1f81fb39.el9 2024-04-24T07:57:19.440 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3116.g1f81fb39.el9 2024-04-24T07:57:19.440 INFO:teuthology.task.install:The correct ceph version 19.0.0-3116.g1f81fb39 is installed. 2024-04-24T07:57:19.442 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T07:57:19.585 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-04-24T07:57:19.610 INFO:teuthology.orchestra.run.smithi194.stdout:19.0.0-3116.g1f81fb39.el9 2024-04-24T07:57:19.610 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-3116.g1f81fb39.el9 2024-04-24T07:57:19.610 INFO:teuthology.task.install:The correct ceph version 19.0.0-3116.g1f81fb39 is installed. 2024-04-24T07:57:19.612 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-04-24T07:57:19.612 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T07:57:19.612 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T07:57:19.693 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-04-24T07:57:19.693 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T07:57:19.724 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-24T07:57:19.725 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-04-24T07:57:19.756 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-04-24T07:57:19.756 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T07:57:19.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T07:57:19.823 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T07:57:19.944 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-04-24T07:57:19.944 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T07:57:19.973 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T07:57:20.039 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-24T07:57:20.039 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/daemon-helper 2024-04-24T07:57:20.064 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-04-24T07:57:20.127 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-04-24T07:57:20.127 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T07:57:20.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T07:57:20.200 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T07:57:20.315 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-04-24T07:57:20.315 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T07:57:20.339 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T07:57:20.413 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-24T07:57:20.413 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/adjust-ulimits 2024-04-24T07:57:20.437 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-04-24T07:57:20.500 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-04-24T07:57:20.501 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T07:57:20.501 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T07:57:20.568 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T07:57:20.674 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-04-24T07:57:20.674 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T07:57:20.698 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T07:57:20.761 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-24T07:57:20.761 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/stdin-killer 2024-04-24T07:57:20.784 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-04-24T07:57:20.848 INFO:teuthology.run_tasks:Running task cephadm... 2024-04-24T07:57:20.952 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'filestore flush min': 0, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd heartbeat grace': 60, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10000}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request', 'SLOW_OPS', 'slow request'], 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0', 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-24T07:57:20.953 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T07:57:20.953 INFO:tasks.cephadm:Cluster fsid is 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T07:57:20.953 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-04-24T07:57:20.953 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.28', 'mon.b': '172.21.15.117', 'mon.c': '172.21.15.194'} 2024-04-24T07:57:20.954 INFO:tasks.cephadm:First mon is mon.a on smithi028 2024-04-24T07:57:20.954 INFO:tasks.cephadm:First mgr is x 2024-04-24T07:57:20.954 INFO:tasks.cephadm:Normalizing hostnames... 2024-04-24T07:57:20.954 DEBUG:teuthology.orchestra.run.smithi028:> sudo hostname $(hostname -s) 2024-04-24T07:57:21.026 DEBUG:teuthology.orchestra.run.smithi117:> sudo hostname $(hostname -s) 2024-04-24T07:57:21.050 DEBUG:teuthology.orchestra.run.smithi194:> sudo hostname $(hostname -s) 2024-04-24T07:57:21.073 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-04-24T07:57:21.073 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T07:57:21.216 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '1f81fb397ae98da7563d451a78c61574c8f4e6e0', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/78823/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-3116-g1f81fb39', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-3116.g1f81fb39'}, 'url': 'https://4.chacra.ceph.com/r/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-04-24 01:00:13.634707', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'rishabh-main-apr17-a654945', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-04-24T07:57:21.450 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref rishabh-main-apr17-a654945, sha1 1f81fb397ae98da7563d451a78c61574c8f4e6e0 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T07:57:21.451 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/x86_64/flavors/default/cephadm 2024-04-24T07:57:21.451 DEBUG:teuthology.orchestra.run.smithi028:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-24T07:57:22.846 INFO:teuthology.orchestra.run.smithi028.stdout:-rw-r--r--. 1 ubuntu ubuntu 781486 Apr 24 07:57 /home/ubuntu/cephtest/cephadm 2024-04-24T07:57:22.846 DEBUG:teuthology.orchestra.run.smithi117:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-24T07:57:24.612 INFO:teuthology.orchestra.run.smithi117.stdout:-rw-r--r--. 1 ubuntu ubuntu 781486 Apr 24 07:57 /home/ubuntu/cephtest/cephadm 2024-04-24T07:57:24.612 DEBUG:teuthology.orchestra.run.smithi194:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/rishabh-main-apr17-a654945/1f81fb397ae98da7563d451a78c61574c8f4e6e0/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-04-24T07:57:26.593 INFO:teuthology.orchestra.run.smithi194.stdout:-rw-r--r--. 1 ubuntu ubuntu 781486 Apr 24 07:57 /home/ubuntu/cephtest/cephadm 2024-04-24T07:57:26.594 DEBUG:teuthology.orchestra.run.smithi028:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-24T07:57:26.650 DEBUG:teuthology.orchestra.run.smithi117:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-24T07:57:26.668 DEBUG:teuthology.orchestra.run.smithi194:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-04-24T07:57:26.702 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 on all hosts... 2024-04-24T07:57:26.702 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 pull 2024-04-24T07:57:26.708 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 pull 2024-04-24T07:57:26.710 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 pull 2024-04-24T07:57:28.208 INFO:teuthology.orchestra.run.smithi194.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0... 2024-04-24T07:57:28.245 INFO:teuthology.orchestra.run.smithi117.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0... 2024-04-24T07:57:29.998 INFO:teuthology.orchestra.run.smithi028.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0... 2024-04-24T07:58:11.386 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T07:58:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T07:58:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: "image_id": "b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02", 2024-04-24T07:58:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: "repo_digests": [ 2024-04-24T07:58:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644" 2024-04-24T07:58:11.387 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-24T07:58:11.387 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T07:58:21.018 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-24T07:58:21.018 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T07:58:21.018 INFO:teuthology.orchestra.run.smithi194.stdout: "image_id": "b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02", 2024-04-24T07:58:21.019 INFO:teuthology.orchestra.run.smithi194.stdout: "repo_digests": [ 2024-04-24T07:58:21.019 INFO:teuthology.orchestra.run.smithi194.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644" 2024-04-24T07:58:21.019 INFO:teuthology.orchestra.run.smithi194.stdout: ] 2024-04-24T07:58:21.019 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-24T07:58:29.929 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-04-24T07:58:29.930 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_version": "ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev)", 2024-04-24T07:58:29.930 INFO:teuthology.orchestra.run.smithi117.stdout: "image_id": "b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02", 2024-04-24T07:58:29.930 INFO:teuthology.orchestra.run.smithi117.stdout: "repo_digests": [ 2024-04-24T07:58:29.930 INFO:teuthology.orchestra.run.smithi117.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644" 2024-04-24T07:58:29.930 INFO:teuthology.orchestra.run.smithi117.stdout: ] 2024-04-24T07:58:29.930 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-04-24T07:58:29.955 DEBUG:teuthology.orchestra.run.smithi028:> sudo mkdir -p /etc/ceph 2024-04-24T07:58:30.047 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph 2024-04-24T07:58:30.073 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph 2024-04-24T07:58:30.105 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 777 /etc/ceph 2024-04-24T07:58:30.197 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 777 /etc/ceph 2024-04-24T07:58:30.220 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /etc/ceph 2024-04-24T07:58:30.248 INFO:tasks.cephadm:Writing seed config... 2024-04-24T07:58:30.249 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-04-24T07:58:30.249 INFO:tasks.cephadm: override: [client] debug client = 20 2024-04-24T07:58:30.249 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-04-24T07:58:30.249 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-04-24T07:58:30.250 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-04-24T07:58:30.250 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-04-24T07:58:30.250 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-04-24T07:58:30.250 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-04-24T07:58:30.250 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-04-24T07:58:30.250 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-04-24T07:58:30.250 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-04-24T07:58:30.250 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-04-24T07:58:30.250 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-04-24T07:58:30.250 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-04-24T07:58:30.250 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-04-24T07:58:30.250 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-04-24T07:58:30.250 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-04-24T07:58:30.250 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-04-24T07:58:30.250 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-04-24T07:58:30.251 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-04-24T07:58:30.251 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-04-24T07:58:30.251 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-04-24T07:58:30.251 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-04-24T07:58:30.251 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-04-24T07:58:30.251 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-04-24T07:58:30.251 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-04-24T07:58:30.251 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-04-24T07:58:30.251 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-04-24T07:58:30.251 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-04-24T07:58:30.251 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-04-24T07:58:30.251 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-04-24T07:58:30.251 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-04-24T07:58:30.251 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-04-24T07:58:30.251 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-04-24T07:58:30.252 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-04-24T07:58:30.252 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-04-24T07:58:30.252 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-04-24T07:58:30.252 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-04-24T07:58:30.252 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-04-24T07:58:30.252 INFO:tasks.cephadm: override: [osd] filestore flush min = 0 2024-04-24T07:58:30.252 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-04-24T07:58:30.252 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-04-24T07:58:30.252 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-04-24T07:58:30.252 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-04-24T07:58:30.252 INFO:tasks.cephadm: override: [osd] osd heartbeat grace = 60 2024-04-24T07:58:30.252 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-04-24T07:58:30.252 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-04-24T07:58:30.252 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-04-24T07:58:30.252 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10000 2024-04-24T07:58:30.253 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T07:58:30.253 DEBUG:teuthology.orchestra.run.smithi028:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-04-24T07:58:30.314 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 473aaf58-0210-11ef-bc93-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 filestore flush min = 0 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd heartbeat grace = 60 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10000 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-04-24T07:58:30.314 DEBUG:teuthology.orchestra.run.smithi028:mon.a> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mon.a.service 2024-04-24T07:58:30.361 DEBUG:teuthology.orchestra.run.smithi028:mgr.x> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mgr.x.service 2024-04-24T07:58:30.406 INFO:tasks.cephadm:Bootstrapping... 2024-04-24T07:58:30.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 -v bootstrap --fsid 473aaf58-0210-11ef-bc93-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.28 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-04-24T07:58:30.679 INFO:teuthology.orchestra.run.smithi028.stdout:-------------------------------------------------------------------------------- 2024-04-24T07:58:30.679 INFO:teuthology.orchestra.run.smithi028.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0', '-v', 'bootstrap', '--fsid', '473aaf58-0210-11ef-bc93-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.28', '--skip-admin-label'] 2024-04-24T07:58:30.680 INFO:teuthology.orchestra.run.smithi028.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-04-24T07:58:30.680 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying podman|docker is present... 2024-04-24T07:58:30.739 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-24T07:58:30.740 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying lvm2 is present... 2024-04-24T07:58:30.740 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying time synchronization is in place... 2024-04-24T07:58:30.756 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-24T07:58:30.756 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-24T07:58:30.771 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-24T07:58:30.771 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-24T07:58:30.787 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-04-24T07:58:30.804 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-04-24T07:58:30.804 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-04-24T07:58:30.804 INFO:teuthology.orchestra.run.smithi028.stdout:Repeating the final host check... 2024-04-24T07:58:30.856 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout 4.9.4-dev 2024-04-24T07:58:30.856 INFO:teuthology.orchestra.run.smithi028.stdout:podman (/bin/podman) version 4.9.4 is present 2024-04-24T07:58:30.857 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl is present 2024-04-24T07:58:30.857 INFO:teuthology.orchestra.run.smithi028.stdout:lvcreate is present 2024-04-24T07:58:30.872 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-04-24T07:58:30.872 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-04-24T07:58:30.892 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-04-24T07:58:30.893 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-24T07:58:30.919 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout enabled 2024-04-24T07:58:30.941 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout active 2024-04-24T07:58:30.942 INFO:teuthology.orchestra.run.smithi028.stdout:Unit chronyd.service is enabled and running 2024-04-24T07:58:30.942 INFO:teuthology.orchestra.run.smithi028.stdout:Host looks OK 2024-04-24T07:58:30.942 INFO:teuthology.orchestra.run.smithi028.stdout:Cluster fsid: 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T07:58:30.943 INFO:teuthology.orchestra.run.smithi028.stdout:Acquiring lock 139697328106656 on /run/cephadm/473aaf58-0210-11ef-bc93-c7b262605968.lock 2024-04-24T07:58:30.944 INFO:teuthology.orchestra.run.smithi028.stdout:Lock 139697328106656 acquired on /run/cephadm/473aaf58-0210-11ef-bc93-c7b262605968.lock 2024-04-24T07:58:30.945 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 3300 ... 2024-04-24T07:58:30.946 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying IP 172.21.15.28 port 6789 ... 2024-04-24T07:58:30.947 INFO:teuthology.orchestra.run.smithi028.stdout:Base mon IP(s) is [172.21.15.28:3300, 172.21.15.28:6789], mon addrv is [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-04-24T07:58:30.962 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.28 metric 100 2024-04-24T07:58:30.962 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.28 metric 100 2024-04-24T07:58:30.975 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-04-24T07:58:30.975 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout fe80::/64 dev ens1f0 proto kernel metric 1024 pref medium 2024-04-24T07:58:30.976 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 100 pref medium 2024-04-24T07:58:30.987 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-04-24T07:58:30.987 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-04-24T07:58:30.988 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-24T07:58:30.988 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout 4: ens1f0: mtu 1500 state UP qlen 1000 2024-04-24T07:58:30.988 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-04-24T07:58:30.988 INFO:teuthology.orchestra.run.smithi028.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-04-24T07:58:30.991 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-04-24T07:58:30.992 INFO:teuthology.orchestra.run.smithi028.stdout:Mon IP `172.21.15.28` is in CIDR network `172.21.0.0/20` 2024-04-24T07:58:30.992 INFO:teuthology.orchestra.run.smithi028.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-04-24T07:58:30.995 INFO:teuthology.orchestra.run.smithi028.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-04-24T07:58:30.998 INFO:teuthology.orchestra.run.smithi028.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0... 2024-04-24T07:58:31.712 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stdout b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 2024-04-24T07:58:31.713 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0... 2024-04-24T07:58:31.713 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Getting image source signatures 2024-04-24T07:58:31.713 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:b727accf024e55f70f86e348fc139074bb9b3de1cca3400f8b76b2ed7d35ac83 2024-04-24T07:58:31.713 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying blob sha256:3d665f4c032644be9cae748af04bee69f2bfe96cdc3616dabe0ab17899633ab5 2024-04-24T07:58:31.713 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Copying config sha256:b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 2024-04-24T07:58:31.713 INFO:teuthology.orchestra.run.smithi028.stdout:/bin/podman: stderr Writing manifest to image destination 2024-04-24T07:58:32.925 INFO:teuthology.orchestra.run.smithi028.stdout:ceph: stdout ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev) 2024-04-24T07:58:32.925 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph version: ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev) 2024-04-24T07:58:32.925 INFO:teuthology.orchestra.run.smithi028.stdout:Extracting ceph user uid/gid from container image... 2024-04-24T07:58:34.050 INFO:teuthology.orchestra.run.smithi028.stdout:stat: stdout 167 167 2024-04-24T07:58:34.050 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial keys... 2024-04-24T07:58:35.126 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQCquyhm3RCXHxAAWcBgEUJaAiV05JV/hsY/nQ== 2024-04-24T07:58:36.516 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQCruyhmKk1zJBAAuE51M1Wy+yH96usCMaRTuA== 2024-04-24T07:58:37.688 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph-authtool: stdout AQCsuyhmEzUSOhAAVX2Re015QaZvCNODkW6pCw== 2024-04-24T07:58:37.691 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial monmap... 2024-04-24T07:58:38.796 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-24T07:58:38.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-04-24T07:58:38.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T07:58:38.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-24T07:58:38.797 INFO:teuthology.orchestra.run.smithi028.stdout:monmaptool for a [v2:172.21.15.28:3300,v1:172.21.15.28:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-04-24T07:58:38.797 INFO:teuthology.orchestra.run.smithi028.stdout:setting min_mon_release = quincy 2024-04-24T07:58:38.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: set fsid to 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T07:58:38.797 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-04-24T07:58:38.797 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:58:38.797 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mon... 2024-04-24T07:58:39.960 INFO:teuthology.orchestra.run.smithi028.stdout:create mon.a on 2024-04-24T07:58:40.320 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-04-24T07:58:40.582 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-04-24T07:58:40.893 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-473aaf58-0210-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-473aaf58-0210-11ef-bc93-c7b262605968.target. 2024-04-24T07:58:40.893 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-473aaf58-0210-11ef-bc93-c7b262605968.target → /etc/systemd/system/ceph-473aaf58-0210-11ef-bc93-c7b262605968.target. 2024-04-24T07:58:41.280 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-473aaf58-0210-11ef-bc93-c7b262605968@mon.a 2024-04-24T07:58:41.280 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-473aaf58-0210-11ef-bc93-c7b262605968@mon.a.service: Unit ceph-473aaf58-0210-11ef-bc93-c7b262605968@mon.a.service not loaded. 2024-04-24T07:58:41.587 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-473aaf58-0210-11ef-bc93-c7b262605968.target.wants/ceph-473aaf58-0210-11ef-bc93-c7b262605968@mon.a.service → /etc/systemd/system/ceph-473aaf58-0210-11ef-bc93-c7b262605968@.service. 2024-04-24T07:58:42.251 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:42 smithi028 podman[27399]: 2024-04-24 07:58:42.239215501 +0000 UTC m=+0.373672887 container init d0e0813f1dba3dad985bc481c8801ac1ff9839b46dd4d90c99ecaf5c004bc7bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-24T07:58:42.399 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:42 smithi028 podman[27399]: 2024-04-24 07:58:42.250059356 +0000 UTC m=+0.384516662 container start d0e0813f1dba3dad985bc481c8801ac1ff9839b46dd4d90c99ecaf5c004bc7bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-04-24T07:58:42.400 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-24T07:58:42.400 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-24T07:58:42.400 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-24T07:58:42.400 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-24T07:58:42.400 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-24T07:58:42.412 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-04-24T07:58:42.413 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon to start... 2024-04-24T07:58:42.413 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mon... 2024-04-24T07:58:42.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:42 smithi028 ceph-mon[27419]: mkfs 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T07:58:42.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:42 smithi028 ceph-mon[27419]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-24T07:58:42.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:42 smithi028 bash[27399]: d0e0813f1dba3dad985bc481c8801ac1ff9839b46dd4d90c99ecaf5c004bc7bb 2024-04-24T07:58:42.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:42 smithi028 systemd[1]: Started Ceph mon.a for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T07:58:43.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:43 smithi028 ceph-mon[27419]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-24T07:58:43.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:43 smithi028 ceph-mon[27419]: monmap epoch 1 2024-04-24T07:58:43.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:43 smithi028 ceph-mon[27419]: fsid 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T07:58:43.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:43 smithi028 ceph-mon[27419]: last_changed 2024-04-24T07:58:38.163806+0000 2024-04-24T07:58:43.494 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:43 smithi028 ceph-mon[27419]: created 2024-04-24T07:58:38.163806+0000 2024-04-24T07:58:43.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:43 smithi028 ceph-mon[27419]: min_mon_release 19 (squid) 2024-04-24T07:58:43.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:43 smithi028 ceph-mon[27419]: election_strategy: 1 2024-04-24T07:58:43.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:43 smithi028 ceph-mon[27419]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-24T07:58:43.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:43 smithi028 ceph-mon[27419]: fsmap 2024-04-24T07:58:43.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:43 smithi028 ceph-mon[27419]: osdmap e1: 0 total, 0 up, 0 in 2024-04-24T07:58:43.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:43 smithi028 ceph-mon[27419]: mgrmap e1: no daemons active 2024-04-24T07:58:43.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:43 smithi028 ceph-mon[27419]: from='client.? 172.21.15.28:0/1308736385' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-04-24T07:58:43.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout cluster: 2024-04-24T07:58:43.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout id: 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T07:58:43.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-04-24T07:58:43.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-24T07:58:43.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout services: 2024-04-24T07:58:43.780 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.779746s) 2024-04-24T07:58:43.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-04-24T07:58:43.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-04-24T07:58:43.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-24T07:58:43.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout data: 2024-04-24T07:58:43.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-04-24T07:58:43.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-04-24T07:58:43.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-04-24T07:58:43.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout pgs: 2024-04-24T07:58:43.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-24T07:58:43.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.002+0000 7efcfca00700 1 Processor -- start 2024-04-24T07:58:43.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.002+0000 7efcfca00700 1 -- start start 2024-04-24T07:58:43.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.003+0000 7efcfca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf810a980 0x7efcf810ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:58:43.781 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.003+0000 7efcfca00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efcf810b370 con 0x7efcf810a980 2024-04-24T07:58:43.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.004+0000 7efcf5e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf810a980 0x7efcf810ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:58:43.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.004+0000 7efcf5e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf810a980 0x7efcf810ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50410/0 (socket says 172.21.15.28:50410) 2024-04-24T07:58:43.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.004+0000 7efcf5e00700 1 -- 172.21.15.28:0/3156007287 learned_addr learned my addr 172.21.15.28:0/3156007287 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:58:43.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.005+0000 7efcf5e00700 1 -- 172.21.15.28:0/3156007287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf810bb30 con 0x7efcf810a980 2024-04-24T07:58:43.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.005+0000 7efcf5e00700 1 --2- 172.21.15.28:0/3156007287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf810a980 0x7efcf810ada0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7efce80098a0 tx=0x7efce8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e309f7cc3ab2bffd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:58:43.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.006+0000 7efcf4a00700 1 -- 172.21.15.28:0/3156007287 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efce8004160 con 0x7efcf810a980 2024-04-24T07:58:43.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.006+0000 7efcf4a00700 1 -- 172.21.15.28:0/3156007287 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efce8004300 con 0x7efcf810a980 2024-04-24T07:58:43.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.006+0000 7efcf4a00700 1 -- 172.21.15.28:0/3156007287 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efce80036a0 con 0x7efcf810a980 2024-04-24T07:58:43.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.007+0000 7efcfca00700 1 -- 172.21.15.28:0/3156007287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf810a980 msgr2=0x7efcf810ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:58:43.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.007+0000 7efcfca00700 1 --2- 172.21.15.28:0/3156007287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf810a980 0x7efcf810ada0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7efce80098a0 tx=0x7efce8009bb0 comp rx=0 tx=0).stop 2024-04-24T07:58:43.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.007+0000 7efcfca00700 1 -- 172.21.15.28:0/3156007287 shutdown_connections 2024-04-24T07:58:43.782 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.007+0000 7efcfca00700 1 --2- 172.21.15.28:0/3156007287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf810a980 0x7efcf810ada0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:58:43.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.007+0000 7efcfca00700 1 -- 172.21.15.28:0/3156007287 >> 172.21.15.28:0/3156007287 conn(0x7efcf807def0 msgr2=0x7efcf81084f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:58:43.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.007+0000 7efcfca00700 1 -- 172.21.15.28:0/3156007287 shutdown_connections 2024-04-24T07:58:43.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.007+0000 7efcfca00700 1 -- 172.21.15.28:0/3156007287 wait complete. 2024-04-24T07:58:43.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.008+0000 7efcfca00700 1 Processor -- start 2024-04-24T07:58:43.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.008+0000 7efcfca00700 1 -- start start 2024-04-24T07:58:43.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.008+0000 7efcfca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf810a980 0x7efcf819dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:58:43.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.008+0000 7efcfca00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efcf810c060 con 0x7efcf810a980 2024-04-24T07:58:43.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.009+0000 7efcf5e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf810a980 0x7efcf819dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:58:43.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.009+0000 7efcf5e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf810a980 0x7efcf819dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50426/0 (socket says 172.21.15.28:50426) 2024-04-24T07:58:43.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.009+0000 7efcf5e00700 1 -- 172.21.15.28:0/1308736385 learned_addr learned my addr 172.21.15.28:0/1308736385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:58:43.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.009+0000 7efcf5e00700 1 -- 172.21.15.28:0/1308736385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efcf819e4e0 con 0x7efcf810a980 2024-04-24T07:58:43.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.010+0000 7efcf5e00700 1 --2- 172.21.15.28:0/1308736385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf810a980 0x7efcf819dfa0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7efce8000c00 tx=0x7efce8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:58:43.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.010+0000 7efceea00700 1 -- 172.21.15.28:0/1308736385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efce8003db0 con 0x7efcf810a980 2024-04-24T07:58:43.783 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.010+0000 7efceea00700 1 -- 172.21.15.28:0/1308736385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7efce8011630 con 0x7efcf810a980 2024-04-24T07:58:43.784 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.010+0000 7efceea00700 1 -- 172.21.15.28:0/1308736385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7efce8011950 con 0x7efcf810a980 2024-04-24T07:58:43.784 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.010+0000 7efcfca00700 1 -- 172.21.15.28:0/1308736385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efcf819e770 con 0x7efcf810a980 2024-04-24T07:58:43.784 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.010+0000 7efcfca00700 1 -- 172.21.15.28:0/1308736385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efcf81a1460 con 0x7efcf810a980 2024-04-24T07:58:43.784 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.011+0000 7efceea00700 1 -- 172.21.15.28:0/1308736385 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7efce8011af0 con 0x7efcf810a980 2024-04-24T07:58:43.784 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.011+0000 7efceea00700 1 -- 172.21.15.28:0/1308736385 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7efce801c450 con 0x7efcf810a980 2024-04-24T07:58:43.784 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.011+0000 7efcfca00700 1 -- 172.21.15.28:0/1308736385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efcf8064150 con 0x7efcf810a980 2024-04-24T07:58:43.784 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.015+0000 7efceea00700 1 -- 172.21.15.28:0/1308736385 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7efce8032020 con 0x7efcf810a980 2024-04-24T07:58:43.784 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.095+0000 7efcfca00700 1 -- 172.21.15.28:0/1308736385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7efcf802e160 con 0x7efcf810a980 2024-04-24T07:58:43.784 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.095+0000 7efceea00700 1 -- 172.21.15.28:0/1308736385 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7efce8004260 con 0x7efcf810a980 2024-04-24T07:58:43.784 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.097+0000 7efcfca00700 1 -- 172.21.15.28:0/1308736385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf810a980 msgr2=0x7efcf819dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:58:43.784 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.097+0000 7efcfca00700 1 --2- 172.21.15.28:0/1308736385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf810a980 0x7efcf819dfa0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7efce8000c00 tx=0x7efce8004750 comp rx=0 tx=0).stop 2024-04-24T07:58:43.784 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.097+0000 7efcfca00700 1 -- 172.21.15.28:0/1308736385 shutdown_connections 2024-04-24T07:58:43.784 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.097+0000 7efcfca00700 1 --2- 172.21.15.28:0/1308736385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efcf810a980 0x7efcf819dfa0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:58:43.784 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.097+0000 7efcfca00700 1 -- 172.21.15.28:0/1308736385 >> 172.21.15.28:0/1308736385 conn(0x7efcf807def0 msgr2=0x7efcf8194be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:58:43.785 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.097+0000 7efcfca00700 1 -- 172.21.15.28:0/1308736385 shutdown_connections 2024-04-24T07:58:43.785 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:43.097+0000 7efcfca00700 1 -- 172.21.15.28:0/1308736385 wait complete. 2024-04-24T07:58:43.785 INFO:teuthology.orchestra.run.smithi028.stdout:mon is available 2024-04-24T07:58:43.785 INFO:teuthology.orchestra.run.smithi028.stdout:Assimilating anything we can from ceph.conf... 2024-04-24T07:58:44.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:44 smithi028 ceph-mon[27419]: from='client.? 172.21.15.28:0/1242896951' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-24T07:58:44.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:44 smithi028 ceph-mon[27419]: from='client.? 172.21.15.28:0/1242896951' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-04-24T07:58:45.071 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-24T07:58:45.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-04-24T07:58:45.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T07:58:45.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.28:3300,v1:172.21.15.28:6789] 2024-04-24T07:58:45.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-24T07:58:45.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-24T07:58:45.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-24T07:58:45.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-24T07:58:45.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-24T07:58:45.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-24T07:58:45.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-24T07:58:45.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-24T07:58:45.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-04-24T07:58:45.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-04-24T07:58:45.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-24T07:58:45.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-24T07:58:45.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-24T07:58:45.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-24T07:58:45.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-24T07:58:45.077 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.353+0000 7effba200700 1 Processor -- start 2024-04-24T07:58:45.077 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.354+0000 7effba200700 1 -- start start 2024-04-24T07:58:45.077 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.354+0000 7effba200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb410a780 0x7effb410aba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:58:45.077 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.354+0000 7effba200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7effb410b170 con 0x7effb410a780 2024-04-24T07:58:45.078 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.354+0000 7effb8e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb410a780 0x7effb410aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:58:45.078 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.355+0000 7effb8e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb410a780 0x7effb410aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50434/0 (socket says 172.21.15.28:50434) 2024-04-24T07:58:45.079 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.355+0000 7effb8e00700 1 -- 172.21.15.28:0/1370018185 learned_addr learned my addr 172.21.15.28:0/1370018185 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:58:45.079 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.355+0000 7effb8e00700 1 -- 172.21.15.28:0/1370018185 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb410b8e0 con 0x7effb410a780 2024-04-24T07:58:45.079 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.355+0000 7effb8e00700 1 --2- 172.21.15.28:0/1370018185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb410a780 0x7effb410aba0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7effa80098a0 tx=0x7effa8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2749441759061f3e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:58:45.080 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.356+0000 7effb3400700 1 -- 172.21.15.28:0/1370018185 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effa8004160 con 0x7effb410a780 2024-04-24T07:58:45.080 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.356+0000 7effb3400700 1 -- 172.21.15.28:0/1370018185 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effa8004300 con 0x7effb410a780 2024-04-24T07:58:45.080 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.356+0000 7effb3400700 1 -- 172.21.15.28:0/1370018185 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effa80036a0 con 0x7effb410a780 2024-04-24T07:58:45.080 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.356+0000 7effba200700 1 -- 172.21.15.28:0/1370018185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb410a780 msgr2=0x7effb410aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:58:45.080 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.356+0000 7effba200700 1 --2- 172.21.15.28:0/1370018185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb410a780 0x7effb410aba0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7effa80098a0 tx=0x7effa8009bb0 comp rx=0 tx=0).stop 2024-04-24T07:58:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.356+0000 7effba200700 1 -- 172.21.15.28:0/1370018185 shutdown_connections 2024-04-24T07:58:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.356+0000 7effba200700 1 --2- 172.21.15.28:0/1370018185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb410a780 0x7effb410aba0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:58:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.356+0000 7effba200700 1 -- 172.21.15.28:0/1370018185 >> 172.21.15.28:0/1370018185 conn(0x7effb407de30 msgr2=0x7effb41084f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:58:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.357+0000 7effba200700 1 -- 172.21.15.28:0/1370018185 shutdown_connections 2024-04-24T07:58:45.081 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.357+0000 7effba200700 1 -- 172.21.15.28:0/1370018185 wait complete. 2024-04-24T07:58:45.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.358+0000 7effba200700 1 Processor -- start 2024-04-24T07:58:45.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.358+0000 7effba200700 1 -- start start 2024-04-24T07:58:45.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.358+0000 7effba200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb410a780 0x7effb419ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:58:45.082 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.358+0000 7effba200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7effb410be10 con 0x7effb410a780 2024-04-24T07:58:45.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.359+0000 7effb8e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb410a780 0x7effb419ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:58:45.083 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.359+0000 7effb8e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb410a780 0x7effb419ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50438/0 (socket says 172.21.15.28:50438) 2024-04-24T07:58:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.359+0000 7effb8e00700 1 -- 172.21.15.28:0/1242896951 learned_addr learned my addr 172.21.15.28:0/1242896951 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:58:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.359+0000 7effb8e00700 1 -- 172.21.15.28:0/1242896951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb419e330 con 0x7effb410a780 2024-04-24T07:58:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.359+0000 7effb8e00700 1 --2- 172.21.15.28:0/1242896951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb410a780 0x7effb419ddf0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7effa8000c00 tx=0x7effa8004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:58:45.084 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.360+0000 7effb1600700 1 -- 172.21.15.28:0/1242896951 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effa8003db0 con 0x7effb410a780 2024-04-24T07:58:45.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.360+0000 7effb1600700 1 -- 172.21.15.28:0/1242896951 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7effa8011630 con 0x7effb410a780 2024-04-24T07:58:45.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.360+0000 7effb1600700 1 -- 172.21.15.28:0/1242896951 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7effa8011950 con 0x7effb410a780 2024-04-24T07:58:45.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.360+0000 7effba200700 1 -- 172.21.15.28:0/1242896951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effb419e5c0 con 0x7effb410a780 2024-04-24T07:58:45.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.360+0000 7effba200700 1 -- 172.21.15.28:0/1242896951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effb41a12b0 con 0x7effb410a780 2024-04-24T07:58:45.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.360+0000 7effb1600700 1 -- 172.21.15.28:0/1242896951 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7effa801a460 con 0x7effb410a780 2024-04-24T07:58:45.085 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.360+0000 7effb1600700 1 -- 172.21.15.28:0/1242896951 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7effa801ba50 con 0x7effb410a780 2024-04-24T07:58:45.086 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.361+0000 7effba200700 1 -- 172.21.15.28:0/1242896951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effb4064150 con 0x7effb410a780 2024-04-24T07:58:45.086 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.364+0000 7effb1600700 1 -- 172.21.15.28:0/1242896951 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7effa8004260 con 0x7effb410a780 2024-04-24T07:58:45.087 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.442+0000 7effba200700 1 -- 172.21.15.28:0/1242896951 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7effb419eaa0 con 0x7effb410a780 2024-04-24T07:58:45.087 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.451+0000 7effb1600700 1 -- 172.21.15.28:0/1242896951 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+454 (secure 0 0 0) 0x7effb419eaa0 con 0x7effb410a780 2024-04-24T07:58:45.087 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.452+0000 7effb1600700 1 -- 172.21.15.28:0/1242896951 <== mon.0 v2:172.21.15.28:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7effa801ada0 con 0x7effb410a780 2024-04-24T07:58:45.087 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.452+0000 7effba200700 1 -- 172.21.15.28:0/1242896951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb410a780 msgr2=0x7effb419ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:58:45.088 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.453+0000 7effba200700 1 --2- 172.21.15.28:0/1242896951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb410a780 0x7effb419ddf0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7effa8000c00 tx=0x7effa8004750 comp rx=0 tx=0).stop 2024-04-24T07:58:45.088 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.453+0000 7effba200700 1 -- 172.21.15.28:0/1242896951 shutdown_connections 2024-04-24T07:58:45.088 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.453+0000 7effba200700 1 --2- 172.21.15.28:0/1242896951 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb410a780 0x7effb419ddf0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:58:45.088 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.453+0000 7effba200700 1 -- 172.21.15.28:0/1242896951 >> 172.21.15.28:0/1242896951 conn(0x7effb407de30 msgr2=0x7effb4194a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:58:45.088 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.453+0000 7effba200700 1 -- 172.21.15.28:0/1242896951 shutdown_connections 2024-04-24T07:58:45.089 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:44.453+0000 7effba200700 1 -- 172.21.15.28:0/1242896951 wait complete. 2024-04-24T07:58:45.089 INFO:teuthology.orchestra.run.smithi028.stdout:Generating new minimal ceph.conf... 2024-04-24T07:58:46.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:45 smithi028 ceph-mon[27419]: from='client.? 172.21.15.28:0/3058956980' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T07:58:46.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.676+0000 7fd7a3600700 1 Processor -- start 2024-04-24T07:58:46.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.677+0000 7fd7a3600700 1 -- start start 2024-04-24T07:58:46.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.677+0000 7fd7a3600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd79c10a900 0x7fd79c10ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:58:46.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.677+0000 7fd7a3600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd79c10b260 con 0x7fd79c10a900 2024-04-24T07:58:46.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.678+0000 7fd7a0c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd79c10a900 0x7fd79c10ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:58:46.410 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.678+0000 7fd7a0c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd79c10a900 0x7fd79c10ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50442/0 (socket says 172.21.15.28:50442) 2024-04-24T07:58:46.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.678+0000 7fd7a0c00700 1 -- 172.21.15.28:0/4008374257 learned_addr learned my addr 172.21.15.28:0/4008374257 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:58:46.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.678+0000 7fd7a0c00700 1 -- 172.21.15.28:0/4008374257 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd79c10b3e0 con 0x7fd79c10a900 2024-04-24T07:58:46.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.679+0000 7fd7a0c00700 1 --2- 172.21.15.28:0/4008374257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd79c10a900 0x7fd79c10ad20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd7900098a0 tx=0x7fd790009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=feaaca1002c86ffc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:58:46.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.679+0000 7fd79b400700 1 -- 172.21.15.28:0/4008374257 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd790004160 con 0x7fd79c10a900 2024-04-24T07:58:46.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.679+0000 7fd79b400700 1 -- 172.21.15.28:0/4008374257 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd790004300 con 0x7fd79c10a900 2024-04-24T07:58:46.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.679+0000 7fd79b400700 1 -- 172.21.15.28:0/4008374257 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd790011b50 con 0x7fd79c10a900 2024-04-24T07:58:46.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.680+0000 7fd7a3600700 1 -- 172.21.15.28:0/4008374257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd79c10a900 msgr2=0x7fd79c10ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:58:46.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.680+0000 7fd7a3600700 1 --2- 172.21.15.28:0/4008374257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd79c10a900 0x7fd79c10ad20 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fd7900098a0 tx=0x7fd790009bb0 comp rx=0 tx=0).stop 2024-04-24T07:58:46.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.680+0000 7fd7a3600700 1 -- 172.21.15.28:0/4008374257 shutdown_connections 2024-04-24T07:58:46.412 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.680+0000 7fd7a3600700 1 --2- 172.21.15.28:0/4008374257 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd79c10a900 0x7fd79c10ad20 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:58:46.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.680+0000 7fd7a3600700 1 -- 172.21.15.28:0/4008374257 >> 172.21.15.28:0/4008374257 conn(0x7fd79c07ded0 msgr2=0x7fd79c1084f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:58:46.415 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.680+0000 7fd7a3600700 1 -- 172.21.15.28:0/4008374257 shutdown_connections 2024-04-24T07:58:46.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.680+0000 7fd7a3600700 1 -- 172.21.15.28:0/4008374257 wait complete. 2024-04-24T07:58:46.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.681+0000 7fd7a3600700 1 Processor -- start 2024-04-24T07:58:46.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.681+0000 7fd7a3600700 1 -- start start 2024-04-24T07:58:46.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.682+0000 7fd7a3600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd79c10a900 0x7fd79c19e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:58:46.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.682+0000 7fd7a3600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd79c10ed70 con 0x7fd79c10a900 2024-04-24T07:58:46.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.682+0000 7fd7a0c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd79c10a900 0x7fd79c19e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:58:46.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.682+0000 7fd7a0c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd79c10a900 0x7fd79c19e7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50452/0 (socket says 172.21.15.28:50452) 2024-04-24T07:58:46.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.682+0000 7fd7a0c00700 1 -- 172.21.15.28:0/3058956980 learned_addr learned my addr 172.21.15.28:0/3058956980 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:58:46.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.682+0000 7fd7a0c00700 1 -- 172.21.15.28:0/3058956980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd79c19ed20 con 0x7fd79c10a900 2024-04-24T07:58:46.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.683+0000 7fd7a0c00700 1 --2- 172.21.15.28:0/3058956980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd79c10a900 0x7fd79c19e7e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd790011670 tx=0x7fd790011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:58:46.416 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.683+0000 7fd799600700 1 -- 172.21.15.28:0/3058956980 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7900039c0 con 0x7fd79c10a900 2024-04-24T07:58:46.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.683+0000 7fd7a3600700 1 -- 172.21.15.28:0/3058956980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd79c19efb0 con 0x7fd79c10a900 2024-04-24T07:58:46.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.683+0000 7fd799600700 1 -- 172.21.15.28:0/3058956980 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fd790011e50 con 0x7fd79c10a900 2024-04-24T07:58:46.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.683+0000 7fd799600700 1 -- 172.21.15.28:0/3058956980 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd79001a5c0 con 0x7fd79c10a900 2024-04-24T07:58:46.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.683+0000 7fd7a3600700 1 -- 172.21.15.28:0/3058956980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd79c1a1ca0 con 0x7fd79c10a900 2024-04-24T07:58:46.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.683+0000 7fd799600700 1 -- 172.21.15.28:0/3058956980 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fd79001ac50 con 0x7fd79c10a900 2024-04-24T07:58:46.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.684+0000 7fd799600700 1 -- 172.21.15.28:0/3058956980 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fd790019700 con 0x7fd79c10a900 2024-04-24T07:58:46.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.684+0000 7fd7a3600700 1 -- 172.21.15.28:0/3058956980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd79c064150 con 0x7fd79c10a900 2024-04-24T07:58:46.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.687+0000 7fd799600700 1 -- 172.21.15.28:0/3058956980 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fd790003b60 con 0x7fd79c10a900 2024-04-24T07:58:46.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.766+0000 7fd7a3600700 1 -- 172.21.15.28:0/3058956980 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fd79c1a1e70 con 0x7fd79c10a900 2024-04-24T07:58:46.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.767+0000 7fd799600700 1 -- 172.21.15.28:0/3058956980 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fd7900199a0 con 0x7fd79c10a900 2024-04-24T07:58:46.417 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.768+0000 7fd7a3600700 1 -- 172.21.15.28:0/3058956980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd79c10a900 msgr2=0x7fd79c19e7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:58:46.421 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.768+0000 7fd7a3600700 1 --2- 172.21.15.28:0/3058956980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd79c10a900 0x7fd79c19e7e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fd790011670 tx=0x7fd790011b60 comp rx=0 tx=0).stop 2024-04-24T07:58:46.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.768+0000 7fd7a3600700 1 -- 172.21.15.28:0/3058956980 shutdown_connections 2024-04-24T07:58:46.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.768+0000 7fd7a3600700 1 --2- 172.21.15.28:0/3058956980 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd79c10a900 0x7fd79c19e7e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:58:46.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.768+0000 7fd7a3600700 1 -- 172.21.15.28:0/3058956980 >> 172.21.15.28:0/3058956980 conn(0x7fd79c07ded0 msgr2=0x7fd79c106b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:58:46.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.769+0000 7fd7a3600700 1 -- 172.21.15.28:0/3058956980 shutdown_connections 2024-04-24T07:58:46.422 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:45.769+0000 7fd7a3600700 1 -- 172.21.15.28:0/3058956980 wait complete. 2024-04-24T07:58:46.422 INFO:teuthology.orchestra.run.smithi028.stdout:Restarting the monitor... 2024-04-24T07:58:46.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:46 smithi028 systemd[1]: Stopping Ceph mon.a for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T07:58:47.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:46 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a[27414]: 2024-04-24T07:58:46.924+0000 7fb297e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T07:58:47.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:46 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a[27414]: 2024-04-24T07:58:46.924+0000 7fb297e00700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-04-24T07:58:47.273 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:47 smithi028 podman[27664]: 2024-04-24 07:58:47.049576365 +0000 UTC m=+0.245760276 container died d0e0813f1dba3dad985bc481c8801ac1ff9839b46dd4d90c99ecaf5c004bc7bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True) 2024-04-24T07:58:47.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:47 smithi028 podman[27664]: 2024-04-24 07:58:47.384551247 +0000 UTC m=+0.580735144 container cleanup d0e0813f1dba3dad985bc481c8801ac1ff9839b46dd4d90c99ecaf5c004bc7bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-04-24T07:58:47.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:47 smithi028 bash[27664]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a 2024-04-24T07:58:47.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:47 smithi028 podman[27679]: 2024-04-24 07:58:47.546481019 +0000 UTC m=+0.482556775 container remove d0e0813f1dba3dad985bc481c8801ac1ff9839b46dd4d90c99ecaf5c004bc7bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-24T07:58:48.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 systemd[1]: ceph-473aaf58-0210-11ef-bc93-c7b262605968@mon.a.service: Deactivated successfully. 2024-04-24T07:58:48.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 systemd[1]: Stopped Ceph mon.a for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T07:58:48.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 systemd[1]: ceph-473aaf58-0210-11ef-bc93-c7b262605968@mon.a.service: Consumed 1.928s CPU time. 2024-04-24T07:58:48.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 systemd[1]: Starting Ceph mon.a for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T07:58:48.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 podman[27767]: 2024-04-24 07:58:48.317656528 +0000 UTC m=+0.032831392 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T07:58:48.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 podman[27767]: 2024-04-24 07:58:48.420173673 +0000 UTC m=+0.135348601 container create 29ad43385eba63f095616e23346567f09deee4aba577f58b5fe3980f1d535002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T07:58:48.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 podman[27767]: 2024-04-24 07:58:48.657264779 +0000 UTC m=+0.372439690 container init 29ad43385eba63f095616e23346567f09deee4aba577f58b5fe3980f1d535002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424) 2024-04-24T07:58:48.670 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 podman[27767]: 2024-04-24 07:58:48.669579506 +0000 UTC m=+0.384754410 container start 29ad43385eba63f095616e23346567f09deee4aba577f58b5fe3980f1d535002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-24T07:58:48.804 INFO:teuthology.orchestra.run.smithi028.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-04-24T07:58:48.959 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: set uid:gid to 167:167 (ceph:ceph) 2024-04-24T07:58:48.959 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: ceph version 19.0.0-3116-g1f81fb39 (1f81fb397ae98da7563d451a78c61574c8f4e6e0) squid (dev), process ceph-mon, pid 2 2024-04-24T07:58:48.959 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: pidfile_write: ignore empty --pid-file 2024-04-24T07:58:48.959 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: load: jerasure load: lrc 2024-04-24T07:58:48.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: RocksDB version: 7.9.2 2024-04-24T07:58:48.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Git sha 0 2024-04-24T07:58:48.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Compile date 2024-04-23 21:06:33 2024-04-24T07:58:48.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: DB SUMMARY 2024-04-24T07:58:48.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: DB Session ID: 562BSJ7NUECMUCKVTFAG 2024-04-24T07:58:48.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: CURRENT file: CURRENT 2024-04-24T07:58:48.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: IDENTITY file: IDENTITY 2024-04-24T07:58:48.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-04-24T07:58:48.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-04-24T07:58:48.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 106126 ; 2024-04-24T07:58:48.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.error_if_exists: 0 2024-04-24T07:58:48.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.create_if_missing: 0 2024-04-24T07:58:48.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.paranoid_checks: 1 2024-04-24T07:58:48.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.flush_verify_memtable_count: 1 2024-04-24T07:58:48.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-04-24T07:58:48.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-04-24T07:58:48.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.env: 0x55c7c202acc0 2024-04-24T07:58:48.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.fs: PosixFileSystem 2024-04-24T07:58:48.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.info_log: 0x55c7fe9b9c60 2024-04-24T07:58:48.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_file_opening_threads: 16 2024-04-24T07:58:48.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.statistics: (nil) 2024-04-24T07:58:48.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.use_fsync: 0 2024-04-24T07:58:48.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_log_file_size: 0 2024-04-24T07:58:48.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-04-24T07:58:48.961 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.log_file_time_to_roll: 0 2024-04-24T07:58:48.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.keep_log_file_num: 1000 2024-04-24T07:58:48.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.recycle_log_file_num: 0 2024-04-24T07:58:48.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.allow_fallocate: 1 2024-04-24T07:58:48.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.allow_mmap_reads: 0 2024-04-24T07:58:48.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.allow_mmap_writes: 0 2024-04-24T07:58:48.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.use_direct_reads: 0 2024-04-24T07:58:48.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-04-24T07:58:48.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.create_missing_column_families: 0 2024-04-24T07:58:48.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.db_log_dir: 2024-04-24T07:58:48.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.wal_dir: 2024-04-24T07:58:48.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.table_cache_numshardbits: 6 2024-04-24T07:58:48.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.WAL_ttl_seconds: 0 2024-04-24T07:58:48.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.WAL_size_limit_MB: 0 2024-04-24T07:58:48.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-04-24T07:58:48.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-04-24T07:58:48.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.is_fd_close_on_exec: 1 2024-04-24T07:58:48.963 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.advise_random_on_open: 1 2024-04-24T07:58:48.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.db_write_buffer_size: 0 2024-04-24T07:58:48.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.write_buffer_manager: 0x55c7fe9881e0 2024-04-24T07:58:48.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-04-24T07:58:48.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-04-24T07:58:48.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.use_adaptive_mutex: 0 2024-04-24T07:58:48.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.rate_limiter: (nil) 2024-04-24T07:58:48.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-04-24T07:58:48.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.wal_recovery_mode: 2 2024-04-24T07:58:48.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.enable_thread_tracking: 0 2024-04-24T07:58:48.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.enable_pipelined_write: 0 2024-04-24T07:58:48.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.unordered_write: 0 2024-04-24T07:58:48.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-04-24T07:58:48.964 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-04-24T07:58:48.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-04-24T07:58:48.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-04-24T07:58:48.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.row_cache: None 2024-04-24T07:58:48.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.wal_filter: None 2024-04-24T07:58:48.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-04-24T07:58:48.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.allow_ingest_behind: 0 2024-04-24T07:58:48.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.two_write_queues: 0 2024-04-24T07:58:48.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.manual_wal_flush: 0 2024-04-24T07:58:48.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.wal_compression: 0 2024-04-24T07:58:48.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.atomic_flush: 0 2024-04-24T07:58:48.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-04-24T07:58:48.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.persist_stats_to_disk: 0 2024-04-24T07:58:48.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.write_dbid_to_manifest: 0 2024-04-24T07:58:48.965 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.log_readahead_size: 0 2024-04-24T07:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-04-24T07:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.best_efforts_recovery: 0 2024-04-24T07:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-04-24T07:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-04-24T07:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.allow_data_in_errors: 0 2024-04-24T07:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.db_host_id: __hostname__ 2024-04-24T07:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.enforce_single_del_contracts: true 2024-04-24T07:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_background_jobs: 2 2024-04-24T07:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_background_compactions: -1 2024-04-24T07:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_subcompactions: 1 2024-04-24T07:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-04-24T07:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-04-24T07:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.delayed_write_rate : 16777216 2024-04-24T07:58:48.966 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_total_wal_size: 0 2024-04-24T07:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-04-24T07:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.stats_dump_period_sec: 600 2024-04-24T07:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.stats_persist_period_sec: 600 2024-04-24T07:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-04-24T07:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_open_files: -1 2024-04-24T07:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.bytes_per_sync: 0 2024-04-24T07:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.wal_bytes_per_sync: 0 2024-04-24T07:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.strict_bytes_per_sync: 0 2024-04-24T07:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compaction_readahead_size: 0 2024-04-24T07:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_background_flushes: -1 2024-04-24T07:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Compression algorithms supported: 2024-04-24T07:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-04-24T07:58:48.967 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: kZSTD supported: 0 2024-04-24T07:58:48.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: kXpressCompression supported: 0 2024-04-24T07:58:48.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: kLZ4HCCompression supported: 1 2024-04-24T07:58:48.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: kZlibCompression supported: 1 2024-04-24T07:58:48.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: kSnappyCompression supported: 1 2024-04-24T07:58:48.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: kLZ4Compression supported: 1 2024-04-24T07:58:48.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: kBZip2Compression supported: 0 2024-04-24T07:58:48.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Fast CRC32 supported: Supported on x86 2024-04-24T07:58:48.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: DMutex implementation: pthread_mutex_t 2024-04-24T07:58:48.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-04-24T07:58:48.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-04-24T07:58:48.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-04-24T07:58:48.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.merge_operator: 2024-04-24T07:58:48.968 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compaction_filter: None 2024-04-24T07:58:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compaction_filter_factory: None 2024-04-24T07:58:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.sst_partitioner_factory: None 2024-04-24T07:58:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.memtable_factory: SkipListFactory 2024-04-24T07:58:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.table_factory: BlockBasedTable 2024-04-24T07:58:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55c7fe9b9e20) 2024-04-24T07:58:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks: 1 2024-04-24T07:58:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-04-24T07:58:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-04-24T07:58:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout: pin_top_level_index_and_filter: 1 2024-04-24T07:58:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_type: 0 2024-04-24T07:58:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_index_type: 0 2024-04-24T07:58:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_shortening: 1 2024-04-24T07:58:48.969 INFO:journalctl@ceph.mon.a.smithi028.stdout: data_block_hash_table_util_ratio: 0.750000 2024-04-24T07:58:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout: checksum: 4 2024-04-24T07:58:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout: no_block_cache: 0 2024-04-24T07:58:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache: 0x55c7fe9bb090 2024-04-24T07:58:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_name: BinnedLRUCache 2024-04-24T07:58:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_options: 2024-04-24T07:58:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout: capacity : 536870912 2024-04-24T07:58:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_shard_bits : 4 2024-04-24T07:58:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout: strict_capacity_limit : 0 2024-04-24T07:58:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout: high_pri_pool_ratio: 0.000 2024-04-24T07:58:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_cache_compressed: (nil) 2024-04-24T07:58:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout: persistent_cache: (nil) 2024-04-24T07:58:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size: 4096 2024-04-24T07:58:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_size_deviation: 10 2024-04-24T07:58:48.970 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_restart_interval: 16 2024-04-24T07:58:48.971 INFO:journalctl@ceph.mon.a.smithi028.stdout: index_block_restart_interval: 1 2024-04-24T07:58:48.971 INFO:journalctl@ceph.mon.a.smithi028.stdout: metadata_block_size: 4096 2024-04-24T07:58:48.971 INFO:journalctl@ceph.mon.a.smithi028.stdout: partition_filters: 0 2024-04-24T07:58:48.971 INFO:journalctl@ceph.mon.a.smithi028.stdout: use_delta_encoding: 1 2024-04-24T07:58:48.971 INFO:journalctl@ceph.mon.a.smithi028.stdout: filter_policy: bloomfilter 2024-04-24T07:58:48.971 INFO:journalctl@ceph.mon.a.smithi028.stdout: whole_key_filtering: 1 2024-04-24T07:58:48.971 INFO:journalctl@ceph.mon.a.smithi028.stdout: verify_compression: 0 2024-04-24T07:58:48.971 INFO:journalctl@ceph.mon.a.smithi028.stdout: read_amp_bytes_per_bit: 0 2024-04-24T07:58:48.971 INFO:journalctl@ceph.mon.a.smithi028.stdout: format_version: 5 2024-04-24T07:58:48.971 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_index_compression: 1 2024-04-24T07:58:48.971 INFO:journalctl@ceph.mon.a.smithi028.stdout: block_align: 0 2024-04-24T07:58:48.971 INFO:journalctl@ceph.mon.a.smithi028.stdout: max_auto_readahead_size: 262144 2024-04-24T07:58:48.971 INFO:journalctl@ceph.mon.a.smithi028.stdout: prepopulate_block_cache: 0 2024-04-24T07:58:48.971 INFO:journalctl@ceph.mon.a.smithi028.stdout: initial_auto_readahead_size: 8192 2024-04-24T07:58:48.971 INFO:journalctl@ceph.mon.a.smithi028.stdout: num_file_reads_for_auto_readahead: 2 2024-04-24T07:58:48.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.write_buffer_size: 33554432 2024-04-24T07:58:48.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_write_buffer_number: 2 2024-04-24T07:58:48.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compression: NoCompression 2024-04-24T07:58:48.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.bottommost_compression: Disabled 2024-04-24T07:58:48.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.prefix_extractor: nullptr 2024-04-24T07:58:48.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-04-24T07:58:48.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.num_levels: 7 2024-04-24T07:58:48.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-04-24T07:58:48.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-04-24T07:58:48.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-04-24T07:58:48.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-04-24T07:58:48.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-04-24T07:58:48.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-04-24T07:58:48.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-04-24T07:58:48.972 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-04-24T07:58:48.973 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-04-24T07:58:48.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-04-24T07:58:48.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-04-24T07:58:48.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-04-24T07:58:48.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compression_opts.window_bits: -14 2024-04-24T07:58:48.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compression_opts.level: 32767 2024-04-24T07:58:48.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compression_opts.strategy: 0 2024-04-24T07:58:48.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-04-24T07:58:48.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-04-24T07:58:48.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-04-24T07:58:48.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-04-24T07:58:48.974 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compression_opts.enabled: false 2024-04-24T07:58:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-04-24T07:58:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-04-24T07:58:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-04-24T07:58:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-04-24T07:58:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.target_file_size_base: 67108864 2024-04-24T07:58:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.target_file_size_multiplier: 1 2024-04-24T07:58:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-04-24T07:58:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-04-24T07:58:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-04-24T07:58:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-04-24T07:58:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-04-24T07:58:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-04-24T07:58:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-04-24T07:58:48.975 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-04-24T07:58:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-04-24T07:58:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-04-24T07:58:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-04-24T07:58:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-04-24T07:58:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-04-24T07:58:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.arena_block_size: 1048576 2024-04-24T07:58:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-04-24T07:58:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-04-24T07:58:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.disable_auto_compactions: 0 2024-04-24T07:58:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-04-24T07:58:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-04-24T07:58:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-04-24T07:58:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-04-24T07:58:48.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-04-24T07:58:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-04-24T07:58:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-04-24T07:58:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-04-24T07:58:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-04-24T07:58:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-04-24T07:58:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.table_properties_collectors: CompactOnDeletionCollector (Sliding window size = 32768 Deletion trigger = 16384 Deletion ratio = 0); 2024-04-24T07:58:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.inplace_update_support: 0 2024-04-24T07:58:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.inplace_update_num_locks: 10000 2024-04-24T07:58:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-04-24T07:58:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-04-24T07:58:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.memtable_huge_page_size: 0 2024-04-24T07:58:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.bloom_locality: 0 2024-04-24T07:58:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.max_successive_merges: 0 2024-04-24T07:58:48.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.optimize_filters_for_hits: 0 2024-04-24T07:58:48.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.paranoid_file_checks: 0 2024-04-24T07:58:48.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.force_consistency_checks: 1 2024-04-24T07:58:48.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.report_bg_io_stats: 0 2024-04-24T07:58:48.978 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.ttl: 2592000 2024-04-24T07:58:48.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.periodic_compaction_seconds: 0 2024-04-24T07:58:48.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-04-24T07:58:48.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-04-24T07:58:48.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.enable_blob_files: false 2024-04-24T07:58:48.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.min_blob_size: 0 2024-04-24T07:58:48.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.blob_file_size: 268435456 2024-04-24T07:58:48.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.blob_compression_type: NoCompression 2024-04-24T07:58:48.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.enable_blob_garbage_collection: false 2024-04-24T07:58:48.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-04-24T07:58:48.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-04-24T07:58:48.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-04-24T07:58:48.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.blob_file_starting_level: 0 2024-04-24T07:58:48.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-04-24T07:58:48.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-04-24T07:58:48.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-04-24T07:58:48.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 58350fb7-3ceb-46b9-828f-aaf673a200a3 2024-04-24T07:58:48.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713945528729426, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-04-24T07:58:48.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-04-24T07:58:48.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713945528730985, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 101126, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 330, "table_properties": {"data_size": 99025, "index_size": 293, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14423, "raw_average_key_size": 49, "raw_value_size": 91228, "raw_average_value_size": 315, "num_data_blocks": 12, "num_entries": 289, "num_filter_entries": 289, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[CompactOnDeletionCollector]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1713945528, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "58350fb7-3ceb-46b9-828f-aaf673a200a3", "db_session_id": "562BSJ7NUECMUCKVTFAG", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-04-24T07:58:48.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: EVENT_LOG_v1 {"time_micros": 1713945528731142, "job": 1, "event": "recovery_finished"} 2024-04-24T07:58:48.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-04-24T07:58:48.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-04-24T07:58:48.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55c7feaa8000 2024-04-24T07:58:48.980 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: DB pointer 0x55c7fea92000 2024-04-24T07:58:48.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-04-24T07:58:48.981 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-04-24T07:58:48.981 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** DB Stats ** 2024-04-24T07:58:48.981 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T07:58:48.981 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-04-24T07:58:48.981 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T07:58:48.981 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T07:58:48.981 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-04-24T07:58:48.982 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-04-24T07:58:48.982 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-04-24T07:58:48.982 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-24T07:58:48.982 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-04-24T07:58:48.982 INFO:journalctl@ceph.mon.a.smithi028.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T07:58:48.982 INFO:journalctl@ceph.mon.a.smithi028.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-04-24T07:58:48.982 INFO:journalctl@ceph.mon.a.smithi028.stdout: L0 2/0 100.62 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 101.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T07:58:48.982 INFO:journalctl@ceph.mon.a.smithi028.stdout: Sum 2/0 100.62 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 101.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T07:58:48.982 INFO:journalctl@ceph.mon.a.smithi028.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 101.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T07:58:48.983 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-24T07:58:48.983 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** Compaction Stats [default] ** 2024-04-24T07:58:48.983 INFO:journalctl@ceph.mon.a.smithi028.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-04-24T07:58:48.983 INFO:journalctl@ceph.mon.a.smithi028.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-04-24T07:58:48.983 INFO:journalctl@ceph.mon.a.smithi028.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 101.9 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-04-24T07:58:48.983 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-24T07:58:48.983 INFO:journalctl@ceph.mon.a.smithi028.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-04-24T07:58:48.983 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-24T07:58:48.983 INFO:journalctl@ceph.mon.a.smithi028.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-04-24T07:58:48.983 INFO:journalctl@ceph.mon.a.smithi028.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-04-24T07:58:48.983 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-04-24T07:58:48.983 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-04-24T07:58:48.983 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-04-24T07:58:48.984 INFO:journalctl@ceph.mon.a.smithi028.stdout: AddFile(Keys): cumulative 0, interval 0 2024-04-24T07:58:48.984 INFO:journalctl@ceph.mon.a.smithi028.stdout: Cumulative compaction: 0.00 GB write, 14.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T07:58:48.984 INFO:journalctl@ceph.mon.a.smithi028.stdout: Interval compaction: 0.00 GB write, 14.23 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-04-24T07:58:48.984 INFO:journalctl@ceph.mon.a.smithi028.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-04-24T07:58:48.984 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache BinnedLRUCache@0x55c7fe9bb090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.1e-05 secs_since: 0 2024-04-24T07:58:48.984 INFO:journalctl@ceph.mon.a.smithi028.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-04-24T07:58:48.984 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-24T07:58:48.984 INFO:journalctl@ceph.mon.a.smithi028.stdout: ** File Read Latency Histogram By Level [default] ** 2024-04-24T07:58:48.984 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: starting mon.a rank 0 at public addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] at bind addrs [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T07:58:48.984 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: mon.a@-1(???) e1 preinit fsid 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T07:58:48.984 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: mon.a@-1(???).mds e1 new map 2024-04-24T07:58:48.984 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: mon.a@-1(???).mds e1 print_map 2024-04-24T07:58:48.984 INFO:journalctl@ceph.mon.a.smithi028.stdout: e1 2024-04-24T07:58:48.984 INFO:journalctl@ceph.mon.a.smithi028.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-04-24T07:58:48.985 INFO:journalctl@ceph.mon.a.smithi028.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-24T07:58:48.985 INFO:journalctl@ceph.mon.a.smithi028.stdout: legacy client fscid: -1 2024-04-24T07:58:48.985 INFO:journalctl@ceph.mon.a.smithi028.stdout: 2024-04-24T07:58:48.985 INFO:journalctl@ceph.mon.a.smithi028.stdout: No filesystems configured 2024-04-24T07:58:48.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-04-24T07:58:48.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T07:58:48.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T07:58:48.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-04-24T07:58:48.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-04-24T07:58:48.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 bash[27767]: 29ad43385eba63f095616e23346567f09deee4aba577f58b5fe3980f1d535002 2024-04-24T07:58:48.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: mon.a is new leader, mons a in quorum (ranks 0) 2024-04-24T07:58:48.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: monmap epoch 1 2024-04-24T07:58:48.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: fsid 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T07:58:48.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: last_changed 2024-04-24T07:58:38.163806+0000 2024-04-24T07:58:48.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: created 2024-04-24T07:58:38.163806+0000 2024-04-24T07:58:48.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: min_mon_release 19 (squid) 2024-04-24T07:58:48.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: election_strategy: 1 2024-04-24T07:58:48.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-24T07:58:48.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: fsmap 2024-04-24T07:58:48.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: osdmap e1: 0 total, 0 up, 0 in 2024-04-24T07:58:48.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 ceph-mon[27786]: mgrmap e1: no daemons active 2024-04-24T07:58:48.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:48 smithi028 systemd[1]: Started Ceph mon.a for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T07:58:50.246 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.501+0000 7f5445400700 1 Processor -- start 2024-04-24T07:58:50.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.502+0000 7f5445400700 1 -- start start 2024-04-24T07:58:50.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.502+0000 7f5445400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544007de50 0x7f544007c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:58:50.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.502+0000 7f5445400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f544007c840 con 0x7f544007de50 2024-04-24T07:58:50.247 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.503+0000 7f543e800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544007de50 0x7f544007c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:58:50.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.503+0000 7f543e800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544007de50 0x7f544007c300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50462/0 (socket says 172.21.15.28:50462) 2024-04-24T07:58:50.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.503+0000 7f543e800700 1 -- 172.21.15.28:0/3925186508 learned_addr learned my addr 172.21.15.28:0/3925186508 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:58:50.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.503+0000 7f543e800700 1 -- 172.21.15.28:0/3925186508 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f544007c9c0 con 0x7f544007de50 2024-04-24T07:58:50.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.504+0000 7f543e800700 1 --2- 172.21.15.28:0/3925186508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544007de50 0x7f544007c300 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f54300098a0 tx=0x7f5430009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1736f84856f137ec server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:58:50.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.504+0000 7f543d400700 1 -- 172.21.15.28:0/3925186508 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5430004160 con 0x7f544007de50 2024-04-24T07:58:50.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.504+0000 7f543d400700 1 -- 172.21.15.28:0/3925186508 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5430004300 con 0x7f544007de50 2024-04-24T07:58:50.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.504+0000 7f543d400700 1 -- 172.21.15.28:0/3925186508 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5430011b50 con 0x7f544007de50 2024-04-24T07:58:50.248 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.505+0000 7f5445400700 1 -- 172.21.15.28:0/3925186508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544007de50 msgr2=0x7f544007c300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:58:50.250 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.505+0000 7f5445400700 1 --2- 172.21.15.28:0/3925186508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544007de50 0x7f544007c300 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f54300098a0 tx=0x7f5430009bb0 comp rx=0 tx=0).stop 2024-04-24T07:58:50.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.505+0000 7f5445400700 1 -- 172.21.15.28:0/3925186508 shutdown_connections 2024-04-24T07:58:50.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.505+0000 7f5445400700 1 --2- 172.21.15.28:0/3925186508 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544007de50 0x7f544007c300 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:58:50.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.505+0000 7f5445400700 1 -- 172.21.15.28:0/3925186508 >> 172.21.15.28:0/3925186508 conn(0x7f5440103d70 msgr2=0x7f5440106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:58:50.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.505+0000 7f5445400700 1 -- 172.21.15.28:0/3925186508 shutdown_connections 2024-04-24T07:58:50.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.505+0000 7f5445400700 1 -- 172.21.15.28:0/3925186508 wait complete. 2024-04-24T07:58:50.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.506+0000 7f5445400700 1 Processor -- start 2024-04-24T07:58:50.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.506+0000 7f5445400700 1 -- start start 2024-04-24T07:58:50.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.507+0000 7f5445400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544007de50 0x7f544019e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:58:50.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.507+0000 7f5445400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f544010ed00 con 0x7f544007de50 2024-04-24T07:58:50.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.507+0000 7f543e800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544007de50 0x7f544019e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:58:50.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.507+0000 7f543e800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544007de50 0x7f544019e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50466/0 (socket says 172.21.15.28:50466) 2024-04-24T07:58:50.251 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.507+0000 7f543e800700 1 -- 172.21.15.28:0/1543734806 learned_addr learned my addr 172.21.15.28:0/1543734806 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:58:50.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.507+0000 7f543e800700 1 -- 172.21.15.28:0/1543734806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f544019ee20 con 0x7f544007de50 2024-04-24T07:58:50.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.508+0000 7f543e800700 1 --2- 172.21.15.28:0/1543734806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544007de50 0x7f544019e8e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5430011b60 tx=0x7f5430011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:58:50.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.508+0000 7f5437400700 1 -- 172.21.15.28:0/1543734806 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5430003970 con 0x7f544007de50 2024-04-24T07:58:50.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.508+0000 7f5445400700 1 -- 172.21.15.28:0/1543734806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f544019f0b0 con 0x7f544007de50 2024-04-24T07:58:50.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.508+0000 7f5437400700 1 -- 172.21.15.28:0/1543734806 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f5430003b10 con 0x7f544007de50 2024-04-24T07:58:50.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.508+0000 7f5437400700 1 -- 172.21.15.28:0/1543734806 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5430024610 con 0x7f544007de50 2024-04-24T07:58:50.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.508+0000 7f5445400700 1 -- 172.21.15.28:0/1543734806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54401a1da0 con 0x7f544007de50 2024-04-24T07:58:50.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.509+0000 7f5437400700 1 -- 172.21.15.28:0/1543734806 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f5430024ca0 con 0x7f544007de50 2024-04-24T07:58:50.253 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.509+0000 7f5437400700 1 -- 172.21.15.28:0/1543734806 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f5430011e00 con 0x7f544007de50 2024-04-24T07:58:50.254 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.509+0000 7f5445400700 1 -- 172.21.15.28:0/1543734806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5440064150 con 0x7f544007de50 2024-04-24T07:58:50.255 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.513+0000 7f5437400700 1 -- 172.21.15.28:0/1543734806 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f543001a9f0 con 0x7f544007de50 2024-04-24T07:58:50.255 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.591+0000 7f5445400700 1 -- 172.21.15.28:0/1543734806 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7f54401a1f70 con 0x7f544007de50 2024-04-24T07:58:50.255 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.595+0000 7f5437400700 1 -- 172.21.15.28:0/1543734806 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7f543001a430 con 0x7f544007de50 2024-04-24T07:58:50.255 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.596+0000 7f5445400700 1 -- 172.21.15.28:0/1543734806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544007de50 msgr2=0x7f544019e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:58:50.255 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.596+0000 7f5445400700 1 --2- 172.21.15.28:0/1543734806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544007de50 0x7f544019e8e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5430011b60 tx=0x7f5430011c40 comp rx=0 tx=0).stop 2024-04-24T07:58:50.255 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.596+0000 7f5445400700 1 -- 172.21.15.28:0/1543734806 shutdown_connections 2024-04-24T07:58:50.255 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.596+0000 7f5445400700 1 --2- 172.21.15.28:0/1543734806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f544007de50 0x7f544019e8e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:58:50.255 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.596+0000 7f5445400700 1 -- 172.21.15.28:0/1543734806 >> 172.21.15.28:0/1543734806 conn(0x7f5440103d70 msgr2=0x7f54401048c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:58:50.255 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.596+0000 7f5445400700 1 -- 172.21.15.28:0/1543734806 shutdown_connections 2024-04-24T07:58:50.255 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:49.597+0000 7f5445400700 1 -- 172.21.15.28:0/1543734806 wait complete. 2024-04-24T07:58:50.256 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote config to /etc/ceph/ceph.conf 2024-04-24T07:58:50.280 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-04-24T07:58:50.280 INFO:teuthology.orchestra.run.smithi028.stdout:Creating mgr... 2024-04-24T07:58:50.282 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:9283 ... 2024-04-24T07:58:50.283 INFO:teuthology.orchestra.run.smithi028.stdout:Verifying port 0.0.0.0:8765 ... 2024-04-24T07:58:50.734 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-473aaf58-0210-11ef-bc93-c7b262605968@mgr.x 2024-04-24T07:58:50.734 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Failed to reset failed state of unit ceph-473aaf58-0210-11ef-bc93-c7b262605968@mgr.x.service: Unit ceph-473aaf58-0210-11ef-bc93-c7b262605968@mgr.x.service not loaded. 2024-04-24T07:58:51.011 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-473aaf58-0210-11ef-bc93-c7b262605968.target.wants/ceph-473aaf58-0210-11ef-bc93-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-473aaf58-0210-11ef-bc93-c7b262605968@.service. 2024-04-24T07:58:51.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:50 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1543734806' entity='client.admin' 2024-04-24T07:58:51.423 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:51 smithi028 systemd[1]: Starting Ceph mgr.x for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T07:58:51.735 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:51 smithi028 podman[27996]: 2024-04-24 07:58:51.328415438 +0000 UTC m=+0.038551342 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T07:58:51.735 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:51 smithi028 podman[27996]: 2024-04-24 07:58:51.424334102 +0000 UTC m=+0.134470060 container create 8e12d69a1c64607dc54909196c173b811b4ae98663f9b173799c1e4bcb47b688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-24T07:58:51.735 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:51 smithi028 podman[27996]: 2024-04-24 07:58:51.660402819 +0000 UTC m=+0.370538729 container init 8e12d69a1c64607dc54909196c173b811b4ae98663f9b173799c1e4bcb47b688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-04-24T07:58:51.735 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:51 smithi028 podman[27996]: 2024-04-24 07:58:51.671074539 +0000 UTC m=+0.381210448 container start 8e12d69a1c64607dc54909196c173b811b4ae98663f9b173799c1e4bcb47b688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T07:58:51.735 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:51 smithi028 bash[27996]: 8e12d69a1c64607dc54909196c173b811b4ae98663f9b173799c1e4bcb47b688 2024-04-24T07:58:51.772 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-24T07:58:51.772 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-24T07:58:51.793 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-24T07:58:51.793 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-24T07:58:51.793 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-24T07:58:51.793 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to enable service . firewalld.service is not available 2024-04-24T07:58:51.824 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-24T07:58:51.824 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-24T07:58:51.846 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-24T07:58:51.846 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-24T07:58:51.846 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-24T07:58:51.846 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-04-24T07:58:51.846 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr to start... 2024-04-24T07:58:51.846 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr... 2024-04-24T07:58:52.014 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:51 smithi028 systemd[1]: Started Ceph mgr.x for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T07:58:52.014 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:51 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:51.945+0000 7f12a4591200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T07:58:52.289 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:52 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:52.207+0000 7f12a4591200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T07:58:52.672 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:52 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:52.355+0000 7f12a4591200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T07:58:53.019 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:52 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2128267106' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T07:58:53.019 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:52 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:52.800+0000 7f12a4591200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T07:58:53.019 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:52 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:52.915+0000 7f12a4591200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T07:58:53.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-24T07:58:53.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-24T07:58:53.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "473aaf58-0210-11ef-bc93-c7b262605968", 2024-04-24T07:58:53.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-24T07:58:53.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-24T07:58:53.345 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-24T07:58:53.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-24T07:58:53.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:58:53.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-24T07:58:53.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-24T07:58:53.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-24T07:58:53.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-24T07:58:53.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-24T07:58:53.346 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-24T07:58:53.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-24T07:58:53.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-04-24T07:58:53.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-24T07:58:53.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T07:58:53.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-24T07:58:53.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-24T07:58:53.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:58:53.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-24T07:58:53.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T07:58:53.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-24T07:58:53.347 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-24T07:58:53.348 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-24T07:58:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-24T07:58:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-24T07:58:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-24T07:58:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:58:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-24T07:58:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-24T07:58:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-24T07:58:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-24T07:58:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-24T07:58:53.356 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-24T07:58:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-24T07:58:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-24T07:58:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-24T07:58:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:58:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-24T07:58:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T07:58:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-24T07:58:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-24T07:58:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:58:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-24T07:58:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-24T07:58:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-24T07:58:53.357 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-24T07:58:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-24T07:58:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-24T07:58:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-24T07:58:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-24T07:58:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T07:58:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:58:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-24T07:58:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T07:58:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-24T07:58:42.319020+0000", 2024-04-24T07:58:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T07:58:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:58:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-24T07:58:53.358 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-24T07:58:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.579+0000 7f92e3a00700 1 Processor -- start 2024-04-24T07:58:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.579+0000 7f92e3a00700 1 -- start start 2024-04-24T07:58:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.579+0000 7f92e3a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc07de70 0x7f92dc07c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:58:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.579+0000 7f92e3a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc07c8f0 con 0x7f92dc07de70 2024-04-24T07:58:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.580+0000 7f92e1000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc07de70 0x7f92dc07c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:58:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.580+0000 7f92e1000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc07de70 0x7f92dc07c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57406/0 (socket says 172.21.15.28:57406) 2024-04-24T07:58:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.580+0000 7f92e1000700 1 -- 172.21.15.28:0/2668984097 learned_addr learned my addr 172.21.15.28:0/2668984097 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:58:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.581+0000 7f92e1000700 1 -- 172.21.15.28:0/2668984097 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92dc07ca70 con 0x7f92dc07de70 2024-04-24T07:58:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.581+0000 7f92e1000700 1 --2- 172.21.15.28:0/2668984097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc07de70 0x7f92dc07c320 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f92d00098a0 tx=0x7f92d0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32442dc86b238c51 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:58:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.581+0000 7f92db400700 1 -- 172.21.15.28:0/2668984097 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92d0004160 con 0x7f92dc07de70 2024-04-24T07:58:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.581+0000 7f92db400700 1 -- 172.21.15.28:0/2668984097 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f92d0004300 con 0x7f92dc07de70 2024-04-24T07:58:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.581+0000 7f92db400700 1 -- 172.21.15.28:0/2668984097 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92d0011b50 con 0x7f92dc07de70 2024-04-24T07:58:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.582+0000 7f92e3a00700 1 -- 172.21.15.28:0/2668984097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc07de70 msgr2=0x7f92dc07c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:58:53.359 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.582+0000 7f92e3a00700 1 --2- 172.21.15.28:0/2668984097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc07de70 0x7f92dc07c320 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f92d00098a0 tx=0x7f92d0009bb0 comp rx=0 tx=0).stop 2024-04-24T07:58:53.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.582+0000 7f92e3a00700 1 -- 172.21.15.28:0/2668984097 shutdown_connections 2024-04-24T07:58:53.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.582+0000 7f92e3a00700 1 --2- 172.21.15.28:0/2668984097 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc07de70 0x7f92dc07c320 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:58:53.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.582+0000 7f92e3a00700 1 -- 172.21.15.28:0/2668984097 >> 172.21.15.28:0/2668984097 conn(0x7f92dc103d10 msgr2=0x7f92dc106130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:58:53.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.583+0000 7f92e3a00700 1 -- 172.21.15.28:0/2668984097 shutdown_connections 2024-04-24T07:58:53.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.583+0000 7f92e3a00700 1 -- 172.21.15.28:0/2668984097 wait complete. 2024-04-24T07:58:53.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.584+0000 7f92e3a00700 1 Processor -- start 2024-04-24T07:58:53.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.584+0000 7f92e3a00700 1 -- start start 2024-04-24T07:58:53.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.584+0000 7f92e3a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc07de70 0x7f92dc19e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:58:53.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.584+0000 7f92e3a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92dc10eca0 con 0x7f92dc07de70 2024-04-24T07:58:53.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.585+0000 7f92e1000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc07de70 0x7f92dc19e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:58:53.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.585+0000 7f92e1000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc07de70 0x7f92dc19e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57410/0 (socket says 172.21.15.28:57410) 2024-04-24T07:58:53.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.585+0000 7f92e1000700 1 -- 172.21.15.28:0/2128267106 learned_addr learned my addr 172.21.15.28:0/2128267106 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:58:53.360 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.585+0000 7f92e1000700 1 -- 172.21.15.28:0/2128267106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92dc19edf0 con 0x7f92dc07de70 2024-04-24T07:58:53.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.585+0000 7f92e1000700 1 --2- 172.21.15.28:0/2128267106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc07de70 0x7f92dc19e8b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f92d0011b60 tx=0x7f92d0011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:58:53.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.586+0000 7f92d9600700 1 -- 172.21.15.28:0/2128267106 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92d0003970 con 0x7f92dc07de70 2024-04-24T07:58:53.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.586+0000 7f92d9600700 1 -- 172.21.15.28:0/2128267106 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f92d0003b10 con 0x7f92dc07de70 2024-04-24T07:58:53.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.586+0000 7f92d9600700 1 -- 172.21.15.28:0/2128267106 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f92d0024610 con 0x7f92dc07de70 2024-04-24T07:58:53.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.586+0000 7f92e3a00700 1 -- 172.21.15.28:0/2128267106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92dc19f080 con 0x7f92dc07de70 2024-04-24T07:58:53.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.586+0000 7f92e3a00700 1 -- 172.21.15.28:0/2128267106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92dc1a1d70 con 0x7f92dc07de70 2024-04-24T07:58:53.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.587+0000 7f92d9600700 1 -- 172.21.15.28:0/2128267106 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f92d00248c0 con 0x7f92dc07de70 2024-04-24T07:58:53.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.587+0000 7f92d9600700 1 -- 172.21.15.28:0/2128267106 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f92d0011e00 con 0x7f92dc07de70 2024-04-24T07:58:53.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.587+0000 7f92e3a00700 1 -- 172.21.15.28:0/2128267106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f92dc064150 con 0x7f92dc07de70 2024-04-24T07:58:53.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.590+0000 7f92d9600700 1 -- 172.21.15.28:0/2128267106 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f92d001a7c0 con 0x7f92dc07de70 2024-04-24T07:58:53.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.671+0000 7f92e3a00700 1 -- 172.21.15.28:0/2128267106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f92dc1a1f40 con 0x7f92dc07de70 2024-04-24T07:58:53.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.672+0000 7f92d9600700 1 -- 172.21.15.28:0/2128267106 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f92d002dd70 con 0x7f92dc07de70 2024-04-24T07:58:53.361 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.673+0000 7f92e3a00700 1 -- 172.21.15.28:0/2128267106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc07de70 msgr2=0x7f92dc19e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:58:53.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.673+0000 7f92e3a00700 1 --2- 172.21.15.28:0/2128267106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc07de70 0x7f92dc19e8b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f92d0011b60 tx=0x7f92d0011c40 comp rx=0 tx=0).stop 2024-04-24T07:58:53.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.673+0000 7f92e3a00700 1 -- 172.21.15.28:0/2128267106 shutdown_connections 2024-04-24T07:58:53.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.674+0000 7f92e3a00700 1 --2- 172.21.15.28:0/2128267106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92dc07de70 0x7f92dc19e8b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:58:53.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.674+0000 7f92e3a00700 1 -- 172.21.15.28:0/2128267106 >> 172.21.15.28:0/2128267106 conn(0x7f92dc103d10 msgr2=0x7f92dc1048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:58:53.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.674+0000 7f92e3a00700 1 -- 172.21.15.28:0/2128267106 shutdown_connections 2024-04-24T07:58:53.362 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:52.674+0000 7f92e3a00700 1 -- 172.21.15.28:0/2128267106 wait complete. 2024-04-24T07:58:53.362 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (1/15)... 2024-04-24T07:58:55.020 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:54 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:54.565+0000 7f12a4591200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T07:58:55.020 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:54 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:54.800+0000 7f12a4591200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T07:58:55.020 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:55 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:55.019+0000 7f12a4591200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T07:58:55.684 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:55 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:55.419+0000 7f12a4591200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T07:58:55.685 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:55 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:55.552+0000 7f12a4591200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T07:58:55.937 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:55 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:55.815+0000 7f12a4591200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T07:58:56.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:56 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/4238898988' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T07:58:56.289 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:56 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:56.086+0000 7f12a4591200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T07:58:56.735 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-24T07:58:56.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-24T07:58:56.736 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "473aaf58-0210-11ef-bc93-c7b262605968", 2024-04-24T07:58:56.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-24T07:58:56.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-24T07:58:56.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-24T07:58:56.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-24T07:58:56.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:58:56.738 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-24T07:58:56.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-24T07:58:56.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-24T07:58:56.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-24T07:58:56.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-24T07:58:56.739 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-24T07:58:56.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-24T07:58:56.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-04-24T07:58:56.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-24T07:58:56.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T07:58:56.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-24T07:58:56.740 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-24T07:58:56.741 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:58:56.741 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-24T07:58:56.741 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T07:58:56.741 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-24T07:58:56.741 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-24T07:58:56.741 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-24T07:58:56.742 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-24T07:58:56.742 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-24T07:58:56.742 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-24T07:58:56.742 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:58:56.742 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-24T07:58:56.742 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-24T07:58:56.742 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-24T07:58:56.742 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-24T07:58:56.742 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-24T07:58:56.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-24T07:58:56.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-24T07:58:56.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-24T07:58:56.748 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-24T07:58:56.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:58:56.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-24T07:58:56.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T07:58:56.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-24T07:58:56.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-24T07:58:56.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:58:56.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-24T07:58:56.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-24T07:58:56.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-24T07:58:56.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-24T07:58:56.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-24T07:58:56.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-24T07:58:56.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-24T07:58:56.749 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-24T07:58:56.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T07:58:56.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:58:56.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-24T07:58:56.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T07:58:56.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-24T07:58:42.319020+0000", 2024-04-24T07:58:56.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T07:58:56.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:58:56.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-24T07:58:56.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-24T07:58:56.750 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.926+0000 7fa459200700 1 Processor -- start 2024-04-24T07:58:56.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.926+0000 7fa459200700 1 -- start start 2024-04-24T07:58:56.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.926+0000 7fa459200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa45407de30 0x7fa45407c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:58:56.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.926+0000 7fa459200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa45407c820 con 0x7fa45407de30 2024-04-24T07:58:56.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.927+0000 7fa451e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa45407de30 0x7fa45407c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:58:56.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.927+0000 7fa451e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa45407de30 0x7fa45407c2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57424/0 (socket says 172.21.15.28:57424) 2024-04-24T07:58:56.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.927+0000 7fa451e00700 1 -- 172.21.15.28:0/3072225314 learned_addr learned my addr 172.21.15.28:0/3072225314 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:58:56.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.928+0000 7fa451e00700 1 -- 172.21.15.28:0/3072225314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa45407c9a0 con 0x7fa45407de30 2024-04-24T07:58:56.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.928+0000 7fa451e00700 1 --2- 172.21.15.28:0/3072225314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa45407de30 0x7fa45407c2e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa4440098a0 tx=0x7fa444009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c2642965722f397 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:58:56.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.928+0000 7fa450a00700 1 -- 172.21.15.28:0/3072225314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa444004160 con 0x7fa45407de30 2024-04-24T07:58:56.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.928+0000 7fa450a00700 1 -- 172.21.15.28:0/3072225314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa444004300 con 0x7fa45407de30 2024-04-24T07:58:56.751 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.928+0000 7fa450a00700 1 -- 172.21.15.28:0/3072225314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa444011b50 con 0x7fa45407de30 2024-04-24T07:58:56.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.929+0000 7fa459200700 1 -- 172.21.15.28:0/3072225314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa45407de30 msgr2=0x7fa45407c2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:58:56.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.929+0000 7fa459200700 1 --2- 172.21.15.28:0/3072225314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa45407de30 0x7fa45407c2e0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa4440098a0 tx=0x7fa444009bb0 comp rx=0 tx=0).stop 2024-04-24T07:58:56.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.929+0000 7fa459200700 1 -- 172.21.15.28:0/3072225314 shutdown_connections 2024-04-24T07:58:56.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.929+0000 7fa459200700 1 --2- 172.21.15.28:0/3072225314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa45407de30 0x7fa45407c2e0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:58:56.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.929+0000 7fa459200700 1 -- 172.21.15.28:0/3072225314 >> 172.21.15.28:0/3072225314 conn(0x7fa454103d10 msgr2=0x7fa454106130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:58:56.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.929+0000 7fa459200700 1 -- 172.21.15.28:0/3072225314 shutdown_connections 2024-04-24T07:58:56.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.929+0000 7fa459200700 1 -- 172.21.15.28:0/3072225314 wait complete. 2024-04-24T07:58:56.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.930+0000 7fa459200700 1 Processor -- start 2024-04-24T07:58:56.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.931+0000 7fa459200700 1 -- start start 2024-04-24T07:58:56.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.931+0000 7fa459200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa45407de30 0x7fa45419e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:58:56.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.931+0000 7fa459200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa45410ece0 con 0x7fa45407de30 2024-04-24T07:58:56.752 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.931+0000 7fa451e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa45407de30 0x7fa45419e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:58:56.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.931+0000 7fa451e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa45407de30 0x7fa45419e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57426/0 (socket says 172.21.15.28:57426) 2024-04-24T07:58:56.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.931+0000 7fa451e00700 1 -- 172.21.15.28:0/4238898988 learned_addr learned my addr 172.21.15.28:0/4238898988 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:58:56.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.932+0000 7fa451e00700 1 -- 172.21.15.28:0/4238898988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa45419edf0 con 0x7fa45407de30 2024-04-24T07:58:56.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.932+0000 7fa451e00700 1 --2- 172.21.15.28:0/4238898988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa45407de30 0x7fa45419e8b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa444011b60 tx=0x7fa444011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:58:56.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.932+0000 7fa44aa00700 1 -- 172.21.15.28:0/4238898988 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa444003970 con 0x7fa45407de30 2024-04-24T07:58:56.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.932+0000 7fa44aa00700 1 -- 172.21.15.28:0/4238898988 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa444003b10 con 0x7fa45407de30 2024-04-24T07:58:56.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.932+0000 7fa44aa00700 1 -- 172.21.15.28:0/4238898988 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa444024610 con 0x7fa45407de30 2024-04-24T07:58:56.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.932+0000 7fa459200700 1 -- 172.21.15.28:0/4238898988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa45419f080 con 0x7fa45407de30 2024-04-24T07:58:56.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.933+0000 7fa459200700 1 -- 172.21.15.28:0/4238898988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa4541a1d70 con 0x7fa45407de30 2024-04-24T07:58:56.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.933+0000 7fa44aa00700 1 -- 172.21.15.28:0/4238898988 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fa444024c50 con 0x7fa45407de30 2024-04-24T07:58:56.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.933+0000 7fa44aa00700 1 -- 172.21.15.28:0/4238898988 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fa444011e00 con 0x7fa45407de30 2024-04-24T07:58:56.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.934+0000 7fa459200700 1 -- 172.21.15.28:0/4238898988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa454064150 con 0x7fa45407de30 2024-04-24T07:58:56.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:55.936+0000 7fa44aa00700 1 -- 172.21.15.28:0/4238898988 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7fa444043e50 con 0x7fa45407de30 2024-04-24T07:58:56.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:56.019+0000 7fa459200700 1 -- 172.21.15.28:0/4238898988 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fa4541a1f40 con 0x7fa45407de30 2024-04-24T07:58:56.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:56.019+0000 7fa44aa00700 1 -- 172.21.15.28:0/4238898988 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fa444032030 con 0x7fa45407de30 2024-04-24T07:58:56.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:56.021+0000 7fa459200700 1 -- 172.21.15.28:0/4238898988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa45407de30 msgr2=0x7fa45419e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:58:56.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:56.021+0000 7fa459200700 1 --2- 172.21.15.28:0/4238898988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa45407de30 0x7fa45419e8b0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fa444011b60 tx=0x7fa444011c40 comp rx=0 tx=0).stop 2024-04-24T07:58:56.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:56.021+0000 7fa459200700 1 -- 172.21.15.28:0/4238898988 shutdown_connections 2024-04-24T07:58:56.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:56.021+0000 7fa459200700 1 --2- 172.21.15.28:0/4238898988 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa45407de30 0x7fa45419e8b0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:58:56.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:56.021+0000 7fa459200700 1 -- 172.21.15.28:0/4238898988 >> 172.21.15.28:0/4238898988 conn(0x7fa454103d10 msgr2=0x7fa454104890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:58:56.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:56.021+0000 7fa459200700 1 -- 172.21.15.28:0/4238898988 shutdown_connections 2024-04-24T07:58:56.758 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:56.021+0000 7fa459200700 1 -- 172.21.15.28:0/4238898988 wait complete. 2024-04-24T07:58:56.758 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (2/15)... 2024-04-24T07:58:57.038 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:56 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:56.933+0000 7f12a4591200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T07:58:57.789 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:57 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:57.503+0000 7f12a4591200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T07:58:57.789 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:57 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:57.633+0000 7f12a4591200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T07:58:58.288 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:57 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:57.907+0000 7f12a4591200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T07:58:58.906 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:58 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:58.574+0000 7f12a4591200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T07:58:58.906 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:58 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:58.702+0000 7f12a4591200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T07:58:59.174 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:59 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:59.059+0000 7f12a4591200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T07:58:59.427 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:59 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:59.262+0000 7f12a4591200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T07:58:59.427 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:59 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:59.393+0000 7f12a4591200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T07:58:59.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:58:59 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2257300551' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T07:58:59.789 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:59 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:59.560+0000 7f12a4591200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T07:59:00.128 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:58:59 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:58:59.937+0000 7f12a4591200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T07:59:00.144 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-24T07:59:00.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-24T07:59:00.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "473aaf58-0210-11ef-bc93-c7b262605968", 2024-04-24T07:59:00.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-24T07:59:00.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-24T07:59:00.145 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-24T07:59:00.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-24T07:59:00.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:59:00.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-24T07:59:00.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-24T07:59:00.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-24T07:59:00.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-24T07:59:00.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-24T07:59:00.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-24T07:59:00.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-24T07:59:00.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-04-24T07:59:00.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-24T07:59:00.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T07:59:00.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-24T07:59:00.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-24T07:59:00.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:59:00.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-24T07:59:00.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T07:59:00.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-24T07:59:00.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-24T07:59:00.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-24T07:59:00.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-24T07:59:00.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-24T07:59:00.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-24T07:59:00.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:59:00.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-24T07:59:00.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-24T07:59:00.153 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-24T07:59:00.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-24T07:59:00.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-24T07:59:00.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-24T07:59:00.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-24T07:59:00.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-24T07:59:00.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-24T07:59:00.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:59:00.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-24T07:59:00.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T07:59:00.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-24T07:59:00.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-24T07:59:00.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:59:00.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-24T07:59:00.154 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": false, 2024-04-24T07:59:00.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-24T07:59:00.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-24T07:59:00.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-24T07:59:00.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-24T07:59:00.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-24T07:59:00.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-24T07:59:00.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T07:59:00.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:59:00.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-24T07:59:00.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T07:59:00.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-24T07:58:42.319020+0000", 2024-04-24T07:59:00.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T07:59:00.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:59:00.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-24T07:59:00.155 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-24T07:59:00.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.336+0000 7f75d0800700 1 Processor -- start 2024-04-24T07:59:00.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.336+0000 7f75d0800700 1 -- start start 2024-04-24T07:59:00.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.336+0000 7f75d0800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c807de70 0x7f75c807c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:00.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.336+0000 7f75d0800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75c807c8f0 con 0x7f75c807de70 2024-04-24T07:59:00.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.337+0000 7f75cde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c807de70 0x7f75c807c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:00.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.337+0000 7f75cde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c807de70 0x7f75c807c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57428/0 (socket says 172.21.15.28:57428) 2024-04-24T07:59:00.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.337+0000 7f75cde00700 1 -- 172.21.15.28:0/4131368772 learned_addr learned my addr 172.21.15.28:0/4131368772 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:00.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.337+0000 7f75cde00700 1 -- 172.21.15.28:0/4131368772 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c807ca70 con 0x7f75c807de70 2024-04-24T07:59:00.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.338+0000 7f75cde00700 1 --2- 172.21.15.28:0/4131368772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c807de70 0x7f75c807c320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0098a0 tx=0x7f75bc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e286a03d5eed4320 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:00.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.338+0000 7f75cca00700 1 -- 172.21.15.28:0/4131368772 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75bc004160 con 0x7f75c807de70 2024-04-24T07:59:00.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.338+0000 7f75cca00700 1 -- 172.21.15.28:0/4131368772 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f75bc004300 con 0x7f75c807de70 2024-04-24T07:59:00.156 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.338+0000 7f75cca00700 1 -- 172.21.15.28:0/4131368772 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75bc011b50 con 0x7f75c807de70 2024-04-24T07:59:00.163 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.339+0000 7f75d0800700 1 -- 172.21.15.28:0/4131368772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c807de70 msgr2=0x7f75c807c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:00.164 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.339+0000 7f75d0800700 1 --2- 172.21.15.28:0/4131368772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c807de70 0x7f75c807c320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f75bc0098a0 tx=0x7f75bc009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:00.164 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.339+0000 7f75d0800700 1 -- 172.21.15.28:0/4131368772 shutdown_connections 2024-04-24T07:59:00.164 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.339+0000 7f75d0800700 1 --2- 172.21.15.28:0/4131368772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c807de70 0x7f75c807c320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:00.164 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.339+0000 7f75d0800700 1 -- 172.21.15.28:0/4131368772 >> 172.21.15.28:0/4131368772 conn(0x7f75c8103d10 msgr2=0x7f75c8106130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:00.164 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.339+0000 7f75d0800700 1 -- 172.21.15.28:0/4131368772 shutdown_connections 2024-04-24T07:59:00.164 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.339+0000 7f75d0800700 1 -- 172.21.15.28:0/4131368772 wait complete. 2024-04-24T07:59:00.164 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.340+0000 7f75d0800700 1 Processor -- start 2024-04-24T07:59:00.164 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.340+0000 7f75d0800700 1 -- start start 2024-04-24T07:59:00.164 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.341+0000 7f75d0800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c807de70 0x7f75c819e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:00.164 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.341+0000 7f75d0800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f75c810eca0 con 0x7f75c807de70 2024-04-24T07:59:00.164 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.341+0000 7f75cde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c807de70 0x7f75c819e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:00.164 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.341+0000 7f75cde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c807de70 0x7f75c819e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57438/0 (socket says 172.21.15.28:57438) 2024-04-24T07:59:00.165 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.341+0000 7f75cde00700 1 -- 172.21.15.28:0/2257300551 learned_addr learned my addr 172.21.15.28:0/2257300551 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:00.165 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.341+0000 7f75cde00700 1 -- 172.21.15.28:0/2257300551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f75c819edf0 con 0x7f75c807de70 2024-04-24T07:59:00.165 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.342+0000 7f75cde00700 1 --2- 172.21.15.28:0/2257300551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c807de70 0x7f75c819e8b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f75bc011b60 tx=0x7f75bc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:00.165 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.342+0000 7f75c2a00700 1 -- 172.21.15.28:0/2257300551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75bc003970 con 0x7f75c807de70 2024-04-24T07:59:00.165 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.342+0000 7f75c2a00700 1 -- 172.21.15.28:0/2257300551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f75bc003b10 con 0x7f75c807de70 2024-04-24T07:59:00.165 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.342+0000 7f75d0800700 1 -- 172.21.15.28:0/2257300551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f75c819f080 con 0x7f75c807de70 2024-04-24T07:59:00.165 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.342+0000 7f75c2a00700 1 -- 172.21.15.28:0/2257300551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f75bc024610 con 0x7f75c807de70 2024-04-24T07:59:00.165 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.342+0000 7f75d0800700 1 -- 172.21.15.28:0/2257300551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f75c81a1d70 con 0x7f75c807de70 2024-04-24T07:59:00.165 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.342+0000 7f75c2a00700 1 -- 172.21.15.28:0/2257300551 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f75bc024c50 con 0x7f75c807de70 2024-04-24T07:59:00.165 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.342+0000 7f75c2a00700 1 -- 172.21.15.28:0/2257300551 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f75bc011e00 con 0x7f75c807de70 2024-04-24T07:59:00.165 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.343+0000 7f75d0800700 1 -- 172.21.15.28:0/2257300551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f75c8064150 con 0x7f75c807de70 2024-04-24T07:59:00.165 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.346+0000 7f75c2a00700 1 -- 172.21.15.28:0/2257300551 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73265 (secure 0 0 0) 0x7f75bc01a9f0 con 0x7f75c807de70 2024-04-24T07:59:00.166 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.426+0000 7f75d0800700 1 -- 172.21.15.28:0/2257300551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f75c81a1f40 con 0x7f75c807de70 2024-04-24T07:59:00.166 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.428+0000 7f75c2a00700 1 -- 172.21.15.28:0/2257300551 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1240 (secure 0 0 0) 0x7f75bc01a430 con 0x7f75c807de70 2024-04-24T07:59:00.166 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.429+0000 7f75d0800700 1 -- 172.21.15.28:0/2257300551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c807de70 msgr2=0x7f75c819e8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:00.166 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.429+0000 7f75d0800700 1 --2- 172.21.15.28:0/2257300551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c807de70 0x7f75c819e8b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f75bc011b60 tx=0x7f75bc011c40 comp rx=0 tx=0).stop 2024-04-24T07:59:00.166 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.429+0000 7f75d0800700 1 -- 172.21.15.28:0/2257300551 shutdown_connections 2024-04-24T07:59:00.166 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.429+0000 7f75d0800700 1 --2- 172.21.15.28:0/2257300551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f75c807de70 0x7f75c819e8b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:00.166 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.429+0000 7f75d0800700 1 -- 172.21.15.28:0/2257300551 >> 172.21.15.28:0/2257300551 conn(0x7f75c8103d10 msgr2=0x7f75c81048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:00.166 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.429+0000 7f75d0800700 1 -- 172.21.15.28:0/2257300551 shutdown_connections 2024-04-24T07:59:00.166 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:58:59.429+0000 7f75d0800700 1 -- 172.21.15.28:0/2257300551 wait complete. 2024-04-24T07:59:00.166 INFO:teuthology.orchestra.run.smithi028.stdout:mgr not available, waiting (3/15)... 2024-04-24T07:59:01.028 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:00 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:00.733+0000 7f12a4591200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T07:59:01.289 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:01 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:01.027+0000 7f12a4591200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T07:59:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:01 smithi028 ceph-mon[27786]: Activating manager daemon x 2024-04-24T07:59:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:01 smithi028 ceph-mon[27786]: mgrmap e2: x(active, starting, since 0.0117572s) 2024-04-24T07:59:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:01 smithi028 ceph-mon[27786]: from='mgr.14100 172.21.15.28:0/3100571876' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T07:59:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:01 smithi028 ceph-mon[27786]: from='mgr.14100 172.21.15.28:0/3100571876' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T07:59:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:01 smithi028 ceph-mon[27786]: from='mgr.14100 172.21.15.28:0/3100571876' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T07:59:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:01 smithi028 ceph-mon[27786]: from='mgr.14100 172.21.15.28:0/3100571876' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T07:59:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:01 smithi028 ceph-mon[27786]: from='mgr.14100 172.21.15.28:0/3100571876' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T07:59:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:01 smithi028 ceph-mon[27786]: Manager daemon x is now available 2024-04-24T07:59:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:01 smithi028 ceph-mon[27786]: from='mgr.14100 172.21.15.28:0/3100571876' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T07:59:01.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:01 smithi028 ceph-mon[27786]: from='mgr.14100 172.21.15.28:0/3100571876' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T07:59:01.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:01 smithi028 ceph-mon[27786]: from='mgr.14100 172.21.15.28:0/3100571876' entity='mgr.x' 2024-04-24T07:59:01.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:01 smithi028 ceph-mon[27786]: from='mgr.14100 172.21.15.28:0/3100571876' entity='mgr.x' 2024-04-24T07:59:02.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:02 smithi028 ceph-mon[27786]: from='mgr.14100 172.21.15.28:0/3100571876' entity='mgr.x' 2024-04-24T07:59:02.481 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:02 smithi028 ceph-mon[27786]: mgrmap e3: x(active, since 1.02095s) 2024-04-24T07:59:03.728 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-24T07:59:03.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-24T07:59:03.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsid": "473aaf58-0210-11ef-bc93-c7b262605968", 2024-04-24T07:59:03.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "health": { 2024-04-24T07:59:03.731 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-04-24T07:59:03.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-04-24T07:59:03.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-04-24T07:59:03.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:59:03.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-04-24T07:59:03.732 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-04-24T07:59:03.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 0 2024-04-24T07:59:03.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-24T07:59:03.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-04-24T07:59:03.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "a" 2024-04-24T07:59:03.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-24T07:59:03.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-04-24T07:59:03.734 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "monmap": { 2024-04-24T07:59:03.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T07:59:03.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-04-24T07:59:03.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-04-24T07:59:03.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:59:03.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-04-24T07:59:03.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T07:59:03.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-04-24T07:59:03.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-04-24T07:59:03.737 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-04-24T07:59:03.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-04-24T07:59:03.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-04-24T07:59:03.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-04-24T07:59:03.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:59:03.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-04-24T07:59:03.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-04-24T07:59:03.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-04-24T07:59:03.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-04-24T07:59:03.743 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-04-24T07:59:03.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-04-24T07:59:03.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-04-24T07:59:03.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-04-24T07:59:03.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-04-24T07:59:03.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:59:03.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-04-24T07:59:03.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T07:59:03.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-04-24T07:59:03.744 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-04-24T07:59:03.745 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:59:03.745 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-04-24T07:59:03.745 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-24T07:59:03.745 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-04-24T07:59:03.745 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modules": [ 2024-04-24T07:59:03.745 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "iostat", 2024-04-24T07:59:03.745 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "nfs", 2024-04-24T07:59:03.745 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "restful" 2024-04-24T07:59:03.745 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ], 2024-04-24T07:59:03.745 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T07:59:03.745 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:59:03.745 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-04-24T07:59:03.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-04-24T07:59:03.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "modified": "2024-04-24T07:58:42.319020+0000", 2024-04-24T07:59:03.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "services": {} 2024-04-24T07:59:03.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout }, 2024-04-24T07:59:03.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-04-24T07:59:03.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-24T07:59:03.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.789+0000 7f4384a00700 1 Processor -- start 2024-04-24T07:59:03.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.789+0000 7f4384a00700 1 -- start start 2024-04-24T07:59:03.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.790+0000 7f4384a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438010efe0 0x7f438010f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:03.746 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.790+0000 7f4384a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f438010f9b0 con 0x7f438010efe0 2024-04-24T07:59:03.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.790+0000 7f437f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438010efe0 0x7f438010f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:03.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.790+0000 7f437f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438010efe0 0x7f438010f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35862/0 (socket says 172.21.15.28:35862) 2024-04-24T07:59:03.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.790+0000 7f437f400700 1 -- 172.21.15.28:0/1437698004 learned_addr learned my addr 172.21.15.28:0/1437698004 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:03.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.791+0000 7f437f400700 1 -- 172.21.15.28:0/1437698004 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f438010fb30 con 0x7f438010efe0 2024-04-24T07:59:03.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.791+0000 7f437f400700 1 --2- 172.21.15.28:0/1437698004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438010efe0 0x7f438010f3e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f43700098a0 tx=0x7f4370009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c4e5b81277acb28 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:03.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.791+0000 7f437e000700 1 -- 172.21.15.28:0/1437698004 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4370004160 con 0x7f438010efe0 2024-04-24T07:59:03.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.791+0000 7f437e000700 1 -- 172.21.15.28:0/1437698004 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4370004300 con 0x7f438010efe0 2024-04-24T07:59:03.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.791+0000 7f437e000700 1 -- 172.21.15.28:0/1437698004 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4370011b50 con 0x7f438010efe0 2024-04-24T07:59:03.747 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.792+0000 7f4384a00700 1 -- 172.21.15.28:0/1437698004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438010efe0 msgr2=0x7f438010f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:03.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.792+0000 7f4384a00700 1 --2- 172.21.15.28:0/1437698004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438010efe0 0x7f438010f3e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f43700098a0 tx=0x7f4370009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:03.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.792+0000 7f4384a00700 1 -- 172.21.15.28:0/1437698004 shutdown_connections 2024-04-24T07:59:03.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.792+0000 7f4384a00700 1 --2- 172.21.15.28:0/1437698004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438010efe0 0x7f438010f3e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:03.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.792+0000 7f4384a00700 1 -- 172.21.15.28:0/1437698004 >> 172.21.15.28:0/1437698004 conn(0x7f438007ddf0 msgr2=0x7f4380080210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:03.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.792+0000 7f4384a00700 1 -- 172.21.15.28:0/1437698004 shutdown_connections 2024-04-24T07:59:03.753 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.792+0000 7f4384a00700 1 -- 172.21.15.28:0/1437698004 wait complete. 2024-04-24T07:59:03.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.793+0000 7f4384a00700 1 Processor -- start 2024-04-24T07:59:03.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.793+0000 7f4384a00700 1 -- start start 2024-04-24T07:59:03.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.794+0000 7f4384a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438010efe0 0x7f43801a2a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:03.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.794+0000 7f4384a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4380112fd0 con 0x7f438010efe0 2024-04-24T07:59:03.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.794+0000 7f437f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438010efe0 0x7f43801a2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:03.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.794+0000 7f437f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438010efe0 0x7f43801a2a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35876/0 (socket says 172.21.15.28:35876) 2024-04-24T07:59:03.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.794+0000 7f437f400700 1 -- 172.21.15.28:0/568568208 learned_addr learned my addr 172.21.15.28:0/568568208 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:03.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.794+0000 7f437f400700 1 -- 172.21.15.28:0/568568208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43801a2fd0 con 0x7f438010efe0 2024-04-24T07:59:03.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.795+0000 7f437f400700 1 --2- 172.21.15.28:0/568568208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438010efe0 0x7f43801a2a90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4370011b60 tx=0x7f4370011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:03.754 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.795+0000 7f4377e00700 1 -- 172.21.15.28:0/568568208 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4370003970 con 0x7f438010efe0 2024-04-24T07:59:03.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.795+0000 7f4377e00700 1 -- 172.21.15.28:0/568568208 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4370003b10 con 0x7f438010efe0 2024-04-24T07:59:03.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.795+0000 7f4377e00700 1 -- 172.21.15.28:0/568568208 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4370024610 con 0x7f438010efe0 2024-04-24T07:59:03.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.795+0000 7f4384a00700 1 -- 172.21.15.28:0/568568208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43801a3260 con 0x7f438010efe0 2024-04-24T07:59:03.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.795+0000 7f4384a00700 1 -- 172.21.15.28:0/568568208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43801a5f50 con 0x7f438010efe0 2024-04-24T07:59:03.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.797+0000 7f4377e00700 1 -- 172.21.15.28:0/568568208 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 3) ==== 46441+0+0 (secure 0 0 0) 0x7f437001a9f0 con 0x7f438010efe0 2024-04-24T07:59:03.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.797+0000 7f4384a00700 1 -- 172.21.15.28:0/568568208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4380113150 con 0x7f438010efe0 2024-04-24T07:59:03.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.797+0000 7f4377e00700 1 --2- 172.21.15.28:0/568568208 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f4364039490 0x7f436403b950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:03.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.797+0000 7f4377e00700 1 -- 172.21.15.28:0/568568208 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f4370011e00 con 0x7f438010efe0 2024-04-24T07:59:03.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.797+0000 7f437ea00700 1 --2- 172.21.15.28:0/568568208 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f4364039490 0x7f436403b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:03.755 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.798+0000 7f437ea00700 1 --2- 172.21.15.28:0/568568208 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f4364039490 0x7f436403b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f436c006fd0 tx=0x7f436c006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:03.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:02.806+0000 7f4377e00700 1 -- 172.21.15.28:0/568568208 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4370016260 con 0x7f438010efe0 2024-04-24T07:59:03.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:03.093+0000 7f4377e00700 1 -- 172.21.15.28:0/568568208 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f4370024c40 con 0x7f438010efe0 2024-04-24T07:59:03.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:03.126+0000 7f4384a00700 1 -- 172.21.15.28:0/568568208 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f43801a6500 con 0x7f438010efe0 2024-04-24T07:59:03.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:03.127+0000 7f4377e00700 1 -- 172.21.15.28:0/568568208 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f437002c880 con 0x7f438010efe0 2024-04-24T07:59:03.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:03.129+0000 7f4384a00700 1 -- 172.21.15.28:0/568568208 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f4364039490 msgr2=0x7f436403b950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:03.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:03.129+0000 7f4384a00700 1 --2- 172.21.15.28:0/568568208 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f4364039490 0x7f436403b950 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f436c006fd0 tx=0x7f436c006e40 comp rx=0 tx=0).stop 2024-04-24T07:59:03.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:03.129+0000 7f4384a00700 1 -- 172.21.15.28:0/568568208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438010efe0 msgr2=0x7f43801a2a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:03.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:03.129+0000 7f4384a00700 1 --2- 172.21.15.28:0/568568208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438010efe0 0x7f43801a2a90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4370011b60 tx=0x7f4370011c40 comp rx=0 tx=0).stop 2024-04-24T07:59:03.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:03.130+0000 7f4384a00700 1 -- 172.21.15.28:0/568568208 shutdown_connections 2024-04-24T07:59:03.756 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:03.130+0000 7f4384a00700 1 --2- 172.21.15.28:0/568568208 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f4364039490 0x7f436403b950 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:03.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:03.130+0000 7f4384a00700 1 --2- 172.21.15.28:0/568568208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f438010efe0 0x7f43801a2a90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:03.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:03.130+0000 7f4384a00700 1 -- 172.21.15.28:0/568568208 >> 172.21.15.28:0/568568208 conn(0x7f438007ddf0 msgr2=0x7f438007f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:03.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:03.130+0000 7f4384a00700 1 -- 172.21.15.28:0/568568208 shutdown_connections 2024-04-24T07:59:03.757 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:03.130+0000 7f4384a00700 1 -- 172.21.15.28:0/568568208 wait complete. 2024-04-24T07:59:03.757 INFO:teuthology.orchestra.run.smithi028.stdout:mgr is available 2024-04-24T07:59:04.361 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:04 smithi028 ceph-mon[27786]: mgrmap e4: x(active, since 2s) 2024-04-24T07:59:04.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:04 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/568568208' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-04-24T07:59:05.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:05 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/4097580113' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-04-24T07:59:05.312 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-24T07:59:05.312 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [global] 2024-04-24T07:59:05.313 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout fsid = 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T07:59:05.313 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-04-24T07:59:05.313 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-04-24T07:59:05.313 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-04-24T07:59:05.313 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-04-24T07:59:05.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-24T07:59:05.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [mgr] 2024-04-24T07:59:05.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-04-24T07:59:05.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 2024-04-24T07:59:05.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout [osd] 2024-04-24T07:59:05.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout filestore_flush_min = 0 2024-04-24T07:59:05.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-04-24T07:59:05.317 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-04-24T07:59:05.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-04-24T07:59:05.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-04-24T07:59:05.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-04-24T07:59:05.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.360+0000 7f110de00700 1 Processor -- start 2024-04-24T07:59:05.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.361+0000 7f110de00700 1 -- start start 2024-04-24T07:59:05.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.361+0000 7f110de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1108107440 0x7f1108107860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:05.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.361+0000 7f110de00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1108107e30 con 0x7f1108107440 2024-04-24T07:59:05.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.362+0000 7f110ca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1108107440 0x7f1108107860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:05.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.362+0000 7f110ca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1108107440 0x7f1108107860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35882/0 (socket says 172.21.15.28:35882) 2024-04-24T07:59:05.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.362+0000 7f110ca00700 1 -- 172.21.15.28:0/3208760375 learned_addr learned my addr 172.21.15.28:0/3208760375 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:05.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.362+0000 7f110ca00700 1 -- 172.21.15.28:0/3208760375 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1108108640 con 0x7f1108107440 2024-04-24T07:59:05.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.362+0000 7f110ca00700 1 --2- 172.21.15.28:0/3208760375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1108107440 0x7f1108107860 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0098a0 tx=0x7f10fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72a03b5520af6364 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:05.318 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.363+0000 7f1107400700 1 -- 172.21.15.28:0/3208760375 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10fc004160 con 0x7f1108107440 2024-04-24T07:59:05.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.363+0000 7f1107400700 1 -- 172.21.15.28:0/3208760375 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f10fc004300 con 0x7f1108107440 2024-04-24T07:59:05.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.363+0000 7f1107400700 1 -- 172.21.15.28:0/3208760375 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10fc011b50 con 0x7f1108107440 2024-04-24T07:59:05.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.363+0000 7f110de00700 1 -- 172.21.15.28:0/3208760375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1108107440 msgr2=0x7f1108107860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:05.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.363+0000 7f110de00700 1 --2- 172.21.15.28:0/3208760375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1108107440 0x7f1108107860 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f10fc0098a0 tx=0x7f10fc009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:05.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.364+0000 7f110de00700 1 -- 172.21.15.28:0/3208760375 shutdown_connections 2024-04-24T07:59:05.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.364+0000 7f110de00700 1 --2- 172.21.15.28:0/3208760375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1108107440 0x7f1108107860 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:05.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.364+0000 7f110de00700 1 -- 172.21.15.28:0/3208760375 >> 172.21.15.28:0/3208760375 conn(0x7f1108102a60 msgr2=0x7f1108104e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:05.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.364+0000 7f110de00700 1 -- 172.21.15.28:0/3208760375 shutdown_connections 2024-04-24T07:59:05.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.364+0000 7f110de00700 1 -- 172.21.15.28:0/3208760375 wait complete. 2024-04-24T07:59:05.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.365+0000 7f110de00700 1 Processor -- start 2024-04-24T07:59:05.319 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.365+0000 7f110de00700 1 -- start start 2024-04-24T07:59:05.320 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.366+0000 7f110de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1108107440 0x7f11081925a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:05.320 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.366+0000 7f110de00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f110810b9f0 con 0x7f1108107440 2024-04-24T07:59:05.320 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.366+0000 7f110ca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1108107440 0x7f11081925a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:05.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.366+0000 7f110ca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1108107440 0x7f11081925a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35886/0 (socket says 172.21.15.28:35886) 2024-04-24T07:59:05.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.366+0000 7f110ca00700 1 -- 172.21.15.28:0/4097580113 learned_addr learned my addr 172.21.15.28:0/4097580113 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:05.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.366+0000 7f110ca00700 1 -- 172.21.15.28:0/4097580113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1108192ae0 con 0x7f1108107440 2024-04-24T07:59:05.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.367+0000 7f110ca00700 1 --2- 172.21.15.28:0/4097580113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1108107440 0x7f11081925a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f10fc011850 tx=0x7f10fc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:05.323 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.367+0000 7f1105600700 1 -- 172.21.15.28:0/4097580113 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10fc003af0 con 0x7f1108107440 2024-04-24T07:59:05.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.367+0000 7f1105600700 1 -- 172.21.15.28:0/4097580113 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f10fc020070 con 0x7f1108107440 2024-04-24T07:59:05.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.367+0000 7f110de00700 1 -- 172.21.15.28:0/4097580113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1108190d20 con 0x7f1108107440 2024-04-24T07:59:05.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.367+0000 7f1105600700 1 -- 172.21.15.28:0/4097580113 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f10fc0195a0 con 0x7f1108107440 2024-04-24T07:59:05.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.367+0000 7f110de00700 1 -- 172.21.15.28:0/4097580113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1108191200 con 0x7f1108107440 2024-04-24T07:59:05.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.368+0000 7f1105600700 1 -- 172.21.15.28:0/4097580113 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7f10fc0198a0 con 0x7f1108107440 2024-04-24T07:59:05.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.368+0000 7f110de00700 1 -- 172.21.15.28:0/4097580113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1108064150 con 0x7f1108107440 2024-04-24T07:59:05.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.369+0000 7f1105600700 1 --2- 172.21.15.28:0/4097580113 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f10f0039560 0x7f10f003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:05.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.369+0000 7f1105600700 1 -- 172.21.15.28:0/4097580113 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f10fc011e00 con 0x7f1108107440 2024-04-24T07:59:05.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.369+0000 7f1107e00700 1 --2- 172.21.15.28:0/4097580113 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f10f0039560 0x7f10f003ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:05.324 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.370+0000 7f1107e00700 1 --2- 172.21.15.28:0/4097580113 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f10f0039560 0x7f10f003ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f10f4006fd0 tx=0x7f10f4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:05.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.375+0000 7f1105600700 1 -- 172.21.15.28:0/4097580113 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f10fc00e260 con 0x7f1108107440 2024-04-24T07:59:05.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.594+0000 7f110de00700 1 -- 172.21.15.28:0/4097580113 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f1108196340 con 0x7f1108107440 2024-04-24T07:59:05.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.595+0000 7f1105600700 1 -- 172.21.15.28:0/4097580113 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+398 (secure 0 0 0) 0x7f10fc04d4b0 con 0x7f1108107440 2024-04-24T07:59:05.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.598+0000 7f110de00700 1 -- 172.21.15.28:0/4097580113 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f10f0039560 msgr2=0x7f10f003ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:05.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.598+0000 7f110de00700 1 --2- 172.21.15.28:0/4097580113 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f10f0039560 0x7f10f003ba20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f10f4006fd0 tx=0x7f10f4006e40 comp rx=0 tx=0).stop 2024-04-24T07:59:05.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.598+0000 7f110de00700 1 -- 172.21.15.28:0/4097580113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1108107440 msgr2=0x7f11081925a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:05.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.598+0000 7f110de00700 1 --2- 172.21.15.28:0/4097580113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1108107440 0x7f11081925a0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f10fc011850 tx=0x7f10fc011c10 comp rx=0 tx=0).stop 2024-04-24T07:59:05.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.598+0000 7f110de00700 1 -- 172.21.15.28:0/4097580113 shutdown_connections 2024-04-24T07:59:05.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.598+0000 7f110de00700 1 --2- 172.21.15.28:0/4097580113 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f10f0039560 0x7f10f003ba20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:05.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.598+0000 7f110de00700 1 --2- 172.21.15.28:0/4097580113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1108107440 0x7f11081925a0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:05.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.598+0000 7f110de00700 1 -- 172.21.15.28:0/4097580113 >> 172.21.15.28:0/4097580113 conn(0x7f1108102a60 msgr2=0x7f11081039d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:05.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.598+0000 7f110de00700 1 -- 172.21.15.28:0/4097580113 shutdown_connections 2024-04-24T07:59:05.325 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:04.598+0000 7f110de00700 1 -- 172.21.15.28:0/4097580113 wait complete. 2024-04-24T07:59:05.325 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling cephadm module... 2024-04-24T07:59:06.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:06 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2222107864' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-04-24T07:59:06.510 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:06 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: ignoring --setuser ceph since I am not root 2024-04-24T07:59:06.510 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:06 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: ignoring --setgroup ceph since I am not root 2024-04-24T07:59:06.510 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:06 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:06.508+0000 7fb0c854b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T07:59:06.995 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.862+0000 7fbb13200700 1 Processor -- start 2024-04-24T07:59:06.995 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.862+0000 7fbb13200700 1 -- start start 2024-04-24T07:59:06.995 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.862+0000 7fbb13200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c10a980 0x7fbb0c10ada0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:06.995 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.863+0000 7fbb13200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c10b370 con 0x7fbb0c10a980 2024-04-24T07:59:06.995 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.863+0000 7fbb0be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c10a980 0x7fbb0c10ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:06.995 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.863+0000 7fbb0be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c10a980 0x7fbb0c10ada0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35896/0 (socket says 172.21.15.28:35896) 2024-04-24T07:59:06.995 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.863+0000 7fbb0be00700 1 -- 172.21.15.28:0/341450768 learned_addr learned my addr 172.21.15.28:0/341450768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:06.995 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.863+0000 7fbb0be00700 1 -- 172.21.15.28:0/341450768 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb0c10bb30 con 0x7fbb0c10a980 2024-04-24T07:59:06.995 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.864+0000 7fbb0be00700 1 --2- 172.21.15.28:0/341450768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c10a980 0x7fbb0c10ada0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0098a0 tx=0x7fbafc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d2d90e4c05126ebc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:06.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.864+0000 7fbb0b400700 1 -- 172.21.15.28:0/341450768 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbafc004160 con 0x7fbb0c10a980 2024-04-24T07:59:06.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.864+0000 7fbb0b400700 1 -- 172.21.15.28:0/341450768 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbafc004300 con 0x7fbb0c10a980 2024-04-24T07:59:06.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.864+0000 7fbb0b400700 1 -- 172.21.15.28:0/341450768 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbafc011b50 con 0x7fbb0c10a980 2024-04-24T07:59:06.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.865+0000 7fbb13200700 1 -- 172.21.15.28:0/341450768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c10a980 msgr2=0x7fbb0c10ada0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:06.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.865+0000 7fbb13200700 1 --2- 172.21.15.28:0/341450768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c10a980 0x7fbb0c10ada0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fbafc0098a0 tx=0x7fbafc009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:06.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.866+0000 7fbb13200700 1 -- 172.21.15.28:0/341450768 shutdown_connections 2024-04-24T07:59:06.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.866+0000 7fbb13200700 1 --2- 172.21.15.28:0/341450768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c10a980 0x7fbb0c10ada0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:06.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.866+0000 7fbb13200700 1 -- 172.21.15.28:0/341450768 >> 172.21.15.28:0/341450768 conn(0x7fbb0c07def0 msgr2=0x7fbb0c1084f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:06.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.866+0000 7fbb13200700 1 -- 172.21.15.28:0/341450768 shutdown_connections 2024-04-24T07:59:06.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.866+0000 7fbb13200700 1 -- 172.21.15.28:0/341450768 wait complete. 2024-04-24T07:59:06.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.867+0000 7fbb13200700 1 Processor -- start 2024-04-24T07:59:06.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.867+0000 7fbb13200700 1 -- start start 2024-04-24T07:59:06.996 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.867+0000 7fbb13200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c10a980 0x7fbb0c19e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:06.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.868+0000 7fbb13200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbb0c10eee0 con 0x7fbb0c10a980 2024-04-24T07:59:06.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.868+0000 7fbb0be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c10a980 0x7fbb0c19e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:06.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.868+0000 7fbb0be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c10a980 0x7fbb0c19e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35910/0 (socket says 172.21.15.28:35910) 2024-04-24T07:59:06.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.868+0000 7fbb0be00700 1 -- 172.21.15.28:0/2222107864 learned_addr learned my addr 172.21.15.28:0/2222107864 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:06.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.868+0000 7fbb0be00700 1 -- 172.21.15.28:0/2222107864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb0c19eda0 con 0x7fbb0c10a980 2024-04-24T07:59:06.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.868+0000 7fbb0be00700 1 --2- 172.21.15.28:0/2222107864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c10a980 0x7fbb0c19e860 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbafc000c00 tx=0x7fbafc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:06.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.869+0000 7fbb09600700 1 -- 172.21.15.28:0/2222107864 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbafc003b20 con 0x7fbb0c10a980 2024-04-24T07:59:06.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.869+0000 7fbb09600700 1 -- 172.21.15.28:0/2222107864 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fbafc003cc0 con 0x7fbb0c10a980 2024-04-24T07:59:06.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.869+0000 7fbb13200700 1 -- 172.21.15.28:0/2222107864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb0c19f030 con 0x7fbb0c10a980 2024-04-24T07:59:06.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.869+0000 7fbb09600700 1 -- 172.21.15.28:0/2222107864 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbafc024910 con 0x7fbb0c10a980 2024-04-24T07:59:06.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.869+0000 7fbb13200700 1 -- 172.21.15.28:0/2222107864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb0c1a1d20 con 0x7fbb0c10a980 2024-04-24T07:59:06.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.870+0000 7fbb13200700 1 -- 172.21.15.28:0/2222107864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbae8005360 con 0x7fbb0c10a980 2024-04-24T07:59:06.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.870+0000 7fbb09600700 1 -- 172.21.15.28:0/2222107864 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 4) ==== 46547+0+0 (secure 0 0 0) 0x7fbafc022020 con 0x7fbb0c10a980 2024-04-24T07:59:06.997 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.871+0000 7fbb09600700 1 --2- 172.21.15.28:0/2222107864 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7fbaf00395b0 0x7fbaf003ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:06.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.871+0000 7fbb09600700 1 -- 172.21.15.28:0/2222107864 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fbafc011e00 con 0x7fbb0c10a980 2024-04-24T07:59:06.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.871+0000 7fbb03e00700 1 --2- 172.21.15.28:0/2222107864 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7fbaf00395b0 0x7fbaf003ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:06.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.872+0000 7fbb03e00700 1 --2- 172.21.15.28:0/2222107864 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7fbaf00395b0 0x7fbaf003ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4006fd0 tx=0x7fbaf4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:06.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:05.877+0000 7fbb09600700 1 -- 172.21.15.28:0/2222107864 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbafc00b3e0 con 0x7fbb0c10a980 2024-04-24T07:59:06.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:06.166+0000 7fbb13200700 1 -- 172.21.15.28:0/2222107864 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7fbae8005650 con 0x7fbb0c10a980 2024-04-24T07:59:06.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:06.220+0000 7fbb09600700 1 -- 172.21.15.28:0/2222107864 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7fbafc02cda0 con 0x7fbb0c10a980 2024-04-24T07:59:06.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:06.223+0000 7fbb13200700 1 -- 172.21.15.28:0/2222107864 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7fbaf00395b0 msgr2=0x7fbaf003ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:06.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:06.223+0000 7fbb13200700 1 --2- 172.21.15.28:0/2222107864 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7fbaf00395b0 0x7fbaf003ba70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbaf4006fd0 tx=0x7fbaf4006e40 comp rx=0 tx=0).stop 2024-04-24T07:59:06.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:06.223+0000 7fbb13200700 1 -- 172.21.15.28:0/2222107864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c10a980 msgr2=0x7fbb0c19e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:06.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:06.224+0000 7fbb13200700 1 --2- 172.21.15.28:0/2222107864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c10a980 0x7fbb0c19e860 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbafc000c00 tx=0x7fbafc011c10 comp rx=0 tx=0).stop 2024-04-24T07:59:06.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:06.225+0000 7fbb13200700 1 -- 172.21.15.28:0/2222107864 shutdown_connections 2024-04-24T07:59:06.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:06.225+0000 7fbb13200700 1 --2- 172.21.15.28:0/2222107864 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7fbaf00395b0 0x7fbaf003ba70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:06.998 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:06.225+0000 7fbb13200700 1 --2- 172.21.15.28:0/2222107864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbb0c10a980 0x7fbb0c19e860 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:06.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:06.225+0000 7fbb13200700 1 -- 172.21.15.28:0/2222107864 >> 172.21.15.28:0/2222107864 conn(0x7fbb0c07def0 msgr2=0x7fbb0c106bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:06.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:06.225+0000 7fbb13200700 1 -- 172.21.15.28:0/2222107864 shutdown_connections 2024-04-24T07:59:06.999 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:06.225+0000 7fbb13200700 1 -- 172.21.15.28:0/2222107864 wait complete. 2024-04-24T07:59:07.038 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:06 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:06.760+0000 7fb0c854b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T07:59:07.038 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:06 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:06.895+0000 7fb0c854b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T07:59:07.422 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:07 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2222107864' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-04-24T07:59:07.423 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:07 smithi028 ceph-mon[27786]: mgrmap e5: x(active, since 5s) 2024-04-24T07:59:07.423 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:07 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:07.328+0000 7fb0c854b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T07:59:07.788 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:07 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:07.446+0000 7fb0c854b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T07:59:08.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:08 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/716433962' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-24T07:59:08.590 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-24T07:59:08.590 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-04-24T07:59:08.590 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-24T07:59:08.591 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-24T07:59:08.591 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-24T07:59:08.591 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-24T07:59:08.591 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.579+0000 7ff691a00700 1 Processor -- start 2024-04-24T07:59:08.591 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.579+0000 7ff691a00700 1 -- start start 2024-04-24T07:59:08.591 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.580+0000 7ff691a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1086f0 0x7ff68c108b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:08.591 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.580+0000 7ff691a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff68c1090e0 con 0x7ff68c1086f0 2024-04-24T07:59:08.591 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.580+0000 7ff68aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1086f0 0x7ff68c108b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:08.591 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.580+0000 7ff68aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1086f0 0x7ff68c108b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35942/0 (socket says 172.21.15.28:35942) 2024-04-24T07:59:08.591 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.580+0000 7ff68aa00700 1 -- 172.21.15.28:0/797543656 learned_addr learned my addr 172.21.15.28:0/797543656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:08.591 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.581+0000 7ff68aa00700 1 -- 172.21.15.28:0/797543656 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff68c1098a0 con 0x7ff68c1086f0 2024-04-24T07:59:08.591 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.581+0000 7ff68aa00700 1 --2- 172.21.15.28:0/797543656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1086f0 0x7ff68c108b10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff67c0098a0 tx=0x7ff67c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d0313f2161a5204 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:08.591 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.581+0000 7ff689600700 1 -- 172.21.15.28:0/797543656 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff67c004160 con 0x7ff68c1086f0 2024-04-24T07:59:08.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.581+0000 7ff689600700 1 -- 172.21.15.28:0/797543656 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff67c004300 con 0x7ff68c1086f0 2024-04-24T07:59:08.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.581+0000 7ff689600700 1 -- 172.21.15.28:0/797543656 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff67c011b50 con 0x7ff68c1086f0 2024-04-24T07:59:08.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.582+0000 7ff691a00700 1 -- 172.21.15.28:0/797543656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1086f0 msgr2=0x7ff68c108b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:08.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.582+0000 7ff691a00700 1 --2- 172.21.15.28:0/797543656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1086f0 0x7ff68c108b10 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7ff67c0098a0 tx=0x7ff67c009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:08.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.582+0000 7ff691a00700 1 -- 172.21.15.28:0/797543656 shutdown_connections 2024-04-24T07:59:08.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.582+0000 7ff691a00700 1 --2- 172.21.15.28:0/797543656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1086f0 0x7ff68c108b10 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:08.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.582+0000 7ff691a00700 1 -- 172.21.15.28:0/797543656 >> 172.21.15.28:0/797543656 conn(0x7ff68c103d10 msgr2=0x7ff68c106130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:08.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.582+0000 7ff691a00700 1 -- 172.21.15.28:0/797543656 shutdown_connections 2024-04-24T07:59:08.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.582+0000 7ff691a00700 1 -- 172.21.15.28:0/797543656 wait complete. 2024-04-24T07:59:08.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.583+0000 7ff691a00700 1 Processor -- start 2024-04-24T07:59:08.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.584+0000 7ff691a00700 1 -- start start 2024-04-24T07:59:08.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.584+0000 7ff691a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1086f0 0x7ff68c19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:08.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.584+0000 7ff691a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff68c10cc50 con 0x7ff68c1086f0 2024-04-24T07:59:08.592 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.584+0000 7ff68aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1086f0 0x7ff68c19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:08.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.584+0000 7ff68aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1086f0 0x7ff68c19c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35952/0 (socket says 172.21.15.28:35952) 2024-04-24T07:59:08.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.584+0000 7ff68aa00700 1 -- 172.21.15.28:0/716433962 learned_addr learned my addr 172.21.15.28:0/716433962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:08.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.585+0000 7ff68aa00700 1 -- 172.21.15.28:0/716433962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff68c19cc30 con 0x7ff68c1086f0 2024-04-24T07:59:08.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.585+0000 7ff68aa00700 1 --2- 172.21.15.28:0/716433962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1086f0 0x7ff68c19c6f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009fb0 tx=0x7ff67c011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:08.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.585+0000 7ff683400700 1 -- 172.21.15.28:0/716433962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff67c0038b0 con 0x7ff68c1086f0 2024-04-24T07:59:08.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.585+0000 7ff683400700 1 -- 172.21.15.28:0/716433962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff67c011e50 con 0x7ff68c1086f0 2024-04-24T07:59:08.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.585+0000 7ff691a00700 1 -- 172.21.15.28:0/716433962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff68c19cec0 con 0x7ff68c1086f0 2024-04-24T07:59:08.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.585+0000 7ff683400700 1 -- 172.21.15.28:0/716433962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff67c019550 con 0x7ff68c1086f0 2024-04-24T07:59:08.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.586+0000 7ff691a00700 1 -- 172.21.15.28:0/716433962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff68c19fbb0 con 0x7ff68c1086f0 2024-04-24T07:59:08.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.587+0000 7ff683400700 1 -- 172.21.15.28:0/716433962 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7ff67c003a50 con 0x7ff68c1086f0 2024-04-24T07:59:08.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.587+0000 7ff691a00700 1 -- 172.21.15.28:0/716433962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff68c064150 con 0x7ff68c1086f0 2024-04-24T07:59:08.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.587+0000 7ff683400700 1 --2- 172.21.15.28:0/716433962 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7ff670039560 0x7ff67003ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:08.593 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.587+0000 7ff683400700 1 -- 172.21.15.28:0/716433962 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff67c04d930 con 0x7ff68c1086f0 2024-04-24T07:59:08.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.587+0000 7ff68a000700 1 -- 172.21.15.28:0/716433962 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7ff670039560 msgr2=0x7ff67003ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2371619813 2024-04-24T07:59:08.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.587+0000 7ff68a000700 1 --2- 172.21.15.28:0/716433962 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7ff670039560 0x7ff67003ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T07:59:08.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.597+0000 7ff683400700 1 -- 172.21.15.28:0/716433962 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff67c020240 con 0x7ff68c1086f0 2024-04-24T07:59:08.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.788+0000 7ff68a000700 1 -- 172.21.15.28:0/716433962 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7ff670039560 msgr2=0x7ff67003ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2371619813 2024-04-24T07:59:08.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.788+0000 7ff68a000700 1 --2- 172.21.15.28:0/716433962 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7ff670039560 0x7ff67003ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T07:59:08.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.888+0000 7ff691a00700 1 -- 172.21.15.28:0/716433962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7ff68c1a01f0 con 0x7ff68c1086f0 2024-04-24T07:59:08.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.889+0000 7ff683400700 1 -- 172.21.15.28:0/716433962 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7ff67c04d350 con 0x7ff68c1086f0 2024-04-24T07:59:08.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.891+0000 7ff691a00700 1 -- 172.21.15.28:0/716433962 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7ff670039560 msgr2=0x7ff67003ba20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T07:59:08.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.891+0000 7ff691a00700 1 --2- 172.21.15.28:0/716433962 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7ff670039560 0x7ff67003ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:08.594 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.891+0000 7ff691a00700 1 -- 172.21.15.28:0/716433962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1086f0 msgr2=0x7ff68c19c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:08.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.891+0000 7ff691a00700 1 --2- 172.21.15.28:0/716433962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1086f0 0x7ff68c19c6f0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7ff67c009fb0 tx=0x7ff67c011bf0 comp rx=0 tx=0).stop 2024-04-24T07:59:08.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.892+0000 7ff691a00700 1 -- 172.21.15.28:0/716433962 shutdown_connections 2024-04-24T07:59:08.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.892+0000 7ff691a00700 1 --2- 172.21.15.28:0/716433962 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7ff670039560 0x7ff67003ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:08.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.892+0000 7ff691a00700 1 --2- 172.21.15.28:0/716433962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff68c1086f0 0x7ff68c19c6f0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:08.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.892+0000 7ff691a00700 1 -- 172.21.15.28:0/716433962 >> 172.21.15.28:0/716433962 conn(0x7ff68c103d10 msgr2=0x7ff68c1048e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:08.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.892+0000 7ff691a00700 1 -- 172.21.15.28:0/716433962 shutdown_connections 2024-04-24T07:59:08.595 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:07.892+0000 7ff691a00700 1 -- 172.21.15.28:0/716433962 wait complete. 2024-04-24T07:59:08.595 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-04-24T07:59:08.595 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 5... 2024-04-24T07:59:09.136 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:09 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:09.042+0000 7fb0c854b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T07:59:09.397 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:09 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:09.280+0000 7fb0c854b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T07:59:09.790 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:09 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:09.501+0000 7fb0c854b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T07:59:10.167 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:09 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:09.893+0000 7fb0c854b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T07:59:10.167 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:10 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:10.018+0000 7fb0c854b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T07:59:10.509 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:10 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:10.279+0000 7fb0c854b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T07:59:10.788 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:10 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:10.546+0000 7fb0c854b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T07:59:11.763 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:11 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:11.364+0000 7fb0c854b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T07:59:12.038 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:11 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:11.944+0000 7fb0c854b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T07:59:12.349 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:12 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:12.074+0000 7fb0c854b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T07:59:12.788 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:12 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:12.348+0000 7fb0c854b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T07:59:13.288 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:13 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:13.010+0000 7fb0c854b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T07:59:13.289 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:13 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:13.138+0000 7fb0c854b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T07:59:13.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:13 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:13.466+0000 7fb0c854b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T07:59:13.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:13 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:13.649+0000 7fb0c854b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T07:59:13.780 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:13 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:13.778+0000 7fb0c854b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T07:59:14.038 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:13 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:13.933+0000 7fb0c854b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T07:59:14.788 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:14 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:14.308+0000 7fb0c854b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T07:59:15.379 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:15 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:15.087+0000 7fb0c854b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T07:59:15.788 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:15 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:15.377+0000 7fb0c854b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T07:59:15.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:15 smithi028 ceph-mon[27786]: Active manager daemon x restarted 2024-04-24T07:59:15.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:15 smithi028 ceph-mon[27786]: Activating manager daemon x 2024-04-24T07:59:15.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:15 smithi028 ceph-mon[27786]: osdmap e2: 0 total, 0 up, 0 in 2024-04-24T07:59:15.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:15 smithi028 ceph-mon[27786]: mgrmap e6: x(active, starting, since 0.00460717s) 2024-04-24T07:59:15.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:15 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T07:59:15.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:15 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T07:59:15.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:15 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T07:59:15.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:15 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T07:59:15.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:15 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T07:59:15.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:15 smithi028 ceph-mon[27786]: Manager daemon x is now available 2024-04-24T07:59:16.743 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:16 smithi028 ceph-mon[27786]: Found migration_current of "None". Setting to last migration. 2024-04-24T07:59:16.743 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:16 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' 2024-04-24T07:59:16.743 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:16 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' 2024-04-24T07:59:16.743 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:16 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T07:59:16.743 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:16 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T07:59:16.743 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:16 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T07:59:16.743 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:16 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T07:59:16.743 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:16 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' 2024-04-24T07:59:16.744 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:16 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' 2024-04-24T07:59:16.744 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:16 smithi028 ceph-mon[27786]: mgrmap e7: x(active, since 1.00995s) 2024-04-24T07:59:17.072 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-24T07:59:17.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-04-24T07:59:17.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-24T07:59:17.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-24T07:59:17.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.187+0000 7f22af200700 1 Processor -- start 2024-04-24T07:59:17.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.188+0000 7f22af200700 1 -- start start 2024-04-24T07:59:17.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.188+0000 7f22af200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b01085a0 0x7f22b01089c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:17.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.188+0000 7f22af200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f22b0108f90 con 0x7f22b01085a0 2024-04-24T07:59:17.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.188+0000 7f22ade00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b01085a0 0x7f22b01089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:17.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.189+0000 7f22ade00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b01085a0 0x7f22b01089c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35964/0 (socket says 172.21.15.28:35964) 2024-04-24T07:59:17.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.189+0000 7f22ade00700 1 -- 172.21.15.28:0/3256281882 learned_addr learned my addr 172.21.15.28:0/3256281882 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:17.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.189+0000 7f22ade00700 1 -- 172.21.15.28:0/3256281882 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b0109700 con 0x7f22b01085a0 2024-04-24T07:59:17.073 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.189+0000 7f22ade00700 1 --2- 172.21.15.28:0/3256281882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b01085a0 0x7f22b01089c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f22a00098a0 tx=0x7f22a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e71bee95192cf544 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:17.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.190+0000 7f22aca00700 1 -- 172.21.15.28:0/3256281882 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22a0004160 con 0x7f22b01085a0 2024-04-24T07:59:17.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.190+0000 7f22aca00700 1 -- 172.21.15.28:0/3256281882 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f22a0004300 con 0x7f22b01085a0 2024-04-24T07:59:17.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.190+0000 7f22aca00700 1 -- 172.21.15.28:0/3256281882 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22a0011b50 con 0x7f22b01085a0 2024-04-24T07:59:17.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.190+0000 7f22af200700 1 -- 172.21.15.28:0/3256281882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b01085a0 msgr2=0x7f22b01089c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:17.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.190+0000 7f22af200700 1 --2- 172.21.15.28:0/3256281882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b01085a0 0x7f22b01089c0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f22a00098a0 tx=0x7f22a0009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:17.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.191+0000 7f22af200700 1 -- 172.21.15.28:0/3256281882 shutdown_connections 2024-04-24T07:59:17.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.191+0000 7f22af200700 1 --2- 172.21.15.28:0/3256281882 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b01085a0 0x7f22b01089c0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:17.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.191+0000 7f22af200700 1 -- 172.21.15.28:0/3256281882 >> 172.21.15.28:0/3256281882 conn(0x7f22b0103be0 msgr2=0x7f22b0106000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:17.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.191+0000 7f22af200700 1 -- 172.21.15.28:0/3256281882 shutdown_connections 2024-04-24T07:59:17.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.191+0000 7f22af200700 1 -- 172.21.15.28:0/3256281882 wait complete. 2024-04-24T07:59:17.074 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.192+0000 7f22af200700 1 Processor -- start 2024-04-24T07:59:17.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.192+0000 7f22af200700 1 -- start start 2024-04-24T07:59:17.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.192+0000 7f22af200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b019e910 0x7f22b019ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:17.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.192+0000 7f22af200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f22b010cab0 con 0x7f22b019e910 2024-04-24T07:59:17.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.192+0000 7f22ade00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b019e910 0x7f22b019ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:17.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.193+0000 7f22ade00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b019e910 0x7f22b019ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35966/0 (socket says 172.21.15.28:35966) 2024-04-24T07:59:17.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.193+0000 7f22ade00700 1 -- 172.21.15.28:0/2255729602 learned_addr learned my addr 172.21.15.28:0/2255729602 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:17.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.193+0000 7f22ade00700 1 -- 172.21.15.28:0/2255729602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22b019f270 con 0x7f22b019e910 2024-04-24T07:59:17.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.193+0000 7f22ade00700 1 --2- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b019e910 0x7f22b019ed30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f22a0011850 tx=0x7f22a0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:17.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.193+0000 7f22a6a00700 1 -- 172.21.15.28:0/2255729602 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22a00036a0 con 0x7f22b019e910 2024-04-24T07:59:17.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.194+0000 7f22af200700 1 -- 172.21.15.28:0/2255729602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22b019f500 con 0x7f22b019e910 2024-04-24T07:59:17.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.194+0000 7f22a6a00700 1 -- 172.21.15.28:0/2255729602 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f22a001b440 con 0x7f22b019e910 2024-04-24T07:59:17.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.194+0000 7f22af200700 1 -- 172.21.15.28:0/2255729602 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22b01a1f90 con 0x7f22b019e910 2024-04-24T07:59:17.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.194+0000 7f22a6a00700 1 -- 172.21.15.28:0/2255729602 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f22a0024430 con 0x7f22b019e910 2024-04-24T07:59:17.075 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.195+0000 7f22a6a00700 1 -- 172.21.15.28:0/2255729602 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 5) ==== 46558+0+0 (secure 0 0 0) 0x7f22a001b9f0 con 0x7f22b019e910 2024-04-24T07:59:17.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.195+0000 7f22a6a00700 1 --2- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f2294039560 0x7f229403ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:17.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.195+0000 7f22a6a00700 1 -- 172.21.15.28:0/2255729602 --> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f229403c130 con 0x7f2294039560 2024-04-24T07:59:17.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.196+0000 7f22a6a00700 1 -- 172.21.15.28:0/2255729602 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f22a004de90 con 0x7f22b019e910 2024-04-24T07:59:17.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.196+0000 7f22ad400700 1 -- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f2294039560 msgr2=0x7f229403ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2371619813 2024-04-24T07:59:17.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.196+0000 7f22ad400700 1 --2- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f2294039560 0x7f229403ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T07:59:17.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.396+0000 7f22ad400700 1 -- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f2294039560 msgr2=0x7f229403ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2371619813 2024-04-24T07:59:17.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.396+0000 7f22ad400700 1 --2- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f2294039560 0x7f229403ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T07:59:17.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.797+0000 7f22ad400700 1 -- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f2294039560 msgr2=0x7f229403ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2371619813 2024-04-24T07:59:17.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:09.797+0000 7f22ad400700 1 --2- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f2294039560 0x7f229403ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-24T07:59:17.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:10.598+0000 7f22ad400700 1 -- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f2294039560 msgr2=0x7f229403ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2371619813 2024-04-24T07:59:17.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:10.598+0000 7f22ad400700 1 --2- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f2294039560 0x7f229403ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-24T07:59:17.076 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:12.200+0000 7f22ad400700 1 -- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f2294039560 msgr2=0x7f229403ba20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/2371619813 2024-04-24T07:59:17.077 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:12.200+0000 7f22ad400700 1 --2- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f2294039560 0x7f229403ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-24T07:59:17.077 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:15.385+0000 7f22a6a00700 1 -- 172.21.15.28:0/2255729602 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 6) ==== 46325+0+0 (secure 0 0 0) 0x7f22a002dcd0 con 0x7f22b019e910 2024-04-24T07:59:17.077 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:15.385+0000 7f22a6a00700 1 -- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f2294039560 msgr2=0x7f229403ba20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T07:59:17.077 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:15.385+0000 7f22a6a00700 1 --2- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f2294039560 0x7f229403ba20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:17.077 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.389+0000 7f22a6a00700 1 -- 172.21.15.28:0/2255729602 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 7) ==== 46452+0+0 (secure 0 0 0) 0x7f22a000eca0 con 0x7f22b019e910 2024-04-24T07:59:17.077 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.390+0000 7f22a6a00700 1 --2- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f229403ce50 0x7f229403f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:17.077 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.390+0000 7f22a6a00700 1 -- 172.21.15.28:0/2255729602 --> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f229403c130 con 0x7f229403ce50 2024-04-24T07:59:17.077 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.391+0000 7f22ad400700 1 --2- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f229403ce50 0x7f229403f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:17.077 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.395+0000 7f22ad400700 1 --2- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f229403ce50 0x7f229403f240 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f229c003a10 tx=0x7f229c009290 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:17.077 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.396+0000 7f22a6a00700 1 -- 172.21.15.28:0/2255729602 <== mgr.14120 v2:172.21.15.28:6800/3308597107 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f229403c130 con 0x7f229403ce50 2024-04-24T07:59:17.077 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.404+0000 7f22af200700 1 -- 172.21.15.28:0/2255729602 --> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f22b0064150 con 0x7f229403ce50 2024-04-24T07:59:17.077 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.405+0000 7f22a6a00700 1 -- 172.21.15.28:0/2255729602 <== mgr.14120 v2:172.21.15.28:6800/3308597107 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7f22b0064150 con 0x7f229403ce50 2024-04-24T07:59:17.078 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.405+0000 7f22af200700 1 -- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f229403ce50 msgr2=0x7f229403f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:17.078 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.406+0000 7f22af200700 1 --2- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f229403ce50 0x7f229403f240 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f229c003a10 tx=0x7f229c009290 comp rx=0 tx=0).stop 2024-04-24T07:59:17.078 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.406+0000 7f22af200700 1 -- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b019e910 msgr2=0x7f22b019ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:17.078 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.406+0000 7f22af200700 1 --2- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b019e910 0x7f22b019ed30 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f22a0011850 tx=0x7f22a0011c10 comp rx=0 tx=0).stop 2024-04-24T07:59:17.078 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.406+0000 7f22af200700 1 -- 172.21.15.28:0/2255729602 shutdown_connections 2024-04-24T07:59:17.078 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.406+0000 7f22af200700 1 --2- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f229403ce50 0x7f229403f240 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:17.078 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.406+0000 7f22af200700 1 --2- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f22b019e910 0x7f22b019ed30 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:17.078 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.406+0000 7f22af200700 1 --2- 172.21.15.28:0/2255729602 >> [v2:172.21.15.28:6800/2371619813,v1:172.21.15.28:6801/2371619813] conn(0x7f2294039560 0x7f229403ba20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:17.078 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.406+0000 7f22af200700 1 -- 172.21.15.28:0/2255729602 >> 172.21.15.28:0/2255729602 conn(0x7f22b0103be0 msgr2=0x7f22b01046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:17.078 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.406+0000 7f22af200700 1 -- 172.21.15.28:0/2255729602 shutdown_connections 2024-04-24T07:59:17.078 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:16.406+0000 7f22af200700 1 -- 172.21.15.28:0/2255729602 wait complete. 2024-04-24T07:59:17.078 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 5 is available 2024-04-24T07:59:17.078 INFO:teuthology.orchestra.run.smithi028.stdout:Setting orchestrator backend to cephadm... 2024-04-24T07:59:17.708 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:17 smithi028 ceph-mon[27786]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-24T07:59:17.708 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:17 smithi028 ceph-mon[27786]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-24T07:59:17.708 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:17 smithi028 ceph-mon[27786]: [24/Apr/2024:07:59:17] ENGINE Bus STARTING 2024-04-24T07:59:18.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:18 smithi028 ceph-mon[27786]: [24/Apr/2024:07:59:17] ENGINE Serving on https://172.21.15.28:7150 2024-04-24T07:59:18.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:18 smithi028 ceph-mon[27786]: mgrmap e8: x(active, since 2s) 2024-04-24T07:59:18.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:18 smithi028 ceph-mon[27786]: [24/Apr/2024:07:59:17] ENGINE Serving on http://172.21.15.28:8765 2024-04-24T07:59:18.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:18 smithi028 ceph-mon[27786]: [24/Apr/2024:07:59:17] ENGINE Bus STARTED 2024-04-24T07:59:18.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:18 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T07:59:18.638 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:18 smithi028 ceph-mon[27786]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T07:59:18.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:18 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' 2024-04-24T07:59:18.639 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:18 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T07:59:18.654 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.705+0000 7f795c400700 1 Processor -- start 2024-04-24T07:59:18.655 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.706+0000 7f795c400700 1 -- start start 2024-04-24T07:59:18.655 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.706+0000 7f795c400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795407e880 0x7f795407eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:18.656 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.706+0000 7f795c400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f795407f270 con 0x7f795407e880 2024-04-24T07:59:18.656 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.706+0000 7f7959a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795407e880 0x7f795407eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:18.656 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.707+0000 7f7959a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795407e880 0x7f795407eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45454/0 (socket says 172.21.15.28:45454) 2024-04-24T07:59:18.656 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.707+0000 7f7959a00700 1 -- 172.21.15.28:0/2352407314 learned_addr learned my addr 172.21.15.28:0/2352407314 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:18.656 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.707+0000 7f7959a00700 1 -- 172.21.15.28:0/2352407314 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f795407fa80 con 0x7f795407e880 2024-04-24T07:59:18.656 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.707+0000 7f7959a00700 1 --2- 172.21.15.28:0/2352407314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795407e880 0x7f795407eca0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f79480098a0 tx=0x7f7948009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa48584e5e1d850e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:18.656 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.708+0000 7f7953e00700 1 -- 172.21.15.28:0/2352407314 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7948004160 con 0x7f795407e880 2024-04-24T07:59:18.656 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.708+0000 7f7953e00700 1 -- 172.21.15.28:0/2352407314 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7948004300 con 0x7f795407e880 2024-04-24T07:59:18.656 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.708+0000 7f7953e00700 1 -- 172.21.15.28:0/2352407314 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7948011b50 con 0x7f795407e880 2024-04-24T07:59:18.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.709+0000 7f795c400700 1 -- 172.21.15.28:0/2352407314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795407e880 msgr2=0x7f795407eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:18.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.709+0000 7f795c400700 1 --2- 172.21.15.28:0/2352407314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795407e880 0x7f795407eca0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f79480098a0 tx=0x7f7948009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:18.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.709+0000 7f795c400700 1 -- 172.21.15.28:0/2352407314 shutdown_connections 2024-04-24T07:59:18.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.709+0000 7f795c400700 1 --2- 172.21.15.28:0/2352407314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795407e880 0x7f795407eca0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:18.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.709+0000 7f795c400700 1 -- 172.21.15.28:0/2352407314 >> 172.21.15.28:0/2352407314 conn(0x7f795407def0 msgr2=0x7f79541084f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:18.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.709+0000 7f795c400700 1 -- 172.21.15.28:0/2352407314 shutdown_connections 2024-04-24T07:59:18.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.709+0000 7f795c400700 1 -- 172.21.15.28:0/2352407314 wait complete. 2024-04-24T07:59:18.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.710+0000 7f795c400700 1 Processor -- start 2024-04-24T07:59:18.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.710+0000 7f795c400700 1 -- start start 2024-04-24T07:59:18.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.710+0000 7f795c400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795407e880 0x7f79541a2cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:18.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.710+0000 7f795c400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7954080e40 con 0x7f795407e880 2024-04-24T07:59:18.661 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.711+0000 7f7959a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795407e880 0x7f79541a2cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:18.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.711+0000 7f7959a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795407e880 0x7f79541a2cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45462/0 (socket says 172.21.15.28:45462) 2024-04-24T07:59:18.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.711+0000 7f7959a00700 1 -- 172.21.15.28:0/3338809780 learned_addr learned my addr 172.21.15.28:0/3338809780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:18.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.711+0000 7f7959a00700 1 -- 172.21.15.28:0/3338809780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79541a3200 con 0x7f795407e880 2024-04-24T07:59:18.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.711+0000 7f7959a00700 1 --2- 172.21.15.28:0/3338809780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795407e880 0x7f79541a2cc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7948011b60 tx=0x7f7948011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:18.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.712+0000 7f7952000700 1 -- 172.21.15.28:0/3338809780 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7948003970 con 0x7f795407e880 2024-04-24T07:59:18.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.712+0000 7f7952000700 1 -- 172.21.15.28:0/3338809780 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7948003b10 con 0x7f795407e880 2024-04-24T07:59:18.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.712+0000 7f795c400700 1 -- 172.21.15.28:0/3338809780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79541a3490 con 0x7f795407e880 2024-04-24T07:59:18.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.712+0000 7f7952000700 1 -- 172.21.15.28:0/3338809780 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7948024610 con 0x7f795407e880 2024-04-24T07:59:18.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.712+0000 7f795c400700 1 -- 172.21.15.28:0/3338809780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79541a6180 con 0x7f795407e880 2024-04-24T07:59:18.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.713+0000 7f7952000700 1 -- 172.21.15.28:0/3338809780 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f79480248c0 con 0x7f795407e880 2024-04-24T07:59:18.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.713+0000 7f795c400700 1 -- 172.21.15.28:0/3338809780 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7954064150 con 0x7f795407e880 2024-04-24T07:59:18.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.713+0000 7f7952000700 1 --2- 172.21.15.28:0/3338809780 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f793c039600 0x7f793c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:18.662 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.713+0000 7f7952000700 1 -- 172.21.15.28:0/3338809780 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f7948011e00 con 0x7f795407e880 2024-04-24T07:59:18.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.714+0000 7f7959000700 1 --2- 172.21.15.28:0/3338809780 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f793c039600 0x7f793c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:18.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.714+0000 7f7959000700 1 --2- 172.21.15.28:0/3338809780 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f793c039600 0x7f793c03bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7940006fd0 tx=0x7f7940006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:18.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.720+0000 7f7952000700 1 -- 172.21.15.28:0/3338809780 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f794800b3e0 con 0x7f795407e880 2024-04-24T07:59:18.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.957+0000 7f795c400700 1 -- 172.21.15.28:0/3338809780 --> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7f79540008d0 con 0x7f793c039600 2024-04-24T07:59:18.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.968+0000 7f7952000700 1 -- 172.21.15.28:0/3338809780 <== mgr.14120 v2:172.21.15.28:6800/3308597107 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f79540008d0 con 0x7f793c039600 2024-04-24T07:59:18.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.971+0000 7f795c400700 1 -- 172.21.15.28:0/3338809780 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f793c039600 msgr2=0x7f793c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:18.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.971+0000 7f795c400700 1 --2- 172.21.15.28:0/3338809780 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f793c039600 0x7f793c03bac0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f7940006fd0 tx=0x7f7940006e40 comp rx=0 tx=0).stop 2024-04-24T07:59:18.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.971+0000 7f795c400700 1 -- 172.21.15.28:0/3338809780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795407e880 msgr2=0x7f79541a2cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:18.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.971+0000 7f795c400700 1 --2- 172.21.15.28:0/3338809780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795407e880 0x7f79541a2cc0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f7948011b60 tx=0x7f7948011c40 comp rx=0 tx=0).stop 2024-04-24T07:59:18.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.971+0000 7f795c400700 1 -- 172.21.15.28:0/3338809780 shutdown_connections 2024-04-24T07:59:18.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.971+0000 7f795c400700 1 --2- 172.21.15.28:0/3338809780 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f793c039600 0x7f793c03bac0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:18.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.971+0000 7f795c400700 1 --2- 172.21.15.28:0/3338809780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f795407e880 0x7f79541a2cc0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:18.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.971+0000 7f795c400700 1 -- 172.21.15.28:0/3338809780 >> 172.21.15.28:0/3338809780 conn(0x7f795407def0 msgr2=0x7f79541999e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:18.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.971+0000 7f795c400700 1 -- 172.21.15.28:0/3338809780 shutdown_connections 2024-04-24T07:59:18.663 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:17.971+0000 7f795c400700 1 -- 172.21.15.28:0/3338809780 wait complete. 2024-04-24T07:59:20.220 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout value unchanged 2024-04-24T07:59:20.220 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.284+0000 7fb538a00700 1 Processor -- start 2024-04-24T07:59:20.221 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.284+0000 7fb538a00700 1 -- start start 2024-04-24T07:59:20.221 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.285+0000 7fb538a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb53410efe0 0x7fb53410f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:20.221 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.285+0000 7fb538a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb53410f9b0 con 0x7fb53410efe0 2024-04-24T07:59:20.221 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.285+0000 7fb532800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb53410efe0 0x7fb53410f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:20.222 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.285+0000 7fb532800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb53410efe0 0x7fb53410f3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45478/0 (socket says 172.21.15.28:45478) 2024-04-24T07:59:20.222 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.285+0000 7fb532800700 1 -- 172.21.15.28:0/331549808 learned_addr learned my addr 172.21.15.28:0/331549808 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:20.222 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.286+0000 7fb532800700 1 -- 172.21.15.28:0/331549808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb53410fb30 con 0x7fb53410efe0 2024-04-24T07:59:20.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.286+0000 7fb532800700 1 --2- 172.21.15.28:0/331549808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb53410efe0 0x7fb53410f3e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb5240098a0 tx=0x7fb524009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24d561b056abf5f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:20.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.287+0000 7fb531e00700 1 -- 172.21.15.28:0/331549808 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb524004160 con 0x7fb53410efe0 2024-04-24T07:59:20.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.287+0000 7fb531e00700 1 -- 172.21.15.28:0/331549808 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb524004300 con 0x7fb53410efe0 2024-04-24T07:59:20.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.287+0000 7fb531e00700 1 -- 172.21.15.28:0/331549808 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb524011b50 con 0x7fb53410efe0 2024-04-24T07:59:20.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.287+0000 7fb538a00700 1 -- 172.21.15.28:0/331549808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb53410efe0 msgr2=0x7fb53410f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:20.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.287+0000 7fb538a00700 1 --2- 172.21.15.28:0/331549808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb53410efe0 0x7fb53410f3e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb5240098a0 tx=0x7fb524009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:20.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.288+0000 7fb538a00700 1 -- 172.21.15.28:0/331549808 shutdown_connections 2024-04-24T07:59:20.224 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.288+0000 7fb538a00700 1 --2- 172.21.15.28:0/331549808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb53410efe0 0x7fb53410f3e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:20.225 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.288+0000 7fb538a00700 1 -- 172.21.15.28:0/331549808 >> 172.21.15.28:0/331549808 conn(0x7fb53407ddf0 msgr2=0x7fb534080210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:20.225 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.288+0000 7fb538a00700 1 -- 172.21.15.28:0/331549808 shutdown_connections 2024-04-24T07:59:20.225 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.288+0000 7fb538a00700 1 -- 172.21.15.28:0/331549808 wait complete. 2024-04-24T07:59:20.225 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.289+0000 7fb538a00700 1 Processor -- start 2024-04-24T07:59:20.225 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.289+0000 7fb538a00700 1 -- start start 2024-04-24T07:59:20.225 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.290+0000 7fb538a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb53410efe0 0x7fb5341a2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:20.227 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.290+0000 7fb538a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb534112fd0 con 0x7fb53410efe0 2024-04-24T07:59:20.227 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.290+0000 7fb532800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb53410efe0 0x7fb5341a2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:20.227 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.290+0000 7fb532800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb53410efe0 0x7fb5341a2ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45486/0 (socket says 172.21.15.28:45486) 2024-04-24T07:59:20.227 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.290+0000 7fb532800700 1 -- 172.21.15.28:0/4169314127 learned_addr learned my addr 172.21.15.28:0/4169314127 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:20.227 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.290+0000 7fb532800700 1 -- 172.21.15.28:0/4169314127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb5341a2ff0 con 0x7fb53410efe0 2024-04-24T07:59:20.227 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.291+0000 7fb532800700 1 --2- 172.21.15.28:0/4169314127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb53410efe0 0x7fb5341a2ab0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb524011b60 tx=0x7fb524011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:20.227 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.291+0000 7fb52be00700 1 -- 172.21.15.28:0/4169314127 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb524003970 con 0x7fb53410efe0 2024-04-24T07:59:20.227 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.291+0000 7fb52be00700 1 -- 172.21.15.28:0/4169314127 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb524003b10 con 0x7fb53410efe0 2024-04-24T07:59:20.227 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.291+0000 7fb52be00700 1 -- 172.21.15.28:0/4169314127 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb524024610 con 0x7fb53410efe0 2024-04-24T07:59:20.228 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.291+0000 7fb538a00700 1 -- 172.21.15.28:0/4169314127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb5341a3280 con 0x7fb53410efe0 2024-04-24T07:59:20.228 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.291+0000 7fb538a00700 1 -- 172.21.15.28:0/4169314127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5341a5f70 con 0x7fb53410efe0 2024-04-24T07:59:20.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.292+0000 7fb52be00700 1 -- 172.21.15.28:0/4169314127 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fb52401a9f0 con 0x7fb53410efe0 2024-04-24T07:59:20.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.292+0000 7fb538a00700 1 -- 172.21.15.28:0/4169314127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb534113150 con 0x7fb53410efe0 2024-04-24T07:59:20.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.293+0000 7fb52be00700 1 --2- 172.21.15.28:0/4169314127 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fb514039600 0x7fb51403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:20.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.293+0000 7fb52be00700 1 -- 172.21.15.28:0/4169314127 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb524011e00 con 0x7fb53410efe0 2024-04-24T07:59:20.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.293+0000 7fb529e00700 1 --2- 172.21.15.28:0/4169314127 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fb514039600 0x7fb51403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:20.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.294+0000 7fb529e00700 1 --2- 172.21.15.28:0/4169314127 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fb514039600 0x7fb51403bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb51c006fd0 tx=0x7fb51c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:20.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.299+0000 7fb52be00700 1 -- 172.21.15.28:0/4169314127 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb524016260 con 0x7fb53410efe0 2024-04-24T07:59:20.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.531+0000 7fb538a00700 1 -- 172.21.15.28:0/4169314127 --> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7fb5340800e0 con 0x7fb514039600 2024-04-24T07:59:20.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.532+0000 7fb52be00700 1 -- 172.21.15.28:0/4169314127 <== mgr.14120 v2:172.21.15.28:6800/3308597107 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7fb5340800e0 con 0x7fb514039600 2024-04-24T07:59:20.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.536+0000 7fb538a00700 1 -- 172.21.15.28:0/4169314127 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fb514039600 msgr2=0x7fb51403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:20.230 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.536+0000 7fb538a00700 1 --2- 172.21.15.28:0/4169314127 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fb514039600 0x7fb51403bac0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb51c006fd0 tx=0x7fb51c006e40 comp rx=0 tx=0).stop 2024-04-24T07:59:20.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.536+0000 7fb538a00700 1 -- 172.21.15.28:0/4169314127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb53410efe0 msgr2=0x7fb5341a2ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:20.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.536+0000 7fb538a00700 1 --2- 172.21.15.28:0/4169314127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb53410efe0 0x7fb5341a2ab0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb524011b60 tx=0x7fb524011c40 comp rx=0 tx=0).stop 2024-04-24T07:59:20.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.536+0000 7fb538a00700 1 -- 172.21.15.28:0/4169314127 shutdown_connections 2024-04-24T07:59:20.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.536+0000 7fb538a00700 1 --2- 172.21.15.28:0/4169314127 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fb514039600 0x7fb51403bac0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:20.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.536+0000 7fb538a00700 1 --2- 172.21.15.28:0/4169314127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb53410efe0 0x7fb5341a2ab0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:20.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.536+0000 7fb538a00700 1 -- 172.21.15.28:0/4169314127 >> 172.21.15.28:0/4169314127 conn(0x7fb53407ddf0 msgr2=0x7fb53407f9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:20.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.536+0000 7fb538a00700 1 -- 172.21.15.28:0/4169314127 shutdown_connections 2024-04-24T07:59:20.231 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:19.537+0000 7fb538a00700 1 -- 172.21.15.28:0/4169314127 wait complete. 2024-04-24T07:59:20.231 INFO:teuthology.orchestra.run.smithi028.stdout:Generating ssh key... 2024-04-24T07:59:20.525 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:20 smithi028 ceph-mon[27786]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T07:59:21.761 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-mon[27786]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T07:59:21.761 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-mon[27786]: Generating ssh key... 2024-04-24T07:59:21.763 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: Generating public/private rsa key pair. 2024-04-24T07:59:21.763 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: Your identification has been saved in /tmp/tmpe4w0gnu4/key. 2024-04-24T07:59:21.763 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: Your public key has been saved in /tmp/tmpe4w0gnu4/key.pub. 2024-04-24T07:59:21.763 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: The key fingerprint is: 2024-04-24T07:59:21.763 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: SHA256:eRWOThMRz9M34C8gCqdjWnMVwG4xQVMOcNtxu31+Lyo ceph-473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T07:59:21.763 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: The key's randomart image is: 2024-04-24T07:59:21.763 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: +---[RSA 3072]----+ 2024-04-24T07:59:21.763 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: | .+B++++.. | 2024-04-24T07:59:21.763 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: | .+* +*oo. | 2024-04-24T07:59:21.763 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: | ..oo=+o*....| 2024-04-24T07:59:21.763 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: | +oo+.o+....| 2024-04-24T07:59:21.763 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: | *.oS o. o o | 2024-04-24T07:59:21.763 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: | + + . + | 2024-04-24T07:59:21.763 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: | . ..| 2024-04-24T07:59:21.764 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: | E . o| 2024-04-24T07:59:21.764 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: | ... ..| 2024-04-24T07:59:21.764 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: +----[SHA256]-----+ 2024-04-24T07:59:22.146 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.830+0000 7eff87e00700 1 Processor -- start 2024-04-24T07:59:22.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.830+0000 7eff87e00700 1 -- start start 2024-04-24T07:59:22.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.831+0000 7eff87e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff88106d70 0x7eff88107190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:22.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.831+0000 7eff87e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff8807beb0 con 0x7eff88106d70 2024-04-24T07:59:22.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.831+0000 7eff86a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff88106d70 0x7eff88107190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:22.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.832+0000 7eff86a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff88106d70 0x7eff88107190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49496/0 (socket says 172.21.15.28:49496) 2024-04-24T07:59:22.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.832+0000 7eff86a00700 1 -- 172.21.15.28:0/2495285560 learned_addr learned my addr 172.21.15.28:0/2495285560 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:22.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.832+0000 7eff86a00700 1 -- 172.21.15.28:0/2495285560 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff881076d0 con 0x7eff88106d70 2024-04-24T07:59:22.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.832+0000 7eff86a00700 1 --2- 172.21.15.28:0/2495285560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff88106d70 0x7eff88107190 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7eff780098a0 tx=0x7eff78009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=596c4a22433e77cc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:22.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.833+0000 7eff85600700 1 -- 172.21.15.28:0/2495285560 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff78004160 con 0x7eff88106d70 2024-04-24T07:59:22.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.833+0000 7eff85600700 1 -- 172.21.15.28:0/2495285560 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7eff78004300 con 0x7eff88106d70 2024-04-24T07:59:22.147 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.833+0000 7eff85600700 1 -- 172.21.15.28:0/2495285560 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff78011b50 con 0x7eff88106d70 2024-04-24T07:59:22.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.833+0000 7eff87e00700 1 -- 172.21.15.28:0/2495285560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff88106d70 msgr2=0x7eff88107190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:22.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.833+0000 7eff87e00700 1 --2- 172.21.15.28:0/2495285560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff88106d70 0x7eff88107190 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7eff780098a0 tx=0x7eff78009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:22.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.834+0000 7eff87e00700 1 -- 172.21.15.28:0/2495285560 shutdown_connections 2024-04-24T07:59:22.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.834+0000 7eff87e00700 1 --2- 172.21.15.28:0/2495285560 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff88106d70 0x7eff88107190 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:22.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.834+0000 7eff87e00700 1 -- 172.21.15.28:0/2495285560 >> 172.21.15.28:0/2495285560 conn(0x7eff88102a10 msgr2=0x7eff88104e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:22.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.834+0000 7eff87e00700 1 -- 172.21.15.28:0/2495285560 shutdown_connections 2024-04-24T07:59:22.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.834+0000 7eff87e00700 1 -- 172.21.15.28:0/2495285560 wait complete. 2024-04-24T07:59:22.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.835+0000 7eff87e00700 1 Processor -- start 2024-04-24T07:59:22.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.835+0000 7eff87e00700 1 -- start start 2024-04-24T07:59:22.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.835+0000 7eff87e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff88106d70 0x7eff8819e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:22.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.835+0000 7eff87e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff88108760 con 0x7eff88106d70 2024-04-24T07:59:22.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.836+0000 7eff86a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff88106d70 0x7eff8819e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:22.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.836+0000 7eff86a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff88106d70 0x7eff8819e6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49508/0 (socket says 172.21.15.28:49508) 2024-04-24T07:59:22.148 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.836+0000 7eff86a00700 1 -- 172.21.15.28:0/425919204 learned_addr learned my addr 172.21.15.28:0/425919204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:22.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.836+0000 7eff86a00700 1 -- 172.21.15.28:0/425919204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff8819ec20 con 0x7eff88106d70 2024-04-24T07:59:22.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.836+0000 7eff86a00700 1 --2- 172.21.15.28:0/425919204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff88106d70 0x7eff8819e6e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7eff78009fb0 tx=0x7eff78011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:22.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.836+0000 7eff8ca00700 1 -- 172.21.15.28:0/425919204 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff780038b0 con 0x7eff88106d70 2024-04-24T07:59:22.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.837+0000 7eff8ca00700 1 -- 172.21.15.28:0/425919204 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7eff78011e50 con 0x7eff88106d70 2024-04-24T07:59:22.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.837+0000 7eff8ca00700 1 -- 172.21.15.28:0/425919204 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7eff78019420 con 0x7eff88106d70 2024-04-24T07:59:22.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.837+0000 7eff87e00700 1 -- 172.21.15.28:0/425919204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff8819eeb0 con 0x7eff88106d70 2024-04-24T07:59:22.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.837+0000 7eff87e00700 1 -- 172.21.15.28:0/425919204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff881a1ba0 con 0x7eff88106d70 2024-04-24T07:59:22.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.838+0000 7eff8ca00700 1 -- 172.21.15.28:0/425919204 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7eff78003a50 con 0x7eff88106d70 2024-04-24T07:59:22.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.838+0000 7eff87e00700 1 -- 172.21.15.28:0/425919204 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7eff88064150 con 0x7eff88106d70 2024-04-24T07:59:22.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.839+0000 7eff8ca00700 1 --2- 172.21.15.28:0/425919204 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7eff6c039560 0x7eff6c03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:22.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.839+0000 7eff8ca00700 1 -- 172.21.15.28:0/425919204 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7eff7804da60 con 0x7eff88106d70 2024-04-24T07:59:22.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.839+0000 7eff86000700 1 --2- 172.21.15.28:0/425919204 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7eff6c039560 0x7eff6c03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:22.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.840+0000 7eff86000700 1 --2- 172.21.15.28:0/425919204 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7eff6c039560 0x7eff6c03ba20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7eff74006fd0 tx=0x7eff74006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:22.149 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:20.845+0000 7eff8ca00700 1 -- 172.21.15.28:0/425919204 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7eff7804d3b0 con 0x7eff88106d70 2024-04-24T07:59:22.150 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:21.078+0000 7eff87e00700 1 -- 172.21.15.28:0/425919204 --> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7eff880008d0 con 0x7eff6c039560 2024-04-24T07:59:22.150 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:21.449+0000 7eff8ca00700 1 -- 172.21.15.28:0/425919204 <== mgr.14120 v2:172.21.15.28:6800/3308597107 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7eff880008d0 con 0x7eff6c039560 2024-04-24T07:59:22.150 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:21.452+0000 7eff87e00700 1 -- 172.21.15.28:0/425919204 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7eff6c039560 msgr2=0x7eff6c03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:22.150 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:21.452+0000 7eff87e00700 1 --2- 172.21.15.28:0/425919204 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7eff6c039560 0x7eff6c03ba20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7eff74006fd0 tx=0x7eff74006e40 comp rx=0 tx=0).stop 2024-04-24T07:59:22.150 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:21.452+0000 7eff87e00700 1 -- 172.21.15.28:0/425919204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff88106d70 msgr2=0x7eff8819e6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:22.150 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:21.452+0000 7eff87e00700 1 --2- 172.21.15.28:0/425919204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff88106d70 0x7eff8819e6e0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7eff78009fb0 tx=0x7eff78011bf0 comp rx=0 tx=0).stop 2024-04-24T07:59:22.150 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:21.452+0000 7eff87e00700 1 -- 172.21.15.28:0/425919204 shutdown_connections 2024-04-24T07:59:22.150 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:21.452+0000 7eff87e00700 1 --2- 172.21.15.28:0/425919204 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7eff6c039560 0x7eff6c03ba20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:22.150 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:21.452+0000 7eff87e00700 1 --2- 172.21.15.28:0/425919204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7eff88106d70 0x7eff8819e6e0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:22.150 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:21.452+0000 7eff87e00700 1 -- 172.21.15.28:0/425919204 >> 172.21.15.28:0/425919204 conn(0x7eff88102a10 msgr2=0x7eff88195370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:22.150 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:21.453+0000 7eff87e00700 1 -- 172.21.15.28:0/425919204 shutdown_connections 2024-04-24T07:59:22.150 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:21.453+0000 7eff87e00700 1 -- 172.21.15.28:0/425919204 wait complete. 2024-04-24T07:59:22.763 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:22 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' 2024-04-24T07:59:22.763 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:22 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' 2024-04-24T07:59:23.680 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:23 smithi028 ceph-mon[27786]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T07:59:23.699 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZYhBI+GUk+9C6auHe/w5oj/55pFiy+mOUtLbhVDX2zrYYUVmdd9K4XwMzlLpSkzVR2UiBCbYk2bKpr9ni2n7FRrW3vIO72eRI07uLEdoe2p1vaud9GTFfsA2vBDDMuQHvF8WQuEBdoYBq7nTc3J13d/Pk+X1KVw8g19xXqSfeWhM05I+PyJ0QFfwuZxK/fCBZbL8+tqpzEuUlObLf61h27K+vNoJeFkEczEqk6vMUiCZwcX1xLry33wqWivlzP5E0735b0LudFXAuF2RLaok6+/5b5EIHRnMctjygQTQkrPikyrDHTD1Rkd2zo7Wyh0H5hCykRhTZoVngHZv4TfQp4KQiKCttXupGM5YAlFKCZ3CxmCv1+b/puPhyssHAOwaVCBmrg1RheOTF5sZZw97gzXWL0xS6hE4JInC+VznSuy2ByBjEVtroPQ0Sr3a79dxE1X7rD2csMQZ8aazsCkB53ff0rT8ktMnxCHC49DxPyxdb2cQ0nXFE3Tiv/ZpCPDs= ceph-473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T07:59:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.761+0000 7ff81b600700 1 Processor -- start 2024-04-24T07:59:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.761+0000 7ff81b600700 1 -- start start 2024-04-24T07:59:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.762+0000 7ff81b600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814109150 0x7ff81410b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.762+0000 7ff81b600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff81407bed0 con 0x7ff814109150 2024-04-24T07:59:23.700 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.763+0000 7ff818c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814109150 0x7ff81410b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:23.701 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.763+0000 7ff818c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814109150 0x7ff81410b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49516/0 (socket says 172.21.15.28:49516) 2024-04-24T07:59:23.701 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.763+0000 7ff818c00700 1 -- 172.21.15.28:0/2176071744 learned_addr learned my addr 172.21.15.28:0/2176071744 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:23.701 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.763+0000 7ff818c00700 1 -- 172.21.15.28:0/2176071744 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81410ba80 con 0x7ff814109150 2024-04-24T07:59:23.701 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.764+0000 7ff818c00700 1 --2- 172.21.15.28:0/2176071744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814109150 0x7ff81410b540 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff8080098a0 tx=0x7ff808009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eebe65a9f113282b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:23.702 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.764+0000 7ff813400700 1 -- 172.21.15.28:0/2176071744 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff808004160 con 0x7ff814109150 2024-04-24T07:59:23.703 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.764+0000 7ff813400700 1 -- 172.21.15.28:0/2176071744 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff808004300 con 0x7ff814109150 2024-04-24T07:59:23.703 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.764+0000 7ff813400700 1 -- 172.21.15.28:0/2176071744 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff808011b50 con 0x7ff814109150 2024-04-24T07:59:23.703 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.765+0000 7ff81b600700 1 -- 172.21.15.28:0/2176071744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814109150 msgr2=0x7ff81410b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:23.703 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.765+0000 7ff81b600700 1 --2- 172.21.15.28:0/2176071744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814109150 0x7ff81410b540 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7ff8080098a0 tx=0x7ff808009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:23.705 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.765+0000 7ff81b600700 1 -- 172.21.15.28:0/2176071744 shutdown_connections 2024-04-24T07:59:23.705 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.765+0000 7ff81b600700 1 --2- 172.21.15.28:0/2176071744 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814109150 0x7ff81410b540 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:23.705 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.765+0000 7ff81b600700 1 -- 172.21.15.28:0/2176071744 >> 172.21.15.28:0/2176071744 conn(0x7ff814102be0 msgr2=0x7ff814105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:23.705 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.766+0000 7ff81b600700 1 -- 172.21.15.28:0/2176071744 shutdown_connections 2024-04-24T07:59:23.705 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.766+0000 7ff81b600700 1 -- 172.21.15.28:0/2176071744 wait complete. 2024-04-24T07:59:23.705 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.767+0000 7ff81b600700 1 Processor -- start 2024-04-24T07:59:23.705 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.767+0000 7ff81b600700 1 -- start start 2024-04-24T07:59:23.705 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.767+0000 7ff81b600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814109150 0x7ff81419e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:23.705 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.767+0000 7ff81b600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff81410cab0 con 0x7ff814109150 2024-04-24T07:59:23.705 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.768+0000 7ff818c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814109150 0x7ff81419e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:23.706 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.768+0000 7ff818c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814109150 0x7ff81419e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49526/0 (socket says 172.21.15.28:49526) 2024-04-24T07:59:23.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.768+0000 7ff818c00700 1 -- 172.21.15.28:0/1955356137 learned_addr learned my addr 172.21.15.28:0/1955356137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:23.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.768+0000 7ff818c00700 1 -- 172.21.15.28:0/1955356137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff81419ede0 con 0x7ff814109150 2024-04-24T07:59:23.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.768+0000 7ff818c00700 1 --2- 172.21.15.28:0/1955356137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814109150 0x7ff81419e8a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff808011b60 tx=0x7ff808011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:23.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.769+0000 7ff811600700 1 -- 172.21.15.28:0/1955356137 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff808003b20 con 0x7ff814109150 2024-04-24T07:59:23.708 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.769+0000 7ff811600700 1 -- 172.21.15.28:0/1955356137 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff808003cc0 con 0x7ff814109150 2024-04-24T07:59:23.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.769+0000 7ff811600700 1 -- 172.21.15.28:0/1955356137 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff808024840 con 0x7ff814109150 2024-04-24T07:59:23.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.769+0000 7ff81b600700 1 -- 172.21.15.28:0/1955356137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff81419f070 con 0x7ff814109150 2024-04-24T07:59:23.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.769+0000 7ff81b600700 1 -- 172.21.15.28:0/1955356137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8141a1d60 con 0x7ff814109150 2024-04-24T07:59:23.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.770+0000 7ff811600700 1 -- 172.21.15.28:0/1955356137 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7ff808022020 con 0x7ff814109150 2024-04-24T07:59:23.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.770+0000 7ff81b600700 1 -- 172.21.15.28:0/1955356137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff814064150 con 0x7ff814109150 2024-04-24T07:59:23.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.771+0000 7ff811600700 1 --2- 172.21.15.28:0/1955356137 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7ff7fc039600 0x7ff7fc03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:23.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.771+0000 7ff811600700 1 -- 172.21.15.28:0/1955356137 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7ff808011e00 con 0x7ff814109150 2024-04-24T07:59:23.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.771+0000 7ff813e00700 1 --2- 172.21.15.28:0/1955356137 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7ff7fc039600 0x7ff7fc03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:23.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.772+0000 7ff813e00700 1 --2- 172.21.15.28:0/1955356137 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7ff7fc039600 0x7ff7fc03bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff800006fd0 tx=0x7ff800006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:23.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:22.776+0000 7ff811600700 1 -- 172.21.15.28:0/1955356137 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff80801a950 con 0x7ff814109150 2024-04-24T07:59:23.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:23.008+0000 7ff81b600700 1 -- 172.21.15.28:0/1955356137 --> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7ff814103dd0 con 0x7ff7fc039600 2024-04-24T07:59:23.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:23.009+0000 7ff811600700 1 -- 172.21.15.28:0/1955356137 <== mgr.14120 v2:172.21.15.28:6800/3308597107 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7ff814103dd0 con 0x7ff7fc039600 2024-04-24T07:59:23.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:23.012+0000 7ff81b600700 1 -- 172.21.15.28:0/1955356137 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7ff7fc039600 msgr2=0x7ff7fc03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:23.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:23.012+0000 7ff81b600700 1 --2- 172.21.15.28:0/1955356137 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7ff7fc039600 0x7ff7fc03bac0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff800006fd0 tx=0x7ff800006e40 comp rx=0 tx=0).stop 2024-04-24T07:59:23.709 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:23.012+0000 7ff81b600700 1 -- 172.21.15.28:0/1955356137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814109150 msgr2=0x7ff81419e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:23.710 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:23.012+0000 7ff81b600700 1 --2- 172.21.15.28:0/1955356137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814109150 0x7ff81419e8a0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7ff808011b60 tx=0x7ff808011c40 comp rx=0 tx=0).stop 2024-04-24T07:59:23.710 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:23.013+0000 7ff81b600700 1 -- 172.21.15.28:0/1955356137 shutdown_connections 2024-04-24T07:59:23.710 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:23.013+0000 7ff81b600700 1 --2- 172.21.15.28:0/1955356137 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7ff7fc039600 0x7ff7fc03bac0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:23.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:23.013+0000 7ff81b600700 1 --2- 172.21.15.28:0/1955356137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff814109150 0x7ff81419e8a0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:23.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:23.013+0000 7ff81b600700 1 -- 172.21.15.28:0/1955356137 >> 172.21.15.28:0/1955356137 conn(0x7ff814102be0 msgr2=0x7ff8141036c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:23.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:23.013+0000 7ff81b600700 1 -- 172.21.15.28:0/1955356137 shutdown_connections 2024-04-24T07:59:23.711 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:23.014+0000 7ff81b600700 1 -- 172.21.15.28:0/1955356137 wait complete. 2024-04-24T07:59:23.711 INFO:teuthology.orchestra.run.smithi028.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-04-24T07:59:23.711 INFO:teuthology.orchestra.run.smithi028.stdout:Adding key to root@localhost authorized_keys... 2024-04-24T07:59:23.724 INFO:teuthology.orchestra.run.smithi028.stdout:Adding host smithi028... 2024-04-24T07:59:25.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:25 smithi028 ceph-mon[27786]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T07:59:26.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:26 smithi028 ceph-mon[27786]: Deploying cephadm binary to smithi028 2024-04-24T07:59:31.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Added host 'smithi028' with addr '172.21.15.28' 2024-04-24T07:59:31.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.298+0000 7fe1e7e00700 1 Processor -- start 2024-04-24T07:59:31.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.298+0000 7fe1e7e00700 1 -- start start 2024-04-24T07:59:31.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.299+0000 7fe1e7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e810a760 0x7fe1e810ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:31.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.299+0000 7fe1e7e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e810b150 con 0x7fe1e810a760 2024-04-24T07:59:31.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.299+0000 7fe1e6a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e810a760 0x7fe1e810ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:31.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.299+0000 7fe1e6a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e810a760 0x7fe1e810ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49538/0 (socket says 172.21.15.28:49538) 2024-04-24T07:59:31.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.299+0000 7fe1e6a00700 1 -- 172.21.15.28:0/4060525143 learned_addr learned my addr 172.21.15.28:0/4060525143 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:31.824 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.300+0000 7fe1e6a00700 1 -- 172.21.15.28:0/4060525143 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e810b910 con 0x7fe1e810a760 2024-04-24T07:59:31.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.300+0000 7fe1e6a00700 1 --2- 172.21.15.28:0/4060525143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e810a760 0x7fe1e810ab80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe1d80098a0 tx=0x7fe1d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23cbd44d268f6d46 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:31.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.300+0000 7fe1e5600700 1 -- 172.21.15.28:0/4060525143 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1d8004160 con 0x7fe1e810a760 2024-04-24T07:59:31.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.300+0000 7fe1e5600700 1 -- 172.21.15.28:0/4060525143 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe1d8004300 con 0x7fe1e810a760 2024-04-24T07:59:31.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.300+0000 7fe1e5600700 1 -- 172.21.15.28:0/4060525143 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1d8011b50 con 0x7fe1e810a760 2024-04-24T07:59:31.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.301+0000 7fe1e7e00700 1 -- 172.21.15.28:0/4060525143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e810a760 msgr2=0x7fe1e810ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:31.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.301+0000 7fe1e7e00700 1 --2- 172.21.15.28:0/4060525143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e810a760 0x7fe1e810ab80 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe1d80098a0 tx=0x7fe1d8009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:31.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.301+0000 7fe1e7e00700 1 -- 172.21.15.28:0/4060525143 shutdown_connections 2024-04-24T07:59:31.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.301+0000 7fe1e7e00700 1 --2- 172.21.15.28:0/4060525143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e810a760 0x7fe1e810ab80 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:31.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.301+0000 7fe1e7e00700 1 -- 172.21.15.28:0/4060525143 >> 172.21.15.28:0/4060525143 conn(0x7fe1e807de30 msgr2=0x7fe1e81084f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:31.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.301+0000 7fe1e7e00700 1 -- 172.21.15.28:0/4060525143 shutdown_connections 2024-04-24T07:59:31.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.302+0000 7fe1e7e00700 1 -- 172.21.15.28:0/4060525143 wait complete. 2024-04-24T07:59:31.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.303+0000 7fe1e7e00700 1 Processor -- start 2024-04-24T07:59:31.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.303+0000 7fe1e7e00700 1 -- start start 2024-04-24T07:59:31.825 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.303+0000 7fe1e7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e810a760 0x7fe1e819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:31.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.303+0000 7fe1e7e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e810ecc0 con 0x7fe1e810a760 2024-04-24T07:59:31.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.304+0000 7fe1e6a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e810a760 0x7fe1e819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:31.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.304+0000 7fe1e6a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e810a760 0x7fe1e819e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:49550/0 (socket says 172.21.15.28:49550) 2024-04-24T07:59:31.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.304+0000 7fe1e6a00700 1 -- 172.21.15.28:0/3471676781 learned_addr learned my addr 172.21.15.28:0/3471676781 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:31.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.304+0000 7fe1e6a00700 1 -- 172.21.15.28:0/3471676781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e819ebf0 con 0x7fe1e810a760 2024-04-24T07:59:31.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.304+0000 7fe1e6a00700 1 --2- 172.21.15.28:0/3471676781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e810a760 0x7fe1e819e6b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8011670 tx=0x7fe1d8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:31.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.305+0000 7fe1eca00700 1 -- 172.21.15.28:0/3471676781 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1d80039c0 con 0x7fe1e810a760 2024-04-24T07:59:31.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.305+0000 7fe1eca00700 1 -- 172.21.15.28:0/3471676781 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe1d8011e50 con 0x7fe1e810a760 2024-04-24T07:59:31.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.305+0000 7fe1e7e00700 1 -- 172.21.15.28:0/3471676781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1e819ee80 con 0x7fe1e810a760 2024-04-24T07:59:31.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.305+0000 7fe1eca00700 1 -- 172.21.15.28:0/3471676781 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1d801a5c0 con 0x7fe1e810a760 2024-04-24T07:59:31.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.305+0000 7fe1e7e00700 1 -- 172.21.15.28:0/3471676781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1e81a1b70 con 0x7fe1e810a760 2024-04-24T07:59:31.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.306+0000 7fe1e7e00700 1 -- 172.21.15.28:0/3471676781 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1e8064150 con 0x7fe1e810a760 2024-04-24T07:59:31.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.306+0000 7fe1eca00700 1 -- 172.21.15.28:0/3471676781 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fe1d801a8c0 con 0x7fe1e810a760 2024-04-24T07:59:31.826 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.307+0000 7fe1eca00700 1 --2- 172.21.15.28:0/3471676781 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fe1cc039560 0x7fe1cc03ba20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:31.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.307+0000 7fe1eca00700 1 -- 172.21.15.28:0/3471676781 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe1d804dcd0 con 0x7fe1e810a760 2024-04-24T07:59:31.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.307+0000 7fe1e6000700 1 --2- 172.21.15.28:0/3471676781 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fe1cc039560 0x7fe1cc03ba20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:31.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.308+0000 7fe1e6000700 1 --2- 172.21.15.28:0/3471676781 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fe1cc039560 0x7fe1cc03ba20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4006fd0 tx=0x7fe1d4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:31.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.312+0000 7fe1eca00700 1 -- 172.21.15.28:0/3471676781 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe1d804d620 con 0x7fe1e810a760 2024-04-24T07:59:31.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:24.548+0000 7fe1e7e00700 1 -- 172.21.15.28:0/3471676781 --> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi028", "addr": "172.21.15.28", "target": ["mon-mgr", ""]}) -- 0x7fe1e81083c0 con 0x7fe1cc039560 2024-04-24T07:59:31.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:30.998+0000 7fe1eca00700 1 -- 172.21.15.28:0/3471676781 <== mgr.14120 v2:172.21.15.28:6800/3308597107 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7fe1e81083c0 con 0x7fe1cc039560 2024-04-24T07:59:31.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:31.002+0000 7fe1e7e00700 1 -- 172.21.15.28:0/3471676781 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fe1cc039560 msgr2=0x7fe1cc03ba20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:31.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:31.002+0000 7fe1e7e00700 1 --2- 172.21.15.28:0/3471676781 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fe1cc039560 0x7fe1cc03ba20 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe1d4006fd0 tx=0x7fe1d4006e40 comp rx=0 tx=0).stop 2024-04-24T07:59:31.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:31.002+0000 7fe1e7e00700 1 -- 172.21.15.28:0/3471676781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e810a760 msgr2=0x7fe1e819e6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:31.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:31.002+0000 7fe1e7e00700 1 --2- 172.21.15.28:0/3471676781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e810a760 0x7fe1e819e6b0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8011670 tx=0x7fe1d8011b60 comp rx=0 tx=0).stop 2024-04-24T07:59:31.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:31.002+0000 7fe1e7e00700 1 -- 172.21.15.28:0/3471676781 shutdown_connections 2024-04-24T07:59:31.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:31.002+0000 7fe1e7e00700 1 --2- 172.21.15.28:0/3471676781 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fe1cc039560 0x7fe1cc03ba20 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:31.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:31.002+0000 7fe1e7e00700 1 --2- 172.21.15.28:0/3471676781 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1e810a760 0x7fe1e819e6b0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:31.827 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:31.002+0000 7fe1e7e00700 1 -- 172.21.15.28:0/3471676781 >> 172.21.15.28:0/3471676781 conn(0x7fe1e807de30 msgr2=0x7fe1e8107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:31.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:31.003+0000 7fe1e7e00700 1 -- 172.21.15.28:0/3471676781 shutdown_connections 2024-04-24T07:59:31.828 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:31.003+0000 7fe1e7e00700 1 -- 172.21.15.28:0/3471676781 wait complete. 2024-04-24T07:59:31.828 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mon service... 2024-04-24T07:59:32.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:31 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' 2024-04-24T07:59:32.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:31 smithi028 ceph-mon[27786]: Added host smithi028 2024-04-24T07:59:32.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:31 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T07:59:33.454 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-04-24T07:59:33.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.458+0000 7f4aa7200700 1 Processor -- start 2024-04-24T07:59:33.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.458+0000 7f4aa7200700 1 -- start start 2024-04-24T07:59:33.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.458+0000 7f4aa7200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010f030 0x7f4aa010f450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:33.455 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.458+0000 7f4aa7200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa010f990 con 0x7f4aa010f030 2024-04-24T07:59:33.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.459+0000 7f4a9fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010f030 0x7f4aa010f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:33.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.459+0000 7f4a9fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010f030 0x7f4aa010f450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60844/0 (socket says 172.21.15.28:60844) 2024-04-24T07:59:33.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.459+0000 7f4a9fe00700 1 -- 172.21.15.28:0/1134217150 learned_addr learned my addr 172.21.15.28:0/1134217150 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:33.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.459+0000 7f4a9fe00700 1 -- 172.21.15.28:0/1134217150 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa010fb10 con 0x7f4aa010f030 2024-04-24T07:59:33.459 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.460+0000 7f4a9fe00700 1 --2- 172.21.15.28:0/1134217150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010f030 0x7f4aa010f450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4a900098a0 tx=0x7f4a90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbe45aa52a68b8db server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:33.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.460+0000 7f4a9ea00700 1 -- 172.21.15.28:0/1134217150 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a90004160 con 0x7f4aa010f030 2024-04-24T07:59:33.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.460+0000 7f4a9ea00700 1 -- 172.21.15.28:0/1134217150 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4a90004300 con 0x7f4aa010f030 2024-04-24T07:59:33.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.460+0000 7f4a9ea00700 1 -- 172.21.15.28:0/1134217150 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a90011b50 con 0x7f4aa010f030 2024-04-24T07:59:33.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.461+0000 7f4aa7200700 1 -- 172.21.15.28:0/1134217150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010f030 msgr2=0x7f4aa010f450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:33.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.461+0000 7f4aa7200700 1 --2- 172.21.15.28:0/1134217150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010f030 0x7f4aa010f450 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f4a900098a0 tx=0x7f4a90009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:33.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.461+0000 7f4aa7200700 1 -- 172.21.15.28:0/1134217150 shutdown_connections 2024-04-24T07:59:33.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.461+0000 7f4aa7200700 1 --2- 172.21.15.28:0/1134217150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010f030 0x7f4aa010f450 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:33.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.461+0000 7f4aa7200700 1 -- 172.21.15.28:0/1134217150 >> 172.21.15.28:0/1134217150 conn(0x7f4aa007def0 msgr2=0x7f4aa0080310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:33.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.461+0000 7f4aa7200700 1 -- 172.21.15.28:0/1134217150 shutdown_connections 2024-04-24T07:59:33.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.461+0000 7f4aa7200700 1 -- 172.21.15.28:0/1134217150 wait complete. 2024-04-24T07:59:33.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.462+0000 7f4aa7200700 1 Processor -- start 2024-04-24T07:59:33.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.462+0000 7f4aa7200700 1 -- start start 2024-04-24T07:59:33.460 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.462+0000 7f4aa7200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010f030 0x7f4aa01a2c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:33.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.462+0000 7f4aa7200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa0110d30 con 0x7f4aa010f030 2024-04-24T07:59:33.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.463+0000 7f4a9fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010f030 0x7f4aa01a2c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:33.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.463+0000 7f4a9fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010f030 0x7f4aa01a2c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60858/0 (socket says 172.21.15.28:60858) 2024-04-24T07:59:33.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.463+0000 7f4a9fe00700 1 -- 172.21.15.28:0/3765019416 learned_addr learned my addr 172.21.15.28:0/3765019416 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:33.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.463+0000 7f4a9fe00700 1 -- 172.21.15.28:0/3765019416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa01a3170 con 0x7f4aa010f030 2024-04-24T07:59:33.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.463+0000 7f4a9fe00700 1 --2- 172.21.15.28:0/3765019416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010f030 0x7f4aa01a2c30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4a90011670 tx=0x7f4a90011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:33.462 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.464+0000 7f4a9cc00700 1 -- 172.21.15.28:0/3765019416 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a900039c0 con 0x7f4aa010f030 2024-04-24T07:59:33.463 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.464+0000 7f4a9cc00700 1 -- 172.21.15.28:0/3765019416 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f4a90011e50 con 0x7f4aa010f030 2024-04-24T07:59:33.463 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.464+0000 7f4a9cc00700 1 -- 172.21.15.28:0/3765019416 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a9001a5c0 con 0x7f4aa010f030 2024-04-24T07:59:33.463 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.464+0000 7f4aa7200700 1 -- 172.21.15.28:0/3765019416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa01a3400 con 0x7f4aa010f030 2024-04-24T07:59:33.463 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.464+0000 7f4aa7200700 1 -- 172.21.15.28:0/3765019416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa01a60f0 con 0x7f4aa010f030 2024-04-24T07:59:33.463 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.465+0000 7f4a9cc00700 1 -- 172.21.15.28:0/3765019416 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f4a90024430 con 0x7f4aa010f030 2024-04-24T07:59:33.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.465+0000 7f4aa7200700 1 -- 172.21.15.28:0/3765019416 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aa0064150 con 0x7f4aa010f030 2024-04-24T07:59:33.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.466+0000 7f4a9cc00700 1 --2- 172.21.15.28:0/3765019416 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f4a84039610 0x7f4a8403bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:33.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.466+0000 7f4a9cc00700 1 -- 172.21.15.28:0/3765019416 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4a9004db80 con 0x7f4aa010f030 2024-04-24T07:59:33.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.466+0000 7f4a9f400700 1 --2- 172.21.15.28:0/3765019416 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f4a84039610 0x7f4a8403bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:33.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.467+0000 7f4a9f400700 1 --2- 172.21.15.28:0/3765019416 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f4a84039610 0x7f4a8403bad0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c006fd0 tx=0x7f4a8c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:33.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.472+0000 7f4a9cc00700 1 -- 172.21.15.28:0/3765019416 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a9002d910 con 0x7f4aa010f030 2024-04-24T07:59:33.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.719+0000 7f4aa7200700 1 -- 172.21.15.28:0/3765019416 --> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7f4aa007f0d0 con 0x7f4a84039610 2024-04-24T07:59:33.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.730+0000 7f4a9cc00700 1 -- 172.21.15.28:0/3765019416 <== mgr.14120 v2:172.21.15.28:6800/3308597107 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f4aa007f0d0 con 0x7f4a84039610 2024-04-24T07:59:33.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.733+0000 7f4aa7200700 1 -- 172.21.15.28:0/3765019416 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f4a84039610 msgr2=0x7f4a8403bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:33.465 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.733+0000 7f4aa7200700 1 --2- 172.21.15.28:0/3765019416 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f4a84039610 0x7f4a8403bad0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c006fd0 tx=0x7f4a8c006e40 comp rx=0 tx=0).stop 2024-04-24T07:59:33.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.733+0000 7f4aa7200700 1 -- 172.21.15.28:0/3765019416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010f030 msgr2=0x7f4aa01a2c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:33.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.733+0000 7f4aa7200700 1 --2- 172.21.15.28:0/3765019416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010f030 0x7f4aa01a2c30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f4a90011670 tx=0x7f4a90011b60 comp rx=0 tx=0).stop 2024-04-24T07:59:33.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.734+0000 7f4aa7200700 1 -- 172.21.15.28:0/3765019416 shutdown_connections 2024-04-24T07:59:33.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.734+0000 7f4aa7200700 1 --2- 172.21.15.28:0/3765019416 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f4a84039610 0x7f4a8403bad0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:33.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.734+0000 7f4aa7200700 1 --2- 172.21.15.28:0/3765019416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010f030 0x7f4aa01a2c30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:33.466 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.734+0000 7f4aa7200700 1 -- 172.21.15.28:0/3765019416 >> 172.21.15.28:0/3765019416 conn(0x7f4aa007def0 msgr2=0x7f4aa007e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:33.467 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.734+0000 7f4aa7200700 1 -- 172.21.15.28:0/3765019416 shutdown_connections 2024-04-24T07:59:33.467 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:32.734+0000 7f4aa7200700 1 -- 172.21.15.28:0/3765019416 wait complete. 2024-04-24T07:59:33.467 INFO:teuthology.orchestra.run.smithi028.stdout:Deploying unmanaged mgr service... 2024-04-24T07:59:33.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:33 smithi028 ceph-mon[27786]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T07:59:33.998 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:33 smithi028 ceph-mon[27786]: Saving service mon spec with placement count:5 2024-04-24T07:59:33.999 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:33 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' 2024-04-24T07:59:35.842 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-04-24T07:59:35.843 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.795+0000 7fe1bce00700 1 Processor -- start 2024-04-24T07:59:35.843 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.796+0000 7fe1bce00700 1 -- start start 2024-04-24T07:59:35.843 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.796+0000 7fe1bce00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b8109010 0x7fe1b810b400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:35.843 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.796+0000 7fe1bce00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b807be90 con 0x7fe1b8109010 2024-04-24T07:59:35.843 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.797+0000 7fe1b7400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b8109010 0x7fe1b810b400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:35.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.797+0000 7fe1b7400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b8109010 0x7fe1b810b400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60862/0 (socket says 172.21.15.28:60862) 2024-04-24T07:59:35.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.797+0000 7fe1b7400700 1 -- 172.21.15.28:0/1342129497 learned_addr learned my addr 172.21.15.28:0/1342129497 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:35.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.797+0000 7fe1b7400700 1 -- 172.21.15.28:0/1342129497 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b810b940 con 0x7fe1b8109010 2024-04-24T07:59:35.844 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.798+0000 7fe1b7400700 1 --2- 172.21.15.28:0/1342129497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b8109010 0x7fe1b810b400 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe1a80098a0 tx=0x7fe1a8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3714596e55a6213d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:35.847 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.798+0000 7fe1b6000700 1 -- 172.21.15.28:0/1342129497 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1a8004160 con 0x7fe1b8109010 2024-04-24T07:59:35.847 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.798+0000 7fe1b6000700 1 -- 172.21.15.28:0/1342129497 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe1a8004300 con 0x7fe1b8109010 2024-04-24T07:59:35.847 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.798+0000 7fe1b6000700 1 -- 172.21.15.28:0/1342129497 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1a8011b50 con 0x7fe1b8109010 2024-04-24T07:59:35.847 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.799+0000 7fe1bce00700 1 -- 172.21.15.28:0/1342129497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b8109010 msgr2=0x7fe1b810b400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:35.847 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.799+0000 7fe1bce00700 1 --2- 172.21.15.28:0/1342129497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b8109010 0x7fe1b810b400 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe1a80098a0 tx=0x7fe1a8009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:35.847 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.799+0000 7fe1bce00700 1 -- 172.21.15.28:0/1342129497 shutdown_connections 2024-04-24T07:59:35.847 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.799+0000 7fe1bce00700 1 --2- 172.21.15.28:0/1342129497 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b8109010 0x7fe1b810b400 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:35.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.799+0000 7fe1bce00700 1 -- 172.21.15.28:0/1342129497 >> 172.21.15.28:0/1342129497 conn(0x7fe1b8102aa0 msgr2=0x7fe1b8104ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:35.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.799+0000 7fe1bce00700 1 -- 172.21.15.28:0/1342129497 shutdown_connections 2024-04-24T07:59:35.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.799+0000 7fe1bce00700 1 -- 172.21.15.28:0/1342129497 wait complete. 2024-04-24T07:59:35.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.800+0000 7fe1bce00700 1 Processor -- start 2024-04-24T07:59:35.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.801+0000 7fe1bce00700 1 -- start start 2024-04-24T07:59:35.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.801+0000 7fe1bce00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b8109010 0x7fe1b8108890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:35.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.801+0000 7fe1bce00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1b810c960 con 0x7fe1b8109010 2024-04-24T07:59:35.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.801+0000 7fe1b7400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b8109010 0x7fe1b8108890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:35.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.802+0000 7fe1b7400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b8109010 0x7fe1b8108890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60864/0 (socket says 172.21.15.28:60864) 2024-04-24T07:59:35.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.802+0000 7fe1b7400700 1 -- 172.21.15.28:0/3166488151 learned_addr learned my addr 172.21.15.28:0/3166488151 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:35.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.802+0000 7fe1b7400700 1 -- 172.21.15.28:0/3166488151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1b8108dd0 con 0x7fe1b8109010 2024-04-24T07:59:35.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.802+0000 7fe1b7400700 1 --2- 172.21.15.28:0/3166488151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b8109010 0x7fe1b8108890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8011b60 tx=0x7fe1a8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:35.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.803+0000 7fe1afe00700 1 -- 172.21.15.28:0/3166488151 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1a8003970 con 0x7fe1b8109010 2024-04-24T07:59:35.848 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.803+0000 7fe1afe00700 1 -- 172.21.15.28:0/3166488151 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe1a8003b10 con 0x7fe1b8109010 2024-04-24T07:59:35.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.803+0000 7fe1afe00700 1 -- 172.21.15.28:0/3166488151 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe1a8024610 con 0x7fe1b8109010 2024-04-24T07:59:35.851 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.803+0000 7fe1bce00700 1 -- 172.21.15.28:0/3166488151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1b8106fc0 con 0x7fe1b8109010 2024-04-24T07:59:35.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.803+0000 7fe1bce00700 1 -- 172.21.15.28:0/3166488151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1b8107500 con 0x7fe1b8109010 2024-04-24T07:59:35.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.804+0000 7fe1afe00700 1 -- 172.21.15.28:0/3166488151 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fe1a801a9f0 con 0x7fe1b8109010 2024-04-24T07:59:35.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.804+0000 7fe1bce00700 1 -- 172.21.15.28:0/3166488151 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1b8064150 con 0x7fe1b8109010 2024-04-24T07:59:35.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.804+0000 7fe1afe00700 1 --2- 172.21.15.28:0/3166488151 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fe19c039600 0x7fe19c03bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:35.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.804+0000 7fe1afe00700 1 -- 172.21.15.28:0/3166488151 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe1a8011e00 con 0x7fe1b8109010 2024-04-24T07:59:35.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.805+0000 7fe1b6a00700 1 --2- 172.21.15.28:0/3166488151 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fe19c039600 0x7fe19c03bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:35.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.806+0000 7fe1b6a00700 1 --2- 172.21.15.28:0/3166488151 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fe19c039600 0x7fe19c03bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4006fd0 tx=0x7fe1a4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:35.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:34.811+0000 7fe1afe00700 1 -- 172.21.15.28:0/3166488151 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe1a800b3e0 con 0x7fe1b8109010 2024-04-24T07:59:35.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:35.007+0000 7fe1afe00700 1 -- 172.21.15.28:0/3166488151 <== mon.0 v2:172.21.15.28:3300/0 7 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1a802cd80 con 0x7fe1b8109010 2024-04-24T07:59:35.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:35.064+0000 7fe1bce00700 1 -- 172.21.15.28:0/3166488151 --> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fe190000d90 con 0x7fe19c039600 2024-04-24T07:59:35.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:35.071+0000 7fe1afe00700 1 -- 172.21.15.28:0/3166488151 <== mgr.14120 v2:172.21.15.28:6800/3308597107 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fe190000d90 con 0x7fe19c039600 2024-04-24T07:59:35.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:35.074+0000 7fe1bce00700 1 -- 172.21.15.28:0/3166488151 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fe19c039600 msgr2=0x7fe19c03bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:35.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:35.074+0000 7fe1bce00700 1 --2- 172.21.15.28:0/3166488151 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fe19c039600 0x7fe19c03bac0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe1a4006fd0 tx=0x7fe1a4006e40 comp rx=0 tx=0).stop 2024-04-24T07:59:35.852 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:35.074+0000 7fe1bce00700 1 -- 172.21.15.28:0/3166488151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b8109010 msgr2=0x7fe1b8108890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:35.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:35.074+0000 7fe1bce00700 1 --2- 172.21.15.28:0/3166488151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b8109010 0x7fe1b8108890 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fe1a8011b60 tx=0x7fe1a8011c40 comp rx=0 tx=0).stop 2024-04-24T07:59:35.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:35.074+0000 7fe1bce00700 1 -- 172.21.15.28:0/3166488151 shutdown_connections 2024-04-24T07:59:35.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:35.074+0000 7fe1bce00700 1 --2- 172.21.15.28:0/3166488151 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fe19c039600 0x7fe19c03bac0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:35.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:35.074+0000 7fe1bce00700 1 --2- 172.21.15.28:0/3166488151 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1b8109010 0x7fe1b8108890 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:35.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:35.074+0000 7fe1bce00700 1 -- 172.21.15.28:0/3166488151 >> 172.21.15.28:0/3166488151 conn(0x7fe1b8102aa0 msgr2=0x7fe1b81035c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:35.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:35.074+0000 7fe1bce00700 1 -- 172.21.15.28:0/3166488151 shutdown_connections 2024-04-24T07:59:35.853 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:35.074+0000 7fe1bce00700 1 -- 172.21.15.28:0/3166488151 wait complete. 2024-04-24T07:59:36.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:36 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' 2024-04-24T07:59:36.181 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:36 smithi028 ceph-mon[27786]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T07:59:36.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:36 smithi028 ceph-mon[27786]: Saving service mgr spec with placement count:2 2024-04-24T07:59:36.182 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:36 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' 2024-04-24T07:59:37.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:37 smithi028 ceph-mon[27786]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T07:59:37.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:37 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' 2024-04-24T07:59:37.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:37 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2743857974' entity='client.admin' 2024-04-24T07:59:37.388 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.457+0000 7fbd02c00700 1 Processor -- start 2024-04-24T07:59:37.388 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.457+0000 7fbd02c00700 1 -- start start 2024-04-24T07:59:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.458+0000 7fbd02c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc07de70 0x7fbcfc07c320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.458+0000 7fbd02c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc07c8f0 con 0x7fbcfc07de70 2024-04-24T07:59:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.458+0000 7fbcfbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc07de70 0x7fbcfc07c320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.458+0000 7fbcfbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc07de70 0x7fbcfc07c320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60876/0 (socket says 172.21.15.28:60876) 2024-04-24T07:59:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.458+0000 7fbcfbe00700 1 -- 172.21.15.28:0/4199995380 learned_addr learned my addr 172.21.15.28:0/4199995380 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.459+0000 7fbcfbe00700 1 -- 172.21.15.28:0/4199995380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcfc07ca70 con 0x7fbcfc07de70 2024-04-24T07:59:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.459+0000 7fbcfbe00700 1 --2- 172.21.15.28:0/4199995380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc07de70 0x7fbcfc07c320 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0098a0 tx=0x7fbcec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5338d157a18da5c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.459+0000 7fbcfaa00700 1 -- 172.21.15.28:0/4199995380 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcec004160 con 0x7fbcfc07de70 2024-04-24T07:59:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.460+0000 7fbcfaa00700 1 -- 172.21.15.28:0/4199995380 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcec004300 con 0x7fbcfc07de70 2024-04-24T07:59:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.460+0000 7fbcfaa00700 1 -- 172.21.15.28:0/4199995380 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcec011b20 con 0x7fbcfc07de70 2024-04-24T07:59:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.460+0000 7fbd02c00700 1 -- 172.21.15.28:0/4199995380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc07de70 msgr2=0x7fbcfc07c320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.460+0000 7fbd02c00700 1 --2- 172.21.15.28:0/4199995380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc07de70 0x7fbcfc07c320 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fbcec0098a0 tx=0x7fbcec009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:37.389 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.461+0000 7fbd02c00700 1 -- 172.21.15.28:0/4199995380 shutdown_connections 2024-04-24T07:59:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.461+0000 7fbd02c00700 1 --2- 172.21.15.28:0/4199995380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc07de70 0x7fbcfc07c320 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.461+0000 7fbd02c00700 1 -- 172.21.15.28:0/4199995380 >> 172.21.15.28:0/4199995380 conn(0x7fbcfc103d70 msgr2=0x7fbcfc106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.461+0000 7fbd02c00700 1 -- 172.21.15.28:0/4199995380 shutdown_connections 2024-04-24T07:59:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.461+0000 7fbd02c00700 1 -- 172.21.15.28:0/4199995380 wait complete. 2024-04-24T07:59:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.462+0000 7fbd02c00700 1 Processor -- start 2024-04-24T07:59:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.462+0000 7fbd02c00700 1 -- start start 2024-04-24T07:59:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.462+0000 7fbd02c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc07de70 0x7fbcfc19e810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.462+0000 7fbd02c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbcfc10ed10 con 0x7fbcfc07de70 2024-04-24T07:59:37.390 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.463+0000 7fbcfbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc07de70 0x7fbcfc19e810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.463+0000 7fbcfbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc07de70 0x7fbcfc19e810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60882/0 (socket says 172.21.15.28:60882) 2024-04-24T07:59:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.463+0000 7fbcfbe00700 1 -- 172.21.15.28:0/2743857974 learned_addr learned my addr 172.21.15.28:0/2743857974 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.463+0000 7fbcfbe00700 1 -- 172.21.15.28:0/2743857974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbcfc19ed50 con 0x7fbcfc07de70 2024-04-24T07:59:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.463+0000 7fbcfbe00700 1 --2- 172.21.15.28:0/2743857974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc07de70 0x7fbcfc19e810 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbcec011b60 tx=0x7fbcec011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.464+0000 7fbcf8c00700 1 -- 172.21.15.28:0/2743857974 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcec003b60 con 0x7fbcfc07de70 2024-04-24T07:59:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.464+0000 7fbcf8c00700 1 -- 172.21.15.28:0/2743857974 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcec01a910 con 0x7fbcfc07de70 2024-04-24T07:59:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.464+0000 7fbd02c00700 1 -- 172.21.15.28:0/2743857974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbcfc19efe0 con 0x7fbcfc07de70 2024-04-24T07:59:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.464+0000 7fbcf8c00700 1 -- 172.21.15.28:0/2743857974 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fbcec024930 con 0x7fbcfc07de70 2024-04-24T07:59:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.464+0000 7fbd02c00700 1 -- 172.21.15.28:0/2743857974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbcfc19f440 con 0x7fbcfc07de70 2024-04-24T07:59:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.465+0000 7fbd02c00700 1 -- 172.21.15.28:0/2743857974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbcfc064150 con 0x7fbcfc07de70 2024-04-24T07:59:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.465+0000 7fbcf8c00700 1 -- 172.21.15.28:0/2743857974 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7fbcec022020 con 0x7fbcfc07de70 2024-04-24T07:59:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.466+0000 7fbcf8c00700 1 --2- 172.21.15.28:0/2743857974 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fbce0039600 0x7fbce003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:37.391 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.466+0000 7fbcf8c00700 1 -- 172.21.15.28:0/2743857974 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fbcec04d8d0 con 0x7fbcfc07de70 2024-04-24T07:59:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.466+0000 7fbcfb400700 1 --2- 172.21.15.28:0/2743857974 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fbce0039600 0x7fbce003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.467+0000 7fbcfb400700 1 --2- 172.21.15.28:0/2743857974 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fbce0039600 0x7fbce003bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbce8006fd0 tx=0x7fbce8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.471+0000 7fbcf8c00700 1 -- 172.21.15.28:0/2743857974 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbcec01a4e0 con 0x7fbcfc07de70 2024-04-24T07:59:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.718+0000 7fbd02c00700 1 -- 172.21.15.28:0/2743857974 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7fbcfc1a1cd0 con 0x7fbcfc07de70 2024-04-24T07:59:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.722+0000 7fbcf8c00700 1 -- 172.21.15.28:0/2743857974 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7fbcec02d3f0 con 0x7fbcfc07de70 2024-04-24T07:59:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.725+0000 7fbd02c00700 1 -- 172.21.15.28:0/2743857974 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fbce0039600 msgr2=0x7fbce003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.725+0000 7fbd02c00700 1 --2- 172.21.15.28:0/2743857974 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fbce0039600 0x7fbce003bac0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fbce8006fd0 tx=0x7fbce8006e40 comp rx=0 tx=0).stop 2024-04-24T07:59:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.725+0000 7fbd02c00700 1 -- 172.21.15.28:0/2743857974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc07de70 msgr2=0x7fbcfc19e810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.725+0000 7fbd02c00700 1 --2- 172.21.15.28:0/2743857974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc07de70 0x7fbcfc19e810 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7fbcec011b60 tx=0x7fbcec011c40 comp rx=0 tx=0).stop 2024-04-24T07:59:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.725+0000 7fbd02c00700 1 -- 172.21.15.28:0/2743857974 shutdown_connections 2024-04-24T07:59:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.725+0000 7fbd02c00700 1 --2- 172.21.15.28:0/2743857974 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fbce0039600 0x7fbce003bac0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.725+0000 7fbd02c00700 1 --2- 172.21.15.28:0/2743857974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbcfc07de70 0x7fbcfc19e810 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.725+0000 7fbd02c00700 1 -- 172.21.15.28:0/2743857974 >> 172.21.15.28:0/2743857974 conn(0x7fbcfc103d70 msgr2=0x7fbcfc104810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.726+0000 7fbd02c00700 1 -- 172.21.15.28:0/2743857974 shutdown_connections 2024-04-24T07:59:37.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:36.726+0000 7fbd02c00700 1 -- 172.21.15.28:0/2743857974 wait complete. 2024-04-24T07:59:38.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.004+0000 7f4a9fe00700 1 Processor -- start 2024-04-24T07:59:38.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.005+0000 7f4a9fe00700 1 -- start start 2024-04-24T07:59:38.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.005+0000 7f4a9fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010a760 0x7f4aa010ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:38.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.005+0000 7f4a9fe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa010b150 con 0x7f4aa010a760 2024-04-24T07:59:38.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.005+0000 7f4a9ea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010a760 0x7f4aa010ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:38.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.006+0000 7f4a9ea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010a760 0x7f4aa010ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60884/0 (socket says 172.21.15.28:60884) 2024-04-24T07:59:38.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.006+0000 7f4a9ea00700 1 -- 172.21.15.28:0/3868041536 learned_addr learned my addr 172.21.15.28:0/3868041536 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:38.959 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.006+0000 7f4a9ea00700 1 -- 172.21.15.28:0/3868041536 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa010b910 con 0x7f4aa010a760 2024-04-24T07:59:38.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.006+0000 7f4a9ea00700 1 --2- 172.21.15.28:0/3868041536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010a760 0x7f4aa010ab80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4a900098a0 tx=0x7f4a90009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1899412488a2dcef server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:38.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.007+0000 7f4a9d600700 1 -- 172.21.15.28:0/3868041536 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a90004160 con 0x7f4aa010a760 2024-04-24T07:59:38.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.007+0000 7f4a9d600700 1 -- 172.21.15.28:0/3868041536 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a90004300 con 0x7f4aa010a760 2024-04-24T07:59:38.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.007+0000 7f4a9d600700 1 -- 172.21.15.28:0/3868041536 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a90011b20 con 0x7f4aa010a760 2024-04-24T07:59:38.960 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.007+0000 7f4a9fe00700 1 -- 172.21.15.28:0/3868041536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010a760 msgr2=0x7f4aa010ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:38.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.007+0000 7f4a9fe00700 1 --2- 172.21.15.28:0/3868041536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010a760 0x7f4aa010ab80 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f4a900098a0 tx=0x7f4a90009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:38.964 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.008+0000 7f4a9fe00700 1 -- 172.21.15.28:0/3868041536 shutdown_connections 2024-04-24T07:59:38.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.008+0000 7f4a9fe00700 1 --2- 172.21.15.28:0/3868041536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010a760 0x7f4aa010ab80 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:38.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.008+0000 7f4a9fe00700 1 -- 172.21.15.28:0/3868041536 >> 172.21.15.28:0/3868041536 conn(0x7f4aa007de30 msgr2=0x7f4aa01084f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:38.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.008+0000 7f4a9fe00700 1 -- 172.21.15.28:0/3868041536 shutdown_connections 2024-04-24T07:59:38.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.008+0000 7f4a9fe00700 1 -- 172.21.15.28:0/3868041536 wait complete. 2024-04-24T07:59:38.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.009+0000 7f4a9fe00700 1 Processor -- start 2024-04-24T07:59:38.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.009+0000 7f4a9fe00700 1 -- start start 2024-04-24T07:59:38.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.010+0000 7f4a9fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010a760 0x7f4aa019e620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:38.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.010+0000 7f4a9fe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4aa010ecc0 con 0x7f4aa010a760 2024-04-24T07:59:38.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.010+0000 7f4a9ea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010a760 0x7f4aa019e620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:38.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.010+0000 7f4a9ea00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010a760 0x7f4aa019e620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:60898/0 (socket says 172.21.15.28:60898) 2024-04-24T07:59:38.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.010+0000 7f4a9ea00700 1 -- 172.21.15.28:0/3734412610 learned_addr learned my addr 172.21.15.28:0/3734412610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:38.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.011+0000 7f4a9ea00700 1 -- 172.21.15.28:0/3734412610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4aa019eb60 con 0x7f4aa010a760 2024-04-24T07:59:38.965 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.011+0000 7f4a9ea00700 1 --2- 172.21.15.28:0/3734412610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010a760 0x7f4aa019e620 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4a900117e0 tx=0x7f4a90011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:38.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.011+0000 7f4a97400700 1 -- 172.21.15.28:0/3734412610 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a90003bf0 con 0x7f4aa010a760 2024-04-24T07:59:38.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.011+0000 7f4a97400700 1 -- 172.21.15.28:0/3734412610 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a9001a910 con 0x7f4aa010a760 2024-04-24T07:59:38.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.011+0000 7f4a97400700 1 -- 172.21.15.28:0/3734412610 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4a90024970 con 0x7f4aa010a760 2024-04-24T07:59:38.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.011+0000 7f4a9fe00700 1 -- 172.21.15.28:0/3734412610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4aa019edf0 con 0x7f4aa010a760 2024-04-24T07:59:38.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.012+0000 7f4a9fe00700 1 -- 172.21.15.28:0/3734412610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4aa019f250 con 0x7f4aa010a760 2024-04-24T07:59:38.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.013+0000 7f4a97400700 1 -- 172.21.15.28:0/3734412610 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f4a90022020 con 0x7f4aa010a760 2024-04-24T07:59:38.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.013+0000 7f4a97400700 1 --2- 172.21.15.28:0/3734412610 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f4a84039600 0x7f4a8403bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:38.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.013+0000 7f4a9fe00700 1 -- 172.21.15.28:0/3734412610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4aa0064150 con 0x7f4aa010a760 2024-04-24T07:59:38.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.013+0000 7f4a97400700 1 -- 172.21.15.28:0/3734412610 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f4a9004db50 con 0x7f4aa010a760 2024-04-24T07:59:38.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.014+0000 7f4a9e000700 1 --2- 172.21.15.28:0/3734412610 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f4a84039600 0x7f4a8403bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:38.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.014+0000 7f4a9e000700 1 --2- 172.21.15.28:0/3734412610 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f4a84039600 0x7f4a8403bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c006fd0 tx=0x7f4a8c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:38.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.019+0000 7f4a97400700 1 -- 172.21.15.28:0/3734412610 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4a9004aba0 con 0x7f4aa010a760 2024-04-24T07:59:38.966 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.248+0000 7f4a9fe00700 1 -- 172.21.15.28:0/3734412610 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f4aa01a1c00 con 0x7f4aa010a760 2024-04-24T07:59:38.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.251+0000 7f4a97400700 1 -- 172.21.15.28:0/3734412610 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f4a9004d130 con 0x7f4aa010a760 2024-04-24T07:59:38.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.255+0000 7f4a9fe00700 1 -- 172.21.15.28:0/3734412610 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f4a84039600 msgr2=0x7f4a8403bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:38.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.255+0000 7f4a9fe00700 1 --2- 172.21.15.28:0/3734412610 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f4a84039600 0x7f4a8403bac0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f4a8c006fd0 tx=0x7f4a8c006e40 comp rx=0 tx=0).stop 2024-04-24T07:59:38.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.255+0000 7f4a9fe00700 1 -- 172.21.15.28:0/3734412610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010a760 msgr2=0x7f4aa019e620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:38.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.255+0000 7f4a9fe00700 1 --2- 172.21.15.28:0/3734412610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010a760 0x7f4aa019e620 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4a900117e0 tx=0x7f4a90011ba0 comp rx=0 tx=0).stop 2024-04-24T07:59:38.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.255+0000 7f4a9fe00700 1 -- 172.21.15.28:0/3734412610 shutdown_connections 2024-04-24T07:59:38.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.255+0000 7f4a9fe00700 1 --2- 172.21.15.28:0/3734412610 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f4a84039600 0x7f4a8403bac0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:38.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.255+0000 7f4a9fe00700 1 --2- 172.21.15.28:0/3734412610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4aa010a760 0x7f4aa019e620 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:38.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.255+0000 7f4a9fe00700 1 -- 172.21.15.28:0/3734412610 >> 172.21.15.28:0/3734412610 conn(0x7f4aa007de30 msgr2=0x7f4aa0107c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:38.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.256+0000 7f4a9fe00700 1 -- 172.21.15.28:0/3734412610 shutdown_connections 2024-04-24T07:59:38.967 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:38.256+0000 7f4a9fe00700 1 -- 172.21.15.28:0/3734412610 wait complete. 2024-04-24T07:59:38.967 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling the dashboard module... 2024-04-24T07:59:39.528 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:39 smithi028 ceph-mon[27786]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T07:59:39.528 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:39 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3734412610' entity='client.admin' 2024-04-24T07:59:40.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:40 smithi028 ceph-mon[27786]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T07:59:40.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:40 smithi028 ceph-mon[27786]: from='mgr.14120 172.21.15.28:0/356975982' entity='mgr.x' 2024-04-24T07:59:40.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:40 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3660590641' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-04-24T07:59:40.943 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:40 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: ignoring --setuser ceph since I am not root 2024-04-24T07:59:40.943 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:40 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: ignoring --setgroup ceph since I am not root 2024-04-24T07:59:40.943 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:40 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:40.942+0000 7f398035f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T07:59:41.392 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.827+0000 7f16aec00700 1 Processor -- start 2024-04-24T07:59:41.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.828+0000 7f16aec00700 1 -- start start 2024-04-24T07:59:41.393 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.828+0000 7f16aec00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a8108520 0x7f16a8108940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:41.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.828+0000 7f16aec00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16a8108f10 con 0x7f16a8108520 2024-04-24T07:59:41.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.828+0000 7f16ad800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a8108520 0x7f16a8108940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:41.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.829+0000 7f16ad800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a8108520 0x7f16a8108940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52328/0 (socket says 172.21.15.28:52328) 2024-04-24T07:59:41.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.829+0000 7f16ad800700 1 -- 172.21.15.28:0/890716589 learned_addr learned my addr 172.21.15.28:0/890716589 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:41.394 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.829+0000 7f16ad800700 1 -- 172.21.15.28:0/890716589 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a81096d0 con 0x7f16a8108520 2024-04-24T07:59:41.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.830+0000 7f16ad800700 1 --2- 172.21.15.28:0/890716589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a8108520 0x7f16a8108940 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f169c0098a0 tx=0x7f169c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f3b2489eacbe1a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:41.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.830+0000 7f16a7e00700 1 -- 172.21.15.28:0/890716589 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f169c004160 con 0x7f16a8108520 2024-04-24T07:59:41.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.830+0000 7f16a7e00700 1 -- 172.21.15.28:0/890716589 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f169c004300 con 0x7f16a8108520 2024-04-24T07:59:41.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.830+0000 7f16a7e00700 1 -- 172.21.15.28:0/890716589 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f169c011b20 con 0x7f16a8108520 2024-04-24T07:59:41.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.831+0000 7f16aec00700 1 -- 172.21.15.28:0/890716589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a8108520 msgr2=0x7f16a8108940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:41.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.831+0000 7f16aec00700 1 --2- 172.21.15.28:0/890716589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a8108520 0x7f16a8108940 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f169c0098a0 tx=0x7f169c009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:41.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.831+0000 7f16aec00700 1 -- 172.21.15.28:0/890716589 shutdown_connections 2024-04-24T07:59:41.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.831+0000 7f16aec00700 1 --2- 172.21.15.28:0/890716589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a8108520 0x7f16a8108940 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:41.396 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.831+0000 7f16aec00700 1 -- 172.21.15.28:0/890716589 >> 172.21.15.28:0/890716589 conn(0x7f16a8103ac0 msgr2=0x7f16a8105f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:41.397 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.831+0000 7f16aec00700 1 -- 172.21.15.28:0/890716589 shutdown_connections 2024-04-24T07:59:41.397 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.831+0000 7f16aec00700 1 -- 172.21.15.28:0/890716589 wait complete. 2024-04-24T07:59:41.397 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.833+0000 7f16aec00700 1 Processor -- start 2024-04-24T07:59:41.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.833+0000 7f16aec00700 1 -- start start 2024-04-24T07:59:41.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.833+0000 7f16aec00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a8108520 0x7f16a819c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:41.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.833+0000 7f16aec00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16a810ca80 con 0x7f16a8108520 2024-04-24T07:59:41.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.834+0000 7f16ad800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a8108520 0x7f16a819c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:41.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.834+0000 7f16ad800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a8108520 0x7f16a819c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52340/0 (socket says 172.21.15.28:52340) 2024-04-24T07:59:41.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.834+0000 7f16ad800700 1 -- 172.21.15.28:0/3660590641 learned_addr learned my addr 172.21.15.28:0/3660590641 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:41.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.834+0000 7f16ad800700 1 -- 172.21.15.28:0/3660590641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a819c940 con 0x7f16a8108520 2024-04-24T07:59:41.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.835+0000 7f16ad800700 1 --2- 172.21.15.28:0/3660590641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a8108520 0x7f16a819c400 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f169c0117e0 tx=0x7f169c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:41.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.835+0000 7f16a6000700 1 -- 172.21.15.28:0/3660590641 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f169c003bf0 con 0x7f16a8108520 2024-04-24T07:59:41.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.835+0000 7f16a6000700 1 -- 172.21.15.28:0/3660590641 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f169c01a910 con 0x7f16a8108520 2024-04-24T07:59:41.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.835+0000 7f16a6000700 1 -- 172.21.15.28:0/3660590641 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f169c024970 con 0x7f16a8108520 2024-04-24T07:59:41.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.835+0000 7f16aec00700 1 -- 172.21.15.28:0/3660590641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a819cbd0 con 0x7f16a8108520 2024-04-24T07:59:41.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.836+0000 7f16aec00700 1 -- 172.21.15.28:0/3660590641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a819d030 con 0x7f16a8108520 2024-04-24T07:59:41.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.837+0000 7f16aec00700 1 -- 172.21.15.28:0/3660590641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16a8064150 con 0x7f16a8108520 2024-04-24T07:59:41.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.838+0000 7f16a6000700 1 -- 172.21.15.28:0/3660590641 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 8) ==== 46558+0+0 (secure 0 0 0) 0x7f169c022020 con 0x7f16a8108520 2024-04-24T07:59:41.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.838+0000 7f16a6000700 1 --2- 172.21.15.28:0/3660590641 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f1690039600 0x7f169003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:41.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.838+0000 7f16a6000700 1 -- 172.21.15.28:0/3660590641 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f169c04db30 con 0x7f16a8108520 2024-04-24T07:59:41.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.838+0000 7f16ace00700 1 --2- 172.21.15.28:0/3660590641 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f1690039600 0x7f169003bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:41.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.839+0000 7f16ace00700 1 --2- 172.21.15.28:0/3660590641 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f1690039600 0x7f169003bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1694006fd0 tx=0x7f1694006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:41.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:39.844+0000 7f16a6000700 1 -- 172.21.15.28:0/3660590641 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f169c0212c0 con 0x7f16a8108520 2024-04-24T07:59:41.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:40.145+0000 7f16aec00700 1 -- 172.21.15.28:0/3660590641 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7f16a819fe10 con 0x7f16a8108520 2024-04-24T07:59:41.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:40.586+0000 7f16a6000700 1 -- 172.21.15.28:0/3660590641 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7f169c04d480 con 0x7f16a8108520 2024-04-24T07:59:41.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:40.590+0000 7f16a6000700 1 -- 172.21.15.28:0/3660590641 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f169c02d3f0 con 0x7f16a8108520 2024-04-24T07:59:41.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:40.591+0000 7f16aec00700 1 -- 172.21.15.28:0/3660590641 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f1690039600 msgr2=0x7f169003bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:41.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:40.591+0000 7f16aec00700 1 --2- 172.21.15.28:0/3660590641 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f1690039600 0x7f169003bac0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f1694006fd0 tx=0x7f1694006e40 comp rx=0 tx=0).stop 2024-04-24T07:59:41.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:40.591+0000 7f16aec00700 1 -- 172.21.15.28:0/3660590641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a8108520 msgr2=0x7f16a819c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:41.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:40.591+0000 7f16aec00700 1 --2- 172.21.15.28:0/3660590641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a8108520 0x7f16a819c400 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f169c0117e0 tx=0x7f169c011ba0 comp rx=0 tx=0).stop 2024-04-24T07:59:41.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:40.592+0000 7f16aec00700 1 -- 172.21.15.28:0/3660590641 shutdown_connections 2024-04-24T07:59:41.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:40.592+0000 7f16aec00700 1 --2- 172.21.15.28:0/3660590641 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f1690039600 0x7f169003bac0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:41.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:40.592+0000 7f16aec00700 1 --2- 172.21.15.28:0/3660590641 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a8108520 0x7f16a819c400 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:41.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:40.592+0000 7f16aec00700 1 -- 172.21.15.28:0/3660590641 >> 172.21.15.28:0/3660590641 conn(0x7f16a8103ac0 msgr2=0x7f16a8104560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:41.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:40.592+0000 7f16aec00700 1 -- 172.21.15.28:0/3660590641 shutdown_connections 2024-04-24T07:59:41.403 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:40.592+0000 7f16aec00700 1 -- 172.21.15.28:0/3660590641 wait complete. 2024-04-24T07:59:41.539 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:41 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:41.202+0000 7f398035f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T07:59:41.539 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:41 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:41.340+0000 7f398035f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T07:59:41.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:41 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3660590641' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-04-24T07:59:41.882 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:41 smithi028 ceph-mon[27786]: mgrmap e9: x(active, since 25s) 2024-04-24T07:59:41.882 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:41 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:41.767+0000 7f398035f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T07:59:42.247 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:41 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:41.884+0000 7f398035f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T07:59:43.000 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:42 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1192733144' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-04-24T07:59:43.014 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-24T07:59:43.014 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-04-24T07:59:43.015 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "available": true, 2024-04-24T07:59:43.015 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-04-24T07:59:43.015 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-04-24T07:59:43.015 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-24T07:59:43.015 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.037+0000 7f3d6f400700 1 Processor -- start 2024-04-24T07:59:43.015 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.037+0000 7f3d6f400700 1 -- start start 2024-04-24T07:59:43.015 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.037+0000 7f3d6f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d68106f60 0x7f3d68107380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:43.015 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.037+0000 7f3d6f400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d6807bed0 con 0x7f3d68106f60 2024-04-24T07:59:43.015 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.038+0000 7f3d6ca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d68106f60 0x7f3d68107380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:43.015 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.038+0000 7f3d6ca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d68106f60 0x7f3d68107380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52366/0 (socket says 172.21.15.28:52366) 2024-04-24T07:59:43.015 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.038+0000 7f3d6ca00700 1 -- 172.21.15.28:0/1721047158 learned_addr learned my addr 172.21.15.28:0/1721047158 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:43.015 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.039+0000 7f3d6ca00700 1 -- 172.21.15.28:0/1721047158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d681078c0 con 0x7f3d68106f60 2024-04-24T07:59:43.016 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.039+0000 7f3d6ca00700 1 --2- 172.21.15.28:0/1721047158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d68106f60 0x7f3d68107380 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c0098a0 tx=0x7f3d5c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=171579e629968976 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:43.016 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.039+0000 7f3d67400700 1 -- 172.21.15.28:0/1721047158 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d5c004160 con 0x7f3d68106f60 2024-04-24T07:59:43.016 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.039+0000 7f3d67400700 1 -- 172.21.15.28:0/1721047158 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d5c004300 con 0x7f3d68106f60 2024-04-24T07:59:43.016 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.039+0000 7f3d67400700 1 -- 172.21.15.28:0/1721047158 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d5c011b20 con 0x7f3d68106f60 2024-04-24T07:59:43.016 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.040+0000 7f3d6f400700 1 -- 172.21.15.28:0/1721047158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d68106f60 msgr2=0x7f3d68107380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:43.016 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.040+0000 7f3d6f400700 1 --2- 172.21.15.28:0/1721047158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d68106f60 0x7f3d68107380 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c0098a0 tx=0x7f3d5c009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:43.016 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.040+0000 7f3d6f400700 1 -- 172.21.15.28:0/1721047158 shutdown_connections 2024-04-24T07:59:43.016 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.040+0000 7f3d6f400700 1 --2- 172.21.15.28:0/1721047158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d68106f60 0x7f3d68107380 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:43.016 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.040+0000 7f3d6f400700 1 -- 172.21.15.28:0/1721047158 >> 172.21.15.28:0/1721047158 conn(0x7f3d68102c00 msgr2=0x7f3d68105020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:43.016 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.040+0000 7f3d6f400700 1 -- 172.21.15.28:0/1721047158 shutdown_connections 2024-04-24T07:59:43.016 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.040+0000 7f3d6f400700 1 -- 172.21.15.28:0/1721047158 wait complete. 2024-04-24T07:59:43.016 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.041+0000 7f3d6f400700 1 Processor -- start 2024-04-24T07:59:43.016 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.042+0000 7f3d6f400700 1 -- start start 2024-04-24T07:59:43.017 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.042+0000 7f3d6f400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d68106f60 0x7f3d6819e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:43.017 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.042+0000 7f3d6f400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d68108990 con 0x7f3d68106f60 2024-04-24T07:59:43.017 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.042+0000 7f3d6ca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d68106f60 0x7f3d6819e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:43.017 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.042+0000 7f3d6ca00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d68106f60 0x7f3d6819e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52378/0 (socket says 172.21.15.28:52378) 2024-04-24T07:59:43.017 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.042+0000 7f3d6ca00700 1 -- 172.21.15.28:0/1192733144 learned_addr learned my addr 172.21.15.28:0/1192733144 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:43.017 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.043+0000 7f3d6ca00700 1 -- 172.21.15.28:0/1192733144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d6819ee00 con 0x7f3d68106f60 2024-04-24T07:59:43.017 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.043+0000 7f3d6ca00700 1 --2- 172.21.15.28:0/1192733144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d68106f60 0x7f3d6819e8c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c011b60 tx=0x7f3d5c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:43.017 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.043+0000 7f3d65600700 1 -- 172.21.15.28:0/1192733144 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d5c003b60 con 0x7f3d68106f60 2024-04-24T07:59:43.017 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.043+0000 7f3d65600700 1 -- 172.21.15.28:0/1192733144 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d5c01a910 con 0x7f3d68106f60 2024-04-24T07:59:43.017 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.043+0000 7f3d65600700 1 -- 172.21.15.28:0/1192733144 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3d5c024930 con 0x7f3d68106f60 2024-04-24T07:59:43.017 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.043+0000 7f3d6f400700 1 -- 172.21.15.28:0/1192733144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d6819f090 con 0x7f3d68106f60 2024-04-24T07:59:43.017 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.044+0000 7f3d6f400700 1 -- 172.21.15.28:0/1192733144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d6819f4f0 con 0x7f3d68106f60 2024-04-24T07:59:43.017 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.045+0000 7f3d65600700 1 -- 172.21.15.28:0/1192733144 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7f3d5c022020 con 0x7f3d68106f60 2024-04-24T07:59:43.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.045+0000 7f3d6f400700 1 -- 172.21.15.28:0/1192733144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d68064150 con 0x7f3d68106f60 2024-04-24T07:59:43.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.045+0000 7f3d65600700 1 --2- 172.21.15.28:0/1192733144 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f3d50039600 0x7f3d5003bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:43.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.046+0000 7f3d65600700 1 -- 172.21.15.28:0/1192733144 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3d5c04da80 con 0x7f3d68106f60 2024-04-24T07:59:43.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.046+0000 7f3d67e00700 1 -- 172.21.15.28:0/1192733144 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f3d50039600 msgr2=0x7f3d5003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3308597107 2024-04-24T07:59:43.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.046+0000 7f3d67e00700 1 --2- 172.21.15.28:0/1192733144 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f3d50039600 0x7f3d5003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T07:59:43.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.051+0000 7f3d65600700 1 -- 172.21.15.28:0/1192733144 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3d5c04fe20 con 0x7f3d68106f60 2024-04-24T07:59:43.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.247+0000 7f3d67e00700 1 -- 172.21.15.28:0/1192733144 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f3d50039600 msgr2=0x7f3d5003bac0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3308597107 2024-04-24T07:59:43.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.247+0000 7f3d67e00700 1 --2- 172.21.15.28:0/1192733144 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f3d50039600 0x7f3d5003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T07:59:43.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.339+0000 7f3d6f400700 1 -- 172.21.15.28:0/1192733144 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f3d68063700 con 0x7f3d68106f60 2024-04-24T07:59:43.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.339+0000 7f3d65600700 1 -- 172.21.15.28:0/1192733144 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f3d5c04d3d0 con 0x7f3d68106f60 2024-04-24T07:59:43.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.343+0000 7f3d6f400700 1 -- 172.21.15.28:0/1192733144 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f3d50039600 msgr2=0x7f3d5003bac0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T07:59:43.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.343+0000 7f3d6f400700 1 --2- 172.21.15.28:0/1192733144 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f3d50039600 0x7f3d5003bac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:43.018 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.343+0000 7f3d6f400700 1 -- 172.21.15.28:0/1192733144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d68106f60 msgr2=0x7f3d6819e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:43.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.343+0000 7f3d6f400700 1 --2- 172.21.15.28:0/1192733144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d68106f60 0x7f3d6819e8c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3d5c011b60 tx=0x7f3d5c011c40 comp rx=0 tx=0).stop 2024-04-24T07:59:43.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.343+0000 7f3d6f400700 1 -- 172.21.15.28:0/1192733144 shutdown_connections 2024-04-24T07:59:43.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.343+0000 7f3d6f400700 1 --2- 172.21.15.28:0/1192733144 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7f3d50039600 0x7f3d5003bac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:43.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.343+0000 7f3d6f400700 1 --2- 172.21.15.28:0/1192733144 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d68106f60 0x7f3d6819e8c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:43.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.343+0000 7f3d6f400700 1 -- 172.21.15.28:0/1192733144 >> 172.21.15.28:0/1192733144 conn(0x7f3d68102c00 msgr2=0x7f3d68195560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:43.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.343+0000 7f3d6f400700 1 -- 172.21.15.28:0/1192733144 shutdown_connections 2024-04-24T07:59:43.019 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:42.343+0000 7f3d6f400700 1 -- 172.21.15.28:0/1192733144 wait complete. 2024-04-24T07:59:43.019 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for the mgr to restart... 2024-04-24T07:59:43.019 INFO:teuthology.orchestra.run.smithi028.stdout:Waiting for mgr epoch 9... 2024-04-24T07:59:43.606 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:43 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:43.530+0000 7f398035f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T07:59:43.987 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:43 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:43.768+0000 7f398035f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T07:59:43.987 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:43 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:43.986+0000 7f398035f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T07:59:44.775 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:44 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:44.384+0000 7f398035f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T07:59:44.775 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:44 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:44.510+0000 7f398035f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T07:59:45.038 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:44 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:44.774+0000 7f398035f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T07:59:45.538 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:45 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:45.037+0000 7f398035f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T07:59:46.288 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:45 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:45.847+0000 7f398035f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T07:59:46.788 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:46 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:46.422+0000 7f398035f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T07:59:46.789 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:46 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:46.554+0000 7f398035f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T07:59:47.288 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:46 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:46.832+0000 7f398035f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T07:59:47.788 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:47 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:47.502+0000 7f398035f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T07:59:47.789 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:47 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:47.631+0000 7f398035f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T07:59:48.286 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:47 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:47.966+0000 7f398035f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T07:59:48.287 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:48 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:48.151+0000 7f398035f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T07:59:48.287 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:48 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:48.284+0000 7f398035f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T07:59:48.538 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:48 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:48.442+0000 7f398035f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T07:59:49.288 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:48 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:48.819+0000 7f398035f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T07:59:49.914 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:49 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:49.613+0000 7f398035f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T07:59:50.288 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 07:59:49 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T07:59:49.910+0000 7f398035f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T07:59:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:49 smithi028 ceph-mon[27786]: Active manager daemon x restarted 2024-04-24T07:59:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:49 smithi028 ceph-mon[27786]: Activating manager daemon x 2024-04-24T07:59:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:49 smithi028 ceph-mon[27786]: osdmap e3: 0 total, 0 up, 0 in 2024-04-24T07:59:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:49 smithi028 ceph-mon[27786]: mgrmap e10: x(active, starting, since 0.00502155s) 2024-04-24T07:59:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T07:59:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T07:59:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T07:59:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T07:59:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T07:59:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:49 smithi028 ceph-mon[27786]: Manager daemon x is now available 2024-04-24T07:59:51.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T07:59:51.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T07:59:51.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T07:59:51.263 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:50 smithi028 ceph-mon[27786]: [24/Apr/2024:07:59:50] ENGINE Bus STARTING 2024-04-24T07:59:51.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:50 smithi028 ceph-mon[27786]: [24/Apr/2024:07:59:50] ENGINE Serving on https://172.21.15.28:7150 2024-04-24T07:59:51.264 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:50 smithi028 ceph-mon[27786]: mgrmap e11: x(active, since 1.01017s) 2024-04-24T07:59:51.625 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout { 2024-04-24T07:59:51.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-04-24T07:59:51.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout "initialized": true 2024-04-24T07:59:51.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout } 2024-04-24T07:59:51.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.605+0000 7fd5f3e00700 1 Processor -- start 2024-04-24T07:59:51.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.606+0000 7fd5f3e00700 1 -- start start 2024-04-24T07:59:51.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.606+0000 7fd5f3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5ec1086f0 0x7fd5ec108b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:51.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.606+0000 7fd5f3e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ec1090e0 con 0x7fd5ec1086f0 2024-04-24T07:59:51.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.607+0000 7fd5f1400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5ec1086f0 0x7fd5ec108b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:51.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.607+0000 7fd5f1400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5ec1086f0 0x7fd5ec108b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52386/0 (socket says 172.21.15.28:52386) 2024-04-24T07:59:51.626 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.607+0000 7fd5f1400700 1 -- 172.21.15.28:0/1312234161 learned_addr learned my addr 172.21.15.28:0/1312234161 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:51.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.607+0000 7fd5f1400700 1 -- 172.21.15.28:0/1312234161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5ec1098a0 con 0x7fd5ec1086f0 2024-04-24T07:59:51.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.608+0000 7fd5f1400700 1 --2- 172.21.15.28:0/1312234161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5ec1086f0 0x7fd5ec108b10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd5e00098a0 tx=0x7fd5e0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20c05f75e8755d5b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:51.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.608+0000 7fd5ebe00700 1 -- 172.21.15.28:0/1312234161 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5e0004160 con 0x7fd5ec1086f0 2024-04-24T07:59:51.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.608+0000 7fd5ebe00700 1 -- 172.21.15.28:0/1312234161 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5e0004300 con 0x7fd5ec1086f0 2024-04-24T07:59:51.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.608+0000 7fd5ebe00700 1 -- 172.21.15.28:0/1312234161 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5e0011b20 con 0x7fd5ec1086f0 2024-04-24T07:59:51.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.609+0000 7fd5f3e00700 1 -- 172.21.15.28:0/1312234161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5ec1086f0 msgr2=0x7fd5ec108b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:51.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.609+0000 7fd5f3e00700 1 --2- 172.21.15.28:0/1312234161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5ec1086f0 0x7fd5ec108b10 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd5e00098a0 tx=0x7fd5e0009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:51.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.609+0000 7fd5f3e00700 1 -- 172.21.15.28:0/1312234161 shutdown_connections 2024-04-24T07:59:51.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.609+0000 7fd5f3e00700 1 --2- 172.21.15.28:0/1312234161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5ec1086f0 0x7fd5ec108b10 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:51.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.609+0000 7fd5f3e00700 1 -- 172.21.15.28:0/1312234161 >> 172.21.15.28:0/1312234161 conn(0x7fd5ec103d10 msgr2=0x7fd5ec106130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:51.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.610+0000 7fd5f3e00700 1 -- 172.21.15.28:0/1312234161 shutdown_connections 2024-04-24T07:59:51.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.610+0000 7fd5f3e00700 1 -- 172.21.15.28:0/1312234161 wait complete. 2024-04-24T07:59:51.627 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.611+0000 7fd5f3e00700 1 Processor -- start 2024-04-24T07:59:51.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.611+0000 7fd5f3e00700 1 -- start start 2024-04-24T07:59:51.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.612+0000 7fd5f3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5ec1086f0 0x7fd5ec19c670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:51.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.612+0000 7fd5f3e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5ec10cc50 con 0x7fd5ec1086f0 2024-04-24T07:59:51.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.612+0000 7fd5f1400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5ec1086f0 0x7fd5ec19c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:51.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.612+0000 7fd5f1400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5ec1086f0 0x7fd5ec19c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52394/0 (socket says 172.21.15.28:52394) 2024-04-24T07:59:51.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.612+0000 7fd5f1400700 1 -- 172.21.15.28:0/3870489518 learned_addr learned my addr 172.21.15.28:0/3870489518 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:51.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.612+0000 7fd5f1400700 1 -- 172.21.15.28:0/3870489518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5ec19cbb0 con 0x7fd5ec1086f0 2024-04-24T07:59:51.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.613+0000 7fd5f1400700 1 --2- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5ec1086f0 0x7fd5ec19c670 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0000c00 tx=0x7fd5e0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:51.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.613+0000 7fd5ea000700 1 -- 172.21.15.28:0/3870489518 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5e0003b60 con 0x7fd5ec1086f0 2024-04-24T07:59:51.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.613+0000 7fd5ea000700 1 -- 172.21.15.28:0/3870489518 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5e001a910 con 0x7fd5ec1086f0 2024-04-24T07:59:51.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.613+0000 7fd5f3e00700 1 -- 172.21.15.28:0/3870489518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5ec19ce40 con 0x7fd5ec1086f0 2024-04-24T07:59:51.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.613+0000 7fd5ea000700 1 -- 172.21.15.28:0/3870489518 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd5e0024930 con 0x7fd5ec1086f0 2024-04-24T07:59:51.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.613+0000 7fd5f3e00700 1 -- 172.21.15.28:0/3870489518 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5ec19d2a0 con 0x7fd5ec1086f0 2024-04-24T07:59:51.628 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.615+0000 7fd5ea000700 1 -- 172.21.15.28:0/3870489518 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 9) ==== 46571+0+0 (secure 0 0 0) 0x7fd5e0022020 con 0x7fd5ec1086f0 2024-04-24T07:59:51.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.615+0000 7fd5ea000700 1 --2- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fd5d4039660 0x7fd5d403bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:51.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.615+0000 7fd5ea000700 1 -- 172.21.15.28:0/3870489518 --> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd5d403c230 con 0x7fd5d4039660 2024-04-24T07:59:51.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.615+0000 7fd5ea000700 1 -- 172.21.15.28:0/3870489518 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fd5e004d970 con 0x7fd5ec1086f0 2024-04-24T07:59:51.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.615+0000 7fd5f0a00700 1 -- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fd5d4039660 msgr2=0x7fd5d403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3308597107 2024-04-24T07:59:51.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.615+0000 7fd5f0a00700 1 --2- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fd5d4039660 0x7fd5d403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T07:59:51.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.816+0000 7fd5f0a00700 1 -- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fd5d4039660 msgr2=0x7fd5d403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3308597107 2024-04-24T07:59:51.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:43.816+0000 7fd5f0a00700 1 --2- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fd5d4039660 0x7fd5d403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T07:59:51.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:44.217+0000 7fd5f0a00700 1 -- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fd5d4039660 msgr2=0x7fd5d403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3308597107 2024-04-24T07:59:51.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:44.217+0000 7fd5f0a00700 1 --2- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fd5d4039660 0x7fd5d403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-24T07:59:51.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:45.018+0000 7fd5f0a00700 1 -- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fd5d4039660 msgr2=0x7fd5d403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3308597107 2024-04-24T07:59:51.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:45.018+0000 7fd5f0a00700 1 --2- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fd5d4039660 0x7fd5d403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-04-24T07:59:51.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:46.619+0000 7fd5f0a00700 1 -- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fd5d4039660 msgr2=0x7fd5d403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3308597107 2024-04-24T07:59:51.629 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:46.619+0000 7fd5f0a00700 1 --2- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fd5d4039660 0x7fd5d403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-04-24T07:59:51.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:49.823+0000 7fd5f0a00700 1 -- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fd5d4039660 msgr2=0x7fd5d403bb20 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/3308597107 2024-04-24T07:59:51.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:49.823+0000 7fd5f0a00700 1 --2- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fd5d4039660 0x7fd5d403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-04-24T07:59:51.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:49.919+0000 7fd5ea000700 1 -- 172.21.15.28:0/3870489518 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mgrmap(e 10) ==== 46338+0+0 (secure 0 0 0) 0x7fd5e002d720 con 0x7fd5ec1086f0 2024-04-24T07:59:51.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:49.920+0000 7fd5ea000700 1 -- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fd5d4039660 msgr2=0x7fd5d403bb20 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T07:59:51.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:49.920+0000 7fd5ea000700 1 --2- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fd5d4039660 0x7fd5d403bb20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:51.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.921+0000 7fd5ea000700 1 -- 172.21.15.28:0/3870489518 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 11) ==== 46465+0+0 (secure 0 0 0) 0x7fd5e0021d00 con 0x7fd5ec1086f0 2024-04-24T07:59:51.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.922+0000 7fd5ea000700 1 --2- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd5d403cff0 0x7fd5d403f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:51.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.922+0000 7fd5ea000700 1 -- 172.21.15.28:0/3870489518 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fd5e001ac60 con 0x7fd5d403cff0 2024-04-24T07:59:51.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.922+0000 7fd5f0a00700 1 --2- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd5d403cff0 0x7fd5d403f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:51.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.924+0000 7fd5f0a00700 1 --2- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd5d403cff0 0x7fd5d403f3e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8003a10 tx=0x7fd5d8009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:51.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.925+0000 7fd5ea000700 1 -- 172.21.15.28:0/3870489518 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7fd5e001ac60 con 0x7fd5d403cff0 2024-04-24T07:59:51.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.933+0000 7fd5f3e00700 1 -- 172.21.15.28:0/3870489518 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7fd5ec064150 con 0x7fd5d403cff0 2024-04-24T07:59:51.630 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.933+0000 7fd5ea000700 1 -- 172.21.15.28:0/3870489518 <== mgr.14152 v2:172.21.15.28:6800/361125225 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7fd5ec064150 con 0x7fd5d403cff0 2024-04-24T07:59:51.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.933+0000 7fd5f3e00700 1 -- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd5d403cff0 msgr2=0x7fd5d403f3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:51.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.933+0000 7fd5f3e00700 1 --2- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd5d403cff0 0x7fd5d403f3e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8003a10 tx=0x7fd5d8009270 comp rx=0 tx=0).stop 2024-04-24T07:59:51.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.933+0000 7fd5f3e00700 1 -- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5ec1086f0 msgr2=0x7fd5ec19c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:51.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.933+0000 7fd5f3e00700 1 --2- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5ec1086f0 0x7fd5ec19c670 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd5e0000c00 tx=0x7fd5e0011c10 comp rx=0 tx=0).stop 2024-04-24T07:59:51.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.934+0000 7fd5f3e00700 1 -- 172.21.15.28:0/3870489518 shutdown_connections 2024-04-24T07:59:51.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.934+0000 7fd5f3e00700 1 --2- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd5d403cff0 0x7fd5d403f3e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:51.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.934+0000 7fd5f3e00700 1 --2- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5ec1086f0 0x7fd5ec19c670 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:51.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.934+0000 7fd5f3e00700 1 --2- 172.21.15.28:0/3870489518 >> [v2:172.21.15.28:6800/3308597107,v1:172.21.15.28:6801/3308597107] conn(0x7fd5d4039660 0x7fd5d403bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:51.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.934+0000 7fd5f3e00700 1 -- 172.21.15.28:0/3870489518 >> 172.21.15.28:0/3870489518 conn(0x7fd5ec103d10 msgr2=0x7fd5ec104870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:51.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.934+0000 7fd5f3e00700 1 -- 172.21.15.28:0/3870489518 shutdown_connections 2024-04-24T07:59:51.631 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:50.934+0000 7fd5f3e00700 1 -- 172.21.15.28:0/3870489518 wait complete. 2024-04-24T07:59:51.631 INFO:teuthology.orchestra.run.smithi028.stdout:mgr epoch 9 is available 2024-04-24T07:59:51.631 INFO:teuthology.orchestra.run.smithi028.stdout:Generating a dashboard self-signed certificate... 2024-04-24T07:59:52.170 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:51 smithi028 ceph-mon[27786]: [24/Apr/2024:07:59:50] ENGINE Serving on http://172.21.15.28:8765 2024-04-24T07:59:52.170 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:51 smithi028 ceph-mon[27786]: [24/Apr/2024:07:59:50] ENGINE Bus STARTED 2024-04-24T07:59:52.170 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:51 smithi028 ceph-mon[27786]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-04-24T07:59:52.170 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:51 smithi028 ceph-mon[27786]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-04-24T07:59:53.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:52 smithi028 ceph-mon[27786]: mgrmap e12: x(active, since 2s) 2024-04-24T07:59:53.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:52 smithi028 ceph-mon[27786]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T07:59:53.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:52 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T07:59:53.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:52 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T07:59:53.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-04-24T07:59:53.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.333+0000 7f863be00700 1 Processor -- start 2024-04-24T07:59:53.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.333+0000 7f863be00700 1 -- start start 2024-04-24T07:59:53.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.333+0000 7f863be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634108710 0x7f8634108b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:53.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.333+0000 7f863be00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8634109100 con 0x7f8634108710 2024-04-24T07:59:53.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.334+0000 7f8639400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634108710 0x7f8634108b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:53.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.334+0000 7f8639400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634108710 0x7f8634108b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44252/0 (socket says 172.21.15.28:44252) 2024-04-24T07:59:53.484 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.334+0000 7f8639400700 1 -- 172.21.15.28:0/2335640466 learned_addr learned my addr 172.21.15.28:0/2335640466 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:53.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.334+0000 7f8639400700 1 -- 172.21.15.28:0/2335640466 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86341098c0 con 0x7f8634108710 2024-04-24T07:59:53.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.335+0000 7f8639400700 1 --2- 172.21.15.28:0/2335640466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634108710 0x7f8634108b30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8628009e80 tx=0x7f862800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6525cceb94f36fc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:53.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.335+0000 7f8633e00700 1 -- 172.21.15.28:0/2335640466 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8628017070 con 0x7f8634108710 2024-04-24T07:59:53.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.335+0000 7f8633e00700 1 -- 172.21.15.28:0/2335640466 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f862800b8c0 con 0x7f8634108710 2024-04-24T07:59:53.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.335+0000 7f8633e00700 1 -- 172.21.15.28:0/2335640466 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f86280045c0 con 0x7f8634108710 2024-04-24T07:59:53.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.336+0000 7f863be00700 1 -- 172.21.15.28:0/2335640466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634108710 msgr2=0x7f8634108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:53.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.336+0000 7f863be00700 1 --2- 172.21.15.28:0/2335640466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634108710 0x7f8634108b30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8628009e80 tx=0x7f862800b330 comp rx=0 tx=0).stop 2024-04-24T07:59:53.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.336+0000 7f863be00700 1 -- 172.21.15.28:0/2335640466 shutdown_connections 2024-04-24T07:59:53.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.336+0000 7f863be00700 1 --2- 172.21.15.28:0/2335640466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634108710 0x7f8634108b30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:53.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.336+0000 7f863be00700 1 -- 172.21.15.28:0/2335640466 >> 172.21.15.28:0/2335640466 conn(0x7f8634103d70 msgr2=0x7f8634106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:53.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.336+0000 7f863be00700 1 -- 172.21.15.28:0/2335640466 shutdown_connections 2024-04-24T07:59:53.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.336+0000 7f863be00700 1 -- 172.21.15.28:0/2335640466 wait complete. 2024-04-24T07:59:53.485 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.337+0000 7f863be00700 1 Processor -- start 2024-04-24T07:59:53.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.337+0000 7f863be00700 1 -- start start 2024-04-24T07:59:53.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.338+0000 7f863be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634108710 0x7f863419e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:53.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.338+0000 7f863be00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f863410cc70 con 0x7f8634108710 2024-04-24T07:59:53.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.338+0000 7f8639400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634108710 0x7f863419e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:53.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.338+0000 7f8639400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634108710 0x7f863419e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44262/0 (socket says 172.21.15.28:44262) 2024-04-24T07:59:53.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.338+0000 7f8639400700 1 -- 172.21.15.28:0/665839379 learned_addr learned my addr 172.21.15.28:0/665839379 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:53.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.338+0000 7f8639400700 1 -- 172.21.15.28:0/665839379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f863419ee20 con 0x7f8634108710 2024-04-24T07:59:53.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.339+0000 7f8639400700 1 --2- 172.21.15.28:0/665839379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634108710 0x7f863419e8e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8628003fa0 tx=0x7f8628004080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:53.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.339+0000 7f8632000700 1 -- 172.21.15.28:0/665839379 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8628020020 con 0x7f8634108710 2024-04-24T07:59:53.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.339+0000 7f8632000700 1 -- 172.21.15.28:0/665839379 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f862800bba0 con 0x7f8634108710 2024-04-24T07:59:53.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.339+0000 7f863be00700 1 -- 172.21.15.28:0/665839379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f863419f0b0 con 0x7f8634108710 2024-04-24T07:59:53.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.339+0000 7f8632000700 1 -- 172.21.15.28:0/665839379 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f8628017040 con 0x7f8634108710 2024-04-24T07:59:53.486 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.339+0000 7f863be00700 1 -- 172.21.15.28:0/665839379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86341a1da0 con 0x7f8634108710 2024-04-24T07:59:53.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.341+0000 7f8632000700 1 -- 172.21.15.28:0/665839379 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f8628023020 con 0x7f8634108710 2024-04-24T07:59:53.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.341+0000 7f863be00700 1 -- 172.21.15.28:0/665839379 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8634064150 con 0x7f8634108710 2024-04-24T07:59:53.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.341+0000 7f8632000700 1 --2- 172.21.15.28:0/665839379 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f861c039660 0x7f861c03bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:53.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.341+0000 7f8632000700 1 -- 172.21.15.28:0/665839379 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f862800f050 con 0x7f8634108710 2024-04-24T07:59:53.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.342+0000 7f8638a00700 1 --2- 172.21.15.28:0/665839379 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f861c039660 0x7f861c03bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:53.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.342+0000 7f8638a00700 1 --2- 172.21.15.28:0/665839379 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f861c039660 0x7f861c03bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8620006fd0 tx=0x7f8620006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:53.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.347+0000 7f8632000700 1 -- 172.21.15.28:0/665839379 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f862802a020 con 0x7f8634108710 2024-04-24T07:59:53.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.594+0000 7f863be00700 1 -- 172.21.15.28:0/665839379 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7f8634104f20 con 0x7f861c039660 2024-04-24T07:59:53.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.772+0000 7f8632000700 1 -- 172.21.15.28:0/665839379 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7f8634104f20 con 0x7f861c039660 2024-04-24T07:59:53.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.775+0000 7f863be00700 1 -- 172.21.15.28:0/665839379 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f861c039660 msgr2=0x7f861c03bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:53.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.775+0000 7f863be00700 1 --2- 172.21.15.28:0/665839379 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f861c039660 0x7f861c03bb20 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f8620006fd0 tx=0x7f8620006e40 comp rx=0 tx=0).stop 2024-04-24T07:59:53.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.775+0000 7f863be00700 1 -- 172.21.15.28:0/665839379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634108710 msgr2=0x7f863419e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:53.487 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.775+0000 7f863be00700 1 --2- 172.21.15.28:0/665839379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634108710 0x7f863419e8e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8628003fa0 tx=0x7f8628004080 comp rx=0 tx=0).stop 2024-04-24T07:59:53.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.775+0000 7f863be00700 1 -- 172.21.15.28:0/665839379 shutdown_connections 2024-04-24T07:59:53.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.775+0000 7f863be00700 1 --2- 172.21.15.28:0/665839379 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f861c039660 0x7f861c03bb20 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:53.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.775+0000 7f863be00700 1 --2- 172.21.15.28:0/665839379 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8634108710 0x7f863419e8e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:53.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.775+0000 7f863be00700 1 -- 172.21.15.28:0/665839379 >> 172.21.15.28:0/665839379 conn(0x7f8634103d70 msgr2=0x7f8634104810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:53.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.775+0000 7f863be00700 1 -- 172.21.15.28:0/665839379 shutdown_connections 2024-04-24T07:59:53.488 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:52.775+0000 7f863be00700 1 -- 172.21.15.28:0/665839379 wait complete. 2024-04-24T07:59:53.488 INFO:teuthology.orchestra.run.smithi028.stdout:Creating initial admin user... 2024-04-24T07:59:55.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:54 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T07:59:55.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:54 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T07:59:55.398 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$GrUMjzDq82fLjw2ZLBfU4./tsMH5gdBtRsfiGwmypXBf5uYyAneIS", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1713945594, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-04-24T07:59:55.398 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.087+0000 7f9bd4400700 1 Processor -- start 2024-04-24T07:59:55.398 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.087+0000 7f9bd4400700 1 -- start start 2024-04-24T07:59:55.398 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.088+0000 7f9bd4400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc108710 0x7f9bcc108b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:55.398 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.088+0000 7f9bd4400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9bcc109100 con 0x7f9bcc108710 2024-04-24T07:59:55.398 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.088+0000 7f9bd1a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc108710 0x7f9bcc108b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:55.398 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.089+0000 7f9bd1a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc108710 0x7f9bcc108b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44278/0 (socket says 172.21.15.28:44278) 2024-04-24T07:59:55.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.089+0000 7f9bd1a00700 1 -- 172.21.15.28:0/1926497857 learned_addr learned my addr 172.21.15.28:0/1926497857 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:55.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.089+0000 7f9bd1a00700 1 -- 172.21.15.28:0/1926497857 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bcc1098c0 con 0x7f9bcc108710 2024-04-24T07:59:55.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.089+0000 7f9bd1a00700 1 --2- 172.21.15.28:0/1926497857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc108710 0x7f9bcc108b30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00098a0 tx=0x7f9bc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c46e5547bd8b9725 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:55.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.090+0000 7f9bcbe00700 1 -- 172.21.15.28:0/1926497857 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bc0004160 con 0x7f9bcc108710 2024-04-24T07:59:55.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.090+0000 7f9bcbe00700 1 -- 172.21.15.28:0/1926497857 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bc0004300 con 0x7f9bcc108710 2024-04-24T07:59:55.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.090+0000 7f9bcbe00700 1 -- 172.21.15.28:0/1926497857 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bc0011b20 con 0x7f9bcc108710 2024-04-24T07:59:55.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.091+0000 7f9bd4400700 1 -- 172.21.15.28:0/1926497857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc108710 msgr2=0x7f9bcc108b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:55.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.091+0000 7f9bd4400700 1 --2- 172.21.15.28:0/1926497857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc108710 0x7f9bcc108b30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f9bc00098a0 tx=0x7f9bc0009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:55.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.091+0000 7f9bd4400700 1 -- 172.21.15.28:0/1926497857 shutdown_connections 2024-04-24T07:59:55.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.091+0000 7f9bd4400700 1 --2- 172.21.15.28:0/1926497857 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc108710 0x7f9bcc108b30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:55.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.091+0000 7f9bd4400700 1 -- 172.21.15.28:0/1926497857 >> 172.21.15.28:0/1926497857 conn(0x7f9bcc103d70 msgr2=0x7f9bcc106190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:55.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.091+0000 7f9bd4400700 1 -- 172.21.15.28:0/1926497857 shutdown_connections 2024-04-24T07:59:55.399 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.091+0000 7f9bd4400700 1 -- 172.21.15.28:0/1926497857 wait complete. 2024-04-24T07:59:55.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.092+0000 7f9bd4400700 1 Processor -- start 2024-04-24T07:59:55.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.093+0000 7f9bd4400700 1 -- start start 2024-04-24T07:59:55.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.093+0000 7f9bd4400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc108710 0x7f9bcc19c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:55.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.093+0000 7f9bd4400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9bcc10cc70 con 0x7f9bcc108710 2024-04-24T07:59:55.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.093+0000 7f9bd1a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc108710 0x7f9bcc19c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:55.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.093+0000 7f9bd1a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc108710 0x7f9bcc19c650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44288/0 (socket says 172.21.15.28:44288) 2024-04-24T07:59:55.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.093+0000 7f9bd1a00700 1 -- 172.21.15.28:0/1459666299 learned_addr learned my addr 172.21.15.28:0/1459666299 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:55.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.094+0000 7f9bd1a00700 1 -- 172.21.15.28:0/1459666299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9bcc19cb90 con 0x7f9bcc108710 2024-04-24T07:59:55.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.094+0000 7f9bd1a00700 1 --2- 172.21.15.28:0/1459666299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc108710 0x7f9bcc19c650 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0000c00 tx=0x7f9bc0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:55.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.094+0000 7f9bca000700 1 -- 172.21.15.28:0/1459666299 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bc0003b60 con 0x7f9bcc108710 2024-04-24T07:59:55.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.094+0000 7f9bca000700 1 -- 172.21.15.28:0/1459666299 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9bc001a910 con 0x7f9bcc108710 2024-04-24T07:59:55.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.094+0000 7f9bca000700 1 -- 172.21.15.28:0/1459666299 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9bc0024930 con 0x7f9bcc108710 2024-04-24T07:59:55.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.095+0000 7f9bd4400700 1 -- 172.21.15.28:0/1459666299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9bcc19ce20 con 0x7f9bcc108710 2024-04-24T07:59:55.400 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.095+0000 7f9bd4400700 1 -- 172.21.15.28:0/1459666299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9bcc19d280 con 0x7f9bcc108710 2024-04-24T07:59:55.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.096+0000 7f9bd4400700 1 -- 172.21.15.28:0/1459666299 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9bcc064150 con 0x7f9bcc108710 2024-04-24T07:59:55.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.096+0000 7f9bca000700 1 -- 172.21.15.28:0/1459666299 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f9bc0022020 con 0x7f9bcc108710 2024-04-24T07:59:55.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.097+0000 7f9bca000700 1 --2- 172.21.15.28:0/1459666299 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9bb4039660 0x7f9bb403bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:55.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.097+0000 7f9bca000700 1 -- 172.21.15.28:0/1459666299 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f9bc004da90 con 0x7f9bcc108710 2024-04-24T07:59:55.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.098+0000 7f9bd1000700 1 --2- 172.21.15.28:0/1459666299 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9bb4039660 0x7f9bb403bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:55.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.099+0000 7f9bd1000700 1 --2- 172.21.15.28:0/1459666299 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9bb4039660 0x7f9bb403bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8006fd0 tx=0x7f9bb8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:55.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.103+0000 7f9bca000700 1 -- 172.21.15.28:0/1459666299 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9bc000b3e0 con 0x7f9bcc108710 2024-04-24T07:59:55.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.354+0000 7f9bd4400700 1 -- 172.21.15.28:0/1459666299 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f9bcc19fc30 con 0x7f9bb4039660 2024-04-24T07:59:55.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.607+0000 7f9bca000700 1 -- 172.21.15.28:0/1459666299 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f9bcc19fc30 con 0x7f9bb4039660 2024-04-24T07:59:55.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.610+0000 7f9bd4400700 1 -- 172.21.15.28:0/1459666299 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9bb4039660 msgr2=0x7f9bb403bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:55.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.610+0000 7f9bd4400700 1 --2- 172.21.15.28:0/1459666299 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9bb4039660 0x7f9bb403bb20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9bb8006fd0 tx=0x7f9bb8006e40 comp rx=0 tx=0).stop 2024-04-24T07:59:55.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.610+0000 7f9bd4400700 1 -- 172.21.15.28:0/1459666299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc108710 msgr2=0x7f9bcc19c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:55.401 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.610+0000 7f9bd4400700 1 --2- 172.21.15.28:0/1459666299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc108710 0x7f9bcc19c650 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f9bc0000c00 tx=0x7f9bc0011c10 comp rx=0 tx=0).stop 2024-04-24T07:59:55.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.610+0000 7f9bd4400700 1 -- 172.21.15.28:0/1459666299 shutdown_connections 2024-04-24T07:59:55.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.610+0000 7f9bd4400700 1 --2- 172.21.15.28:0/1459666299 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9bb4039660 0x7f9bb403bb20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:55.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.610+0000 7f9bd4400700 1 --2- 172.21.15.28:0/1459666299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9bcc108710 0x7f9bcc19c650 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:55.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.610+0000 7f9bd4400700 1 -- 172.21.15.28:0/1459666299 >> 172.21.15.28:0/1459666299 conn(0x7f9bcc103d70 msgr2=0x7f9bcc104840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:55.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.611+0000 7f9bd4400700 1 -- 172.21.15.28:0/1459666299 shutdown_connections 2024-04-24T07:59:55.402 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:54.611+0000 7f9bd4400700 1 -- 172.21.15.28:0/1459666299 wait complete. 2024-04-24T07:59:55.402 INFO:teuthology.orchestra.run.smithi028.stdout:Fetching dashboard port number... 2024-04-24T07:59:55.962 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:55 smithi028 ceph-mon[27786]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T07:59:56.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:56 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T07:59:56.878 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:56 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3471004559' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-04-24T07:59:56.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stdout 8443 2024-04-24T07:59:56.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.960+0000 7f6543a00700 1 Processor -- start 2024-04-24T07:59:56.888 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.960+0000 7f6543a00700 1 -- start start 2024-04-24T07:59:56.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.960+0000 7f6543a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c1086f0 0x7f653c108b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:56.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.960+0000 7f6543a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f653c1090e0 con 0x7f653c1086f0 2024-04-24T07:59:56.889 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.961+0000 7f6541000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c1086f0 0x7f653c108b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:56.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.961+0000 7f6541000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c1086f0 0x7f653c108b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44290/0 (socket says 172.21.15.28:44290) 2024-04-24T07:59:56.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.961+0000 7f6541000700 1 -- 172.21.15.28:0/2203493840 learned_addr learned my addr 172.21.15.28:0/2203493840 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:56.890 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.961+0000 7f6541000700 1 -- 172.21.15.28:0/2203493840 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653c1098a0 con 0x7f653c1086f0 2024-04-24T07:59:56.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.962+0000 7f6541000700 1 --2- 172.21.15.28:0/2203493840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c1086f0 0x7f653c108b10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f65300098a0 tx=0x7f6530009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11c768a32e2ffc01 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:56.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.962+0000 7f653b400700 1 -- 172.21.15.28:0/2203493840 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6530004160 con 0x7f653c1086f0 2024-04-24T07:59:56.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.962+0000 7f653b400700 1 -- 172.21.15.28:0/2203493840 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6530004300 con 0x7f653c1086f0 2024-04-24T07:59:56.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.962+0000 7f653b400700 1 -- 172.21.15.28:0/2203493840 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6530011b20 con 0x7f653c1086f0 2024-04-24T07:59:56.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.963+0000 7f6543a00700 1 -- 172.21.15.28:0/2203493840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c1086f0 msgr2=0x7f653c108b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:56.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.963+0000 7f6543a00700 1 --2- 172.21.15.28:0/2203493840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c1086f0 0x7f653c108b10 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f65300098a0 tx=0x7f6530009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:56.897 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.963+0000 7f6543a00700 1 -- 172.21.15.28:0/2203493840 shutdown_connections 2024-04-24T07:59:56.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.963+0000 7f6543a00700 1 --2- 172.21.15.28:0/2203493840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c1086f0 0x7f653c108b10 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:56.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.963+0000 7f6543a00700 1 -- 172.21.15.28:0/2203493840 >> 172.21.15.28:0/2203493840 conn(0x7f653c103d10 msgr2=0x7f653c106130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:56.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.963+0000 7f6543a00700 1 -- 172.21.15.28:0/2203493840 shutdown_connections 2024-04-24T07:59:56.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.964+0000 7f6543a00700 1 -- 172.21.15.28:0/2203493840 wait complete. 2024-04-24T07:59:56.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.965+0000 7f6543a00700 1 Processor -- start 2024-04-24T07:59:56.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.965+0000 7f6543a00700 1 -- start start 2024-04-24T07:59:56.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.965+0000 7f6543a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c1086f0 0x7f653c19e860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:56.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.965+0000 7f6543a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f653c10cc50 con 0x7f653c1086f0 2024-04-24T07:59:56.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.966+0000 7f6541000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c1086f0 0x7f653c19e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:56.898 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.966+0000 7f6541000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c1086f0 0x7f653c19e860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44302/0 (socket says 172.21.15.28:44302) 2024-04-24T07:59:56.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.966+0000 7f6541000700 1 -- 172.21.15.28:0/3471004559 learned_addr learned my addr 172.21.15.28:0/3471004559 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:56.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.966+0000 7f6541000700 1 -- 172.21.15.28:0/3471004559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f653c19eda0 con 0x7f653c1086f0 2024-04-24T07:59:56.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.966+0000 7f6541000700 1 --2- 172.21.15.28:0/3471004559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c1086f0 0x7f653c19e860 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6530000c00 tx=0x7f6530011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:56.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.967+0000 7f6539600700 1 -- 172.21.15.28:0/3471004559 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6530003b60 con 0x7f653c1086f0 2024-04-24T07:59:56.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.967+0000 7f6539600700 1 -- 172.21.15.28:0/3471004559 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f653001a910 con 0x7f653c1086f0 2024-04-24T07:59:56.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.967+0000 7f6539600700 1 -- 172.21.15.28:0/3471004559 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6530024930 con 0x7f653c1086f0 2024-04-24T07:59:56.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.967+0000 7f6543a00700 1 -- 172.21.15.28:0/3471004559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f653c19f030 con 0x7f653c1086f0 2024-04-24T07:59:56.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.967+0000 7f6543a00700 1 -- 172.21.15.28:0/3471004559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f653c19f490 con 0x7f653c1086f0 2024-04-24T07:59:56.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.968+0000 7f6543a00700 1 -- 172.21.15.28:0/3471004559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f653c064150 con 0x7f653c1086f0 2024-04-24T07:59:56.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.969+0000 7f6539600700 1 -- 172.21.15.28:0/3471004559 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 12) ==== 46571+0+0 (secure 0 0 0) 0x7f6530022020 con 0x7f653c1086f0 2024-04-24T07:59:56.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.969+0000 7f6539600700 1 --2- 172.21.15.28:0/3471004559 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6524039660 0x7f652403bb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:56.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.969+0000 7f6539600700 1 -- 172.21.15.28:0/3471004559 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f653004db10 con 0x7f653c1086f0 2024-04-24T07:59:56.899 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.970+0000 7f653be00700 1 --2- 172.21.15.28:0/3471004559 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6524039660 0x7f652403bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:56.900 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.970+0000 7f653be00700 1 --2- 172.21.15.28:0/3471004559 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6524039660 0x7f652403bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6528006fd0 tx=0x7f6528006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:56.900 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:55.975+0000 7f6539600700 1 -- 172.21.15.28:0/3471004559 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f653000b3e0 con 0x7f653c1086f0 2024-04-24T07:59:56.900 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:56.203+0000 7f6543a00700 1 -- 172.21.15.28:0/3471004559 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f653c1a1f60 con 0x7f653c1086f0 2024-04-24T07:59:56.900 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:56.204+0000 7f6539600700 1 -- 172.21.15.28:0/3471004559 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f653002ddb0 con 0x7f653c1086f0 2024-04-24T07:59:56.900 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:56.209+0000 7f6543a00700 1 -- 172.21.15.28:0/3471004559 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6524039660 msgr2=0x7f652403bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:56.900 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:56.209+0000 7f6543a00700 1 --2- 172.21.15.28:0/3471004559 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6524039660 0x7f652403bb20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6528006fd0 tx=0x7f6528006e40 comp rx=0 tx=0).stop 2024-04-24T07:59:56.900 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:56.209+0000 7f6543a00700 1 -- 172.21.15.28:0/3471004559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c1086f0 msgr2=0x7f653c19e860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:56.900 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:56.209+0000 7f6543a00700 1 --2- 172.21.15.28:0/3471004559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c1086f0 0x7f653c19e860 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6530000c00 tx=0x7f6530011c10 comp rx=0 tx=0).stop 2024-04-24T07:59:56.900 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:56.209+0000 7f6543a00700 1 -- 172.21.15.28:0/3471004559 shutdown_connections 2024-04-24T07:59:56.900 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:56.209+0000 7f6543a00700 1 --2- 172.21.15.28:0/3471004559 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6524039660 0x7f652403bb20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:56.901 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:56.209+0000 7f6543a00700 1 --2- 172.21.15.28:0/3471004559 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f653c1086f0 0x7f653c19e860 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:56.901 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:56.209+0000 7f6543a00700 1 -- 172.21.15.28:0/3471004559 >> 172.21.15.28:0/3471004559 conn(0x7f653c103d10 msgr2=0x7f653c104820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:56.901 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:56.209+0000 7f6543a00700 1 -- 172.21.15.28:0/3471004559 shutdown_connections 2024-04-24T07:59:56.901 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:56.209+0000 7f6543a00700 1 -- 172.21.15.28:0/3471004559 wait complete. 2024-04-24T07:59:56.919 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-04-24T07:59:56.919 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout disabled 2024-04-24T07:59:56.937 INFO:teuthology.orchestra.run.smithi028.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-04-24T07:59:56.938 INFO:teuthology.orchestra.run.smithi028.stdout:systemctl: stdout inactive 2024-04-24T07:59:56.938 INFO:teuthology.orchestra.run.smithi028.stdout:firewalld.service is not enabled 2024-04-24T07:59:56.938 INFO:teuthology.orchestra.run.smithi028.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-04-24T07:59:56.943 INFO:teuthology.orchestra.run.smithi028.stdout:Ceph Dashboard is now available at: 2024-04-24T07:59:56.943 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:59:56.943 INFO:teuthology.orchestra.run.smithi028.stdout: URL: https://smithi028.front.sepia.ceph.com:8443/ 2024-04-24T07:59:56.943 INFO:teuthology.orchestra.run.smithi028.stdout: User: admin 2024-04-24T07:59:56.943 INFO:teuthology.orchestra.run.smithi028.stdout: Password: p9lhagcdwi 2024-04-24T07:59:56.943 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T07:59:56.944 INFO:teuthology.orchestra.run.smithi028.stdout:Saving cluster configuration to /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config directory 2024-04-24T07:59:56.945 INFO:teuthology.orchestra.run.smithi028.stdout:Enabling autotune for osd_memory_target 2024-04-24T07:59:57.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 07:59:57 smithi028 ceph-mon[27786]: mgrmap e13: x(active, since 6s) 2024-04-24T07:59:58.855 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.570+0000 7f6ddbe00700 1 Processor -- start 2024-04-24T07:59:58.855 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.570+0000 7f6ddbe00700 1 -- start start 2024-04-24T07:59:58.855 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.570+0000 7f6ddbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ddc10a730 0x7f6ddc10ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:58.855 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.571+0000 7f6ddbe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ddc10b120 con 0x7f6ddc10a730 2024-04-24T07:59:58.856 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.571+0000 7f6ddaa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ddc10a730 0x7f6ddc10ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:58.856 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.571+0000 7f6ddaa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ddc10a730 0x7f6ddc10ab50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44316/0 (socket says 172.21.15.28:44316) 2024-04-24T07:59:58.856 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.571+0000 7f6ddaa00700 1 -- 172.21.15.28:0/2041008328 learned_addr learned my addr 172.21.15.28:0/2041008328 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:58.856 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.571+0000 7f6ddaa00700 1 -- 172.21.15.28:0/2041008328 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ddc10b8e0 con 0x7f6ddc10a730 2024-04-24T07:59:58.856 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.572+0000 7f6ddaa00700 1 --2- 172.21.15.28:0/2041008328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ddc10a730 0x7f6ddc10ab50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc0098a0 tx=0x7f6dcc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38635903234542a7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:58.856 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.572+0000 7f6dd9600700 1 -- 172.21.15.28:0/2041008328 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6dcc004160 con 0x7f6ddc10a730 2024-04-24T07:59:58.856 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.572+0000 7f6dd9600700 1 -- 172.21.15.28:0/2041008328 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6dcc004300 con 0x7f6ddc10a730 2024-04-24T07:59:58.856 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.572+0000 7f6dd9600700 1 -- 172.21.15.28:0/2041008328 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6dcc011b20 con 0x7f6ddc10a730 2024-04-24T07:59:58.856 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.573+0000 7f6ddbe00700 1 -- 172.21.15.28:0/2041008328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ddc10a730 msgr2=0x7f6ddc10ab50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:58.856 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.573+0000 7f6ddbe00700 1 --2- 172.21.15.28:0/2041008328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ddc10a730 0x7f6ddc10ab50 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc0098a0 tx=0x7f6dcc009bb0 comp rx=0 tx=0).stop 2024-04-24T07:59:58.856 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.573+0000 7f6ddbe00700 1 -- 172.21.15.28:0/2041008328 shutdown_connections 2024-04-24T07:59:58.856 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.573+0000 7f6ddbe00700 1 --2- 172.21.15.28:0/2041008328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ddc10a730 0x7f6ddc10ab50 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:58.856 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.573+0000 7f6ddbe00700 1 -- 172.21.15.28:0/2041008328 >> 172.21.15.28:0/2041008328 conn(0x7f6ddc07ddf0 msgr2=0x7f6ddc1084f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:58.857 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.573+0000 7f6ddbe00700 1 -- 172.21.15.28:0/2041008328 shutdown_connections 2024-04-24T07:59:58.857 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.573+0000 7f6ddbe00700 1 -- 172.21.15.28:0/2041008328 wait complete. 2024-04-24T07:59:58.857 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.574+0000 7f6ddbe00700 1 Processor -- start 2024-04-24T07:59:58.857 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.574+0000 7f6ddbe00700 1 -- start start 2024-04-24T07:59:58.857 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.574+0000 7f6ddbe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ddc10a730 0x7f6ddc19e690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:58.857 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.574+0000 7f6ddbe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ddc10ec90 con 0x7f6ddc10a730 2024-04-24T07:59:58.857 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.575+0000 7f6ddaa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ddc10a730 0x7f6ddc19e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:58.857 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.575+0000 7f6ddaa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ddc10a730 0x7f6ddc19e690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44328/0 (socket says 172.21.15.28:44328) 2024-04-24T07:59:58.857 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.575+0000 7f6ddaa00700 1 -- 172.21.15.28:0/1815673455 learned_addr learned my addr 172.21.15.28:0/1815673455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T07:59:58.857 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.575+0000 7f6ddaa00700 1 -- 172.21.15.28:0/1815673455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ddc19ebd0 con 0x7f6ddc10a730 2024-04-24T07:59:58.857 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.575+0000 7f6ddaa00700 1 --2- 172.21.15.28:0/1815673455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ddc10a730 0x7f6ddc19e690 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc000c00 tx=0x7f6dcc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:58.857 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.576+0000 7f6dd3400700 1 -- 172.21.15.28:0/1815673455 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6dcc003b60 con 0x7f6ddc10a730 2024-04-24T07:59:58.857 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.576+0000 7f6dd3400700 1 -- 172.21.15.28:0/1815673455 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6dcc01a910 con 0x7f6ddc10a730 2024-04-24T07:59:58.857 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.576+0000 7f6ddbe00700 1 -- 172.21.15.28:0/1815673455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ddc19ee60 con 0x7f6ddc10a730 2024-04-24T07:59:58.858 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.576+0000 7f6dd3400700 1 -- 172.21.15.28:0/1815673455 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6dcc024930 con 0x7f6ddc10a730 2024-04-24T07:59:58.858 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.576+0000 7f6ddbe00700 1 -- 172.21.15.28:0/1815673455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ddc19f2c0 con 0x7f6ddc10a730 2024-04-24T07:59:58.858 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.577+0000 7f6ddbe00700 1 -- 172.21.15.28:0/1815673455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ddc064150 con 0x7f6ddc10a730 2024-04-24T07:59:58.858 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.577+0000 7f6dd3400700 1 -- 172.21.15.28:0/1815673455 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6dcc022020 con 0x7f6ddc10a730 2024-04-24T07:59:58.858 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.578+0000 7f6dd3400700 1 --2- 172.21.15.28:0/1815673455 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6dc00396a0 0x7f6dc003bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T07:59:58.858 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.578+0000 7f6dd3400700 1 -- 172.21.15.28:0/1815673455 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f6dcc04dbb0 con 0x7f6ddc10a730 2024-04-24T07:59:58.858 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.578+0000 7f6dda000700 1 --2- 172.21.15.28:0/1815673455 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6dc00396a0 0x7f6dc003bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T07:59:58.858 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.579+0000 7f6dda000700 1 --2- 172.21.15.28:0/1815673455 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6dc00396a0 0x7f6dc003bb60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8006fd0 tx=0x7f6dc8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T07:59:58.858 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.583+0000 7f6dd3400700 1 -- 172.21.15.28:0/1815673455 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6dcc0212c0 con 0x7f6ddc10a730 2024-04-24T07:59:58.858 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.815+0000 7f6ddbe00700 1 -- 172.21.15.28:0/1815673455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f6ddc1a1c90 con 0x7f6ddc10a730 2024-04-24T07:59:58.858 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.816+0000 7f6dd3400700 1 -- 172.21.15.28:0/1815673455 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f6dcc01aac0 con 0x7f6ddc10a730 2024-04-24T07:59:58.858 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.819+0000 7f6ddbe00700 1 -- 172.21.15.28:0/1815673455 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6dc00396a0 msgr2=0x7f6dc003bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:58.858 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.819+0000 7f6ddbe00700 1 --2- 172.21.15.28:0/1815673455 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6dc00396a0 0x7f6dc003bb60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f6dc8006fd0 tx=0x7f6dc8006e40 comp rx=0 tx=0).stop 2024-04-24T07:59:58.859 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.819+0000 7f6ddbe00700 1 -- 172.21.15.28:0/1815673455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ddc10a730 msgr2=0x7f6ddc19e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T07:59:58.859 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.819+0000 7f6ddbe00700 1 --2- 172.21.15.28:0/1815673455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ddc10a730 0x7f6ddc19e690 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc000c00 tx=0x7f6dcc011c10 comp rx=0 tx=0).stop 2024-04-24T07:59:58.859 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.819+0000 7f6ddbe00700 1 -- 172.21.15.28:0/1815673455 shutdown_connections 2024-04-24T07:59:58.859 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.819+0000 7f6ddbe00700 1 --2- 172.21.15.28:0/1815673455 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6dc00396a0 0x7f6dc003bb60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:58.859 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.819+0000 7f6ddbe00700 1 --2- 172.21.15.28:0/1815673455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ddc10a730 0x7f6ddc19e690 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T07:59:58.859 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.819+0000 7f6ddbe00700 1 -- 172.21.15.28:0/1815673455 >> 172.21.15.28:0/1815673455 conn(0x7f6ddc07ddf0 msgr2=0x7f6ddc107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T07:59:58.859 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.819+0000 7f6ddbe00700 1 -- 172.21.15.28:0/1815673455 shutdown_connections 2024-04-24T07:59:58.859 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:57.820+0000 7f6ddbe00700 1 -- 172.21.15.28:0/1815673455 wait complete. 2024-04-24T08:00:00.904 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.914+0000 7f0157800700 1 Processor -- start 2024-04-24T08:00:00.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.914+0000 7f0157800700 1 -- start start 2024-04-24T08:00:00.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.915+0000 7f0157800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015010a910 0x7f015010ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:00.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.915+0000 7f0157800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f015010b300 con 0x7f015010a910 2024-04-24T08:00:00.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.915+0000 7f0154e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015010a910 0x7f015010ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:00.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.915+0000 7f0154e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015010a910 0x7f015010ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58840/0 (socket says 172.21.15.28:58840) 2024-04-24T08:00:00.910 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.915+0000 7f0154e00700 1 -- 172.21.15.28:0/876793867 learned_addr learned my addr 172.21.15.28:0/876793867 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:00:00.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.916+0000 7f0154e00700 1 -- 172.21.15.28:0/876793867 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f015010bac0 con 0x7f015010a910 2024-04-24T08:00:00.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.916+0000 7f0154e00700 1 --2- 172.21.15.28:0/876793867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015010a910 0x7f015010ad30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f01440098a0 tx=0x7f0144009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d584ce202de1a267 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:00.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.917+0000 7f014f400700 1 -- 172.21.15.28:0/876793867 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0144004160 con 0x7f015010a910 2024-04-24T08:00:00.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.917+0000 7f014f400700 1 -- 172.21.15.28:0/876793867 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0144004300 con 0x7f015010a910 2024-04-24T08:00:00.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.917+0000 7f014f400700 1 -- 172.21.15.28:0/876793867 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0144011b20 con 0x7f015010a910 2024-04-24T08:00:00.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.918+0000 7f0157800700 1 -- 172.21.15.28:0/876793867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015010a910 msgr2=0x7f015010ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:00.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.918+0000 7f0157800700 1 --2- 172.21.15.28:0/876793867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015010a910 0x7f015010ad30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f01440098a0 tx=0x7f0144009bb0 comp rx=0 tx=0).stop 2024-04-24T08:00:00.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.918+0000 7f0157800700 1 -- 172.21.15.28:0/876793867 shutdown_connections 2024-04-24T08:00:00.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.918+0000 7f0157800700 1 --2- 172.21.15.28:0/876793867 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015010a910 0x7f015010ad30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:00.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.918+0000 7f0157800700 1 -- 172.21.15.28:0/876793867 >> 172.21.15.28:0/876793867 conn(0x7f015007de90 msgr2=0x7f01501084f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:00.911 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.918+0000 7f0157800700 1 -- 172.21.15.28:0/876793867 shutdown_connections 2024-04-24T08:00:00.912 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.918+0000 7f0157800700 1 -- 172.21.15.28:0/876793867 wait complete. 2024-04-24T08:00:00.912 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.919+0000 7f0157800700 1 Processor -- start 2024-04-24T08:00:00.912 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.920+0000 7f0157800700 1 -- start start 2024-04-24T08:00:00.912 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.920+0000 7f0157800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015010a910 0x7f015019e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:00.912 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.920+0000 7f0157800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f015010ee70 con 0x7f015010a910 2024-04-24T08:00:00.912 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.920+0000 7f0154e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015010a910 0x7f015019e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:00.912 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.920+0000 7f0154e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015010a910 0x7f015019e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58842/0 (socket says 172.21.15.28:58842) 2024-04-24T08:00:00.912 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.920+0000 7f0154e00700 1 -- 172.21.15.28:0/3950365282 learned_addr learned my addr 172.21.15.28:0/3950365282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:00:00.912 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.921+0000 7f0154e00700 1 -- 172.21.15.28:0/3950365282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f015019ed30 con 0x7f015010a910 2024-04-24T08:00:00.912 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.921+0000 7f0154e00700 1 --2- 172.21.15.28:0/3950365282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015010a910 0x7f015019e7f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f01440117e0 tx=0x7f0144011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:00.912 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.921+0000 7f014d600700 1 -- 172.21.15.28:0/3950365282 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0144003bf0 con 0x7f015010a910 2024-04-24T08:00:00.913 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.921+0000 7f014d600700 1 -- 172.21.15.28:0/3950365282 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f014401a910 con 0x7f015010a910 2024-04-24T08:00:00.913 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.921+0000 7f0157800700 1 -- 172.21.15.28:0/3950365282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f015019efc0 con 0x7f015010a910 2024-04-24T08:00:00.913 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.922+0000 7f014d600700 1 -- 172.21.15.28:0/3950365282 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01440249a0 con 0x7f015010a910 2024-04-24T08:00:00.913 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.922+0000 7f0157800700 1 -- 172.21.15.28:0/3950365282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f015019f420 con 0x7f015010a910 2024-04-24T08:00:00.913 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.923+0000 7f014d600700 1 -- 172.21.15.28:0/3950365282 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0144022020 con 0x7f015010a910 2024-04-24T08:00:00.913 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.923+0000 7f0157800700 1 -- 172.21.15.28:0/3950365282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0150064150 con 0x7f015010a910 2024-04-24T08:00:00.913 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.924+0000 7f014d600700 1 --2- 172.21.15.28:0/3950365282 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f01380396b0 0x7f013803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:00.913 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.924+0000 7f014d600700 1 -- 172.21.15.28:0/3950365282 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f014404dba0 con 0x7f015010a910 2024-04-24T08:00:00.913 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.924+0000 7f014fe00700 1 --2- 172.21.15.28:0/3950365282 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f01380396b0 0x7f013803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:00.913 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.925+0000 7f014fe00700 1 --2- 172.21.15.28:0/3950365282 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f01380396b0 0x7f013803bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f013c006fd0 tx=0x7f013c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:00.913 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T07:59:59.929+0000 7f014d600700 1 -- 172.21.15.28:0/3950365282 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0144011e20 con 0x7f015010a910 2024-04-24T08:00:00.913 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T08:00:00.281+0000 7f0157800700 1 -- 172.21.15.28:0/3950365282 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f01501a1eb0 con 0x7f015010a910 2024-04-24T08:00:00.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T08:00:00.285+0000 7f014d600700 1 -- 172.21.15.28:0/3950365282 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v27)=0 set mgr/dashboard/cluster/status v27) ==== 153+0+0 (secure 0 0 0) 0x7f0144004300 con 0x7f015010a910 2024-04-24T08:00:00.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T08:00:00.288+0000 7f0157800700 1 -- 172.21.15.28:0/3950365282 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f01380396b0 msgr2=0x7f013803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:00.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T08:00:00.288+0000 7f0157800700 1 --2- 172.21.15.28:0/3950365282 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f01380396b0 0x7f013803bb70 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f013c006fd0 tx=0x7f013c006e40 comp rx=0 tx=0).stop 2024-04-24T08:00:00.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T08:00:00.288+0000 7f0157800700 1 -- 172.21.15.28:0/3950365282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015010a910 msgr2=0x7f015019e7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:00.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T08:00:00.288+0000 7f0157800700 1 --2- 172.21.15.28:0/3950365282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015010a910 0x7f015019e7f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f01440117e0 tx=0x7f0144011c10 comp rx=0 tx=0).stop 2024-04-24T08:00:00.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T08:00:00.288+0000 7f0157800700 1 -- 172.21.15.28:0/3950365282 shutdown_connections 2024-04-24T08:00:00.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T08:00:00.288+0000 7f0157800700 1 --2- 172.21.15.28:0/3950365282 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f01380396b0 0x7f013803bb70 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:00.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T08:00:00.288+0000 7f0157800700 1 --2- 172.21.15.28:0/3950365282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f015010a910 0x7f015019e7f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:00.914 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T08:00:00.288+0000 7f0157800700 1 -- 172.21.15.28:0/3950365282 >> 172.21.15.28:0/3950365282 conn(0x7f015007de90 msgr2=0x7f0150106b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:00.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T08:00:00.288+0000 7f0157800700 1 -- 172.21.15.28:0/3950365282 shutdown_connections 2024-04-24T08:00:00.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr 2024-04-24T08:00:00.288+0000 7f0157800700 1 -- 172.21.15.28:0/3950365282 wait complete. 2024-04-24T08:00:00.915 INFO:teuthology.orchestra.run.smithi028.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-04-24T08:00:00.915 INFO:teuthology.orchestra.run.smithi028.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-04-24T08:00:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:00:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-04-24T08:00:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:00:00.915 INFO:teuthology.orchestra.run.smithi028.stdout:Or, if you are only running a single cluster on this host: 2024-04-24T08:00:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:00:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-04-24T08:00:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:00:00.915 INFO:teuthology.orchestra.run.smithi028.stdout:Please consider enabling telemetry to help improve Ceph: 2024-04-24T08:00:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:00:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: ceph telemetry on 2024-04-24T08:00:00.915 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:00:00.916 INFO:teuthology.orchestra.run.smithi028.stdout:For more information see: 2024-04-24T08:00:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:00:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-04-24T08:00:00.916 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:00:00.916 INFO:teuthology.orchestra.run.smithi028.stdout:Bootstrap complete. 2024-04-24T08:00:00.997 INFO:tasks.cephadm:Fetching config... 2024-04-24T08:00:00.998 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T08:00:00.998 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-04-24T08:00:01.048 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-04-24T08:00:01.048 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T08:00:01.048 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-04-24T08:00:01.096 INFO:tasks.cephadm:Fetching mon keyring... 2024-04-24T08:00:01.096 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T08:00:01.096 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/keyring of=/dev/stdout 2024-04-24T08:00:01.203 INFO:tasks.cephadm:Fetching pub ssh key... 2024-04-24T08:00:01.203 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T08:00:01.203 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-04-24T08:00:01.299 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-04-24T08:00:01.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZYhBI+GUk+9C6auHe/w5oj/55pFiy+mOUtLbhVDX2zrYYUVmdd9K4XwMzlLpSkzVR2UiBCbYk2bKpr9ni2n7FRrW3vIO72eRI07uLEdoe2p1vaud9GTFfsA2vBDDMuQHvF8WQuEBdoYBq7nTc3J13d/Pk+X1KVw8g19xXqSfeWhM05I+PyJ0QFfwuZxK/fCBZbL8+tqpzEuUlObLf61h27K+vNoJeFkEczEqk6vMUiCZwcX1xLry33wqWivlzP5E0735b0LudFXAuF2RLaok6+/5b5EIHRnMctjygQTQkrPikyrDHTD1Rkd2zo7Wyh0H5hCykRhTZoVngHZv4TfQp4KQiKCttXupGM5YAlFKCZ3CxmCv1+b/puPhyssHAOwaVCBmrg1RheOTF5sZZw97gzXWL0xS6hE4JInC+VznSuy2ByBjEVtroPQ0Sr3a79dxE1X7rD2csMQZ8aazsCkB53ff0rT8ktMnxCHC49DxPyxdb2cQ0nXFE3Tiv/ZpCPDs= ceph-473aaf58-0210-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-24T08:00:01.419 INFO:teuthology.orchestra.run.smithi028.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZYhBI+GUk+9C6auHe/w5oj/55pFiy+mOUtLbhVDX2zrYYUVmdd9K4XwMzlLpSkzVR2UiBCbYk2bKpr9ni2n7FRrW3vIO72eRI07uLEdoe2p1vaud9GTFfsA2vBDDMuQHvF8WQuEBdoYBq7nTc3J13d/Pk+X1KVw8g19xXqSfeWhM05I+PyJ0QFfwuZxK/fCBZbL8+tqpzEuUlObLf61h27K+vNoJeFkEczEqk6vMUiCZwcX1xLry33wqWivlzP5E0735b0LudFXAuF2RLaok6+/5b5EIHRnMctjygQTQkrPikyrDHTD1Rkd2zo7Wyh0H5hCykRhTZoVngHZv4TfQp4KQiKCttXupGM5YAlFKCZ3CxmCv1+b/puPhyssHAOwaVCBmrg1RheOTF5sZZw97gzXWL0xS6hE4JInC+VznSuy2ByBjEVtroPQ0Sr3a79dxE1X7rD2csMQZ8aazsCkB53ff0rT8ktMnxCHC49DxPyxdb2cQ0nXFE3Tiv/ZpCPDs= ceph-473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:00:01.446 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:01 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3950365282' entity='client.admin' 2024-04-24T08:00:01.454 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZYhBI+GUk+9C6auHe/w5oj/55pFiy+mOUtLbhVDX2zrYYUVmdd9K4XwMzlLpSkzVR2UiBCbYk2bKpr9ni2n7FRrW3vIO72eRI07uLEdoe2p1vaud9GTFfsA2vBDDMuQHvF8WQuEBdoYBq7nTc3J13d/Pk+X1KVw8g19xXqSfeWhM05I+PyJ0QFfwuZxK/fCBZbL8+tqpzEuUlObLf61h27K+vNoJeFkEczEqk6vMUiCZwcX1xLry33wqWivlzP5E0735b0LudFXAuF2RLaok6+/5b5EIHRnMctjygQTQkrPikyrDHTD1Rkd2zo7Wyh0H5hCykRhTZoVngHZv4TfQp4KQiKCttXupGM5YAlFKCZ3CxmCv1+b/puPhyssHAOwaVCBmrg1RheOTF5sZZw97gzXWL0xS6hE4JInC+VznSuy2ByBjEVtroPQ0Sr3a79dxE1X7rD2csMQZ8aazsCkB53ff0rT8ktMnxCHC49DxPyxdb2cQ0nXFE3Tiv/ZpCPDs= ceph-473aaf58-0210-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-24T08:00:01.496 INFO:teuthology.orchestra.run.smithi117.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZYhBI+GUk+9C6auHe/w5oj/55pFiy+mOUtLbhVDX2zrYYUVmdd9K4XwMzlLpSkzVR2UiBCbYk2bKpr9ni2n7FRrW3vIO72eRI07uLEdoe2p1vaud9GTFfsA2vBDDMuQHvF8WQuEBdoYBq7nTc3J13d/Pk+X1KVw8g19xXqSfeWhM05I+PyJ0QFfwuZxK/fCBZbL8+tqpzEuUlObLf61h27K+vNoJeFkEczEqk6vMUiCZwcX1xLry33wqWivlzP5E0735b0LudFXAuF2RLaok6+/5b5EIHRnMctjygQTQkrPikyrDHTD1Rkd2zo7Wyh0H5hCykRhTZoVngHZv4TfQp4KQiKCttXupGM5YAlFKCZ3CxmCv1+b/puPhyssHAOwaVCBmrg1RheOTF5sZZw97gzXWL0xS6hE4JInC+VznSuy2ByBjEVtroPQ0Sr3a79dxE1X7rD2csMQZ8aazsCkB53ff0rT8ktMnxCHC49DxPyxdb2cQ0nXFE3Tiv/ZpCPDs= ceph-473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:00:01.510 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZYhBI+GUk+9C6auHe/w5oj/55pFiy+mOUtLbhVDX2zrYYUVmdd9K4XwMzlLpSkzVR2UiBCbYk2bKpr9ni2n7FRrW3vIO72eRI07uLEdoe2p1vaud9GTFfsA2vBDDMuQHvF8WQuEBdoYBq7nTc3J13d/Pk+X1KVw8g19xXqSfeWhM05I+PyJ0QFfwuZxK/fCBZbL8+tqpzEuUlObLf61h27K+vNoJeFkEczEqk6vMUiCZwcX1xLry33wqWivlzP5E0735b0LudFXAuF2RLaok6+/5b5EIHRnMctjygQTQkrPikyrDHTD1Rkd2zo7Wyh0H5hCykRhTZoVngHZv4TfQp4KQiKCttXupGM5YAlFKCZ3CxmCv1+b/puPhyssHAOwaVCBmrg1RheOTF5sZZw97gzXWL0xS6hE4JInC+VznSuy2ByBjEVtroPQ0Sr3a79dxE1X7rD2csMQZ8aazsCkB53ff0rT8ktMnxCHC49DxPyxdb2cQ0nXFE3Tiv/ZpCPDs= ceph-473aaf58-0210-11ef-bc93-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-04-24T08:00:01.554 INFO:teuthology.orchestra.run.smithi194.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDZYhBI+GUk+9C6auHe/w5oj/55pFiy+mOUtLbhVDX2zrYYUVmdd9K4XwMzlLpSkzVR2UiBCbYk2bKpr9ni2n7FRrW3vIO72eRI07uLEdoe2p1vaud9GTFfsA2vBDDMuQHvF8WQuEBdoYBq7nTc3J13d/Pk+X1KVw8g19xXqSfeWhM05I+PyJ0QFfwuZxK/fCBZbL8+tqpzEuUlObLf61h27K+vNoJeFkEczEqk6vMUiCZwcX1xLry33wqWivlzP5E0735b0LudFXAuF2RLaok6+/5b5EIHRnMctjygQTQkrPikyrDHTD1Rkd2zo7Wyh0H5hCykRhTZoVngHZv4TfQp4KQiKCttXupGM5YAlFKCZ3CxmCv1+b/puPhyssHAOwaVCBmrg1RheOTF5sZZw97gzXWL0xS6hE4JInC+VznSuy2ByBjEVtroPQ0Sr3a79dxE1X7rD2csMQZ8aazsCkB53ff0rT8ktMnxCHC49DxPyxdb2cQ0nXFE3Tiv/ZpCPDs= ceph-473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:00:01.567 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-04-24T08:00:01.986 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:00:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.622+0000 7fedac600700 1 -- 172.21.15.28:0/4271961050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda4102550 msgr2=0x7feda4102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.622+0000 7fedac600700 1 --2- 172.21.15.28:0/4271961050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda4102550 0x7feda4102970 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fed98009910 tx=0x7fed98009c20 comp rx=0 tx=0).stop 2024-04-24T08:00:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.622+0000 7fedac600700 1 -- 172.21.15.28:0/4271961050 shutdown_connections 2024-04-24T08:00:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.622+0000 7fedac600700 1 --2- 172.21.15.28:0/4271961050 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda4102550 0x7feda4102970 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.622+0000 7fedac600700 1 -- 172.21.15.28:0/4271961050 >> 172.21.15.28:0/4271961050 conn(0x7feda40fdaf0 msgr2=0x7feda40fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.622+0000 7fedac600700 1 -- 172.21.15.28:0/4271961050 shutdown_connections 2024-04-24T08:00:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.622+0000 7fedac600700 1 -- 172.21.15.28:0/4271961050 wait complete. 2024-04-24T08:00:04.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.624+0000 7fedac600700 1 Processor -- start 2024-04-24T08:00:04.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.624+0000 7fedac600700 1 -- start start 2024-04-24T08:00:04.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.624+0000 7fedac600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda4102550 0x7feda4199e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:04.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.624+0000 7fedac600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feda41085e0 con 0x7feda4102550 2024-04-24T08:00:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.625+0000 7feda9c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda4102550 0x7feda4199e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.625+0000 7feda9c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda4102550 0x7feda4199e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58864/0 (socket says 172.21.15.28:58864) 2024-04-24T08:00:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.625+0000 7feda9c00700 1 -- 172.21.15.28:0/597481302 learned_addr learned my addr 172.21.15.28:0/597481302 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:00:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.625+0000 7feda9c00700 1 -- 172.21.15.28:0/597481302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda419a3c0 con 0x7feda4102550 2024-04-24T08:00:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.625+0000 7feda9c00700 1 --2- 172.21.15.28:0/597481302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda4102550 0x7feda4199e80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fed98005850 tx=0x7fed98005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.626+0000 7feda2000700 1 -- 172.21.15.28:0/597481302 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed98026070 con 0x7feda4102550 2024-04-24T08:00:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.626+0000 7feda2000700 1 -- 172.21.15.28:0/597481302 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed980044a0 con 0x7feda4102550 2024-04-24T08:00:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.626+0000 7feda2000700 1 -- 172.21.15.28:0/597481302 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fed98003ab0 con 0x7feda4102550 2024-04-24T08:00:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.626+0000 7fedac600700 1 -- 172.21.15.28:0/597481302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feda419a650 con 0x7feda4102550 2024-04-24T08:00:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.626+0000 7fedac600700 1 -- 172.21.15.28:0/597481302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feda419d340 con 0x7feda4102550 2024-04-24T08:00:04.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.627+0000 7feda2000700 1 -- 172.21.15.28:0/597481302 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fed98029070 con 0x7feda4102550 2024-04-24T08:00:04.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.627+0000 7fedac600700 1 -- 172.21.15.28:0/597481302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feda4108d60 con 0x7feda4102550 2024-04-24T08:00:04.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.628+0000 7feda2000700 1 --2- 172.21.15.28:0/597481302 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fed8c039700 0x7fed8c03bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:04.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.628+0000 7feda2000700 1 -- 172.21.15.28:0/597481302 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fed9801e070 con 0x7feda4102550 2024-04-24T08:00:04.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.628+0000 7feda9200700 1 --2- 172.21.15.28:0/597481302 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fed8c039700 0x7fed8c03bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:04.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.629+0000 7feda9200700 1 --2- 172.21.15.28:0/597481302 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fed8c039700 0x7fed8c03bbc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fed90006fd0 tx=0x7fed90006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:04.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.635+0000 7feda2000700 1 -- 172.21.15.28:0/597481302 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed98013070 con 0x7feda4102550 2024-04-24T08:00:04.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.860+0000 7fedac600700 1 -- 172.21.15.28:0/597481302 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7feda402e0f0 con 0x7feda4102550 2024-04-24T08:00:04.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.867+0000 7feda2000700 1 -- 172.21.15.28:0/597481302 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fed980300c0 con 0x7feda4102550 2024-04-24T08:00:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.870+0000 7fedac600700 1 -- 172.21.15.28:0/597481302 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fed8c039700 msgr2=0x7fed8c03bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.870+0000 7fedac600700 1 --2- 172.21.15.28:0/597481302 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fed8c039700 0x7fed8c03bbc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fed90006fd0 tx=0x7fed90006e40 comp rx=0 tx=0).stop 2024-04-24T08:00:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.870+0000 7fedac600700 1 -- 172.21.15.28:0/597481302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda4102550 msgr2=0x7feda4199e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:04.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.870+0000 7fedac600700 1 --2- 172.21.15.28:0/597481302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda4102550 0x7feda4199e80 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fed98005850 tx=0x7fed98005030 comp rx=0 tx=0).stop 2024-04-24T08:00:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.870+0000 7fedac600700 1 -- 172.21.15.28:0/597481302 shutdown_connections 2024-04-24T08:00:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.870+0000 7fedac600700 1 --2- 172.21.15.28:0/597481302 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fed8c039700 0x7fed8c03bbc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.870+0000 7fedac600700 1 --2- 172.21.15.28:0/597481302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda4102550 0x7feda4199e80 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.870+0000 7fedac600700 1 -- 172.21.15.28:0/597481302 >> 172.21.15.28:0/597481302 conn(0x7feda40fdaf0 msgr2=0x7feda40fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.870+0000 7fedac600700 1 -- 172.21.15.28:0/597481302 shutdown_connections 2024-04-24T08:00:04.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:04.870+0000 7fedac600700 1 -- 172.21.15.28:0/597481302 wait complete. 2024-04-24T08:00:05.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:05.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:05.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi028", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:00:05.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:05.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:00:05.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:05.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:05 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/597481302' entity='client.admin' 2024-04-24T08:00:05.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:00:05.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:05.625 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-04-24T08:00:05.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-04-24T08:00:06.074 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:00:07.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.935+0000 7f6079c00700 1 -- 172.21.15.28:0/2410007277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6074102550 msgr2=0x7f6074102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:07.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.935+0000 7f6079c00700 1 --2- 172.21.15.28:0/2410007277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6074102550 0x7f6074102970 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f6064009910 tx=0x7f6064009c20 comp rx=0 tx=0).stop 2024-04-24T08:00:07.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.935+0000 7f6079c00700 1 -- 172.21.15.28:0/2410007277 shutdown_connections 2024-04-24T08:00:07.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.935+0000 7f6079c00700 1 --2- 172.21.15.28:0/2410007277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6074102550 0x7f6074102970 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:07.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.935+0000 7f6079c00700 1 -- 172.21.15.28:0/2410007277 >> 172.21.15.28:0/2410007277 conn(0x7f60740fdaf0 msgr2=0x7f60740fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:07.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.935+0000 7f6079c00700 1 -- 172.21.15.28:0/2410007277 shutdown_connections 2024-04-24T08:00:07.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.935+0000 7f6079c00700 1 -- 172.21.15.28:0/2410007277 wait complete. 2024-04-24T08:00:07.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.936+0000 7f6079c00700 1 Processor -- start 2024-04-24T08:00:07.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.937+0000 7f6079c00700 1 -- start start 2024-04-24T08:00:07.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.937+0000 7f6079c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6074102550 0x7f6074197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:07.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.937+0000 7f6079c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f60741085e0 con 0x7f6074102550 2024-04-24T08:00:07.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.937+0000 7f6072a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6074102550 0x7f6074197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:07.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.937+0000 7f6072a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6074102550 0x7f6074197c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58888/0 (socket says 172.21.15.28:58888) 2024-04-24T08:00:07.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.937+0000 7f6072a00700 1 -- 172.21.15.28:0/3261488751 learned_addr learned my addr 172.21.15.28:0/3261488751 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:00:07.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.938+0000 7f6072a00700 1 -- 172.21.15.28:0/3261488751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60741981a0 con 0x7f6074102550 2024-04-24T08:00:07.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.938+0000 7f6072a00700 1 --2- 172.21.15.28:0/3261488751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6074102550 0x7f6074197c60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6064005850 tx=0x7f6064005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:07.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.938+0000 7f606b400700 1 -- 172.21.15.28:0/3261488751 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6064026070 con 0x7f6074102550 2024-04-24T08:00:07.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.938+0000 7f606b400700 1 -- 172.21.15.28:0/3261488751 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60640044a0 con 0x7f6074102550 2024-04-24T08:00:07.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.939+0000 7f606b400700 1 -- 172.21.15.28:0/3261488751 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f6064003ab0 con 0x7f6074102550 2024-04-24T08:00:07.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.939+0000 7f6079c00700 1 -- 172.21.15.28:0/3261488751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6074198430 con 0x7f6074102550 2024-04-24T08:00:07.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.939+0000 7f6079c00700 1 -- 172.21.15.28:0/3261488751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f607419b120 con 0x7f6074102550 2024-04-24T08:00:07.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.940+0000 7f606b400700 1 -- 172.21.15.28:0/3261488751 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f6064029070 con 0x7f6074102550 2024-04-24T08:00:07.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.940+0000 7f6079c00700 1 -- 172.21.15.28:0/3261488751 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6074108d60 con 0x7f6074102550 2024-04-24T08:00:07.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.941+0000 7f606b400700 1 --2- 172.21.15.28:0/3261488751 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f60580396b0 0x7f605803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:07.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.941+0000 7f606b400700 1 -- 172.21.15.28:0/3261488751 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f606401e070 con 0x7f6074102550 2024-04-24T08:00:07.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.941+0000 7f606be00700 1 --2- 172.21.15.28:0/3261488751 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f60580396b0 0x7f605803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:07.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.942+0000 7f606be00700 1 --2- 172.21.15.28:0/3261488751 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f60580396b0 0x7f605803bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f605c006fd0 tx=0x7f605c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:07.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:07.946+0000 7f606b400700 1 -- 172.21.15.28:0/3261488751 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6064013070 con 0x7f6074102550 2024-04-24T08:00:08.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:08.183+0000 7f6079c00700 1 -- 172.21.15.28:0/3261488751 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f607402e180 con 0x7f60580396b0 2024-04-24T08:00:08.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:08.191+0000 7f606b400700 1 -- 172.21.15.28:0/3261488751 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f607402e180 con 0x7f60580396b0 2024-04-24T08:00:08.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:08.195+0000 7f6079c00700 1 -- 172.21.15.28:0/3261488751 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f60580396b0 msgr2=0x7f605803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:08.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:08.195+0000 7f6079c00700 1 --2- 172.21.15.28:0/3261488751 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f60580396b0 0x7f605803bb70 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f605c006fd0 tx=0x7f605c006e40 comp rx=0 tx=0).stop 2024-04-24T08:00:08.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:08.195+0000 7f6079c00700 1 -- 172.21.15.28:0/3261488751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6074102550 msgr2=0x7f6074197c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:08.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:08.195+0000 7f6079c00700 1 --2- 172.21.15.28:0/3261488751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6074102550 0x7f6074197c60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6064005850 tx=0x7f6064005030 comp rx=0 tx=0).stop 2024-04-24T08:00:08.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:08.195+0000 7f6079c00700 1 -- 172.21.15.28:0/3261488751 shutdown_connections 2024-04-24T08:00:08.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:08.195+0000 7f6079c00700 1 --2- 172.21.15.28:0/3261488751 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f60580396b0 0x7f605803bb70 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:08.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:08.195+0000 7f6079c00700 1 --2- 172.21.15.28:0/3261488751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6074102550 0x7f6074197c60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:08.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:08.195+0000 7f6079c00700 1 -- 172.21.15.28:0/3261488751 >> 172.21.15.28:0/3261488751 conn(0x7f60740fdaf0 msgr2=0x7f60740fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:08.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:08.195+0000 7f6079c00700 1 -- 172.21.15.28:0/3261488751 shutdown_connections 2024-04-24T08:00:08.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:08.195+0000 7f6079c00700 1 -- 172.21.15.28:0/3261488751 wait complete. 2024-04-24T08:00:08.995 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi117 2024-04-24T08:00:08.995 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-04-24T08:00:08.995 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/etc/ceph/ceph.conf 2024-04-24T08:00:09.015 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-04-24T08:00:09.015 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:00:09.072 INFO:tasks.cephadm:Adding host smithi117 to orchestrator... 2024-04-24T08:00:09.073 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch host add smithi117 2024-04-24T08:00:09.502 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:00:09.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:09 smithi028 ceph-mon[27786]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:00:09.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:09 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:09.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:09 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:00:09.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:09 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:00:09.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:09 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:00:09.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:09 smithi028 ceph-mon[27786]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-24T08:00:10.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:10 smithi028 ceph-mon[27786]: Updating smithi028:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:00:10.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:10 smithi028 ceph-mon[27786]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:00:11.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:11 smithi028 ceph-mon[27786]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:11.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:11 smithi028 ceph-mon[27786]: Updating smithi028:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:00:11.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:11 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:11.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:11 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:11.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:11 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:11.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.528+0000 7f4e8be00700 1 -- 172.21.15.28:0/2129555498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c074f20 msgr2=0x7f4e8c075300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:11.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.528+0000 7f4e8be00700 1 --2- 172.21.15.28:0/2129555498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c074f20 0x7f4e8c075300 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c009910 tx=0x7f4e7c009c20 comp rx=0 tx=0).stop 2024-04-24T08:00:11.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.528+0000 7f4e8be00700 1 -- 172.21.15.28:0/2129555498 shutdown_connections 2024-04-24T08:00:11.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.528+0000 7f4e8be00700 1 --2- 172.21.15.28:0/2129555498 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c074f20 0x7f4e8c075300 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:11.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.528+0000 7f4e8be00700 1 -- 172.21.15.28:0/2129555498 >> 172.21.15.28:0/2129555498 conn(0x7f4e8c0fde50 msgr2=0x7f4e8c100270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:11.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.528+0000 7f4e8be00700 1 -- 172.21.15.28:0/2129555498 shutdown_connections 2024-04-24T08:00:11.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.528+0000 7f4e8be00700 1 -- 172.21.15.28:0/2129555498 wait complete. 2024-04-24T08:00:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.529+0000 7f4e8be00700 1 Processor -- start 2024-04-24T08:00:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.529+0000 7f4e8be00700 1 -- start start 2024-04-24T08:00:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.530+0000 7f4e8be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c074f20 0x7f4e8c19a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.530+0000 7f4e8be00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4e8c10ebf0 con 0x7f4e8c074f20 2024-04-24T08:00:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.530+0000 7f4e8aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c074f20 0x7f4e8c19a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.530+0000 7f4e8aa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c074f20 0x7f4e8c19a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40336/0 (socket says 172.21.15.28:40336) 2024-04-24T08:00:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.530+0000 7f4e8aa00700 1 -- 172.21.15.28:0/4100098584 learned_addr learned my addr 172.21.15.28:0/4100098584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:00:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.530+0000 7f4e8aa00700 1 -- 172.21.15.28:0/4100098584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e8c19a630 con 0x7f4e8c074f20 2024-04-24T08:00:11.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.531+0000 7f4e8aa00700 1 --2- 172.21.15.28:0/4100098584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c074f20 0x7f4e8c19a0f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c006010 tx=0x7f4e7c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:11.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.531+0000 7f4e83400700 1 -- 172.21.15.28:0/4100098584 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e7c026070 con 0x7f4e8c074f20 2024-04-24T08:00:11.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.531+0000 7f4e83400700 1 -- 172.21.15.28:0/4100098584 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e7c00bc30 con 0x7f4e8c074f20 2024-04-24T08:00:11.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.531+0000 7f4e8be00700 1 -- 172.21.15.28:0/4100098584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e8c19a8c0 con 0x7f4e8c074f20 2024-04-24T08:00:11.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.531+0000 7f4e83400700 1 -- 172.21.15.28:0/4100098584 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f4e7c0039a0 con 0x7f4e8c074f20 2024-04-24T08:00:11.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.531+0000 7f4e8be00700 1 -- 172.21.15.28:0/4100098584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e8c19ad20 con 0x7f4e8c074f20 2024-04-24T08:00:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.532+0000 7f4e83400700 1 -- 172.21.15.28:0/4100098584 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f4e7c029070 con 0x7f4e8c074f20 2024-04-24T08:00:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.532+0000 7f4e8be00700 1 -- 172.21.15.28:0/4100098584 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4e8c0765d0 con 0x7f4e8c074f20 2024-04-24T08:00:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.533+0000 7f4e83400700 1 --2- 172.21.15.28:0/4100098584 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4e70039650 0x7f4e7003bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.533+0000 7f4e83400700 1 -- 172.21.15.28:0/4100098584 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f4e7c01e070 con 0x7f4e8c074f20 2024-04-24T08:00:11.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.533+0000 7f4e83e00700 1 --2- 172.21.15.28:0/4100098584 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4e70039650 0x7f4e7003bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:11.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.534+0000 7f4e83e00700 1 --2- 172.21.15.28:0/4100098584 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4e70039650 0x7f4e7003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4e74006fd0 tx=0x7f4e74006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:11.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.539+0000 7f4e83400700 1 -- 172.21.15.28:0/4100098584 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4e7c013070 con 0x7f4e8c074f20 2024-04-24T08:00:11.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:11.776+0000 7f4e8be00700 1 -- 172.21.15.28:0/4100098584 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi117", "target": ["mon-mgr", ""]}) -- 0x7f4e8c19b4c0 con 0x7f4e70039650 2024-04-24T08:00:13.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:13 smithi028 ceph-mon[27786]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi117", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:00:13.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:13 smithi028 ceph-mon[27786]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:13.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:13 smithi028 ceph-mon[27786]: Deploying cephadm binary to smithi117 2024-04-24T08:00:15.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:15 smithi028 ceph-mon[27786]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:16.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:16.625+0000 7f4e83400700 1 -- 172.21.15.28:0/4100098584 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f4e8c19b4c0 con 0x7f4e70039650 2024-04-24T08:00:16.626 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi117' with addr '172.21.15.117' 2024-04-24T08:00:16.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:16.628+0000 7f4e8be00700 1 -- 172.21.15.28:0/4100098584 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4e70039650 msgr2=0x7f4e7003bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:16.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:16.628+0000 7f4e8be00700 1 --2- 172.21.15.28:0/4100098584 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4e70039650 0x7f4e7003bb10 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f4e74006fd0 tx=0x7f4e74006e40 comp rx=0 tx=0).stop 2024-04-24T08:00:16.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:16.628+0000 7f4e8be00700 1 -- 172.21.15.28:0/4100098584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c074f20 msgr2=0x7f4e8c19a0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:16.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:16.628+0000 7f4e8be00700 1 --2- 172.21.15.28:0/4100098584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c074f20 0x7f4e8c19a0f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f4e7c006010 tx=0x7f4e7c005030 comp rx=0 tx=0).stop 2024-04-24T08:00:16.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:16.629+0000 7f4e8be00700 1 -- 172.21.15.28:0/4100098584 shutdown_connections 2024-04-24T08:00:16.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:16.629+0000 7f4e8be00700 1 --2- 172.21.15.28:0/4100098584 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4e70039650 0x7f4e7003bb10 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:16.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:16.629+0000 7f4e8be00700 1 --2- 172.21.15.28:0/4100098584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e8c074f20 0x7f4e8c19a0f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:16.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:16.629+0000 7f4e8be00700 1 -- 172.21.15.28:0/4100098584 >> 172.21.15.28:0/4100098584 conn(0x7f4e8c0fde50 msgr2=0x7f4e8c1065e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:16.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:16.629+0000 7f4e8be00700 1 -- 172.21.15.28:0/4100098584 shutdown_connections 2024-04-24T08:00:16.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:16.629+0000 7f4e8be00700 1 -- 172.21.15.28:0/4100098584 wait complete. 2024-04-24T08:00:17.398 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-24T08:00:17.862 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:00:18.022 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:17 smithi028 ceph-mon[27786]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:18.022 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:17 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:18.022 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:17 smithi028 ceph-mon[27786]: Added host smithi117 2024-04-24T08:00:18.022 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:17 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:00:18.022 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:17 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:19.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:18 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.867+0000 7f3834600700 1 -- 172.21.15.28:0/1896015195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f382c074e90 msgr2=0x7f382c075270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.867+0000 7f3834600700 1 --2- 172.21.15.28:0/1896015195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f382c074e90 0x7f382c075270 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3820009910 tx=0x7f3820009c20 comp rx=0 tx=0).stop 2024-04-24T08:00:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.867+0000 7f3834600700 1 -- 172.21.15.28:0/1896015195 shutdown_connections 2024-04-24T08:00:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.867+0000 7f3834600700 1 --2- 172.21.15.28:0/1896015195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f382c074e90 0x7f382c075270 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.867+0000 7f3834600700 1 -- 172.21.15.28:0/1896015195 >> 172.21.15.28:0/1896015195 conn(0x7f382c0fe030 msgr2=0x7f382c100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:19.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.867+0000 7f3834600700 1 -- 172.21.15.28:0/1896015195 shutdown_connections 2024-04-24T08:00:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.867+0000 7f3834600700 1 -- 172.21.15.28:0/1896015195 wait complete. 2024-04-24T08:00:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.869+0000 7f3834600700 1 Processor -- start 2024-04-24T08:00:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.869+0000 7f3834600700 1 -- start start 2024-04-24T08:00:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.869+0000 7f3834600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f382c074e90 0x7f382c19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:19.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.869+0000 7f3834600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f382c1131a0 con 0x7f382c074e90 2024-04-24T08:00:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.870+0000 7f3831c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f382c074e90 0x7f382c19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.870+0000 7f3831c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f382c074e90 0x7f382c19e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47060/0 (socket says 172.21.15.28:47060) 2024-04-24T08:00:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.870+0000 7f3831c00700 1 -- 172.21.15.28:0/2145074359 learned_addr learned my addr 172.21.15.28:0/2145074359 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:00:19.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.870+0000 7f3831c00700 1 -- 172.21.15.28:0/2145074359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f382c19ec10 con 0x7f382c074e90 2024-04-24T08:00:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.870+0000 7f3831c00700 1 --2- 172.21.15.28:0/2145074359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f382c074e90 0x7f382c19e6d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3820006010 tx=0x7f3820005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.871+0000 7f382a000700 1 -- 172.21.15.28:0/2145074359 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3820026070 con 0x7f382c074e90 2024-04-24T08:00:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.871+0000 7f382a000700 1 -- 172.21.15.28:0/2145074359 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f382000bc30 con 0x7f382c074e90 2024-04-24T08:00:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.871+0000 7f382a000700 1 -- 172.21.15.28:0/2145074359 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3820003ab0 con 0x7f382c074e90 2024-04-24T08:00:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.871+0000 7f3834600700 1 -- 172.21.15.28:0/2145074359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f382c19eea0 con 0x7f382c074e90 2024-04-24T08:00:19.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.871+0000 7f3834600700 1 -- 172.21.15.28:0/2145074359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f382c19f300 con 0x7f382c074e90 2024-04-24T08:00:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.872+0000 7f382a000700 1 -- 172.21.15.28:0/2145074359 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f3820029070 con 0x7f382c074e90 2024-04-24T08:00:19.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.872+0000 7f3834600700 1 -- 172.21.15.28:0/2145074359 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f382c0765d0 con 0x7f382c074e90 2024-04-24T08:00:19.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.873+0000 7f382a000700 1 --2- 172.21.15.28:0/2145074359 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f38140396b0 0x7f381403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:19.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.873+0000 7f382a000700 1 -- 172.21.15.28:0/2145074359 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f382001e070 con 0x7f382c074e90 2024-04-24T08:00:19.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.873+0000 7f3831200700 1 --2- 172.21.15.28:0/2145074359 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f38140396b0 0x7f381403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:19.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.874+0000 7f3831200700 1 --2- 172.21.15.28:0/2145074359 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f38140396b0 0x7f381403bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3818006fd0 tx=0x7f3818006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:19.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:19.879+0000 7f382a000700 1 -- 172.21.15.28:0/2145074359 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3820013070 con 0x7f382c074e90 2024-04-24T08:00:20.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:19 smithi028 ceph-mon[27786]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:20.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:19 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:20.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:20.114+0000 7f3834600700 1 -- 172.21.15.28:0/2145074359 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f382c19fa30 con 0x7f38140396b0 2024-04-24T08:00:20.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:20.115+0000 7f382a000700 1 -- 172.21.15.28:0/2145074359 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f382c19fa30 con 0x7f38140396b0 2024-04-24T08:00:20.116 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:00:20.116 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.117", "hostname": "smithi117", "labels": [], "status": ""}] 2024-04-24T08:00:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:20.118+0000 7f3834600700 1 -- 172.21.15.28:0/2145074359 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f38140396b0 msgr2=0x7f381403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:20.118+0000 7f3834600700 1 --2- 172.21.15.28:0/2145074359 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f38140396b0 0x7f381403bb70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3818006fd0 tx=0x7f3818006e40 comp rx=0 tx=0).stop 2024-04-24T08:00:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:20.118+0000 7f3834600700 1 -- 172.21.15.28:0/2145074359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f382c074e90 msgr2=0x7f382c19e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:20.118+0000 7f3834600700 1 --2- 172.21.15.28:0/2145074359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f382c074e90 0x7f382c19e6d0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f3820006010 tx=0x7f3820005050 comp rx=0 tx=0).stop 2024-04-24T08:00:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:20.118+0000 7f3834600700 1 -- 172.21.15.28:0/2145074359 shutdown_connections 2024-04-24T08:00:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:20.118+0000 7f3834600700 1 --2- 172.21.15.28:0/2145074359 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f38140396b0 0x7f381403bb70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:20.118+0000 7f3834600700 1 --2- 172.21.15.28:0/2145074359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f382c074e90 0x7f382c19e6d0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:20.118+0000 7f3834600700 1 -- 172.21.15.28:0/2145074359 >> 172.21.15.28:0/2145074359 conn(0x7f382c0fe030 msgr2=0x7f382c0ffa60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:20.119+0000 7f3834600700 1 -- 172.21.15.28:0/2145074359 shutdown_connections 2024-04-24T08:00:20.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:20.119+0000 7f3834600700 1 -- 172.21.15.28:0/2145074359 wait complete. 2024-04-24T08:00:20.859 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi194 2024-04-24T08:00:20.860 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-24T08:00:20.860 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.conf 2024-04-24T08:00:20.879 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-24T08:00:20.879 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:00:20.936 INFO:tasks.cephadm:Adding host smithi194 to orchestrator... 2024-04-24T08:00:20.937 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch host add smithi194 2024-04-24T08:00:21.375 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:00:21.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:21 smithi028 ceph-mon[27786]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:21.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:21 smithi028 ceph-mon[27786]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:00:23.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.410+0000 7f0f87000700 1 -- 172.21.15.28:0/2084375807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8010a710 msgr2=0x7f0f8010aaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:23.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.410+0000 7f0f87000700 1 --2- 172.21.15.28:0/2084375807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8010a710 0x7f0f8010aaf0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0f70009910 tx=0x7f0f70009c20 comp rx=0 tx=0).stop 2024-04-24T08:00:23.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.410+0000 7f0f87000700 1 -- 172.21.15.28:0/2084375807 shutdown_connections 2024-04-24T08:00:23.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.410+0000 7f0f87000700 1 --2- 172.21.15.28:0/2084375807 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8010a710 0x7f0f8010aaf0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:23.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.410+0000 7f0f87000700 1 -- 172.21.15.28:0/2084375807 >> 172.21.15.28:0/2084375807 conn(0x7f0f80100200 msgr2=0x7f0f80102620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:23.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.410+0000 7f0f87000700 1 -- 172.21.15.28:0/2084375807 shutdown_connections 2024-04-24T08:00:23.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.410+0000 7f0f87000700 1 -- 172.21.15.28:0/2084375807 wait complete. 2024-04-24T08:00:23.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.411+0000 7f0f87000700 1 Processor -- start 2024-04-24T08:00:23.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.412+0000 7f0f87000700 1 -- start start 2024-04-24T08:00:23.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.412+0000 7f0f87000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8010a710 0x7f0f8019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:23.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.412+0000 7f0f87000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0f8010ed50 con 0x7f0f8010a710 2024-04-24T08:00:23.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.413+0000 7f0f7fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8010a710 0x7f0f8019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:23.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.413+0000 7f0f7fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8010a710 0x7f0f8019a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47090/0 (socket says 172.21.15.28:47090) 2024-04-24T08:00:23.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.413+0000 7f0f7fe00700 1 -- 172.21.15.28:0/1958738846 learned_addr learned my addr 172.21.15.28:0/1958738846 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:00:23.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.413+0000 7f0f7fe00700 1 -- 172.21.15.28:0/1958738846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f8019a790 con 0x7f0f8010a710 2024-04-24T08:00:23.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.414+0000 7f0f7fe00700 1 --2- 172.21.15.28:0/1958738846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8010a710 0x7f0f8019a250 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0f70006010 tx=0x7f0f70005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.416+0000 7f0f7cc00700 1 -- 172.21.15.28:0/1958738846 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f70026070 con 0x7f0f8010a710 2024-04-24T08:00:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.416+0000 7f0f7cc00700 1 -- 172.21.15.28:0/1958738846 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f7000bbd0 con 0x7f0f8010a710 2024-04-24T08:00:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.416+0000 7f0f87000700 1 -- 172.21.15.28:0/1958738846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f8019aa80 con 0x7f0f8010a710 2024-04-24T08:00:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.416+0000 7f0f7cc00700 1 -- 172.21.15.28:0/1958738846 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0f70003910 con 0x7f0f8010a710 2024-04-24T08:00:23.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.416+0000 7f0f87000700 1 -- 172.21.15.28:0/1958738846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f8019eb20 con 0x7f0f8010a710 2024-04-24T08:00:23.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.417+0000 7f0f7cc00700 1 -- 172.21.15.28:0/1958738846 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f0f70029070 con 0x7f0f8010a710 2024-04-24T08:00:23.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.418+0000 7f0f87000700 1 -- 172.21.15.28:0/1958738846 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0f80105e50 con 0x7f0f8010a710 2024-04-24T08:00:23.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.418+0000 7f0f7cc00700 1 --2- 172.21.15.28:0/1958738846 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0f640396b0 0x7f0f6403bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:23.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.418+0000 7f0f7cc00700 1 -- 172.21.15.28:0/1958738846 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0f7001e070 con 0x7f0f8010a710 2024-04-24T08:00:23.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.418+0000 7f0f7f400700 1 --2- 172.21.15.28:0/1958738846 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0f640396b0 0x7f0f6403bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:23.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.419+0000 7f0f7f400700 1 --2- 172.21.15.28:0/1958738846 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0f640396b0 0x7f0f6403bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c006fd0 tx=0x7f0f6c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:23.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.424+0000 7f0f7cc00700 1 -- 172.21.15.28:0/1958738846 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0f70013070 con 0x7f0f8010a710 2024-04-24T08:00:23.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:23 smithi028 ceph-mon[27786]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:23.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:23 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:23.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:23 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:23.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:23 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:23.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:23 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:23.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:23 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi117", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:00:23.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:23 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:00:23.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:23 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:00:23.499 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:23 smithi028 ceph-mon[27786]: Updating smithi117:/etc/ceph/ceph.conf 2024-04-24T08:00:23.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:23.709+0000 7f0f87000700 1 -- 172.21.15.28:0/1958738846 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}) -- 0x7f0f8019b5b0 con 0x7f0f640396b0 2024-04-24T08:00:25.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:24 smithi028 ceph-mon[27786]: Updating smithi117:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:00:25.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:24 smithi028 ceph-mon[27786]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:00:25.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:24 smithi028 ceph-mon[27786]: Updating smithi117:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:00:25.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:24 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:25.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:24 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:25.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:24 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:25.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:24 smithi028 ceph-mon[27786]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:00:26.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:25 smithi028 ceph-mon[27786]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:26.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:25 smithi028 ceph-mon[27786]: Deploying cephadm binary to smithi194 2024-04-24T08:00:28.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:27 smithi028 ceph-mon[27786]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:28.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:28.547+0000 7f0f7cc00700 1 -- 172.21.15.28:0/1958738846 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7f0f8019b5b0 con 0x7f0f640396b0 2024-04-24T08:00:28.550 INFO:teuthology.orchestra.run.smithi028.stdout:Added host 'smithi194' with addr '172.21.15.194' 2024-04-24T08:00:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:28.551+0000 7f0f87000700 1 -- 172.21.15.28:0/1958738846 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0f640396b0 msgr2=0x7f0f6403bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:28.552+0000 7f0f87000700 1 --2- 172.21.15.28:0/1958738846 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0f640396b0 0x7f0f6403bb70 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0f6c006fd0 tx=0x7f0f6c006e40 comp rx=0 tx=0).stop 2024-04-24T08:00:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:28.552+0000 7f0f87000700 1 -- 172.21.15.28:0/1958738846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8010a710 msgr2=0x7f0f8019a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:28.552+0000 7f0f87000700 1 --2- 172.21.15.28:0/1958738846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8010a710 0x7f0f8019a250 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f0f70006010 tx=0x7f0f70005030 comp rx=0 tx=0).stop 2024-04-24T08:00:28.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:28.552+0000 7f0f87000700 1 -- 172.21.15.28:0/1958738846 shutdown_connections 2024-04-24T08:00:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:28.552+0000 7f0f87000700 1 --2- 172.21.15.28:0/1958738846 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0f640396b0 0x7f0f6403bb70 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:28.552+0000 7f0f87000700 1 --2- 172.21.15.28:0/1958738846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0f8010a710 0x7f0f8019a250 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:28.552+0000 7f0f87000700 1 -- 172.21.15.28:0/1958738846 >> 172.21.15.28:0/1958738846 conn(0x7f0f80100200 msgr2=0x7f0f80100ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:28.552+0000 7f0f87000700 1 -- 172.21.15.28:0/1958738846 shutdown_connections 2024-04-24T08:00:28.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:28.552+0000 7f0f87000700 1 -- 172.21.15.28:0/1958738846 wait complete. 2024-04-24T08:00:29.331 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch host ls --format=json 2024-04-24T08:00:29.795 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:00:29.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:29 smithi028 ceph-mon[27786]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:29.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:29.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:29 smithi028 ceph-mon[27786]: Added host smithi194 2024-04-24T08:00:29.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:00:29.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:30.960 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:30 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:31.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.858+0000 7fea6ee00700 1 -- 172.21.15.28:0/250577917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea68108580 msgr2=0x7fea68108960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:31.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.858+0000 7fea6ee00700 1 --2- 172.21.15.28:0/250577917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea68108580 0x7fea68108960 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fea58009910 tx=0x7fea58009c20 comp rx=0 tx=0).stop 2024-04-24T08:00:31.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.858+0000 7fea6ee00700 1 -- 172.21.15.28:0/250577917 shutdown_connections 2024-04-24T08:00:31.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.858+0000 7fea6ee00700 1 --2- 172.21.15.28:0/250577917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea68108580 0x7fea68108960 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:31.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.858+0000 7fea6ee00700 1 -- 172.21.15.28:0/250577917 >> 172.21.15.28:0/250577917 conn(0x7fea68076f50 msgr2=0x7fea68077360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:31.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.858+0000 7fea6ee00700 1 -- 172.21.15.28:0/250577917 shutdown_connections 2024-04-24T08:00:31.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.858+0000 7fea6ee00700 1 -- 172.21.15.28:0/250577917 wait complete. 2024-04-24T08:00:31.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.859+0000 7fea6ee00700 1 Processor -- start 2024-04-24T08:00:31.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.860+0000 7fea6ee00700 1 -- start start 2024-04-24T08:00:31.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.860+0000 7fea6ee00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea68108580 0x7fea6819a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:31.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.860+0000 7fea6ee00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fea6810cac0 con 0x7fea68108580 2024-04-24T08:00:31.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.860+0000 7fea67e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea68108580 0x7fea6819a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:31.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.860+0000 7fea67e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea68108580 0x7fea6819a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38162/0 (socket says 172.21.15.28:38162) 2024-04-24T08:00:31.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.860+0000 7fea67e00700 1 -- 172.21.15.28:0/1904551943 learned_addr learned my addr 172.21.15.28:0/1904551943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:00:31.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.861+0000 7fea67e00700 1 -- 172.21.15.28:0/1904551943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fea6819a6e0 con 0x7fea68108580 2024-04-24T08:00:31.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.861+0000 7fea67e00700 1 --2- 172.21.15.28:0/1904551943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea68108580 0x7fea6819a1a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fea58006010 tx=0x7fea58005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:31.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.861+0000 7fea64c00700 1 -- 172.21.15.28:0/1904551943 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea58026070 con 0x7fea68108580 2024-04-24T08:00:31.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.861+0000 7fea64c00700 1 -- 172.21.15.28:0/1904551943 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fea5800bc30 con 0x7fea68108580 2024-04-24T08:00:31.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.861+0000 7fea6ee00700 1 -- 172.21.15.28:0/1904551943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fea6819a9d0 con 0x7fea68108580 2024-04-24T08:00:31.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.861+0000 7fea64c00700 1 -- 172.21.15.28:0/1904551943 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fea580039d0 con 0x7fea68108580 2024-04-24T08:00:31.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.862+0000 7fea6ee00700 1 -- 172.21.15.28:0/1904551943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fea6819ea70 con 0x7fea68108580 2024-04-24T08:00:31.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.863+0000 7fea6ee00700 1 -- 172.21.15.28:0/1904551943 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fea68103c40 con 0x7fea68108580 2024-04-24T08:00:31.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.863+0000 7fea64c00700 1 -- 172.21.15.28:0/1904551943 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fea58029070 con 0x7fea68108580 2024-04-24T08:00:31.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.864+0000 7fea64c00700 1 --2- 172.21.15.28:0/1904551943 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fea4c039650 0x7fea4c03bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:31.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.864+0000 7fea64c00700 1 -- 172.21.15.28:0/1904551943 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fea5801e070 con 0x7fea68108580 2024-04-24T08:00:31.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.864+0000 7fea67400700 1 --2- 172.21.15.28:0/1904551943 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fea4c039650 0x7fea4c03bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:31.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.865+0000 7fea67400700 1 --2- 172.21.15.28:0/1904551943 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fea4c039650 0x7fea4c03bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fea54006fd0 tx=0x7fea54006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:31.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:31.869+0000 7fea64c00700 1 -- 172.21.15.28:0/1904551943 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fea58013070 con 0x7fea68108580 2024-04-24T08:00:32.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:31 smithi028 ceph-mon[27786]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:32.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:31 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:32.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:32.106+0000 7fea6ee00700 1 -- 172.21.15.28:0/1904551943 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fea6819b500 con 0x7fea4c039650 2024-04-24T08:00:32.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:32.108+0000 7fea64c00700 1 -- 172.21.15.28:0/1904551943 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7fea6819b500 con 0x7fea4c039650 2024-04-24T08:00:32.109 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:00:32.110 INFO:teuthology.orchestra.run.smithi028.stdout:[{"addr": "172.21.15.28", "hostname": "smithi028", "labels": [], "status": ""}, {"addr": "172.21.15.117", "hostname": "smithi117", "labels": [], "status": ""}, {"addr": "172.21.15.194", "hostname": "smithi194", "labels": [], "status": ""}] 2024-04-24T08:00:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:32.111+0000 7fea6ee00700 1 -- 172.21.15.28:0/1904551943 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fea4c039650 msgr2=0x7fea4c03bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:32.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:32.111+0000 7fea6ee00700 1 --2- 172.21.15.28:0/1904551943 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fea4c039650 0x7fea4c03bb10 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fea54006fd0 tx=0x7fea54006e40 comp rx=0 tx=0).stop 2024-04-24T08:00:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:32.111+0000 7fea6ee00700 1 -- 172.21.15.28:0/1904551943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea68108580 msgr2=0x7fea6819a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:32.111+0000 7fea6ee00700 1 --2- 172.21.15.28:0/1904551943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea68108580 0x7fea6819a1a0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fea58006010 tx=0x7fea58005030 comp rx=0 tx=0).stop 2024-04-24T08:00:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:32.111+0000 7fea6ee00700 1 -- 172.21.15.28:0/1904551943 shutdown_connections 2024-04-24T08:00:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:32.111+0000 7fea6ee00700 1 --2- 172.21.15.28:0/1904551943 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fea4c039650 0x7fea4c03bb10 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:32.111+0000 7fea6ee00700 1 --2- 172.21.15.28:0/1904551943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fea68108580 0x7fea6819a1a0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:32.111+0000 7fea6ee00700 1 -- 172.21.15.28:0/1904551943 >> 172.21.15.28:0/1904551943 conn(0x7fea68076f50 msgr2=0x7fea6810ab60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:32.111+0000 7fea6ee00700 1 -- 172.21.15.28:0/1904551943 shutdown_connections 2024-04-24T08:00:32.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:32.111+0000 7fea6ee00700 1 -- 172.21.15.28:0/1904551943 wait complete. 2024-04-24T08:00:32.874 INFO:tasks.cephadm:Setting crush tunables to default 2024-04-24T08:00:32.874 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd crush tunables default 2024-04-24T08:00:33.320 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:00:33.739 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:33 smithi028 ceph-mon[27786]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:33.739 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:33 smithi028 ceph-mon[27786]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:00:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.297+0000 7f5a7a200700 1 -- 172.21.15.28:0/2589456897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a741085e0 msgr2=0x7f5a741089c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.297+0000 7f5a7a200700 1 --2- 172.21.15.28:0/2589456897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a741085e0 0x7f5a741089c0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f5a6400b0a0 tx=0x7f5a6400b3b0 comp rx=0 tx=0).stop 2024-04-24T08:00:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.297+0000 7f5a7a200700 1 -- 172.21.15.28:0/2589456897 shutdown_connections 2024-04-24T08:00:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.297+0000 7f5a7a200700 1 --2- 172.21.15.28:0/2589456897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a741085e0 0x7f5a741089c0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.297+0000 7f5a7a200700 1 -- 172.21.15.28:0/2589456897 >> 172.21.15.28:0/2589456897 conn(0x7f5a740fe020 msgr2=0x7f5a74100440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.297+0000 7f5a7a200700 1 -- 172.21.15.28:0/2589456897 shutdown_connections 2024-04-24T08:00:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.297+0000 7f5a7a200700 1 -- 172.21.15.28:0/2589456897 wait complete. 2024-04-24T08:00:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.299+0000 7f5a7a200700 1 Processor -- start 2024-04-24T08:00:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.299+0000 7f5a7a200700 1 -- start start 2024-04-24T08:00:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.299+0000 7f5a7a200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a741085e0 0x7f5a74074df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:35.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.299+0000 7f5a7a200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a7410cba0 con 0x7f5a741085e0 2024-04-24T08:00:35.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.300+0000 7f5a73400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a741085e0 0x7f5a74074df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:35.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.300+0000 7f5a73400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a741085e0 0x7f5a74074df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38184/0 (socket says 172.21.15.28:38184) 2024-04-24T08:00:35.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.300+0000 7f5a73400700 1 -- 172.21.15.28:0/4081447519 learned_addr learned my addr 172.21.15.28:0/4081447519 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:00:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.300+0000 7f5a73400700 1 -- 172.21.15.28:0/4081447519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a74076ce0 con 0x7f5a741085e0 2024-04-24T08:00:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.301+0000 7f5a73400700 1 --2- 172.21.15.28:0/4081447519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a741085e0 0x7f5a74074df0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5a64000f80 tx=0x7f5a64003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.301+0000 7f5a6be00700 1 -- 172.21.15.28:0/4081447519 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a64017040 con 0x7f5a741085e0 2024-04-24T08:00:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.301+0000 7f5a6be00700 1 -- 172.21.15.28:0/4081447519 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a64022070 con 0x7f5a741085e0 2024-04-24T08:00:35.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.301+0000 7f5a6be00700 1 -- 172.21.15.28:0/4081447519 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f5a6401d440 con 0x7f5a741085e0 2024-04-24T08:00:35.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.301+0000 7f5a7a200700 1 -- 172.21.15.28:0/4081447519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a74075330 con 0x7f5a741085e0 2024-04-24T08:00:35.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.301+0000 7f5a7a200700 1 -- 172.21.15.28:0/4081447519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a740757f0 con 0x7f5a741085e0 2024-04-24T08:00:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.303+0000 7f5a7a200700 1 -- 172.21.15.28:0/4081447519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a74103c50 con 0x7f5a741085e0 2024-04-24T08:00:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.303+0000 7f5a6be00700 1 -- 172.21.15.28:0/4081447519 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f5a64004000 con 0x7f5a741085e0 2024-04-24T08:00:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.303+0000 7f5a6be00700 1 --2- 172.21.15.28:0/4081447519 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5a58039700 0x7f5a5803bbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.303+0000 7f5a6be00700 1 -- 172.21.15.28:0/4081447519 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f5a6400f050 con 0x7f5a741085e0 2024-04-24T08:00:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.304+0000 7f5a72a00700 1 --2- 172.21.15.28:0/4081447519 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5a58039700 0x7f5a5803bbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.305+0000 7f5a72a00700 1 --2- 172.21.15.28:0/4081447519 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5a58039700 0x7f5a5803bbc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5a60006fd0 tx=0x7f5a60006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:35.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.309+0000 7f5a6be00700 1 -- 172.21.15.28:0/4081447519 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5a64007e20 con 0x7f5a741085e0 2024-04-24T08:00:35.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:35.588+0000 7f5a7a200700 1 -- 172.21.15.28:0/4081447519 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7f5a740763c0 con 0x7f5a741085e0 2024-04-24T08:00:35.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:35 smithi028 ceph-mon[27786]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:35.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:35.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:35.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:35.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:35.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:00:35.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:00:35.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:00:35.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:35 smithi028 ceph-mon[27786]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-24T08:00:35.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:35 smithi028 ceph-mon[27786]: Updating smithi194:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:00:36.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:36.424+0000 7f5a6be00700 1 -- 172.21.15.28:0/4081447519 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7f5a640102f0 con 0x7f5a741085e0 2024-04-24T08:00:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:36.428+0000 7f5a7a200700 1 -- 172.21.15.28:0/4081447519 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5a58039700 msgr2=0x7f5a5803bbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:36.428+0000 7f5a7a200700 1 --2- 172.21.15.28:0/4081447519 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5a58039700 0x7f5a5803bbc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f5a60006fd0 tx=0x7f5a60006e40 comp rx=0 tx=0).stop 2024-04-24T08:00:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:36.428+0000 7f5a7a200700 1 -- 172.21.15.28:0/4081447519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a741085e0 msgr2=0x7f5a74074df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:36.428+0000 7f5a7a200700 1 --2- 172.21.15.28:0/4081447519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a741085e0 0x7f5a74074df0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5a64000f80 tx=0x7f5a64003ce0 comp rx=0 tx=0).stop 2024-04-24T08:00:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:36.428+0000 7f5a7a200700 1 -- 172.21.15.28:0/4081447519 shutdown_connections 2024-04-24T08:00:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:36.428+0000 7f5a7a200700 1 --2- 172.21.15.28:0/4081447519 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5a58039700 0x7f5a5803bbc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:36.428+0000 7f5a7a200700 1 --2- 172.21.15.28:0/4081447519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a741085e0 0x7f5a74074df0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:36.428+0000 7f5a7a200700 1 -- 172.21.15.28:0/4081447519 >> 172.21.15.28:0/4081447519 conn(0x7f5a740fe020 msgr2=0x7f5a740fe400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:36.428+0000 7f5a7a200700 1 -- 172.21.15.28:0/4081447519 shutdown_connections 2024-04-24T08:00:36.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:00:36.429+0000 7f5a7a200700 1 -- 172.21.15.28:0/4081447519 wait complete. 2024-04-24T08:00:36.431 INFO:teuthology.orchestra.run.smithi028.stderr:adjusted tunables profile to default 2024-04-24T08:00:36.751 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:36 smithi028 ceph-mon[27786]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:00:36.751 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:36 smithi028 ceph-mon[27786]: Updating smithi194:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:00:36.752 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:36 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/4081447519' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-24T08:00:36.752 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:36 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:36.752 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:36 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:36.752 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:36 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:37.038 INFO:tasks.cephadm:Adding mon.a on smithi028 2024-04-24T08:00:37.039 INFO:tasks.cephadm:Adding mon.b on smithi117 2024-04-24T08:00:37.039 INFO:tasks.cephadm:Adding mon.c on smithi194 2024-04-24T08:00:37.039 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch apply mon '3;smithi028:172.21.15.28=a;smithi117:172.21.15.117=b;smithi194:172.21.15.194=c' 2024-04-24T08:00:37.275 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:00:37.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:37 smithi028 ceph-mon[27786]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:37.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:37 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/4081447519' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-24T08:00:37.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:37 smithi028 ceph-mon[27786]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:00:38.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.836+0000 7fb231a60700 1 -- 172.21.15.194:0/151575272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c1085d0 msgr2=0x7fb22c1089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:38.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.836+0000 7fb231a60700 1 --2- 172.21.15.194:0/151575272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c1085d0 0x7fb22c1089b0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb21c00b0a0 tx=0x7fb21c00b3b0 comp rx=0 tx=0).stop 2024-04-24T08:00:38.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.836+0000 7fb231a60700 1 -- 172.21.15.194:0/151575272 shutdown_connections 2024-04-24T08:00:38.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.836+0000 7fb231a60700 1 --2- 172.21.15.194:0/151575272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c1085d0 0x7fb22c1089b0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:38.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.836+0000 7fb231a60700 1 -- 172.21.15.194:0/151575272 >> 172.21.15.194:0/151575272 conn(0x7fb22c0fe030 msgr2=0x7fb22c100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:38.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.837+0000 7fb231a60700 1 -- 172.21.15.194:0/151575272 shutdown_connections 2024-04-24T08:00:38.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.837+0000 7fb231a60700 1 -- 172.21.15.194:0/151575272 wait complete. 2024-04-24T08:00:38.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.837+0000 7fb231a60700 1 Processor -- start 2024-04-24T08:00:38.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.837+0000 7fb231a60700 1 -- start start 2024-04-24T08:00:38.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.838+0000 7fb231a60700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c1085d0 0x7fb22c197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:38.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.838+0000 7fb231a60700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb22c10cb90 con 0x7fb22c1085d0 2024-04-24T08:00:38.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.838+0000 7fb22affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c1085d0 0x7fb22c197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:38.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.838+0000 7fb22affd700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c1085d0 0x7fb22c197f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.194:56246/0 (socket says 172.21.15.194:56246) 2024-04-24T08:00:38.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.838+0000 7fb22affd700 1 -- 172.21.15.194:0/997928789 learned_addr learned my addr 172.21.15.194:0/997928789 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-24T08:00:38.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.839+0000 7fb22affd700 1 -- 172.21.15.194:0/997928789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb22c1984b0 con 0x7fb22c1085d0 2024-04-24T08:00:38.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.839+0000 7fb22affd700 1 --2- 172.21.15.194:0/997928789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c1085d0 0x7fb22c197f70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb21c000f80 tx=0x7fb21c003ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:38.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.839+0000 7fb230a5e700 1 -- 172.21.15.194:0/997928789 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb21c017040 con 0x7fb22c1085d0 2024-04-24T08:00:38.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.840+0000 7fb231a60700 1 -- 172.21.15.194:0/997928789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb22c198740 con 0x7fb22c1085d0 2024-04-24T08:00:38.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.840+0000 7fb231a60700 1 -- 172.21.15.194:0/997928789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb22c19c840 con 0x7fb22c1085d0 2024-04-24T08:00:38.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.840+0000 7fb230a5e700 1 -- 172.21.15.194:0/997928789 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb21c022070 con 0x7fb22c1085d0 2024-04-24T08:00:38.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.840+0000 7fb230a5e700 1 -- 172.21.15.194:0/997928789 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb21c004000 con 0x7fb22c1085d0 2024-04-24T08:00:38.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.840+0000 7fb231a60700 1 -- 172.21.15.194:0/997928789 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb22c103c40 con 0x7fb22c1085d0 2024-04-24T08:00:38.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.841+0000 7fb230a5e700 1 -- 172.21.15.194:0/997928789 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fb21c027460 con 0x7fb22c1085d0 2024-04-24T08:00:38.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.841+0000 7fb230a5e700 1 --2- 172.21.15.194:0/997928789 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2180396a0 0x7fb21803bb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:38.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.841+0000 7fb230a5e700 1 -- 172.21.15.194:0/997928789 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fb21c00f050 con 0x7fb22c1085d0 2024-04-24T08:00:38.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.842+0000 7fb22a7fc700 1 --2- 172.21.15.194:0/997928789 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2180396a0 0x7fb21803bb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:38.843 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.843+0000 7fb22a7fc700 1 --2- 172.21.15.194:0/997928789 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2180396a0 0x7fb21803bb60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb214006fd0 tx=0x7fb214006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:38.857 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:38.857+0000 7fb230a5e700 1 -- 172.21.15.194:0/997928789 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb21c007e20 con 0x7fb22c1085d0 2024-04-24T08:00:39.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:39.067+0000 7fb231a60700 1 -- 172.21.15.194:0/997928789 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi117:172.21.15.117=b;smithi194:172.21.15.194=c", "target": ["mon-mgr", ""]}) -- 0x7fb22c1992d0 con 0x7fb2180396a0 2024-04-24T08:00:39.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:39.082+0000 7fb230a5e700 1 -- 172.21.15.194:0/997928789 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb22c1992d0 con 0x7fb2180396a0 2024-04-24T08:00:39.083 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mon update... 2024-04-24T08:00:39.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:39.084+0000 7fb231a60700 1 -- 172.21.15.194:0/997928789 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2180396a0 msgr2=0x7fb21803bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:39.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:39.084+0000 7fb231a60700 1 --2- 172.21.15.194:0/997928789 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2180396a0 0x7fb21803bb60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb214006fd0 tx=0x7fb214006e40 comp rx=0 tx=0).stop 2024-04-24T08:00:39.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:39.084+0000 7fb231a60700 1 -- 172.21.15.194:0/997928789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c1085d0 msgr2=0x7fb22c197f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:39.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:39.084+0000 7fb231a60700 1 --2- 172.21.15.194:0/997928789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c1085d0 0x7fb22c197f70 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fb21c000f80 tx=0x7fb21c003ce0 comp rx=0 tx=0).stop 2024-04-24T08:00:39.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:39.084+0000 7fb231a60700 1 -- 172.21.15.194:0/997928789 shutdown_connections 2024-04-24T08:00:39.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:39.084+0000 7fb231a60700 1 --2- 172.21.15.194:0/997928789 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2180396a0 0x7fb21803bb60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:39.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:39.084+0000 7fb231a60700 1 --2- 172.21.15.194:0/997928789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb22c1085d0 0x7fb22c197f70 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:39.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:39.084+0000 7fb231a60700 1 -- 172.21.15.194:0/997928789 >> 172.21.15.194:0/997928789 conn(0x7fb22c0fe030 msgr2=0x7fb22c0ff790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:39.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:39.085+0000 7fb231a60700 1 -- 172.21.15.194:0/997928789 shutdown_connections 2024-04-24T08:00:39.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:39.085+0000 7fb231a60700 1 -- 172.21.15.194:0/997928789 wait complete. 2024-04-24T08:00:39.742 DEBUG:teuthology.orchestra.run.smithi117:mon.b> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mon.b.service 2024-04-24T08:00:39.744 DEBUG:teuthology.orchestra.run.smithi194:mon.c> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mon.c.service 2024-04-24T08:00:39.747 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-24T08:00:39.747 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-24T08:00:39.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:39 smithi028 ceph-mon[27786]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:39.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:39 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:39.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:39 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:00:39.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:39 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:00:39.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:39 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:00:39.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:39 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:39.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:39 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:00:39.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:39 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:00:40.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:40 smithi028 ceph-mon[27786]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi117:172.21.15.117=b;smithi194:172.21.15.194=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:00:40.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:40 smithi028 ceph-mon[27786]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi117:172.21.15.117=b;smithi194:172.21.15.194=c;count:3 2024-04-24T08:00:40.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:40 smithi028 ceph-mon[27786]: Deploying daemon mon.c on smithi194 2024-04-24T08:00:41.413 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:00:41.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:41 smithi028 ceph-mon[27786]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:43.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.633+0000 7f09fee53700 1 -- 172.21.15.194:0/1729942502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09f81033f0 msgr2=0x7f09f81037d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:43.635 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.633+0000 7f09fee53700 1 --2- 172.21.15.194:0/1729942502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09f81033f0 0x7f09f81037d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f09ec009910 tx=0x7f09ec009c20 comp rx=0 tx=0).stop 2024-04-24T08:00:43.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.633+0000 7f09fee53700 1 -- 172.21.15.194:0/1729942502 shutdown_connections 2024-04-24T08:00:43.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.633+0000 7f09fee53700 1 --2- 172.21.15.194:0/1729942502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09f81033f0 0x7f09f81037d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:43.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.633+0000 7f09fee53700 1 -- 172.21.15.194:0/1729942502 >> 172.21.15.194:0/1729942502 conn(0x7f09f8077b70 msgr2=0x7f09f8077f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:43.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.633+0000 7f09fee53700 1 -- 172.21.15.194:0/1729942502 shutdown_connections 2024-04-24T08:00:43.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.633+0000 7f09fee53700 1 -- 172.21.15.194:0/1729942502 wait complete. 2024-04-24T08:00:43.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.634+0000 7f09fee53700 1 Processor -- start 2024-04-24T08:00:43.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.634+0000 7f09fee53700 1 -- start start 2024-04-24T08:00:43.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.634+0000 7f09fee53700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09f81033f0 0x7f09f819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:43.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.634+0000 7f09fee53700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f09f810edb0 con 0x7f09f81033f0 2024-04-24T08:00:43.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.635+0000 7f09fcbef700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09f81033f0 0x7f09f819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:43.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.635+0000 7f09fcbef700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09f81033f0 0x7f09f819a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.194:56272/0 (socket says 172.21.15.194:56272) 2024-04-24T08:00:43.636 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.635+0000 7f09fcbef700 1 -- 172.21.15.194:0/2868557011 learned_addr learned my addr 172.21.15.194:0/2868557011 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-24T08:00:43.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.636+0000 7f09fcbef700 1 -- 172.21.15.194:0/2868557011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09f819a7d0 con 0x7f09f81033f0 2024-04-24T08:00:43.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.636+0000 7f09fcbef700 1 --2- 172.21.15.194:0/2868557011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09f81033f0 0x7f09f819a290 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f09ec006010 tx=0x7f09ec005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:43.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.636+0000 7f09f5ffb700 1 -- 172.21.15.194:0/2868557011 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09ec026070 con 0x7f09f81033f0 2024-04-24T08:00:43.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.636+0000 7f09fee53700 1 -- 172.21.15.194:0/2868557011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09f819aa60 con 0x7f09f81033f0 2024-04-24T08:00:43.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.637+0000 7f09fee53700 1 -- 172.21.15.194:0/2868557011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09f819aec0 con 0x7f09f81033f0 2024-04-24T08:00:43.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.637+0000 7f09f5ffb700 1 -- 172.21.15.194:0/2868557011 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09ec00bc30 con 0x7f09f81033f0 2024-04-24T08:00:43.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.637+0000 7f09f5ffb700 1 -- 172.21.15.194:0/2868557011 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f09ec003950 con 0x7f09f81033f0 2024-04-24T08:00:43.638 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.637+0000 7f09fee53700 1 -- 172.21.15.194:0/2868557011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f09dc005330 con 0x7f09f81033f0 2024-04-24T08:00:43.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.638+0000 7f09f5ffb700 1 -- 172.21.15.194:0/2868557011 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f09ec029070 con 0x7f09f81033f0 2024-04-24T08:00:43.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.638+0000 7f09f5ffb700 1 --2- 172.21.15.194:0/2868557011 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f09e80396b0 0x7f09e803bb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:43.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.638+0000 7f09f5ffb700 1 -- 172.21.15.194:0/2868557011 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f09ec01e070 con 0x7f09f81033f0 2024-04-24T08:00:43.639 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.639+0000 7f09f7fff700 1 --2- 172.21.15.194:0/2868557011 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f09e80396b0 0x7f09e803bb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:43.640 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.640+0000 7f09f7fff700 1 --2- 172.21.15.194:0/2868557011 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f09e80396b0 0x7f09e803bb70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f09e4006fd0 tx=0x7f09e4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:43.653 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.653+0000 7f09f5ffb700 1 -- 172.21.15.194:0/2868557011 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f09ec013070 con 0x7f09f81033f0 2024-04-24T08:00:43.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:43 smithi028 ceph-mon[27786]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:43.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.952+0000 7f09fee53700 1 -- 172.21.15.194:0/2868557011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f09dc0058f0 con 0x7f09f81033f0 2024-04-24T08:00:43.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.964+0000 7f09f5ffb700 1 -- 172.21.15.194:0/2868557011 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7f09ec00e3e0 con 0x7f09f81033f0 2024-04-24T08:00:43.965 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T08:00:43.966 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","modified":"2024-04-24T07:58:38.163806Z","created":"2024-04-24T07:58:38.163806Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-04-24T08:00:43.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.967+0000 7f09fee53700 1 -- 172.21.15.194:0/2868557011 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f09e80396b0 msgr2=0x7f09e803bb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:43.967 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.967+0000 7f09fee53700 1 --2- 172.21.15.194:0/2868557011 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f09e80396b0 0x7f09e803bb70 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f09e4006fd0 tx=0x7f09e4006e40 comp rx=0 tx=0).stop 2024-04-24T08:00:43.968 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.967+0000 7f09fee53700 1 -- 172.21.15.194:0/2868557011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09f81033f0 msgr2=0x7f09f819a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:43.968 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.967+0000 7f09fee53700 1 --2- 172.21.15.194:0/2868557011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09f81033f0 0x7f09f819a290 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f09ec006010 tx=0x7f09ec005030 comp rx=0 tx=0).stop 2024-04-24T08:00:43.968 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.967+0000 7f09fee53700 1 -- 172.21.15.194:0/2868557011 shutdown_connections 2024-04-24T08:00:43.968 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.967+0000 7f09fee53700 1 --2- 172.21.15.194:0/2868557011 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f09e80396b0 0x7f09e803bb70 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:43.968 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.967+0000 7f09fee53700 1 --2- 172.21.15.194:0/2868557011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f09f81033f0 0x7f09f819a290 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:43.968 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.967+0000 7f09fee53700 1 -- 172.21.15.194:0/2868557011 >> 172.21.15.194:0/2868557011 conn(0x7f09f8077b70 msgr2=0x7f09f810a1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:43.968 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.967+0000 7f09fee53700 1 -- 172.21.15.194:0/2868557011 shutdown_connections 2024-04-24T08:00:43.968 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:43.967+0000 7f09fee53700 1 -- 172.21.15.194:0/2868557011 wait complete. 2024-04-24T08:00:43.968 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-04-24T08:00:45.603 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-24T08:00:45.604 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-24T08:00:45.842 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:00:49.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:49.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:00:49.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: mon.a calling monitor election 2024-04-24T08:00:49.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:00:49.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:00:49.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:49.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:00:49.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: mon.c calling monitor election 2024-04-24T08:00:49.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:00:49.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:49.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:00:49.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:00:49.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T08:00:49.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: monmap epoch 2 2024-04-24T08:00:49.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: fsid 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:00:49.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: last_changed 2024-04-24T08:00:43.994093+0000 2024-04-24T08:00:49.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: created 2024-04-24T07:58:38.163806+0000 2024-04-24T08:00:49.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: min_mon_release 19 (squid) 2024-04-24T08:00:49.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: election_strategy: 1 2024-04-24T08:00:49.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-24T08:00:49.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-04-24T08:00:49.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: fsmap 2024-04-24T08:00:49.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:00:49.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: mgrmap e13: x(active, since 59s) 2024-04-24T08:00:49.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: overall HEALTH_OK 2024-04-24T08:00:49.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:49.542 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:50.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.313+0000 7f79c159d700 1 -- 172.21.15.194:0/1242588815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79a40035c0 msgr2=0x7f79a4005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:50.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.313+0000 7f79c159d700 1 --2- 172.21.15.194:0/1242588815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79a40035c0 0x7f79a4005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f79bc101460 tx=0x7f79ac00feb0 comp rx=0 tx=0).stop 2024-04-24T08:00:50.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.313+0000 7f79c159d700 1 -- 172.21.15.194:0/1242588815 shutdown_connections 2024-04-24T08:00:50.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.313+0000 7f79c159d700 1 --2- 172.21.15.194:0/1242588815 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79a40035c0 0x7f79a4005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:50.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.313+0000 7f79c159d700 1 -- 172.21.15.194:0/1242588815 >> 172.21.15.194:0/1242588815 conn(0x7f79bc0fd040 msgr2=0x7f79bc0ff460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:50.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.313+0000 7f79c159d700 1 -- 172.21.15.194:0/1242588815 shutdown_connections 2024-04-24T08:00:50.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.313+0000 7f79c159d700 1 -- 172.21.15.194:0/1242588815 wait complete. 2024-04-24T08:00:50.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.314+0000 7f79c159d700 1 Processor -- start 2024-04-24T08:00:50.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.314+0000 7f79c159d700 1 -- start start 2024-04-24T08:00:50.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.315+0000 7f79c159d700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79a40035c0 0x7f79bc1031c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:50.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.315+0000 7f79c159d700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79bc103700 0x7f79bc103b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:50.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.315+0000 7f79c159d700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f79bc1095d0 con 0x7f79bc103700 2024-04-24T08:00:50.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.315+0000 7f79c159d700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f79bc109450 con 0x7f79a40035c0 2024-04-24T08:00:50.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.315+0000 7f79baffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79a40035c0 0x7f79bc1031c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:50.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.315+0000 7f79baffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79a40035c0 0x7f79bc1031c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:54748/0 (socket says 172.21.15.194:54748) 2024-04-24T08:00:50.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.315+0000 7f79baffd700 1 -- 172.21.15.194:0/3753951099 learned_addr learned my addr 172.21.15.194:0/3753951099 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-24T08:00:50.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.315+0000 7f79baffd700 1 -- 172.21.15.194:0/3753951099 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79a40035c0 msgr2=0x7f79bc1031c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-04-24T08:00:50.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.315+0000 7f79baffd700 1 -- 172.21.15.194:0/3753951099 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79a40035c0 msgr2=0x7f79bc1031c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-24T08:00:50.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.315+0000 7f79baffd700 1 --2- 172.21.15.194:0/3753951099 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79a40035c0 0x7f79bc1031c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T08:00:50.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.315+0000 7f79baffd700 1 --2- 172.21.15.194:0/3753951099 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79a40035c0 0x7f79bc1031c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:00:50.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.315+0000 7f79ba7fc700 1 --2- 172.21.15.194:0/3753951099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79bc103700 0x7f79bc103b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:50.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.316+0000 7f79ba7fc700 1 -- 172.21.15.194:0/3753951099 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79a40035c0 msgr2=0x7f79bc1031c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:00:50.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.316+0000 7f79ba7fc700 1 --2- 172.21.15.194:0/3753951099 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79a40035c0 0x7f79bc1031c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:50.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.316+0000 7f79ba7fc700 1 -- 172.21.15.194:0/3753951099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f79bc1050e0 con 0x7f79bc103700 2024-04-24T08:00:50.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.316+0000 7f79ba7fc700 1 --2- 172.21.15.194:0/3753951099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79bc103700 0x7f79bc103b80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f79b000b4d0 tx=0x7f79b000b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:50.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.317+0000 7f79a3fff700 1 -- 172.21.15.194:0/3753951099 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f79b0011630 con 0x7f79bc103700 2024-04-24T08:00:50.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.317+0000 7f79c159d700 1 -- 172.21.15.194:0/3753951099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f79bc1a4460 con 0x7f79bc103700 2024-04-24T08:00:50.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.317+0000 7f79c159d700 1 -- 172.21.15.194:0/3753951099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f79bc1a49a0 con 0x7f79bc103700 2024-04-24T08:00:50.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.317+0000 7f79a3fff700 1 -- 172.21.15.194:0/3753951099 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f79b0011cb0 con 0x7f79bc103700 2024-04-24T08:00:50.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.317+0000 7f79a3fff700 1 -- 172.21.15.194:0/3753951099 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7f79b0010e60 con 0x7f79bc103700 2024-04-24T08:00:50.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.318+0000 7f79c159d700 1 -- 172.21.15.194:0/3753951099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f79bc102c70 con 0x7f79bc103700 2024-04-24T08:00:50.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.318+0000 7f79a3fff700 1 -- 172.21.15.194:0/3753951099 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f79b0018020 con 0x7f79bc103700 2024-04-24T08:00:50.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.319+0000 7f79a3fff700 1 --2- 172.21.15.194:0/3753951099 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f79a80393a0 0x7f79a803b860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:50.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.319+0000 7f79a3fff700 1 -- 172.21.15.194:0/3753951099 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f79b004c3b0 con 0x7f79bc103700 2024-04-24T08:00:50.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.319+0000 7f79baffd700 1 --2- 172.21.15.194:0/3753951099 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f79a80393a0 0x7f79a803b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:50.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.321+0000 7f79baffd700 1 --2- 172.21.15.194:0/3753951099 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f79a80393a0 0x7f79a803b860 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f79ac00fa60 tx=0x7f79ac005a90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:50.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.335+0000 7f79a3fff700 1 -- 172.21.15.194:0/3753951099 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f79b001f020 con 0x7f79bc103700 2024-04-24T08:00:50.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:50.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:00:50.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:00:50.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:50 smithi028 ceph-mon[27786]: Deploying daemon mon.b on smithi117 2024-04-24T08:00:50.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:00:50.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.616+0000 7f79c159d700 1 -- 172.21.15.194:0/3753951099 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f79bc1a4d90 con 0x7f79bc103700 2024-04-24T08:00:50.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.618+0000 7f79a3fff700 1 -- 172.21.15.194:0/3753951099 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7f79b001e1b0 con 0x7f79bc103700 2024-04-24T08:00:50.618 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T08:00:50.618 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":2,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","modified":"2024-04-24T08:00:43.994093Z","created":"2024-04-24T07:58:38.163806Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-24T08:00:50.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.620+0000 7f79c159d700 1 -- 172.21.15.194:0/3753951099 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f79a80393a0 msgr2=0x7f79a803b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:50.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.620+0000 7f79c159d700 1 --2- 172.21.15.194:0/3753951099 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f79a80393a0 0x7f79a803b860 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f79ac00fa60 tx=0x7f79ac005a90 comp rx=0 tx=0).stop 2024-04-24T08:00:50.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.620+0000 7f79c159d700 1 -- 172.21.15.194:0/3753951099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79bc103700 msgr2=0x7f79bc103b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:50.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.620+0000 7f79c159d700 1 --2- 172.21.15.194:0/3753951099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79bc103700 0x7f79bc103b80 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f79b000b4d0 tx=0x7f79b000b890 comp rx=0 tx=0).stop 2024-04-24T08:00:50.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.620+0000 7f79c159d700 1 -- 172.21.15.194:0/3753951099 shutdown_connections 2024-04-24T08:00:50.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.620+0000 7f79c159d700 1 --2- 172.21.15.194:0/3753951099 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f79a80393a0 0x7f79a803b860 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:50.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.620+0000 7f79c159d700 1 --2- 172.21.15.194:0/3753951099 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f79a40035c0 0x7f79bc1031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:50.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.620+0000 7f79c159d700 1 --2- 172.21.15.194:0/3753951099 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f79bc103700 0x7f79bc103b80 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:50.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.620+0000 7f79c159d700 1 -- 172.21.15.194:0/3753951099 >> 172.21.15.194:0/3753951099 conn(0x7f79bc0fd040 msgr2=0x7f79bc06b100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:50.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.620+0000 7f79c159d700 1 -- 172.21.15.194:0/3753951099 shutdown_connections 2024-04-24T08:00:50.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:50.620+0000 7f79c159d700 1 -- 172.21.15.194:0/3753951099 wait complete. 2024-04-24T08:00:50.621 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 2 2024-04-24T08:00:51.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:51 smithi194 ceph-mon[32836]: from='client.? 172.21.15.194:0/3753951099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T08:00:51.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:51 smithi028 ceph-mon[27786]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:51.367 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:51 smithi028 ceph-mon[27786]: from='client.? 172.21.15.194:0/3753951099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T08:00:52.235 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-04-24T08:00:52.236 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph mon dump -f json 2024-04-24T08:00:52.478 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:00:53.133 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: Updating smithi117:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:00:53.133 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.133 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.134 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.134 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:00:53.134 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:53.134 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: Deploying cephadm binary to smithi194 2024-04-24T08:00:53.134 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:53.134 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:53.134 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.134 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: Added host smithi194 2024-04-24T08:00:53.134 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:00:53.134 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.134 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.135 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:53.135 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.135 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:53.135 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:00:53.135 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:53.135 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.135 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.135 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.135 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.135 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi194", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:00:53.135 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:00:53.135 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:00:53.135 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-24T08:00:53.136 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: Updating smithi194:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:00:53.136 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:00:53.136 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: Updating smithi194:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:00:53.136 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/4081447519' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-04-24T08:00:53.136 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.136 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.136 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.136 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:53.136 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/4081447519' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-04-24T08:00:53.136 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:00:53.136 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:53.136 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.136 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:00:53.136 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:00:53.137 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:00:53.137 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.137 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:00:53.137 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:00:53.137 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi028:172.21.15.28=a;smithi117:172.21.15.117=b;smithi194:172.21.15.194=c", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:00:53.137 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: Saving service mon spec with placement smithi028:172.21.15.28=a;smithi117:172.21.15.117=b;smithi194:172.21.15.194=c;count:3 2024-04-24T08:00:53.137 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: Deploying daemon mon.c on smithi194 2024-04-24T08:00:53.137 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:53.137 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:53.137 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:53.137 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:00:53.137 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: mon.a calling monitor election 2024-04-24T08:00:53.137 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:00:53.137 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:00:53.138 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:53.138 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:00:53.138 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: mon.c calling monitor election 2024-04-24T08:00:53.138 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:00:53.138 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:53.138 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:00:53.138 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:00:53.138 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T08:00:53.138 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: monmap epoch 2 2024-04-24T08:00:53.138 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: fsid 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:00:53.138 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: last_changed 2024-04-24T08:00:43.994093+0000 2024-04-24T08:00:53.138 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: created 2024-04-24T07:58:38.163806+0000 2024-04-24T08:00:53.138 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: min_mon_release 19 (squid) 2024-04-24T08:00:53.139 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: election_strategy: 1 2024-04-24T08:00:53.139 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-24T08:00:53.139 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-04-24T08:00:53.139 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: fsmap 2024-04-24T08:00:53.139 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:00:53.139 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: mgrmap e13: x(active, since 59s) 2024-04-24T08:00:53.139 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: overall HEALTH_OK 2024-04-24T08:00:53.139 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.139 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.139 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:53.140 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:00:53.140 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:00:53.140 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: Deploying daemon mon.b on smithi117 2024-04-24T08:00:53.140 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:00:53.140 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:53.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:53 smithi194 ceph-mon[32836]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:53.439 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: from='client.? 172.21.15.194:0/3753951099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T08:00:53.439 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:53.439 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 ceph-mon[32438]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-04-24T08:00:53.439 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 bash[32417]: abc0e1bdf477d099e6c87e2d251156b31f37696cb0f0292bcab10324a32a57db 2024-04-24T08:00:53.440 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:00:53 smithi117 systemd[1]: Started Ceph mon.b for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T08:00:53.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:53 smithi028 ceph-mon[27786]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:58.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.245+0000 7f78d8fa0700 1 -- 172.21.15.194:0/1499770459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc0035c0 msgr2=0x7f78bc005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:58.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.245+0000 7f78d8fa0700 1 --2- 172.21.15.194:0/1499770459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc0035c0 0x7f78bc005a50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f78d4102540 tx=0x7f78c800fe10 comp rx=0 tx=0).stop 2024-04-24T08:00:58.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.246+0000 7f78d8fa0700 1 -- 172.21.15.194:0/1499770459 shutdown_connections 2024-04-24T08:00:58.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.246+0000 7f78d8fa0700 1 --2- 172.21.15.194:0/1499770459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc0035c0 0x7f78bc005a50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:58.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.246+0000 7f78d8fa0700 1 -- 172.21.15.194:0/1499770459 >> 172.21.15.194:0/1499770459 conn(0x7f78d40fe100 msgr2=0x7f78d4100520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:58.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.246+0000 7f78d8fa0700 1 -- 172.21.15.194:0/1499770459 shutdown_connections 2024-04-24T08:00:58.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.246+0000 7f78d8fa0700 1 -- 172.21.15.194:0/1499770459 wait complete. 2024-04-24T08:00:58.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.247+0000 7f78d8fa0700 1 Processor -- start 2024-04-24T08:00:58.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.247+0000 7f78d8fa0700 1 -- start start 2024-04-24T08:00:58.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.247+0000 7f78d8fa0700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc0035c0 0x7f78d41981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:58.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.247+0000 7f78d8fa0700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78d4198720 0x7f78d419cab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:58.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.247+0000 7f78d8fa0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78d419cff0 0x7f78d419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:58.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.247+0000 7f78d8fa0700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f78d410a680 con 0x7f78bc0035c0 2024-04-24T08:00:58.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.247+0000 7f78d8fa0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f78d410a500 con 0x7f78d419cff0 2024-04-24T08:00:58.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.247+0000 7f78d8fa0700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f78d410a800 con 0x7f78d4198720 2024-04-24T08:00:58.249 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.248+0000 7f78d1d9b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78d4198720 0x7f78d419cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:58.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.248+0000 7f78d1d9b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78d4198720 0x7f78d419cab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:46608/0 (socket says 172.21.15.194:46608) 2024-04-24T08:00:58.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.248+0000 7f78d1d9b700 1 -- 172.21.15.194:0/2915543938 learned_addr learned my addr 172.21.15.194:0/2915543938 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-24T08:00:58.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.248+0000 7f78d2d9d700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78d419cff0 0x7f78d419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:58.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.248+0000 7f78d1d9b700 1 -- 172.21.15.194:0/2915543938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78d4198720 msgr2=0x7f78d419cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-04-24T08:00:58.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.248+0000 7f78d1d9b700 1 -- 172.21.15.194:0/2915543938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78d4198720 msgr2=0x7f78d419cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-24T08:00:58.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.248+0000 7f78d259c700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc0035c0 0x7f78d41981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:58.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.248+0000 7f78d1d9b700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78d4198720 0x7f78d419cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T08:00:58.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.248+0000 7f78d2d9d700 1 -- 172.21.15.194:0/2915543938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78d419cff0 msgr2=0x7f78d419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-04-24T08:00:58.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.248+0000 7f78d2d9d700 1 -- 172.21.15.194:0/2915543938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78d419cff0 msgr2=0x7f78d419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-04-24T08:00:58.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.248+0000 7f78d2d9d700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78d419cff0 0x7f78d419d4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-04-24T08:00:58.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.248+0000 7f78d1d9b700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78d4198720 0x7f78d419cab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:00:58.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.248+0000 7f78d2d9d700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78d419cff0 0x7f78d419d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:00:58.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.249+0000 7f78d259c700 1 -- 172.21.15.194:0/2915543938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78d4198720 msgr2=0x7f78d419cab0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:00:58.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.249+0000 7f78d259c700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78d4198720 0x7f78d419cab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:58.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.249+0000 7f78d259c700 1 -- 172.21.15.194:0/2915543938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78d419cff0 msgr2=0x7f78d419d4a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:00:58.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.249+0000 7f78d259c700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78d419cff0 0x7f78d419d4a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:58.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.249+0000 7f78d259c700 1 -- 172.21.15.194:0/2915543938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78d41a4d60 con 0x7f78bc0035c0 2024-04-24T08:00:58.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.249+0000 7f78d259c700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc0035c0 0x7f78d41981e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f78c801f000 tx=0x7f78c800ffb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:58.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.249+0000 7f78c37fe700 1 -- 172.21.15.194:0/2915543938 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c8022040 con 0x7f78bc0035c0 2024-04-24T08:00:58.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.250+0000 7f78d8fa0700 1 -- 172.21.15.194:0/2915543938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78d41a4f30 con 0x7f78bc0035c0 2024-04-24T08:00:58.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.250+0000 7f78c37fe700 1 -- 172.21.15.194:0/2915543938 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f78c802c070 con 0x7f78bc0035c0 2024-04-24T08:00:58.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.250+0000 7f78c37fe700 1 -- 172.21.15.194:0/2915543938 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78c801dc30 con 0x7f78bc0035c0 2024-04-24T08:00:58.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.250+0000 7f78d8fa0700 1 -- 172.21.15.194:0/2915543938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78d41a5460 con 0x7f78bc0035c0 2024-04-24T08:00:58.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.251+0000 7f78d8fa0700 1 -- 172.21.15.194:0/2915543938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78d4064150 con 0x7f78bc0035c0 2024-04-24T08:00:58.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.251+0000 7f78c37fe700 1 -- 172.21.15.194:0/2915543938 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f78c8003880 con 0x7f78bc0035c0 2024-04-24T08:00:58.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.251+0000 7f78c37fe700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f78b4039ae0 0x7f78b403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:00:58.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.251+0000 7f78c37fe700 1 -- 172.21.15.194:0/2915543938 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f78c804f260 con 0x7f78bc0035c0 2024-04-24T08:00:58.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.252+0000 7f78d1d9b700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f78b4039ae0 0x7f78b403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:00:58.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.252+0000 7f78d1d9b700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f78b4039ae0 0x7f78b403bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009180 tx=0x7f78c4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:00:58.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.266+0000 7f78c37fe700 1 -- 172.21.15.194:0/2915543938 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f78c801ce10 con 0x7f78bc0035c0 2024-04-24T08:00:58.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: mon.c calling monitor election 2024-04-24T08:00:58.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:00:58.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: mon.a calling monitor election 2024-04-24T08:00:58.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:00:58.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:00:58.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:58.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:00:58.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:00:58.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:58.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:00:58.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:00:58.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:00:58.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T08:00:58.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: monmap epoch 3 2024-04-24T08:00:58.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: fsid 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:00:58.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: last_changed 2024-04-24T08:00:53.136948+0000 2024-04-24T08:00:58.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: created 2024-04-24T07:58:38.163806+0000 2024-04-24T08:00:58.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: min_mon_release 19 (squid) 2024-04-24T08:00:58.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: election_strategy: 1 2024-04-24T08:00:58.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-24T08:00:58.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-04-24T08:00:58.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: 2: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.b 2024-04-24T08:00:58.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: fsmap 2024-04-24T08:00:58.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:00:58.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: mgrmap e13: x(active, since 68s) 2024-04-24T08:00:58.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: overall HEALTH_OK 2024-04-24T08:00:58.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:58.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:58.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:58.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:58.553 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.552+0000 7f78d8fa0700 1 -- 172.21.15.194:0/2915543938 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f78d402dfe0 con 0x7f78bc0035c0 2024-04-24T08:00:58.553 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: mon.c calling monitor election 2024-04-24T08:00:58.553 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:00:58.553 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: mon.a calling monitor election 2024-04-24T08:00:58.553 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:00:58.553 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:00:58.553 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:58.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:00:58.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:00:58.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:58.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:00:58.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:00:58.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:00:58.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T08:00:58.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: monmap epoch 3 2024-04-24T08:00:58.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: fsid 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:00:58.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: last_changed 2024-04-24T08:00:53.136948+0000 2024-04-24T08:00:58.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: created 2024-04-24T07:58:38.163806+0000 2024-04-24T08:00:58.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: min_mon_release 19 (squid) 2024-04-24T08:00:58.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: election_strategy: 1 2024-04-24T08:00:58.554 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-24T08:00:58.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-04-24T08:00:58.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: 2: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.b 2024-04-24T08:00:58.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: fsmap 2024-04-24T08:00:58.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:00:58.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: mgrmap e13: x(active, since 68s) 2024-04-24T08:00:58.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: overall HEALTH_OK 2024-04-24T08:00:58.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:58.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:58.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:58.555 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:00:58.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.553+0000 7f78c37fe700 1 -- 172.21.15.194:0/2915543938 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1283 (secure 0 0 0) 0x7f78c801d400 con 0x7f78bc0035c0 2024-04-24T08:00:58.556 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-04-24T08:00:58.556 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":3,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","modified":"2024-04-24T08:00:53.136948Z","created":"2024-04-24T07:58:38.163806Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:3300","nonce":0},{"type":"v1","addr":"172.21.15.28:6789","nonce":0}]},"addr":"172.21.15.28:6789/0","public_addr":"172.21.15.28:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:3300","nonce":0},{"type":"v1","addr":"172.21.15.194:6789","nonce":0}]},"addr":"172.21.15.194:6789/0","public_addr":"172.21.15.194:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:3300","nonce":0},{"type":"v1","addr":"172.21.15.117:6789","nonce":0}]},"addr":"172.21.15.117:6789/0","public_addr":"172.21.15.117:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-04-24T08:00:58.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.555+0000 7f78d8fa0700 1 -- 172.21.15.194:0/2915543938 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f78b4039ae0 msgr2=0x7f78b403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:58.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.555+0000 7f78d8fa0700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f78b4039ae0 0x7f78b403bfa0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f78c4009180 tx=0x7f78c4008040 comp rx=0 tx=0).stop 2024-04-24T08:00:58.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.555+0000 7f78d8fa0700 1 -- 172.21.15.194:0/2915543938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc0035c0 msgr2=0x7f78d41981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:00:58.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.555+0000 7f78d8fa0700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc0035c0 0x7f78d41981e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f78c801f000 tx=0x7f78c800ffb0 comp rx=0 tx=0).stop 2024-04-24T08:00:58.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.555+0000 7f78d8fa0700 1 -- 172.21.15.194:0/2915543938 shutdown_connections 2024-04-24T08:00:58.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.555+0000 7f78d8fa0700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f78b4039ae0 0x7f78b403bfa0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:58.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.555+0000 7f78d8fa0700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78d419cff0 0x7f78d419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:58.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.555+0000 7f78d8fa0700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78bc0035c0 0x7f78d41981e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:58.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.555+0000 7f78d8fa0700 1 --2- 172.21.15.194:0/2915543938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f78d4198720 0x7f78d419cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:00:58.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.555+0000 7f78d8fa0700 1 -- 172.21.15.194:0/2915543938 >> 172.21.15.194:0/2915543938 conn(0x7f78d40fe100 msgr2=0x7f78d40fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:00:58.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.555+0000 7f78d8fa0700 1 -- 172.21.15.194:0/2915543938 shutdown_connections 2024-04-24T08:00:58.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:00:58.555+0000 7f78d8fa0700 1 -- 172.21.15.194:0/2915543938 wait complete. 2024-04-24T08:00:58.558 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 3 2024-04-24T08:00:59.165 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-04-24T08:00:59.165 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph config generate-minimal-conf 2024-04-24T08:00:59.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:59 smithi194 ceph-mon[32836]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:59.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:00:59.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:59 smithi194 ceph-mon[32836]: from='client.? 172.21.15.194:0/2915543938' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T08:00:59.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:00:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:00:59.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:59 smithi028 ceph-mon[27786]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:00:59.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:00:59.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:59 smithi028 ceph-mon[27786]: from='client.? 172.21.15.194:0/2915543938' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T08:00:59.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:00:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:00:59.606 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:01:00.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: mon.c calling monitor election 2024-04-24T08:01:00.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:01:00.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: mon.a calling monitor election 2024-04-24T08:01:00.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:01:00.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:01:00.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:00.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:01:00.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:01:00.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:00.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:01:00.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:01:00.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:01:00.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-04-24T08:01:00.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: monmap epoch 3 2024-04-24T08:01:00.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: fsid 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:01:00.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: last_changed 2024-04-24T08:00:53.136948+0000 2024-04-24T08:01:00.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: created 2024-04-24T07:58:38.163806+0000 2024-04-24T08:01:00.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: min_mon_release 19 (squid) 2024-04-24T08:01:00.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: election_strategy: 1 2024-04-24T08:01:00.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-24T08:01:00.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-04-24T08:01:00.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: 2: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.b 2024-04-24T08:01:00.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: fsmap 2024-04-24T08:01:00.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:01:00.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: mgrmap e13: x(active, since 68s) 2024-04-24T08:01:00.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: overall HEALTH_OK 2024-04-24T08:01:00.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:00.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:00.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:00.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:00.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:00.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:01:00.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='client.? 172.21.15.194:0/2915543938' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-04-24T08:01:00.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:01:01.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: mon.b calling monitor election 2024-04-24T08:01:01.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-24T08:01:01.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: Updating smithi117:/etc/ceph/ceph.conf 2024-04-24T08:01:01.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-24T08:01:01.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:01.138 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: mon.b calling monitor election 2024-04-24T08:01:01.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: mon.c calling monitor election 2024-04-24T08:01:01.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: mon.a calling monitor election 2024-04-24T08:01:01.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T08:01:01.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: Updating smithi117:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:01:01.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: monmap epoch 3 2024-04-24T08:01:01.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: fsid 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:01:01.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: last_changed 2024-04-24T08:00:53.136948+0000 2024-04-24T08:01:01.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: created 2024-04-24T07:58:38.163806+0000 2024-04-24T08:01:01.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: min_mon_release 19 (squid) 2024-04-24T08:01:01.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: election_strategy: 1 2024-04-24T08:01:01.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-24T08:01:01.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-04-24T08:01:01.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: 2: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.b 2024-04-24T08:01:01.139 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: fsmap 2024-04-24T08:01:01.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:01:01.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: mgrmap e13: x(active, since 70s) 2024-04-24T08:01:01.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: overall HEALTH_OK 2024-04-24T08:01:01.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: Updating smithi194:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:01:01.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:01.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:01.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:01.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:01.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:00 smithi028 ceph-mon[27786]: Updating smithi028:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:01:01.295 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: mon.b calling monitor election 2024-04-24T08:01:01.295 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-24T08:01:01.295 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: Updating smithi117:/etc/ceph/ceph.conf 2024-04-24T08:01:01.295 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-24T08:01:01.295 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:01.295 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: mon.b calling monitor election 2024-04-24T08:01:01.295 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: mon.c calling monitor election 2024-04-24T08:01:01.295 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: mon.a calling monitor election 2024-04-24T08:01:01.295 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T08:01:01.295 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: Updating smithi117:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:01:01.296 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: monmap epoch 3 2024-04-24T08:01:01.296 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: fsid 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:01:01.296 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: last_changed 2024-04-24T08:00:53.136948+0000 2024-04-24T08:01:01.296 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: created 2024-04-24T07:58:38.163806+0000 2024-04-24T08:01:01.296 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: min_mon_release 19 (squid) 2024-04-24T08:01:01.296 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: election_strategy: 1 2024-04-24T08:01:01.296 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-24T08:01:01.296 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-04-24T08:01:01.296 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: 2: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.b 2024-04-24T08:01:01.296 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: fsmap 2024-04-24T08:01:01.296 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:01:01.296 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: mgrmap e13: x(active, since 70s) 2024-04-24T08:01:01.296 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: overall HEALTH_OK 2024-04-24T08:01:01.296 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: Updating smithi194:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:01:01.297 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:01.297 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:01.297 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:01.297 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:01.297 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:00 smithi117 ceph-mon[32438]: Updating smithi028:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:01:01.416 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: mon.b calling monitor election 2024-04-24T08:01:01.416 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-24T08:01:01.416 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: Updating smithi117:/etc/ceph/ceph.conf 2024-04-24T08:01:01.416 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-24T08:01:01.416 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:01.416 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: mon.b calling monitor election 2024-04-24T08:01:01.416 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: mon.c calling monitor election 2024-04-24T08:01:01.416 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: mon.a calling monitor election 2024-04-24T08:01:01.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-04-24T08:01:01.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: Updating smithi117:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:01:01.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: monmap epoch 3 2024-04-24T08:01:01.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: fsid 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:01:01.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: last_changed 2024-04-24T08:00:53.136948+0000 2024-04-24T08:01:01.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: created 2024-04-24T07:58:38.163806+0000 2024-04-24T08:01:01.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: min_mon_release 19 (squid) 2024-04-24T08:01:01.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: election_strategy: 1 2024-04-24T08:01:01.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: 0: [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] mon.a 2024-04-24T08:01:01.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: 1: [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] mon.c 2024-04-24T08:01:01.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: 2: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.b 2024-04-24T08:01:01.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: fsmap 2024-04-24T08:01:01.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: osdmap e4: 0 total, 0 up, 0 in 2024-04-24T08:01:01.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: mgrmap e13: x(active, since 70s) 2024-04-24T08:01:01.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: overall HEALTH_OK 2024-04-24T08:01:01.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: Updating smithi194:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:01:01.418 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:01.418 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:01.418 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:01.418 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:01.418 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:00 smithi194 ceph-mon[32836]: Updating smithi028:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:01:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.683+0000 7f84a3c00700 1 -- 172.21.15.28:0/3399183277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c1085d0 msgr2=0x7f849c1089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.683+0000 7f84a3c00700 1 --2- 172.21.15.28:0/3399183277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c1085d0 0x7f849c1089b0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f8490009910 tx=0x7f8490009c20 comp rx=0 tx=0).stop 2024-04-24T08:01:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.683+0000 7f84a3c00700 1 -- 172.21.15.28:0/3399183277 shutdown_connections 2024-04-24T08:01:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.683+0000 7f84a3c00700 1 --2- 172.21.15.28:0/3399183277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c1085d0 0x7f849c1089b0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.683+0000 7f84a3c00700 1 -- 172.21.15.28:0/3399183277 >> 172.21.15.28:0/3399183277 conn(0x7f849c0fe030 msgr2=0x7f849c100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:01:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.683+0000 7f84a3c00700 1 -- 172.21.15.28:0/3399183277 shutdown_connections 2024-04-24T08:01:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.684+0000 7f84a3c00700 1 -- 172.21.15.28:0/3399183277 wait complete. 2024-04-24T08:01:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.685+0000 7f84a3c00700 1 Processor -- start 2024-04-24T08:01:01.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.685+0000 7f84a3c00700 1 -- start start 2024-04-24T08:01:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.685+0000 7f84a3c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f849c1085d0 0x7f849c19a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:01:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.685+0000 7f84a3c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f849c19a9a0 0x7f849c19ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:01:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.685+0000 7f84a3c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c19f270 0x7f849c19f720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:01:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.685+0000 7f84a3c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f849c10cb40 con 0x7f849c19f270 2024-04-24T08:01:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.685+0000 7f84a3c00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f849c10c9c0 con 0x7f849c1085d0 2024-04-24T08:01:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.685+0000 7f84a3c00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f849c10ccc0 con 0x7f849c19a9a0 2024-04-24T08:01:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.686+0000 7f849be00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f849c19a9a0 0x7f849c19ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:01:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.686+0000 7f84a1c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c19f270 0x7f849c19f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:01:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.686+0000 7f849be00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f849c19a9a0 0x7f849c19ed30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60298/0 (socket says 172.21.15.28:60298) 2024-04-24T08:01:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.686+0000 7f849be00700 1 -- 172.21.15.28:0/2422079650 learned_addr learned my addr 172.21.15.28:0/2422079650 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:01:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.686+0000 7f84a1200700 1 --2- 172.21.15.28:0/2422079650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f849c1085d0 0x7f849c19a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:01:01.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.686+0000 7f849be00700 1 -- 172.21.15.28:0/2422079650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f849c1085d0 msgr2=0x7f849c19a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:01.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.686+0000 7f849be00700 1 --2- 172.21.15.28:0/2422079650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f849c1085d0 0x7f849c19a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:01.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.686+0000 7f849be00700 1 -- 172.21.15.28:0/2422079650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c19f270 msgr2=0x7f849c19f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:01.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.686+0000 7f849be00700 1 --2- 172.21.15.28:0/2422079650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c19f270 0x7f849c19f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:01.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.686+0000 7f849be00700 1 -- 172.21.15.28:0/2422079650 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f849c1a6fe0 con 0x7f849c19a9a0 2024-04-24T08:01:01.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.687+0000 7f849be00700 1 --2- 172.21.15.28:0/2422079650 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f849c19a9a0 0x7f849c19ed30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f848800b780 tx=0x7f848800ba90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:01:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.687+0000 7f8499600700 1 -- 172.21.15.28:0/2422079650 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488004230 con 0x7f849c19a9a0 2024-04-24T08:01:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.687+0000 7f8499600700 1 -- 172.21.15.28:0/2422079650 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84880043d0 con 0x7f849c19a9a0 2024-04-24T08:01:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.687+0000 7f8499600700 1 -- 172.21.15.28:0/2422079650 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8488011c90 con 0x7f849c19a9a0 2024-04-24T08:01:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.687+0000 7f84a3c00700 1 -- 172.21.15.28:0/2422079650 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f849c1a7210 con 0x7f849c19a9a0 2024-04-24T08:01:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.687+0000 7f84a3c00700 1 -- 172.21.15.28:0/2422079650 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f849c1a7780 con 0x7f849c19a9a0 2024-04-24T08:01:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.688+0000 7f84a3c00700 1 -- 172.21.15.28:0/2422079650 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f849c064150 con 0x7f849c19a9a0 2024-04-24T08:01:01.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.688+0000 7f8499600700 1 -- 172.21.15.28:0/2422079650 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f848801a400 con 0x7f849c19a9a0 2024-04-24T08:01:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.689+0000 7f8499600700 1 --2- 172.21.15.28:0/2422079650 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8484039ae0 0x7f848403bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:01:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.689+0000 7f8499600700 1 -- 172.21.15.28:0/2422079650 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f848804c540 con 0x7f849c19a9a0 2024-04-24T08:01:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.689+0000 7f84a1200700 1 --2- 172.21.15.28:0/2422079650 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8484039ae0 0x7f848403bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:01:01.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.690+0000 7f84a1200700 1 --2- 172.21.15.28:0/2422079650 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8484039ae0 0x7f848403bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8490009750 tx=0x7f849000b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:01:01.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.694+0000 7f8499600700 1 -- 172.21.15.28:0/2422079650 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8488018350 con 0x7f849c19a9a0 2024-04-24T08:01:01.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.931+0000 7f84a3c00700 1 -- 172.21.15.28:0/2422079650 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f849c1a7a70 con 0x7f849c19a9a0 2024-04-24T08:01:01.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.932+0000 7f8499600700 1 -- 172.21.15.28:0/2422079650 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f848801a7f0 con 0x7f849c19a9a0 2024-04-24T08:01:01.933 INFO:teuthology.orchestra.run.smithi028.stdout:# minimal ceph.conf for 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:01:01.933 INFO:teuthology.orchestra.run.smithi028.stdout:[global] 2024-04-24T08:01:01.934 INFO:teuthology.orchestra.run.smithi028.stdout: fsid = 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:01:01.934 INFO:teuthology.orchestra.run.smithi028.stdout: mon_host = [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] 2024-04-24T08:01:01.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.935+0000 7f84a3c00700 1 -- 172.21.15.28:0/2422079650 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8484039ae0 msgr2=0x7f848403bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:01.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.935+0000 7f84a3c00700 1 --2- 172.21.15.28:0/2422079650 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8484039ae0 0x7f848403bfa0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f8490009750 tx=0x7f849000b540 comp rx=0 tx=0).stop 2024-04-24T08:01:01.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.935+0000 7f84a3c00700 1 -- 172.21.15.28:0/2422079650 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f849c19a9a0 msgr2=0x7f849c19ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:01.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.935+0000 7f84a3c00700 1 --2- 172.21.15.28:0/2422079650 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f849c19a9a0 0x7f849c19ed30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f848800b780 tx=0x7f848800ba90 comp rx=0 tx=0).stop 2024-04-24T08:01:01.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.935+0000 7f84a3c00700 1 -- 172.21.15.28:0/2422079650 shutdown_connections 2024-04-24T08:01:01.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.935+0000 7f84a3c00700 1 --2- 172.21.15.28:0/2422079650 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8484039ae0 0x7f848403bfa0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:01.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.935+0000 7f84a3c00700 1 --2- 172.21.15.28:0/2422079650 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f849c19f270 0x7f849c19f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:01.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.935+0000 7f84a3c00700 1 --2- 172.21.15.28:0/2422079650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f849c1085d0 0x7f849c19a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:01.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.935+0000 7f84a3c00700 1 --2- 172.21.15.28:0/2422079650 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f849c19a9a0 0x7f849c19ed30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:01.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.935+0000 7f84a3c00700 1 -- 172.21.15.28:0/2422079650 >> 172.21.15.28:0/2422079650 conn(0x7f849c0fe030 msgr2=0x7f849c0fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:01:01.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.935+0000 7f84a3c00700 1 -- 172.21.15.28:0/2422079650 shutdown_connections 2024-04-24T08:01:01.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:01.935+0000 7f84a3c00700 1 -- 172.21.15.28:0/2422079650 wait complete. 2024-04-24T08:01:02.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:01:02.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:01 smithi194 ceph-mon[32836]: Reconfiguring mon.a (unknown last config time)... 2024-04-24T08:01:02.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:01:02.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:01:02.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:02.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:01 smithi194 ceph-mon[32836]: Reconfiguring daemon mon.a on smithi028 2024-04-24T08:01:02.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:01 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2422079650' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:02.253 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:01:02 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:01:02.136+0000 7f393c800700 -1 mgr.server handle_report got status from non-daemon mon.b 2024-04-24T08:01:02.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:01:02.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.253 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:01 smithi028 ceph-mon[27786]: Reconfiguring mon.a (unknown last config time)... 2024-04-24T08:01:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:01:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:01:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:01 smithi028 ceph-mon[27786]: Reconfiguring daemon mon.a on smithi028 2024-04-24T08:01:02.254 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:01 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2422079650' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:02.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:01:02.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:02.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:01 smithi117 ceph-mon[32438]: Reconfiguring mon.a (unknown last config time)... 2024-04-24T08:01:02.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:01:02.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:01:02.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:02.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:01 smithi117 ceph-mon[32438]: Reconfiguring daemon mon.a on smithi028 2024-04-24T08:01:02.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:01 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2422079650' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:02.632 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-04-24T08:01:02.632 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T08:01:02.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.conf 2024-04-24T08:01:02.720 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T08:01:02.720 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:01:02.845 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-04-24T08:01:02.845 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.conf 2024-04-24T08:01:02.877 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-04-24T08:01:02.877 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:01:02.943 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-24T08:01:02.943 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.conf 2024-04-24T08:01:02.970 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-24T08:01:02.970 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:01:03.034 INFO:tasks.cephadm:Adding mgr.x on smithi028 2024-04-24T08:01:03.034 INFO:tasks.cephadm:Adding mgr.y on smithi117 2024-04-24T08:01:03.034 INFO:tasks.cephadm:Adding mgr.z on smithi194 2024-04-24T08:01:03.035 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch apply mgr '3;smithi028=x;smithi117=y;smithi194=z' 2024-04-24T08:01:03.301 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:01:03.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:03 smithi194 ceph-mon[32836]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:03.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:03 smithi117 ceph-mon[32438]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:03.908 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:03 smithi028 ceph-mon[27786]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:04.857 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.855+0000 7f2aba7d7700 1 -- 172.21.15.194:0/10014907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ab4101570 msgr2=0x7f2ab4101990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:04.857 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.855+0000 7f2aba7d7700 1 --2- 172.21.15.194:0/10014907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ab4101570 0x7f2ab4101990 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f2aa4009910 tx=0x7f2aa4009c20 comp rx=0 tx=0).stop 2024-04-24T08:01:04.857 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.855+0000 7f2aba7d7700 1 -- 172.21.15.194:0/10014907 shutdown_connections 2024-04-24T08:01:04.857 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.855+0000 7f2aba7d7700 1 --2- 172.21.15.194:0/10014907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ab4101570 0x7f2ab4101990 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:04.857 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.855+0000 7f2aba7d7700 1 -- 172.21.15.194:0/10014907 >> 172.21.15.194:0/10014907 conn(0x7f2ab40fcb30 msgr2=0x7f2ab40fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:01:04.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.855+0000 7f2aba7d7700 1 -- 172.21.15.194:0/10014907 shutdown_connections 2024-04-24T08:01:04.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.855+0000 7f2aba7d7700 1 -- 172.21.15.194:0/10014907 wait complete. 2024-04-24T08:01:04.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.855+0000 7f2aba7d7700 1 Processor -- start 2024-04-24T08:01:04.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.855+0000 7f2aba7d7700 1 -- start start 2024-04-24T08:01:04.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2aba7d7700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ab4101570 0x7f2ab4197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:01:04.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2aba7d7700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ab4198430 0x7f2ab419d4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:01:04.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2aba7d7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ab41988b0 0x7f2ab419fa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:01:04.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2aba7d7700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab41075b0 con 0x7f2ab4198430 2024-04-24T08:01:04.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2aba7d7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab4107430 con 0x7f2ab41988b0 2024-04-24T08:01:04.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2aba7d7700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2ab4107730 con 0x7f2ab4101570 2024-04-24T08:01:04.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2ab3fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ab4101570 0x7f2ab4197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:01:04.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2ab3fff700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ab4101570 0x7f2ab4197ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:46622/0 (socket says 172.21.15.194:46622) 2024-04-24T08:01:04.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2ab3fff700 1 -- 172.21.15.194:0/1066990910 learned_addr learned my addr 172.21.15.194:0/1066990910 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-24T08:01:04.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2ab8d74700 1 --2- 172.21.15.194:0/1066990910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ab41988b0 0x7f2ab419fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:01:04.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2ab3fff700 1 -- 172.21.15.194:0/1066990910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ab41988b0 msgr2=0x7f2ab419fa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:04.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2ab3fff700 1 --2- 172.21.15.194:0/1066990910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ab41988b0 0x7f2ab419fa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:04.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2ab3fff700 1 -- 172.21.15.194:0/1066990910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ab4198430 msgr2=0x7f2ab419d4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:04.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2ab3fff700 1 --2- 172.21.15.194:0/1066990910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ab4198430 0x7f2ab419d4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:04.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2ab3fff700 1 -- 172.21.15.194:0/1066990910 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ab41a0300 con 0x7f2ab4101570 2024-04-24T08:01:04.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2ab3fff700 1 --2- 172.21.15.194:0/1066990910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ab4101570 0x7f2ab4197ef0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2aa400be40 tx=0x7f2aa400be70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:01:04.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2ab17fa700 1 -- 172.21.15.194:0/1066990910 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa4027070 con 0x7f2ab4101570 2024-04-24T08:01:04.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2aba7d7700 1 -- 172.21.15.194:0/1066990910 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ab41a0590 con 0x7f2ab4101570 2024-04-24T08:01:04.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2ab17fa700 1 -- 172.21.15.194:0/1066990910 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2aa4005da0 con 0x7f2ab4101570 2024-04-24T08:01:04.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.856+0000 7f2ab17fa700 1 -- 172.21.15.194:0/1066990910 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2aa400f460 con 0x7f2ab4101570 2024-04-24T08:01:04.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.857+0000 7f2aba7d7700 1 -- 172.21.15.194:0/1066990910 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ab41a0b40 con 0x7f2ab4101570 2024-04-24T08:01:04.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.857+0000 7f2ab17fa700 1 -- 172.21.15.194:0/1066990910 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7f2aa402a070 con 0x7f2ab4101570 2024-04-24T08:01:04.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.857+0000 7f2ab17fa700 1 --2- 172.21.15.194:0/1066990910 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2a9c039ae0 0x7f2a9c03bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:01:04.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.857+0000 7f2aba7d7700 1 -- 172.21.15.194:0/1066990910 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ab4064150 con 0x7f2ab4101570 2024-04-24T08:01:04.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.858+0000 7f2ab17fa700 1 -- 172.21.15.194:0/1066990910 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f2aa404dad0 con 0x7f2ab4101570 2024-04-24T08:01:04.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.858+0000 7f2ab37fe700 1 --2- 172.21.15.194:0/1066990910 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2a9c039ae0 0x7f2a9c03bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:01:04.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.859+0000 7f2ab37fe700 1 --2- 172.21.15.194:0/1066990910 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2a9c039ae0 0x7f2a9c03bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2aa80098e0 tx=0x7f2aa8006dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:01:04.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:04.863+0000 7f2ab17fa700 1 -- 172.21.15.194:0/1066990910 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2aa4013070 con 0x7f2ab4101570 2024-04-24T08:01:05.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:05.076+0000 7f2aba7d7700 1 -- 172.21.15.194:0/1066990910 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi117=y;smithi194=z", "target": ["mon-mgr", ""]}) -- 0x7f2ab4062fc0 con 0x7f2a9c039ae0 2024-04-24T08:01:05.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:05.090+0000 7f2ab17fa700 1 -- 172.21.15.194:0/1066990910 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f2ab4062fc0 con 0x7f2a9c039ae0 2024-04-24T08:01:05.090 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mgr update... 2024-04-24T08:01:05.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:05.092+0000 7f2aba7d7700 1 -- 172.21.15.194:0/1066990910 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2a9c039ae0 msgr2=0x7f2a9c03bfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:05.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:05.092+0000 7f2aba7d7700 1 --2- 172.21.15.194:0/1066990910 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2a9c039ae0 0x7f2a9c03bfa0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f2aa80098e0 tx=0x7f2aa8006dd0 comp rx=0 tx=0).stop 2024-04-24T08:01:05.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:05.092+0000 7f2aba7d7700 1 -- 172.21.15.194:0/1066990910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ab4101570 msgr2=0x7f2ab4197ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:05.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:05.092+0000 7f2aba7d7700 1 --2- 172.21.15.194:0/1066990910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ab4101570 0x7f2ab4197ef0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2aa400be40 tx=0x7f2aa400be70 comp rx=0 tx=0).stop 2024-04-24T08:01:05.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:05.092+0000 7f2aba7d7700 1 -- 172.21.15.194:0/1066990910 shutdown_connections 2024-04-24T08:01:05.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:05.092+0000 7f2aba7d7700 1 --2- 172.21.15.194:0/1066990910 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2a9c039ae0 0x7f2a9c03bfa0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:05.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:05.092+0000 7f2aba7d7700 1 --2- 172.21.15.194:0/1066990910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ab41988b0 0x7f2ab419fa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:05.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:05.092+0000 7f2aba7d7700 1 --2- 172.21.15.194:0/1066990910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ab4101570 0x7f2ab4197ef0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:05.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:05.092+0000 7f2aba7d7700 1 --2- 172.21.15.194:0/1066990910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ab4198430 0x7f2ab419d4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:05.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:05.092+0000 7f2aba7d7700 1 -- 172.21.15.194:0/1066990910 >> 172.21.15.194:0/1066990910 conn(0x7f2ab40fcb30 msgr2=0x7f2ab406b280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:01:05.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:05.092+0000 7f2aba7d7700 1 -- 172.21.15.194:0/1066990910 shutdown_connections 2024-04-24T08:01:05.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:01:05.092+0000 7f2aba7d7700 1 -- 172.21.15.194:0/1066990910 wait complete. 2024-04-24T08:01:05.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:05 smithi028 ceph-mon[27786]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:05.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:05.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:05.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:05 smithi028 ceph-mon[27786]: Reconfiguring mon.b (monmap changed)... 2024-04-24T08:01:05.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:01:05.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:01:05.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:05.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:05 smithi028 ceph-mon[27786]: Reconfiguring daemon mon.b on smithi117 2024-04-24T08:01:05.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:05 smithi117 ceph-mon[32438]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:05.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:05 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:05.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:05 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:05.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:05 smithi117 ceph-mon[32438]: Reconfiguring mon.b (monmap changed)... 2024-04-24T08:01:05.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:05 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:01:05.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:05 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:01:05.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:05 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:05.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:05 smithi117 ceph-mon[32438]: Reconfiguring daemon mon.b on smithi117 2024-04-24T08:01:05.431 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:05 smithi194 ceph-mon[32836]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:05.431 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:05 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:05.431 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:05 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:05.432 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:05 smithi194 ceph-mon[32836]: Reconfiguring mon.b (monmap changed)... 2024-04-24T08:01:05.432 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:05 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:01:05.432 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:05 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:01:05.432 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:05 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:05.432 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:05 smithi194 ceph-mon[32836]: Reconfiguring daemon mon.b on smithi117 2024-04-24T08:01:05.633 DEBUG:teuthology.orchestra.run.smithi117:mgr.y> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mgr.y.service 2024-04-24T08:01:05.635 DEBUG:teuthology.orchestra.run.smithi194:mgr.z> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mgr.z.service 2024-04-24T08:01:05.638 INFO:tasks.cephadm:Deploying OSDs... 2024-04-24T08:01:05.638 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T08:01:05.638 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T08:01:05.703 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T08:01:05.703 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_1 2024-04-24T08:01:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T08:01:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:01:05.755 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 633 Links: 1 2024-04-24T08:01:05.755 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:01:05.755 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:01:05.755 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-24 08:00:02.616507319 +0000 2024-04-24T08:01:05.756 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-24 07:54:18.286140072 +0000 2024-04-24T08:01:05.756 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-24 07:54:18.286140072 +0000 2024-04-24T08:01:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-24 07:54:18.286140072 +0000 2024-04-24T08:01:05.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T08:01:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-24T08:01:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-24T08:01:05.841 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000280871 s, 1.8 MB/s 2024-04-24T08:01:05.845 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T08:01:05.905 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_2 2024-04-24T08:01:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T08:01:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:01:05.960 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 650 Links: 1 2024-04-24T08:01:05.960 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:01:05.960 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:01:05.960 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-24 08:00:02.689505274 +0000 2024-04-24T08:01:05.960 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-24 07:54:18.930122198 +0000 2024-04-24T08:01:05.960 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-24 07:54:18.930122198 +0000 2024-04-24T08:01:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-24 07:54:18.930122198 +0000 2024-04-24T08:01:05.961 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T08:01:06.045 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-24T08:01:06.045 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-24T08:01:06.045 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000287344 s, 1.8 MB/s 2024-04-24T08:01:06.050 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T08:01:06.108 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_3 2024-04-24T08:01:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T08:01:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:01:06.159 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 672 Links: 1 2024-04-24T08:01:06.159 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:01:06.159 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:01:06.159 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-24 08:00:02.767503090 +0000 2024-04-24T08:01:06.159 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-24 07:54:19.541105241 +0000 2024-04-24T08:01:06.159 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-24 07:54:19.541105241 +0000 2024-04-24T08:01:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-24 07:54:19.541105241 +0000 2024-04-24T08:01:06.159 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T08:01:06.235 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-24T08:01:06.236 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-24T08:01:06.236 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000276547 s, 1.9 MB/s 2024-04-24T08:01:06.239 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T08:01:06.297 DEBUG:teuthology.orchestra.run.smithi028:> stat /dev/vg_nvme/lv_4 2024-04-24T08:01:06.342 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:06 smithi194 ceph-mon[32836]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi117=y;smithi194=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:01:06.342 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:06 smithi194 ceph-mon[32836]: Saving service mgr spec with placement smithi028=x;smithi117=y;smithi194=z;count:3 2024-04-24T08:01:06.342 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:06.342 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:06.342 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:06.342 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:06 smithi194 ceph-mon[32836]: Reconfiguring mon.c (monmap changed)... 2024-04-24T08:01:06.342 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:01:06.342 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:01:06.342 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:06.343 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:06 smithi194 ceph-mon[32836]: Reconfiguring daemon mon.c on smithi194 2024-04-24T08:01:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T08:01:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:01:06.350 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 6h/6d Inode: 688 Links: 1 2024-04-24T08:01:06.350 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:01:06.350 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:01:06.350 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-24 08:00:02.847500849 +0000 2024-04-24T08:01:06.350 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-24 07:54:20.125089032 +0000 2024-04-24T08:01:06.351 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-24 07:54:20.125089032 +0000 2024-04-24T08:01:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-24 07:54:20.125089032 +0000 2024-04-24T08:01:06.351 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T08:01:06.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:06 smithi117 ceph-mon[32438]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi117=y;smithi194=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:01:06.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:06 smithi117 ceph-mon[32438]: Saving service mgr spec with placement smithi028=x;smithi117=y;smithi194=z;count:3 2024-04-24T08:01:06.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:06.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:06.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:06.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:06 smithi117 ceph-mon[32438]: Reconfiguring mon.c (monmap changed)... 2024-04-24T08:01:06.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:01:06.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:01:06.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:06.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:06 smithi117 ceph-mon[32438]: Reconfiguring daemon mon.c on smithi194 2024-04-24T08:01:06.428 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:06 smithi028 ceph-mon[27786]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi028=x;smithi117=y;smithi194=z", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:01:06.429 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:06 smithi028 ceph-mon[27786]: Saving service mgr spec with placement smithi028=x;smithi117=y;smithi194=z;count:3 2024-04-24T08:01:06.429 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:06.429 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:06.429 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:06.429 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:06 smithi028 ceph-mon[27786]: Reconfiguring mon.c (monmap changed)... 2024-04-24T08:01:06.429 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-04-24T08:01:06.429 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-04-24T08:01:06.429 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:06.429 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:06 smithi028 ceph-mon[27786]: Reconfiguring daemon mon.c on smithi194 2024-04-24T08:01:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records in 2024-04-24T08:01:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:1+0 records out 2024-04-24T08:01:06.439 INFO:teuthology.orchestra.run.smithi028.stderr:512 bytes copied, 0.000272735 s, 1.9 MB/s 2024-04-24T08:01:06.443 DEBUG:teuthology.orchestra.run.smithi028:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T08:01:06.504 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2024-04-24T08:01:06.504 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T08:01:06.520 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T08:01:06.520 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_1 2024-04-24T08:01:06.576 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T08:01:06.576 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:01:06.576 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-24T08:01:06.576 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:01:06.576 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:01:06.576 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-04-24 08:00:52.561891133 +0000 2024-04-24T08:01:06.576 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-04-24 07:53:04.739157940 +0000 2024-04-24T08:01:06.576 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-04-24 07:53:04.739157940 +0000 2024-04-24T08:01:06.576 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-04-24 07:53:04.739157940 +0000 2024-04-24T08:01:06.577 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T08:01:06.640 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-04-24T08:01:06.640 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-04-24T08:01:06.640 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000220425 s, 2.3 MB/s 2024-04-24T08:01:06.642 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T08:01:06.698 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_2 2024-04-24T08:01:06.754 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T08:01:06.754 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:01:06.754 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-04-24T08:01:06.755 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:01:06.755 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:01:06.755 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-04-24 08:00:52.562891113 +0000 2024-04-24T08:01:06.755 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-04-24 07:53:05.097151028 +0000 2024-04-24T08:01:06.755 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-04-24 07:53:05.097151028 +0000 2024-04-24T08:01:06.755 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-04-24 07:53:05.097151028 +0000 2024-04-24T08:01:06.756 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T08:01:06.817 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-04-24T08:01:06.817 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-04-24T08:01:06.817 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000219452 s, 2.3 MB/s 2024-04-24T08:01:06.819 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T08:01:06.875 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_3 2024-04-24T08:01:06.930 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T08:01:06.931 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:01:06.931 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-24T08:01:06.931 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:01:06.931 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:01:06.931 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-04-24 08:00:52.562891113 +0000 2024-04-24T08:01:06.931 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-04-24 07:53:05.428144638 +0000 2024-04-24T08:01:06.931 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-04-24 07:53:05.428144638 +0000 2024-04-24T08:01:06.931 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-04-24 07:53:05.428144638 +0000 2024-04-24T08:01:06.931 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T08:01:06.992 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-04-24T08:01:06.993 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-04-24T08:01:06.993 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000245052 s, 2.1 MB/s 2024-04-24T08:01:06.994 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T08:01:07.050 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_4 2024-04-24T08:01:07.104 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T08:01:07.104 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:01:07.104 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-24T08:01:07.104 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:01:07.104 INFO:teuthology.orchestra.run.smithi117.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:01:07.104 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2024-04-24 08:00:52.562891113 +0000 2024-04-24T08:01:07.104 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2024-04-24 07:53:05.751138402 +0000 2024-04-24T08:01:07.104 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2024-04-24 07:53:05.751138402 +0000 2024-04-24T08:01:07.105 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: 2024-04-24 07:53:05.751138402 +0000 2024-04-24T08:01:07.105 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T08:01:07.166 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2024-04-24T08:01:07.167 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2024-04-24T08:01:07.167 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000199749 s, 2.6 MB/s 2024-04-24T08:01:07.168 DEBUG:teuthology.orchestra.run.smithi117:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T08:01:07.224 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-04-24T08:01:07.224 DEBUG:teuthology.orchestra.run.smithi194:> dd if=/scratch_devs of=/dev/stdout 2024-04-24T08:01:07.240 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-04-24T08:01:07.241 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_1 2024-04-24T08:01:07.296 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-04-24T08:01:07.296 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:01:07.296 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 663 Links: 1 2024-04-24T08:01:07.297 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:01:07.297 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:01:07.297 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-04-24 08:00:42.839470875 +0000 2024-04-24T08:01:07.297 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-04-24 07:53:16.574173041 +0000 2024-04-24T08:01:07.297 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-04-24 07:53:16.574173041 +0000 2024-04-24T08:01:07.297 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-04-24 07:53:16.574173041 +0000 2024-04-24T08:01:07.297 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-04-24T08:01:07.360 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-04-24T08:01:07.360 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-04-24T08:01:07.361 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000274182 s, 1.9 MB/s 2024-04-24T08:01:07.362 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-04-24T08:01:07.419 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_2 2024-04-24T08:01:07.474 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-04-24T08:01:07.474 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:01:07.474 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-04-24T08:01:07.474 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:01:07.474 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:01:07.474 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-04-24 08:00:42.840470855 +0000 2024-04-24T08:01:07.474 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-04-24 07:53:16.916166157 +0000 2024-04-24T08:01:07.474 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-04-24 07:53:16.916166157 +0000 2024-04-24T08:01:07.474 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-04-24 07:53:16.916166157 +0000 2024-04-24T08:01:07.475 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-04-24T08:01:07.536 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-04-24T08:01:07.537 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-04-24T08:01:07.537 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000224625 s, 2.3 MB/s 2024-04-24T08:01:07.538 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-04-24T08:01:07.593 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_3 2024-04-24T08:01:07.649 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-04-24T08:01:07.649 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:01:07.649 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-04-24T08:01:07.649 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:01:07.649 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:01:07.649 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-04-24 08:00:42.840470855 +0000 2024-04-24T08:01:07.649 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-04-24 07:53:17.254159353 +0000 2024-04-24T08:01:07.649 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-04-24 07:53:17.254159353 +0000 2024-04-24T08:01:07.649 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-04-24 07:53:17.254159353 +0000 2024-04-24T08:01:07.650 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-04-24T08:01:07.713 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-04-24T08:01:07.713 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-04-24T08:01:07.714 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.00020334 s, 2.5 MB/s 2024-04-24T08:01:07.715 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-04-24T08:01:07.773 DEBUG:teuthology.orchestra.run.smithi194:> stat /dev/vg_nvme/lv_4 2024-04-24T08:01:07.828 INFO:teuthology.orchestra.run.smithi194.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-04-24T08:01:07.828 INFO:teuthology.orchestra.run.smithi194.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-04-24T08:01:07.828 INFO:teuthology.orchestra.run.smithi194.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-04-24T08:01:07.828 INFO:teuthology.orchestra.run.smithi194.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:01:07.828 INFO:teuthology.orchestra.run.smithi194.stdout:Context: system_u:object_r:device_t:s0 2024-04-24T08:01:07.828 INFO:teuthology.orchestra.run.smithi194.stdout:Access: 2024-04-24 08:00:42.840470855 +0000 2024-04-24T08:01:07.828 INFO:teuthology.orchestra.run.smithi194.stdout:Modify: 2024-04-24 07:53:17.594152508 +0000 2024-04-24T08:01:07.828 INFO:teuthology.orchestra.run.smithi194.stdout:Change: 2024-04-24 07:53:17.594152508 +0000 2024-04-24T08:01:07.828 INFO:teuthology.orchestra.run.smithi194.stdout: Birth: 2024-04-24 07:53:17.594152508 +0000 2024-04-24T08:01:07.829 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-04-24T08:01:07.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:07 smithi117 ceph-mon[32438]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:01:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:01:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:01:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T08:01:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:01:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:07 smithi194 ceph-mon[32836]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:01:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:01:07.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:07.888 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:01:07.888 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T08:01:07.888 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:01:07.888 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:07.890 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records in 2024-04-24T08:01:07.891 INFO:teuthology.orchestra.run.smithi194.stderr:1+0 records out 2024-04-24T08:01:07.891 INFO:teuthology.orchestra.run.smithi194.stderr:512 bytes copied, 0.000234014 s, 2.2 MB/s 2024-04-24T08:01:07.892 DEBUG:teuthology.orchestra.run.smithi194:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-04-24T08:01:07.948 INFO:tasks.cephadm:Deploying osd.0 on smithi028 with /dev/vg_nvme/lv_4... 2024-04-24T08:01:07.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-24T08:01:08.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:07 smithi028 ceph-mon[27786]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:08.031 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:08.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:08.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:01:08.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:08.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:01:08.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:08.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:01:08.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T08:01:08.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:01:08.032 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:08.413 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:01:08.875 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:08 smithi194 ceph-mon[32836]: Deploying daemon mgr.z on smithi194 2024-04-24T08:01:08.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:08 smithi117 ceph-mon[32438]: Deploying daemon mgr.z on smithi194 2024-04-24T08:01:08.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:08 smithi028 ceph-mon[27786]: Deploying daemon mgr.z on smithi194 2024-04-24T08:01:09.779 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:09 smithi194 ceph-mon[32836]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:09.780 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:09 smithi194 systemd[1]: Started Ceph mgr.z for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T08:01:09.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:09 smithi117 ceph-mon[32438]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:10.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:09 smithi028 ceph-mon[27786]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:10.067 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:09 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:09.777+0000 7f0be82a3200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T08:01:10.067 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:09 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:09.963+0000 7f0be82a3200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:01:10.067 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:10 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:10.065+0000 7f0be82a3200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T08:01:10.690 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:10 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:10.375+0000 7f0be82a3200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T08:01:10.690 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:10 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:10.460+0000 7f0be82a3200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T08:01:11.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:10 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:11.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:10 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:11.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:10 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:11.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:10 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:01:11.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:10 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T08:01:11.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:10 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:01:11.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:10 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:11.081 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:10 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:11.081 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:10 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:11.081 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:10 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:11.081 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:10 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:01:11.081 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:10 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T08:01:11.081 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:10 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:01:11.081 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:10 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:11.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:10 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:11.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:10 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:11.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:10 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:11.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:10 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:01:11.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:10 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-04-24T08:01:11.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:10 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:01:11.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:10 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:11.372 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-24T08:01:11.372 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-24T08:01:11.372 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-24T08:01:11.372 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-24T08:01:11.373 INFO:teuthology.orchestra.run.smithi028.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0390446 s, 269 MB/s 2024-04-24T08:01:11.373 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-24T08:01:11.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:11 smithi194 ceph-mon[32836]: Deploying daemon mgr.y on smithi117 2024-04-24T08:01:11.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:11 smithi194 ceph-mon[32836]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:11.940 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:11 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:11.653+0000 7f0be82a3200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T08:01:11.940 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:11 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:11.825+0000 7f0be82a3200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T08:01:12.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:11 smithi028 ceph-mon[27786]: Deploying daemon mgr.y on smithi117 2024-04-24T08:01:12.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:11 smithi028 ceph-mon[27786]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:12.042 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:11 smithi117 ceph-mon[32438]: Deploying daemon mgr.y on smithi117 2024-04-24T08:01:12.042 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:11 smithi117 ceph-mon[32438]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:12.118 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_4 2024-04-24T08:01:12.291 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:11 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:11.997+0000 7f0be82a3200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T08:01:12.291 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:12 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:12.286+0000 7f0be82a3200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T08:01:12.482 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:12 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:12.481+0000 7f2194f22200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:01:12.588 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:12 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:12.380+0000 7f0be82a3200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T08:01:12.589 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:12 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:12.587+0000 7f0be82a3200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T08:01:12.591 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:01:12.878 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:12 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:12.584+0000 7f2194f22200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T08:01:12.940 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:12 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:12.787+0000 7f0be82a3200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:01:13.276 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:12 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:12.908+0000 7f2194f22200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T08:01:13.277 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:12 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:12.995+0000 7f2194f22200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T08:01:13.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:13 smithi028 ceph-mon[27786]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:13.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:13 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:13.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:13 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:13.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:13 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:13.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:13 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:13.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:13 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:01:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:13 smithi117 ceph-mon[32438]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:13 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:13 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:13 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:13 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:13 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:01:13.690 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:13 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:13.424+0000 7f0be82a3200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T08:01:13.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:13 smithi194 ceph-mon[32836]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:13.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:13 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:13.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:13 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:13.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:13 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:13.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:13 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:13.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:13 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:01:14.167 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:13 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:13.864+0000 7f0be82a3200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T08:01:14.167 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:13 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:13.961+0000 7f0be82a3200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T08:01:14.167 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:14.165+0000 7f0be82a3200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T08:01:14.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.496+0000 7fd34ba00700 1 -- 172.21.15.28:0/244362302 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd344109190 msgr2=0x7fd3441095b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.496+0000 7fd34ba00700 1 --2- 172.21.15.28:0/244362302 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd344109190 0x7fd3441095b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd338009990 tx=0x7fd338009ca0 comp rx=0 tx=0).stop 2024-04-24T08:01:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.497+0000 7fd34ba00700 1 -- 172.21.15.28:0/244362302 shutdown_connections 2024-04-24T08:01:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.497+0000 7fd34ba00700 1 --2- 172.21.15.28:0/244362302 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd34410ad30 0x7fd3441115e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.497+0000 7fd34ba00700 1 --2- 172.21.15.28:0/244362302 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd344109190 0x7fd3441095b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.497+0000 7fd34ba00700 1 --2- 172.21.15.28:0/244362302 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34410a390 0x7fd34410a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.497+0000 7fd34ba00700 1 -- 172.21.15.28:0/244362302 >> 172.21.15.28:0/244362302 conn(0x7fd344078160 msgr2=0x7fd34407a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:01:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.497+0000 7fd34ba00700 1 -- 172.21.15.28:0/244362302 shutdown_connections 2024-04-24T08:01:14.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.497+0000 7fd34ba00700 1 -- 172.21.15.28:0/244362302 wait complete. 2024-04-24T08:01:14.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.498+0000 7fd34ba00700 1 Processor -- start 2024-04-24T08:01:14.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.498+0000 7fd34ba00700 1 -- start start 2024-04-24T08:01:14.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.498+0000 7fd34ba00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd344109190 0x7fd34419f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:01:14.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.498+0000 7fd34ba00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34410a390 0x7fd34419f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:01:14.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.498+0000 7fd34ba00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd34410ad30 0x7fd3441a6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:01:14.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.498+0000 7fd34ba00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd344114dc0 con 0x7fd34410a390 2024-04-24T08:01:14.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.498+0000 7fd34ba00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd344114c40 con 0x7fd344109190 2024-04-24T08:01:14.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.498+0000 7fd34ba00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd344114f40 con 0x7fd34410ad30 2024-04-24T08:01:14.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.499+0000 7fd349000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd344109190 0x7fd34419f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:01:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.499+0000 7fd343e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34410a390 0x7fd34419f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:01:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.499+0000 7fd349000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd344109190 0x7fd34419f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:38778/0 (socket says 172.21.15.28:38778) 2024-04-24T08:01:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.499+0000 7fd349000700 1 -- 172.21.15.28:0/813846474 learned_addr learned my addr 172.21.15.28:0/813846474 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:01:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.499+0000 7fd349a00700 1 --2- 172.21.15.28:0/813846474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd34410ad30 0x7fd3441a6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:01:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.499+0000 7fd349000700 1 -- 172.21.15.28:0/813846474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd34410ad30 msgr2=0x7fd3441a6990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.499+0000 7fd349000700 1 --2- 172.21.15.28:0/813846474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd34410ad30 0x7fd3441a6990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.499+0000 7fd349000700 1 -- 172.21.15.28:0/813846474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34410a390 msgr2=0x7fd34419f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.499+0000 7fd349000700 1 --2- 172.21.15.28:0/813846474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34410a390 0x7fd34419f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.499+0000 7fd349000700 1 -- 172.21.15.28:0/813846474 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd3441a7030 con 0x7fd344109190 2024-04-24T08:01:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.499+0000 7fd349000700 1 --2- 172.21.15.28:0/813846474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd344109190 0x7fd34419f3d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd338000c00 tx=0x7fd338009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:01:14.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.500+0000 7fd341600700 1 -- 172.21.15.28:0/813846474 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd338026070 con 0x7fd344109190 2024-04-24T08:01:14.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.500+0000 7fd341600700 1 -- 172.21.15.28:0/813846474 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd33800fde0 con 0x7fd344109190 2024-04-24T08:01:14.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.500+0000 7fd341600700 1 -- 172.21.15.28:0/813846474 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd338017650 con 0x7fd344109190 2024-04-24T08:01:14.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.500+0000 7fd34ba00700 1 -- 172.21.15.28:0/813846474 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd3441a72c0 con 0x7fd344109190 2024-04-24T08:01:14.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.500+0000 7fd34ba00700 1 -- 172.21.15.28:0/813846474 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3441a76c0 con 0x7fd344109190 2024-04-24T08:01:14.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.501+0000 7fd341600700 1 -- 172.21.15.28:0/813846474 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 13) ==== 46614+0+0 (secure 0 0 0) 0x7fd338029070 con 0x7fd344109190 2024-04-24T08:01:14.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.501+0000 7fd34ba00700 1 -- 172.21.15.28:0/813846474 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd344069820 con 0x7fd344109190 2024-04-24T08:01:14.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.501+0000 7fd341600700 1 --2- 172.21.15.28:0/813846474 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd32c039890 0x7fd32c03bd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:01:14.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.502+0000 7fd341600700 1 -- 172.21.15.28:0/813846474 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fd33804da60 con 0x7fd344109190 2024-04-24T08:01:14.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.502+0000 7fd343e00700 1 --2- 172.21.15.28:0/813846474 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd32c039890 0x7fd32c03bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:01:14.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.503+0000 7fd343e00700 1 --2- 172.21.15.28:0/813846474 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd32c039890 0x7fd32c03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd330009c60 tx=0x7fd330009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:01:14.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.507+0000 7fd341600700 1 -- 172.21.15.28:0/813846474 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd338013070 con 0x7fd344109190 2024-04-24T08:01:14.567 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:14.229+0000 7f2194f22200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T08:01:14.567 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:14.397+0000 7f2194f22200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T08:01:14.567 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:14.566+0000 7f2194f22200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T08:01:14.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:14.751+0000 7fd34ba00700 1 -- 172.21.15.28:0/813846474 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fd344062f30 con 0x7fd32c039890 2024-04-24T08:01:15.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:14 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:15.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:14 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:15.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:14 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:15.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:14 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:15.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:14 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:15.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:14 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:01:15.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:14 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:15.014 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:14 smithi028 ceph-mon[27786]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:15.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:14 smithi028 ceph-mon[27786]: Reconfiguring mgr.x (unknown last config time)... 2024-04-24T08:01:15.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:14 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:01:15.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:14 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:01:15.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:14 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:15.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:14 smithi028 ceph-mon[27786]: Reconfiguring daemon mgr.x on smithi028 2024-04-24T08:01:15.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:14 smithi028 ceph-mon[27786]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:01:15.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:14 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:01:15.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:14 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:01:15.015 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:14 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:15.031 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:14.692+0000 7f0be82a3200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T08:01:15.031 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:14.784+0000 7f0be82a3200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T08:01:15.031 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:15.031 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:15.032 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:15.032 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:15.032 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:15.032 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:01:15.032 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:15.032 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-mon[32836]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:15.032 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-mon[32836]: Reconfiguring mgr.x (unknown last config time)... 2024-04-24T08:01:15.032 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:01:15.032 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:01:15.032 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:15.032 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-mon[32836]: Reconfiguring daemon mgr.x on smithi028 2024-04-24T08:01:15.032 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-mon[32836]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:01:15.032 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:01:15.033 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:01:15.033 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:14 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:15.127 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:14.850+0000 7f2194f22200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T08:01:15.128 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:14.940+0000 7f2194f22200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T08:01:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:01:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-mon[32438]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-mon[32438]: Reconfiguring mgr.x (unknown last config time)... 2024-04-24T08:01:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-04-24T08:01:15.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-04-24T08:01:15.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:15.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-mon[32438]: Reconfiguring daemon mgr.x on smithi028 2024-04-24T08:01:15.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-mon[32438]: from='client.24119 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:01:15.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:01:15.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:01:15.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:14 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:15.379 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:15 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:15.029+0000 7f0be82a3200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T08:01:15.380 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:15 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:15.164+0000 7f0be82a3200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T08:01:15.380 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:15 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:15.260+0000 7f0be82a3200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T08:01:15.380 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:15 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:15.378+0000 7f0be82a3200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T08:01:15.627 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:15 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:15.140+0000 7f2194f22200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T08:01:15.628 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:15 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:15.330+0000 7f2194f22200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:01:16.004 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:15 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:15.709+0000 7f0be82a3200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T08:01:16.218 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:16 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:16.314 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:16 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:16.342 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:16 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:16.343 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:15 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:15.920+0000 7f2194f22200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T08:01:16.627 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:16 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:16.341+0000 7f2194f22200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T08:01:16.627 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:16 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:16.436+0000 7f2194f22200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T08:01:16.690 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:16 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:16.312+0000 7f0be82a3200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T08:01:16.690 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:01:16 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:01:16.530+0000 7f0be82a3200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T08:01:17.119 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:16 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:16.635+0000 7f2194f22200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T08:01:17.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:17.157+0000 7fd341600700 1 -- 172.21.15.28:0/813846474 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mgrmap(e 14) ==== 92602+0+0 (secure 0 0 0) 0x7fd338009d10 con 0x7fd344109190 2024-04-24T08:01:17.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:17 smithi028 ceph-mon[27786]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:17.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:17 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:17.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:17 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:17.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:17 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:01:17.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:17 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:17.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:17 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:01:17.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:17 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:17.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:17 smithi028 ceph-mon[27786]: Standby manager daemon z started 2024-04-24T08:01:17.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:17 smithi028 ceph-mon[27786]: from='mgr.? 172.21.15.194:0/90756049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-24T08:01:17.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:17 smithi028 ceph-mon[27786]: from='mgr.? 172.21.15.194:0/90756049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:01:17.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:17 smithi028 ceph-mon[27786]: from='mgr.? 172.21.15.194:0/90756049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-24T08:01:17.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:17 smithi028 ceph-mon[27786]: from='mgr.? 172.21.15.194:0/90756049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:01:17.377 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:17.160+0000 7f2194f22200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T08:01:17.378 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:17.251+0000 7f2194f22200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T08:01:17.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-mon[32438]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:17.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:17.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:17.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:01:17.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:17.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:01:17.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:17.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-mon[32438]: Standby manager daemon z started 2024-04-24T08:01:17.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-mon[32438]: from='mgr.? 172.21.15.194:0/90756049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-24T08:01:17.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-mon[32438]: from='mgr.? 172.21.15.194:0/90756049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:01:17.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-mon[32438]: from='mgr.? 172.21.15.194:0/90756049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-24T08:01:17.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-mon[32438]: from='mgr.? 172.21.15.194:0/90756049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:01:17.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:17 smithi194 ceph-mon[32836]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:17.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:17 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:17.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:17 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:17.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:17 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:01:17.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:17 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:17.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:17 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:01:17.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:17 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:17.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:17 smithi194 ceph-mon[32836]: Standby manager daemon z started 2024-04-24T08:01:17.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:17 smithi194 ceph-mon[32836]: from='mgr.? 172.21.15.194:0/90756049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-24T08:01:17.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:17 smithi194 ceph-mon[32836]: from='mgr.? 172.21.15.194:0/90756049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:01:17.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:17 smithi194 ceph-mon[32836]: from='mgr.? 172.21.15.194:0/90756049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-24T08:01:17.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:17 smithi194 ceph-mon[32836]: from='mgr.? 172.21.15.194:0/90756049' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:01:17.830 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:17.489+0000 7f2194f22200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T08:01:17.830 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:17.620+0000 7f2194f22200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T08:01:17.830 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:17.714+0000 7f2194f22200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T08:01:17.830 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:17 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:17.829+0000 7f2194f22200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T08:01:18.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:18 smithi194 ceph-mon[32836]: mgrmap e14: x(active, since 87s), standbys: z 2024-04-24T08:01:18.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:18 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-24T08:01:18.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:18 smithi028 ceph-mon[27786]: mgrmap e14: x(active, since 87s), standbys: z 2024-04-24T08:01:18.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:18 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-24T08:01:18.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:18 smithi117 ceph-mon[32438]: mgrmap e14: x(active, since 87s), standbys: z 2024-04-24T08:01:18.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:18 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-24T08:01:18.628 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:18 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:18.129+0000 7f2194f22200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T08:01:19.128 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:18 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:18.722+0000 7f2194f22200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T08:01:19.128 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:01:18 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:01:18.935+0000 7f2194f22200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T08:01:19.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:19.179+0000 7fd341600700 1 -- 172.21.15.28:0/813846474 <== mon.2 v2:172.21.15.117:3300/0 8 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fd33802b7c0 con 0x7fd344109190 2024-04-24T08:01:19.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:19 smithi194 ceph-mon[32836]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:19.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:19 smithi194 ceph-mon[32836]: from='mgr.? 172.21.15.117:0/2555422910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T08:01:19.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:19 smithi194 ceph-mon[32836]: Standby manager daemon y started 2024-04-24T08:01:19.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:19 smithi194 ceph-mon[32836]: from='mgr.? 172.21.15.117:0/2555422910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:01:19.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:19 smithi194 ceph-mon[32836]: from='mgr.? 172.21.15.117:0/2555422910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T08:01:19.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:19 smithi194 ceph-mon[32836]: from='mgr.? 172.21.15.117:0/2555422910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:01:19.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:19 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2955292719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a"}]: dispatch 2024-04-24T08:01:19.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:19 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2955292719' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a"}]': finished 2024-04-24T08:01:19.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:19 smithi194 ceph-mon[32836]: osdmap e5: 1 total, 0 up, 1 in 2024-04-24T08:01:19.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:19 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:01:19.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:19 smithi028 ceph-mon[27786]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:19.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:19 smithi028 ceph-mon[27786]: from='mgr.? 172.21.15.117:0/2555422910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T08:01:19.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:19 smithi028 ceph-mon[27786]: Standby manager daemon y started 2024-04-24T08:01:19.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:19 smithi028 ceph-mon[27786]: from='mgr.? 172.21.15.117:0/2555422910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:01:19.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:19 smithi028 ceph-mon[27786]: from='mgr.? 172.21.15.117:0/2555422910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T08:01:19.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:19 smithi028 ceph-mon[27786]: from='mgr.? 172.21.15.117:0/2555422910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:01:19.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:19 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2955292719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a"}]: dispatch 2024-04-24T08:01:19.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:19 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2955292719' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a"}]': finished 2024-04-24T08:01:19.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:19 smithi028 ceph-mon[27786]: osdmap e5: 1 total, 0 up, 1 in 2024-04-24T08:01:19.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:19 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:01:19.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:19 smithi117 ceph-mon[32438]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:19.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:19 smithi117 ceph-mon[32438]: from='mgr.? 172.21.15.117:0/2555422910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T08:01:19.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:19 smithi117 ceph-mon[32438]: Standby manager daemon y started 2024-04-24T08:01:19.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:19 smithi117 ceph-mon[32438]: from='mgr.? 172.21.15.117:0/2555422910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:01:19.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:19 smithi117 ceph-mon[32438]: from='mgr.? 172.21.15.117:0/2555422910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T08:01:19.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:19 smithi117 ceph-mon[32438]: from='mgr.? 172.21.15.117:0/2555422910' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:01:19.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:19 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2955292719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a"}]: dispatch 2024-04-24T08:01:19.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:19 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2955292719' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a"}]': finished 2024-04-24T08:01:19.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:19 smithi117 ceph-mon[32438]: osdmap e5: 1 total, 0 up, 1 in 2024-04-24T08:01:19.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:19 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:01:20.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:20 smithi194 ceph-mon[32836]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-24T08:01:20.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:20 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T08:01:20.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:20 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2482597088' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:01:20.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:20 smithi028 ceph-mon[27786]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-24T08:01:20.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:20 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T08:01:20.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:20 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2482597088' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:01:20.570 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:20 smithi117 ceph-mon[32438]: mgrmap e15: x(active, since 89s), standbys: z, y 2024-04-24T08:01:20.570 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:20 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T08:01:20.571 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:20 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2482597088' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:01:21.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:21 smithi194 ceph-mon[32836]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:21.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:21 smithi028 ceph-mon[27786]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:21.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:21 smithi117 ceph-mon[32438]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:23.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:23 smithi194 ceph-mon[32836]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:23.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:23 smithi028 ceph-mon[27786]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:23.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:23 smithi117 ceph-mon[32438]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:25.452 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:25 smithi028 ceph-mon[27786]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:25.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:25 smithi117 ceph-mon[32438]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:25.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:25 smithi194 ceph-mon[32836]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:27.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:27 smithi028 ceph-mon[27786]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:27.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:27 smithi117 ceph-mon[32438]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:27.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:27 smithi194 ceph-mon[32836]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:29.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:29 smithi028 ceph-mon[27786]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:29.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T08:01:29.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:29.469 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:29 smithi028 ceph-mon[27786]: Deploying daemon osd.0 on smithi028 2024-04-24T08:01:29.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:29 smithi117 ceph-mon[32438]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:29.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T08:01:29.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:29.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:29 smithi117 ceph-mon[32438]: Deploying daemon osd.0 on smithi028 2024-04-24T08:01:29.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:29 smithi194 ceph-mon[32836]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:29.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-04-24T08:01:29.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:29.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:29 smithi194 ceph-mon[32836]: Deploying daemon osd.0 on smithi028 2024-04-24T08:01:31.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:31 smithi028 ceph-mon[27786]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:31.492 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:31 smithi194 ceph-mon[32836]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:31.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:31 smithi117 ceph-mon[32438]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:33.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:33 smithi028 ceph-mon[27786]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:33.643 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:33 smithi117 ceph-mon[32438]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:33.679 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:33 smithi194 ceph-mon[32836]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:35.517 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:35 smithi028 ceph-mon[27786]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:35.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:35 smithi117 ceph-mon[32438]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:35.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:35 smithi194 ceph-mon[32836]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:37.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:37 smithi028 ceph-mon[27786]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:37.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:37 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:37.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:37 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:01:37.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:37 smithi117 ceph-mon[32438]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:37.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:37 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:37.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:37 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:01:37.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:37 smithi194 ceph-mon[32836]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:37.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:37 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:37.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:37 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:01:38.477 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:38 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:38.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:38 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:38.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:38 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:39.324 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:39 smithi028 ceph-mon[27786]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:39.529 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:39 smithi117 ceph-mon[32438]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:39.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:39 smithi194 ceph-mon[32836]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:41.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:41 smithi028 ceph-mon[27786]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:41.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:41 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:41.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:41 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:41.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:41 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:41.419 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:41 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:01:41.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:41 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:41.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:41 smithi028 ceph-mon[27786]: from='osd.0 [v2:172.21.15.28:6802/1414246664,v1:172.21.15.28:6803/1414246664]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T08:01:41.420 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:41 smithi028 ceph-mon[27786]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T08:01:41.534 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:41 smithi117 ceph-mon[32438]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:41.535 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:41 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:41.535 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:41 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:41.535 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:41 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:41.535 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:41 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:01:41.535 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:41 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:41.535 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:41 smithi117 ceph-mon[32438]: from='osd.0 [v2:172.21.15.28:6802/1414246664,v1:172.21.15.28:6803/1414246664]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T08:01:41.535 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:41 smithi117 ceph-mon[32438]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T08:01:41.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:41 smithi194 ceph-mon[32836]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:41.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:41 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:41.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:41 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:41.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:41 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:41.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:41 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:01:41.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:41 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:41.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:41 smithi194 ceph-mon[32836]: from='osd.0 [v2:172.21.15.28:6802/1414246664,v1:172.21.15.28:6803/1414246664]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T08:01:41.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:41 smithi194 ceph-mon[32836]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-04-24T08:01:42.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:42.261+0000 7fd341600700 1 -- 172.21.15.28:0/813846474 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fd344062f30 con 0x7fd32c039890 2024-04-24T08:01:42.262 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 0 on host 'smithi028' 2024-04-24T08:01:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:42.265+0000 7fd34ba00700 1 -- 172.21.15.28:0/813846474 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd32c039890 msgr2=0x7fd32c03bd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:42.265+0000 7fd34ba00700 1 --2- 172.21.15.28:0/813846474 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd32c039890 0x7fd32c03bd50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd330009c60 tx=0x7fd330009260 comp rx=0 tx=0).stop 2024-04-24T08:01:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:42.265+0000 7fd34ba00700 1 -- 172.21.15.28:0/813846474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd344109190 msgr2=0x7fd34419f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:42.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:42.265+0000 7fd34ba00700 1 --2- 172.21.15.28:0/813846474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd344109190 0x7fd34419f3d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd338000c00 tx=0x7fd338009580 comp rx=0 tx=0).stop 2024-04-24T08:01:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:42.265+0000 7fd34ba00700 1 -- 172.21.15.28:0/813846474 shutdown_connections 2024-04-24T08:01:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:42.265+0000 7fd34ba00700 1 --2- 172.21.15.28:0/813846474 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd32c039890 0x7fd32c03bd50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:42.265+0000 7fd34ba00700 1 --2- 172.21.15.28:0/813846474 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd34410ad30 0x7fd3441a6990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:42.265+0000 7fd34ba00700 1 --2- 172.21.15.28:0/813846474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd344109190 0x7fd34419f3d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:42.265+0000 7fd34ba00700 1 --2- 172.21.15.28:0/813846474 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd34410a390 0x7fd34419f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:42.265+0000 7fd34ba00700 1 -- 172.21.15.28:0/813846474 >> 172.21.15.28:0/813846474 conn(0x7fd344078160 msgr2=0x7fd34410d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:01:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:42.265+0000 7fd34ba00700 1 -- 172.21.15.28:0/813846474 shutdown_connections 2024-04-24T08:01:42.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:01:42.265+0000 7fd34ba00700 1 -- 172.21.15.28:0/813846474 wait complete. 2024-04-24T08:01:42.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:42 smithi028 ceph-mon[27786]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-24T08:01:42.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:42 smithi028 ceph-mon[27786]: from='osd.0 [v2:172.21.15.28:6802/1414246664,v1:172.21.15.28:6803/1414246664]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:01:42.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:42 smithi028 ceph-mon[27786]: osdmap e6: 1 total, 0 up, 1 in 2024-04-24T08:01:42.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:42 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:01:42.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:42 smithi028 ceph-mon[27786]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:01:42.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:42 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:01:42.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:42 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:42.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:42 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:42.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:42 smithi117 ceph-mon[32438]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-24T08:01:42.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:42 smithi117 ceph-mon[32438]: from='osd.0 [v2:172.21.15.28:6802/1414246664,v1:172.21.15.28:6803/1414246664]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:01:42.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:42 smithi117 ceph-mon[32438]: osdmap e6: 1 total, 0 up, 1 in 2024-04-24T08:01:42.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:42 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:01:42.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:42 smithi117 ceph-mon[32438]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:01:42.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:42 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:01:42.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:42 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:42.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:42 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:42.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:42 smithi194 ceph-mon[32836]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-04-24T08:01:42.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:42 smithi194 ceph-mon[32836]: from='osd.0 [v2:172.21.15.28:6802/1414246664,v1:172.21.15.28:6803/1414246664]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:01:42.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:42 smithi194 ceph-mon[32836]: osdmap e6: 1 total, 0 up, 1 in 2024-04-24T08:01:42.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:42 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:01:42.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:42 smithi194 ceph-mon[32836]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:01:42.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:42 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:01:42.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:42 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:42.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:42 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:43.089 DEBUG:teuthology.orchestra.run.smithi028:osd.0> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.0.service 2024-04-24T08:01:43.094 INFO:tasks.cephadm:Deploying osd.1 on smithi117 with /dev/vg_nvme/lv_4... 2024-04-24T08:01:43.094 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-24T08:01:43.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:43 smithi028 ceph-mon[27786]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:43.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:43 smithi028 ceph-mon[27786]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-24T08:01:43.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:43 smithi028 ceph-mon[27786]: osdmap e7: 1 total, 0 up, 1 in 2024-04-24T08:01:43.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:43 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:01:43.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:43 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:01:43.338 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:01:43.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:43 smithi117 ceph-mon[32438]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:43.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:43 smithi117 ceph-mon[32438]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-24T08:01:43.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:43 smithi117 ceph-mon[32438]: osdmap e7: 1 total, 0 up, 1 in 2024-04-24T08:01:43.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:43 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:01:43.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:43 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:01:43.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:43 smithi194 ceph-mon[32836]: pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:43.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:43 smithi194 ceph-mon[32836]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-24T08:01:43.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:43 smithi194 ceph-mon[32836]: osdmap e7: 1 total, 0 up, 1 in 2024-04-24T08:01:43.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:43 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:01:43.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:43 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:01:44.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:44 smithi028 ceph-mon[27786]: purged_snaps scrub starts 2024-04-24T08:01:44.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:44 smithi028 ceph-mon[27786]: purged_snaps scrub ok 2024-04-24T08:01:44.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:44 smithi028 ceph-mon[27786]: osd.0 [v2:172.21.15.28:6802/1414246664,v1:172.21.15.28:6803/1414246664] boot 2024-04-24T08:01:44.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:44 smithi028 ceph-mon[27786]: osdmap e8: 1 total, 1 up, 1 in 2024-04-24T08:01:44.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:44 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:01:44.622 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:44 smithi117 ceph-mon[32438]: purged_snaps scrub starts 2024-04-24T08:01:44.622 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:44 smithi117 ceph-mon[32438]: purged_snaps scrub ok 2024-04-24T08:01:44.622 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:44 smithi117 ceph-mon[32438]: osd.0 [v2:172.21.15.28:6802/1414246664,v1:172.21.15.28:6803/1414246664] boot 2024-04-24T08:01:44.622 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:44 smithi117 ceph-mon[32438]: osdmap e8: 1 total, 1 up, 1 in 2024-04-24T08:01:44.622 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:44 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:01:44.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:44 smithi194 ceph-mon[32836]: purged_snaps scrub starts 2024-04-24T08:01:44.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:44 smithi194 ceph-mon[32836]: purged_snaps scrub ok 2024-04-24T08:01:44.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:44 smithi194 ceph-mon[32836]: osd.0 [v2:172.21.15.28:6802/1414246664,v1:172.21.15.28:6803/1414246664] boot 2024-04-24T08:01:44.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:44 smithi194 ceph-mon[32836]: osdmap e8: 1 total, 1 up, 1 in 2024-04-24T08:01:44.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:44 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:01:45.173 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-24T08:01:45.173 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-24T08:01:45.173 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-04-24T08:01:45.173 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-04-24T08:01:45.173 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0134044 s, 782 MB/s 2024-04-24T08:01:45.173 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 2024-04-24T08:01:45.173 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-04-24T08:01:45.485 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:45 smithi117 ceph-mon[32438]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:45.486 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:45 smithi117 ceph-mon[32438]: osdmap e9: 1 total, 1 up, 1 in 2024-04-24T08:01:45.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:45 smithi028 ceph-mon[27786]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:45.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:45 smithi028 ceph-mon[27786]: osdmap e9: 1 total, 1 up, 1 in 2024-04-24T08:01:45.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:45 smithi194 ceph-mon[32836]: pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-04-24T08:01:45.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:45 smithi194 ceph-mon[32836]: osdmap e9: 1 total, 1 up, 1 in 2024-04-24T08:01:45.797 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi117:vg_nvme/lv_4 2024-04-24T08:01:46.030 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:01:47.486 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:47 smithi117 ceph-mon[32438]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:47.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.506+0000 7fa1c8a6f700 1 -- 172.21.15.117:0/4068698117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1c4103660 msgr2=0x7fa1c4105a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:47.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.506+0000 7fa1c8a6f700 1 --2- 172.21.15.117:0/4068698117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1c4103660 0x7fa1c4105a50 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fa1b4009990 tx=0x7fa1b4009ca0 comp rx=0 tx=0).stop 2024-04-24T08:01:47.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.506+0000 7fa1c8a6f700 1 -- 172.21.15.117:0/4068698117 shutdown_connections 2024-04-24T08:01:47.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.506+0000 7fa1c8a6f700 1 --2- 172.21.15.117:0/4068698117 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1c41088c0 0x7fa1c410acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:47.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.506+0000 7fa1c8a6f700 1 --2- 172.21.15.117:0/4068698117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1c4103660 0x7fa1c4105a50 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:47.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.506+0000 7fa1c8a6f700 1 --2- 172.21.15.117:0/4068698117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1c4105f90 0x7fa1c4108380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:47.507 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.506+0000 7fa1c8a6f700 1 -- 172.21.15.117:0/4068698117 >> 172.21.15.117:0/4068698117 conn(0x7fa1c40fcff0 msgr2=0x7fa1c40ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:01:47.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.506+0000 7fa1c8a6f700 1 -- 172.21.15.117:0/4068698117 shutdown_connections 2024-04-24T08:01:47.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.506+0000 7fa1c8a6f700 1 -- 172.21.15.117:0/4068698117 wait complete. 2024-04-24T08:01:47.508 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1c8a6f700 1 Processor -- start 2024-04-24T08:01:47.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1c8a6f700 1 -- start start 2024-04-24T08:01:47.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1c8a6f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1c4103660 0x7fa1c4198ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:01:47.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1c8a6f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1c4105f90 0x7fa1c4199410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:01:47.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1c8a6f700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1c41088c0 0x7fa1c41a0490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:01:47.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1c8a6f700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c410e820 con 0x7fa1c41088c0 2024-04-24T08:01:47.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1c8a6f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c410e6a0 con 0x7fa1c4105f90 2024-04-24T08:01:47.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1c8a6f700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa1c410e9a0 con 0x7fa1c4103660 2024-04-24T08:01:47.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1b9d9b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1c4105f90 0x7fa1c4199410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:01:47.510 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1b9d9b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1c4105f90 0x7fa1c4199410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:38980/0 (socket says 172.21.15.117:38980) 2024-04-24T08:01:47.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1b9d9b700 1 -- 172.21.15.117:0/3653295167 learned_addr learned my addr 172.21.15.117:0/3653295167 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-04-24T08:01:47.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1c259c700 1 --2- 172.21.15.117:0/3653295167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1c4103660 0x7fa1c4198ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:01:47.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1b9d9b700 1 -- 172.21.15.117:0/3653295167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1c4103660 msgr2=0x7fa1c4198ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:47.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1b9d9b700 1 --2- 172.21.15.117:0/3653295167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1c4103660 0x7fa1c4198ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:47.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1b9d9b700 1 -- 172.21.15.117:0/3653295167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1c41088c0 msgr2=0x7fa1c41a0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:01:47.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1b9d9b700 1 --2- 172.21.15.117:0/3653295167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1c41088c0 0x7fa1c41a0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:01:47.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1b9d9b700 1 -- 172.21.15.117:0/3653295167 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1c41a0b30 con 0x7fa1c4105f90 2024-04-24T08:01:47.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.507+0000 7fa1b9d9b700 1 --2- 172.21.15.117:0/3653295167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1c4105f90 0x7fa1c4199410 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00b530 tx=0x7fa1ac00b8f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:01:47.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.508+0000 7fa1bbfff700 1 -- 172.21.15.117:0/3653295167 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac015020 con 0x7fa1c4105f90 2024-04-24T08:01:47.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.508+0000 7fa1c8a6f700 1 -- 172.21.15.117:0/3653295167 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1c41a0e20 con 0x7fa1c4105f90 2024-04-24T08:01:47.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.508+0000 7fa1bbfff700 1 -- 172.21.15.117:0/3653295167 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1ac011d10 con 0x7fa1c4105f90 2024-04-24T08:01:47.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.508+0000 7fa1bbfff700 1 -- 172.21.15.117:0/3653295167 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1ac00f2e0 con 0x7fa1c4105f90 2024-04-24T08:01:47.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.508+0000 7fa1c8a6f700 1 -- 172.21.15.117:0/3653295167 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1c41a1350 con 0x7fa1c4105f90 2024-04-24T08:01:47.511 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.509+0000 7fa1c8a6f700 1 -- 172.21.15.117:0/3653295167 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1c40697e0 con 0x7fa1c4105f90 2024-04-24T08:01:47.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.510+0000 7fa1bbfff700 1 -- 172.21.15.117:0/3653295167 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fa1ac018020 con 0x7fa1c4105f90 2024-04-24T08:01:47.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.510+0000 7fa1bbfff700 1 --2- 172.21.15.117:0/3653295167 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa1b00a3bb0 0x7fa1b00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:01:47.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.511+0000 7fa1bbfff700 1 -- 172.21.15.117:0/3653295167 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7fa1ac0ce170 con 0x7fa1c4105f90 2024-04-24T08:01:47.512 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.511+0000 7fa1c259c700 1 --2- 172.21.15.117:0/3653295167 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa1b00a3bb0 0x7fa1b00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:01:47.513 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.512+0000 7fa1c259c700 1 --2- 172.21.15.117:0/3653295167 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa1b00a3bb0 0x7fa1b00a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa1b400b5c0 tx=0x7fa1b4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:01:47.516 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.515+0000 7fa1bbfff700 1 -- 172.21.15.117:0/3653295167 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa1ac01f020 con 0x7fa1c4105f90 2024-04-24T08:01:47.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:47 smithi028 ceph-mon[27786]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:47.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:47 smithi194 ceph-mon[32836]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:47.724 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:01:47.723+0000 7fa1c8a6f700 1 -- 172.21.15.117:0/3653295167 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7fa1c4062f30 con 0x7fa1b00a3bb0 2024-04-24T08:01:48.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:48 smithi194 ceph-mon[32836]: Detected new or changed devices on smithi028 2024-04-24T08:01:48.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:48 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:48.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:48 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:48.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:48 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:01:48.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:48 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi028 to 17094M 2024-04-24T08:01:48.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:48 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:48.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:48 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:48.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:48 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:01:48.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:48 smithi194 ceph-mon[32836]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:01:48.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:48 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:48.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:48 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:01:48.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:48 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:01:48.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:48 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:49.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:48 smithi028 ceph-mon[27786]: Detected new or changed devices on smithi028 2024-04-24T08:01:49.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:48 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:49.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:48 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:49.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:48 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:01:49.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:48 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi028 to 17094M 2024-04-24T08:01:49.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:48 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:49.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:48 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:49.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:48 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:01:49.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:48 smithi028 ceph-mon[27786]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:01:49.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:48 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:49.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:48 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:01:49.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:48 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:01:49.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:48 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:49.092 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:48 smithi117 ceph-mon[32438]: Detected new or changed devices on smithi028 2024-04-24T08:01:49.093 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:48 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:49.093 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:48 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:49.093 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:48 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:01:49.093 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:48 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi028 to 17094M 2024-04-24T08:01:49.093 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:48 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:49.093 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:48 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:49.093 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:48 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:01:49.093 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:48 smithi117 ceph-mon[32438]: from='client.24146 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:01:49.093 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:48 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:01:49.093 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:48 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:01:49.093 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:48 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:01:49.093 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:48 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:49.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:49 smithi117 ceph-mon[32438]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:50.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:49 smithi028 ceph-mon[27786]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:50.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:49 smithi194 ceph-mon[32836]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:51.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:50 smithi028 ceph-mon[27786]: from='client.? 172.21.15.117:0/1164996629' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1b6b23d-4deb-4210-af96-64b7217b06aa"}]: dispatch 2024-04-24T08:01:51.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:50 smithi028 ceph-mon[27786]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1b6b23d-4deb-4210-af96-64b7217b06aa"}]: dispatch 2024-04-24T08:01:51.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:50 smithi028 ceph-mon[27786]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a1b6b23d-4deb-4210-af96-64b7217b06aa"}]': finished 2024-04-24T08:01:51.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:50 smithi028 ceph-mon[27786]: osdmap e10: 2 total, 1 up, 2 in 2024-04-24T08:01:51.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:01:51.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:50 smithi117 ceph-mon[32438]: from='client.? 172.21.15.117:0/1164996629' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1b6b23d-4deb-4210-af96-64b7217b06aa"}]: dispatch 2024-04-24T08:01:51.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:50 smithi117 ceph-mon[32438]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1b6b23d-4deb-4210-af96-64b7217b06aa"}]: dispatch 2024-04-24T08:01:51.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:50 smithi117 ceph-mon[32438]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a1b6b23d-4deb-4210-af96-64b7217b06aa"}]': finished 2024-04-24T08:01:51.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:50 smithi117 ceph-mon[32438]: osdmap e10: 2 total, 1 up, 2 in 2024-04-24T08:01:51.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:50 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:01:51.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:50 smithi194 ceph-mon[32836]: from='client.? 172.21.15.117:0/1164996629' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1b6b23d-4deb-4210-af96-64b7217b06aa"}]: dispatch 2024-04-24T08:01:51.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:50 smithi194 ceph-mon[32836]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "a1b6b23d-4deb-4210-af96-64b7217b06aa"}]: dispatch 2024-04-24T08:01:51.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:50 smithi194 ceph-mon[32836]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "a1b6b23d-4deb-4210-af96-64b7217b06aa"}]': finished 2024-04-24T08:01:51.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:50 smithi194 ceph-mon[32836]: osdmap e10: 2 total, 1 up, 2 in 2024-04-24T08:01:51.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:50 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:01:52.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:51 smithi028 ceph-mon[27786]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:52.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:51 smithi028 ceph-mon[27786]: from='client.? 172.21.15.117:0/1310687563' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:01:52.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:51 smithi117 ceph-mon[32438]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:52.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:51 smithi117 ceph-mon[32438]: from='client.? 172.21.15.117:0/1310687563' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:01:52.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:51 smithi194 ceph-mon[32836]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:52.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:51 smithi194 ceph-mon[32836]: from='client.? 172.21.15.117:0/1310687563' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:01:54.026 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:53 smithi117 ceph-mon[32438]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:54.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:53 smithi028 ceph-mon[27786]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:54.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:53 smithi194 ceph-mon[32836]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:55.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:55 smithi117 ceph-mon[32438]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:56.042 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:55 smithi028 ceph-mon[27786]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:56.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:55 smithi194 ceph-mon[32836]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:58.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:57 smithi028 ceph-mon[27786]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:58.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:57 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T08:01:58.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:57 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:58.080 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:57 smithi117 ceph-mon[32438]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:58.080 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:57 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T08:01:58.080 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:57 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:58.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:57 smithi194 ceph-mon[32836]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:01:58.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:57 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-04-24T08:01:58.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:57 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:01:58.971 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:58 smithi117 ceph-mon[32438]: Deploying daemon osd.1 on smithi117 2024-04-24T08:01:59.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:58 smithi028 ceph-mon[27786]: Deploying daemon osd.1 on smithi117 2024-04-24T08:01:59.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:58 smithi194 ceph-mon[32836]: Deploying daemon osd.1 on smithi117 2024-04-24T08:01:59.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:01:59 smithi117 ceph-mon[32438]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:00.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:01:59 smithi028 ceph-mon[27786]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:00.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:01:59 smithi194 ceph-mon[32836]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:01.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:01 smithi117 ceph-mon[32438]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:02.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:01 smithi028 ceph-mon[27786]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:02.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:01 smithi194 ceph-mon[32836]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:04.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:03 smithi028 ceph-mon[27786]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:04.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:03 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:04.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:03 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:04.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:03 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:02:04.091 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:03 smithi117 ceph-mon[32438]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:04.091 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:03 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:04.091 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:03 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:04.091 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:03 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:02:04.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:03 smithi194 ceph-mon[32836]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:04.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:03 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:04.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:03 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:04.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:03 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:02:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:05 smithi117 ceph-mon[32438]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:05 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:05 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:05 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:05 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:02:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:05 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:06.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:05 smithi028 ceph-mon[27786]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:06.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:06.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:06.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:06.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:02:06.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:06.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:05 smithi194 ceph-mon[32836]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:06.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:05 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:06.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:05 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:06.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:05 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:06.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:05 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:02:06.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:05 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:06.266 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:02:06.265+0000 7fa1bbfff700 1 -- 172.21.15.117:0/3653295167 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa1c4062f30 con 0x7fa1b00a3bb0 2024-04-24T08:02:06.270 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 1 on host 'smithi117' 2024-04-24T08:02:06.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:02:06.267+0000 7fa1c8a6f700 1 -- 172.21.15.117:0/3653295167 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa1b00a3bb0 msgr2=0x7fa1b00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:02:06.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:02:06.267+0000 7fa1c8a6f700 1 --2- 172.21.15.117:0/3653295167 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa1b00a3bb0 0x7fa1b00a6070 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fa1b400b5c0 tx=0x7fa1b4005fb0 comp rx=0 tx=0).stop 2024-04-24T08:02:06.271 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:02:06.267+0000 7fa1c8a6f700 1 -- 172.21.15.117:0/3653295167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1c4105f90 msgr2=0x7fa1c4199410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:02:06.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:02:06.267+0000 7fa1c8a6f700 1 --2- 172.21.15.117:0/3653295167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1c4105f90 0x7fa1c4199410 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa1ac00b530 tx=0x7fa1ac00b8f0 comp rx=0 tx=0).stop 2024-04-24T08:02:06.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:02:06.267+0000 7fa1c8a6f700 1 -- 172.21.15.117:0/3653295167 shutdown_connections 2024-04-24T08:02:06.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:02:06.267+0000 7fa1c8a6f700 1 --2- 172.21.15.117:0/3653295167 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa1b00a3bb0 0x7fa1b00a6070 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:06.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:02:06.267+0000 7fa1c8a6f700 1 --2- 172.21.15.117:0/3653295167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1c41088c0 0x7fa1c41a0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:06.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:02:06.267+0000 7fa1c8a6f700 1 --2- 172.21.15.117:0/3653295167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1c4103660 0x7fa1c4198ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:06.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:02:06.267+0000 7fa1c8a6f700 1 --2- 172.21.15.117:0/3653295167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1c4105f90 0x7fa1c4199410 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:06.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:02:06.267+0000 7fa1c8a6f700 1 -- 172.21.15.117:0/3653295167 >> 172.21.15.117:0/3653295167 conn(0x7fa1c40fcff0 msgr2=0x7fa1c4104080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:02:06.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:02:06.267+0000 7fa1c8a6f700 1 -- 172.21.15.117:0/3653295167 shutdown_connections 2024-04-24T08:02:06.272 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:02:06.267+0000 7fa1c8a6f700 1 -- 172.21.15.117:0/3653295167 wait complete. 2024-04-24T08:02:06.828 DEBUG:teuthology.orchestra.run.smithi117:osd.1> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.1.service 2024-04-24T08:02:06.830 INFO:tasks.cephadm:Deploying osd.2 on smithi194 with /dev/vg_nvme/lv_4... 2024-04-24T08:02:06.830 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-04-24T08:02:07.073 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:02:07.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:07 smithi117 ceph-mon[32438]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:07.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:07.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:07.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:02:07.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:07 smithi117 ceph-mon[32438]: from='osd.1 [v2:172.21.15.117:6800/3269221814,v1:172.21.15.117:6801/3269221814]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T08:02:07.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:07 smithi117 ceph-mon[32438]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T08:02:07.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:07 smithi194 ceph-mon[32836]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:07.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:07.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:07.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:02:07.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:07 smithi194 ceph-mon[32836]: from='osd.1 [v2:172.21.15.117:6800/3269221814,v1:172.21.15.117:6801/3269221814]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T08:02:07.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:07 smithi194 ceph-mon[32836]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T08:02:07.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:07 smithi028 ceph-mon[27786]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:07.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:07.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:07.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:02:07.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:07 smithi028 ceph-mon[27786]: from='osd.1 [v2:172.21.15.117:6800/3269221814,v1:172.21.15.117:6801/3269221814]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T08:02:07.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:07 smithi028 ceph-mon[27786]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-04-24T08:02:08.343 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:08 smithi194 ceph-mon[32836]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-24T08:02:08.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:08 smithi194 ceph-mon[32836]: from='osd.1 [v2:172.21.15.117:6800/3269221814,v1:172.21.15.117:6801/3269221814]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:02:08.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:08 smithi194 ceph-mon[32836]: osdmap e11: 2 total, 1 up, 2 in 2024-04-24T08:02:08.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:08 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:02:08.344 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:08 smithi194 ceph-mon[32836]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:02:08.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:08 smithi117 ceph-mon[32438]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-24T08:02:08.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:08 smithi117 ceph-mon[32438]: from='osd.1 [v2:172.21.15.117:6800/3269221814,v1:172.21.15.117:6801/3269221814]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:02:08.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:08 smithi117 ceph-mon[32438]: osdmap e11: 2 total, 1 up, 2 in 2024-04-24T08:02:08.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:08 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:02:08.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:08 smithi117 ceph-mon[32438]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:02:08.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:08 smithi028 ceph-mon[27786]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-04-24T08:02:08.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:08 smithi028 ceph-mon[27786]: from='osd.1 [v2:172.21.15.117:6800/3269221814,v1:172.21.15.117:6801/3269221814]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:02:08.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:08 smithi028 ceph-mon[27786]: osdmap e11: 2 total, 1 up, 2 in 2024-04-24T08:02:08.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:08 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:02:08.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:08 smithi028 ceph-mon[27786]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:02:08.908 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-04-24T08:02:08.908 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-04-24T08:02:08.908 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-04-24T08:02:08.908 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-04-24T08:02:08.909 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114071 s, 919 MB/s 2024-04-24T08:02:08.909 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-04-24T08:02:09.128 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:02:08 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-1[37090]: 2024-04-24T08:02:08.685+0000 7fd7d3cec700 -1 osd.1 0 waiting for initial osdmap 2024-04-24T08:02:09.128 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:02:08 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-1[37090]: 2024-04-24T08:02:08.689+0000 7fd7cbad7700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:02:09.398 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:09 smithi117 ceph-mon[32438]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:09.398 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:09 smithi117 ceph-mon[32438]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-04-24T08:02:09.398 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:09 smithi117 ceph-mon[32438]: osdmap e12: 2 total, 1 up, 2 in 2024-04-24T08:02:09.399 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:09 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:02:09.399 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:09 smithi117 ceph-mon[32438]: from='osd.1 ' entity='osd.1' 2024-04-24T08:02:09.399 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:09 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:02:09.516 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:09 smithi194 ceph-mon[32836]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:09.516 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:09 smithi194 ceph-mon[32836]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-04-24T08:02:09.516 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:09 smithi194 ceph-mon[32836]: osdmap e12: 2 total, 1 up, 2 in 2024-04-24T08:02:09.516 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:09 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:02:09.516 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:09 smithi194 ceph-mon[32836]: from='osd.1 ' entity='osd.1' 2024-04-24T08:02:09.516 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:09 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:02:09.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:09 smithi028 ceph-mon[27786]: pgmap v70: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:09.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:09 smithi028 ceph-mon[27786]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-04-24T08:02:09.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:09 smithi028 ceph-mon[27786]: osdmap e12: 2 total, 1 up, 2 in 2024-04-24T08:02:09.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:09 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:02:09.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:09 smithi028 ceph-mon[27786]: from='osd.1 ' entity='osd.1' 2024-04-24T08:02:09.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:09 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:02:09.547 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_4 2024-04-24T08:02:09.779 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:02:10.414 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:10 smithi028 ceph-mon[27786]: purged_snaps scrub starts 2024-04-24T08:02:10.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:10 smithi028 ceph-mon[27786]: purged_snaps scrub ok 2024-04-24T08:02:10.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:10 smithi028 ceph-mon[27786]: osd.1 [v2:172.21.15.117:6800/3269221814,v1:172.21.15.117:6801/3269221814] boot 2024-04-24T08:02:10.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:10 smithi028 ceph-mon[27786]: osdmap e13: 2 total, 2 up, 2 in 2024-04-24T08:02:10.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:10 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:02:10.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:10 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:10.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:10 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:10.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:10 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:02:10.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:10 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:10.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:10 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:10.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:10 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:02:10.415 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:10 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:10.561 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:10 smithi194 ceph-mon[32836]: purged_snaps scrub starts 2024-04-24T08:02:10.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:10 smithi194 ceph-mon[32836]: purged_snaps scrub ok 2024-04-24T08:02:10.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:10 smithi194 ceph-mon[32836]: osd.1 [v2:172.21.15.117:6800/3269221814,v1:172.21.15.117:6801/3269221814] boot 2024-04-24T08:02:10.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:10 smithi194 ceph-mon[32836]: osdmap e13: 2 total, 2 up, 2 in 2024-04-24T08:02:10.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:10 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:02:10.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:10 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:10.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:10 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:10.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:10 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:02:10.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:10 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:10.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:10 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:10.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:10 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:02:10.562 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:10 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:10.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:10 smithi117 ceph-mon[32438]: purged_snaps scrub starts 2024-04-24T08:02:10.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:10 smithi117 ceph-mon[32438]: purged_snaps scrub ok 2024-04-24T08:02:10.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:10 smithi117 ceph-mon[32438]: osd.1 [v2:172.21.15.117:6800/3269221814,v1:172.21.15.117:6801/3269221814] boot 2024-04-24T08:02:10.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:10 smithi117 ceph-mon[32438]: osdmap e13: 2 total, 2 up, 2 in 2024-04-24T08:02:10.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:10 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:02:10.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:10 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:10.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:10 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:10.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:10 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:02:10.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:10 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:10.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:10 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:10.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:10 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:02:10.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:10 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:11.354 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.351+0000 7f1f8b9c3700 1 -- 172.21.15.194:0/3169150351 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f84103d50 msgr2=0x7f1f841041b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:02:11.354 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.351+0000 7f1f8b9c3700 1 --2- 172.21.15.194:0/3169150351 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f84103d50 0x7f1f841041b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f1f74009990 tx=0x7f1f74009ca0 comp rx=0 tx=0).stop 2024-04-24T08:02:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.351+0000 7f1f8b9c3700 1 -- 172.21.15.194:0/3169150351 shutdown_connections 2024-04-24T08:02:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.351+0000 7f1f8b9c3700 1 --2- 172.21.15.194:0/3169150351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f841046f0 0x7f1f8410afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.351+0000 7f1f8b9c3700 1 --2- 172.21.15.194:0/3169150351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f84102b50 0x7f1f84102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.351+0000 7f1f8b9c3700 1 --2- 172.21.15.194:0/3169150351 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f84103d50 0x7f1f841041b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.351+0000 7f1f8b9c3700 1 -- 172.21.15.194:0/3169150351 >> 172.21.15.194:0/3169150351 conn(0x7f1f840fe0f0 msgr2=0x7f1f84100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:02:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.352+0000 7f1f8b9c3700 1 -- 172.21.15.194:0/3169150351 shutdown_connections 2024-04-24T08:02:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.352+0000 7f1f8b9c3700 1 -- 172.21.15.194:0/3169150351 wait complete. 2024-04-24T08:02:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.352+0000 7f1f8b9c3700 1 Processor -- start 2024-04-24T08:02:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.352+0000 7f1f8b9c3700 1 -- start start 2024-04-24T08:02:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.352+0000 7f1f8b9c3700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f84102b50 0x7f1f84198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:02:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f8b9c3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f84103d50 0x7f1f84199490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:02:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f8b9c3700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f841046f0 0x7f1f841a04c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:02:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f8b9c3700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8410e830 con 0x7f1f841046f0 2024-04-24T08:02:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f8b9c3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8410e6b0 con 0x7f1f84103d50 2024-04-24T08:02:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f8b9c3700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1f8410e9b0 con 0x7f1f84102b50 2024-04-24T08:02:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f8975f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f84102b50 0x7f1f84198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:02:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f8975f700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f84102b50 0x7f1f84198f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:55802/0 (socket says 172.21.15.194:55802) 2024-04-24T08:02:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f8975f700 1 -- 172.21.15.194:0/1064731892 learned_addr learned my addr 172.21.15.194:0/1064731892 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-24T08:02:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f88f5e700 1 --2- 172.21.15.194:0/1064731892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f84103d50 0x7f1f84199490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:02:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f8975f700 1 -- 172.21.15.194:0/1064731892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f84103d50 msgr2=0x7f1f84199490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:02:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f8975f700 1 --2- 172.21.15.194:0/1064731892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f84103d50 0x7f1f84199490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f8975f700 1 -- 172.21.15.194:0/1064731892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f841046f0 msgr2=0x7f1f841a04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:02:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f8975f700 1 --2- 172.21.15.194:0/1064731892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f841046f0 0x7f1f841a04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f8975f700 1 -- 172.21.15.194:0/1064731892 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f841a0b60 con 0x7f1f84102b50 2024-04-24T08:02:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f88f5e700 1 --2- 172.21.15.194:0/1064731892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f84103d50 0x7f1f84199490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:02:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f8975f700 1 --2- 172.21.15.194:0/1064731892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f84102b50 0x7f1f84198f50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1f8000b530 tx=0x7f1f8000b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:02:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f7a7fc700 1 -- 172.21.15.194:0/1064731892 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f80015020 con 0x7f1f84102b50 2024-04-24T08:02:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f7a7fc700 1 -- 172.21.15.194:0/1064731892 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f80011d10 con 0x7f1f84102b50 2024-04-24T08:02:11.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f8b9c3700 1 -- 172.21.15.194:0/1064731892 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f841a0e50 con 0x7f1f84102b50 2024-04-24T08:02:11.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f7a7fc700 1 -- 172.21.15.194:0/1064731892 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f8000f2e0 con 0x7f1f84102b50 2024-04-24T08:02:11.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.353+0000 7f1f8b9c3700 1 -- 172.21.15.194:0/1064731892 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f841a1380 con 0x7f1f84102b50 2024-04-24T08:02:11.358 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.354+0000 7f1f8b9c3700 1 -- 172.21.15.194:0/1064731892 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1f840697e0 con 0x7f1f84102b50 2024-04-24T08:02:11.358 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.355+0000 7f1f7a7fc700 1 -- 172.21.15.194:0/1064731892 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7f1f80018020 con 0x7f1f84102b50 2024-04-24T08:02:11.358 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.356+0000 7f1f7a7fc700 1 --2- 172.21.15.194:0/1064731892 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1f700a3c30 0x7f1f700a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:02:11.358 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.356+0000 7f1f7a7fc700 1 -- 172.21.15.194:0/1064731892 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f1f800ce2a0 con 0x7f1f84102b50 2024-04-24T08:02:11.358 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.356+0000 7f1f88f5e700 1 --2- 172.21.15.194:0/1064731892 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1f700a3c30 0x7f1f700a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:02:11.358 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.357+0000 7f1f88f5e700 1 --2- 172.21.15.194:0/1064731892 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1f700a3c30 0x7f1f700a60f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1f740039c0 tx=0x7f1f74005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:02:11.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.361+0000 7f1f7a7fc700 1 -- 172.21.15.194:0/1064731892 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1f8001f020 con 0x7f1f84102b50 2024-04-24T08:02:11.573 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:11.572+0000 7f1f8b9c3700 1 -- 172.21.15.194:0/1064731892 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f1f84062f30 con 0x7f1f700a3c30 2024-04-24T08:02:12.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:11 smithi028 ceph-mon[27786]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:12.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:11 smithi028 ceph-mon[27786]: Detected new or changed devices on smithi117 2024-04-24T08:02:12.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:11 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi117 to 17088M 2024-04-24T08:02:12.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:11 smithi028 ceph-mon[27786]: osdmap e14: 2 total, 2 up, 2 in 2024-04-24T08:02:12.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:11 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:02:12.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:11 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:02:12.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:11 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:12.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:11 smithi117 ceph-mon[32438]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:12.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:11 smithi117 ceph-mon[32438]: Detected new or changed devices on smithi117 2024-04-24T08:02:12.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:11 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi117 to 17088M 2024-04-24T08:02:12.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:11 smithi117 ceph-mon[32438]: osdmap e14: 2 total, 2 up, 2 in 2024-04-24T08:02:12.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:11 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:02:12.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:11 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:02:12.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:11 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:12.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:11 smithi194 ceph-mon[32836]: pgmap v73: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-04-24T08:02:12.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:11 smithi194 ceph-mon[32836]: Detected new or changed devices on smithi117 2024-04-24T08:02:12.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:11 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi117 to 17088M 2024-04-24T08:02:12.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:11 smithi194 ceph-mon[32836]: osdmap e14: 2 total, 2 up, 2 in 2024-04-24T08:02:12.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:11 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:02:12.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:11 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:02:12.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:11 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:12.887 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:12 smithi194 ceph-mon[32836]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:02:13.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:12 smithi028 ceph-mon[27786]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:02:13.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:12 smithi117 ceph-mon[32438]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:02:14.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:13 smithi028 ceph-mon[27786]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:14.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:13 smithi117 ceph-mon[32438]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:14.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:13 smithi194 ceph-mon[32836]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:14.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:14 smithi194 ceph-mon[32836]: from='client.? 172.21.15.194:0/3810380852' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "223ab66d-7bb6-4225-b292-009846c746d4"}]: dispatch 2024-04-24T08:02:14.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:14 smithi194 ceph-mon[32836]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "223ab66d-7bb6-4225-b292-009846c746d4"}]: dispatch 2024-04-24T08:02:14.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:14 smithi194 ceph-mon[32836]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "223ab66d-7bb6-4225-b292-009846c746d4"}]': finished 2024-04-24T08:02:14.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:14 smithi194 ceph-mon[32836]: osdmap e15: 3 total, 2 up, 3 in 2024-04-24T08:02:14.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:14 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:02:15.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:14 smithi028 ceph-mon[27786]: from='client.? 172.21.15.194:0/3810380852' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "223ab66d-7bb6-4225-b292-009846c746d4"}]: dispatch 2024-04-24T08:02:15.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:14 smithi028 ceph-mon[27786]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "223ab66d-7bb6-4225-b292-009846c746d4"}]: dispatch 2024-04-24T08:02:15.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:14 smithi028 ceph-mon[27786]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "223ab66d-7bb6-4225-b292-009846c746d4"}]': finished 2024-04-24T08:02:15.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:14 smithi028 ceph-mon[27786]: osdmap e15: 3 total, 2 up, 3 in 2024-04-24T08:02:15.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:14 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:02:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:14 smithi117 ceph-mon[32438]: from='client.? 172.21.15.194:0/3810380852' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "223ab66d-7bb6-4225-b292-009846c746d4"}]: dispatch 2024-04-24T08:02:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:14 smithi117 ceph-mon[32438]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "223ab66d-7bb6-4225-b292-009846c746d4"}]: dispatch 2024-04-24T08:02:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:14 smithi117 ceph-mon[32438]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "223ab66d-7bb6-4225-b292-009846c746d4"}]': finished 2024-04-24T08:02:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:14 smithi117 ceph-mon[32438]: osdmap e15: 3 total, 2 up, 3 in 2024-04-24T08:02:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:14 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:02:16.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:15 smithi028 ceph-mon[27786]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:16.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:15 smithi028 ceph-mon[27786]: from='client.? 172.21.15.194:0/2807953600' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:02:16.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:15 smithi117 ceph-mon[32438]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:16.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:15 smithi117 ceph-mon[32438]: from='client.? 172.21.15.194:0/2807953600' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:02:16.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:15 smithi194 ceph-mon[32836]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:16.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:15 smithi194 ceph-mon[32836]: from='client.? 172.21.15.194:0/2807953600' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:02:18.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:17 smithi028 ceph-mon[27786]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:18.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:17 smithi117 ceph-mon[32438]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:18.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:17 smithi194 ceph-mon[32836]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:20.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:19 smithi028 ceph-mon[27786]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:20.063 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:19 smithi194 ceph-mon[32836]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:20.132 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:19 smithi117 ceph-mon[32438]: pgmap v79: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:22.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:21 smithi028 ceph-mon[27786]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:22.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:21 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T08:02:22.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:21 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:22.057 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:21 smithi194 ceph-mon[32836]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:22.057 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:21 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T08:02:22.057 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:21 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:22.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:21 smithi117 ceph-mon[32438]: pgmap v80: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:22.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:21 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-04-24T08:02:22.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:21 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:23.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:22 smithi028 ceph-mon[27786]: Deploying daemon osd.2 on smithi194 2024-04-24T08:02:23.080 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:22 smithi194 ceph-mon[32836]: Deploying daemon osd.2 on smithi194 2024-04-24T08:02:23.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:22 smithi117 ceph-mon[32438]: Deploying daemon osd.2 on smithi194 2024-04-24T08:02:24.018 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:23 smithi194 ceph-mon[32836]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:24.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:23 smithi028 ceph-mon[27786]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:24.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:23 smithi117 ceph-mon[32438]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:26.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:25 smithi028 ceph-mon[27786]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:26.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:25 smithi194 ceph-mon[32836]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:26.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:25 smithi117 ceph-mon[32438]: pgmap v82: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:27.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:27 smithi194 ceph-mon[32836]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:27.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:27 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:27.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:27 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:02:27.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:27 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:28.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:27 smithi028 ceph-mon[27786]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:28.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:27 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:28.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:27 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:02:28.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:27 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:28.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:27 smithi117 ceph-mon[32438]: pgmap v83: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:28.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:27 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:28.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:27 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:02:28.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:27 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:29.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:29 smithi194 ceph-mon[32836]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:29.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:29.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:29.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:29.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:02:29.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:30.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:29 smithi028 ceph-mon[27786]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:30.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:30.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:30.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:30.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:02:30.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:30.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:29 smithi117 ceph-mon[32438]: pgmap v84: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:02:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:30.413 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 2 on host 'smithi194' 2024-04-24T08:02:30.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:30.412+0000 7f1f7a7fc700 1 -- 172.21.15.194:0/1064731892 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1f84062f30 con 0x7f1f700a3c30 2024-04-24T08:02:30.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:30.414+0000 7f1f8b9c3700 1 -- 172.21.15.194:0/1064731892 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1f700a3c30 msgr2=0x7f1f700a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:02:30.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:30.414+0000 7f1f8b9c3700 1 --2- 172.21.15.194:0/1064731892 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1f700a3c30 0x7f1f700a60f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f1f740039c0 tx=0x7f1f74005fb0 comp rx=0 tx=0).stop 2024-04-24T08:02:30.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:30.414+0000 7f1f8b9c3700 1 -- 172.21.15.194:0/1064731892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f84102b50 msgr2=0x7f1f84198f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:02:30.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:30.414+0000 7f1f8b9c3700 1 --2- 172.21.15.194:0/1064731892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f84102b50 0x7f1f84198f50 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f1f8000b530 tx=0x7f1f8000b8f0 comp rx=0 tx=0).stop 2024-04-24T08:02:30.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:30.415+0000 7f1f8b9c3700 1 -- 172.21.15.194:0/1064731892 shutdown_connections 2024-04-24T08:02:30.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:30.415+0000 7f1f8b9c3700 1 --2- 172.21.15.194:0/1064731892 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1f700a3c30 0x7f1f700a60f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:30.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:30.415+0000 7f1f8b9c3700 1 --2- 172.21.15.194:0/1064731892 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f841046f0 0x7f1f841a04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:30.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:30.415+0000 7f1f8b9c3700 1 --2- 172.21.15.194:0/1064731892 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f84102b50 0x7f1f84198f50 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:30.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:30.415+0000 7f1f8b9c3700 1 --2- 172.21.15.194:0/1064731892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f84103d50 0x7f1f84199490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:30.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:30.415+0000 7f1f8b9c3700 1 -- 172.21.15.194:0/1064731892 >> 172.21.15.194:0/1064731892 conn(0x7f1f840fe0f0 msgr2=0x7f1f84107170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:02:30.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:30.415+0000 7f1f8b9c3700 1 -- 172.21.15.194:0/1064731892 shutdown_connections 2024-04-24T08:02:30.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:02:30.415+0000 7f1f8b9c3700 1 -- 172.21.15.194:0/1064731892 wait complete. 2024-04-24T08:02:31.042 DEBUG:teuthology.orchestra.run.smithi194:osd.2> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.2.service 2024-04-24T08:02:31.044 INFO:tasks.cephadm:Deploying osd.3 on smithi028 with /dev/vg_nvme/lv_3... 2024-04-24T08:02:31.045 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-24T08:02:31.500 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:02:31.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:31 smithi028 ceph-mon[27786]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:31.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:31 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:02:31.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:31 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:31.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:31 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:31.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:31 smithi028 ceph-mon[27786]: from='osd.2 [v2:172.21.15.194:6800/65787773,v1:172.21.15.194:6801/65787773]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T08:02:31.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:31 smithi028 ceph-mon[27786]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T08:02:31.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:31 smithi117 ceph-mon[32438]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:31.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:31 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:02:31.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:31 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:31.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:31 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:31.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:31 smithi117 ceph-mon[32438]: from='osd.2 [v2:172.21.15.194:6800/65787773,v1:172.21.15.194:6801/65787773]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T08:02:31.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:31 smithi117 ceph-mon[32438]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T08:02:31.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:31 smithi194 ceph-mon[32836]: pgmap v85: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:31.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:31 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:02:31.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:31 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:31.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:31 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:31.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:31 smithi194 ceph-mon[32836]: from='osd.2 [v2:172.21.15.194:6800/65787773,v1:172.21.15.194:6801/65787773]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T08:02:31.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:31 smithi194 ceph-mon[32836]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-04-24T08:02:32.489 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:32 smithi194 ceph-mon[32836]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-24T08:02:32.489 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:32 smithi194 ceph-mon[32836]: osdmap e16: 3 total, 2 up, 3 in 2024-04-24T08:02:32.489 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:32 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:02:32.489 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:32 smithi194 ceph-mon[32836]: from='osd.2 [v2:172.21.15.194:6800/65787773,v1:172.21.15.194:6801/65787773]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:02:32.489 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:32 smithi194 ceph-mon[32836]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:02:32.489 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:02:32 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-2[38007]: 2024-04-24T08:02:32.395+0000 7f743ae06700 -1 osd.2 0 waiting for initial osdmap 2024-04-24T08:02:32.490 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:02:32 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-2[38007]: 2024-04-24T08:02:32.400+0000 7f7436bf9700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:02:32.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:32 smithi028 ceph-mon[27786]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-24T08:02:32.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:32 smithi028 ceph-mon[27786]: osdmap e16: 3 total, 2 up, 3 in 2024-04-24T08:02:32.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:32 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:02:32.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:32 smithi028 ceph-mon[27786]: from='osd.2 [v2:172.21.15.194:6800/65787773,v1:172.21.15.194:6801/65787773]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:02:32.662 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:32 smithi028 ceph-mon[27786]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:02:32.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:32 smithi117 ceph-mon[32438]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-04-24T08:02:32.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:32 smithi117 ceph-mon[32438]: osdmap e16: 3 total, 2 up, 3 in 2024-04-24T08:02:32.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:32 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:02:32.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:32 smithi117 ceph-mon[32438]: from='osd.2 [v2:172.21.15.194:6800/65787773,v1:172.21.15.194:6801/65787773]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:02:32.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:32 smithi117 ceph-mon[32438]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:02:33.471 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:33 smithi028 ceph-mon[27786]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:33.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:33 smithi028 ceph-mon[27786]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-24T08:02:33.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:33 smithi028 ceph-mon[27786]: osdmap e17: 3 total, 2 up, 3 in 2024-04-24T08:02:33.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:33 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:02:33.472 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:33 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:02:33.676 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:33 smithi194 ceph-mon[32836]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:33.676 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:33 smithi194 ceph-mon[32836]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-24T08:02:33.677 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:33 smithi194 ceph-mon[32836]: osdmap e17: 3 total, 2 up, 3 in 2024-04-24T08:02:33.677 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:33 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:02:33.677 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:33 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:02:33.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:33 smithi117 ceph-mon[32438]: pgmap v87: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:33.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:33 smithi117 ceph-mon[32438]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-24T08:02:33.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:33 smithi117 ceph-mon[32438]: osdmap e17: 3 total, 2 up, 3 in 2024-04-24T08:02:33.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:33 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:02:33.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:33 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:02:34.631 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-24T08:02:34.631 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-24T08:02:34.631 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-24T08:02:34.632 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-24T08:02:34.632 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0342774 s, 306 MB/s 2024-04-24T08:02:34.632 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-24T08:02:34.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:34 smithi194 ceph-mon[32836]: purged_snaps scrub starts 2024-04-24T08:02:34.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:34 smithi194 ceph-mon[32836]: purged_snaps scrub ok 2024-04-24T08:02:34.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:34 smithi194 ceph-mon[32836]: osd.2 [v2:172.21.15.194:6800/65787773,v1:172.21.15.194:6801/65787773] boot 2024-04-24T08:02:34.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:34 smithi194 ceph-mon[32836]: osdmap e18: 3 total, 3 up, 3 in 2024-04-24T08:02:34.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:34 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:02:34.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:34 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:34.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:34 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:34.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:34 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:02:34.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:34 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:02:34.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:34 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:34.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:34 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:34.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:34 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:02:34.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:34 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:34.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:34 smithi028 ceph-mon[27786]: purged_snaps scrub starts 2024-04-24T08:02:34.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:34 smithi028 ceph-mon[27786]: purged_snaps scrub ok 2024-04-24T08:02:34.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:34 smithi028 ceph-mon[27786]: osd.2 [v2:172.21.15.194:6800/65787773,v1:172.21.15.194:6801/65787773] boot 2024-04-24T08:02:34.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:34 smithi028 ceph-mon[27786]: osdmap e18: 3 total, 3 up, 3 in 2024-04-24T08:02:34.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:34 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:02:34.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:34 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:34.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:34 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:34.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:34 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:02:34.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:34 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:02:34.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:34 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:34.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:34 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:34.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:34 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:02:34.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:34 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:34.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:34 smithi117 ceph-mon[32438]: purged_snaps scrub starts 2024-04-24T08:02:34.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:34 smithi117 ceph-mon[32438]: purged_snaps scrub ok 2024-04-24T08:02:34.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:34 smithi117 ceph-mon[32438]: osd.2 [v2:172.21.15.194:6800/65787773,v1:172.21.15.194:6801/65787773] boot 2024-04-24T08:02:34.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:34 smithi117 ceph-mon[32438]: osdmap e18: 3 total, 3 up, 3 in 2024-04-24T08:02:34.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:34 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:02:34.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:34 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:34.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:34 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:34.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:34 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:02:34.880 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:34 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:02:34.880 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:34 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:34.880 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:34 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:34.880 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:34 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:02:34.880 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:34 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:35.523 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_3 2024-04-24T08:02:35.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:35 smithi194 ceph-mon[32836]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:35.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:35 smithi194 ceph-mon[32836]: Detected new or changed devices on smithi194 2024-04-24T08:02:35.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:35 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi194 to 17088M 2024-04-24T08:02:35.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-24T08:02:35.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:35 smithi194 ceph-mon[32836]: osdmap e19: 3 total, 3 up, 3 in 2024-04-24T08:02:35.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:02:35.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:35 smithi028 ceph-mon[27786]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:35.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:35 smithi028 ceph-mon[27786]: Detected new or changed devices on smithi194 2024-04-24T08:02:35.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:35 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi194 to 17088M 2024-04-24T08:02:35.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-24T08:02:35.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:35 smithi028 ceph-mon[27786]: osdmap e19: 3 total, 3 up, 3 in 2024-04-24T08:02:35.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:02:35.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:35 smithi117 ceph-mon[32438]: pgmap v90: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-04-24T08:02:35.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:35 smithi117 ceph-mon[32438]: Detected new or changed devices on smithi194 2024-04-24T08:02:35.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:35 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi194 to 17088M 2024-04-24T08:02:35.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-04-24T08:02:35.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:35 smithi117 ceph-mon[32438]: osdmap e19: 3 total, 3 up, 3 in 2024-04-24T08:02:35.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:02:35.951 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:02:36.429 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:02:36 smithi028 sudo[44545]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T08:02:36.429 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:02:36 smithi028 sudo[44545]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:02:36.429 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:02:36 smithi028 sudo[44545]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:02:36.429 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:02:36 smithi028 sudo[44545]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:02:36.429 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:02:36 smithi028 sudo[44552]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T08:02:36.430 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:02:36 smithi028 sudo[44552]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:02:36.430 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:02:36 smithi028 sudo[44552]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:02:36.430 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:02:36 smithi028 sudo[44552]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:02:36.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:36 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-24T08:02:36.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:36 smithi117 ceph-mon[32438]: osdmap e20: 3 total, 3 up, 3 in 2024-04-24T08:02:36.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:36 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:36.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:36 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:36.628 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:02:36 smithi117 sudo[41236]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T08:02:36.628 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:02:36 smithi117 sudo[41236]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:02:36.628 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:02:36 smithi117 sudo[41236]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:02:36.628 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:02:36 smithi117 sudo[41236]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:02:36.628 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:02:36 smithi117 sudo[41239]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T08:02:36.628 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:02:36 smithi117 sudo[41239]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:02:36.628 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:02:36 smithi117 sudo[41239]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:02:36.628 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:02:36 smithi117 sudo[41239]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:02:36.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:36 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-24T08:02:36.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:36 smithi028 ceph-mon[27786]: osdmap e20: 3 total, 3 up, 3 in 2024-04-24T08:02:36.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:36 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:36.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:36 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:36.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:36 smithi028 sudo[44567]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T08:02:36.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:36 smithi028 sudo[44567]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:02:36.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:36 smithi028 sudo[44567]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:02:36.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:36 smithi028 sudo[44567]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:02:36.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:36 smithi028 sudo[44570]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T08:02:36.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:36 smithi028 sudo[44570]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:02:36.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:36 smithi028 sudo[44570]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:02:36.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:36 smithi028 sudo[44570]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:02:36.941 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:02:36 smithi194 sudo[42065]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T08:02:36.941 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:02:36 smithi194 sudo[42065]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:02:36.941 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:02:36 smithi194 sudo[42065]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:02:36.941 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:02:36 smithi194 sudo[42065]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:02:36.941 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:02:36 smithi194 sudo[42068]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T08:02:36.941 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:02:36 smithi194 sudo[42068]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:02:36.941 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:02:36 smithi194 sudo[42068]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:02:36.941 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:02:36 smithi194 sudo[42068]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:02:36.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:36 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-04-24T08:02:36.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:36 smithi194 ceph-mon[32836]: osdmap e20: 3 total, 3 up, 3 in 2024-04-24T08:02:36.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:36 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:36.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:36 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:36.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:36 smithi194 sudo[42071]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T08:02:36.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:36 smithi194 sudo[42071]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:02:36.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:36 smithi194 sudo[42071]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:02:36.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:36 smithi194 sudo[42071]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:02:36.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:36 smithi194 sudo[42074]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T08:02:36.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:36 smithi194 sudo[42074]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:02:36.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:36 smithi194 sudo[42074]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:02:36.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:36 smithi194 sudo[42074]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:02:37.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:36 smithi117 sudo[41242]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-04-24T08:02:37.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:36 smithi117 sudo[41242]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:02:37.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:36 smithi117 sudo[41242]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:02:37.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:36 smithi117 sudo[41242]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:02:37.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:36 smithi117 sudo[41245]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-04-24T08:02:37.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:36 smithi117 sudo[41245]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-04-24T08:02:37.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:36 smithi117 sudo[41245]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-04-24T08:02:37.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:36 smithi117 sudo[41245]: pam_unix(sudo:session): session closed for user root 2024-04-24T08:02:37.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: pgmap v93: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:37.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: osdmap e21: 3 total, 3 up, 3 in 2024-04-24T08:02:37.675 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:37.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:02:37.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:02:37.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:02:37.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:02:37.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:02:37.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:02:37.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:02:37.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:02:37.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:02:37.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:02:37.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:02:37.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:02:37.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:02:37.676 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:02:37.677 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:37 smithi028 ceph-mon[27786]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:02:37.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: pgmap v93: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:37.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: osdmap e21: 3 total, 3 up, 3 in 2024-04-24T08:02:37.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:37.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:02:37.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:02:37.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:02:37.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:02:37.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:02:37.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:02:37.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:02:37.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:02:37.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:02:37.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:02:37.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:02:37.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:02:37.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:02:37.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:02:37.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:37 smithi117 ceph-mon[32438]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:02:37.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: pgmap v93: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:37.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: osdmap e21: 3 total, 3 up, 3 in 2024-04-24T08:02:37.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:02:37.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:02:37.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:02:37.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:02:37.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:02:37.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:02:37.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:02:37.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:02:37.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:02:37.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:02:37.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:02:37.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-04-24T08:02:37.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:02:37.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:02:37.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:02:37.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:37 smithi194 ceph-mon[32836]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-04-24T08:02:38.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.160+0000 7fb998800700 1 -- 172.21.15.28:0/302628548 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb990108900 msgr2=0x7fb99010acf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:02:38.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.160+0000 7fb998800700 1 --2- 172.21.15.28:0/302628548 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb990108900 0x7fb99010acf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fb98c009f90 tx=0x7fb98c00b330 comp rx=0 tx=0).stop 2024-04-24T08:02:38.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.160+0000 7fb998800700 1 -- 172.21.15.28:0/302628548 shutdown_connections 2024-04-24T08:02:38.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.160+0000 7fb998800700 1 --2- 172.21.15.28:0/302628548 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb990108900 0x7fb99010acf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:38.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.160+0000 7fb998800700 1 --2- 172.21.15.28:0/302628548 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9901036a0 0x7fb990105a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:38.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.160+0000 7fb998800700 1 --2- 172.21.15.28:0/302628548 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb990105fd0 0x7fb9901083c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:38.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.160+0000 7fb998800700 1 -- 172.21.15.28:0/302628548 >> 172.21.15.28:0/302628548 conn(0x7fb9900fd010 msgr2=0x7fb9900ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:02:38.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.160+0000 7fb998800700 1 -- 172.21.15.28:0/302628548 shutdown_connections 2024-04-24T08:02:38.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.160+0000 7fb998800700 1 -- 172.21.15.28:0/302628548 wait complete. 2024-04-24T08:02:38.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.163+0000 7fb998800700 1 Processor -- start 2024-04-24T08:02:38.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.163+0000 7fb998800700 1 -- start start 2024-04-24T08:02:38.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.163+0000 7fb998800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9901036a0 0x7fb990103000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:02:38.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.163+0000 7fb998800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb990105fd0 0x7fb990101650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:02:38.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.163+0000 7fb998800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb990108900 0x7fb990101b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:02:38.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.164+0000 7fb998800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb99010e740 con 0x7fb990108900 2024-04-24T08:02:38.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.164+0000 7fb998800700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb99010e5c0 con 0x7fb990105fd0 2024-04-24T08:02:38.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.164+0000 7fb998800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb99010e8c0 con 0x7fb9901036a0 2024-04-24T08:02:38.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.164+0000 7fb996800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb990108900 0x7fb990101b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:02:38.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.164+0000 7fb995400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb990105fd0 0x7fb990101650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:02:38.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.164+0000 7fb996800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb990108900 0x7fb990101b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:43332/0 (socket says 172.21.15.28:43332) 2024-04-24T08:02:38.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.164+0000 7fb996800700 1 -- 172.21.15.28:0/1538687455 learned_addr learned my addr 172.21.15.28:0/1538687455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:02:38.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.164+0000 7fb995e00700 1 --2- 172.21.15.28:0/1538687455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9901036a0 0x7fb990103000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:02:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.164+0000 7fb996800700 1 -- 172.21.15.28:0/1538687455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9901036a0 msgr2=0x7fb990103000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:02:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.164+0000 7fb996800700 1 --2- 172.21.15.28:0/1538687455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9901036a0 0x7fb990103000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.164+0000 7fb996800700 1 -- 172.21.15.28:0/1538687455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb990105fd0 msgr2=0x7fb990101650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:02:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.164+0000 7fb996800700 1 --2- 172.21.15.28:0/1538687455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb990105fd0 0x7fb990101650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:02:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.164+0000 7fb996800700 1 -- 172.21.15.28:0/1538687455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb990102440 con 0x7fb990108900 2024-04-24T08:02:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.165+0000 7fb996800700 1 --2- 172.21.15.28:0/1538687455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb990108900 0x7fb990101b90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb98c007c50 tx=0x7fb98c007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:02:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.165+0000 7fb98aa00700 1 -- 172.21.15.28:0/1538687455 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c021020 con 0x7fb990108900 2024-04-24T08:02:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.165+0000 7fb98aa00700 1 -- 172.21.15.28:0/1538687455 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb98c01d950 con 0x7fb990108900 2024-04-24T08:02:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.165+0000 7fb98aa00700 1 -- 172.21.15.28:0/1538687455 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb98c0053d0 con 0x7fb990108900 2024-04-24T08:02:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.165+0000 7fb998800700 1 -- 172.21.15.28:0/1538687455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb9901a2df0 con 0x7fb990108900 2024-04-24T08:02:38.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.165+0000 7fb998800700 1 -- 172.21.15.28:0/1538687455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb9901a31f0 con 0x7fb990108900 2024-04-24T08:02:38.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.166+0000 7fb998800700 1 -- 172.21.15.28:0/1538687455 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9900697e0 con 0x7fb990108900 2024-04-24T08:02:38.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.169+0000 7fb98aa00700 1 -- 172.21.15.28:0/1538687455 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 15) ==== 138590+0+0 (secure 0 0 0) 0x7fb98c024020 con 0x7fb990108900 2024-04-24T08:02:38.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.170+0000 7fb98aa00700 1 --2- 172.21.15.28:0/1538687455 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb9780a3bb0 0x7fb9780a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:02:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.170+0000 7fb98aa00700 1 -- 172.21.15.28:0/1538687455 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(21..21 src has 1..21) ==== 3237+0+0 (secure 0 0 0) 0x7fb98c00f020 con 0x7fb990108900 2024-04-24T08:02:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.171+0000 7fb995e00700 1 --2- 172.21.15.28:0/1538687455 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb9780a3bb0 0x7fb9780a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:02:38.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.171+0000 7fb995e00700 1 --2- 172.21.15.28:0/1538687455 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb9780a3bb0 0x7fb9780a6070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb984005fd0 tx=0x7fb984009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:02:38.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.174+0000 7fb98aa00700 1 -- 172.21.15.28:0/1538687455 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb98c09e610 con 0x7fb990108900 2024-04-24T08:02:38.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.413+0000 7fb998800700 1 -- 172.21.15.28:0/1538687455 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fb990062f30 con 0x7fb9780a3bb0 2024-04-24T08:02:38.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:02:38.462+0000 7fb98aa00700 1 -- 172.21.15.28:0/1538687455 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb98c09a580 con 0x7fb990108900 2024-04-24T08:02:38.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:38 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:02:38.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:38 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:02:38.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:38 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:38.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:38 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:02:38.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:38 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:02:38.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:38 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:38.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:38 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:02:38.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:38 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:02:38.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:38 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:39.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:39 smithi028 ceph-mon[27786]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:39.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:39 smithi028 ceph-mon[27786]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:02:39.684 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:39 smithi028 ceph-mon[27786]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-24T08:02:39.779 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:39 smithi117 ceph-mon[32438]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:39.779 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:39 smithi117 ceph-mon[32438]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:02:39.779 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:39 smithi117 ceph-mon[32438]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-24T08:02:39.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:39 smithi194 ceph-mon[32836]: pgmap v95: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:39.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:39 smithi194 ceph-mon[32836]: from='client.14250 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:02:39.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:39 smithi194 ceph-mon[32836]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-04-24T08:02:41.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:41 smithi028 ceph-mon[27786]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:41.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:41 smithi117 ceph-mon[32438]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:41.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:41 smithi194 ceph-mon[32836]: pgmap v96: 1 pgs: 1 creating+peering; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:43.770 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:43 smithi194 ceph-mon[32836]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:43.770 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:43 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2236266513' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32e5f179-3598-4a9b-93c4-48728d942883"}]: dispatch 2024-04-24T08:02:43.770 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:43 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2236266513' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32e5f179-3598-4a9b-93c4-48728d942883"}]': finished 2024-04-24T08:02:43.771 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:43 smithi194 ceph-mon[32836]: osdmap e22: 4 total, 3 up, 4 in 2024-04-24T08:02:43.771 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:43 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:02:43.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:43 smithi028 ceph-mon[27786]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:43.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:43 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2236266513' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32e5f179-3598-4a9b-93c4-48728d942883"}]: dispatch 2024-04-24T08:02:43.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:43 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2236266513' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32e5f179-3598-4a9b-93c4-48728d942883"}]': finished 2024-04-24T08:02:43.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:43 smithi028 ceph-mon[27786]: osdmap e22: 4 total, 3 up, 4 in 2024-04-24T08:02:43.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:43 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:02:43.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:43 smithi117 ceph-mon[32438]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:43.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:43 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2236266513' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "32e5f179-3598-4a9b-93c4-48728d942883"}]: dispatch 2024-04-24T08:02:43.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:43 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2236266513' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "32e5f179-3598-4a9b-93c4-48728d942883"}]': finished 2024-04-24T08:02:43.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:43 smithi117 ceph-mon[32438]: osdmap e22: 4 total, 3 up, 4 in 2024-04-24T08:02:43.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:43 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:02:44.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:44 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/4104646976' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:02:44.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:44 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/4104646976' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:02:44.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:44 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/4104646976' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:02:45.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:45 smithi028 ceph-mon[27786]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:45.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:45 smithi117 ceph-mon[32438]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:45.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:45 smithi194 ceph-mon[32836]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:47.671 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:47 smithi028 ceph-mon[27786]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:47.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:47 smithi117 ceph-mon[32438]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:47.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:47 smithi194 ceph-mon[32836]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:49.777 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:49 smithi028 ceph-mon[27786]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:49.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:49 smithi117 ceph-mon[32438]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:49.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:49 smithi194 ceph-mon[32836]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:51.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:51 smithi028 ceph-mon[27786]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:51.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:51 smithi117 ceph-mon[32438]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:51.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:51 smithi194 ceph-mon[32836]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:52.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:52 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T08:02:52.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:52 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:52.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:52 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T08:02:52.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:52 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:52.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:52 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-04-24T08:02:52.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:52 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:02:53.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:53 smithi028 ceph-mon[27786]: Deploying daemon osd.3 on smithi028 2024-04-24T08:02:53.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:53 smithi028 ceph-mon[27786]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:53.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:53 smithi117 ceph-mon[32438]: Deploying daemon osd.3 on smithi028 2024-04-24T08:02:53.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:53 smithi117 ceph-mon[32438]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:53.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:53 smithi194 ceph-mon[32836]: Deploying daemon osd.3 on smithi028 2024-04-24T08:02:53.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:53 smithi194 ceph-mon[32836]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:55.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:55 smithi028 ceph-mon[27786]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:55.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:55 smithi117 ceph-mon[32438]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:55.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:55 smithi194 ceph-mon[32836]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:57.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:57 smithi028 ceph-mon[27786]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:57.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:57 smithi117 ceph-mon[32438]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:57.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:57 smithi194 ceph-mon[32836]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:59.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:02:59 smithi028 ceph-mon[27786]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:59.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:02:59 smithi117 ceph-mon[32438]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:02:59.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:02:59 smithi194 ceph-mon[32836]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:03:01.757 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:01 smithi194 ceph-mon[32836]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:03:01.757 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:01.757 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:01.757 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:01.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:01 smithi028 ceph-mon[27786]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:03:01.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:01.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:01.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:01.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:01 smithi117 ceph-mon[32438]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:03:01.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:01.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:01.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:03.778 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:03 smithi028 ceph-mon[27786]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:03:03.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:03 smithi117 ceph-mon[32438]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:03:03.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:03 smithi194 ceph-mon[32836]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:03:05.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:05 smithi028 ceph-mon[27786]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:03:05.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:05.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:05.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:05 smithi028 ceph-mon[27786]: from='osd.3 [v2:172.21.15.28:6810/3962695585,v1:172.21.15.28:6811/3962695585]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T08:03:05.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:05 smithi028 ceph-mon[27786]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T08:03:05.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:05 smithi117 ceph-mon[32438]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:03:05.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:05 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:05.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:05 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:05.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:05 smithi117 ceph-mon[32438]: from='osd.3 [v2:172.21.15.28:6810/3962695585,v1:172.21.15.28:6811/3962695585]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T08:03:05.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:05 smithi117 ceph-mon[32438]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T08:03:05.418 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 3 on host 'smithi028' 2024-04-24T08:03:05.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:03:05.417+0000 7fb98aa00700 1 -- 172.21.15.28:0/1538687455 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fb990062f30 con 0x7fb9780a3bb0 2024-04-24T08:03:05.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:03:05.419+0000 7fb998800700 1 -- 172.21.15.28:0/1538687455 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb9780a3bb0 msgr2=0x7fb9780a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:03:05.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:03:05.419+0000 7fb998800700 1 --2- 172.21.15.28:0/1538687455 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb9780a3bb0 0x7fb9780a6070 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fb984005fd0 tx=0x7fb984009310 comp rx=0 tx=0).stop 2024-04-24T08:03:05.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:03:05.419+0000 7fb998800700 1 -- 172.21.15.28:0/1538687455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb990108900 msgr2=0x7fb990101b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:03:05.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:03:05.419+0000 7fb998800700 1 --2- 172.21.15.28:0/1538687455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb990108900 0x7fb990101b90 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb98c007c50 tx=0x7fb98c007c80 comp rx=0 tx=0).stop 2024-04-24T08:03:05.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:03:05.419+0000 7fb998800700 1 -- 172.21.15.28:0/1538687455 shutdown_connections 2024-04-24T08:03:05.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:03:05.419+0000 7fb998800700 1 --2- 172.21.15.28:0/1538687455 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb9780a3bb0 0x7fb9780a6070 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:05.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:03:05.419+0000 7fb998800700 1 --2- 172.21.15.28:0/1538687455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb990108900 0x7fb990101b90 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:05.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:03:05.419+0000 7fb998800700 1 --2- 172.21.15.28:0/1538687455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb9901036a0 0x7fb990103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:05.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:03:05.419+0000 7fb998800700 1 --2- 172.21.15.28:0/1538687455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb990105fd0 0x7fb990101650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:05.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:03:05.419+0000 7fb998800700 1 -- 172.21.15.28:0/1538687455 >> 172.21.15.28:0/1538687455 conn(0x7fb9900fd010 msgr2=0x7fb990103f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:03:05.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:03:05.420+0000 7fb998800700 1 -- 172.21.15.28:0/1538687455 shutdown_connections 2024-04-24T08:03:05.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:03:05.420+0000 7fb998800700 1 -- 172.21.15.28:0/1538687455 wait complete. 2024-04-24T08:03:05.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:05 smithi194 ceph-mon[32836]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:03:05.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:05 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:05.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:05 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:05.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:05 smithi194 ceph-mon[32836]: from='osd.3 [v2:172.21.15.28:6810/3962695585,v1:172.21.15.28:6811/3962695585]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T08:03:05.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:05 smithi194 ceph-mon[32836]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-04-24T08:03:06.166 DEBUG:teuthology.orchestra.run.smithi028:osd.3> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.3.service 2024-04-24T08:03:06.172 INFO:tasks.cephadm:Deploying osd.4 on smithi117 with /dev/vg_nvme/lv_3... 2024-04-24T08:03:06.172 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-24T08:03:06.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:06.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:06.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:06.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:06.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:06.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:06 smithi117 ceph-mon[32438]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-24T08:03:06.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:06 smithi117 ceph-mon[32438]: from='osd.3 [v2:172.21.15.28:6810/3962695585,v1:172.21.15.28:6811/3962695585]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:03:06.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:06 smithi117 ceph-mon[32438]: osdmap e23: 4 total, 3 up, 4 in 2024-04-24T08:03:06.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:03:06.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:06 smithi117 ceph-mon[32438]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:03:06.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:06.414 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:03:06.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:06.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:06.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:06.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:06.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:06.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:06 smithi194 ceph-mon[32836]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-24T08:03:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:06 smithi194 ceph-mon[32836]: from='osd.3 [v2:172.21.15.28:6810/3962695585,v1:172.21.15.28:6811/3962695585]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:03:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:06 smithi194 ceph-mon[32836]: osdmap e23: 4 total, 3 up, 4 in 2024-04-24T08:03:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:03:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:06 smithi194 ceph-mon[32836]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:03:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:06.517 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:06.517 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:06.517 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:06.517 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:06.517 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:06.517 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:06 smithi028 ceph-mon[27786]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-04-24T08:03:06.517 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:06 smithi028 ceph-mon[27786]: from='osd.3 [v2:172.21.15.28:6810/3962695585,v1:172.21.15.28:6811/3962695585]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:03:06.517 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:06 smithi028 ceph-mon[27786]: osdmap e23: 4 total, 3 up, 4 in 2024-04-24T08:03:06.517 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:03:06.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:06 smithi028 ceph-mon[27786]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:03:06.518 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:06.790 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 24 08:03:06 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-3[46879]: 2024-04-24T08:03:06.542+0000 7f4c5c600700 -1 osd.3 0 waiting for initial osdmap 2024-04-24T08:03:06.790 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 24 08:03:06 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-3[46879]: 2024-04-24T08:03:06.560+0000 7f4c57400700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:03:07.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:07 smithi028 ceph-mon[27786]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:03:07.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:07 smithi028 ceph-mon[27786]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-24T08:03:07.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:07 smithi028 ceph-mon[27786]: osdmap e24: 4 total, 3 up, 4 in 2024-04-24T08:03:07.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:03:07.725 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:03:07.736 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:07 smithi117 ceph-mon[32438]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:03:07.737 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:07 smithi117 ceph-mon[32438]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-24T08:03:07.737 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:07 smithi117 ceph-mon[32438]: osdmap e24: 4 total, 3 up, 4 in 2024-04-24T08:03:07.737 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:03:07.737 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:03:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:07 smithi194 ceph-mon[32836]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 84 MiB used, 268 GiB / 268 GiB avail 2024-04-24T08:03:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:07 smithi194 ceph-mon[32836]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-24T08:03:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:07 smithi194 ceph-mon[32836]: osdmap e24: 4 total, 3 up, 4 in 2024-04-24T08:03:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:03:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:03:08.319 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-24T08:03:08.319 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-24T08:03:08.319 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-04-24T08:03:08.319 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-04-24T08:03:08.320 INFO:teuthology.orchestra.run.smithi117.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011696 s, 897 MB/s 2024-04-24T08:03:08.320 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-04-24T08:03:08.657 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:08 smithi117 ceph-mon[32438]: purged_snaps scrub starts 2024-04-24T08:03:08.657 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:08 smithi117 ceph-mon[32438]: purged_snaps scrub ok 2024-04-24T08:03:08.657 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:08 smithi117 ceph-mon[32438]: osd.3 [v2:172.21.15.28:6810/3962695585,v1:172.21.15.28:6811/3962695585] boot 2024-04-24T08:03:08.658 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:08 smithi117 ceph-mon[32438]: osdmap e25: 4 total, 4 up, 4 in 2024-04-24T08:03:08.658 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:08 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:03:08.658 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:08 smithi117 ceph-mon[32438]: osdmap e26: 4 total, 4 up, 4 in 2024-04-24T08:03:08.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:08 smithi028 ceph-mon[27786]: purged_snaps scrub starts 2024-04-24T08:03:08.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:08 smithi028 ceph-mon[27786]: purged_snaps scrub ok 2024-04-24T08:03:08.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:08 smithi028 ceph-mon[27786]: osd.3 [v2:172.21.15.28:6810/3962695585,v1:172.21.15.28:6811/3962695585] boot 2024-04-24T08:03:08.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:08 smithi028 ceph-mon[27786]: osdmap e25: 4 total, 4 up, 4 in 2024-04-24T08:03:08.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:08 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:03:08.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:08 smithi028 ceph-mon[27786]: osdmap e26: 4 total, 4 up, 4 in 2024-04-24T08:03:08.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:08 smithi194 ceph-mon[32836]: purged_snaps scrub starts 2024-04-24T08:03:08.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:08 smithi194 ceph-mon[32836]: purged_snaps scrub ok 2024-04-24T08:03:08.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:08 smithi194 ceph-mon[32836]: osd.3 [v2:172.21.15.28:6810/3962695585,v1:172.21.15.28:6811/3962695585] boot 2024-04-24T08:03:08.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:08 smithi194 ceph-mon[32836]: osdmap e25: 4 total, 4 up, 4 in 2024-04-24T08:03:08.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:08 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:03:08.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:08 smithi194 ceph-mon[32836]: osdmap e26: 4 total, 4 up, 4 in 2024-04-24T08:03:08.960 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi117:vg_nvme/lv_3 2024-04-24T08:03:09.205 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:03:09.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:09 smithi117 ceph-mon[32438]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:09.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:09 smithi117 ceph-mon[32438]: osdmap e27: 4 total, 4 up, 4 in 2024-04-24T08:03:09.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:09 smithi194 ceph-mon[32836]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:09.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:09 smithi194 ceph-mon[32836]: osdmap e27: 4 total, 4 up, 4 in 2024-04-24T08:03:10.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:09 smithi028 ceph-mon[27786]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:10.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:09 smithi028 ceph-mon[27786]: osdmap e27: 4 total, 4 up, 4 in 2024-04-24T08:03:10.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.770+0000 7f8d1cb5e700 1 -- 172.21.15.117:0/2677501662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d181088e0 msgr2=0x7f8d1810acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:03:10.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.770+0000 7f8d1cb5e700 1 --2- 172.21.15.117:0/2677501662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d181088e0 0x7f8d1810acd0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f8d1000b0a0 tx=0x7f8d1000b3b0 comp rx=0 tx=0).stop 2024-04-24T08:03:10.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.770+0000 7f8d1cb5e700 1 -- 172.21.15.117:0/2677501662 shutdown_connections 2024-04-24T08:03:10.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.770+0000 7f8d1cb5e700 1 --2- 172.21.15.117:0/2677501662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d181088e0 0x7f8d1810acd0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:10.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.770+0000 7f8d1cb5e700 1 --2- 172.21.15.117:0/2677501662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d18103680 0x7f8d18105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:10.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.770+0000 7f8d1cb5e700 1 --2- 172.21.15.117:0/2677501662 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d18105fb0 0x7f8d181083a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:10.773 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.770+0000 7f8d1cb5e700 1 -- 172.21.15.117:0/2677501662 >> 172.21.15.117:0/2677501662 conn(0x7f8d180fcff0 msgr2=0x7f8d180ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:03:10.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.770+0000 7f8d1cb5e700 1 -- 172.21.15.117:0/2677501662 shutdown_connections 2024-04-24T08:03:10.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.770+0000 7f8d1cb5e700 1 -- 172.21.15.117:0/2677501662 wait complete. 2024-04-24T08:03:10.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d1cb5e700 1 Processor -- start 2024-04-24T08:03:10.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d1cb5e700 1 -- start start 2024-04-24T08:03:10.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d1cb5e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d18103680 0x7f8d18198e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:03:10.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d1cb5e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d18105fb0 0x7f8d18199350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:03:10.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d1cb5e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d181088e0 0x7f8d181a03d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:03:10.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d1cb5e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1810e7a0 con 0x7f8d18105fb0 2024-04-24T08:03:10.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d1cb5e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1810e620 con 0x7f8d181088e0 2024-04-24T08:03:10.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d1cb5e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1810e920 con 0x7f8d18103680 2024-04-24T08:03:10.774 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d16d9d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d181088e0 0x7f8d181a03d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:03:10.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d16d9d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d181088e0 0x7f8d181a03d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:36150/0 (socket says 172.21.15.117:36150) 2024-04-24T08:03:10.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d16d9d700 1 -- 172.21.15.117:0/2084832242 learned_addr learned my addr 172.21.15.117:0/2084832242 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-04-24T08:03:10.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d1659c700 1 --2- 172.21.15.117:0/2084832242 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d18103680 0x7f8d18198e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:03:10.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d16d9d700 1 -- 172.21.15.117:0/2084832242 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d18103680 msgr2=0x7f8d18198e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:03:10.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d16d9d700 1 --2- 172.21.15.117:0/2084832242 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d18103680 0x7f8d18198e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:10.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d16d9d700 1 -- 172.21.15.117:0/2084832242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d18105fb0 msgr2=0x7f8d18199350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:03:10.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d16d9d700 1 --2- 172.21.15.117:0/2084832242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d18105fb0 0x7f8d18199350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:10.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d16d9d700 1 -- 172.21.15.117:0/2084832242 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d181a0a70 con 0x7f8d181088e0 2024-04-24T08:03:10.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d16d9d700 1 --2- 172.21.15.117:0/2084832242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d181088e0 0x7f8d181a03d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8d10007fa0 tx=0x7f8d10004dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:03:10.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d077fe700 1 -- 172.21.15.117:0/2084832242 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d10021070 con 0x7f8d181088e0 2024-04-24T08:03:10.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d1cb5e700 1 -- 172.21.15.117:0/2084832242 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d181a0d00 con 0x7f8d181088e0 2024-04-24T08:03:10.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d077fe700 1 -- 172.21.15.117:0/2084832242 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d10007c00 con 0x7f8d181088e0 2024-04-24T08:03:10.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.771+0000 7f8d077fe700 1 -- 172.21.15.117:0/2084832242 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1001d450 con 0x7f8d181088e0 2024-04-24T08:03:10.775 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.772+0000 7f8d1cb5e700 1 -- 172.21.15.117:0/2084832242 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d181a11b0 con 0x7f8d181088e0 2024-04-24T08:03:10.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.772+0000 7f8d1cb5e700 1 -- 172.21.15.117:0/2084832242 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d180697e0 con 0x7f8d181088e0 2024-04-24T08:03:10.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.774+0000 7f8d077fe700 1 -- 172.21.15.117:0/2084832242 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8d10004000 con 0x7f8d181088e0 2024-04-24T08:03:10.776 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.774+0000 7f8d077fe700 1 --2- 172.21.15.117:0/2084832242 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8d000a3c40 0x7f8d000a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:03:10.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.774+0000 7f8d077fe700 1 -- 172.21.15.117:0/2084832242 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7f8d100d0270 con 0x7f8d181088e0 2024-04-24T08:03:10.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.774+0000 7f8d1659c700 1 --2- 172.21.15.117:0/2084832242 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8d000a3c40 0x7f8d000a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:03:10.777 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.775+0000 7f8d1659c700 1 --2- 172.21.15.117:0/2084832242 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8d000a3c40 0x7f8d000a6100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009a10 tx=0x7f8d08009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:03:10.779 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.778+0000 7f8d077fe700 1 -- 172.21.15.117:0/2084832242 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8d1009df00 con 0x7f8d181088e0 2024-04-24T08:03:10.984 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:10.983+0000 7f8d1cb5e700 1 -- 172.21.15.117:0/2084832242 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f8d18062f30 con 0x7f8d000a3c40 2024-04-24T08:03:11.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:11 smithi028 ceph-mon[27786]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:11.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:11 smithi028 ceph-mon[27786]: osdmap e28: 4 total, 4 up, 4 in 2024-04-24T08:03:11.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:11 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:03:11.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:11 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:03:11.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:11 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:11.820 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:11 smithi117 ceph-mon[32438]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:11.820 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:11 smithi117 ceph-mon[32438]: osdmap e28: 4 total, 4 up, 4 in 2024-04-24T08:03:11.820 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:11 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:03:11.820 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:11 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:03:11.820 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:11 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:11.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:11 smithi194 ceph-mon[32836]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:11.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:11 smithi194 ceph-mon[32836]: osdmap e28: 4 total, 4 up, 4 in 2024-04-24T08:03:11.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:11 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:03:11.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:11 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:03:11.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:11 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:12.833 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:12 smithi117 ceph-mon[32438]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:03:12.833 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:12 smithi117 ceph-mon[32438]: Detected new or changed devices on smithi028 2024-04-24T08:03:12.833 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:12 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:12.833 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:12 smithi117 ceph-mon[32438]: osdmap e29: 4 total, 4 up, 4 in 2024-04-24T08:03:12.833 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:12 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:12.833 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:12 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:12.833 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:12 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:12.833 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:12 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi028 to 8547M 2024-04-24T08:03:12.833 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:12 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:12.833 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:12 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:12.833 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:12 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:12.833 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:12 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:12.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:12 smithi194 ceph-mon[32836]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:03:12.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:12 smithi194 ceph-mon[32836]: Detected new or changed devices on smithi028 2024-04-24T08:03:12.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:12 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:12.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:12 smithi194 ceph-mon[32836]: osdmap e29: 4 total, 4 up, 4 in 2024-04-24T08:03:12.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:12 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:12.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:12 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:12.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:12 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:12.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:12 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi028 to 8547M 2024-04-24T08:03:12.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:12 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:12.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:12 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:12.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:12 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:12.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:12 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:13.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:12 smithi028 ceph-mon[27786]: from='client.24221 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:03:13.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:12 smithi028 ceph-mon[27786]: Detected new or changed devices on smithi028 2024-04-24T08:03:13.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:12 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:13.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:12 smithi028 ceph-mon[27786]: osdmap e29: 4 total, 4 up, 4 in 2024-04-24T08:03:13.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:12 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:13.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:12 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:13.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:12 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:13.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:12 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi028 to 8547M 2024-04-24T08:03:13.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:12 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:13.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:12 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:13.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:12 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:13.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:12 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:13.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:13 smithi117 ceph-mon[32438]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-24T08:03:13.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:13 smithi194 ceph-mon[32836]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-24T08:03:14.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:13 smithi028 ceph-mon[27786]: pgmap v120: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-24T08:03:14.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:14 smithi117 ceph-mon[32438]: from='client.? 172.21.15.117:0/1111504677' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c07ffaec-979d-4a1d-b2fd-0d92b3fd303a"}]: dispatch 2024-04-24T08:03:14.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:14 smithi117 ceph-mon[32438]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c07ffaec-979d-4a1d-b2fd-0d92b3fd303a"}]: dispatch 2024-04-24T08:03:14.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:14 smithi117 ceph-mon[32438]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c07ffaec-979d-4a1d-b2fd-0d92b3fd303a"}]': finished 2024-04-24T08:03:14.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:14 smithi117 ceph-mon[32438]: osdmap e30: 5 total, 4 up, 5 in 2024-04-24T08:03:14.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:14 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:03:14.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:14 smithi117 ceph-mon[32438]: from='client.? 172.21.15.117:0/3999144230' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:03:14.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:14 smithi194 ceph-mon[32836]: from='client.? 172.21.15.117:0/1111504677' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c07ffaec-979d-4a1d-b2fd-0d92b3fd303a"}]: dispatch 2024-04-24T08:03:14.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:14 smithi194 ceph-mon[32836]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c07ffaec-979d-4a1d-b2fd-0d92b3fd303a"}]: dispatch 2024-04-24T08:03:14.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:14 smithi194 ceph-mon[32836]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c07ffaec-979d-4a1d-b2fd-0d92b3fd303a"}]': finished 2024-04-24T08:03:14.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:14 smithi194 ceph-mon[32836]: osdmap e30: 5 total, 4 up, 5 in 2024-04-24T08:03:14.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:14 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:03:14.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:14 smithi194 ceph-mon[32836]: from='client.? 172.21.15.117:0/3999144230' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:03:15.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:14 smithi028 ceph-mon[27786]: from='client.? 172.21.15.117:0/1111504677' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c07ffaec-979d-4a1d-b2fd-0d92b3fd303a"}]: dispatch 2024-04-24T08:03:15.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:14 smithi028 ceph-mon[27786]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c07ffaec-979d-4a1d-b2fd-0d92b3fd303a"}]: dispatch 2024-04-24T08:03:15.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:14 smithi028 ceph-mon[27786]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c07ffaec-979d-4a1d-b2fd-0d92b3fd303a"}]': finished 2024-04-24T08:03:15.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:14 smithi028 ceph-mon[27786]: osdmap e30: 5 total, 4 up, 5 in 2024-04-24T08:03:15.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:14 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:03:15.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:14 smithi028 ceph-mon[27786]: from='client.? 172.21.15.117:0/3999144230' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:03:15.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:15 smithi194 ceph-mon[32836]: pgmap v122: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-24T08:03:16.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:15 smithi028 ceph-mon[27786]: pgmap v122: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-24T08:03:16.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:15 smithi117 ceph-mon[32438]: pgmap v122: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-04-24T08:03:18.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:17 smithi028 ceph-mon[27786]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T08:03:18.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:17 smithi117 ceph-mon[32438]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T08:03:18.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:17 smithi194 ceph-mon[32836]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T08:03:19.849 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:19 smithi117 ceph-mon[32438]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:20.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:19 smithi028 ceph-mon[27786]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:20.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:19 smithi194 ceph-mon[32836]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:21.940 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:21 smithi117 ceph-mon[32438]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:21.940 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:21 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T08:03:21.940 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:21 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:22.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:21 smithi028 ceph-mon[27786]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:22.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:21 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T08:03:22.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:21 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:22.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:21 smithi194 ceph-mon[32836]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:22.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:21 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-04-24T08:03:22.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:21 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:22.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:22 smithi117 ceph-mon[32438]: Deploying daemon osd.4 on smithi117 2024-04-24T08:03:23.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:22 smithi028 ceph-mon[27786]: Deploying daemon osd.4 on smithi117 2024-04-24T08:03:23.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:22 smithi194 ceph-mon[32836]: Deploying daemon osd.4 on smithi117 2024-04-24T08:03:23.959 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:23 smithi117 ceph-mon[32438]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:24.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:23 smithi028 ceph-mon[27786]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:24.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:23 smithi194 ceph-mon[32836]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:26.001 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:25 smithi117 ceph-mon[32438]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:26.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:25 smithi028 ceph-mon[27786]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:26.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:25 smithi194 ceph-mon[32836]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:27.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:27 smithi117 ceph-mon[32438]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:27.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:27 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:27.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:27 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:27.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:27 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:27.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:27 smithi194 ceph-mon[32836]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:27.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:27 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:27.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:27 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:27.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:27 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:28.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:27 smithi028 ceph-mon[27786]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:28.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:27 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:28.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:27 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:28.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:27 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:29 smithi117 ceph-mon[32438]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:29.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:29 smithi194 ceph-mon[32836]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:30.021 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:30.021+0000 7f8d077fe700 1 -- 172.21.15.117:0/2084832242 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f8d18062f30 con 0x7f8d000a3c40 2024-04-24T08:03:30.022 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 4 on host 'smithi117' 2024-04-24T08:03:30.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:30.023+0000 7f8d1cb5e700 1 -- 172.21.15.117:0/2084832242 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8d000a3c40 msgr2=0x7f8d000a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:03:30.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:30.023+0000 7f8d1cb5e700 1 --2- 172.21.15.117:0/2084832242 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8d000a3c40 0x7f8d000a6100 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009a10 tx=0x7f8d08009190 comp rx=0 tx=0).stop 2024-04-24T08:03:30.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:30.023+0000 7f8d1cb5e700 1 -- 172.21.15.117:0/2084832242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d181088e0 msgr2=0x7f8d181a03d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:03:30.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:30.023+0000 7f8d1cb5e700 1 --2- 172.21.15.117:0/2084832242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d181088e0 0x7f8d181a03d0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f8d10007fa0 tx=0x7f8d10004dd0 comp rx=0 tx=0).stop 2024-04-24T08:03:30.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:30.023+0000 7f8d1cb5e700 1 -- 172.21.15.117:0/2084832242 shutdown_connections 2024-04-24T08:03:30.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:30.023+0000 7f8d1cb5e700 1 --2- 172.21.15.117:0/2084832242 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8d000a3c40 0x7f8d000a6100 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:30.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:30.023+0000 7f8d1cb5e700 1 --2- 172.21.15.117:0/2084832242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d181088e0 0x7f8d181a03d0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:30.024 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:30.023+0000 7f8d1cb5e700 1 --2- 172.21.15.117:0/2084832242 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8d18103680 0x7f8d18198e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:30.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:30.023+0000 7f8d1cb5e700 1 --2- 172.21.15.117:0/2084832242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8d18105fb0 0x7f8d18199350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:30.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:30.023+0000 7f8d1cb5e700 1 -- 172.21.15.117:0/2084832242 >> 172.21.15.117:0/2084832242 conn(0x7f8d180fcff0 msgr2=0x7f8d18104120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:03:30.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:30.023+0000 7f8d1cb5e700 1 -- 172.21.15.117:0/2084832242 shutdown_connections 2024-04-24T08:03:30.025 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:03:30.023+0000 7f8d1cb5e700 1 -- 172.21.15.117:0/2084832242 wait complete. 2024-04-24T08:03:30.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:29 smithi028 ceph-mon[27786]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:30.679 DEBUG:teuthology.orchestra.run.smithi117:osd.4> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.4.service 2024-04-24T08:03:30.681 INFO:tasks.cephadm:Deploying osd.5 on smithi194 with /dev/vg_nvme/lv_3... 2024-04-24T08:03:30.681 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-04-24T08:03:30.927 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:03:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:30 smithi028 ceph-mon[27786]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:30 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:30 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:30 smithi028 ceph-mon[27786]: from='osd.4 [v2:172.21.15.117:6808/1649446681,v1:172.21.15.117:6809/1649446681]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T08:03:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:30 smithi028 ceph-mon[27786]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T08:03:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:30 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:31.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:30 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:31.291 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:30 smithi194 ceph-mon[32836]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:31.291 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:30 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:31.291 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:30 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:31.291 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:30 smithi194 ceph-mon[32836]: from='osd.4 [v2:172.21.15.117:6808/1649446681,v1:172.21.15.117:6809/1649446681]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T08:03:31.291 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:30 smithi194 ceph-mon[32836]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T08:03:31.291 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:30 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:31.291 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:30 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:31.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:30 smithi117 ceph-mon[32438]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:31.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:30 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:31.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:30 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:31.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:30 smithi117 ceph-mon[32438]: from='osd.4 [v2:172.21.15.117:6808/1649446681,v1:172.21.15.117:6809/1649446681]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T08:03:31.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:30 smithi117 ceph-mon[32438]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-04-24T08:03:31.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:30 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:31.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:30 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:32.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:32 smithi117 ceph-mon[32438]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-24T08:03:32.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:32 smithi117 ceph-mon[32438]: from='osd.4 [v2:172.21.15.117:6808/1649446681,v1:172.21.15.117:6809/1649446681]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:03:32.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:32 smithi117 ceph-mon[32438]: osdmap e31: 5 total, 4 up, 5 in 2024-04-24T08:03:32.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:32 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:03:32.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:32 smithi117 ceph-mon[32438]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:03:32.128 INFO:journalctl@ceph.osd.4.smithi117.stdout:Apr 24 08:03:32 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-4[44079]: 2024-04-24T08:03:32.006+0000 7f5551629700 -1 osd.4 0 waiting for initial osdmap 2024-04-24T08:03:32.128 INFO:journalctl@ceph.osd.4.smithi117.stdout:Apr 24 08:03:32 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-4[44079]: 2024-04-24T08:03:32.011+0000 7f554d41c700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:03:32.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:32 smithi028 ceph-mon[27786]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-24T08:03:32.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:32 smithi028 ceph-mon[27786]: from='osd.4 [v2:172.21.15.117:6808/1649446681,v1:172.21.15.117:6809/1649446681]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:03:32.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:32 smithi028 ceph-mon[27786]: osdmap e31: 5 total, 4 up, 5 in 2024-04-24T08:03:32.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:32 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:03:32.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:32 smithi028 ceph-mon[27786]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:03:32.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:32 smithi194 ceph-mon[32836]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-04-24T08:03:32.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:32 smithi194 ceph-mon[32836]: from='osd.4 [v2:172.21.15.117:6808/1649446681,v1:172.21.15.117:6809/1649446681]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:03:32.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:32 smithi194 ceph-mon[32836]: osdmap e31: 5 total, 4 up, 5 in 2024-04-24T08:03:32.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:32 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:03:32.417 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:32 smithi194 ceph-mon[32836]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:03:33.094 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-04-24T08:03:33.094 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-04-24T08:03:33.094 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-04-24T08:03:33.095 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-04-24T08:03:33.095 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0134105 s, 782 MB/s 2024-04-24T08:03:33.095 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-04-24T08:03:33.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:33 smithi117 ceph-mon[32438]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:33.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:33 smithi117 ceph-mon[32438]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-04-24T08:03:33.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:33 smithi117 ceph-mon[32438]: osdmap e32: 5 total, 4 up, 5 in 2024-04-24T08:03:33.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:33 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:03:33.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:33 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:03:33.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:33 smithi028 ceph-mon[27786]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:33.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:33 smithi028 ceph-mon[27786]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-04-24T08:03:33.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:33 smithi028 ceph-mon[27786]: osdmap e32: 5 total, 4 up, 5 in 2024-04-24T08:03:33.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:33 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:03:33.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:33 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:03:33.435 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:33 smithi194 ceph-mon[32836]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:33.435 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:33 smithi194 ceph-mon[32836]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-04-24T08:03:33.435 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:33 smithi194 ceph-mon[32836]: osdmap e32: 5 total, 4 up, 5 in 2024-04-24T08:03:33.435 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:33 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:03:33.435 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:33 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:03:33.777 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_3 2024-04-24T08:03:34.013 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:03:34.275 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:34 smithi117 ceph-mon[32438]: purged_snaps scrub starts 2024-04-24T08:03:34.275 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:34 smithi117 ceph-mon[32438]: purged_snaps scrub ok 2024-04-24T08:03:34.275 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:34 smithi117 ceph-mon[32438]: osd.4 [v2:172.21.15.117:6808/1649446681,v1:172.21.15.117:6809/1649446681] boot 2024-04-24T08:03:34.275 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:34 smithi117 ceph-mon[32438]: osdmap e33: 5 total, 5 up, 5 in 2024-04-24T08:03:34.275 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:34 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:03:34.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:34 smithi028 ceph-mon[27786]: purged_snaps scrub starts 2024-04-24T08:03:34.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:34 smithi028 ceph-mon[27786]: purged_snaps scrub ok 2024-04-24T08:03:34.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:34 smithi028 ceph-mon[27786]: osd.4 [v2:172.21.15.117:6808/1649446681,v1:172.21.15.117:6809/1649446681] boot 2024-04-24T08:03:34.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:34 smithi028 ceph-mon[27786]: osdmap e33: 5 total, 5 up, 5 in 2024-04-24T08:03:34.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:34 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:03:34.354 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:34 smithi194 ceph-mon[32836]: purged_snaps scrub starts 2024-04-24T08:03:34.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:34 smithi194 ceph-mon[32836]: purged_snaps scrub ok 2024-04-24T08:03:34.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:34 smithi194 ceph-mon[32836]: osd.4 [v2:172.21.15.117:6808/1649446681,v1:172.21.15.117:6809/1649446681] boot 2024-04-24T08:03:34.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:34 smithi194 ceph-mon[32836]: osdmap e33: 5 total, 5 up, 5 in 2024-04-24T08:03:34.355 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:34 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:03:35.117 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:35 smithi194 ceph-mon[32836]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:35.117 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:35 smithi194 ceph-mon[32836]: osdmap e34: 5 total, 5 up, 5 in 2024-04-24T08:03:35.117 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:35 smithi194 ceph-mon[32836]: Detected new or changed devices on smithi117 2024-04-24T08:03:35.117 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:35.117 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:35.117 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:35.117 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:35.117 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:35 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi117 to 8544M 2024-04-24T08:03:35.118 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:35.118 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:35.118 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:35.118 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:35.118 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:35.118 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:35.118 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:35.118 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:35.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:35 smithi028 ceph-mon[27786]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:35.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:35 smithi028 ceph-mon[27786]: osdmap e34: 5 total, 5 up, 5 in 2024-04-24T08:03:35.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:35 smithi028 ceph-mon[27786]: Detected new or changed devices on smithi117 2024-04-24T08:03:35.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:35.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:35.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:35.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:35.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:35 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi117 to 8544M 2024-04-24T08:03:35.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:35.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:35.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:35.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:35.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:35.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:35.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:35.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:35.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:35 smithi117 ceph-mon[32438]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-04-24T08:03:35.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:35 smithi117 ceph-mon[32438]: osdmap e34: 5 total, 5 up, 5 in 2024-04-24T08:03:35.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:35 smithi117 ceph-mon[32438]: Detected new or changed devices on smithi117 2024-04-24T08:03:35.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:35.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:35.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:35.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:35.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:35 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi117 to 8544M 2024-04-24T08:03:35.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:35.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:35.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:35.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:35.380 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:35.380 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:35.380 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:35.380 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:35.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.587+0000 7f43cd941700 1 -- 172.21.15.194:0/2126060685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43c8103d50 msgr2=0x7f43c81041b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:03:35.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.587+0000 7f43cd941700 1 --2- 172.21.15.194:0/2126060685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43c8103d50 0x7f43c81041b0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f43bc009990 tx=0x7f43bc009ca0 comp rx=0 tx=0).stop 2024-04-24T08:03:35.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.587+0000 7f43cd941700 1 -- 172.21.15.194:0/2126060685 shutdown_connections 2024-04-24T08:03:35.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.587+0000 7f43cd941700 1 --2- 172.21.15.194:0/2126060685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43c81046f0 0x7f43c810afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:35.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.587+0000 7f43cd941700 1 --2- 172.21.15.194:0/2126060685 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43c8102b50 0x7f43c8102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:35.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.587+0000 7f43cd941700 1 --2- 172.21.15.194:0/2126060685 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43c8103d50 0x7f43c81041b0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:35.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.587+0000 7f43cd941700 1 -- 172.21.15.194:0/2126060685 >> 172.21.15.194:0/2126060685 conn(0x7f43c80fe0f0 msgr2=0x7f43c8100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:03:35.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.588+0000 7f43cd941700 1 -- 172.21.15.194:0/2126060685 shutdown_connections 2024-04-24T08:03:35.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.588+0000 7f43cd941700 1 -- 172.21.15.194:0/2126060685 wait complete. 2024-04-24T08:03:35.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.588+0000 7f43cd941700 1 Processor -- start 2024-04-24T08:03:35.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.588+0000 7f43cd941700 1 -- start start 2024-04-24T08:03:35.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.588+0000 7f43cd941700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43c8102b50 0x7f43c8198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:03:35.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.588+0000 7f43cd941700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43c8103d50 0x7f43c8199460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:03:35.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.588+0000 7f43cd941700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43c81046f0 0x7f43c81a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:03:35.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.588+0000 7f43cd941700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f43c810e880 con 0x7f43c81046f0 2024-04-24T08:03:35.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.588+0000 7f43cd941700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f43c810e700 con 0x7f43c8103d50 2024-04-24T08:03:35.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.588+0000 7f43cd941700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f43c810ea00 con 0x7f43c8102b50 2024-04-24T08:03:35.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.589+0000 7f43c6ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43c8102b50 0x7f43c8198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:03:35.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.589+0000 7f43c6ffd700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43c8102b50 0x7f43c8198f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:58108/0 (socket says 172.21.15.194:58108) 2024-04-24T08:03:35.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.589+0000 7f43c6ffd700 1 -- 172.21.15.194:0/257209294 learned_addr learned my addr 172.21.15.194:0/257209294 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-24T08:03:35.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.589+0000 7f43c67fc700 1 --2- 172.21.15.194:0/257209294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43c8103d50 0x7f43c8199460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:03:35.593 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.589+0000 7f43c6ffd700 1 -- 172.21.15.194:0/257209294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43c8103d50 msgr2=0x7f43c8199460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:03:35.593 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.589+0000 7f43c6ffd700 1 --2- 172.21.15.194:0/257209294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43c8103d50 0x7f43c8199460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:35.593 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.589+0000 7f43c6ffd700 1 -- 172.21.15.194:0/257209294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43c81046f0 msgr2=0x7f43c81a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:03:35.593 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.589+0000 7f43c6ffd700 1 --2- 172.21.15.194:0/257209294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43c81046f0 0x7f43c81a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:35.593 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.589+0000 7f43c6ffd700 1 -- 172.21.15.194:0/257209294 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f43c81a0b80 con 0x7f43c8102b50 2024-04-24T08:03:35.593 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.589+0000 7f43c6ffd700 1 --2- 172.21.15.194:0/257209294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43c8102b50 0x7f43c8198f20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f43b800b530 tx=0x7f43b800b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:03:35.593 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.589+0000 7f43cc93f700 1 -- 172.21.15.194:0/257209294 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b8015020 con 0x7f43c8102b50 2024-04-24T08:03:35.593 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.589+0000 7f43cd941700 1 -- 172.21.15.194:0/257209294 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f43c81a0e70 con 0x7f43c8102b50 2024-04-24T08:03:35.593 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.589+0000 7f43cc93f700 1 -- 172.21.15.194:0/257209294 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f43b8011d10 con 0x7f43c8102b50 2024-04-24T08:03:35.593 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.589+0000 7f43cc93f700 1 -- 172.21.15.194:0/257209294 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f43b800f2e0 con 0x7f43c8102b50 2024-04-24T08:03:35.593 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.589+0000 7f43cd941700 1 -- 172.21.15.194:0/257209294 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f43c81a13a0 con 0x7f43c8102b50 2024-04-24T08:03:35.593 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.590+0000 7f43cd941700 1 -- 172.21.15.194:0/257209294 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f43c80697e0 con 0x7f43c8102b50 2024-04-24T08:03:35.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.592+0000 7f43cc93f700 1 -- 172.21.15.194:0/257209294 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f43b8018020 con 0x7f43c8102b50 2024-04-24T08:03:35.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.592+0000 7f43cc93f700 1 --2- 172.21.15.194:0/257209294 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f43b00a3c40 0x7f43b00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:03:35.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.593+0000 7f43cc93f700 1 -- 172.21.15.194:0/257209294 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f43b80cf0c0 con 0x7f43c8102b50 2024-04-24T08:03:35.594 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.593+0000 7f43c67fc700 1 --2- 172.21.15.194:0/257209294 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f43b00a3c40 0x7f43b00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:03:35.595 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.594+0000 7f43c67fc700 1 --2- 172.21.15.194:0/257209294 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f43b00a3c40 0x7f43b00a6100 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f43bc000c00 tx=0x7f43bc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:03:35.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.596+0000 7f43cc93f700 1 -- 172.21.15.194:0/257209294 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f43b801f020 con 0x7f43c8102b50 2024-04-24T08:03:35.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:35.803+0000 7f43cd941700 1 -- 172.21.15.194:0/257209294 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f43c8062f30 con 0x7f43b00a3c40 2024-04-24T08:03:36.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:36 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:03:36.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:36 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:03:36.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:36 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:36.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:36 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:03:36.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:36 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:03:36.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:36 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:36.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:36 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:03:36.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:36 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:03:36.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:36 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:37.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:37 smithi117 ceph-mon[32438]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:03:37.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:37 smithi117 ceph-mon[32438]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:37.401 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:37 smithi194 ceph-mon[32836]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:03:37.401 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:37 smithi194 ceph-mon[32836]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:37.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:37 smithi028 ceph-mon[27786]: from='client.24253 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:03:37.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:37 smithi028 ceph-mon[27786]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 137 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:39.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:39 smithi194 ceph-mon[32836]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:39.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:39 smithi194 ceph-mon[32836]: from='client.? 172.21.15.194:0/953471508' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcbda2ee-300e-498a-bb5c-c1d112d03f05"}]: dispatch 2024-04-24T08:03:39.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:39 smithi194 ceph-mon[32836]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcbda2ee-300e-498a-bb5c-c1d112d03f05"}]: dispatch 2024-04-24T08:03:39.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:39 smithi194 ceph-mon[32836]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcbda2ee-300e-498a-bb5c-c1d112d03f05"}]': finished 2024-04-24T08:03:39.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:39 smithi194 ceph-mon[32836]: osdmap e35: 6 total, 5 up, 6 in 2024-04-24T08:03:39.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:39 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:03:39.342 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:39 smithi028 ceph-mon[27786]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:39.342 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:39 smithi028 ceph-mon[27786]: from='client.? 172.21.15.194:0/953471508' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcbda2ee-300e-498a-bb5c-c1d112d03f05"}]: dispatch 2024-04-24T08:03:39.342 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:39 smithi028 ceph-mon[27786]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcbda2ee-300e-498a-bb5c-c1d112d03f05"}]: dispatch 2024-04-24T08:03:39.342 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:39 smithi028 ceph-mon[27786]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcbda2ee-300e-498a-bb5c-c1d112d03f05"}]': finished 2024-04-24T08:03:39.342 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:39 smithi028 ceph-mon[27786]: osdmap e35: 6 total, 5 up, 6 in 2024-04-24T08:03:39.342 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:39 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:03:39.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:39 smithi117 ceph-mon[32438]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:39.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:39 smithi117 ceph-mon[32438]: from='client.? 172.21.15.194:0/953471508' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcbda2ee-300e-498a-bb5c-c1d112d03f05"}]: dispatch 2024-04-24T08:03:39.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:39 smithi117 ceph-mon[32438]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fcbda2ee-300e-498a-bb5c-c1d112d03f05"}]: dispatch 2024-04-24T08:03:39.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:39 smithi117 ceph-mon[32438]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fcbda2ee-300e-498a-bb5c-c1d112d03f05"}]': finished 2024-04-24T08:03:39.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:39 smithi117 ceph-mon[32438]: osdmap e35: 6 total, 5 up, 6 in 2024-04-24T08:03:39.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:39 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:03:40.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:40 smithi117 ceph-mon[32438]: from='client.? 172.21.15.194:0/2370704335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:03:40.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:40 smithi194 ceph-mon[32836]: from='client.? 172.21.15.194:0/2370704335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:03:40.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:40 smithi028 ceph-mon[27786]: from='client.? 172.21.15.194:0/2370704335' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:03:41.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:41 smithi117 ceph-mon[32438]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:41.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:41 smithi194 ceph-mon[32836]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:41.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:41 smithi028 ceph-mon[27786]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:43.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:43 smithi117 ceph-mon[32438]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:43.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:43 smithi194 ceph-mon[32836]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:43.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:43 smithi028 ceph-mon[27786]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:45.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:45 smithi117 ceph-mon[32438]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:45.418 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:45 smithi194 ceph-mon[32836]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:45.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:45 smithi028 ceph-mon[27786]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:46.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:46 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T08:03:46.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:46 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:46.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:46 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T08:03:46.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:46 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:46.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:46 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-04-24T08:03:46.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:46 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:47.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:47 smithi117 ceph-mon[32438]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:47.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:47 smithi117 ceph-mon[32438]: Deploying daemon osd.5 on smithi194 2024-04-24T08:03:47.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:47 smithi194 ceph-mon[32836]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:47.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:47 smithi194 ceph-mon[32836]: Deploying daemon osd.5 on smithi194 2024-04-24T08:03:47.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:47 smithi028 ceph-mon[27786]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:47.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:47 smithi028 ceph-mon[27786]: Deploying daemon osd.5 on smithi194 2024-04-24T08:03:49.330 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:49 smithi194 ceph-mon[32836]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:49.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:49 smithi117 ceph-mon[32438]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:49.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:49 smithi028 ceph-mon[27786]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:51.353 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:51 smithi194 ceph-mon[32836]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:51.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:51 smithi117 ceph-mon[32438]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:51.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:51 smithi028 ceph-mon[27786]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:52.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:52 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:52.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:52 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:52.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:52 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:53.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:52 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:53.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:52 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:53.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:52 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:53.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:52 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:53.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:52 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:53.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:52 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:53.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:53 smithi194 ceph-mon[32836]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:54.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:53 smithi028 ceph-mon[27786]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:54.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:53 smithi117 ceph-mon[32438]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:55.143 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 5 on host 'smithi194' 2024-04-24T08:03:55.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:55.142+0000 7f43cc93f700 1 -- 172.21.15.194:0/257209294 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f43c8062f30 con 0x7f43b00a3c40 2024-04-24T08:03:55.146 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:55.145+0000 7f43cd941700 1 -- 172.21.15.194:0/257209294 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f43b00a3c40 msgr2=0x7f43b00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:03:55.146 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:55.145+0000 7f43cd941700 1 --2- 172.21.15.194:0/257209294 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f43b00a3c40 0x7f43b00a6100 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f43bc000c00 tx=0x7f43bc005fb0 comp rx=0 tx=0).stop 2024-04-24T08:03:55.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:55.145+0000 7f43cd941700 1 -- 172.21.15.194:0/257209294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43c8102b50 msgr2=0x7f43c8198f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:03:55.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:55.145+0000 7f43cd941700 1 --2- 172.21.15.194:0/257209294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43c8102b50 0x7f43c8198f20 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f43b800b530 tx=0x7f43b800b8f0 comp rx=0 tx=0).stop 2024-04-24T08:03:55.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:55.145+0000 7f43cd941700 1 -- 172.21.15.194:0/257209294 shutdown_connections 2024-04-24T08:03:55.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:55.145+0000 7f43cd941700 1 --2- 172.21.15.194:0/257209294 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f43b00a3c40 0x7f43b00a6100 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:55.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:55.145+0000 7f43cd941700 1 --2- 172.21.15.194:0/257209294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f43c81046f0 0x7f43c81a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:55.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:55.145+0000 7f43cd941700 1 --2- 172.21.15.194:0/257209294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f43c8102b50 0x7f43c8198f20 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:55.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:55.145+0000 7f43cd941700 1 --2- 172.21.15.194:0/257209294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43c8103d50 0x7f43c8199460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:03:55.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:55.146+0000 7f43cd941700 1 -- 172.21.15.194:0/257209294 >> 172.21.15.194:0/257209294 conn(0x7f43c80fe0f0 msgr2=0x7f43c81070d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:03:55.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:55.146+0000 7f43cd941700 1 -- 172.21.15.194:0/257209294 shutdown_connections 2024-04-24T08:03:55.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:03:55.146+0000 7f43cd941700 1 -- 172.21.15.194:0/257209294 wait complete. 2024-04-24T08:03:55.795 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:55 smithi194 ceph-mon[32836]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:55.795 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:55 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:55.796 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:55 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:55.796 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:55 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:55.796 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:55 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:55.796 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:55 smithi194 ceph-mon[32836]: from='osd.5 [v2:172.21.15.194:6808/1399608260,v1:172.21.15.194:6809/1399608260]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T08:03:55.796 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:55 smithi194 ceph-mon[32836]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T08:03:55.825 DEBUG:teuthology.orchestra.run.smithi194:osd.5> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.5.service 2024-04-24T08:03:55.827 INFO:tasks.cephadm:Deploying osd.6 on smithi028 with /dev/vg_nvme/lv_2... 2024-04-24T08:03:55.828 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-24T08:03:56.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:55 smithi028 ceph-mon[27786]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:56.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:55 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:56.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:55 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:56.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:55 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:56.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:55 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:56.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:55 smithi028 ceph-mon[27786]: from='osd.5 [v2:172.21.15.194:6808/1399608260,v1:172.21.15.194:6809/1399608260]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T08:03:56.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:55 smithi028 ceph-mon[27786]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T08:03:56.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:55 smithi117 ceph-mon[32438]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:56.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:55 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:56.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:55 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:56.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:55 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:56.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:55 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:56.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:55 smithi117 ceph-mon[32438]: from='osd.5 [v2:172.21.15.194:6808/1399608260,v1:172.21.15.194:6809/1399608260]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T08:03:56.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:55 smithi117 ceph-mon[32438]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-04-24T08:03:56.300 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:03:57.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:57 smithi194 ceph-mon[32836]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:57.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:57 smithi194 ceph-mon[32836]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-24T08:03:57.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:57 smithi194 ceph-mon[32836]: from='osd.5 [v2:172.21.15.194:6808/1399608260,v1:172.21.15.194:6809/1399608260]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:03:57.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:57 smithi194 ceph-mon[32836]: osdmap e36: 6 total, 5 up, 6 in 2024-04-24T08:03:57.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:57 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:03:57.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:57 smithi194 ceph-mon[32836]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:03:57.753 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:57 smithi028 ceph-mon[27786]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:57.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:57 smithi028 ceph-mon[27786]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-24T08:03:57.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:57 smithi028 ceph-mon[27786]: from='osd.5 [v2:172.21.15.194:6808/1399608260,v1:172.21.15.194:6809/1399608260]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:03:57.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:57 smithi028 ceph-mon[27786]: osdmap e36: 6 total, 5 up, 6 in 2024-04-24T08:03:57.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:57 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:03:57.754 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:57 smithi028 ceph-mon[27786]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:03:57.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:57 smithi117 ceph-mon[32438]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:57.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:57 smithi117 ceph-mon[32438]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-04-24T08:03:57.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:57 smithi117 ceph-mon[32438]: from='osd.5 [v2:172.21.15.194:6808/1399608260,v1:172.21.15.194:6809/1399608260]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:03:57.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:57 smithi117 ceph-mon[32438]: osdmap e36: 6 total, 5 up, 6 in 2024-04-24T08:03:57.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:57 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:03:57.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:57 smithi117 ceph-mon[32438]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:03:58.191 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 24 08:03:57 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-5[44967]: 2024-04-24T08:03:57.935+0000 7f638ea9d700 -1 osd.5 0 waiting for initial osdmap 2024-04-24T08:03:58.191 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 24 08:03:57 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-5[44967]: 2024-04-24T08:03:57.941+0000 7f6386087700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:03:58.601 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:58 smithi194 ceph-mon[32836]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-24T08:03:58.601 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:58 smithi194 ceph-mon[32836]: osdmap e37: 6 total, 5 up, 6 in 2024-04-24T08:03:58.601 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:03:58.601 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:03:58.601 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:58 smithi194 ceph-mon[32836]: from='osd.5 ' entity='osd.5' 2024-04-24T08:03:58.601 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:58 smithi194 ceph-mon[32836]: osd.5 [v2:172.21.15.194:6808/1399608260,v1:172.21.15.194:6809/1399608260] boot 2024-04-24T08:03:58.601 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:58 smithi194 ceph-mon[32836]: osdmap e38: 6 total, 6 up, 6 in 2024-04-24T08:03:58.601 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:03:58.697 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:58 smithi028 ceph-mon[27786]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-24T08:03:58.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:58 smithi028 ceph-mon[27786]: osdmap e37: 6 total, 5 up, 6 in 2024-04-24T08:03:58.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:03:58.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:03:58.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:58 smithi028 ceph-mon[27786]: from='osd.5 ' entity='osd.5' 2024-04-24T08:03:58.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:58 smithi028 ceph-mon[27786]: osd.5 [v2:172.21.15.194:6808/1399608260,v1:172.21.15.194:6809/1399608260] boot 2024-04-24T08:03:58.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:58 smithi028 ceph-mon[27786]: osdmap e38: 6 total, 6 up, 6 in 2024-04-24T08:03:58.698 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:03:58.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:58 smithi117 ceph-mon[32438]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-24T08:03:58.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:58 smithi117 ceph-mon[32438]: osdmap e37: 6 total, 5 up, 6 in 2024-04-24T08:03:58.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:58 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:03:58.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:58 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:03:58.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:58 smithi117 ceph-mon[32438]: from='osd.5 ' entity='osd.5' 2024-04-24T08:03:58.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:58 smithi117 ceph-mon[32438]: osd.5 [v2:172.21.15.194:6808/1399608260,v1:172.21.15.194:6809/1399608260] boot 2024-04-24T08:03:58.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:58 smithi117 ceph-mon[32438]: osdmap e38: 6 total, 6 up, 6 in 2024-04-24T08:03:58.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:58 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:03:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-24T08:03:59.347 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-24T08:03:59.348 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-24T08:03:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-24T08:03:59.348 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0580551 s, 181 MB/s 2024-04-24T08:03:59.348 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-24T08:03:59.726 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:59 smithi028 ceph-mon[27786]: purged_snaps scrub starts 2024-04-24T08:03:59.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:59 smithi028 ceph-mon[27786]: purged_snaps scrub ok 2024-04-24T08:03:59.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:59 smithi028 ceph-mon[27786]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:59.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:59 smithi028 ceph-mon[27786]: osdmap e39: 6 total, 6 up, 6 in 2024-04-24T08:03:59.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:59.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:59.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:59.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:59.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:59.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:59.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:59.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:59.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:59.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:59.727 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:59.728 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:03:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:59.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:59 smithi117 ceph-mon[32438]: purged_snaps scrub starts 2024-04-24T08:03:59.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:59 smithi117 ceph-mon[32438]: purged_snaps scrub ok 2024-04-24T08:03:59.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:59 smithi117 ceph-mon[32438]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:59.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:59 smithi117 ceph-mon[32438]: osdmap e39: 6 total, 6 up, 6 in 2024-04-24T08:03:59.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:59 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:59.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:59 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:59.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:59 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:59.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:59 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:59.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:59 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:59.882 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:59 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:59.882 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:59 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:59.882 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:59 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:59.882 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:59 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:59.882 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:59 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:59.882 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:59 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:59.883 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:03:59 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:59.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:59 smithi194 ceph-mon[32836]: purged_snaps scrub starts 2024-04-24T08:03:59.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:59 smithi194 ceph-mon[32836]: purged_snaps scrub ok 2024-04-24T08:03:59.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:59 smithi194 ceph-mon[32836]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 138 MiB used, 447 GiB / 447 GiB avail 2024-04-24T08:03:59.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:59 smithi194 ceph-mon[32836]: osdmap e39: 6 total, 6 up, 6 in 2024-04-24T08:03:59.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:59.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:59.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:59.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:03:59.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:59.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:59.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:59.943 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:03:59.943 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:03:59.943 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:03:59.943 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:03:59.943 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:03:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:00.339 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_2 2024-04-24T08:04:00.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:00 smithi028 ceph-mon[27786]: Detected new or changed devices on smithi194 2024-04-24T08:04:00.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:00 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi194 to 8544M 2024-04-24T08:04:00.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:00 smithi028 ceph-mon[27786]: osdmap e40: 6 total, 6 up, 6 in 2024-04-24T08:04:00.801 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:04:00.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:00 smithi117 ceph-mon[32438]: Detected new or changed devices on smithi194 2024-04-24T08:04:00.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:00 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi194 to 8544M 2024-04-24T08:04:00.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:00 smithi117 ceph-mon[32438]: osdmap e40: 6 total, 6 up, 6 in 2024-04-24T08:04:00.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:00 smithi194 ceph-mon[32836]: Detected new or changed devices on smithi194 2024-04-24T08:04:00.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:00 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi194 to 8544M 2024-04-24T08:04:00.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:00 smithi194 ceph-mon[32836]: osdmap e40: 6 total, 6 up, 6 in 2024-04-24T08:04:01.716 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:01 smithi028 ceph-mon[27786]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:01.841 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:01 smithi194 ceph-mon[32836]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:01.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:01 smithi117 ceph-mon[32438]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:02.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:02 smithi028 ceph-mon[27786]: osdmap e41: 6 total, 6 up, 6 in 2024-04-24T08:04:02.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:02 smithi117 ceph-mon[32438]: osdmap e41: 6 total, 6 up, 6 in 2024-04-24T08:04:02.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:02 smithi194 ceph-mon[32836]: osdmap e41: 6 total, 6 up, 6 in 2024-04-24T08:04:02.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.971+0000 7f4cb8600700 1 -- 172.21.15.28:0/970862937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cb0076b20 msgr2=0x7f4cb0074f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:04:02.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.971+0000 7f4cb8600700 1 --2- 172.21.15.28:0/970862937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cb0076b20 0x7f4cb0074f80 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4009990 tx=0x7f4ca4009ca0 comp rx=0 tx=0).stop 2024-04-24T08:04:02.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.971+0000 7f4cb8600700 1 -- 172.21.15.28:0/970862937 shutdown_connections 2024-04-24T08:04:02.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.971+0000 7f4cb8600700 1 --2- 172.21.15.28:0/970862937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cb010f150 0x7f4cb0111540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:02.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.971+0000 7f4cb8600700 1 --2- 172.21.15.28:0/970862937 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cb0076b20 0x7f4cb0074f80 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:02.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.971+0000 7f4cb8600700 1 --2- 172.21.15.28:0/970862937 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cb0075550 0x7f4cb00759d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:02.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.971+0000 7f4cb8600700 1 -- 172.21.15.28:0/970862937 >> 172.21.15.28:0/970862937 conn(0x7f4cb00fe110 msgr2=0x7f4cb0100550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:04:02.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.971+0000 7f4cb8600700 1 -- 172.21.15.28:0/970862937 shutdown_connections 2024-04-24T08:04:02.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.971+0000 7f4cb8600700 1 -- 172.21.15.28:0/970862937 wait complete. 2024-04-24T08:04:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.972+0000 7f4cb8600700 1 Processor -- start 2024-04-24T08:04:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.973+0000 7f4cb8600700 1 -- start start 2024-04-24T08:04:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.973+0000 7f4cb8600700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cb0075550 0x7f4cb019f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:04:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.973+0000 7f4cb8600700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cb0076b20 0x7f4cb019f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:04:02.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.973+0000 7f4cb8600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cb010f150 0x7f4cb01a6960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:04:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.973+0000 7f4cb8600700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb0114da0 con 0x7f4cb0076b20 2024-04-24T08:04:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.973+0000 7f4cb8600700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb0114c20 con 0x7f4cb0075550 2024-04-24T08:04:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.973+0000 7f4cb8600700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4cb0114f20 con 0x7f4cb010f150 2024-04-24T08:04:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.973+0000 7f4cb6600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cb010f150 0x7f4cb01a6960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:04:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.973+0000 7f4cb5c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cb0075550 0x7f4cb019f3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:04:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.973+0000 7f4cb5200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cb0076b20 0x7f4cb019f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:04:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.973+0000 7f4cb6600700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cb010f150 0x7f4cb01a6960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:39748/0 (socket says 172.21.15.28:39748) 2024-04-24T08:04:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.973+0000 7f4cb6600700 1 -- 172.21.15.28:0/1734931864 learned_addr learned my addr 172.21.15.28:0/1734931864 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:04:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.973+0000 7f4cb5c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cb0075550 0x7f4cb019f3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:38792/0 (socket says 172.21.15.28:38792) 2024-04-24T08:04:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.974+0000 7f4cb6600700 1 -- 172.21.15.28:0/1734931864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cb0075550 msgr2=0x7f4cb019f3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:04:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.974+0000 7f4cb6600700 1 --2- 172.21.15.28:0/1734931864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cb0075550 0x7f4cb019f3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.974+0000 7f4cb6600700 1 -- 172.21.15.28:0/1734931864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cb0076b20 msgr2=0x7f4cb019f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:04:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.974+0000 7f4cb6600700 1 --2- 172.21.15.28:0/1734931864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cb0076b20 0x7f4cb019f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:02.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.974+0000 7f4cb6600700 1 -- 172.21.15.28:0/1734931864 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4cb01a7000 con 0x7f4cb010f150 2024-04-24T08:04:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.974+0000 7f4cb5c00700 1 --2- 172.21.15.28:0/1734931864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cb0075550 0x7f4cb019f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:04:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.974+0000 7f4cb5200700 1 --2- 172.21.15.28:0/1734931864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cb0076b20 0x7f4cb019f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:04:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.974+0000 7f4cb6600700 1 --2- 172.21.15.28:0/1734931864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cb010f150 0x7f4cb01a6960 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4ca800ef60 tx=0x7f4ca800ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:04:02.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.974+0000 7f4cae000700 1 -- 172.21.15.28:0/1734931864 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca80092e0 con 0x7f4cb010f150 2024-04-24T08:04:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.975+0000 7f4cae000700 1 -- 172.21.15.28:0/1734931864 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ca8004bb0 con 0x7f4cb010f150 2024-04-24T08:04:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.975+0000 7f4cae000700 1 -- 172.21.15.28:0/1734931864 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ca8004280 con 0x7f4cb010f150 2024-04-24T08:04:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.975+0000 7f4cb8600700 1 -- 172.21.15.28:0/1734931864 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4cb01a7290 con 0x7f4cb010f150 2024-04-24T08:04:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.975+0000 7f4cb8600700 1 -- 172.21.15.28:0/1734931864 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4cb01a7780 con 0x7f4cb010f150 2024-04-24T08:04:02.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.976+0000 7f4cb8600700 1 -- 172.21.15.28:0/1734931864 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4cb00697e0 con 0x7f4cb010f150 2024-04-24T08:04:02.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.979+0000 7f4cae000700 1 -- 172.21.15.28:0/1734931864 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4ca801b030 con 0x7f4cb010f150 2024-04-24T08:04:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.980+0000 7f4cae000700 1 --2- 172.21.15.28:0/1734931864 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4c980a3cc0 0x7f4c980a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:04:02.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.980+0000 7f4cae000700 1 -- 172.21.15.28:0/1734931864 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(42..42 src has 1..42) ==== 4533+0+0 (secure 0 0 0) 0x7f4ca80cf910 con 0x7f4cb010f150 2024-04-24T08:04:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.980+0000 7f4cb5c00700 1 --2- 172.21.15.28:0/1734931864 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4c980a3cc0 0x7f4c980a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:04:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.981+0000 7f4cb5c00700 1 --2- 172.21.15.28:0/1734931864 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4c980a3cc0 0x7f4c980a6180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4000c00 tx=0x7f4ca4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:04:02.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:02.981+0000 7f4cae000700 1 -- 172.21.15.28:0/1734931864 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4ca8022020 con 0x7f4cb010f150 2024-04-24T08:04:03.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:03.225+0000 7f4cb8600700 1 -- 172.21.15.28:0/1734931864 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f4cb0062f30 con 0x7f4c980a3cc0 2024-04-24T08:04:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:03 smithi028 ceph-mon[27786]: pgmap v157: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-24T08:04:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:03 smithi028 ceph-mon[27786]: osdmap e42: 6 total, 6 up, 6 in 2024-04-24T08:04:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:03 smithi028 ceph-mon[27786]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-24T08:04:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:03 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:04:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:03 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:04:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:03 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:03.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:03 smithi117 ceph-mon[32438]: pgmap v157: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-24T08:04:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:03 smithi117 ceph-mon[32438]: osdmap e42: 6 total, 6 up, 6 in 2024-04-24T08:04:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:03 smithi117 ceph-mon[32438]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-24T08:04:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:03 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:04:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:03 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:04:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:03 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:03.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:03 smithi194 ceph-mon[32836]: pgmap v157: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 174 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-24T08:04:03.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:03 smithi194 ceph-mon[32836]: osdmap e42: 6 total, 6 up, 6 in 2024-04-24T08:04:03.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:03 smithi194 ceph-mon[32836]: Health check failed: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded (PG_DEGRADED) 2024-04-24T08:04:03.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:03 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:04:03.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:03 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:04:03.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:03 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:04.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:04 smithi028 ceph-mon[27786]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:04:04.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:04 smithi117 ceph-mon[32438]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:04:04.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:04 smithi194 ceph-mon[32836]: from='client.24280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:04:05.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:05 smithi028 ceph-mon[27786]: pgmap v159: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-24T08:04:05.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:05 smithi117 ceph-mon[32438]: pgmap v159: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-24T08:04:05.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:05 smithi194 ceph-mon[32836]: pgmap v159: 1 pgs: 1 activating+undersized+degraded+remapped; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 2/6 objects degraded (33.333%) 2024-04-24T08:04:06.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:06 smithi117 ceph-mon[32438]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-24T08:04:06.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:06 smithi117 ceph-mon[32438]: Cluster is now healthy 2024-04-24T08:04:06.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:06 smithi194 ceph-mon[32836]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-24T08:04:06.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:06 smithi194 ceph-mon[32836]: Cluster is now healthy 2024-04-24T08:04:07.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:06 smithi028 ceph-mon[27786]: Health check cleared: PG_DEGRADED (was: Degraded data redundancy: 2/6 objects degraded (33.333%), 1 pg degraded) 2024-04-24T08:04:07.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:06 smithi028 ceph-mon[27786]: Cluster is now healthy 2024-04-24T08:04:07.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:07 smithi028 ceph-mon[27786]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-04-24T08:04:07.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:07 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3668869463' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4"}]: dispatch 2024-04-24T08:04:07.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:07 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3668869463' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4"}]': finished 2024-04-24T08:04:07.837 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:07 smithi028 ceph-mon[27786]: osdmap e43: 7 total, 6 up, 7 in 2024-04-24T08:04:07.838 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:07.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:07 smithi117 ceph-mon[32438]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-04-24T08:04:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:07 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3668869463' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4"}]: dispatch 2024-04-24T08:04:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:07 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3668869463' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4"}]': finished 2024-04-24T08:04:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:07 smithi117 ceph-mon[32438]: osdmap e43: 7 total, 6 up, 7 in 2024-04-24T08:04:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:07 smithi194 ceph-mon[32836]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 94 KiB/s, 0 objects/s recovering 2024-04-24T08:04:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:07 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3668869463' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4"}]: dispatch 2024-04-24T08:04:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:07 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3668869463' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4"}]': finished 2024-04-24T08:04:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:07 smithi194 ceph-mon[32836]: osdmap e43: 7 total, 6 up, 7 in 2024-04-24T08:04:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:08.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:08 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/4061571933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:04:08.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:08 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/4061571933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:04:09.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:08 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/4061571933' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:04:09.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:09 smithi117 ceph-mon[32438]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-24T08:04:09.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:09 smithi194 ceph-mon[32836]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-24T08:04:10.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:09 smithi028 ceph-mon[27786]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 89 KiB/s, 0 objects/s recovering 2024-04-24T08:04:11.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:11 smithi117 ceph-mon[32438]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T08:04:11.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:11 smithi194 ceph-mon[32836]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T08:04:12.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:11 smithi028 ceph-mon[27786]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T08:04:13.871 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:13 smithi028 ceph-mon[27786]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-24T08:04:13.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:13 smithi117 ceph-mon[32438]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-24T08:04:13.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:13 smithi194 ceph-mon[32836]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 61 KiB/s, 0 objects/s recovering 2024-04-24T08:04:15.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:15 smithi028 ceph-mon[27786]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-24T08:04:15.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:15 smithi117 ceph-mon[32438]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-24T08:04:15.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:15 smithi194 ceph-mon[32836]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-24T08:04:17.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:17 smithi117 ceph-mon[32438]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:17.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:17 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T08:04:17.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:17 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:17.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:17 smithi194 ceph-mon[32836]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:17.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:17 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T08:04:17.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:17 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:17.976 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:17 smithi028 ceph-mon[27786]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:17.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:17 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-04-24T08:04:17.977 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:17 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:18.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:18 smithi117 ceph-mon[32438]: Deploying daemon osd.6 on smithi028 2024-04-24T08:04:18.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:18 smithi194 ceph-mon[32836]: Deploying daemon osd.6 on smithi028 2024-04-24T08:04:19.034 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:18 smithi028 ceph-mon[27786]: Deploying daemon osd.6 on smithi028 2024-04-24T08:04:19.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:19 smithi117 ceph-mon[32438]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:19.880 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:19 smithi028 ceph-mon[27786]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:19.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:19 smithi194 ceph-mon[32836]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:21.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:21 smithi028 ceph-mon[27786]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:21.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:21 smithi194 ceph-mon[32836]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:22.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:21 smithi117 ceph-mon[32438]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:23.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:23 smithi194 ceph-mon[32836]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:24.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:23 smithi028 ceph-mon[27786]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:24.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:23 smithi117 ceph-mon[32438]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:25.880 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:25 smithi028 ceph-mon[27786]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:25.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:25 smithi194 ceph-mon[32836]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:26.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:25 smithi117 ceph-mon[32438]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:26.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:04:26.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:26.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:26.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:04:26.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:26.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:27.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:04:27.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:27.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:27.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:27 smithi028 ceph-mon[27786]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:27.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:27 smithi194 ceph-mon[32836]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:28.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:27 smithi117 ceph-mon[32438]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:29.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:29 smithi194 ceph-mon[32836]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:29.987 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:29 smithi028 ceph-mon[27786]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:29 smithi117 ceph-mon[32438]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:30.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:30.970+0000 7f4cae000700 1 -- 172.21.15.28:0/1734931864 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4cb0062f30 con 0x7f4c980a3cc0 2024-04-24T08:04:30.973 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 6 on host 'smithi028' 2024-04-24T08:04:30.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:30.973+0000 7f4cb8600700 1 -- 172.21.15.28:0/1734931864 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4c980a3cc0 msgr2=0x7f4c980a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:04:30.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:30.973+0000 7f4cb8600700 1 --2- 172.21.15.28:0/1734931864 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4c980a3cc0 0x7f4c980a6180 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4ca4000c00 tx=0x7f4ca4005fb0 comp rx=0 tx=0).stop 2024-04-24T08:04:30.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:30.973+0000 7f4cb8600700 1 -- 172.21.15.28:0/1734931864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cb010f150 msgr2=0x7f4cb01a6960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:04:30.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:30.973+0000 7f4cb8600700 1 --2- 172.21.15.28:0/1734931864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cb010f150 0x7f4cb01a6960 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f4ca800ef60 tx=0x7f4ca800ef90 comp rx=0 tx=0).stop 2024-04-24T08:04:30.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:30.973+0000 7f4cb8600700 1 -- 172.21.15.28:0/1734931864 shutdown_connections 2024-04-24T08:04:30.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:30.973+0000 7f4cb8600700 1 --2- 172.21.15.28:0/1734931864 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4c980a3cc0 0x7f4c980a6180 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:30.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:30.973+0000 7f4cb8600700 1 --2- 172.21.15.28:0/1734931864 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4cb010f150 0x7f4cb01a6960 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:30.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:30.973+0000 7f4cb8600700 1 --2- 172.21.15.28:0/1734931864 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cb0075550 0x7f4cb019f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:30.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:30.973+0000 7f4cb8600700 1 --2- 172.21.15.28:0/1734931864 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4cb0076b20 0x7f4cb019f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:30.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:30.973+0000 7f4cb8600700 1 -- 172.21.15.28:0/1734931864 >> 172.21.15.28:0/1734931864 conn(0x7f4cb00fe110 msgr2=0x7f4cb01024f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:04:30.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:30.974+0000 7f4cb8600700 1 -- 172.21.15.28:0/1734931864 shutdown_connections 2024-04-24T08:04:30.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:04:30.974+0000 7f4cb8600700 1 -- 172.21.15.28:0/1734931864 wait complete. 2024-04-24T08:04:31.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:30 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:31.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:30 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:31.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:30 smithi117 ceph-mon[32438]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:31.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:30 smithi117 ceph-mon[32438]: from='osd.6 [v2:172.21.15.28:6818/1356466279,v1:172.21.15.28:6819/1356466279]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T08:04:31.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:30 smithi117 ceph-mon[32438]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T08:04:31.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:30 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:31.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:30 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:31.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:30 smithi194 ceph-mon[32836]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:31.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:30 smithi194 ceph-mon[32836]: from='osd.6 [v2:172.21.15.28:6818/1356466279,v1:172.21.15.28:6819/1356466279]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T08:04:31.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:30 smithi194 ceph-mon[32836]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T08:04:31.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:30 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:31.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:30 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:31.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:30 smithi028 ceph-mon[27786]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:30 smithi028 ceph-mon[27786]: from='osd.6 [v2:172.21.15.28:6818/1356466279,v1:172.21.15.28:6819/1356466279]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T08:04:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:30 smithi028 ceph-mon[27786]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-04-24T08:04:31.945 DEBUG:teuthology.orchestra.run.smithi028:osd.6> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.6.service 2024-04-24T08:04:31.950 INFO:tasks.cephadm:Deploying osd.7 on smithi117 with /dev/vg_nvme/lv_2... 2024-04-24T08:04:31.950 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-24T08:04:32.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:31 smithi028 ceph-mon[27786]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-24T08:04:32.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:31 smithi028 ceph-mon[27786]: osdmap e44: 7 total, 6 up, 7 in 2024-04-24T08:04:32.134 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:31 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:32.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:31 smithi028 ceph-mon[27786]: from='osd.6 [v2:172.21.15.28:6818/1356466279,v1:172.21.15.28:6819/1356466279]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:04:32.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:31 smithi028 ceph-mon[27786]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:04:32.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:31 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:32.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:31 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:32.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:31 smithi194 ceph-mon[32836]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-24T08:04:32.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:31 smithi194 ceph-mon[32836]: osdmap e44: 7 total, 6 up, 7 in 2024-04-24T08:04:32.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:31 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:32.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:31 smithi194 ceph-mon[32836]: from='osd.6 [v2:172.21.15.28:6818/1356466279,v1:172.21.15.28:6819/1356466279]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:04:32.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:31 smithi194 ceph-mon[32836]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:04:32.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:31 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:32.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:31 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:32.197 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:04:32.319 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:31 smithi117 ceph-mon[32438]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-04-24T08:04:32.320 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:31 smithi117 ceph-mon[32438]: osdmap e44: 7 total, 6 up, 7 in 2024-04-24T08:04:32.320 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:31 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:32.320 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:31 smithi117 ceph-mon[32438]: from='osd.6 [v2:172.21.15.28:6818/1356466279,v1:172.21.15.28:6819/1356466279]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:04:32.320 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:31 smithi117 ceph-mon[32438]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:04:32.320 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:31 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:32.320 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:31 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:33.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:32 smithi028 ceph-mon[27786]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-24T08:04:33.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:32 smithi028 ceph-mon[27786]: osdmap e45: 7 total, 6 up, 7 in 2024-04-24T08:04:33.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:32 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:33.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:32 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:33.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:32 smithi028 ceph-mon[27786]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:33.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:32 smithi117 ceph-mon[32438]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-24T08:04:33.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:32 smithi117 ceph-mon[32438]: osdmap e45: 7 total, 6 up, 7 in 2024-04-24T08:04:33.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:32 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:33.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:32 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:33.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:32 smithi117 ceph-mon[32438]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:33.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:32 smithi194 ceph-mon[32836]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-24T08:04:33.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:32 smithi194 ceph-mon[32836]: osdmap e45: 7 total, 6 up, 7 in 2024-04-24T08:04:33.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:32 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:33.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:32 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:33.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:32 smithi194 ceph-mon[32836]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-04-24T08:04:34.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:33 smithi028 ceph-mon[27786]: purged_snaps scrub starts 2024-04-24T08:04:34.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:33 smithi028 ceph-mon[27786]: purged_snaps scrub ok 2024-04-24T08:04:34.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:33 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:34.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:33 smithi028 ceph-mon[27786]: osd.6 [v2:172.21.15.28:6818/1356466279,v1:172.21.15.28:6819/1356466279] boot 2024-04-24T08:04:34.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:33 smithi028 ceph-mon[27786]: osdmap e46: 7 total, 7 up, 7 in 2024-04-24T08:04:34.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:33 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:34.120 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-24T08:04:34.120 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-24T08:04:34.120 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-04-24T08:04:34.120 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-04-24T08:04:34.120 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116087 s, 903 MB/s 2024-04-24T08:04:34.121 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-04-24T08:04:34.145 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:33 smithi117 ceph-mon[32438]: purged_snaps scrub starts 2024-04-24T08:04:34.145 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:33 smithi117 ceph-mon[32438]: purged_snaps scrub ok 2024-04-24T08:04:34.145 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:33 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:34.145 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:33 smithi117 ceph-mon[32438]: osd.6 [v2:172.21.15.28:6818/1356466279,v1:172.21.15.28:6819/1356466279] boot 2024-04-24T08:04:34.145 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:33 smithi117 ceph-mon[32438]: osdmap e46: 7 total, 7 up, 7 in 2024-04-24T08:04:34.145 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:33 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:34.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:33 smithi194 ceph-mon[32836]: purged_snaps scrub starts 2024-04-24T08:04:34.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:33 smithi194 ceph-mon[32836]: purged_snaps scrub ok 2024-04-24T08:04:34.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:33 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:34.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:33 smithi194 ceph-mon[32836]: osd.6 [v2:172.21.15.28:6818/1356466279,v1:172.21.15.28:6819/1356466279] boot 2024-04-24T08:04:34.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:33 smithi194 ceph-mon[32836]: osdmap e46: 7 total, 7 up, 7 in 2024-04-24T08:04:34.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:33 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:04:34.818 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi117:vg_nvme/lv_2 2024-04-24T08:04:35.058 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:04:35.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:34 smithi117 ceph-mon[32438]: osdmap e47: 7 total, 7 up, 7 in 2024-04-24T08:04:35.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:34 smithi117 ceph-mon[32438]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:35.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:34 smithi194 ceph-mon[32836]: osdmap e47: 7 total, 7 up, 7 in 2024-04-24T08:04:35.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:34 smithi194 ceph-mon[32836]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:35.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:34 smithi028 ceph-mon[27786]: osdmap e47: 7 total, 7 up, 7 in 2024-04-24T08:04:35.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:34 smithi028 ceph-mon[27786]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:36.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.735+0000 7fe197e3b700 1 -- 172.21.15.117:0/2721850349 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe190101a50 msgr2=0x7fe190101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:04:36.739 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.735+0000 7fe197e3b700 1 --2- 172.21.15.117:0/2721850349 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe190101a50 0x7fe190101e70 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0099c0 tx=0x7fe18c009cd0 comp rx=0 tx=0).stop 2024-04-24T08:04:36.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.735+0000 7fe197e3b700 1 -- 172.21.15.117:0/2721850349 shutdown_connections 2024-04-24T08:04:36.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.735+0000 7fe197e3b700 1 --2- 172.21.15.117:0/2721850349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1901035f0 0x7fe190109ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:36.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.735+0000 7fe197e3b700 1 --2- 172.21.15.117:0/2721850349 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe190101a50 0x7fe190101e70 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:36.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.735+0000 7fe197e3b700 1 --2- 172.21.15.117:0/2721850349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe190102c50 0x7fe1901030b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:36.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.735+0000 7fe197e3b700 1 -- 172.21.15.117:0/2721850349 >> 172.21.15.117:0/2721850349 conn(0x7fe1900fcff0 msgr2=0x7fe1900ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:04:36.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.736+0000 7fe197e3b700 1 -- 172.21.15.117:0/2721850349 shutdown_connections 2024-04-24T08:04:36.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.736+0000 7fe197e3b700 1 -- 172.21.15.117:0/2721850349 wait complete. 2024-04-24T08:04:36.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.736+0000 7fe197e3b700 1 Processor -- start 2024-04-24T08:04:36.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.736+0000 7fe197e3b700 1 -- start start 2024-04-24T08:04:36.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.736+0000 7fe197e3b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe190101a50 0x7fe190197d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:04:36.740 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe197e3b700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe190102c50 0x7fe190198240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:04:36.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe197e3b700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1901035f0 0x7fe19019f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:04:36.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe197e3b700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe19010d690 con 0x7fe1901035f0 2024-04-24T08:04:36.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe197e3b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe19010d510 con 0x7fe190101a50 2024-04-24T08:04:36.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe197e3b700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe19010d810 con 0x7fe190102c50 2024-04-24T08:04:36.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe195bd7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe190101a50 0x7fe190197d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:04:36.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe195bd7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe190101a50 0x7fe190197d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:50234/0 (socket says 172.21.15.117:50234) 2024-04-24T08:04:36.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe195bd7700 1 -- 172.21.15.117:0/3936602709 learned_addr learned my addr 172.21.15.117:0/3936602709 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-04-24T08:04:36.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe1953d6700 1 --2- 172.21.15.117:0/3936602709 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe190102c50 0x7fe190198240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:04:36.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe195bd7700 1 -- 172.21.15.117:0/3936602709 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe190102c50 msgr2=0x7fe190198240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:04:36.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe195bd7700 1 --2- 172.21.15.117:0/3936602709 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe190102c50 0x7fe190198240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:36.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe195bd7700 1 -- 172.21.15.117:0/3936602709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1901035f0 msgr2=0x7fe19019f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:04:36.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe195bd7700 1 --2- 172.21.15.117:0/3936602709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1901035f0 0x7fe19019f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:36.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe195bd7700 1 -- 172.21.15.117:0/3936602709 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe19019f950 con 0x7fe190101a50 2024-04-24T08:04:36.741 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe1953d6700 1 --2- 172.21.15.117:0/3936602709 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe190102c50 0x7fe190198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:04:36.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe195bd7700 1 --2- 172.21.15.117:0/3936602709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe190101a50 0x7fe190197d00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe18c003890 tx=0x7fe18c009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:04:36.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe186ffd700 1 -- 172.21.15.117:0/3936602709 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c027070 con 0x7fe190101a50 2024-04-24T08:04:36.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe197e3b700 1 -- 172.21.15.117:0/3936602709 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe19019fbe0 con 0x7fe190101a50 2024-04-24T08:04:36.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe186ffd700 1 -- 172.21.15.117:0/3936602709 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe18c004150 con 0x7fe190101a50 2024-04-24T08:04:36.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe186ffd700 1 -- 172.21.15.117:0/3936602709 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18c022dc0 con 0x7fe190101a50 2024-04-24T08:04:36.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.737+0000 7fe197e3b700 1 -- 172.21.15.117:0/3936602709 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1901a0110 con 0x7fe190101a50 2024-04-24T08:04:36.742 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.738+0000 7fe197e3b700 1 -- 172.21.15.117:0/3936602709 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1900697e0 con 0x7fe190101a50 2024-04-24T08:04:36.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.739+0000 7fe186ffd700 1 -- 172.21.15.117:0/3936602709 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fe18c02a070 con 0x7fe190101a50 2024-04-24T08:04:36.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.740+0000 7fe186ffd700 1 --2- 172.21.15.117:0/3936602709 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe17c0a3cc0 0x7fe17c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:04:36.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.740+0000 7fe186ffd700 1 -- 172.21.15.117:0/3936602709 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fe18c0d03e0 con 0x7fe190101a50 2024-04-24T08:04:36.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.740+0000 7fe1953d6700 1 --2- 172.21.15.117:0/3936602709 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe17c0a3cc0 0x7fe17c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:04:36.743 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.741+0000 7fe1953d6700 1 --2- 172.21.15.117:0/3936602709 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe17c0a3cc0 0x7fe17c0a6180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe180005d90 tx=0x7fe180005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:04:36.745 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.745+0000 7fe186ffd700 1 -- 172.21.15.117:0/3936602709 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe18c013070 con 0x7fe190101a50 2024-04-24T08:04:36.953 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:36.952+0000 7fe197e3b700 1 -- 172.21.15.117:0/3936602709 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fe190062f30 con 0x7fe17c0a3cc0 2024-04-24T08:04:37.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:37 smithi117 ceph-mon[32438]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:37.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:37 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:04:37.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:37 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:04:37.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:37 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:37.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:37 smithi194 ceph-mon[32836]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:37.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:37 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:04:37.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:37 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:04:37.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:37 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:37.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:37 smithi028 ceph-mon[27786]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:37.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:37 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:04:37.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:37 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:04:37.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:37 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:38.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:38 smithi028 ceph-mon[27786]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:04:38.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:38 smithi028 ceph-mon[27786]: Detected new or changed devices on smithi028 2024-04-24T08:04:38.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:38 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:38.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:38 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:38.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:38 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:04:38.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:38 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:04:38.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:38 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:04:38.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:38 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi028 to 5698M 2024-04-24T08:04:38.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:38 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:38.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:38 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:38.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:38 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:04:38.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:38 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:38.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:38 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:04:38.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:38 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:38.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:38 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:04:38.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:38 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:38.591 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:38 smithi117 ceph-mon[32438]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:04:38.591 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:38 smithi117 ceph-mon[32438]: Detected new or changed devices on smithi028 2024-04-24T08:04:38.591 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:38 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:38.592 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:38 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:38.592 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:38 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:04:38.592 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:38 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:04:38.592 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:38 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:04:38.592 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:38 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi028 to 5698M 2024-04-24T08:04:38.592 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:38 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:38.592 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:38 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:38.592 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:38 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:04:38.592 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:38 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:38.592 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:38 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:04:38.592 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:38 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:38.592 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:38 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:04:38.592 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:38 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:38.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:38 smithi194 ceph-mon[32836]: from='client.24284 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:04:38.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:38 smithi194 ceph-mon[32836]: Detected new or changed devices on smithi028 2024-04-24T08:04:38.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:38 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:38.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:38 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:38.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:38 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:04:38.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:38 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:04:38.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:38 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:04:38.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:38 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi028 to 5698M 2024-04-24T08:04:38.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:38 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:38.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:38 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:38.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:38 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:04:38.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:38 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:38.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:38 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:04:38.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:38 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:38.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:38 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:04:38.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:38 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:39.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:39 smithi117 ceph-mon[32438]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:39.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:39 smithi194 ceph-mon[32836]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:40.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:39 smithi028 ceph-mon[27786]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:40.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:40 smithi028 ceph-mon[27786]: from='client.? 172.21.15.117:0/4274700157' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "657dc8f6-2eae-4b01-b57c-9bd7fbc5c541"}]: dispatch 2024-04-24T08:04:40.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:40 smithi028 ceph-mon[27786]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "657dc8f6-2eae-4b01-b57c-9bd7fbc5c541"}]: dispatch 2024-04-24T08:04:40.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:40 smithi028 ceph-mon[27786]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "657dc8f6-2eae-4b01-b57c-9bd7fbc5c541"}]': finished 2024-04-24T08:04:40.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:40 smithi028 ceph-mon[27786]: osdmap e48: 8 total, 7 up, 8 in 2024-04-24T08:04:40.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:40 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:04:40.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:40 smithi028 ceph-mon[27786]: from='client.? 172.21.15.117:0/3343938859' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:04:40.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:40 smithi117 ceph-mon[32438]: from='client.? 172.21.15.117:0/4274700157' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "657dc8f6-2eae-4b01-b57c-9bd7fbc5c541"}]: dispatch 2024-04-24T08:04:40.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:40 smithi117 ceph-mon[32438]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "657dc8f6-2eae-4b01-b57c-9bd7fbc5c541"}]: dispatch 2024-04-24T08:04:40.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:40 smithi117 ceph-mon[32438]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "657dc8f6-2eae-4b01-b57c-9bd7fbc5c541"}]': finished 2024-04-24T08:04:40.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:40 smithi117 ceph-mon[32438]: osdmap e48: 8 total, 7 up, 8 in 2024-04-24T08:04:40.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:40 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:04:40.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:40 smithi117 ceph-mon[32438]: from='client.? 172.21.15.117:0/3343938859' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:04:40.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:40 smithi194 ceph-mon[32836]: from='client.? 172.21.15.117:0/4274700157' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "657dc8f6-2eae-4b01-b57c-9bd7fbc5c541"}]: dispatch 2024-04-24T08:04:40.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:40 smithi194 ceph-mon[32836]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "657dc8f6-2eae-4b01-b57c-9bd7fbc5c541"}]: dispatch 2024-04-24T08:04:40.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:40 smithi194 ceph-mon[32836]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "657dc8f6-2eae-4b01-b57c-9bd7fbc5c541"}]': finished 2024-04-24T08:04:40.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:40 smithi194 ceph-mon[32836]: osdmap e48: 8 total, 7 up, 8 in 2024-04-24T08:04:40.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:40 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:04:40.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:40 smithi194 ceph-mon[32836]: from='client.? 172.21.15.117:0/3343938859' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:04:42.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:41 smithi028 ceph-mon[27786]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:42.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:41 smithi117 ceph-mon[32438]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:42.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:41 smithi194 ceph-mon[32836]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:44.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:43 smithi028 ceph-mon[27786]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:44.100 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:43 smithi117 ceph-mon[32438]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:44.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:43 smithi194 ceph-mon[32836]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:46.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:45 smithi028 ceph-mon[27786]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:46.057 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:45 smithi117 ceph-mon[32438]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:46.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:45 smithi194 ceph-mon[32836]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:48.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:47 smithi028 ceph-mon[27786]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:48.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:47 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T08:04:48.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:47 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:48.114 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:47 smithi117 ceph-mon[32438]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:48.114 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:47 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T08:04:48.114 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:47 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:48.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:47 smithi194 ceph-mon[32836]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:48.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:47 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-04-24T08:04:48.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:47 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:04:49.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:48 smithi028 ceph-mon[27786]: Deploying daemon osd.7 on smithi117 2024-04-24T08:04:49.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:48 smithi117 ceph-mon[32438]: Deploying daemon osd.7 on smithi117 2024-04-24T08:04:49.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:48 smithi194 ceph-mon[32836]: Deploying daemon osd.7 on smithi117 2024-04-24T08:04:50.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:49 smithi028 ceph-mon[27786]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:50.112 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:49 smithi117 ceph-mon[32438]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:50.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:49 smithi194 ceph-mon[32836]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:52.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:51 smithi028 ceph-mon[27786]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:52.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:51 smithi117 ceph-mon[32438]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:52.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:51 smithi194 ceph-mon[32836]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:54.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:53 smithi028 ceph-mon[27786]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:54.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:53 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:54.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:53 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:04:54.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:53 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:54.079 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:53 smithi117 ceph-mon[32438]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:54.079 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:54.079 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:04:54.079 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:54.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:53 smithi194 ceph-mon[32836]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:54.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:53 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:54.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:53 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:04:54.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:53 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:56.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:55 smithi028 ceph-mon[27786]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:56.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:55 smithi117 ceph-mon[32438]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:56.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:55 smithi194 ceph-mon[32836]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:56.947 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 7 on host 'smithi117' 2024-04-24T08:04:56.947 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:56.945+0000 7fe186ffd700 1 -- 172.21.15.117:0/3936602709 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fe190062f30 con 0x7fe17c0a3cc0 2024-04-24T08:04:56.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:56.948+0000 7fe197e3b700 1 -- 172.21.15.117:0/3936602709 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe17c0a3cc0 msgr2=0x7fe17c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:04:56.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:56.948+0000 7fe197e3b700 1 --2- 172.21.15.117:0/3936602709 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe17c0a3cc0 0x7fe17c0a6180 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fe180005d90 tx=0x7fe180005d20 comp rx=0 tx=0).stop 2024-04-24T08:04:56.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:56.948+0000 7fe197e3b700 1 -- 172.21.15.117:0/3936602709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe190101a50 msgr2=0x7fe190197d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:04:56.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:56.948+0000 7fe197e3b700 1 --2- 172.21.15.117:0/3936602709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe190101a50 0x7fe190197d00 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fe18c003890 tx=0x7fe18c009580 comp rx=0 tx=0).stop 2024-04-24T08:04:56.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:56.948+0000 7fe197e3b700 1 -- 172.21.15.117:0/3936602709 shutdown_connections 2024-04-24T08:04:56.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:56.948+0000 7fe197e3b700 1 --2- 172.21.15.117:0/3936602709 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe17c0a3cc0 0x7fe17c0a6180 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:56.949 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:56.948+0000 7fe197e3b700 1 --2- 172.21.15.117:0/3936602709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1901035f0 0x7fe19019f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:56.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:56.948+0000 7fe197e3b700 1 --2- 172.21.15.117:0/3936602709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe190101a50 0x7fe190197d00 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:56.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:56.948+0000 7fe197e3b700 1 --2- 172.21.15.117:0/3936602709 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe190102c50 0x7fe190198240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:04:56.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:56.948+0000 7fe197e3b700 1 -- 172.21.15.117:0/3936602709 >> 172.21.15.117:0/3936602709 conn(0x7fe1900fcff0 msgr2=0x7fe190105fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:04:56.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:56.948+0000 7fe197e3b700 1 -- 172.21.15.117:0/3936602709 shutdown_connections 2024-04-24T08:04:56.950 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:04:56.948+0000 7fe197e3b700 1 -- 172.21.15.117:0/3936602709 wait complete. 2024-04-24T08:04:57.677 DEBUG:teuthology.orchestra.run.smithi117:osd.7> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.7.service 2024-04-24T08:04:57.680 INFO:tasks.cephadm:Deploying osd.8 on smithi194 with /dev/vg_nvme/lv_2... 2024-04-24T08:04:57.680 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-04-24T08:04:57.928 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:04:58.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:57 smithi194 ceph-mon[32836]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:58.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:57 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:58.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:57 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:58.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:57 smithi194 ceph-mon[32836]: from='osd.7 [v2:172.21.15.117:6816/1472212690,v1:172.21.15.117:6817/1472212690]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T08:04:58.051 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:57 smithi194 ceph-mon[32836]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T08:04:58.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:57 smithi117 ceph-mon[32438]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:58.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:57 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:58.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:57 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:58.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:57 smithi117 ceph-mon[32438]: from='osd.7 [v2:172.21.15.117:6816/1472212690,v1:172.21.15.117:6817/1472212690]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T08:04:58.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:57 smithi117 ceph-mon[32438]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T08:04:58.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:57 smithi028 ceph-mon[27786]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:58.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:57 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:58.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:57 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:58.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:57 smithi028 ceph-mon[27786]: from='osd.7 [v2:172.21.15.117:6816/1472212690,v1:172.21.15.117:6817/1472212690]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T08:04:58.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:57 smithi028 ceph-mon[27786]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-04-24T08:04:59.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:59.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:59.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:58 smithi194 ceph-mon[32836]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-24T08:04:59.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:58 smithi194 ceph-mon[32836]: osdmap e49: 8 total, 7 up, 8 in 2024-04-24T08:04:59.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:04:59.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:58 smithi194 ceph-mon[32836]: from='osd.7 [v2:172.21.15.117:6816/1472212690,v1:172.21.15.117:6817/1472212690]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:04:59.067 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:58 smithi194 ceph-mon[32836]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:04:59.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:58 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:59.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:58 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:59.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:58 smithi117 ceph-mon[32438]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-24T08:04:59.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:58 smithi117 ceph-mon[32438]: osdmap e49: 8 total, 7 up, 8 in 2024-04-24T08:04:59.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:58 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:04:59.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:58 smithi117 ceph-mon[32438]: from='osd.7 [v2:172.21.15.117:6816/1472212690,v1:172.21.15.117:6817/1472212690]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:04:59.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:58 smithi117 ceph-mon[32438]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:04:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:04:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:58 smithi028 ceph-mon[27786]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-04-24T08:04:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:58 smithi028 ceph-mon[27786]: osdmap e49: 8 total, 7 up, 8 in 2024-04-24T08:04:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:04:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:58 smithi028 ceph-mon[27786]: from='osd.7 [v2:172.21.15.117:6816/1472212690,v1:172.21.15.117:6817/1472212690]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:04:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:58 smithi028 ceph-mon[27786]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:04:59.649 INFO:journalctl@ceph.osd.7.smithi117.stdout:Apr 24 08:04:59 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-7[51090]: 2024-04-24T08:04:59.386+0000 7f1e7bbab700 -1 osd.7 0 waiting for initial osdmap 2024-04-24T08:04:59.649 INFO:journalctl@ceph.osd.7.smithi117.stdout:Apr 24 08:04:59 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-7[51090]: 2024-04-24T08:04:59.396+0000 7f1e73996700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:04:59.869 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-04-24T08:04:59.869 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-04-24T08:04:59.870 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-04-24T08:04:59.870 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-04-24T08:04:59.870 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115193 s, 910 MB/s 2024-04-24T08:04:59.870 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-04-24T08:04:59.968 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:59 smithi117 ceph-mon[32438]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:04:59.968 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:59 smithi117 ceph-mon[32438]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-04-24T08:04:59.968 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:59 smithi117 ceph-mon[32438]: osdmap e50: 8 total, 7 up, 8 in 2024-04-24T08:04:59.968 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:59 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:04:59.968 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:59 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:04:59.968 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:04:59 smithi117 ceph-mon[32438]: from='osd.7 ' entity='osd.7' 2024-04-24T08:05:00.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:59 smithi194 ceph-mon[32836]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:05:00.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:59 smithi194 ceph-mon[32836]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-04-24T08:05:00.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:59 smithi194 ceph-mon[32836]: osdmap e50: 8 total, 7 up, 8 in 2024-04-24T08:05:00.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:05:00.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:05:00.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:04:59 smithi194 ceph-mon[32836]: from='osd.7 ' entity='osd.7' 2024-04-24T08:05:00.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:59 smithi028 ceph-mon[27786]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:05:00.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:59 smithi028 ceph-mon[27786]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-04-24T08:05:00.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:59 smithi028 ceph-mon[27786]: osdmap e50: 8 total, 7 up, 8 in 2024-04-24T08:05:00.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:05:00.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:05:00.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:04:59 smithi028 ceph-mon[27786]: from='osd.7 ' entity='osd.7' 2024-04-24T08:05:00.562 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_2 2024-04-24T08:05:00.798 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:05:01.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:00 smithi117 ceph-mon[32438]: purged_snaps scrub starts 2024-04-24T08:05:01.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:00 smithi117 ceph-mon[32438]: purged_snaps scrub ok 2024-04-24T08:05:01.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:05:01.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:00 smithi117 ceph-mon[32438]: osd.7 [v2:172.21.15.117:6816/1472212690,v1:172.21.15.117:6817/1472212690] boot 2024-04-24T08:05:01.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:00 smithi117 ceph-mon[32438]: osdmap e51: 8 total, 8 up, 8 in 2024-04-24T08:05:01.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:05:01.130 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:00 smithi194 ceph-mon[32836]: purged_snaps scrub starts 2024-04-24T08:05:01.130 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:00 smithi194 ceph-mon[32836]: purged_snaps scrub ok 2024-04-24T08:05:01.130 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:00 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:05:01.130 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:00 smithi194 ceph-mon[32836]: osd.7 [v2:172.21.15.117:6816/1472212690,v1:172.21.15.117:6817/1472212690] boot 2024-04-24T08:05:01.130 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:00 smithi194 ceph-mon[32836]: osdmap e51: 8 total, 8 up, 8 in 2024-04-24T08:05:01.130 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:00 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:05:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:00 smithi028 ceph-mon[27786]: purged_snaps scrub starts 2024-04-24T08:05:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:00 smithi028 ceph-mon[27786]: purged_snaps scrub ok 2024-04-24T08:05:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:00 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:05:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:00 smithi028 ceph-mon[27786]: osd.7 [v2:172.21.15.117:6816/1472212690,v1:172.21.15.117:6817/1472212690] boot 2024-04-24T08:05:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:00 smithi028 ceph-mon[27786]: osdmap e51: 8 total, 8 up, 8 in 2024-04-24T08:05:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:00 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:05:02.097 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:01 smithi194 ceph-mon[32836]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:05:02.097 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:01 smithi194 ceph-mon[32836]: Detected new or changed devices on smithi117 2024-04-24T08:05:02.097 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:02.097 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:02.097 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:02.097 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:02.097 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:02.098 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:01 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi117 to 5696M 2024-04-24T08:05:02.098 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:02.098 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:02.098 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:05:02.098 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:02.098 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:01 smithi194 ceph-mon[32836]: osdmap e52: 8 total, 8 up, 8 in 2024-04-24T08:05:02.098 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:05:02.098 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:02.098 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:05:02.098 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:02.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:01 smithi117 ceph-mon[32438]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:05:02.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:01 smithi117 ceph-mon[32438]: Detected new or changed devices on smithi117 2024-04-24T08:05:02.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:02.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:02.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:02.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:02.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:02.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:01 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi117 to 5696M 2024-04-24T08:05:02.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:02.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:02.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:05:02.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:02.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:01 smithi117 ceph-mon[32438]: osdmap e52: 8 total, 8 up, 8 in 2024-04-24T08:05:02.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:05:02.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:02.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:05:02.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:02.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:01 smithi028 ceph-mon[27786]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-04-24T08:05:02.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:01 smithi028 ceph-mon[27786]: Detected new or changed devices on smithi117 2024-04-24T08:05:02.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:02.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:02.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:02.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:02.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:02.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:01 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi117 to 5696M 2024-04-24T08:05:02.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:02.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:02.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:05:02.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:02.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:01 smithi028 ceph-mon[27786]: osdmap e52: 8 total, 8 up, 8 in 2024-04-24T08:05:02.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:05:02.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:02.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:05:02.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:02.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.436+0000 7ffad0f57700 1 -- 172.21.15.194:0/2926501079 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffacc1035e0 msgr2=0x7ffacc109e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:05:02.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.436+0000 7ffad0f57700 1 --2- 172.21.15.194:0/2926501079 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffacc1035e0 0x7ffacc109e90 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ffac400b0a0 tx=0x7ffac400b3b0 comp rx=0 tx=0).stop 2024-04-24T08:05:02.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.436+0000 7ffad0f57700 1 -- 172.21.15.194:0/2926501079 shutdown_connections 2024-04-24T08:05:02.442 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.436+0000 7ffad0f57700 1 --2- 172.21.15.194:0/2926501079 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffacc1035e0 0x7ffacc109e90 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:02.442 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.436+0000 7ffad0f57700 1 --2- 172.21.15.194:0/2926501079 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffacc101a40 0x7ffacc101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:02.442 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.436+0000 7ffad0f57700 1 --2- 172.21.15.194:0/2926501079 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffacc102c40 0x7ffacc1030a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:02.442 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.436+0000 7ffad0f57700 1 -- 172.21.15.194:0/2926501079 >> 172.21.15.194:0/2926501079 conn(0x7ffacc0fcfe0 msgr2=0x7ffacc0ff420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:05:02.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.436+0000 7ffad0f57700 1 -- 172.21.15.194:0/2926501079 shutdown_connections 2024-04-24T08:05:02.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.436+0000 7ffad0f57700 1 -- 172.21.15.194:0/2926501079 wait complete. 2024-04-24T08:05:02.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.437+0000 7ffad0f57700 1 Processor -- start 2024-04-24T08:05:02.443 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.437+0000 7ffad0f57700 1 -- start start 2024-04-24T08:05:02.444 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.437+0000 7ffad0f57700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffacc101a40 0x7ffacc198d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:05:02.444 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.437+0000 7ffad0f57700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffacc102c40 0x7ffacc1992d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:05:02.444 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.437+0000 7ffad0f57700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffacc1035e0 0x7ffacc1a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:05:02.444 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.437+0000 7ffad0f57700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc10e7a0 con 0x7ffacc1035e0 2024-04-24T08:05:02.444 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.437+0000 7ffad0f57700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc10e620 con 0x7ffacc102c40 2024-04-24T08:05:02.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.437+0000 7ffad0f57700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ffacc10e920 con 0x7ffacc101a40 2024-04-24T08:05:02.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.438+0000 7ffaca59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffacc101a40 0x7ffacc198d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:05:02.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.438+0000 7ffaca59c700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffacc101a40 0x7ffacc198d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:34564/0 (socket says 172.21.15.194:34564) 2024-04-24T08:05:02.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.438+0000 7ffaca59c700 1 -- 172.21.15.194:0/789903660 learned_addr learned my addr 172.21.15.194:0/789903660 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-24T08:05:02.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.438+0000 7ffac9d9b700 1 --2- 172.21.15.194:0/789903660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffacc102c40 0x7ffacc1992d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:05:02.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.438+0000 7ffaca59c700 1 -- 172.21.15.194:0/789903660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffacc102c40 msgr2=0x7ffacc1992d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:05:02.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.438+0000 7ffaca59c700 1 --2- 172.21.15.194:0/789903660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffacc102c40 0x7ffacc1992d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:02.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.438+0000 7ffaca59c700 1 -- 172.21.15.194:0/789903660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffacc1035e0 msgr2=0x7ffacc1a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:05:02.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.438+0000 7ffaca59c700 1 --2- 172.21.15.194:0/789903660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffacc1035e0 0x7ffacc1a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:02.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.438+0000 7ffaca59c700 1 -- 172.21.15.194:0/789903660 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffacc1a09e0 con 0x7ffacc101a40 2024-04-24T08:05:02.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.438+0000 7ffaca59c700 1 --2- 172.21.15.194:0/789903660 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffacc101a40 0x7ffacc198d90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ffac000c970 tx=0x7ffac000cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:05:02.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.438+0000 7ffabb7fe700 1 -- 172.21.15.194:0/789903660 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac0004220 con 0x7ffacc101a40 2024-04-24T08:05:02.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.438+0000 7ffabb7fe700 1 -- 172.21.15.194:0/789903660 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffac0007d80 con 0x7ffacc101a40 2024-04-24T08:05:02.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.438+0000 7ffad0f57700 1 -- 172.21.15.194:0/789903660 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffacc1a0cd0 con 0x7ffacc101a40 2024-04-24T08:05:02.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.438+0000 7ffabb7fe700 1 -- 172.21.15.194:0/789903660 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffac0003e50 con 0x7ffacc101a40 2024-04-24T08:05:02.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.438+0000 7ffad0f57700 1 -- 172.21.15.194:0/789903660 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffacc1a11c0 con 0x7ffacc101a40 2024-04-24T08:05:02.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.439+0000 7ffad0f57700 1 -- 172.21.15.194:0/789903660 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffacc069820 con 0x7ffacc101a40 2024-04-24T08:05:02.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.441+0000 7ffabb7fe700 1 -- 172.21.15.194:0/789903660 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ffac00078a0 con 0x7ffacc101a40 2024-04-24T08:05:02.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.442+0000 7ffabb7fe700 1 --2- 172.21.15.194:0/789903660 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffab40a3de0 0x7ffab40a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:05:02.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.442+0000 7ffabb7fe700 1 -- 172.21.15.194:0/789903660 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7ffac00cf920 con 0x7ffacc101a40 2024-04-24T08:05:02.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.442+0000 7ffac9d9b700 1 --2- 172.21.15.194:0/789903660 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffab40a3de0 0x7ffab40a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:05:02.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.443+0000 7ffac9d9b700 1 --2- 172.21.15.194:0/789903660 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffab40a3de0 0x7ffab40a62a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffabc006fd0 tx=0x7ffabc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:05:02.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.446+0000 7ffabb7fe700 1 -- 172.21.15.194:0/789903660 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffac0014030 con 0x7ffacc101a40 2024-04-24T08:05:02.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:02.656+0000 7ffad0f57700 1 -- 172.21.15.194:0/789903660 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ffacc062f30 con 0x7ffab40a3de0 2024-04-24T08:05:03.625 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:03 smithi194 ceph-mon[32836]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:03.625 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:03 smithi194 ceph-mon[32836]: osdmap e53: 8 total, 8 up, 8 in 2024-04-24T08:05:03.625 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:03 smithi194 ceph-mon[32836]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:05:03.625 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:03 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:05:03.625 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:03 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:05:03.625 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:03 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:03 smithi028 ceph-mon[27786]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:03 smithi028 ceph-mon[27786]: osdmap e53: 8 total, 8 up, 8 in 2024-04-24T08:05:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:03 smithi028 ceph-mon[27786]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:05:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:03 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:05:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:03 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:05:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:03 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:03 smithi117 ceph-mon[32438]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:03 smithi117 ceph-mon[32438]: osdmap e53: 8 total, 8 up, 8 in 2024-04-24T08:05:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:03 smithi117 ceph-mon[32438]: from='client.24325 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:05:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:03 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:05:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:03 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:05:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:03 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:04.539 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:04 smithi194 ceph-mon[32836]: osdmap e54: 8 total, 8 up, 8 in 2024-04-24T08:05:04.540 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:04 smithi194 ceph-mon[32836]: osdmap e55: 8 total, 8 up, 8 in 2024-04-24T08:05:04.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:04 smithi028 ceph-mon[27786]: osdmap e54: 8 total, 8 up, 8 in 2024-04-24T08:05:04.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:04 smithi028 ceph-mon[27786]: osdmap e55: 8 total, 8 up, 8 in 2024-04-24T08:05:04.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:04 smithi117 ceph-mon[32438]: osdmap e54: 8 total, 8 up, 8 in 2024-04-24T08:05:04.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:04 smithi117 ceph-mon[32438]: osdmap e55: 8 total, 8 up, 8 in 2024-04-24T08:05:05.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:05 smithi194 ceph-mon[32836]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:05.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:05 smithi194 ceph-mon[32836]: from='client.? 172.21.15.194:0/3772683359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7bd54baf-23e4-4f9d-8c03-79bc64308158"}]: dispatch 2024-04-24T08:05:05.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:05 smithi194 ceph-mon[32836]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7bd54baf-23e4-4f9d-8c03-79bc64308158"}]: dispatch 2024-04-24T08:05:05.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:05 smithi194 ceph-mon[32836]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7bd54baf-23e4-4f9d-8c03-79bc64308158"}]': finished 2024-04-24T08:05:05.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:05 smithi194 ceph-mon[32836]: osdmap e56: 9 total, 8 up, 9 in 2024-04-24T08:05:05.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:05 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:05.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:05 smithi028 ceph-mon[27786]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:05.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:05 smithi028 ceph-mon[27786]: from='client.? 172.21.15.194:0/3772683359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7bd54baf-23e4-4f9d-8c03-79bc64308158"}]: dispatch 2024-04-24T08:05:05.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:05 smithi028 ceph-mon[27786]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7bd54baf-23e4-4f9d-8c03-79bc64308158"}]: dispatch 2024-04-24T08:05:05.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:05 smithi028 ceph-mon[27786]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7bd54baf-23e4-4f9d-8c03-79bc64308158"}]': finished 2024-04-24T08:05:05.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:05 smithi028 ceph-mon[27786]: osdmap e56: 9 total, 8 up, 9 in 2024-04-24T08:05:05.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:05 smithi117 ceph-mon[32438]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:05 smithi117 ceph-mon[32438]: from='client.? 172.21.15.194:0/3772683359' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7bd54baf-23e4-4f9d-8c03-79bc64308158"}]: dispatch 2024-04-24T08:05:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:05 smithi117 ceph-mon[32438]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7bd54baf-23e4-4f9d-8c03-79bc64308158"}]: dispatch 2024-04-24T08:05:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:05 smithi117 ceph-mon[32438]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7bd54baf-23e4-4f9d-8c03-79bc64308158"}]': finished 2024-04-24T08:05:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:05 smithi117 ceph-mon[32438]: osdmap e56: 9 total, 8 up, 9 in 2024-04-24T08:05:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:05 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:06.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:06 smithi194 ceph-mon[32836]: from='client.? 172.21.15.194:0/1774705482' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:05:06.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:06 smithi028 ceph-mon[27786]: from='client.? 172.21.15.194:0/1774705482' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:05:06.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:06 smithi117 ceph-mon[32438]: from='client.? 172.21.15.194:0/1774705482' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:05:07.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:07 smithi194 ceph-mon[32836]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:07.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:07 smithi028 ceph-mon[27786]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:07.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:07 smithi117 ceph-mon[32438]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:09.694 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:09 smithi117 ceph-mon[32438]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:09.737 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:09 smithi194 ceph-mon[32836]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:09.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:09 smithi028 ceph-mon[27786]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:11.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:11 smithi194 ceph-mon[32836]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:11.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:11 smithi028 ceph-mon[27786]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:11.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:11 smithi117 ceph-mon[32438]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:13.705 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:13 smithi194 ceph-mon[32836]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:13.705 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:13 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-24T08:05:13.705 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:13 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:13.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:13 smithi028 ceph-mon[27786]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:13.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:13 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-24T08:05:13.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:13 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:13.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:13 smithi117 ceph-mon[32438]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:13.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:13 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-04-24T08:05:13.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:13 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:14.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:14 smithi028 ceph-mon[27786]: Deploying daemon osd.8 on smithi194 2024-04-24T08:05:14.735 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:14 smithi194 ceph-mon[32836]: Deploying daemon osd.8 on smithi194 2024-04-24T08:05:14.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:14 smithi117 ceph-mon[32438]: Deploying daemon osd.8 on smithi194 2024-04-24T08:05:15.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:15 smithi028 ceph-mon[27786]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:15.866 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:15 smithi194 ceph-mon[32836]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:15.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:15 smithi117 ceph-mon[32438]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:17.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:17 smithi194 ceph-mon[32836]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:17.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:17 smithi028 ceph-mon[27786]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:17.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:17 smithi117 ceph-mon[32438]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:20.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:19 smithi028 ceph-mon[27786]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:20.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:19 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:20.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:19 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:05:20.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:19 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:20.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:19 smithi117 ceph-mon[32438]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:20.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:19 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:20.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:19 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:05:20.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:19 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:20.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:19 smithi194 ceph-mon[32836]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:20.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:19 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:20.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:19 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:05:20.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:19 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:22.003 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:21 smithi194 ceph-mon[32836]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:22.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:21 smithi028 ceph-mon[27786]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:22.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:21 smithi117 ceph-mon[32438]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:22.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:22.745+0000 7ffabb7fe700 1 -- 172.21.15.194:0/789903660 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ffacc062f30 con 0x7ffab40a3de0 2024-04-24T08:05:22.746 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 8 on host 'smithi194' 2024-04-24T08:05:22.748 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:22.747+0000 7ffad0f57700 1 -- 172.21.15.194:0/789903660 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffab40a3de0 msgr2=0x7ffab40a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:05:22.748 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:22.747+0000 7ffad0f57700 1 --2- 172.21.15.194:0/789903660 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffab40a3de0 0x7ffab40a62a0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7ffabc006fd0 tx=0x7ffabc008040 comp rx=0 tx=0).stop 2024-04-24T08:05:22.748 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:22.747+0000 7ffad0f57700 1 -- 172.21.15.194:0/789903660 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffacc101a40 msgr2=0x7ffacc198d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:05:22.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:22.747+0000 7ffad0f57700 1 --2- 172.21.15.194:0/789903660 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffacc101a40 0x7ffacc198d90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ffac000c970 tx=0x7ffac000cd30 comp rx=0 tx=0).stop 2024-04-24T08:05:22.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:22.747+0000 7ffad0f57700 1 -- 172.21.15.194:0/789903660 shutdown_connections 2024-04-24T08:05:22.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:22.747+0000 7ffad0f57700 1 --2- 172.21.15.194:0/789903660 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffab40a3de0 0x7ffab40a62a0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:22.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:22.747+0000 7ffad0f57700 1 --2- 172.21.15.194:0/789903660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffacc1035e0 0x7ffacc1a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:22.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:22.747+0000 7ffad0f57700 1 --2- 172.21.15.194:0/789903660 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffacc101a40 0x7ffacc198d90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:22.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:22.747+0000 7ffad0f57700 1 --2- 172.21.15.194:0/789903660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffacc102c40 0x7ffacc1992d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:22.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:22.747+0000 7ffad0f57700 1 -- 172.21.15.194:0/789903660 >> 172.21.15.194:0/789903660 conn(0x7ffacc0fcfe0 msgr2=0x7ffacc105ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:05:22.749 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:22.747+0000 7ffad0f57700 1 -- 172.21.15.194:0/789903660 shutdown_connections 2024-04-24T08:05:22.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:05:22.747+0000 7ffad0f57700 1 -- 172.21.15.194:0/789903660 wait complete. 2024-04-24T08:05:22.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:22 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:22.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:22 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:22.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:22 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:22.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:22 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:05:22.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:22 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:22.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:22 smithi194 ceph-mon[32836]: from='osd.8 [v2:172.21.15.194:6816/301057688,v1:172.21.15.194:6817/301057688]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T08:05:22.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:22 smithi194 ceph-mon[32836]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T08:05:22.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:22 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:22.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:22 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:23.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:22 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:23.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:22 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:23.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:22 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:23.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:22 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:05:23.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:22 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:23.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:22 smithi028 ceph-mon[27786]: from='osd.8 [v2:172.21.15.194:6816/301057688,v1:172.21.15.194:6817/301057688]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T08:05:23.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:22 smithi028 ceph-mon[27786]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T08:05:23.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:22 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:23.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:22 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:23.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:22 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:23.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:22 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:23.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:22 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:23.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:22 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:05:23.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:22 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:23.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:22 smithi117 ceph-mon[32438]: from='osd.8 [v2:172.21.15.194:6816/301057688,v1:172.21.15.194:6817/301057688]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T08:05:23.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:22 smithi117 ceph-mon[32438]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-04-24T08:05:23.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:22 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:23.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:22 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:23.435 DEBUG:teuthology.orchestra.run.smithi194:osd.8> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.8.service 2024-04-24T08:05:23.438 INFO:tasks.cephadm:Deploying osd.9 on smithi028 with /dev/vg_nvme/lv_1... 2024-04-24T08:05:23.438 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-24T08:05:23.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:23 smithi028 ceph-mon[27786]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:23.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:23 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:05:23.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:23 smithi028 ceph-mon[27786]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-24T08:05:23.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:23 smithi028 ceph-mon[27786]: from='osd.8 [v2:172.21.15.194:6816/301057688,v1:172.21.15.194:6817/301057688]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:05:23.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:23 smithi028 ceph-mon[27786]: osdmap e57: 9 total, 8 up, 9 in 2024-04-24T08:05:23.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:23 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:23.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:23 smithi028 ceph-mon[27786]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:05:23.896 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:05:24.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:23 smithi117 ceph-mon[32438]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:24.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:23 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:05:24.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:23 smithi117 ceph-mon[32438]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-24T08:05:24.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:23 smithi117 ceph-mon[32438]: from='osd.8 [v2:172.21.15.194:6816/301057688,v1:172.21.15.194:6817/301057688]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:05:24.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:23 smithi117 ceph-mon[32438]: osdmap e57: 9 total, 8 up, 9 in 2024-04-24T08:05:24.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:23 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:24.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:23 smithi117 ceph-mon[32438]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:05:24.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:23 smithi194 ceph-mon[32836]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:24.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:23 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:05:24.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:23 smithi194 ceph-mon[32836]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-04-24T08:05:24.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:23 smithi194 ceph-mon[32836]: from='osd.8 [v2:172.21.15.194:6816/301057688,v1:172.21.15.194:6817/301057688]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:05:24.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:23 smithi194 ceph-mon[32836]: osdmap e57: 9 total, 8 up, 9 in 2024-04-24T08:05:24.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:23 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:24.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:23 smithi194 ceph-mon[32836]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:05:24.643 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 24 08:05:24 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-8[51672]: 2024-04-24T08:05:24.210+0000 7fce3d54c700 -1 osd.8 0 waiting for initial osdmap 2024-04-24T08:05:24.643 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 24 08:05:24 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-8[51672]: 2024-04-24T08:05:24.221+0000 7fce35337700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:05:24.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:24 smithi028 ceph-mon[27786]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-24T08:05:24.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:24 smithi028 ceph-mon[27786]: osdmap e58: 9 total, 8 up, 9 in 2024-04-24T08:05:24.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:24 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:24.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:24 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:24.884 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:24 smithi028 ceph-mon[27786]: osdmap e59: 9 total, 8 up, 9 in 2024-04-24T08:05:24.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:24 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:24.885 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:24 smithi028 ceph-mon[27786]: from='osd.8 ' entity='osd.8' 2024-04-24T08:05:24.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:24 smithi194 ceph-mon[32836]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-24T08:05:24.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:24 smithi194 ceph-mon[32836]: osdmap e58: 9 total, 8 up, 9 in 2024-04-24T08:05:24.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:24 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:24.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:24 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:24.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:24 smithi194 ceph-mon[32836]: osdmap e59: 9 total, 8 up, 9 in 2024-04-24T08:05:24.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:24 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:24.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:24 smithi194 ceph-mon[32836]: from='osd.8 ' entity='osd.8' 2024-04-24T08:05:25.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:24 smithi117 ceph-mon[32438]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-24T08:05:25.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:24 smithi117 ceph-mon[32438]: osdmap e58: 9 total, 8 up, 9 in 2024-04-24T08:05:25.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:24 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:25.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:24 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:25.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:24 smithi117 ceph-mon[32438]: osdmap e59: 9 total, 8 up, 9 in 2024-04-24T08:05:25.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:24 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:25.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:24 smithi117 ceph-mon[32438]: from='osd.8 ' entity='osd.8' 2024-04-24T08:05:25.897 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:25 smithi028 ceph-mon[27786]: purged_snaps scrub starts 2024-04-24T08:05:25.898 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:25 smithi028 ceph-mon[27786]: purged_snaps scrub ok 2024-04-24T08:05:25.898 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:25 smithi028 ceph-mon[27786]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:25.898 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:25 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:25.898 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:25 smithi028 ceph-mon[27786]: osd.8 [v2:172.21.15.194:6816/301057688,v1:172.21.15.194:6817/301057688] boot 2024-04-24T08:05:25.898 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:25 smithi028 ceph-mon[27786]: osdmap e60: 9 total, 9 up, 9 in 2024-04-24T08:05:25.898 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:25 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:26.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:25 smithi117 ceph-mon[32438]: purged_snaps scrub starts 2024-04-24T08:05:26.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:25 smithi117 ceph-mon[32438]: purged_snaps scrub ok 2024-04-24T08:05:26.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:25 smithi117 ceph-mon[32438]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:26.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:25 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:26.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:25 smithi117 ceph-mon[32438]: osd.8 [v2:172.21.15.194:6816/301057688,v1:172.21.15.194:6817/301057688] boot 2024-04-24T08:05:26.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:25 smithi117 ceph-mon[32438]: osdmap e60: 9 total, 9 up, 9 in 2024-04-24T08:05:26.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:25 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:26.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:25 smithi194 ceph-mon[32836]: purged_snaps scrub starts 2024-04-24T08:05:26.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:25 smithi194 ceph-mon[32836]: purged_snaps scrub ok 2024-04-24T08:05:26.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:25 smithi194 ceph-mon[32836]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-04-24T08:05:26.167 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:25 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:26.168 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:25 smithi194 ceph-mon[32836]: osd.8 [v2:172.21.15.194:6816/301057688,v1:172.21.15.194:6817/301057688] boot 2024-04-24T08:05:26.168 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:25 smithi194 ceph-mon[32836]: osdmap e60: 9 total, 9 up, 9 in 2024-04-24T08:05:26.168 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:25 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:05:26.960 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-24T08:05:26.960 INFO:teuthology.orchestra.run.smithi028.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-24T08:05:26.960 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10+0 records in 2024-04-24T08:05:26.960 INFO:teuthology.orchestra.run.smithi028.stderr:10+0 records out 2024-04-24T08:05:26.960 INFO:teuthology.orchestra.run.smithi028.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0561836 s, 187 MB/s 2024-04-24T08:05:26.960 INFO:teuthology.orchestra.run.smithi028.stderr:--> Zapping successful for: 2024-04-24T08:05:27.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:26 smithi117 ceph-mon[32438]: osdmap e61: 9 total, 9 up, 9 in 2024-04-24T08:05:27.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:26 smithi117 ceph-mon[32438]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:27.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:26 smithi117 ceph-mon[32438]: Detected new or changed devices on smithi194 2024-04-24T08:05:27.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:27.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:27.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:27.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:27.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:27.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:26 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi194 to 5696M 2024-04-24T08:05:27.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:27.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:27.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:05:27.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:27.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:26 smithi194 ceph-mon[32836]: osdmap e61: 9 total, 9 up, 9 in 2024-04-24T08:05:27.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:26 smithi194 ceph-mon[32836]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:27.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:26 smithi194 ceph-mon[32836]: Detected new or changed devices on smithi194 2024-04-24T08:05:27.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:27.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:27.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:27.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:27.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:27.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:26 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi194 to 5696M 2024-04-24T08:05:27.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:27.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:27.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:05:27.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:27.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:26 smithi028 ceph-mon[27786]: osdmap e61: 9 total, 9 up, 9 in 2024-04-24T08:05:27.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:26 smithi028 ceph-mon[27786]: pgmap v219: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:27.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:26 smithi028 ceph-mon[27786]: Detected new or changed devices on smithi194 2024-04-24T08:05:27.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:27.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:27.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:27.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:27.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:05:27.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:26 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi194 to 5696M 2024-04-24T08:05:27.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:27.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:27.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:05:27.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:27.742 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi028:vg_nvme/lv_1 2024-04-24T08:05:28.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:27 smithi028 ceph-mon[27786]: osdmap e62: 9 total, 9 up, 9 in 2024-04-24T08:05:28.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:27 smithi117 ceph-mon[32438]: osdmap e62: 9 total, 9 up, 9 in 2024-04-24T08:05:28.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:27 smithi194 ceph-mon[32836]: osdmap e62: 9 total, 9 up, 9 in 2024-04-24T08:05:28.220 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:05:29.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:28 smithi028 ceph-mon[27786]: osdmap e63: 9 total, 9 up, 9 in 2024-04-24T08:05:29.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:28 smithi028 ceph-mon[27786]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:29.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:28 smithi194 ceph-mon[32836]: osdmap e63: 9 total, 9 up, 9 in 2024-04-24T08:05:29.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:28 smithi194 ceph-mon[32836]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:29.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:28 smithi117 ceph-mon[32438]: osdmap e63: 9 total, 9 up, 9 in 2024-04-24T08:05:29.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:28 smithi117 ceph-mon[32438]: pgmap v222: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:29.995 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:29 smithi028 ceph-mon[27786]: osdmap e64: 9 total, 9 up, 9 in 2024-04-24T08:05:30.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:29 smithi194 ceph-mon[32836]: osdmap e64: 9 total, 9 up, 9 in 2024-04-24T08:05:30.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:29 smithi117 ceph-mon[32438]: osdmap e64: 9 total, 9 up, 9 in 2024-04-24T08:05:30.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.533+0000 7f5083400700 1 -- 172.21.15.28:0/4073139842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f507c106930 msgr2=0x7f507c10d1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:05:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.533+0000 7f5083400700 1 --2- 172.21.15.28:0/4073139842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f507c106930 0x7f507c10d1e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f507400b0a0 tx=0x7f507400b3b0 comp rx=0 tx=0).stop 2024-04-24T08:05:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.533+0000 7f5083400700 1 -- 172.21.15.28:0/4073139842 shutdown_connections 2024-04-24T08:05:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.533+0000 7f5083400700 1 --2- 172.21.15.28:0/4073139842 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f507c106930 0x7f507c10d1e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.533+0000 7f5083400700 1 --2- 172.21.15.28:0/4073139842 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f507c104d90 0x7f507c1051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.533+0000 7f5083400700 1 --2- 172.21.15.28:0/4073139842 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c105f90 0x7f507c1063f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.533+0000 7f5083400700 1 -- 172.21.15.28:0/4073139842 >> 172.21.15.28:0/4073139842 conn(0x7f507c100350 msgr2=0x7f507c102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:05:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.533+0000 7f5083400700 1 -- 172.21.15.28:0/4073139842 shutdown_connections 2024-04-24T08:05:30.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.533+0000 7f5083400700 1 -- 172.21.15.28:0/4073139842 wait complete. 2024-04-24T08:05:30.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.535+0000 7f5083400700 1 Processor -- start 2024-04-24T08:05:30.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.535+0000 7f5083400700 1 -- start start 2024-04-24T08:05:30.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.535+0000 7f5083400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f507c104d90 0x7f507c19af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:05:30.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.535+0000 7f5083400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c105f90 0x7f507c19b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:05:30.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.535+0000 7f5083400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f507c106930 0x7f507c1a24d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:05:30.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.535+0000 7f5083400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f507c1109b0 con 0x7f507c105f90 2024-04-24T08:05:30.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.535+0000 7f5083400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f507c110830 con 0x7f507c106930 2024-04-24T08:05:30.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.535+0000 7f5083400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f507c110b30 con 0x7f507c104d90 2024-04-24T08:05:30.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.536+0000 7f507be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c105f90 0x7f507c19b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:05:30.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.536+0000 7f5081400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f507c106930 0x7f507c1a24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:05:30.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.536+0000 7f507be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c105f90 0x7f507c19b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35612/0 (socket says 172.21.15.28:35612) 2024-04-24T08:05:30.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.536+0000 7f507be00700 1 -- 172.21.15.28:0/2629101677 learned_addr learned my addr 172.21.15.28:0/2629101677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:05:30.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.536+0000 7f5080a00700 1 --2- 172.21.15.28:0/2629101677 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f507c104d90 0x7f507c19af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:05:30.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.536+0000 7f507be00700 1 -- 172.21.15.28:0/2629101677 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f507c104d90 msgr2=0x7f507c19af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:05:30.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.536+0000 7f507be00700 1 --2- 172.21.15.28:0/2629101677 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f507c104d90 0x7f507c19af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:30.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.536+0000 7f507be00700 1 -- 172.21.15.28:0/2629101677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f507c106930 msgr2=0x7f507c1a24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:05:30.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.536+0000 7f507be00700 1 --2- 172.21.15.28:0/2629101677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f507c106930 0x7f507c1a24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:30.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.536+0000 7f507be00700 1 -- 172.21.15.28:0/2629101677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f507c1a2b70 con 0x7f507c105f90 2024-04-24T08:05:30.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.537+0000 7f507be00700 1 --2- 172.21.15.28:0/2629101677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c105f90 0x7f507c19b450 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f506800eb60 tx=0x7f506800ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:05:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.537+0000 7f5079600700 1 -- 172.21.15.28:0/2629101677 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f506800cbd0 con 0x7f507c105f90 2024-04-24T08:05:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.537+0000 7f5079600700 1 -- 172.21.15.28:0/2629101677 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5068004500 con 0x7f507c105f90 2024-04-24T08:05:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.537+0000 7f5083400700 1 -- 172.21.15.28:0/2629101677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f507c1a2e60 con 0x7f507c105f90 2024-04-24T08:05:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.537+0000 7f5079600700 1 -- 172.21.15.28:0/2629101677 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5068010430 con 0x7f507c105f90 2024-04-24T08:05:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.537+0000 7f5083400700 1 -- 172.21.15.28:0/2629101677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f507c1a3310 con 0x7f507c105f90 2024-04-24T08:05:30.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.538+0000 7f5083400700 1 -- 172.21.15.28:0/2629101677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f507c0697e0 con 0x7f507c105f90 2024-04-24T08:05:30.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.541+0000 7f5079600700 1 -- 172.21.15.28:0/2629101677 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5068003680 con 0x7f507c105f90 2024-04-24T08:05:30.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.543+0000 7f5079600700 1 --2- 172.21.15.28:0/2629101677 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f50640a3c40 0x7f50640a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:05:30.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.543+0000 7f5079600700 1 -- 172.21.15.28:0/2629101677 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(64..64 src has 1..64) ==== 5829+0+0 (secure 0 0 0) 0x7f50680cfd80 con 0x7f507c105f90 2024-04-24T08:05:30.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.543+0000 7f5080a00700 1 --2- 172.21.15.28:0/2629101677 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f50640a3c40 0x7f50640a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:05:30.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.544+0000 7f5080a00700 1 --2- 172.21.15.28:0/2629101677 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f50640a3c40 0x7f50640a6100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f507c105df0 tx=0x7f5070009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:05:30.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.546+0000 7f5079600700 1 -- 172.21.15.28:0/2629101677 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f506800cd70 con 0x7f507c105f90 2024-04-24T08:05:30.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:30.788+0000 7f5083400700 1 -- 172.21.15.28:0/2629101677 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f507c062f30 con 0x7f50640a3c40 2024-04-24T08:05:31.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:30 smithi194 ceph-mon[32836]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:31.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:30 smithi194 ceph-mon[32836]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:05:31.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:30 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:05:31.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:30 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:05:31.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:30 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:30 smithi028 ceph-mon[27786]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:30 smithi028 ceph-mon[27786]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:05:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:30 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:05:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:30 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:05:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:30 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:31.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:30 smithi117 ceph-mon[32438]: pgmap v224: 1 pgs: 1 remapped+peering; 577 KiB data, 259 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:31.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:30 smithi117 ceph-mon[32438]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi028:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:05:31.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:30 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:05:31.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:30 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:05:31.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:30 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:33.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:33 smithi028 ceph-mon[27786]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T08:05:33.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:33 smithi117 ceph-mon[32438]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T08:05:33.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:33 smithi194 ceph-mon[32836]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T08:05:35.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:35 smithi117 ceph-mon[32438]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-24T08:05:35.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:35 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1560273108' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9268da7-6a4a-4f31-b4ff-99e934adc71e"}]: dispatch 2024-04-24T08:05:35.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:35 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1560273108' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9268da7-6a4a-4f31-b4ff-99e934adc71e"}]': finished 2024-04-24T08:05:35.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:35 smithi117 ceph-mon[32438]: osdmap e65: 10 total, 9 up, 10 in 2024-04-24T08:05:35.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:05:35.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:35 smithi194 ceph-mon[32836]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-24T08:05:35.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:35 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1560273108' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9268da7-6a4a-4f31-b4ff-99e934adc71e"}]: dispatch 2024-04-24T08:05:35.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:35 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1560273108' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9268da7-6a4a-4f31-b4ff-99e934adc71e"}]': finished 2024-04-24T08:05:35.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:35 smithi194 ceph-mon[32836]: osdmap e65: 10 total, 9 up, 10 in 2024-04-24T08:05:35.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:05:35.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:35 smithi028 ceph-mon[27786]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 81 KiB/s, 0 objects/s recovering 2024-04-24T08:05:35.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:35 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1560273108' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9268da7-6a4a-4f31-b4ff-99e934adc71e"}]: dispatch 2024-04-24T08:05:35.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:35 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1560273108' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9268da7-6a4a-4f31-b4ff-99e934adc71e"}]': finished 2024-04-24T08:05:35.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:35 smithi028 ceph-mon[27786]: osdmap e65: 10 total, 9 up, 10 in 2024-04-24T08:05:35.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:05:36.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:36 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1956882805' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:05:36.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:36 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1956882805' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:05:36.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:36 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1956882805' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:05:37.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:37 smithi117 ceph-mon[32438]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T08:05:37.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:37 smithi194 ceph-mon[32836]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T08:05:37.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:37 smithi028 ceph-mon[27786]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-04-24T08:05:39.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:39 smithi117 ceph-mon[32438]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-24T08:05:39.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:39 smithi194 ceph-mon[32836]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-24T08:05:39.465 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:39 smithi028 ceph-mon[27786]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-04-24T08:05:41.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:41 smithi117 ceph-mon[32438]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-24T08:05:41.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:41 smithi194 ceph-mon[32836]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-24T08:05:41.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:41 smithi028 ceph-mon[27786]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-04-24T08:05:43.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:43 smithi028 ceph-mon[27786]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:43.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:43 smithi117 ceph-mon[32438]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:43.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:43 smithi194 ceph-mon[32836]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:45.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:45 smithi117 ceph-mon[32438]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:45.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:45 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-24T08:05:45.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:45 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:45.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:45 smithi194 ceph-mon[32836]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:45.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:45 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-24T08:05:45.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:45 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:45.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:45 smithi028 ceph-mon[27786]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:45.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:45 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-04-24T08:05:45.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:45 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:05:46.271 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:46 smithi028 ceph-mon[27786]: Deploying daemon osd.9 on smithi028 2024-04-24T08:05:46.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:46 smithi117 ceph-mon[32438]: Deploying daemon osd.9 on smithi028 2024-04-24T08:05:46.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:46 smithi194 ceph-mon[32836]: Deploying daemon osd.9 on smithi028 2024-04-24T08:05:47.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:47 smithi117 ceph-mon[32438]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:47.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:47 smithi194 ceph-mon[32836]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:47.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:47 smithi028 ceph-mon[27786]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:49.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:49 smithi117 ceph-mon[32438]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:49.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:49 smithi194 ceph-mon[32836]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:49.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:49 smithi028 ceph-mon[27786]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:51.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:51 smithi028 ceph-mon[27786]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:51.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:51 smithi117 ceph-mon[32438]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:51.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:51 smithi194 ceph-mon[32836]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:53.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:53 smithi028 ceph-mon[27786]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:53.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:53 smithi117 ceph-mon[32438]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:53.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:53 smithi194 ceph-mon[32836]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:55.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:54 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:55.284 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:54 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:54 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:05:55.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:54 smithi028 ceph-mon[27786]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:55.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:54 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:55.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:54 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:55.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:54 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:05:55.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:54 smithi117 ceph-mon[32438]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:55.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:54 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:55.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:54 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:55.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:54 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:05:55.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:54 smithi194 ceph-mon[32836]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:57.135 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:57 smithi028 ceph-mon[27786]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:57.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:57 smithi117 ceph-mon[32438]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:57.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:57 smithi194 ceph-mon[32836]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:59.158 INFO:teuthology.orchestra.run.smithi028.stdout:Created osd(s) 9 on host 'smithi028' 2024-04-24T08:05:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:59.156+0000 7f5079600700 1 -- 172.21.15.28:0/2629101677 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f507c062f30 con 0x7f50640a3c40 2024-04-24T08:05:59.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:59.159+0000 7f5083400700 1 -- 172.21.15.28:0/2629101677 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f50640a3c40 msgr2=0x7f50640a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:05:59.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:59.159+0000 7f5083400700 1 --2- 172.21.15.28:0/2629101677 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f50640a3c40 0x7f50640a6100 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f507c105df0 tx=0x7f5070009260 comp rx=0 tx=0).stop 2024-04-24T08:05:59.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:59.159+0000 7f5083400700 1 -- 172.21.15.28:0/2629101677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c105f90 msgr2=0x7f507c19b450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:05:59.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:59.159+0000 7f5083400700 1 --2- 172.21.15.28:0/2629101677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c105f90 0x7f507c19b450 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f506800eb60 tx=0x7f506800ee70 comp rx=0 tx=0).stop 2024-04-24T08:05:59.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:59.159+0000 7f5083400700 1 -- 172.21.15.28:0/2629101677 shutdown_connections 2024-04-24T08:05:59.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:59.159+0000 7f5083400700 1 --2- 172.21.15.28:0/2629101677 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f50640a3c40 0x7f50640a6100 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:59.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:59.159+0000 7f5083400700 1 --2- 172.21.15.28:0/2629101677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f507c106930 0x7f507c1a24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:59.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:59.159+0000 7f5083400700 1 --2- 172.21.15.28:0/2629101677 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f507c104d90 0x7f507c19af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:59.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:59.159+0000 7f5083400700 1 --2- 172.21.15.28:0/2629101677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f507c105f90 0x7f507c19b450 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:05:59.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:59.159+0000 7f5083400700 1 -- 172.21.15.28:0/2629101677 >> 172.21.15.28:0/2629101677 conn(0x7f507c100350 msgr2=0x7f507c1091c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:05:59.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:59.160+0000 7f5083400700 1 -- 172.21.15.28:0/2629101677 shutdown_connections 2024-04-24T08:05:59.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:05:59.160+0000 7f5083400700 1 -- 172.21.15.28:0/2629101677 wait complete. 2024-04-24T08:05:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:58 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:58 smithi028 ceph-mon[27786]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:59.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:58 smithi028 ceph-mon[27786]: from='osd.9 [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-24T08:05:59.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:58 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:59.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:58 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:59.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:58 smithi117 ceph-mon[32438]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:59.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:58 smithi117 ceph-mon[32438]: from='osd.9 [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-24T08:05:59.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:59.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:58 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:05:59.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:58 smithi194 ceph-mon[32836]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:05:59.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:58 smithi194 ceph-mon[32836]: from='osd.9 [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-04-24T08:05:59.934 DEBUG:teuthology.orchestra.run.smithi028:osd.9> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.9.service 2024-04-24T08:05:59.939 INFO:tasks.cephadm:Deploying osd.10 on smithi117 with /dev/vg_nvme/lv_1... 2024-04-24T08:05:59.939 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-24T08:06:00.184 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:06:00.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:59 smithi028 ceph-mon[27786]: from='osd.9 [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-24T08:06:00.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:59 smithi028 ceph-mon[27786]: osdmap e66: 10 total, 9 up, 10 in 2024-04-24T08:06:00.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:59 smithi028 ceph-mon[27786]: from='osd.9 [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:06:00.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:06:00.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:00.252 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:05:59 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:00.306 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:59 smithi117 ceph-mon[32438]: from='osd.9 [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-24T08:06:00.306 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:59 smithi117 ceph-mon[32438]: osdmap e66: 10 total, 9 up, 10 in 2024-04-24T08:06:00.306 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:59 smithi117 ceph-mon[32438]: from='osd.9 [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:06:00.306 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:59 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:06:00.307 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:59 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:00.307 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:05:59 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:00.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:59 smithi194 ceph-mon[32836]: from='osd.9 [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-04-24T08:06:00.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:59 smithi194 ceph-mon[32836]: osdmap e66: 10 total, 9 up, 10 in 2024-04-24T08:06:00.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:59 smithi194 ceph-mon[32836]: from='osd.9 [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]: dispatch 2024-04-24T08:06:00.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:06:00.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:00.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:05:59 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:00 smithi028 ceph-mon[27786]: from='osd.9 [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-24T08:06:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:00 smithi028 ceph-mon[27786]: osdmap e67: 10 total, 9 up, 10 in 2024-04-24T08:06:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:00 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:06:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:00 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:06:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:00 smithi028 ceph-mon[27786]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:06:01.335 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:00 smithi117 ceph-mon[32438]: from='osd.9 [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-24T08:06:01.336 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:00 smithi117 ceph-mon[32438]: osdmap e67: 10 total, 9 up, 10 in 2024-04-24T08:06:01.336 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:06:01.336 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:00 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:06:01.336 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:00 smithi117 ceph-mon[32438]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:06:01.341 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:00 smithi194 ceph-mon[32836]: from='osd.9 [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi028", "root=default"]}]': finished 2024-04-24T08:06:01.341 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:00 smithi194 ceph-mon[32836]: osdmap e67: 10 total, 9 up, 10 in 2024-04-24T08:06:01.341 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:00 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:06:01.341 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:00 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:06:01.342 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:00 smithi194 ceph-mon[32836]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 260 MiB used, 804 GiB / 805 GiB avail 2024-04-24T08:06:02.203 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:01 smithi028 ceph-mon[27786]: purged_snaps scrub starts 2024-04-24T08:06:02.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:01 smithi028 ceph-mon[27786]: purged_snaps scrub ok 2024-04-24T08:06:02.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:06:02.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:01 smithi028 ceph-mon[27786]: osd.9 [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688] boot 2024-04-24T08:06:02.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:01 smithi028 ceph-mon[27786]: osdmap e68: 10 total, 10 up, 10 in 2024-04-24T08:06:02.204 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:01 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:06:02.281 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:01 smithi117 ceph-mon[32438]: purged_snaps scrub starts 2024-04-24T08:06:02.281 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:01 smithi117 ceph-mon[32438]: purged_snaps scrub ok 2024-04-24T08:06:02.282 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:06:02.282 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:01 smithi117 ceph-mon[32438]: osd.9 [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688] boot 2024-04-24T08:06:02.282 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:01 smithi117 ceph-mon[32438]: osdmap e68: 10 total, 10 up, 10 in 2024-04-24T08:06:02.282 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:01 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:06:02.282 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-24T08:06:02.282 INFO:teuthology.orchestra.run.smithi117.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-24T08:06:02.282 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10+0 records in 2024-04-24T08:06:02.282 INFO:teuthology.orchestra.run.smithi117.stderr:10+0 records out 2024-04-24T08:06:02.282 INFO:teuthology.orchestra.run.smithi117.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0115223 s, 910 MB/s 2024-04-24T08:06:02.282 INFO:teuthology.orchestra.run.smithi117.stderr:--> Zapping successful for: 2024-04-24T08:06:02.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:01 smithi194 ceph-mon[32836]: purged_snaps scrub starts 2024-04-24T08:06:02.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:01 smithi194 ceph-mon[32836]: purged_snaps scrub ok 2024-04-24T08:06:02.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:06:02.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:01 smithi194 ceph-mon[32836]: osd.9 [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688] boot 2024-04-24T08:06:02.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:01 smithi194 ceph-mon[32836]: osdmap e68: 10 total, 10 up, 10 in 2024-04-24T08:06:02.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:01 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:06:02.950 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi117:vg_nvme/lv_1 2024-04-24T08:06:03.202 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:06:03.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:03 smithi028 ceph-mon[27786]: osdmap e69: 10 total, 10 up, 10 in 2024-04-24T08:06:03.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:03 smithi028 ceph-mon[27786]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:03.322 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:03 smithi117 ceph-mon[32438]: osdmap e69: 10 total, 10 up, 10 in 2024-04-24T08:06:03.323 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:03 smithi117 ceph-mon[32438]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:03.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:03 smithi194 ceph-mon[32836]: osdmap e69: 10 total, 10 up, 10 in 2024-04-24T08:06:03.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:03 smithi194 ceph-mon[32836]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:05.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.031+0000 7f9ca6064700 1 -- 172.21.15.117:0/1007167021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ca01046f0 msgr2=0x7f9ca010afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:05.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.032+0000 7f9ca6064700 1 --2- 172.21.15.117:0/1007167021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ca01046f0 0x7f9ca010afa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800b0d0 tx=0x7f9c9800b3e0 comp rx=0 tx=0).stop 2024-04-24T08:06:05.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.032+0000 7f9ca6064700 1 -- 172.21.15.117:0/1007167021 shutdown_connections 2024-04-24T08:06:05.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.032+0000 7f9ca6064700 1 --2- 172.21.15.117:0/1007167021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ca01046f0 0x7f9ca010afa0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:05.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.032+0000 7f9ca6064700 1 --2- 172.21.15.117:0/1007167021 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ca0102b50 0x7f9ca0102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:05.035 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.032+0000 7f9ca6064700 1 --2- 172.21.15.117:0/1007167021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ca0103d50 0x7f9ca01041b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:05.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.032+0000 7f9ca6064700 1 -- 172.21.15.117:0/1007167021 >> 172.21.15.117:0/1007167021 conn(0x7f9ca00fe0f0 msgr2=0x7f9ca0100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:06:05.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.032+0000 7f9ca6064700 1 -- 172.21.15.117:0/1007167021 shutdown_connections 2024-04-24T08:06:05.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.032+0000 7f9ca6064700 1 -- 172.21.15.117:0/1007167021 wait complete. 2024-04-24T08:06:05.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.032+0000 7f9ca6064700 1 Processor -- start 2024-04-24T08:06:05.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.032+0000 7f9ca6064700 1 -- start start 2024-04-24T08:06:05.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9ca6064700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ca0102b50 0x7f9ca0198da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:06:05.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9ca6064700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ca0103d50 0x7f9ca01992e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:06:05.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9ca6064700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ca01046f0 0x7f9ca01a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:06:05.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9ca6064700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca010e7e0 con 0x7f9ca0103d50 2024-04-24T08:06:05.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9ca6064700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca010e660 con 0x7f9ca0102b50 2024-04-24T08:06:05.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9ca6064700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9ca010e960 con 0x7f9ca01046f0 2024-04-24T08:06:05.036 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9c9f7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ca0102b50 0x7f9ca0198da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:06:05.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9c9f7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ca0102b50 0x7f9ca0198da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:39670/0 (socket says 172.21.15.117:39670) 2024-04-24T08:06:05.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9c9f7fe700 1 -- 172.21.15.117:0/3546164399 learned_addr learned my addr 172.21.15.117:0/3546164399 (peer_addr_for_me v2:172.21.15.117:0/0) 2024-04-24T08:06:05.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9c9ffff700 1 --2- 172.21.15.117:0/3546164399 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ca01046f0 0x7f9ca01a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:06:05.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9c9effd700 1 --2- 172.21.15.117:0/3546164399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ca0103d50 0x7f9ca01992e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:06:05.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9c9f7fe700 1 -- 172.21.15.117:0/3546164399 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ca01046f0 msgr2=0x7f9ca01a0360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:05.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9c9f7fe700 1 --2- 172.21.15.117:0/3546164399 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ca01046f0 0x7f9ca01a0360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:05.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9c9f7fe700 1 -- 172.21.15.117:0/3546164399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ca0103d50 msgr2=0x7f9ca01992e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:05.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9c9f7fe700 1 --2- 172.21.15.117:0/3546164399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ca0103d50 0x7f9ca01992e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:05.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9c9f7fe700 1 -- 172.21.15.117:0/3546164399 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ca01a0a00 con 0x7f9ca0102b50 2024-04-24T08:06:05.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9c9f7fe700 1 --2- 172.21.15.117:0/3546164399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ca0102b50 0x7f9ca0198da0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000d980 tx=0x7f9c9000dd40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:06:05.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.033+0000 7f9c9cff9700 1 -- 172.21.15.117:0/3546164399 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90017030 con 0x7f9ca0102b50 2024-04-24T08:06:05.037 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.034+0000 7f9c9cff9700 1 -- 172.21.15.117:0/3546164399 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c90004500 con 0x7f9ca0102b50 2024-04-24T08:06:05.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.034+0000 7f9c9cff9700 1 -- 172.21.15.117:0/3546164399 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c90003b20 con 0x7f9ca0102b50 2024-04-24T08:06:05.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.034+0000 7f9ca6064700 1 -- 172.21.15.117:0/3546164399 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ca01a0cf0 con 0x7f9ca0102b50 2024-04-24T08:06:05.038 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.034+0000 7f9ca6064700 1 -- 172.21.15.117:0/3546164399 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ca01a1200 con 0x7f9ca0102b50 2024-04-24T08:06:05.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.035+0000 7f9ca6064700 1 -- 172.21.15.117:0/3546164399 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9ca002dfe0 con 0x7f9ca0102b50 2024-04-24T08:06:05.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.036+0000 7f9c9cff9700 1 -- 172.21.15.117:0/3546164399 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9c9001a070 con 0x7f9ca0102b50 2024-04-24T08:06:05.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.036+0000 7f9c9cff9700 1 --2- 172.21.15.117:0/3546164399 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9c880a3c40 0x7f9c880a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:06:05.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.036+0000 7f9c9cff9700 1 -- 172.21.15.117:0/3546164399 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(69..69 src has 1..69) ==== 6261+0+0 (secure 0 0 0) 0x7f9c900cf610 con 0x7f9ca0102b50 2024-04-24T08:06:05.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.037+0000 7f9c9effd700 1 --2- 172.21.15.117:0/3546164399 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9c880a3c40 0x7f9c880a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:06:05.039 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.037+0000 7f9c9effd700 1 --2- 172.21.15.117:0/3546164399 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9c880a3c40 0x7f9c880a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9c94006fd0 tx=0x7f9c94008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:06:05.041 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.041+0000 7f9c9cff9700 1 -- 172.21.15.117:0/3546164399 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9c90009940 con 0x7f9ca0102b50 2024-04-24T08:06:05.252 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:05.252+0000 7f9ca6064700 1 -- 172.21.15.117:0/3546164399 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f9ca0062f30 con 0x7f9c880a3c40 2024-04-24T08:06:05.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:05 smithi028 ceph-mon[27786]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:05.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:05 smithi117 ceph-mon[32438]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:05.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:05 smithi194 ceph-mon[32836]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:06.229 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: Detected new or changed devices on smithi028 2024-04-24T08:06:06.229 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:06.229 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:06.230 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:06.230 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:06.230 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:06.230 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:06.230 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi028 to 4273M 2024-04-24T08:06:06.230 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:06.230 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:06.230 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:06.230 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:06.230 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:06:06.230 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:06:06.230 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:06:06.230 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:06.231 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:06:06.231 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:06.231 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:06.231 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:06 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:06.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: Detected new or changed devices on smithi028 2024-04-24T08:06:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi028 to 4273M 2024-04-24T08:06:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:06:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:06:06.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:06:06.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:06.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:06:06.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:06.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:06.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:06 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:06.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: Detected new or changed devices on smithi028 2024-04-24T08:06:06.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi028 to 4273M 2024-04-24T08:06:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='client.24350 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi117:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:06:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:06:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:06:06.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:06.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:06:06.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:06.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:06.541 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:06 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:07.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:07 smithi028 ceph-mon[27786]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:07.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:07 smithi117 ceph-mon[32438]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:07 smithi194 ceph-mon[32836]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:08.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:08 smithi028 ceph-mon[27786]: from='client.? 172.21.15.117:0/3293711903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "107dfd4b-d52f-4fdd-925e-d1fcac3cffe7"}]: dispatch 2024-04-24T08:06:08.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:08 smithi028 ceph-mon[27786]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "107dfd4b-d52f-4fdd-925e-d1fcac3cffe7"}]: dispatch 2024-04-24T08:06:08.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:08 smithi028 ceph-mon[27786]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "107dfd4b-d52f-4fdd-925e-d1fcac3cffe7"}]': finished 2024-04-24T08:06:08.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:08 smithi028 ceph-mon[27786]: osdmap e70: 11 total, 10 up, 11 in 2024-04-24T08:06:08.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:08 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:06:08.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:08 smithi117 ceph-mon[32438]: from='client.? 172.21.15.117:0/3293711903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "107dfd4b-d52f-4fdd-925e-d1fcac3cffe7"}]: dispatch 2024-04-24T08:06:08.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:08 smithi117 ceph-mon[32438]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "107dfd4b-d52f-4fdd-925e-d1fcac3cffe7"}]: dispatch 2024-04-24T08:06:08.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:08 smithi117 ceph-mon[32438]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "107dfd4b-d52f-4fdd-925e-d1fcac3cffe7"}]': finished 2024-04-24T08:06:08.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:08 smithi117 ceph-mon[32438]: osdmap e70: 11 total, 10 up, 11 in 2024-04-24T08:06:08.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:08 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:06:08.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:08 smithi194 ceph-mon[32836]: from='client.? 172.21.15.117:0/3293711903' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "107dfd4b-d52f-4fdd-925e-d1fcac3cffe7"}]: dispatch 2024-04-24T08:06:08.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:08 smithi194 ceph-mon[32836]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "107dfd4b-d52f-4fdd-925e-d1fcac3cffe7"}]: dispatch 2024-04-24T08:06:08.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:08 smithi194 ceph-mon[32836]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "107dfd4b-d52f-4fdd-925e-d1fcac3cffe7"}]': finished 2024-04-24T08:06:08.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:08 smithi194 ceph-mon[32836]: osdmap e70: 11 total, 10 up, 11 in 2024-04-24T08:06:08.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:08 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:06:09.779 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:09 smithi117 ceph-mon[32438]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:09.779 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:09 smithi117 ceph-mon[32438]: from='client.? 172.21.15.117:0/1737757179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:06:09.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:09 smithi028 ceph-mon[27786]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:09.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:09 smithi028 ceph-mon[27786]: from='client.? 172.21.15.117:0/1737757179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:06:09.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:09 smithi194 ceph-mon[32836]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:09.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:09 smithi194 ceph-mon[32836]: from='client.? 172.21.15.117:0/1737757179' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:06:11.767 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:11 smithi117 ceph-mon[32438]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:11.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:11 smithi028 ceph-mon[27786]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:11.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:11 smithi194 ceph-mon[32836]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:13.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:13 smithi028 ceph-mon[27786]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:13.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:13 smithi117 ceph-mon[32438]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:13.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:13 smithi194 ceph-mon[32836]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:15.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:15 smithi028 ceph-mon[27786]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:15.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:15 smithi117 ceph-mon[32438]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:15.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:15 smithi194 ceph-mon[32836]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:16.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:16 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-24T08:06:16.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:16 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:16.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:16 smithi028 ceph-mon[27786]: Deploying daemon osd.10 on smithi117 2024-04-24T08:06:16.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:16 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-24T08:06:16.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:16 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:16.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:16 smithi117 ceph-mon[32438]: Deploying daemon osd.10 on smithi117 2024-04-24T08:06:16.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:16 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-04-24T08:06:16.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:16 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:16.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:16 smithi194 ceph-mon[32836]: Deploying daemon osd.10 on smithi117 2024-04-24T08:06:17.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:17 smithi028 ceph-mon[27786]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:17.852 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:17 smithi117 ceph-mon[32438]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:17.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:17 smithi194 ceph-mon[32836]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:19.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:19 smithi117 ceph-mon[32438]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:19.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:19 smithi194 ceph-mon[32836]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:20.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:19 smithi028 ceph-mon[27786]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:21.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:21 smithi117 ceph-mon[32438]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:21.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:21 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:21.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:21 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:21.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:21 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:06:21.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:21 smithi194 ceph-mon[32836]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:21.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:21 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:21.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:21 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:21.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:21 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:06:22.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:21 smithi028 ceph-mon[27786]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:22.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:21 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:22.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:21 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:22.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:21 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:06:23.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:23 smithi028 ceph-mon[27786]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:23.839 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:23 smithi117 ceph-mon[32438]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:23.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:23 smithi194 ceph-mon[32836]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:25.291 INFO:teuthology.orchestra.run.smithi117.stdout:Created osd(s) 10 on host 'smithi117' 2024-04-24T08:06:25.292 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:25.290+0000 7f9c9cff9700 1 -- 172.21.15.117:0/3546164399 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f9ca0062f30 con 0x7f9c880a3c40 2024-04-24T08:06:25.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:25.292+0000 7f9ca6064700 1 -- 172.21.15.117:0/3546164399 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9c880a3c40 msgr2=0x7f9c880a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:25.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:25.292+0000 7f9ca6064700 1 --2- 172.21.15.117:0/3546164399 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9c880a3c40 0x7f9c880a6100 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f9c94006fd0 tx=0x7f9c94008040 comp rx=0 tx=0).stop 2024-04-24T08:06:25.293 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:25.292+0000 7f9ca6064700 1 -- 172.21.15.117:0/3546164399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ca0102b50 msgr2=0x7f9ca0198da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:25.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:25.292+0000 7f9ca6064700 1 --2- 172.21.15.117:0/3546164399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ca0102b50 0x7f9ca0198da0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9c9000d980 tx=0x7f9c9000dd40 comp rx=0 tx=0).stop 2024-04-24T08:06:25.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:25.292+0000 7f9ca6064700 1 -- 172.21.15.117:0/3546164399 shutdown_connections 2024-04-24T08:06:25.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:25.292+0000 7f9ca6064700 1 --2- 172.21.15.117:0/3546164399 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9c880a3c40 0x7f9c880a6100 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:25.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:25.292+0000 7f9ca6064700 1 --2- 172.21.15.117:0/3546164399 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9ca01046f0 0x7f9ca01a0360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:25.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:25.292+0000 7f9ca6064700 1 --2- 172.21.15.117:0/3546164399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ca0102b50 0x7f9ca0198da0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:25.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:25.292+0000 7f9ca6064700 1 --2- 172.21.15.117:0/3546164399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9ca0103d50 0x7f9ca01992e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:25.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:25.292+0000 7f9ca6064700 1 -- 172.21.15.117:0/3546164399 >> 172.21.15.117:0/3546164399 conn(0x7f9ca00fe0f0 msgr2=0x7f9ca0107050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:06:25.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:25.293+0000 7f9ca6064700 1 -- 172.21.15.117:0/3546164399 shutdown_connections 2024-04-24T08:06:25.294 INFO:teuthology.orchestra.run.smithi117.stderr:2024-04-24T08:06:25.293+0000 7f9ca6064700 1 -- 172.21.15.117:0/3546164399 wait complete. 2024-04-24T08:06:25.606 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:25 smithi117 ceph-mon[32438]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:25.606 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:25 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:25.606 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:25 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:25.606 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:25 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:25.606 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:25 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:25.606 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:25 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:25.606 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:25 smithi117 ceph-mon[32438]: from='osd.10 [v2:172.21.15.117:6824/2345072654,v1:172.21.15.117:6825/2345072654]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T08:06:25.606 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:25 smithi117 ceph-mon[32438]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T08:06:25.607 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:25 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:25.607 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:25 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:25.607 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:25 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:06:25.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:25 smithi194 ceph-mon[32836]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:25.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:25 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:25.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:25 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:25.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:25 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:25.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:25 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:25.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:25 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:25.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:25 smithi194 ceph-mon[32836]: from='osd.10 [v2:172.21.15.117:6824/2345072654,v1:172.21.15.117:6825/2345072654]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T08:06:25.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:25 smithi194 ceph-mon[32836]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T08:06:25.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:25 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:25.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:25 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:25.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:25 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:06:25.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:25 smithi028 ceph-mon[27786]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:25.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:25 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:25.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:25 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:25.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:25 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:25.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:25 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:25.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:25 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:25.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:25 smithi028 ceph-mon[27786]: from='osd.10 [v2:172.21.15.117:6824/2345072654,v1:172.21.15.117:6825/2345072654]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T08:06:25.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:25 smithi028 ceph-mon[27786]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-04-24T08:06:25.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:25 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:25.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:25 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:25.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:25 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:06:25.940 DEBUG:teuthology.orchestra.run.smithi117:osd.10> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.10.service 2024-04-24T08:06:25.942 INFO:tasks.cephadm:Deploying osd.11 on smithi194 with /dev/vg_nvme/lv_1... 2024-04-24T08:06:25.943 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-04-24T08:06:26.187 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:06:26.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:26 smithi117 ceph-mon[32438]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-24T08:06:26.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:26 smithi117 ceph-mon[32438]: osdmap e71: 11 total, 10 up, 11 in 2024-04-24T08:06:26.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:26 smithi117 ceph-mon[32438]: from='osd.10 [v2:172.21.15.117:6824/2345072654,v1:172.21.15.117:6825/2345072654]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:06:26.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:06:26.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:26 smithi117 ceph-mon[32438]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:06:26.878 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:06:26 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:06:26.563+0000 7ff43db21700 -1 osd.10 0 waiting for initial osdmap 2024-04-24T08:06:26.878 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:06:26 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:06:26.576+0000 7ff439914700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:06:26.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:26 smithi194 ceph-mon[32836]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-24T08:06:26.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:26 smithi194 ceph-mon[32836]: osdmap e71: 11 total, 10 up, 11 in 2024-04-24T08:06:26.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:26 smithi194 ceph-mon[32836]: from='osd.10 [v2:172.21.15.117:6824/2345072654,v1:172.21.15.117:6825/2345072654]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:06:26.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:06:26.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:26 smithi194 ceph-mon[32836]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:06:27.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:26 smithi028 ceph-mon[27786]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-04-24T08:06:27.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:26 smithi028 ceph-mon[27786]: osdmap e71: 11 total, 10 up, 11 in 2024-04-24T08:06:27.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:26 smithi028 ceph-mon[27786]: from='osd.10 [v2:172.21.15.117:6824/2345072654,v1:172.21.15.117:6825/2345072654]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:06:27.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:06:27.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:26 smithi028 ceph-mon[27786]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]: dispatch 2024-04-24T08:06:27.639 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:27 smithi194 ceph-mon[32836]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:27.639 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:27 smithi194 ceph-mon[32836]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-04-24T08:06:27.639 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:27 smithi194 ceph-mon[32836]: osdmap e72: 11 total, 10 up, 11 in 2024-04-24T08:06:27.639 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:27 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:06:27.639 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:27 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:06:27.639 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:27 smithi194 ceph-mon[32836]: osd.10 [v2:172.21.15.117:6824/2345072654,v1:172.21.15.117:6825/2345072654] boot 2024-04-24T08:06:27.639 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:27 smithi194 ceph-mon[32836]: osdmap e73: 11 total, 11 up, 11 in 2024-04-24T08:06:27.639 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:27 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:06:27.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:27 smithi117 ceph-mon[32438]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:27.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:27 smithi117 ceph-mon[32438]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-04-24T08:06:27.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:27 smithi117 ceph-mon[32438]: osdmap e72: 11 total, 10 up, 11 in 2024-04-24T08:06:27.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:27 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:06:27.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:27 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:06:27.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:27 smithi117 ceph-mon[32438]: osd.10 [v2:172.21.15.117:6824/2345072654,v1:172.21.15.117:6825/2345072654] boot 2024-04-24T08:06:27.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:27 smithi117 ceph-mon[32438]: osdmap e73: 11 total, 11 up, 11 in 2024-04-24T08:06:27.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:27 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:06:28.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:27 smithi028 ceph-mon[27786]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-04-24T08:06:28.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:27 smithi028 ceph-mon[27786]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi117", "root=default"]}]': finished 2024-04-24T08:06:28.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:27 smithi028 ceph-mon[27786]: osdmap e72: 11 total, 10 up, 11 in 2024-04-24T08:06:28.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:27 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:06:28.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:27 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:06:28.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:27 smithi028 ceph-mon[27786]: osd.10 [v2:172.21.15.117:6824/2345072654,v1:172.21.15.117:6825/2345072654] boot 2024-04-24T08:06:28.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:27 smithi028 ceph-mon[27786]: osdmap e73: 11 total, 11 up, 11 in 2024-04-24T08:06:28.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:27 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:06:28.386 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-04-24T08:06:28.386 INFO:teuthology.orchestra.run.smithi194.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-04-24T08:06:28.386 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10+0 records in 2024-04-24T08:06:28.386 INFO:teuthology.orchestra.run.smithi194.stderr:10+0 records out 2024-04-24T08:06:28.387 INFO:teuthology.orchestra.run.smithi194.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113743 s, 922 MB/s 2024-04-24T08:06:28.387 INFO:teuthology.orchestra.run.smithi194.stderr:--> Zapping successful for: 2024-04-24T08:06:28.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:28 smithi194 ceph-mon[32836]: purged_snaps scrub starts 2024-04-24T08:06:28.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:28 smithi194 ceph-mon[32836]: purged_snaps scrub ok 2024-04-24T08:06:28.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:28 smithi194 ceph-mon[32836]: osdmap e74: 11 total, 11 up, 11 in 2024-04-24T08:06:28.806 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:28 smithi117 ceph-mon[32438]: purged_snaps scrub starts 2024-04-24T08:06:28.806 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:28 smithi117 ceph-mon[32438]: purged_snaps scrub ok 2024-04-24T08:06:28.807 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:28 smithi117 ceph-mon[32438]: osdmap e74: 11 total, 11 up, 11 in 2024-04-24T08:06:29.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:28 smithi028 ceph-mon[27786]: purged_snaps scrub starts 2024-04-24T08:06:29.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:28 smithi028 ceph-mon[27786]: purged_snaps scrub ok 2024-04-24T08:06:29.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:28 smithi028 ceph-mon[27786]: osdmap e74: 11 total, 11 up, 11 in 2024-04-24T08:06:29.048 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch daemon add osd smithi194:vg_nvme/lv_1 2024-04-24T08:06:29.289 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:06:29.662 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:29 smithi194 ceph-mon[32836]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:29.663 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:29.663 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:29.663 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:29.663 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:29.663 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:29.663 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:29.663 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:29.663 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:29.663 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:29.663 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:29.663 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:29 smithi194 ceph-mon[32836]: osdmap e75: 11 total, 11 up, 11 in 2024-04-24T08:06:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:29 smithi117 ceph-mon[32438]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:29 smithi117 ceph-mon[32438]: osdmap e75: 11 total, 11 up, 11 in 2024-04-24T08:06:30.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:29 smithi028 ceph-mon[27786]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:30.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:30.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:30.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:30.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:30.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:30.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:30.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:30.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:30.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:30.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:30.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:29 smithi028 ceph-mon[27786]: osdmap e75: 11 total, 11 up, 11 in 2024-04-24T08:06:30.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:30 smithi194 ceph-mon[32836]: Detected new or changed devices on smithi117 2024-04-24T08:06:30.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:30 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi117 to 4272M 2024-04-24T08:06:30.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:30 smithi194 ceph-mon[32836]: osdmap e76: 11 total, 11 up, 11 in 2024-04-24T08:06:30.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:30 smithi117 ceph-mon[32438]: Detected new or changed devices on smithi117 2024-04-24T08:06:30.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:30 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi117 to 4272M 2024-04-24T08:06:30.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:30 smithi117 ceph-mon[32438]: osdmap e76: 11 total, 11 up, 11 in 2024-04-24T08:06:30.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.950+0000 7f9d18539700 1 -- 172.21.15.194:0/2910486788 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d10102da0 msgr2=0x7f9d10109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:30.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.950+0000 7f9d18539700 1 --2- 172.21.15.194:0/2910486788 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d10102da0 0x7f9d10109650 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9d0800b0a0 tx=0x7f9d0800b3b0 comp rx=0 tx=0).stop 2024-04-24T08:06:30.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.950+0000 7f9d18539700 1 -- 172.21.15.194:0/2910486788 shutdown_connections 2024-04-24T08:06:30.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.950+0000 7f9d18539700 1 --2- 172.21.15.194:0/2910486788 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d10102da0 0x7f9d10109650 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:30.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.950+0000 7f9d18539700 1 --2- 172.21.15.194:0/2910486788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d10101200 0x7f9d10101620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:30.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.950+0000 7f9d18539700 1 --2- 172.21.15.194:0/2910486788 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d10102400 0x7f9d10102860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:30.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.950+0000 7f9d18539700 1 -- 172.21.15.194:0/2910486788 >> 172.21.15.194:0/2910486788 conn(0x7f9d100fc7a0 msgr2=0x7f9d100febe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:06:30.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.950+0000 7f9d18539700 1 -- 172.21.15.194:0/2910486788 shutdown_connections 2024-04-24T08:06:30.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.950+0000 7f9d18539700 1 -- 172.21.15.194:0/2910486788 wait complete. 2024-04-24T08:06:30.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.951+0000 7f9d18539700 1 Processor -- start 2024-04-24T08:06:30.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.951+0000 7f9d18539700 1 -- start start 2024-04-24T08:06:30.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.951+0000 7f9d18539700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d10101200 0x7f9d100fe840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:06:30.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.951+0000 7f9d18539700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d10102400 0x7f9d1019dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:06:30.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.951+0000 7f9d18539700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d10102da0 0x7f9d101a0200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:06:30.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.951+0000 7f9d18539700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1010cee0 con 0x7f9d10102400 2024-04-24T08:06:30.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.951+0000 7f9d18539700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1010cd60 con 0x7f9d10102da0 2024-04-24T08:06:30.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.951+0000 7f9d18539700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1010d060 con 0x7f9d10101200 2024-04-24T08:06:30.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.951+0000 7f9d162d5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d10101200 0x7f9d100fe840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:06:30.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.951+0000 7f9d162d5700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d10101200 0x7f9d100fe840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:57914/0 (socket says 172.21.15.194:57914) 2024-04-24T08:06:30.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.951+0000 7f9d162d5700 1 -- 172.21.15.194:0/4228558768 learned_addr learned my addr 172.21.15.194:0/4228558768 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-24T08:06:30.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.952+0000 7f9d16ad6700 1 --2- 172.21.15.194:0/4228558768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d10102da0 0x7f9d101a0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:06:30.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.952+0000 7f9d162d5700 1 -- 172.21.15.194:0/4228558768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d10102da0 msgr2=0x7f9d101a0200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:30.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.952+0000 7f9d162d5700 1 --2- 172.21.15.194:0/4228558768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d10102da0 0x7f9d101a0200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:30.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.952+0000 7f9d162d5700 1 -- 172.21.15.194:0/4228558768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d10102400 msgr2=0x7f9d1019dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:30.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.952+0000 7f9d162d5700 1 --2- 172.21.15.194:0/4228558768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d10102400 0x7f9d1019dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:30.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.952+0000 7f9d162d5700 1 -- 172.21.15.194:0/4228558768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d101a0ad0 con 0x7f9d10101200 2024-04-24T08:06:30.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.952+0000 7f9d162d5700 1 --2- 172.21.15.194:0/4228558768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d10101200 0x7f9d100fe840 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000b500 tx=0x7f9d0000b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:06:30.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.952+0000 7f9d077fe700 1 -- 172.21.15.194:0/4228558768 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d00015020 con 0x7f9d10101200 2024-04-24T08:06:30.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.952+0000 7f9d18539700 1 -- 172.21.15.194:0/4228558768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d101a0dc0 con 0x7f9d10101200 2024-04-24T08:06:30.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.952+0000 7f9d077fe700 1 -- 172.21.15.194:0/4228558768 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d00011d10 con 0x7f9d10101200 2024-04-24T08:06:30.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.952+0000 7f9d077fe700 1 -- 172.21.15.194:0/4228558768 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0000f2e0 con 0x7f9d10101200 2024-04-24T08:06:30.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.952+0000 7f9d18539700 1 -- 172.21.15.194:0/4228558768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d101a12f0 con 0x7f9d10101200 2024-04-24T08:06:30.956 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.953+0000 7f9d18539700 1 -- 172.21.15.194:0/4228558768 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d100697e0 con 0x7f9d10101200 2024-04-24T08:06:30.957 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.954+0000 7f9d077fe700 1 -- 172.21.15.194:0/4228558768 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9d00018020 con 0x7f9d10101200 2024-04-24T08:06:30.957 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.955+0000 7f9d077fe700 1 --2- 172.21.15.194:0/4228558768 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9cfc0a3c40 0x7f9cfc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:06:30.957 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.955+0000 7f9d077fe700 1 -- 172.21.15.194:0/4228558768 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(76..76 src has 1..76) ==== 6694+0+0 (secure 0 0 0) 0x7f9d000cfba0 con 0x7f9d10101200 2024-04-24T08:06:30.957 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.955+0000 7f9d15ad4700 1 --2- 172.21.15.194:0/4228558768 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9cfc0a3c40 0x7f9cfc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:06:30.957 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.956+0000 7f9d15ad4700 1 --2- 172.21.15.194:0/4228558768 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9cfc0a3c40 0x7f9cfc0a6100 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0097b0 tx=0x7f9d0c006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:06:30.959 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:30.959+0000 7f9d077fe700 1 -- 172.21.15.194:0/4228558768 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d0001f020 con 0x7f9d10101200 2024-04-24T08:06:31.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:30 smithi028 ceph-mon[27786]: Detected new or changed devices on smithi117 2024-04-24T08:06:31.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:30 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi117 to 4272M 2024-04-24T08:06:31.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:30 smithi028 ceph-mon[27786]: osdmap e76: 11 total, 11 up, 11 in 2024-04-24T08:06:31.165 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:31.165+0000 7f9d18539700 1 -- 172.21.15.194:0/4228558768 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f9d10062f30 con 0x7f9cfc0a3c40 2024-04-24T08:06:31.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:31 smithi117 ceph-mon[32438]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:31.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:31 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:06:31.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:31 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:06:31.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:31 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:31.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:31 smithi117 ceph-mon[32438]: osdmap e77: 11 total, 11 up, 11 in 2024-04-24T08:06:31.916 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:31 smithi194 ceph-mon[32836]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:31.917 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:31 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:06:31.917 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:31 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:06:31.917 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:31 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:31.917 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:31 smithi194 ceph-mon[32836]: osdmap e77: 11 total, 11 up, 11 in 2024-04-24T08:06:32.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:31 smithi028 ceph-mon[27786]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:32.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:31 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-04-24T08:06:32.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:31 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-04-24T08:06:32.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:31 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:32.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:31 smithi028 ceph-mon[27786]: osdmap e77: 11 total, 11 up, 11 in 2024-04-24T08:06:32.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:32 smithi117 ceph-mon[32438]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:06:32.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:32 smithi194 ceph-mon[32836]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:06:33.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:32 smithi028 ceph-mon[27786]: from='client.24397 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi194:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:06:33.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:33 smithi117 ceph-mon[32438]: pgmap v268: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-24T08:06:33.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:33 smithi194 ceph-mon[32836]: pgmap v268: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-24T08:06:34.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:33 smithi028 ceph-mon[27786]: pgmap v268: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-04-24T08:06:34.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:34 smithi194 ceph-mon[32836]: from='client.? 172.21.15.194:0/88884860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22674865-b8f4-445d-8152-ba0a903586b4"}]: dispatch 2024-04-24T08:06:34.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:34 smithi194 ceph-mon[32836]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22674865-b8f4-445d-8152-ba0a903586b4"}]: dispatch 2024-04-24T08:06:34.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:34 smithi194 ceph-mon[32836]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22674865-b8f4-445d-8152-ba0a903586b4"}]': finished 2024-04-24T08:06:34.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:34 smithi194 ceph-mon[32836]: osdmap e78: 12 total, 11 up, 12 in 2024-04-24T08:06:34.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:34 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:34.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:34 smithi117 ceph-mon[32438]: from='client.? 172.21.15.194:0/88884860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22674865-b8f4-445d-8152-ba0a903586b4"}]: dispatch 2024-04-24T08:06:34.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:34 smithi117 ceph-mon[32438]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22674865-b8f4-445d-8152-ba0a903586b4"}]: dispatch 2024-04-24T08:06:34.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:34 smithi117 ceph-mon[32438]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22674865-b8f4-445d-8152-ba0a903586b4"}]': finished 2024-04-24T08:06:34.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:34 smithi117 ceph-mon[32438]: osdmap e78: 12 total, 11 up, 12 in 2024-04-24T08:06:34.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:34 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:35.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:34 smithi028 ceph-mon[27786]: from='client.? 172.21.15.194:0/88884860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22674865-b8f4-445d-8152-ba0a903586b4"}]: dispatch 2024-04-24T08:06:35.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:34 smithi028 ceph-mon[27786]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "22674865-b8f4-445d-8152-ba0a903586b4"}]: dispatch 2024-04-24T08:06:35.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:34 smithi028 ceph-mon[27786]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "22674865-b8f4-445d-8152-ba0a903586b4"}]': finished 2024-04-24T08:06:35.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:34 smithi028 ceph-mon[27786]: osdmap e78: 12 total, 11 up, 12 in 2024-04-24T08:06:35.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:34 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:35.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:35 smithi117 ceph-mon[32438]: pgmap v270: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-24T08:06:35.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:35 smithi117 ceph-mon[32438]: from='client.? 172.21.15.194:0/2758288074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:06:35.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:35 smithi194 ceph-mon[32836]: pgmap v270: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-24T08:06:35.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:35 smithi194 ceph-mon[32836]: from='client.? 172.21.15.194:0/2758288074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:06:36.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:35 smithi028 ceph-mon[27786]: pgmap v270: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 129 KiB/s, 0 objects/s recovering 2024-04-24T08:06:36.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:35 smithi028 ceph-mon[27786]: from='client.? 172.21.15.194:0/2758288074' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-04-24T08:06:37.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:37 smithi117 ceph-mon[32438]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T08:06:37.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:37 smithi194 ceph-mon[32836]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T08:06:38.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:37 smithi028 ceph-mon[27786]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-04-24T08:06:39.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:39 smithi117 ceph-mon[32438]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-24T08:06:39.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:39 smithi194 ceph-mon[32836]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-24T08:06:40.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:39 smithi028 ceph-mon[27786]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 77 KiB/s, 0 objects/s recovering 2024-04-24T08:06:41.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:41 smithi194 ceph-mon[32836]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:41.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:41 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-24T08:06:41.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:41 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:41.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:41 smithi117 ceph-mon[32438]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:41.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:41 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-24T08:06:41.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:41 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:42.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:41 smithi028 ceph-mon[27786]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:42.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:41 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-04-24T08:06:42.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:41 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:42.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:42 smithi194 ceph-mon[32836]: Deploying daemon osd.11 on smithi194 2024-04-24T08:06:43.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:42 smithi028 ceph-mon[27786]: Deploying daemon osd.11 on smithi194 2024-04-24T08:06:43.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:42 smithi117 ceph-mon[32438]: Deploying daemon osd.11 on smithi194 2024-04-24T08:06:43.902 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:43 smithi194 ceph-mon[32836]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:44.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:43 smithi028 ceph-mon[27786]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:44.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:43 smithi117 ceph-mon[32438]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:45.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:44 smithi194 ceph-mon[32836]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:45.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:44 smithi028 ceph-mon[27786]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:45.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:44 smithi117 ceph-mon[32438]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:47.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:47 smithi117 ceph-mon[32438]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:47.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:47 smithi194 ceph-mon[32836]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:47.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:47 smithi028 ceph-mon[27786]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:48.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:48 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:48.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:48 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:48.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:48 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:06:48.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:48 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:48.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:48 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:48.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:48 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:06:48.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:48 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:48.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:48 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:48.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:48 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:06:49.543 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:49 smithi194 ceph-mon[32836]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:49.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:49 smithi117 ceph-mon[32438]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:49.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:49 smithi028 ceph-mon[27786]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:51.207 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:51.206+0000 7f9d077fe700 1 -- 172.21.15.194:0/4228558768 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f9d10062f30 con 0x7f9cfc0a3c40 2024-04-24T08:06:51.208 INFO:teuthology.orchestra.run.smithi194.stdout:Created osd(s) 11 on host 'smithi194' 2024-04-24T08:06:51.211 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:51.209+0000 7f9d18539700 1 -- 172.21.15.194:0/4228558768 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9cfc0a3c40 msgr2=0x7f9cfc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:51.211 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:51.209+0000 7f9d18539700 1 --2- 172.21.15.194:0/4228558768 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9cfc0a3c40 0x7f9cfc0a6100 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c0097b0 tx=0x7f9d0c006d20 comp rx=0 tx=0).stop 2024-04-24T08:06:51.211 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:51.210+0000 7f9d18539700 1 -- 172.21.15.194:0/4228558768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d10101200 msgr2=0x7f9d100fe840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:51.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:51.210+0000 7f9d18539700 1 --2- 172.21.15.194:0/4228558768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d10101200 0x7f9d100fe840 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9d0000b500 tx=0x7f9d0000b8c0 comp rx=0 tx=0).stop 2024-04-24T08:06:51.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:51.210+0000 7f9d18539700 1 -- 172.21.15.194:0/4228558768 shutdown_connections 2024-04-24T08:06:51.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:51.210+0000 7f9d18539700 1 --2- 172.21.15.194:0/4228558768 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9cfc0a3c40 0x7f9cfc0a6100 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:51.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:51.210+0000 7f9d18539700 1 --2- 172.21.15.194:0/4228558768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d10102da0 0x7f9d101a0200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:51.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:51.210+0000 7f9d18539700 1 --2- 172.21.15.194:0/4228558768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d10101200 0x7f9d100fe840 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:51.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:51.210+0000 7f9d18539700 1 --2- 172.21.15.194:0/4228558768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d10102400 0x7f9d1019dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:51.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:51.210+0000 7f9d18539700 1 -- 172.21.15.194:0/4228558768 >> 172.21.15.194:0/4228558768 conn(0x7f9d100fc7a0 msgr2=0x7f9d10105ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:06:51.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:51.210+0000 7f9d18539700 1 -- 172.21.15.194:0/4228558768 shutdown_connections 2024-04-24T08:06:51.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:06:51.210+0000 7f9d18539700 1 -- 172.21.15.194:0/4228558768 wait complete. 2024-04-24T08:06:51.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:51 smithi194 ceph-mon[32836]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:51.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:51.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:51.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:51.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:51.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:51.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:51.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:51.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:06:51.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:51 smithi194 ceph-mon[32836]: from='osd.11 [v2:172.21.15.194:6824/275366773,v1:172.21.15.194:6825/275366773]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T08:06:51.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:51 smithi194 ceph-mon[32836]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T08:06:51.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:51 smithi028 ceph-mon[27786]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:51.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:51.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:51.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:51.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:51.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:51.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:51.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:51.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:06:51.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:51 smithi028 ceph-mon[27786]: from='osd.11 [v2:172.21.15.194:6824/275366773,v1:172.21.15.194:6825/275366773]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T08:06:51.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:51 smithi028 ceph-mon[27786]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T08:06:51.791 DEBUG:teuthology.orchestra.run.smithi194:osd.11> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.11.service 2024-04-24T08:06:51.793 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-04-24T08:06:51.793 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd stat -f json 2024-04-24T08:06:51.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:51 smithi117 ceph-mon[32438]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:51.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:51.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:51.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:51.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:51.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:51.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:51.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:51.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:06:51.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:51 smithi117 ceph-mon[32438]: from='osd.11 [v2:172.21.15.194:6824/275366773,v1:172.21.15.194:6825/275366773]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T08:06:51.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:51 smithi117 ceph-mon[32438]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-04-24T08:06:52.270 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:06:53.326 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:53 smithi194 ceph-mon[32836]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:53.326 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:53 smithi194 ceph-mon[32836]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-24T08:06:53.326 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:53 smithi194 ceph-mon[32836]: osdmap e79: 12 total, 11 up, 12 in 2024-04-24T08:06:53.326 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:53 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:53.326 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:53 smithi194 ceph-mon[32836]: from='osd.11 [v2:172.21.15.194:6824/275366773,v1:172.21.15.194:6825/275366773]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:06:53.326 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:53 smithi194 ceph-mon[32836]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:06:53.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:53 smithi028 ceph-mon[27786]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:53.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:53 smithi028 ceph-mon[27786]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-24T08:06:53.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:53 smithi028 ceph-mon[27786]: osdmap e79: 12 total, 11 up, 12 in 2024-04-24T08:06:53.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:53 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:53.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:53 smithi028 ceph-mon[27786]: from='osd.11 [v2:172.21.15.194:6824/275366773,v1:172.21.15.194:6825/275366773]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:06:53.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:53 smithi028 ceph-mon[27786]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:06:53.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:53 smithi117 ceph-mon[32438]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:53.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:53 smithi117 ceph-mon[32438]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-04-24T08:06:53.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:53 smithi117 ceph-mon[32438]: osdmap e79: 12 total, 11 up, 12 in 2024-04-24T08:06:53.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:53.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:53 smithi117 ceph-mon[32438]: from='osd.11 [v2:172.21.15.194:6824/275366773,v1:172.21.15.194:6825/275366773]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:06:53.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:53 smithi117 ceph-mon[32438]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]: dispatch 2024-04-24T08:06:54.164 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:06:53 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:06:53.736+0000 7f8da0437700 -1 osd.11 0 waiting for initial osdmap 2024-04-24T08:06:54.164 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:06:53 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:06:53.748+0000 7f8d98222700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-04-24T08:06:54.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:54 smithi028 ceph-mon[27786]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-24T08:06:54.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:54 smithi028 ceph-mon[27786]: osdmap e80: 12 total, 11 up, 12 in 2024-04-24T08:06:54.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:54 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:54.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:54 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:54.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:54 smithi028 ceph-mon[27786]: from='osd.11 ' entity='osd.11' 2024-04-24T08:06:54.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:54 smithi194 ceph-mon[32836]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-24T08:06:54.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:54 smithi194 ceph-mon[32836]: osdmap e80: 12 total, 11 up, 12 in 2024-04-24T08:06:54.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:54 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:54.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:54 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:54.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:54 smithi194 ceph-mon[32836]: from='osd.11 ' entity='osd.11' 2024-04-24T08:06:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.487+0000 7f77f4200700 1 -- 172.21.15.28:0/3801237060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77ec1035e0 msgr2=0x7f77ec1059d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.487+0000 7f77f4200700 1 --2- 172.21.15.28:0/3801237060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77ec1035e0 0x7f77ec1059d0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f77e0009990 tx=0x7f77e0009ca0 comp rx=0 tx=0).stop 2024-04-24T08:06:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.488+0000 7f77f4200700 1 -- 172.21.15.28:0/3801237060 shutdown_connections 2024-04-24T08:06:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.488+0000 7f77f4200700 1 --2- 172.21.15.28:0/3801237060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77ec108840 0x7f77ec10acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.488+0000 7f77f4200700 1 --2- 172.21.15.28:0/3801237060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77ec1035e0 0x7f77ec1059d0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.488+0000 7f77f4200700 1 --2- 172.21.15.28:0/3801237060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77ec105f10 0x7f77ec108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.488+0000 7f77f4200700 1 -- 172.21.15.28:0/3801237060 >> 172.21.15.28:0/3801237060 conn(0x7f77ec0fcff0 msgr2=0x7f77ec0ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:06:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.488+0000 7f77f4200700 1 -- 172.21.15.28:0/3801237060 shutdown_connections 2024-04-24T08:06:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.488+0000 7f77f4200700 1 -- 172.21.15.28:0/3801237060 wait complete. 2024-04-24T08:06:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.489+0000 7f77f4200700 1 Processor -- start 2024-04-24T08:06:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.489+0000 7f77f4200700 1 -- start start 2024-04-24T08:06:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.489+0000 7f77f4200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77ec1035e0 0x7f77ec102f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:06:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.489+0000 7f77f4200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77ec105f10 0x7f77ec101590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:06:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.489+0000 7f77f4200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77ec108840 0x7f77ec101ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:06:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.489+0000 7f77f4200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f77ec10e790 con 0x7f77ec108840 2024-04-24T08:06:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.489+0000 7f77f4200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f77ec10e610 con 0x7f77ec1035e0 2024-04-24T08:06:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.489+0000 7f77f4200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f77ec10e910 con 0x7f77ec105f10 2024-04-24T08:06:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.490+0000 7f77f2200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77ec108840 0x7f77ec101ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:06:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.490+0000 7f77f0e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77ec105f10 0x7f77ec101590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:06:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.490+0000 7f77f2200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77ec108840 0x7f77ec101ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45606/0 (socket says 172.21.15.28:45606) 2024-04-24T08:06:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.490+0000 7f77f2200700 1 -- 172.21.15.28:0/2468193813 learned_addr learned my addr 172.21.15.28:0/2468193813 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:06:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.490+0000 7f77f0e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77ec105f10 0x7f77ec101590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:55566/0 (socket says 172.21.15.28:55566) 2024-04-24T08:06:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.490+0000 7f77f1800700 1 --2- 172.21.15.28:0/2468193813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77ec1035e0 0x7f77ec102f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:06:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.490+0000 7f77f0e00700 1 -- 172.21.15.28:0/2468193813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77ec1035e0 msgr2=0x7f77ec102f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.490+0000 7f77f0e00700 1 --2- 172.21.15.28:0/2468193813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77ec1035e0 0x7f77ec102f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.490+0000 7f77f0e00700 1 -- 172.21.15.28:0/2468193813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77ec108840 msgr2=0x7f77ec101ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.490+0000 7f77f0e00700 1 --2- 172.21.15.28:0/2468193813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77ec108840 0x7f77ec101ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.490+0000 7f77f0e00700 1 -- 172.21.15.28:0/2468193813 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f77ec102380 con 0x7f77ec105f10 2024-04-24T08:06:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.490+0000 7f77f0e00700 1 --2- 172.21.15.28:0/2468193813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77ec105f10 0x7f77ec101590 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f77d800e9f0 tx=0x7f77d800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:06:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.491+0000 7f77ea000700 1 -- 172.21.15.28:0/2468193813 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d8004220 con 0x7f77ec105f10 2024-04-24T08:06:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.491+0000 7f77ea000700 1 -- 172.21.15.28:0/2468193813 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f77d8004d10 con 0x7f77ec105f10 2024-04-24T08:06:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.491+0000 7f77f4200700 1 -- 172.21.15.28:0/2468193813 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f77ec1a2f60 con 0x7f77ec105f10 2024-04-24T08:06:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.491+0000 7f77ea000700 1 -- 172.21.15.28:0/2468193813 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77d8005020 con 0x7f77ec105f10 2024-04-24T08:06:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.491+0000 7f77f4200700 1 -- 172.21.15.28:0/2468193813 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f77ec1a33c0 con 0x7f77ec105f10 2024-04-24T08:06:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.492+0000 7f77f4200700 1 -- 172.21.15.28:0/2468193813 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f77ec0697e0 con 0x7f77ec105f10 2024-04-24T08:06:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.494+0000 7f77ea000700 1 -- 172.21.15.28:0/2468193813 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f77d8004750 con 0x7f77ec105f10 2024-04-24T08:06:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.495+0000 7f77ea000700 1 --2- 172.21.15.28:0/2468193813 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f77d40a3c40 0x7f77d40a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:06:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.495+0000 7f77ea000700 1 -- 172.21.15.28:0/2468193813 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 6847+0+0 (secure 0 0 0) 0x7f77d80cfd70 con 0x7f77ec105f10 2024-04-24T08:06:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.496+0000 7f77f1800700 1 --2- 172.21.15.28:0/2468193813 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f77d40a3c40 0x7f77d40a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:06:54.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.496+0000 7f77f1800700 1 --2- 172.21.15.28:0/2468193813 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f77d40a3c40 0x7f77d40a6100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f77e000b7b0 tx=0x7f77e000b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:06:54.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.499+0000 7f77ea000700 1 -- 172.21.15.28:0/2468193813 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f77d809cc70 con 0x7f77ec105f10 2024-04-24T08:06:54.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:54 smithi117 ceph-mon[32438]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi194", "root=default"]}]': finished 2024-04-24T08:06:54.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:54 smithi117 ceph-mon[32438]: osdmap e80: 12 total, 11 up, 12 in 2024-04-24T08:06:54.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:54 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:54.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:54 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:54.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:54 smithi117 ceph-mon[32438]: from='osd.11 ' entity='osd.11' 2024-04-24T08:06:54.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.740+0000 7f77f4200700 1 -- 172.21.15.28:0/2468193813 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f77ec063700 con 0x7f77ec105f10 2024-04-24T08:06:54.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.745+0000 7f77ea000700 1 -- 172.21.15.28:0/2468193813 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) ==== 74+0+133 (secure 0 0 0) 0x7f77d80cf090 con 0x7f77ec105f10 2024-04-24T08:06:54.747 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:06:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.749+0000 7f77f4200700 1 -- 172.21.15.28:0/2468193813 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f77d40a3c40 msgr2=0x7f77d40a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.749+0000 7f77f4200700 1 --2- 172.21.15.28:0/2468193813 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f77d40a3c40 0x7f77d40a6100 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f77e000b7b0 tx=0x7f77e000b880 comp rx=0 tx=0).stop 2024-04-24T08:06:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.749+0000 7f77f4200700 1 -- 172.21.15.28:0/2468193813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77ec105f10 msgr2=0x7f77ec101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:54.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.749+0000 7f77f4200700 1 --2- 172.21.15.28:0/2468193813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77ec105f10 0x7f77ec101590 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f77d800e9f0 tx=0x7f77d800ed00 comp rx=0 tx=0).stop 2024-04-24T08:06:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.749+0000 7f77f4200700 1 -- 172.21.15.28:0/2468193813 shutdown_connections 2024-04-24T08:06:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.749+0000 7f77f4200700 1 --2- 172.21.15.28:0/2468193813 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f77d40a3c40 0x7f77d40a6100 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.749+0000 7f77f4200700 1 --2- 172.21.15.28:0/2468193813 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f77ec108840 0x7f77ec101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.749+0000 7f77f4200700 1 --2- 172.21.15.28:0/2468193813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77ec1035e0 0x7f77ec102f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.749+0000 7f77f4200700 1 --2- 172.21.15.28:0/2468193813 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f77ec105f10 0x7f77ec101590 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.749+0000 7f77f4200700 1 -- 172.21.15.28:0/2468193813 >> 172.21.15.28:0/2468193813 conn(0x7f77ec0fcff0 msgr2=0x7f77ec104020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:06:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.749+0000 7f77f4200700 1 -- 172.21.15.28:0/2468193813 shutdown_connections 2024-04-24T08:06:54.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:54.749+0000 7f77f4200700 1 -- 172.21.15.28:0/2468193813 wait complete. 2024-04-24T08:06:55.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:55 smithi194 ceph-mon[32836]: purged_snaps scrub starts 2024-04-24T08:06:55.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:55 smithi194 ceph-mon[32836]: purged_snaps scrub ok 2024-04-24T08:06:55.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:55 smithi194 ceph-mon[32836]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:55.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:55 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:55.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:55 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2468193813' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-24T08:06:55.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:55 smithi194 ceph-mon[32836]: osd.11 [v2:172.21.15.194:6824/275366773,v1:172.21.15.194:6825/275366773] boot 2024-04-24T08:06:55.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:55 smithi194 ceph-mon[32836]: osdmap e81: 12 total, 12 up, 12 in 2024-04-24T08:06:55.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:55 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:55.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:55 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:55.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:55 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:55.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:55 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:55.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:55 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:55.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:55 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:55.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:55 smithi028 ceph-mon[27786]: purged_snaps scrub starts 2024-04-24T08:06:55.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:55 smithi028 ceph-mon[27786]: purged_snaps scrub ok 2024-04-24T08:06:55.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:55 smithi028 ceph-mon[27786]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:55.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:55 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:55.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:55 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2468193813' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-24T08:06:55.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:55 smithi028 ceph-mon[27786]: osd.11 [v2:172.21.15.194:6824/275366773,v1:172.21.15.194:6825/275366773] boot 2024-04-24T08:06:55.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:55 smithi028 ceph-mon[27786]: osdmap e81: 12 total, 12 up, 12 in 2024-04-24T08:06:55.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:55 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:55.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:55 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:55.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:55 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:55.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:55 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:55.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:55 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:55.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:55 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:55.615 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1713946014,"num_in_osds":12,"osd_in_since":1713945994,"num_remapped_pgs":0} 2024-04-24T08:06:55.616 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-24T08:06:55.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:55 smithi117 ceph-mon[32438]: purged_snaps scrub starts 2024-04-24T08:06:55.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:55 smithi117 ceph-mon[32438]: purged_snaps scrub ok 2024-04-24T08:06:55.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:55 smithi117 ceph-mon[32438]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-04-24T08:06:55.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:55 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:55.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:55 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2468193813' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-04-24T08:06:55.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:55 smithi117 ceph-mon[32438]: osd.11 [v2:172.21.15.194:6824/275366773,v1:172.21.15.194:6825/275366773] boot 2024-04-24T08:06:55.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:55 smithi117 ceph-mon[32438]: osdmap e81: 12 total, 12 up, 12 in 2024-04-24T08:06:55.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:55 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:06:55.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:55 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:55.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:55 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:55.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:55 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:55.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:55 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:55.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:55 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:56.054 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:06:56.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:56 smithi028 ceph-mon[27786]: Detected new or changed devices on smithi194 2024-04-24T08:06:56.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:56 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:56.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:56 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi194 to 4272M 2024-04-24T08:06:56.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:56 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:56.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:56 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:56.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:56 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:56.521 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:56 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:56.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:56 smithi117 ceph-mon[32438]: Detected new or changed devices on smithi194 2024-04-24T08:06:56.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:56 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:56.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:56 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi194 to 4272M 2024-04-24T08:06:56.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:56 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:56.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:56 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:56.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:56 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:56.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:56 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:56.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:56 smithi194 ceph-mon[32836]: Detected new or changed devices on smithi194 2024-04-24T08:06:56.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:56 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:06:56.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:56 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi194 to 4272M 2024-04-24T08:06:56.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:56 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:56.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:56 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:06:56.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:56 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:06:56.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:56 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:06:57.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:57 smithi028 ceph-mon[27786]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:06:57.432 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:57 smithi028 ceph-mon[27786]: osdmap e82: 12 total, 12 up, 12 in 2024-04-24T08:06:57.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:57 smithi117 ceph-mon[32438]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:06:57.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:57 smithi117 ceph-mon[32438]: osdmap e82: 12 total, 12 up, 12 in 2024-04-24T08:06:57.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:57 smithi194 ceph-mon[32836]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:06:57.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:57 smithi194 ceph-mon[32836]: osdmap e82: 12 total, 12 up, 12 in 2024-04-24T08:06:58.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.294+0000 7fbbf0c00700 1 -- 172.21.15.28:0/1457590411 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbec075730 msgr2=0x7fbbec10ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.294+0000 7fbbf0c00700 1 --2- 172.21.15.28:0/1457590411 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbec075730 0x7fbbec10ed80 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fbbd8009990 tx=0x7fbbd8009ca0 comp rx=0 tx=0).stop 2024-04-24T08:06:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.294+0000 7fbbf0c00700 1 -- 172.21.15.28:0/1457590411 shutdown_connections 2024-04-24T08:06:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.294+0000 7fbbf0c00700 1 --2- 172.21.15.28:0/1457590411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec10f2c0 0x7fbbec1116b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.294+0000 7fbbf0c00700 1 --2- 172.21.15.28:0/1457590411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbec074e10 0x7fbbec0751f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.294+0000 7fbbf0c00700 1 --2- 172.21.15.28:0/1457590411 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbec075730 0x7fbbec10ed80 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.294+0000 7fbbf0c00700 1 -- 172.21.15.28:0/1457590411 >> 172.21.15.28:0/1457590411 conn(0x7fbbec0fe510 msgr2=0x7fbbec100930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:06:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.295+0000 7fbbf0c00700 1 -- 172.21.15.28:0/1457590411 shutdown_connections 2024-04-24T08:06:58.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.295+0000 7fbbf0c00700 1 -- 172.21.15.28:0/1457590411 wait complete. 2024-04-24T08:06:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.296+0000 7fbbf0c00700 1 Processor -- start 2024-04-24T08:06:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.296+0000 7fbbf0c00700 1 -- start start 2024-04-24T08:06:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.296+0000 7fbbf0c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbec074e10 0x7fbbec19b3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:06:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.296+0000 7fbbf0c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbec075730 0x7fbbec19b930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:06:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.296+0000 7fbbf0c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec10f2c0 0x7fbbec19fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:06:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.296+0000 7fbbf0c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec114f60 con 0x7fbbec074e10 2024-04-24T08:06:58.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.296+0000 7fbbf0c00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec114de0 con 0x7fbbec10f2c0 2024-04-24T08:06:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.296+0000 7fbbf0c00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbbec1150e0 con 0x7fbbec075730 2024-04-24T08:06:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.297+0000 7fbbeb400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbec074e10 0x7fbbec19b3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:06:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.297+0000 7fbbeb400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbec074e10 0x7fbbec19b3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45630/0 (socket says 172.21.15.28:45630) 2024-04-24T08:06:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.297+0000 7fbbeb400700 1 -- 172.21.15.28:0/2983543848 learned_addr learned my addr 172.21.15.28:0/2983543848 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:06:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.297+0000 7fbbeaa00700 1 --2- 172.21.15.28:0/2983543848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbec075730 0x7fbbec19b930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:06:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.297+0000 7fbbebe00700 1 --2- 172.21.15.28:0/2983543848 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec10f2c0 0x7fbbec19fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:06:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.297+0000 7fbbeb400700 1 -- 172.21.15.28:0/2983543848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbec075730 msgr2=0x7fbbec19b930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.297+0000 7fbbeb400700 1 --2- 172.21.15.28:0/2983543848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbec075730 0x7fbbec19b930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.297+0000 7fbbeb400700 1 -- 172.21.15.28:0/2983543848 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec10f2c0 msgr2=0x7fbbec19fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.297+0000 7fbbeb400700 1 --2- 172.21.15.28:0/2983543848 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec10f2c0 0x7fbbec19fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.297+0000 7fbbeb400700 1 -- 172.21.15.28:0/2983543848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbbec1a0400 con 0x7fbbec074e10 2024-04-24T08:06:58.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.297+0000 7fbbeb400700 1 --2- 172.21.15.28:0/2983543848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbec074e10 0x7fbbec19b3f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00e9a0 tx=0x7fbbdc00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:06:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.298+0000 7fbbe3e00700 1 -- 172.21.15.28:0/2983543848 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc00cbd0 con 0x7fbbec074e10 2024-04-24T08:06:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.298+0000 7fbbe3e00700 1 -- 172.21.15.28:0/2983543848 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbbdc00cd70 con 0x7fbbec074e10 2024-04-24T08:06:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.298+0000 7fbbe3e00700 1 -- 172.21.15.28:0/2983543848 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbbdc018750 con 0x7fbbec074e10 2024-04-24T08:06:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.298+0000 7fbbf0c00700 1 -- 172.21.15.28:0/2983543848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbbec1a06f0 con 0x7fbbec074e10 2024-04-24T08:06:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.298+0000 7fbbf0c00700 1 -- 172.21.15.28:0/2983543848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbbec1a7f70 con 0x7fbbec074e10 2024-04-24T08:06:58.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.299+0000 7fbbf0c00700 1 -- 172.21.15.28:0/2983543848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbbec0697e0 con 0x7fbbec074e10 2024-04-24T08:06:58.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.301+0000 7fbbe3e00700 1 -- 172.21.15.28:0/2983543848 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbbdc010430 con 0x7fbbec074e10 2024-04-24T08:06:58.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.303+0000 7fbbe3e00700 1 --2- 172.21.15.28:0/2983543848 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbbd00a3cc0 0x7fbbd00a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:06:58.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.303+0000 7fbbe3e00700 1 -- 172.21.15.28:0/2983543848 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fbbdc0d0660 con 0x7fbbec074e10 2024-04-24T08:06:58.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.303+0000 7fbbeaa00700 1 --2- 172.21.15.28:0/2983543848 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbbd00a3cc0 0x7fbbd00a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:06:58.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.304+0000 7fbbeaa00700 1 --2- 172.21.15.28:0/2983543848 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbbd00a3cc0 0x7fbbd00a6180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbbec1105e0 tx=0x7fbbd8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:06:58.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.306+0000 7fbbe3e00700 1 -- 172.21.15.28:0/2983543848 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbbdc09d440 con 0x7fbbec074e10 2024-04-24T08:06:58.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.544+0000 7fbbf0c00700 1 -- 172.21.15.28:0/2983543848 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbbec02dfe0 con 0x7fbbec074e10 2024-04-24T08:06:58.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.547+0000 7fbbe3e00700 1 -- 172.21.15.28:0/2983543848 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18444 (secure 0 0 0) 0x7fbbdc09cd40 con 0x7fbbec074e10 2024-04-24T08:06:58.549 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:06:58.549 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":82,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:06:56.246184+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:06:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.550+0000 7fbbf0c00700 1 -- 172.21.15.28:0/2983543848 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbbd00a3cc0 msgr2=0x7fbbd00a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.550+0000 7fbbf0c00700 1 --2- 172.21.15.28:0/2983543848 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbbd00a3cc0 0x7fbbd00a6180 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbbec1105e0 tx=0x7fbbd8005fd0 comp rx=0 tx=0).stop 2024-04-24T08:06:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.550+0000 7fbbf0c00700 1 -- 172.21.15.28:0/2983543848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbec074e10 msgr2=0x7fbbec19b3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:06:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.550+0000 7fbbf0c00700 1 --2- 172.21.15.28:0/2983543848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbec074e10 0x7fbbec19b3f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fbbdc00e9a0 tx=0x7fbbdc00ecb0 comp rx=0 tx=0).stop 2024-04-24T08:06:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.551+0000 7fbbf0c00700 1 -- 172.21.15.28:0/2983543848 shutdown_connections 2024-04-24T08:06:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.551+0000 7fbbf0c00700 1 --2- 172.21.15.28:0/2983543848 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbbd00a3cc0 0x7fbbd00a6180 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.551+0000 7fbbf0c00700 1 --2- 172.21.15.28:0/2983543848 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbbec10f2c0 0x7fbbec19fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:58.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.551+0000 7fbbf0c00700 1 --2- 172.21.15.28:0/2983543848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbbec074e10 0x7fbbec19b3f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.551+0000 7fbbf0c00700 1 --2- 172.21.15.28:0/2983543848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbbec075730 0x7fbbec19b930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:06:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.551+0000 7fbbf0c00700 1 -- 172.21.15.28:0/2983543848 >> 172.21.15.28:0/2983543848 conn(0x7fbbec0fe510 msgr2=0x7fbbec0ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:06:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.551+0000 7fbbf0c00700 1 -- 172.21.15.28:0/2983543848 shutdown_connections 2024-04-24T08:06:58.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:06:58.551+0000 7fbbf0c00700 1 -- 172.21.15.28:0/2983543848 wait complete. 2024-04-24T08:06:59.346 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:02:34.086371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:06:59.346 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-04-24T08:06:59.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:59 smithi028 ceph-mon[27786]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:06:59.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:06:59 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2983543848' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:06:59.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:59 smithi117 ceph-mon[32438]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:06:59.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:06:59 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2983543848' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:06:59.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:59 smithi194 ceph-mon[32836]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:06:59.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:06:59 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2983543848' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:06:59.811 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:07:01.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:01 smithi028 ceph-mon[27786]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:01.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:01 smithi117 ceph-mon[32438]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:01.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:01 smithi194 ceph-mon[32836]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:02.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.005+0000 7efd5da00700 1 -- 172.21.15.28:0/2646135404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd58074e90 msgr2=0x7efd58075270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:02.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.005+0000 7efd5da00700 1 --2- 172.21.15.28:0/2646135404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd58074e90 0x7efd58075270 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7efd48009990 tx=0x7efd48009ca0 comp rx=0 tx=0).stop 2024-04-24T08:07:02.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.005+0000 7efd5da00700 1 -- 172.21.15.28:0/2646135404 shutdown_connections 2024-04-24T08:07:02.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.005+0000 7efd5da00700 1 --2- 172.21.15.28:0/2646135404 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd58113a30 0x7efd58115e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:02.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.005+0000 7efd5da00700 1 --2- 172.21.15.28:0/2646135404 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd58074e90 0x7efd58075270 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:02.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.005+0000 7efd5da00700 1 --2- 172.21.15.28:0/2646135404 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd580757b0 0x7efd581134f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:02.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.005+0000 7efd5da00700 1 -- 172.21.15.28:0/2646135404 >> 172.21.15.28:0/2646135404 conn(0x7efd580fe570 msgr2=0x7efd58100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:02.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.005+0000 7efd5da00700 1 -- 172.21.15.28:0/2646135404 shutdown_connections 2024-04-24T08:07:02.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.005+0000 7efd5da00700 1 -- 172.21.15.28:0/2646135404 wait complete. 2024-04-24T08:07:02.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.006+0000 7efd5da00700 1 Processor -- start 2024-04-24T08:07:02.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.006+0000 7efd5da00700 1 -- start start 2024-04-24T08:07:02.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.007+0000 7efd5da00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd58074e90 0x7efd581a5430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:02.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.007+0000 7efd5da00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd580757b0 0x7efd581a5970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:02.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.007+0000 7efd5da00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd58113a30 0x7efd5819f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:02.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.007+0000 7efd5da00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd58119350 con 0x7efd58074e90 2024-04-24T08:07:02.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.007+0000 7efd5da00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efd581191d0 con 0x7efd58113a30 2024-04-24T08:07:02.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.007+0000 7efd5da00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efd581194d0 con 0x7efd580757b0 2024-04-24T08:07:02.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.007+0000 7efd4f400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd580757b0 0x7efd581a5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:02.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.007+0000 7efd4f400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd580757b0 0x7efd581a5970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:33674/0 (socket says 172.21.15.28:33674) 2024-04-24T08:07:02.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.007+0000 7efd4f400700 1 -- 172.21.15.28:0/953140336 learned_addr learned my addr 172.21.15.28:0/953140336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:02.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.007+0000 7efd57e00700 1 --2- 172.21.15.28:0/953140336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd58074e90 0x7efd581a5430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.008+0000 7efd5d000700 1 --2- 172.21.15.28:0/953140336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd58113a30 0x7efd5819f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.008+0000 7efd4f400700 1 -- 172.21.15.28:0/953140336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd58113a30 msgr2=0x7efd5819f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.008+0000 7efd4f400700 1 --2- 172.21.15.28:0/953140336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd58113a30 0x7efd5819f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.008+0000 7efd4f400700 1 -- 172.21.15.28:0/953140336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd58074e90 msgr2=0x7efd581a5430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.008+0000 7efd4f400700 1 --2- 172.21.15.28:0/953140336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd58074e90 0x7efd581a5430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.008+0000 7efd4f400700 1 -- 172.21.15.28:0/953140336 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd5819fd50 con 0x7efd580757b0 2024-04-24T08:07:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.008+0000 7efd5d000700 1 --2- 172.21.15.28:0/953140336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd58113a30 0x7efd5819f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:07:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.008+0000 7efd4f400700 1 --2- 172.21.15.28:0/953140336 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd580757b0 0x7efd581a5970 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efd4000eb30 tx=0x7efd4000ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:02.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.009+0000 7efd55600700 1 -- 172.21.15.28:0/953140336 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd4000cc90 con 0x7efd580757b0 2024-04-24T08:07:02.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.009+0000 7efd55600700 1 -- 172.21.15.28:0/953140336 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd4000ce30 con 0x7efd580757b0 2024-04-24T08:07:02.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.009+0000 7efd55600700 1 -- 172.21.15.28:0/953140336 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd40018840 con 0x7efd580757b0 2024-04-24T08:07:02.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.009+0000 7efd5da00700 1 -- 172.21.15.28:0/953140336 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd581a0040 con 0x7efd580757b0 2024-04-24T08:07:02.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.009+0000 7efd5da00700 1 -- 172.21.15.28:0/953140336 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd581ac300 con 0x7efd580757b0 2024-04-24T08:07:02.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.010+0000 7efd5da00700 1 -- 172.21.15.28:0/953140336 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd580697e0 con 0x7efd580757b0 2024-04-24T08:07:02.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.012+0000 7efd55600700 1 -- 172.21.15.28:0/953140336 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7efd4001f020 con 0x7efd580757b0 2024-04-24T08:07:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.013+0000 7efd55600700 1 --2- 172.21.15.28:0/953140336 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd3c0a3cc0 0x7efd3c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.014+0000 7efd55600700 1 -- 172.21.15.28:0/953140336 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7efd400cff80 con 0x7efd580757b0 2024-04-24T08:07:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.014+0000 7efd57e00700 1 --2- 172.21.15.28:0/953140336 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd3c0a3cc0 0x7efd3c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:02.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.014+0000 7efd57e00700 1 --2- 172.21.15.28:0/953140336 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd3c0a3cc0 0x7efd3c0a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efd480098e0 tx=0x7efd48005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:02.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.017+0000 7efd55600700 1 -- 172.21.15.28:0/953140336 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efd4009cd60 con 0x7efd580757b0 2024-04-24T08:07:02.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.247+0000 7efd5da00700 1 -- 172.21.15.28:0/953140336 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7efd5802dfe0 con 0x7efd580757b0 2024-04-24T08:07:02.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.248+0000 7efd55600700 1 -- 172.21.15.28:0/953140336 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7efd4009c660 con 0x7efd580757b0 2024-04-24T08:07:02.249 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-24T08:07:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.251+0000 7efd5da00700 1 -- 172.21.15.28:0/953140336 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd3c0a3cc0 msgr2=0x7efd3c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.251+0000 7efd5da00700 1 --2- 172.21.15.28:0/953140336 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd3c0a3cc0 0x7efd3c0a6180 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7efd480098e0 tx=0x7efd48005960 comp rx=0 tx=0).stop 2024-04-24T08:07:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.251+0000 7efd5da00700 1 -- 172.21.15.28:0/953140336 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd580757b0 msgr2=0x7efd581a5970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.251+0000 7efd5da00700 1 --2- 172.21.15.28:0/953140336 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd580757b0 0x7efd581a5970 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7efd4000eb30 tx=0x7efd4000ee40 comp rx=0 tx=0).stop 2024-04-24T08:07:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.251+0000 7efd5da00700 1 -- 172.21.15.28:0/953140336 shutdown_connections 2024-04-24T08:07:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.251+0000 7efd5da00700 1 --2- 172.21.15.28:0/953140336 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd3c0a3cc0 0x7efd3c0a6180 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.251+0000 7efd5da00700 1 --2- 172.21.15.28:0/953140336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd58113a30 0x7efd5819f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:02.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.251+0000 7efd5da00700 1 --2- 172.21.15.28:0/953140336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd58074e90 0x7efd581a5430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:02.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.251+0000 7efd5da00700 1 --2- 172.21.15.28:0/953140336 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd580757b0 0x7efd581a5970 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:02.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.251+0000 7efd5da00700 1 -- 172.21.15.28:0/953140336 >> 172.21.15.28:0/953140336 conn(0x7efd580fe570 msgr2=0x7efd580ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:02.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.252+0000 7efd5da00700 1 -- 172.21.15.28:0/953140336 shutdown_connections 2024-04-24T08:07:02.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:02.252+0000 7efd5da00700 1 -- 172.21.15.28:0/953140336 wait complete. 2024-04-24T08:07:02.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:02 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/953140336' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:07:02.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:02 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/953140336' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:07:02.678 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:02 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/953140336' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:07:03.075 INFO:tasks.cephadm:Adding mds.a on smithi028 2024-04-24T08:07:03.075 INFO:tasks.cephadm:Adding mds.d on smithi028 2024-04-24T08:07:03.075 INFO:tasks.cephadm:Adding mds.g on smithi028 2024-04-24T08:07:03.075 INFO:tasks.cephadm:Adding mds.j on smithi028 2024-04-24T08:07:03.075 INFO:tasks.cephadm:Adding mds.b on smithi117 2024-04-24T08:07:03.075 INFO:tasks.cephadm:Adding mds.e on smithi117 2024-04-24T08:07:03.076 INFO:tasks.cephadm:Adding mds.h on smithi117 2024-04-24T08:07:03.076 INFO:tasks.cephadm:Adding mds.k on smithi117 2024-04-24T08:07:03.076 INFO:tasks.cephadm:Adding mds.c on smithi194 2024-04-24T08:07:03.076 INFO:tasks.cephadm:Adding mds.f on smithi194 2024-04-24T08:07:03.076 INFO:tasks.cephadm:Adding mds.i on smithi194 2024-04-24T08:07:03.076 INFO:tasks.cephadm:Adding mds.l on smithi194 2024-04-24T08:07:03.076 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph orch apply mds all '12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l' 2024-04-24T08:07:03.314 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:07:03.435 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:03 smithi194 ceph-mon[32836]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:03.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:03 smithi028 ceph-mon[27786]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:03.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:03 smithi117 ceph-mon[32438]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:04.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.960+0000 7fd8b864e700 1 -- 172.21.15.194:0/3541406075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8b0102c50 msgr2=0x7fd8b01030b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:04.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.960+0000 7fd8b864e700 1 --2- 172.21.15.194:0/3541406075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8b0102c50 0x7fd8b01030b0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac009990 tx=0x7fd8ac009ca0 comp rx=0 tx=0).stop 2024-04-24T08:07:04.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.960+0000 7fd8b864e700 1 -- 172.21.15.194:0/3541406075 shutdown_connections 2024-04-24T08:07:04.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.960+0000 7fd8b864e700 1 --2- 172.21.15.194:0/3541406075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8b01035f0 0x7fd8b0109ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:04.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.960+0000 7fd8b864e700 1 --2- 172.21.15.194:0/3541406075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8b0101a50 0x7fd8b0101e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:04.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.960+0000 7fd8b864e700 1 --2- 172.21.15.194:0/3541406075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8b0102c50 0x7fd8b01030b0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:04.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.960+0000 7fd8b864e700 1 -- 172.21.15.194:0/3541406075 >> 172.21.15.194:0/3541406075 conn(0x7fd8b00fcff0 msgr2=0x7fd8b00ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:04.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.960+0000 7fd8b864e700 1 -- 172.21.15.194:0/3541406075 shutdown_connections 2024-04-24T08:07:04.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.960+0000 7fd8b864e700 1 -- 172.21.15.194:0/3541406075 wait complete. 2024-04-24T08:07:04.963 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.960+0000 7fd8b864e700 1 Processor -- start 2024-04-24T08:07:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.960+0000 7fd8b864e700 1 -- start start 2024-04-24T08:07:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.960+0000 7fd8b864e700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8b0101a50 0x7fd8b0105990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.960+0000 7fd8b864e700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8b0102c50 0x7fd8b0104030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.960+0000 7fd8b864e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8b01035f0 0x7fd8b0104570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.960+0000 7fd8b864e700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b010d730 con 0x7fd8b0102c50 2024-04-24T08:07:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.960+0000 7fd8b864e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b010d5b0 con 0x7fd8b01035f0 2024-04-24T08:07:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.960+0000 7fd8b864e700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd8b010d8b0 con 0x7fd8b0101a50 2024-04-24T08:07:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.961+0000 7fd8b63ea700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8b0101a50 0x7fd8b0105990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.961+0000 7fd8b63ea700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8b0101a50 0x7fd8b0105990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.194:36766/0 (socket says 172.21.15.194:36766) 2024-04-24T08:07:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.961+0000 7fd8b63ea700 1 -- 172.21.15.194:0/599105920 learned_addr learned my addr 172.21.15.194:0/599105920 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-04-24T08:07:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.961+0000 7fd8b6beb700 1 --2- 172.21.15.194:0/599105920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8b01035f0 0x7fd8b0104570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.961+0000 7fd8b63ea700 1 -- 172.21.15.194:0/599105920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8b01035f0 msgr2=0x7fd8b0104570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.961+0000 7fd8b63ea700 1 --2- 172.21.15.194:0/599105920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8b01035f0 0x7fd8b0104570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.961+0000 7fd8b63ea700 1 -- 172.21.15.194:0/599105920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8b0102c50 msgr2=0x7fd8b0104030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:04.964 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.961+0000 7fd8b63ea700 1 --2- 172.21.15.194:0/599105920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8b0102c50 0x7fd8b0104030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:04.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.961+0000 7fd8b63ea700 1 -- 172.21.15.194:0/599105920 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8b0104e40 con 0x7fd8b0101a50 2024-04-24T08:07:04.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.961+0000 7fd8b63ea700 1 --2- 172.21.15.194:0/599105920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8b0101a50 0x7fd8b0105990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000b500 tx=0x7fd8a000b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:04.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.961+0000 7fd8a77fe700 1 -- 172.21.15.194:0/599105920 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a0015020 con 0x7fd8b0101a50 2024-04-24T08:07:04.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.961+0000 7fd8b864e700 1 -- 172.21.15.194:0/599105920 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8b01a2fd0 con 0x7fd8b0101a50 2024-04-24T08:07:04.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.961+0000 7fd8a77fe700 1 -- 172.21.15.194:0/599105920 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8a0011d10 con 0x7fd8b0101a50 2024-04-24T08:07:04.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.961+0000 7fd8a77fe700 1 -- 172.21.15.194:0/599105920 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8a000f2e0 con 0x7fd8b0101a50 2024-04-24T08:07:04.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.961+0000 7fd8b864e700 1 -- 172.21.15.194:0/599105920 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8b01a3450 con 0x7fd8b0101a50 2024-04-24T08:07:04.965 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.962+0000 7fd8b864e700 1 -- 172.21.15.194:0/599105920 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8b01a3710 con 0x7fd8b0101a50 2024-04-24T08:07:04.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.964+0000 7fd8a77fe700 1 -- 172.21.15.194:0/599105920 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fd8a0018020 con 0x7fd8b0101a50 2024-04-24T08:07:04.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.964+0000 7fd8a77fe700 1 --2- 172.21.15.194:0/599105920 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd89c0a3c40 0x7fd89c0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:04.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.964+0000 7fd8a77fe700 1 -- 172.21.15.194:0/599105920 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fd8a00cfdb0 con 0x7fd8b0101a50 2024-04-24T08:07:04.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.965+0000 7fd8b5be9700 1 --2- 172.21.15.194:0/599105920 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd89c0a3c40 0x7fd89c0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:04.966 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.966+0000 7fd8b5be9700 1 --2- 172.21.15.194:0/599105920 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd89c0a3c40 0x7fd89c0a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac005f50 tx=0x7fd8ac005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:04.968 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:04.968+0000 7fd8a77fe700 1 -- 172.21.15.194:0/599105920 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd8a001f020 con 0x7fd8b0101a50 2024-04-24T08:07:05.174 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:05.173+0000 7fd8b864e700 1 -- 172.21.15.194:0/599105920 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l", "target": ["mon-mgr", ""]}) -- 0x7fd8b00ff2f0 con 0x7fd89c0a3c40 2024-04-24T08:07:05.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:05.190+0000 7fd8a77fe700 1 -- 172.21.15.194:0/599105920 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fd8b00ff2f0 con 0x7fd89c0a3c40 2024-04-24T08:07:05.191 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mds.all update... 2024-04-24T08:07:05.193 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:05.193+0000 7fd8b864e700 1 -- 172.21.15.194:0/599105920 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd89c0a3c40 msgr2=0x7fd89c0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:05.193 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:05.193+0000 7fd8b864e700 1 --2- 172.21.15.194:0/599105920 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd89c0a3c40 0x7fd89c0a6100 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fd8ac005f50 tx=0x7fd8ac005e80 comp rx=0 tx=0).stop 2024-04-24T08:07:05.194 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:05.193+0000 7fd8b864e700 1 -- 172.21.15.194:0/599105920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8b0101a50 msgr2=0x7fd8b0105990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:05.194 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:05.193+0000 7fd8b864e700 1 --2- 172.21.15.194:0/599105920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8b0101a50 0x7fd8b0105990 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd8a000b500 tx=0x7fd8a000b8c0 comp rx=0 tx=0).stop 2024-04-24T08:07:05.194 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:05.193+0000 7fd8b864e700 1 -- 172.21.15.194:0/599105920 shutdown_connections 2024-04-24T08:07:05.194 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:05.193+0000 7fd8b864e700 1 --2- 172.21.15.194:0/599105920 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd89c0a3c40 0x7fd89c0a6100 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:05.194 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:05.193+0000 7fd8b864e700 1 --2- 172.21.15.194:0/599105920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8b01035f0 0x7fd8b0104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:05.194 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:05.193+0000 7fd8b864e700 1 --2- 172.21.15.194:0/599105920 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd8b0101a50 0x7fd8b0105990 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:05.194 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:05.193+0000 7fd8b864e700 1 --2- 172.21.15.194:0/599105920 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd8b0102c50 0x7fd8b0104030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:05.194 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:05.193+0000 7fd8b864e700 1 -- 172.21.15.194:0/599105920 >> 172.21.15.194:0/599105920 conn(0x7fd8b00fcff0 msgr2=0x7fd8b00feae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:05.194 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:05.193+0000 7fd8b864e700 1 -- 172.21.15.194:0/599105920 shutdown_connections 2024-04-24T08:07:05.194 INFO:teuthology.orchestra.run.smithi194.stderr:2024-04-24T08:07:05.193+0000 7fd8b864e700 1 -- 172.21.15.194:0/599105920 wait complete. 2024-04-24T08:07:05.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:05 smithi194 ceph-mon[32836]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:05.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:05 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:05.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:05 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:07:05.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:05 smithi028 ceph-mon[27786]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:05.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:05.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:05 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:07:05.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:05 smithi117 ceph-mon[32438]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:05.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:05 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:05.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:05 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:07:05.863 DEBUG:teuthology.orchestra.run.smithi028:mds.a> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.a.service 2024-04-24T08:07:05.871 DEBUG:teuthology.orchestra.run.smithi028:mds.d> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.d.service 2024-04-24T08:07:05.917 DEBUG:teuthology.orchestra.run.smithi028:mds.g> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.g.service 2024-04-24T08:07:05.964 DEBUG:teuthology.orchestra.run.smithi028:mds.j> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.j.service 2024-04-24T08:07:06.012 DEBUG:teuthology.orchestra.run.smithi117:mds.b> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.b.service 2024-04-24T08:07:06.015 DEBUG:teuthology.orchestra.run.smithi117:mds.e> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.e.service 2024-04-24T08:07:06.057 DEBUG:teuthology.orchestra.run.smithi117:mds.h> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.h.service 2024-04-24T08:07:06.100 DEBUG:teuthology.orchestra.run.smithi117:mds.k> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.k.service 2024-04-24T08:07:06.144 DEBUG:teuthology.orchestra.run.smithi194:mds.c> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.c.service 2024-04-24T08:07:06.147 DEBUG:teuthology.orchestra.run.smithi194:mds.f> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.f.service 2024-04-24T08:07:06.189 DEBUG:teuthology.orchestra.run.smithi194:mds.i> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.i.service 2024-04-24T08:07:06.232 DEBUG:teuthology.orchestra.run.smithi194:mds.l> sudo journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.l.service 2024-04-24T08:07:06.275 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-04-24T08:07:06.275 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:07:06.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:06 smithi194 ceph-mon[32836]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:07:06.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:06 smithi194 ceph-mon[32836]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l;count:12 2024-04-24T08:07:06.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.495+0000 7ff541c00640 1 -- 172.21.15.28:0/3567218445 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff53c104080 msgr2=0x7ff53c1044a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:06.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.495+0000 7ff541c00640 1 --2- 172.21.15.28:0/3567218445 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff53c104080 0x7ff53c1044a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff52c009900 tx=0x7ff52c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:07:06.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.495+0000 7ff541c00640 1 -- 172.21.15.28:0/3567218445 shutdown_connections 2024-04-24T08:07:06.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.495+0000 7ff541c00640 1 --2- 172.21.15.28:0/3567218445 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff53c105c30 0x7ff53c10c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:06.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.495+0000 7ff541c00640 1 --2- 172.21.15.28:0/3567218445 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff53c105270 0x7ff53c1056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:06.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.495+0000 7ff541c00640 1 --2- 172.21.15.28:0/3567218445 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff53c104080 0x7ff53c1044a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:06.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.495+0000 7ff541c00640 1 -- 172.21.15.28:0/3567218445 >> 172.21.15.28:0/3567218445 conn(0x7ff53c0ff610 msgr2=0x7ff53c101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:06.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.495+0000 7ff541c00640 1 -- 172.21.15.28:0/3567218445 shutdown_connections 2024-04-24T08:07:06.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.495+0000 7ff541c00640 1 -- 172.21.15.28:0/3567218445 wait complete. 2024-04-24T08:07:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.496+0000 7ff541c00640 1 Processor -- start 2024-04-24T08:07:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.497+0000 7ff541c00640 1 -- start start 2024-04-24T08:07:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.497+0000 7ff541c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff53c104080 0x7ff53c19a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.497+0000 7ff541c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff53c105270 0x7ff53c19a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.497+0000 7ff541c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff53c105c30 0x7ff53c1a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.497+0000 7ff541c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff53c10fd60 con 0x7ff53c105270 2024-04-24T08:07:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.497+0000 7ff541c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff53c10fbe0 con 0x7ff53c105c30 2024-04-24T08:07:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.497+0000 7ff541c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff53c10fee0 con 0x7ff53c104080 2024-04-24T08:07:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.497+0000 7ff53be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff53c104080 0x7ff53c19a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.497+0000 7ff533400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff53c105270 0x7ff53c19a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.497+0000 7ff53be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff53c104080 0x7ff53c19a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:33694/0 (socket says 172.21.15.28:33694) 2024-04-24T08:07:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.497+0000 7ff53be00640 1 -- 172.21.15.28:0/3855113818 learned_addr learned my addr 172.21.15.28:0/3855113818 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:06.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.497+0000 7ff541200640 1 --2- 172.21.15.28:0/3855113818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff53c105c30 0x7ff53c1a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.498+0000 7ff533400640 1 -- 172.21.15.28:0/3855113818 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff53c104080 msgr2=0x7ff53c19a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.498+0000 7ff533400640 1 --2- 172.21.15.28:0/3855113818 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff53c104080 0x7ff53c19a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.498+0000 7ff533400640 1 -- 172.21.15.28:0/3855113818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff53c105c30 msgr2=0x7ff53c1a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.498+0000 7ff533400640 1 --2- 172.21.15.28:0/3855113818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff53c105c30 0x7ff53c1a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.498+0000 7ff533400640 1 -- 172.21.15.28:0/3855113818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff53c1a2000 con 0x7ff53c105270 2024-04-24T08:07:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.498+0000 7ff541200640 1 --2- 172.21.15.28:0/3855113818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff53c105c30 0x7ff53c1a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:06.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.498+0000 7ff533400640 1 --2- 172.21.15.28:0/3855113818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff53c105270 0x7ff53c19a920 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff52400e9f0 tx=0x7ff52400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.500+0000 7ff539600640 1 -- 172.21.15.28:0/3855113818 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff52400ccc0 con 0x7ff53c105270 2024-04-24T08:07:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.501+0000 7ff539600640 1 -- 172.21.15.28:0/3855113818 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff524005020 con 0x7ff53c105270 2024-04-24T08:07:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.501+0000 7ff541c00640 1 -- 172.21.15.28:0/3855113818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff53c1a22f0 con 0x7ff53c105270 2024-04-24T08:07:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.501+0000 7ff539600640 1 -- 172.21.15.28:0/3855113818 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5240049e0 con 0x7ff53c105270 2024-04-24T08:07:06.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.501+0000 7ff541c00640 1 -- 172.21.15.28:0/3855113818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff53c1a2840 con 0x7ff53c105270 2024-04-24T08:07:06.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.502+0000 7ff541c00640 1 -- 172.21.15.28:0/3855113818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff53c10b3a0 con 0x7ff53c105270 2024-04-24T08:07:06.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.504+0000 7ff539600640 1 -- 172.21.15.28:0/3855113818 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ff524005560 con 0x7ff53c105270 2024-04-24T08:07:06.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.505+0000 7ff539600640 1 --2- 172.21.15.28:0/3855113818 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ff5200a3d00 0x7ff5200a61c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:06.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.506+0000 7ff539600640 1 -- 172.21.15.28:0/3855113818 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ff5240cfe90 con 0x7ff53c105270 2024-04-24T08:07:06.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.506+0000 7ff53be00640 1 --2- 172.21.15.28:0/3855113818 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ff5200a3d00 0x7ff5200a61c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:06.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.506+0000 7ff53be00640 1 --2- 172.21.15.28:0/3855113818 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ff5200a3d00 0x7ff5200a61c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff52c000c00 tx=0x7ff52c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:06.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.510+0000 7ff539600640 1 -- 172.21.15.28:0/3855113818 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff5240051d0 con 0x7ff53c105270 2024-04-24T08:07:06.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:06 smithi028 ceph-mon[27786]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:07:06.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:06 smithi028 ceph-mon[27786]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l;count:12 2024-04-24T08:07:06.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:06 smithi117 ceph-mon[32438]: from='client.24442 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:07:06.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:06 smithi117 ceph-mon[32438]: Saving service mds.all spec with placement smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l;count:12 2024-04-24T08:07:06.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.716+0000 7ff541c00640 1 -- 172.21.15.28:0/3855113818 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff53c062c00 con 0x7ff53c105270 2024-04-24T08:07:06.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.718+0000 7ff539600640 1 -- 172.21.15.28:0/3855113818 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18444 (secure 0 0 0) 0x7ff5240cf090 con 0x7ff53c105270 2024-04-24T08:07:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:06.720 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":82,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:06:56.246184+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:07:06.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.721+0000 7ff541c00640 1 -- 172.21.15.28:0/3855113818 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ff5200a3d00 msgr2=0x7ff5200a61c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:06.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.721+0000 7ff541c00640 1 --2- 172.21.15.28:0/3855113818 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ff5200a3d00 0x7ff5200a61c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7ff52c000c00 tx=0x7ff52c0315b0 comp rx=0 tx=0).stop 2024-04-24T08:07:06.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.721+0000 7ff541c00640 1 -- 172.21.15.28:0/3855113818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff53c105270 msgr2=0x7ff53c19a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:06.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.721+0000 7ff541c00640 1 --2- 172.21.15.28:0/3855113818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff53c105270 0x7ff53c19a920 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7ff52400e9f0 tx=0x7ff52400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:07:06.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.721+0000 7ff541c00640 1 -- 172.21.15.28:0/3855113818 shutdown_connections 2024-04-24T08:07:06.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.721+0000 7ff541c00640 1 --2- 172.21.15.28:0/3855113818 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ff5200a3d00 0x7ff5200a61c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:06.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.721+0000 7ff541c00640 1 --2- 172.21.15.28:0/3855113818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff53c105270 0x7ff53c19a920 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:06.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.721+0000 7ff541c00640 1 --2- 172.21.15.28:0/3855113818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff53c105c30 0x7ff53c1a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:06.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.721+0000 7ff541c00640 1 --2- 172.21.15.28:0/3855113818 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff53c104080 0x7ff53c19a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:06.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.721+0000 7ff541c00640 1 -- 172.21.15.28:0/3855113818 >> 172.21.15.28:0/3855113818 conn(0x7ff53c0ff610 msgr2=0x7ff53c108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:06.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.721+0000 7ff541c00640 1 -- 172.21.15.28:0/3855113818 shutdown_connections 2024-04-24T08:07:06.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.721+0000 7ff541c00640 1 -- 172.21.15.28:0/3855113818 wait complete. 2024-04-24T08:07:06.744 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:02:34.086371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:07:06.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:07:06.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.956+0000 7f29f1800640 1 -- 172.21.15.28:0/1753032967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29ec102050 msgr2=0x7f29ec102470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:06.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.956+0000 7f29f1800640 1 --2- 172.21.15.28:0/1753032967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29ec102050 0x7f29ec102470 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f29dc009900 tx=0x7f29dc02a400 comp rx=0 tx=0).stop 2024-04-24T08:07:06.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.957+0000 7f29f1800640 1 -- 172.21.15.28:0/1753032967 shutdown_connections 2024-04-24T08:07:06.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.957+0000 7f29f1800640 1 --2- 172.21.15.28:0/1753032967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29ec103240 0x7f29ec1036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:06.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.957+0000 7f29f1800640 1 --2- 172.21.15.28:0/1753032967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ec103c00 0x7f29ec10a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:06.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.957+0000 7f29f1800640 1 --2- 172.21.15.28:0/1753032967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29ec102050 0x7f29ec102470 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:06.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.957+0000 7f29f1800640 1 -- 172.21.15.28:0/1753032967 >> 172.21.15.28:0/1753032967 conn(0x7f29ec0fd5e0 msgr2=0x7f29ec0ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:06.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.957+0000 7f29f1800640 1 -- 172.21.15.28:0/1753032967 shutdown_connections 2024-04-24T08:07:06.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.957+0000 7f29f1800640 1 -- 172.21.15.28:0/1753032967 wait complete. 2024-04-24T08:07:06.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.958+0000 7f29f1800640 1 Processor -- start 2024-04-24T08:07:06.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.958+0000 7f29f1800640 1 -- start start 2024-04-24T08:07:06.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.958+0000 7f29f1800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ec102050 0x7f29ec075db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:06.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.958+0000 7f29f1800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29ec103240 0x7f29ec0743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:06.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.958+0000 7f29f1800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29ec103c00 0x7f29ec074930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.958+0000 7f29f1800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec10dd30 con 0x7f29ec103c00 2024-04-24T08:07:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.958+0000 7f29f1800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec10dbb0 con 0x7f29ec102050 2024-04-24T08:07:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.958+0000 7f29f1800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f29ec10deb0 con 0x7f29ec103240 2024-04-24T08:07:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.959+0000 7f29ea000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29ec103240 0x7f29ec0743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.959+0000 7f29ea000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29ec103240 0x7f29ec0743f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:33716/0 (socket says 172.21.15.28:33716) 2024-04-24T08:07:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.959+0000 7f29ea000640 1 -- 172.21.15.28:0/3984685841 learned_addr learned my addr 172.21.15.28:0/3984685841 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.959+0000 7f29eaa00640 1 --2- 172.21.15.28:0/3984685841 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ec102050 0x7f29ec075db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.959+0000 7f29eb400640 1 --2- 172.21.15.28:0/3984685841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29ec103c00 0x7f29ec074930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.959+0000 7f29ea000640 1 -- 172.21.15.28:0/3984685841 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ec102050 msgr2=0x7f29ec075db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.959+0000 7f29ea000640 1 --2- 172.21.15.28:0/3984685841 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ec102050 0x7f29ec075db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:06.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.959+0000 7f29ea000640 1 -- 172.21.15.28:0/3984685841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29ec103c00 msgr2=0x7f29ec074930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.959+0000 7f29ea000640 1 --2- 172.21.15.28:0/3984685841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29ec103c00 0x7f29ec074930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.959+0000 7f29ea000640 1 -- 172.21.15.28:0/3984685841 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f29ec0751c0 con 0x7f29ec103240 2024-04-24T08:07:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.959+0000 7f29eaa00640 1 --2- 172.21.15.28:0/3984685841 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ec102050 0x7f29ec075db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.959+0000 7f29ea000640 1 --2- 172.21.15.28:0/3984685841 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29ec103240 0x7f29ec0743f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f29d800e9f0 tx=0x7f29d800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.960+0000 7f29e3400640 1 -- 172.21.15.28:0/3984685841 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d800cd80 con 0x7f29ec103240 2024-04-24T08:07:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.960+0000 7f29f1800640 1 -- 172.21.15.28:0/3984685841 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f29ec1a24a0 con 0x7f29ec103240 2024-04-24T08:07:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.960+0000 7f29e3400640 1 -- 172.21.15.28:0/3984685841 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29d8010910 con 0x7f29ec103240 2024-04-24T08:07:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.960+0000 7f29e3400640 1 -- 172.21.15.28:0/3984685841 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29d8018950 con 0x7f29ec103240 2024-04-24T08:07:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.960+0000 7f29f1800640 1 -- 172.21.15.28:0/3984685841 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29ec1a29f0 con 0x7f29ec103240 2024-04-24T08:07:06.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.961+0000 7f29f1800640 1 -- 172.21.15.28:0/3984685841 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f29ec109370 con 0x7f29ec103240 2024-04-24T08:07:06.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.964+0000 7f29e3400640 1 -- 172.21.15.28:0/3984685841 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f29d8010ac0 con 0x7f29ec103240 2024-04-24T08:07:06.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.965+0000 7f29e3400640 1 --2- 172.21.15.28:0/3984685841 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f29d00a3bb0 0x7f29d00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:06.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.965+0000 7f29e3400640 1 -- 172.21.15.28:0/3984685841 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f29d80d0560 con 0x7f29ec103240 2024-04-24T08:07:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.965+0000 7f29eaa00640 1 --2- 172.21.15.28:0/3984685841 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f29d00a3bb0 0x7f29d00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:06.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.966+0000 7f29eaa00640 1 --2- 172.21.15.28:0/3984685841 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f29d00a3bb0 0x7f29d00a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f29dc02a9f0 tx=0x7f29dc0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:06.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:06.967+0000 7f29e3400640 1 -- 172.21.15.28:0/3984685841 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f29d809ccf0 con 0x7f29ec103240 2024-04-24T08:07:07.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.175+0000 7f29f1800640 1 -- 172.21.15.28:0/3984685841 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f29ec02d870 con 0x7f29ec103240 2024-04-24T08:07:07.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.176+0000 7f29e3400640 1 -- 172.21.15.28:0/3984685841 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f29d80a0ff0 con 0x7f29ec103240 2024-04-24T08:07:07.178 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-24T08:07:07.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.179+0000 7f29f1800640 1 -- 172.21.15.28:0/3984685841 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f29d00a3bb0 msgr2=0x7f29d00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:07.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.179+0000 7f29f1800640 1 --2- 172.21.15.28:0/3984685841 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f29d00a3bb0 0x7f29d00a6070 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f29dc02a9f0 tx=0x7f29dc0315b0 comp rx=0 tx=0).stop 2024-04-24T08:07:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.179+0000 7f29f1800640 1 -- 172.21.15.28:0/3984685841 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29ec103240 msgr2=0x7f29ec0743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.179+0000 7f29f1800640 1 --2- 172.21.15.28:0/3984685841 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29ec103240 0x7f29ec0743f0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f29d800e9f0 tx=0x7f29d800eeb0 comp rx=0 tx=0).stop 2024-04-24T08:07:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.179+0000 7f29f1800640 1 -- 172.21.15.28:0/3984685841 shutdown_connections 2024-04-24T08:07:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.179+0000 7f29f1800640 1 --2- 172.21.15.28:0/3984685841 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f29d00a3bb0 0x7f29d00a6070 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.179+0000 7f29f1800640 1 --2- 172.21.15.28:0/3984685841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f29ec103c00 0x7f29ec074930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.179+0000 7f29f1800640 1 --2- 172.21.15.28:0/3984685841 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f29ec103240 0x7f29ec0743f0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.179+0000 7f29f1800640 1 --2- 172.21.15.28:0/3984685841 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29ec102050 0x7f29ec075db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.179+0000 7f29f1800640 1 -- 172.21.15.28:0/3984685841 >> 172.21.15.28:0/3984685841 conn(0x7f29ec0fd5e0 msgr2=0x7f29ec106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.179+0000 7f29f1800640 1 -- 172.21.15.28:0/3984685841 shutdown_connections 2024-04-24T08:07:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.179+0000 7f29f1800640 1 -- 172.21.15.28:0/3984685841 wait complete. 2024-04-24T08:07:07.199 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:07.374 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:07 smithi194 ceph-mon[32836]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:07.374 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:07.374 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:07:07.374 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:07.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:07.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:07.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:07 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:07.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:07 smithi194 ceph-mon[32836]: Deploying daemon mds.l on smithi194 2024-04-24T08:07:07.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:07 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3855113818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:07:07.375 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:07 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3984685841' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:07:07.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.404+0000 7f34ea600640 1 -- 172.21.15.28:0/532566120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34e40769c0 msgr2=0x7f34e4076de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:07.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.404+0000 7f34ea600640 1 --2- 172.21.15.28:0/532566120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34e40769c0 0x7f34e4076de0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f34dc009ca0 tx=0x7f34dc02a8c0 comp rx=0 tx=0).stop 2024-04-24T08:07:07.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.404+0000 7f34ea600640 1 -- 172.21.15.28:0/532566120 shutdown_connections 2024-04-24T08:07:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.404+0000 7f34ea600640 1 --2- 172.21.15.28:0/532566120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34e40785a0 0x7f34e4078e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.404+0000 7f34ea600640 1 --2- 172.21.15.28:0/532566120 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34e4077c00 0x7f34e4078060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.404+0000 7f34ea600640 1 --2- 172.21.15.28:0/532566120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34e40769c0 0x7f34e4076de0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.404+0000 7f34ea600640 1 -- 172.21.15.28:0/532566120 >> 172.21.15.28:0/532566120 conn(0x7f34e40ff7f0 msgr2=0x7f34e4101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.405+0000 7f34ea600640 1 -- 172.21.15.28:0/532566120 shutdown_connections 2024-04-24T08:07:07.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.405+0000 7f34ea600640 1 -- 172.21.15.28:0/532566120 wait complete. 2024-04-24T08:07:07.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.405+0000 7f34ea600640 1 Processor -- start 2024-04-24T08:07:07.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.406+0000 7f34ea600640 1 -- start start 2024-04-24T08:07:07.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.406+0000 7f34ea600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34e40769c0 0x7f34e419e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:07.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.406+0000 7f34ea600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34e4077c00 0x7f34e419eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.406+0000 7f34ea600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34e40785a0 0x7f34e41a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.406+0000 7f34ea600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34e4114330 con 0x7f34e40769c0 2024-04-24T08:07:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.406+0000 7f34ea600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f34e41141b0 con 0x7f34e40785a0 2024-04-24T08:07:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.406+0000 7f34ea600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f34e41144b0 con 0x7f34e4077c00 2024-04-24T08:07:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.406+0000 7f34e3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34e40769c0 0x7f34e419e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.406+0000 7f34e2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34e4077c00 0x7f34e419eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.406+0000 7f34e2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34e4077c00 0x7f34e419eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:33728/0 (socket says 172.21.15.28:33728) 2024-04-24T08:07:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.406+0000 7f34e2a00640 1 -- 172.21.15.28:0/2392907143 learned_addr learned my addr 172.21.15.28:0/2392907143 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.406+0000 7f34e3e00640 1 --2- 172.21.15.28:0/2392907143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34e40785a0 0x7f34e41a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.407+0000 7f34e2a00640 1 -- 172.21.15.28:0/2392907143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34e40785a0 msgr2=0x7f34e41a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.407+0000 7f34e2a00640 1 --2- 172.21.15.28:0/2392907143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34e40785a0 0x7f34e41a5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:07.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.407+0000 7f34e2a00640 1 -- 172.21.15.28:0/2392907143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34e40769c0 msgr2=0x7f34e419e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.407+0000 7f34e2a00640 1 --2- 172.21.15.28:0/2392907143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34e40769c0 0x7f34e419e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.407+0000 7f34e2a00640 1 -- 172.21.15.28:0/2392907143 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34e41a6580 con 0x7f34e4077c00 2024-04-24T08:07:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.407+0000 7f34e3e00640 1 --2- 172.21.15.28:0/2392907143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34e40785a0 0x7f34e41a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:07:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.407+0000 7f34e2a00640 1 --2- 172.21.15.28:0/2392907143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34e4077c00 0x7f34e419eea0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f34d400e9f0 tx=0x7f34d400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.407+0000 7f34dbe00640 1 -- 172.21.15.28:0/2392907143 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d400cd80 con 0x7f34e4077c00 2024-04-24T08:07:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.407+0000 7f34ea600640 1 -- 172.21.15.28:0/2392907143 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34e41a6870 con 0x7f34e4077c00 2024-04-24T08:07:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.407+0000 7f34dbe00640 1 -- 172.21.15.28:0/2392907143 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34d4010910 con 0x7f34e4077c00 2024-04-24T08:07:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.407+0000 7f34dbe00640 1 -- 172.21.15.28:0/2392907143 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34d40188f0 con 0x7f34e4077c00 2024-04-24T08:07:07.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.407+0000 7f34ea600640 1 -- 172.21.15.28:0/2392907143 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34e41a6e10 con 0x7f34e4077c00 2024-04-24T08:07:07.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.408+0000 7f34ea600640 1 -- 172.21.15.28:0/2392907143 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f34e4110d60 con 0x7f34e4077c00 2024-04-24T08:07:07.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.410+0000 7f34dbe00640 1 -- 172.21.15.28:0/2392907143 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f34d4010ac0 con 0x7f34e4077c00 2024-04-24T08:07:07.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.412+0000 7f34dbe00640 1 --2- 172.21.15.28:0/2392907143 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f34c80a3c30 0x7f34c80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:07.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.412+0000 7f34dbe00640 1 -- 172.21.15.28:0/2392907143 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f34d40d05b0 con 0x7f34e4077c00 2024-04-24T08:07:07.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.412+0000 7f34e3400640 1 --2- 172.21.15.28:0/2392907143 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f34c80a3c30 0x7f34c80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:07.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.413+0000 7f34e3400640 1 --2- 172.21.15.28:0/2392907143 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f34c80a3c30 0x7f34c80a60f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f34dc005e30 tx=0x7f34dc005f50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:07.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.414+0000 7f34dbe00640 1 -- 172.21.15.28:0/2392907143 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f34d409d390 con 0x7f34e4077c00 2024-04-24T08:07:07.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:07 smithi028 ceph-mon[27786]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:07.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:07.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:07:07.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:07.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:07.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:07.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:07 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:07.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:07 smithi028 ceph-mon[27786]: Deploying daemon mds.l on smithi194 2024-04-24T08:07:07.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:07 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3855113818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:07:07.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:07 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3984685841' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:07:07.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:07 smithi117 ceph-mon[32438]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:07.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:07.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:07:07.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:07.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:07.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:07.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:07 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:07.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:07 smithi117 ceph-mon[32438]: Deploying daemon mds.l on smithi194 2024-04-24T08:07:07.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:07 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3855113818' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:07:07.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:07 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3984685841' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:07:07.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.664+0000 7f34ea600640 1 -- 172.21.15.28:0/2392907143 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f34e402d870 con 0x7f34e4077c00 2024-04-24T08:07:07.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.664+0000 7f34dbe00640 1 -- 172.21.15.28:0/2392907143 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7f34d409cc90 con 0x7f34e4077c00 2024-04-24T08:07:07.666 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:07.666 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-04-24T08:07:07.666 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 1 2024-04-24T08:07:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.668+0000 7f34ea600640 1 -- 172.21.15.28:0/2392907143 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f34c80a3c30 msgr2=0x7f34c80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.668+0000 7f34ea600640 1 --2- 172.21.15.28:0/2392907143 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f34c80a3c30 0x7f34c80a60f0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f34dc005e30 tx=0x7f34dc005f50 comp rx=0 tx=0).stop 2024-04-24T08:07:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.668+0000 7f34ea600640 1 -- 172.21.15.28:0/2392907143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34e4077c00 msgr2=0x7f34e419eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.668+0000 7f34ea600640 1 --2- 172.21.15.28:0/2392907143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34e4077c00 0x7f34e419eea0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f34d400e9f0 tx=0x7f34d400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:07:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.668+0000 7f34ea600640 1 -- 172.21.15.28:0/2392907143 shutdown_connections 2024-04-24T08:07:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.668+0000 7f34ea600640 1 --2- 172.21.15.28:0/2392907143 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f34c80a3c30 0x7f34c80a60f0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.668+0000 7f34ea600640 1 --2- 172.21.15.28:0/2392907143 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34e40785a0 0x7f34e41a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.668+0000 7f34ea600640 1 --2- 172.21.15.28:0/2392907143 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34e4077c00 0x7f34e419eea0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.668+0000 7f34ea600640 1 --2- 172.21.15.28:0/2392907143 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34e40769c0 0x7f34e419e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.668+0000 7f34ea600640 1 -- 172.21.15.28:0/2392907143 >> 172.21.15.28:0/2392907143 conn(0x7f34e40ff7f0 msgr2=0x7f34e4100ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.668+0000 7f34ea600640 1 -- 172.21.15.28:0/2392907143 shutdown_connections 2024-04-24T08:07:07.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:07.668+0000 7f34ea600640 1 -- 172.21.15.28:0/2392907143 wait complete. 2024-04-24T08:07:08.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:08 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2392907143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:08.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:08 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2392907143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:08.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:08 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2392907143' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:08.882 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 24 08:07:08 smithi194 systemd[1]: Starting Ceph mds.l for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:07:09.176 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 24 08:07:08 smithi194 podman[63213]: 2024-04-24 08:07:08.881383648 +0000 UTC m=+0.094519400 container create 95974d00367b0e2b6faefea8308efc63834ee3dd1cdc790790031b074b5aa24a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-l, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-24T08:07:09.176 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 24 08:07:08 smithi194 podman[63213]: 2024-04-24 08:07:08.804651112 +0000 UTC m=+0.017786862 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:07:09.176 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 24 08:07:09 smithi194 podman[63213]: 2024-04-24 08:07:09.083926099 +0000 UTC m=+0.297061850 container init 95974d00367b0e2b6faefea8308efc63834ee3dd1cdc790790031b074b5aa24a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2) 2024-04-24T08:07:09.176 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 24 08:07:09 smithi194 podman[63213]: 2024-04-24 08:07:09.086715854 +0000 UTC m=+0.299851613 container start 95974d00367b0e2b6faefea8308efc63834ee3dd1cdc790790031b074b5aa24a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD) 2024-04-24T08:07:09.176 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 24 08:07:09 smithi194 ceph-mds[63231]: -- 172.21.15.194:0/3232465183 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55972f133a40 con 0x55972f107000 2024-04-24T08:07:09.176 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 24 08:07:09 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-l[63227]: starting mds.l at 2024-04-24T08:07:09.177 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 24 08:07:09 smithi194 bash[63213]: 95974d00367b0e2b6faefea8308efc63834ee3dd1cdc790790031b074b5aa24a 2024-04-24T08:07:09.440 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 24 08:07:09 smithi194 systemd[1]: Started Ceph mds.l for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T08:07:09.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:09 smithi194 ceph-mon[32836]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:09.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:09 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.194:6832/2281745957,v1:172.21.15.194:6833/2281745957] up:boot 2024-04-24T08:07:09.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:09 smithi194 ceph-mon[32836]: fsmap 1 up:standby 2024-04-24T08:07:09.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:09 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T08:07:09.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:09 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:09.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:09 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:09.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:09 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:09.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:09 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:09.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:09 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:09.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:09 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:09.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:09 smithi117 ceph-mon[32438]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:09.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:09 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.194:6832/2281745957,v1:172.21.15.194:6833/2281745957] up:boot 2024-04-24T08:07:09.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:09 smithi117 ceph-mon[32438]: fsmap 1 up:standby 2024-04-24T08:07:09.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:09 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T08:07:09.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:09 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:09.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:09 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:09.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:09 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:09.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:09 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:09.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:09 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:09.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:09 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:09.688 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:09.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:09 smithi028 ceph-mon[27786]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:09.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:09 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.194:6832/2281745957,v1:172.21.15.194:6833/2281745957] up:boot 2024-04-24T08:07:09.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:09 smithi028 ceph-mon[27786]: fsmap 1 up:standby 2024-04-24T08:07:09.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:09 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T08:07:09.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:09 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:09.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:09 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:09.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:09 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:09.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:09 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:09.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:09 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:09.769 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:09 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.911+0000 7fce7fe00640 1 -- 172.21.15.28:0/3444485182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce80109670 msgr2=0x7fce80109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.911+0000 7fce7fe00640 1 --2- 172.21.15.28:0/3444485182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce80109670 0x7fce80109af0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fce70009900 tx=0x7fce7002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:07:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.912+0000 7fce7fe00640 1 -- 172.21.15.28:0/3444485182 shutdown_connections 2024-04-24T08:07:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.912+0000 7fce7fe00640 1 --2- 172.21.15.28:0/3444485182 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce8010a030 0x7fce801108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.912+0000 7fce7fe00640 1 --2- 172.21.15.28:0/3444485182 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce80109670 0x7fce80109af0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:09.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.912+0000 7fce7fe00640 1 --2- 172.21.15.28:0/3444485182 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce80108480 0x7fce801088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:09.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.912+0000 7fce7fe00640 1 -- 172.21.15.28:0/3444485182 >> 172.21.15.28:0/3444485182 conn(0x7fce80077440 msgr2=0x7fce80079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:09.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.912+0000 7fce7fe00640 1 -- 172.21.15.28:0/3444485182 shutdown_connections 2024-04-24T08:07:09.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.912+0000 7fce7fe00640 1 -- 172.21.15.28:0/3444485182 wait complete. 2024-04-24T08:07:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.913+0000 7fce7fe00640 1 Processor -- start 2024-04-24T08:07:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.913+0000 7fce7fe00640 1 -- start start 2024-04-24T08:07:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.913+0000 7fce7fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce80108480 0x7fce8019e850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.913+0000 7fce7fe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce80109670 0x7fce8019ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.913+0000 7fce7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce8010a030 0x7fce801a5d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.913+0000 7fce7fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fce80114160 con 0x7fce8010a030 2024-04-24T08:07:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.913+0000 7fce7fe00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fce80113fe0 con 0x7fce80109670 2024-04-24T08:07:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.913+0000 7fce7fe00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fce801142e0 con 0x7fce80108480 2024-04-24T08:07:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.914+0000 7fce7e000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce80109670 0x7fce8019ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.914+0000 7fce7e000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce80109670 0x7fce8019ed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:47024/0 (socket says 172.21.15.28:47024) 2024-04-24T08:07:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.914+0000 7fce7ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce80108480 0x7fce8019e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.914+0000 7fce7f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce8010a030 0x7fce801a5d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.914+0000 7fce7e000640 1 -- 172.21.15.28:0/922277924 learned_addr learned my addr 172.21.15.28:0/922277924 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.914+0000 7fce7e000640 1 -- 172.21.15.28:0/922277924 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce80108480 msgr2=0x7fce8019e850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.914+0000 7fce7e000640 1 --2- 172.21.15.28:0/922277924 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce80108480 0x7fce8019e850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.914+0000 7fce7e000640 1 -- 172.21.15.28:0/922277924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce8010a030 msgr2=0x7fce801a5d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.914+0000 7fce7e000640 1 --2- 172.21.15.28:0/922277924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce8010a030 0x7fce801a5d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.914+0000 7fce7e000640 1 -- 172.21.15.28:0/922277924 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fce801a6380 con 0x7fce80109670 2024-04-24T08:07:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.914+0000 7fce7ea00640 1 --2- 172.21.15.28:0/922277924 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce80108480 0x7fce8019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.914+0000 7fce7f400640 1 --2- 172.21.15.28:0/922277924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce8010a030 0x7fce801a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.914+0000 7fce7e000640 1 --2- 172.21.15.28:0/922277924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce80109670 0x7fce8019ed90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fce70031440 tx=0x7fce70032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.915+0000 7fce77400640 1 -- 172.21.15.28:0/922277924 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce7003f030 con 0x7fce80109670 2024-04-24T08:07:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.915+0000 7fce7fe00640 1 -- 172.21.15.28:0/922277924 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fce801a6610 con 0x7fce80109670 2024-04-24T08:07:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.915+0000 7fce77400640 1 -- 172.21.15.28:0/922277924 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fce70003ce0 con 0x7fce80109670 2024-04-24T08:07:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.915+0000 7fce77400640 1 -- 172.21.15.28:0/922277924 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fce70035070 con 0x7fce80109670 2024-04-24T08:07:09.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.915+0000 7fce7fe00640 1 -- 172.21.15.28:0/922277924 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fce801a6b00 con 0x7fce80109670 2024-04-24T08:07:09.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.916+0000 7fce7fe00640 1 -- 172.21.15.28:0/922277924 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fce8010f740 con 0x7fce80109670 2024-04-24T08:07:09.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.918+0000 7fce77400640 1 -- 172.21.15.28:0/922277924 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fce70042030 con 0x7fce80109670 2024-04-24T08:07:09.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.919+0000 7fce77400640 1 --2- 172.21.15.28:0/922277924 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fce640a3bb0 0x7fce640a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:09.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.920+0000 7fce77400640 1 -- 172.21.15.28:0/922277924 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fce700f0a60 con 0x7fce80109670 2024-04-24T08:07:09.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.920+0000 7fce7ea00640 1 --2- 172.21.15.28:0/922277924 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fce640a3bb0 0x7fce640a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:09.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.920+0000 7fce7ea00640 1 --2- 172.21.15.28:0/922277924 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fce640a3bb0 0x7fce640a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fce801094d0 tx=0x7fce68005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:09.926+0000 7fce77400640 1 -- 172.21.15.28:0/922277924 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fce700bd1f0 con 0x7fce80109670 2024-04-24T08:07:10.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:10.178+0000 7fce7fe00640 1 -- 172.21.15.28:0/922277924 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fce8002d620 con 0x7fce80109670 2024-04-24T08:07:10.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:10.179+0000 7fce77400640 1 -- 172.21.15.28:0/922277924 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7fce700c14f0 con 0x7fce80109670 2024-04-24T08:07:10.181 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:10.181 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-24T08:07:10.181 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 2 2024-04-24T08:07:10.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:10.182+0000 7fce7fe00640 1 -- 172.21.15.28:0/922277924 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fce640a3bb0 msgr2=0x7fce640a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:10.182+0000 7fce7fe00640 1 --2- 172.21.15.28:0/922277924 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fce640a3bb0 0x7fce640a6070 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fce801094d0 tx=0x7fce68005f90 comp rx=0 tx=0).stop 2024-04-24T08:07:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:10.182+0000 7fce7fe00640 1 -- 172.21.15.28:0/922277924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce80109670 msgr2=0x7fce8019ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:10.182+0000 7fce7fe00640 1 --2- 172.21.15.28:0/922277924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce80109670 0x7fce8019ed90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fce70031440 tx=0x7fce70032ce0 comp rx=0 tx=0).stop 2024-04-24T08:07:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:10.182+0000 7fce7fe00640 1 -- 172.21.15.28:0/922277924 shutdown_connections 2024-04-24T08:07:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:10.182+0000 7fce7fe00640 1 --2- 172.21.15.28:0/922277924 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fce640a3bb0 0x7fce640a6070 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:10.182+0000 7fce7fe00640 1 --2- 172.21.15.28:0/922277924 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fce8010a030 0x7fce801a5d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:10.182+0000 7fce7fe00640 1 --2- 172.21.15.28:0/922277924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce80109670 0x7fce8019ed90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:10.182+0000 7fce7fe00640 1 --2- 172.21.15.28:0/922277924 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fce80108480 0x7fce8019e850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:10.182+0000 7fce7fe00640 1 -- 172.21.15.28:0/922277924 >> 172.21.15.28:0/922277924 conn(0x7fce80077440 msgr2=0x7fce8010c910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:10.183+0000 7fce7fe00640 1 -- 172.21.15.28:0/922277924 shutdown_connections 2024-04-24T08:07:10.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:10.183+0000 7fce7fe00640 1 -- 172.21.15.28:0/922277924 wait complete. 2024-04-24T08:07:10.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:10 smithi028 ceph-mon[27786]: Deploying daemon mds.k on smithi117 2024-04-24T08:07:10.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:10 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/922277924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:10.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:10 smithi117 ceph-mon[32438]: Deploying daemon mds.k on smithi117 2024-04-24T08:07:10.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:10 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/922277924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:10.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:10 smithi194 ceph-mon[32836]: Deploying daemon mds.k on smithi117 2024-04-24T08:07:10.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:10 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/922277924' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:11.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:11 smithi117 ceph-mon[32438]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:11.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:11 smithi194 ceph-mon[32836]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:11.785 INFO:journalctl@ceph.mds.k.smithi117.stdout:Apr 24 08:07:11 smithi117 systemd[1]: Starting Ceph mds.k for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:07:11.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:11 smithi028 ceph-mon[27786]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:12.047 INFO:journalctl@ceph.mds.k.smithi117.stdout:Apr 24 08:07:11 smithi117 podman[61933]: 2024-04-24 08:07:11.785202735 +0000 UTC m=+0.108341627 container create 8a8f543f281c41933ac1c8af908d432b83250f318ee1bdba92439c296b16c7ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-24T08:07:12.047 INFO:journalctl@ceph.mds.k.smithi117.stdout:Apr 24 08:07:11 smithi117 podman[61933]: 2024-04-24 08:07:11.694718087 +0000 UTC m=+0.017856990 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:07:12.047 INFO:journalctl@ceph.mds.k.smithi117.stdout:Apr 24 08:07:12 smithi117 podman[61933]: 2024-04-24 08:07:12.0211498 +0000 UTC m=+0.344288694 container init 8a8f543f281c41933ac1c8af908d432b83250f318ee1bdba92439c296b16c7ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-k, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:07:12.047 INFO:journalctl@ceph.mds.k.smithi117.stdout:Apr 24 08:07:12 smithi117 podman[61933]: 2024-04-24 08:07:12.023760541 +0000 UTC m=+0.346899441 container start 8a8f543f281c41933ac1c8af908d432b83250f318ee1bdba92439c296b16c7ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-k, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2) 2024-04-24T08:07:12.207 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:12.377 INFO:journalctl@ceph.mds.k.smithi117.stdout:Apr 24 08:07:12 smithi117 ceph-mds[61952]: -- 172.21.15.117:0/1141921167 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d7518d3a40 con 0x55d7518a6c00 2024-04-24T08:07:12.378 INFO:journalctl@ceph.mds.k.smithi117.stdout:Apr 24 08:07:12 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-k[61948]: starting mds.k at 2024-04-24T08:07:12.378 INFO:journalctl@ceph.mds.k.smithi117.stdout:Apr 24 08:07:12 smithi117 bash[61933]: 8a8f543f281c41933ac1c8af908d432b83250f318ee1bdba92439c296b16c7ce 2024-04-24T08:07:12.378 INFO:journalctl@ceph.mds.k.smithi117.stdout:Apr 24 08:07:12 smithi117 systemd[1]: Started Ceph mds.k for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T08:07:12.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.411+0000 7f534a800640 1 -- 172.21.15.28:0/3150992365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5344103240 msgr2=0x7f53441036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:12.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.411+0000 7f534a800640 1 --2- 172.21.15.28:0/3150992365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5344103240 0x7f53441036c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f5330009900 tx=0x7f533002a410 comp rx=0 tx=0).stop 2024-04-24T08:07:12.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.411+0000 7f534a800640 1 -- 172.21.15.28:0/3150992365 shutdown_connections 2024-04-24T08:07:12.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.411+0000 7f534a800640 1 --2- 172.21.15.28:0/3150992365 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5344103240 0x7f53441036c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:12.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.411+0000 7f534a800640 1 --2- 172.21.15.28:0/3150992365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5344103c00 0x7f534410a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:12.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.411+0000 7f534a800640 1 --2- 172.21.15.28:0/3150992365 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5344102050 0x7f5344102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:12.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.411+0000 7f534a800640 1 -- 172.21.15.28:0/3150992365 >> 172.21.15.28:0/3150992365 conn(0x7f53440fd5e0 msgr2=0x7f53440ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:12.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.411+0000 7f534a800640 1 -- 172.21.15.28:0/3150992365 shutdown_connections 2024-04-24T08:07:12.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.411+0000 7f534a800640 1 -- 172.21.15.28:0/3150992365 wait complete. 2024-04-24T08:07:12.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.412+0000 7f534a800640 1 Processor -- start 2024-04-24T08:07:12.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.412+0000 7f534a800640 1 -- start start 2024-04-24T08:07:12.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.412+0000 7f534a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5344102050 0x7f5344107dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:12.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.413+0000 7f534a800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5344103240 0x7f5344108310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:12.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.413+0000 7f534a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5344103c00 0x7f5344104940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:12.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.413+0000 7f534a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f534410dd30 con 0x7f5344103c00 2024-04-24T08:07:12.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.413+0000 7f534a800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f534410dbb0 con 0x7f5344103240 2024-04-24T08:07:12.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.413+0000 7f534a800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f534410deb0 con 0x7f5344102050 2024-04-24T08:07:12.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.413+0000 7f5343e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5344103c00 0x7f5344104940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:12.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.413+0000 7f5342a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5344103240 0x7f5344108310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:12.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.413+0000 7f5343400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5344102050 0x7f5344107dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:12.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.413+0000 7f5343e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5344103c00 0x7f5344104940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36666/0 (socket says 172.21.15.28:36666) 2024-04-24T08:07:12.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.413+0000 7f5343e00640 1 -- 172.21.15.28:0/1457505957 learned_addr learned my addr 172.21.15.28:0/1457505957 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:12.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.413+0000 7f5343e00640 1 -- 172.21.15.28:0/1457505957 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5344102050 msgr2=0x7f5344107dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:12.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.413+0000 7f5343e00640 1 --2- 172.21.15.28:0/1457505957 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5344102050 0x7f5344107dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:12.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.413+0000 7f5343e00640 1 -- 172.21.15.28:0/1457505957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5344103240 msgr2=0x7f5344108310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:12.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.413+0000 7f5343e00640 1 --2- 172.21.15.28:0/1457505957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5344103240 0x7f5344108310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:12.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.413+0000 7f5343e00640 1 -- 172.21.15.28:0/1457505957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5344105200 con 0x7f5344103c00 2024-04-24T08:07:12.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.414+0000 7f5343e00640 1 --2- 172.21.15.28:0/1457505957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5344103c00 0x7f5344104940 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f533c00efd0 tx=0x7f533c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:12.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.414+0000 7f533be00640 1 -- 172.21.15.28:0/1457505957 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c003b70 con 0x7f5344103c00 2024-04-24T08:07:12.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.414+0000 7f533be00640 1 -- 172.21.15.28:0/1457505957 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f533c003d10 con 0x7f5344103c00 2024-04-24T08:07:12.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.414+0000 7f534a800640 1 -- 172.21.15.28:0/1457505957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53441a69d0 con 0x7f5344103c00 2024-04-24T08:07:12.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.414+0000 7f533be00640 1 -- 172.21.15.28:0/1457505957 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f533c0109f0 con 0x7f5344103c00 2024-04-24T08:07:12.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.414+0000 7f534a800640 1 -- 172.21.15.28:0/1457505957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53441a6d80 con 0x7f5344103c00 2024-04-24T08:07:12.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.416+0000 7f534a800640 1 -- 172.21.15.28:0/1457505957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5324005290 con 0x7f5344103c00 2024-04-24T08:07:12.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.418+0000 7f533be00640 1 -- 172.21.15.28:0/1457505957 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f533c010b90 con 0x7f5344103c00 2024-04-24T08:07:12.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.420+0000 7f533be00640 1 --2- 172.21.15.28:0/1457505957 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f53280a3bb0 0x7f53280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:12.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.420+0000 7f533be00640 1 -- 172.21.15.28:0/1457505957 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f533c0d0370 con 0x7f5344103c00 2024-04-24T08:07:12.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.420+0000 7f5343400640 1 --2- 172.21.15.28:0/1457505957 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f53280a3bb0 0x7f53280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:12.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.421+0000 7f5343400640 1 --2- 172.21.15.28:0/1457505957 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f53280a3bb0 0x7f53280a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f53441030a0 tx=0x7f5334005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:12.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.425+0000 7f533be00640 1 -- 172.21.15.28:0/1457505957 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f533c09d150 con 0x7f5344103c00 2024-04-24T08:07:12.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.679+0000 7f534a800640 1 -- 172.21.15.28:0/1457505957 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5324005580 con 0x7f5344103c00 2024-04-24T08:07:12.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.680+0000 7f533be00640 1 -- 172.21.15.28:0/1457505957 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f533c09ca50 con 0x7f5344103c00 2024-04-24T08:07:12.681 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:12.681 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-24T08:07:12.682 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-04-24T08:07:12.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.682+0000 7f534a800640 1 -- 172.21.15.28:0/1457505957 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f53280a3bb0 msgr2=0x7f53280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:12.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.682+0000 7f534a800640 1 --2- 172.21.15.28:0/1457505957 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f53280a3bb0 0x7f53280a6070 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f53441030a0 tx=0x7f5334005f90 comp rx=0 tx=0).stop 2024-04-24T08:07:12.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.682+0000 7f534a800640 1 -- 172.21.15.28:0/1457505957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5344103c00 msgr2=0x7f5344104940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:12.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.682+0000 7f534a800640 1 --2- 172.21.15.28:0/1457505957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5344103c00 0x7f5344104940 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f533c00efd0 tx=0x7f533c00c6a0 comp rx=0 tx=0).stop 2024-04-24T08:07:12.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.683+0000 7f534a800640 1 -- 172.21.15.28:0/1457505957 shutdown_connections 2024-04-24T08:07:12.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.683+0000 7f534a800640 1 --2- 172.21.15.28:0/1457505957 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f53280a3bb0 0x7f53280a6070 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:12.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.683+0000 7f534a800640 1 --2- 172.21.15.28:0/1457505957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5344103c00 0x7f5344104940 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:12.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.683+0000 7f534a800640 1 --2- 172.21.15.28:0/1457505957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5344103240 0x7f5344108310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:12.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.683+0000 7f534a800640 1 --2- 172.21.15.28:0/1457505957 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5344102050 0x7f5344107dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:12.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.683+0000 7f534a800640 1 -- 172.21.15.28:0/1457505957 >> 172.21.15.28:0/1457505957 conn(0x7f53440fd5e0 msgr2=0x7f5344074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:12.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.683+0000 7f534a800640 1 -- 172.21.15.28:0/1457505957 shutdown_connections 2024-04-24T08:07:12.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:12.683+0000 7f534a800640 1 -- 172.21.15.28:0/1457505957 wait complete. 2024-04-24T08:07:13.179 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:13 smithi028 ceph-mon[27786]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:13.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:13 smithi194 ceph-mon[32836]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:13.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:13 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:13.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:13 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:13.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:13 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:13.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:13 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:13.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:13 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:13.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:13 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:13.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:13 smithi194 ceph-mon[32836]: Deploying daemon mds.j on smithi028 2024-04-24T08:07:13.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:13 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.117:6832/4248331494,v1:172.21.15.117:6833/4248331494] up:boot 2024-04-24T08:07:13.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:13 smithi194 ceph-mon[32836]: fsmap 2 up:standby 2024-04-24T08:07:13.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:13 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T08:07:13.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:13 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1457505957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:13.479 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:13 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:13.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:13 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:13.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:13 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:13.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:13 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:13.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:13 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:13.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:13 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:13.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:13 smithi028 ceph-mon[27786]: Deploying daemon mds.j on smithi028 2024-04-24T08:07:13.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:13 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.117:6832/4248331494,v1:172.21.15.117:6833/4248331494] up:boot 2024-04-24T08:07:13.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:13 smithi028 ceph-mon[27786]: fsmap 2 up:standby 2024-04-24T08:07:13.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:13 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T08:07:13.480 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:13 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1457505957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:13 smithi117 ceph-mon[32438]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:13 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:13 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:13 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:13 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:13 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:13 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:13 smithi117 ceph-mon[32438]: Deploying daemon mds.j on smithi028 2024-04-24T08:07:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:13 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.117:6832/4248331494,v1:172.21.15.117:6833/4248331494] up:boot 2024-04-24T08:07:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:13 smithi117 ceph-mon[32438]: fsmap 2 up:standby 2024-04-24T08:07:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:13 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T08:07:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:13 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1457505957' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:14.704 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:14.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.931+0000 7f51d3600640 1 -- 172.21.15.28:0/3762108100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51cc103260 msgr2=0x7f51cc1056d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:14.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.931+0000 7f51d3600640 1 --2- 172.21.15.28:0/3762108100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51cc103260 0x7f51cc1056d0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f51b8009900 tx=0x7f51b802a420 comp rx=0 tx=0).stop 2024-04-24T08:07:14.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.931+0000 7f51d3600640 1 -- 172.21.15.28:0/3762108100 shutdown_connections 2024-04-24T08:07:14.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.931+0000 7f51d3600640 1 --2- 172.21.15.28:0/3762108100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51cc105c10 0x7f51cc106090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:14.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.931+0000 7f51d3600640 1 --2- 172.21.15.28:0/3762108100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51cc103260 0x7f51cc1056d0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:14.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.931+0000 7f51d3600640 1 --2- 172.21.15.28:0/3762108100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51cc1008f0 0x7f51cc102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:14.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.931+0000 7f51d3600640 1 -- 172.21.15.28:0/3762108100 >> 172.21.15.28:0/3762108100 conn(0x7f51cc0fc4f0 msgr2=0x7f51cc0fe930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:14.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.935+0000 7f51d3600640 1 -- 172.21.15.28:0/3762108100 shutdown_connections 2024-04-24T08:07:14.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.935+0000 7f51d3600640 1 -- 172.21.15.28:0/3762108100 wait complete. 2024-04-24T08:07:14.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.937+0000 7f51d3600640 1 Processor -- start 2024-04-24T08:07:14.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.937+0000 7f51d3600640 1 -- start start 2024-04-24T08:07:14.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.938+0000 7f51d3600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51cc1008f0 0x7f51cc19e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:14.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.938+0000 7f51d3600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51cc103260 0x7f51cc19eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:14.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.938+0000 7f51d3600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51cc105c10 0x7f51cc1a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:14.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.938+0000 7f51d3600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f51cc114390 con 0x7f51cc103260 2024-04-24T08:07:14.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.938+0000 7f51d3600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f51cc114210 con 0x7f51cc105c10 2024-04-24T08:07:14.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.938+0000 7f51d3600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f51cc114510 con 0x7f51cc1008f0 2024-04-24T08:07:14.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.938+0000 7f51cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51cc103260 0x7f51cc19eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:14.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.938+0000 7f51d1600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51cc105c10 0x7f51cc1a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:14.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.938+0000 7f51d1600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51cc105c10 0x7f51cc1a5f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:47074/0 (socket says 172.21.15.28:47074) 2024-04-24T08:07:14.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.938+0000 7f51d1600640 1 -- 172.21.15.28:0/3648498385 learned_addr learned my addr 172.21.15.28:0/3648498385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.939+0000 7f51cbe00640 1 -- 172.21.15.28:0/3648498385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51cc1008f0 msgr2=0x7f51cc19e990 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T08:07:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.939+0000 7f51cbe00640 1 --2- 172.21.15.28:0/3648498385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51cc1008f0 0x7f51cc19e990 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.939+0000 7f51cbe00640 1 -- 172.21.15.28:0/3648498385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51cc105c10 msgr2=0x7f51cc1a5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.939+0000 7f51cbe00640 1 --2- 172.21.15.28:0/3648498385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51cc105c10 0x7f51cc1a5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.939+0000 7f51cbe00640 1 -- 172.21.15.28:0/3648498385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f51cc1a65b0 con 0x7f51cc103260 2024-04-24T08:07:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.939+0000 7f51cbe00640 1 --2- 172.21.15.28:0/3648498385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51cc103260 0x7f51cc19eed0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f51b8009850 tx=0x7f51b8004b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:14.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.940+0000 7f51c9600640 1 -- 172.21.15.28:0/3648498385 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b8041070 con 0x7f51cc103260 2024-04-24T08:07:14.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.940+0000 7f51c9600640 1 -- 172.21.15.28:0/3648498385 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f51b803d450 con 0x7f51cc103260 2024-04-24T08:07:14.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.940+0000 7f51c9600640 1 -- 172.21.15.28:0/3648498385 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f51b80316a0 con 0x7f51cc103260 2024-04-24T08:07:14.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.940+0000 7f51d3600640 1 -- 172.21.15.28:0/3648498385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f51cc1a6840 con 0x7f51cc103260 2024-04-24T08:07:14.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.940+0000 7f51d3600640 1 -- 172.21.15.28:0/3648498385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51cc1a6d40 con 0x7f51cc103260 2024-04-24T08:07:14.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.942+0000 7f51d3600640 1 -- 172.21.15.28:0/3648498385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f51cc0fdfe0 con 0x7f51cc103260 2024-04-24T08:07:14.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.945+0000 7f51c9600640 1 -- 172.21.15.28:0/3648498385 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f51b8044070 con 0x7f51cc103260 2024-04-24T08:07:14.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.946+0000 7f51c9600640 1 --2- 172.21.15.28:0/3648498385 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f51b00a3c70 0x7f51b00a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:14.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.946+0000 7f51c9600640 1 -- 172.21.15.28:0/3648498385 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f51b80f0950 con 0x7f51cc103260 2024-04-24T08:07:14.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.947+0000 7f51d0c00640 1 --2- 172.21.15.28:0/3648498385 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f51b00a3c70 0x7f51b00a6130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:14.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.947+0000 7f51d0c00640 1 --2- 172.21.15.28:0/3648498385 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f51b00a3c70 0x7f51b00a6130 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f51bc0096b0 tx=0x7f51bc006c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:14.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:14.951+0000 7f51c9600640 1 -- 172.21.15.28:0/3648498385 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f51b80bd7c0 con 0x7f51cc103260 2024-04-24T08:07:15.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:15.210+0000 7f51d3600640 1 -- 172.21.15.28:0/3648498385 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f51cc02d620 con 0x7f51cc103260 2024-04-24T08:07:15.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:15.210+0000 7f51c9600640 1 -- 172.21.15.28:0/3648498385 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7f51b8003610 con 0x7f51cc103260 2024-04-24T08:07:15.211 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:15.211 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-24T08:07:15.211 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 3 2024-04-24T08:07:15.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:15.213+0000 7f51d3600640 1 -- 172.21.15.28:0/3648498385 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f51b00a3c70 msgr2=0x7f51b00a6130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:15.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:15.213+0000 7f51d3600640 1 --2- 172.21.15.28:0/3648498385 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f51b00a3c70 0x7f51b00a6130 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f51bc0096b0 tx=0x7f51bc006c20 comp rx=0 tx=0).stop 2024-04-24T08:07:15.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:15.213+0000 7f51d3600640 1 -- 172.21.15.28:0/3648498385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51cc103260 msgr2=0x7f51cc19eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:15.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:15.213+0000 7f51d3600640 1 --2- 172.21.15.28:0/3648498385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51cc103260 0x7f51cc19eed0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f51b8009850 tx=0x7f51b8004b00 comp rx=0 tx=0).stop 2024-04-24T08:07:15.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:15.213+0000 7f51d3600640 1 -- 172.21.15.28:0/3648498385 shutdown_connections 2024-04-24T08:07:15.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:15.213+0000 7f51d3600640 1 --2- 172.21.15.28:0/3648498385 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f51b00a3c70 0x7f51b00a6130 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:15.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:15.213+0000 7f51d3600640 1 --2- 172.21.15.28:0/3648498385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f51cc103260 0x7f51cc19eed0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:15.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:15.213+0000 7f51d3600640 1 --2- 172.21.15.28:0/3648498385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51cc105c10 0x7f51cc1a5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:15.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:15.213+0000 7f51d3600640 1 --2- 172.21.15.28:0/3648498385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f51cc1008f0 0x7f51cc19e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:15.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:15.213+0000 7f51d3600640 1 -- 172.21.15.28:0/3648498385 >> 172.21.15.28:0/3648498385 conn(0x7f51cc0fc4f0 msgr2=0x7f51cc101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:15.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:15.213+0000 7f51d3600640 1 -- 172.21.15.28:0/3648498385 shutdown_connections 2024-04-24T08:07:15.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:15.214+0000 7f51d3600640 1 -- 172.21.15.28:0/3648498385 wait complete. 2024-04-24T08:07:15.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:15 smithi194 ceph-mon[32836]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:15.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:15 smithi028 ceph-mon[27786]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:15.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:15 smithi117 ceph-mon[32438]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:15.789 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 24 08:07:15 smithi028 podman[65679]: 2024-04-24 08:07:15.535952998 +0000 UTC m=+0.141063994 container create 98ab3eba529b945687d27c86e6572556c2144bc77bbdfe4ad1e72effc3908b95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-j, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:07:16.190 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 24 08:07:15 smithi028 podman[65679]: 2024-04-24 08:07:15.827260478 +0000 UTC m=+0.432371451 container init 98ab3eba529b945687d27c86e6572556c2144bc77bbdfe4ad1e72effc3908b95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-j, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-04-24T08:07:16.191 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 24 08:07:15 smithi028 podman[65679]: 2024-04-24 08:07:15.837694519 +0000 UTC m=+0.442805481 container start 98ab3eba529b945687d27c86e6572556c2144bc77bbdfe4ad1e72effc3908b95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:07:16.191 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 24 08:07:15 smithi028 ceph-mds[65698]: -- 172.21.15.28:0/1372808401 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x565362921a40 con 0x5653628f4c00 2024-04-24T08:07:16.191 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 24 08:07:15 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-j[65694]: starting mds.j at 2024-04-24T08:07:16.191 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 24 08:07:15 smithi028 bash[65679]: 98ab3eba529b945687d27c86e6572556c2144bc77bbdfe4ad1e72effc3908b95 2024-04-24T08:07:16.191 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 24 08:07:15 smithi028 systemd[1]: Started Ceph mds.j for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T08:07:16.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:16 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3648498385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:16.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:16 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:16.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:16 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:16.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:16 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:16.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:16 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:16.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:16 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:16.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:16 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:16.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:16 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3648498385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:16.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:16 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:16.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:16 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:16.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:16 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:16.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:16 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:16.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:16 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:16.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:16 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:16.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:16 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3648498385' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:16.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:16 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:16.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:16 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:16.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:16 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:16.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:16 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:16.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:16 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:16.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:16 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:17.235 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.463+0000 7f4ac6800640 1 -- 172.21.15.28:0/737454456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ac01040a0 msgr2=0x7f4ac01044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.463+0000 7f4ac6800640 1 --2- 172.21.15.28:0/737454456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ac01040a0 0x7f4ac01044c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4009900 tx=0x7f4ab402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:07:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.463+0000 7f4ac6800640 1 -- 172.21.15.28:0/737454456 shutdown_connections 2024-04-24T08:07:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.463+0000 7f4ac6800640 1 --2- 172.21.15.28:0/737454456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ac0105c50 0x7f4ac010c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.463+0000 7f4ac6800640 1 --2- 172.21.15.28:0/737454456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ac0105290 0x7f4ac0105710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.463+0000 7f4ac6800640 1 --2- 172.21.15.28:0/737454456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ac01040a0 0x7f4ac01044c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.463+0000 7f4ac6800640 1 -- 172.21.15.28:0/737454456 >> 172.21.15.28:0/737454456 conn(0x7f4ac00ff610 msgr2=0x7f4ac0101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:17.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.463+0000 7f4ac6800640 1 -- 172.21.15.28:0/737454456 shutdown_connections 2024-04-24T08:07:17.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.463+0000 7f4ac6800640 1 -- 172.21.15.28:0/737454456 wait complete. 2024-04-24T08:07:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.464+0000 7f4ac6800640 1 Processor -- start 2024-04-24T08:07:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.464+0000 7f4ac6800640 1 -- start start 2024-04-24T08:07:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.465+0000 7f4ac6800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ac01040a0 0x7f4ac019a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.465+0000 7f4ac6800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ac0105290 0x7f4ac019a920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.465+0000 7f4ac6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ac0105c50 0x7f4ac01a1990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.465+0000 7f4ac6800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac010fd80 con 0x7f4ac0105c50 2024-04-24T08:07:17.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.465+0000 7f4ac6800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac010fc00 con 0x7f4ac01040a0 2024-04-24T08:07:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.465+0000 7f4ac6800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4ac010ff00 con 0x7f4ac0105290 2024-04-24T08:07:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.465+0000 7f4ac5400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ac01040a0 0x7f4ac019a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.465+0000 7f4ac5400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ac01040a0 0x7f4ac019a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:47114/0 (socket says 172.21.15.28:47114) 2024-04-24T08:07:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.465+0000 7f4ac5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ac0105c50 0x7f4ac01a1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.465+0000 7f4ac5400640 1 -- 172.21.15.28:0/2162889202 learned_addr learned my addr 172.21.15.28:0/2162889202 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.465+0000 7f4ac4a00640 1 --2- 172.21.15.28:0/2162889202 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ac0105290 0x7f4ac019a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.465+0000 7f4ac5400640 1 -- 172.21.15.28:0/2162889202 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ac0105290 msgr2=0x7f4ac019a920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.465+0000 7f4ac5400640 1 --2- 172.21.15.28:0/2162889202 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ac0105290 0x7f4ac019a920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.465+0000 7f4ac5400640 1 -- 172.21.15.28:0/2162889202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ac0105c50 msgr2=0x7f4ac01a1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.465+0000 7f4ac5400640 1 --2- 172.21.15.28:0/2162889202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ac0105c50 0x7f4ac01a1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.465+0000 7f4ac5400640 1 -- 172.21.15.28:0/2162889202 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4ac01a2000 con 0x7f4ac01040a0 2024-04-24T08:07:17.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.466+0000 7f4ac5400640 1 --2- 172.21.15.28:0/2162889202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ac01040a0 0x7f4ac019a3e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4031440 tx=0x7f4ab4032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.466+0000 7f4abe000640 1 -- 172.21.15.28:0/2162889202 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab403f030 con 0x7f4ac01040a0 2024-04-24T08:07:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.466+0000 7f4abe000640 1 -- 172.21.15.28:0/2162889202 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ab4003ce0 con 0x7f4ac01040a0 2024-04-24T08:07:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.466+0000 7f4abe000640 1 -- 172.21.15.28:0/2162889202 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ab4035070 con 0x7f4ac01040a0 2024-04-24T08:07:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.466+0000 7f4ac6800640 1 -- 172.21.15.28:0/2162889202 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4ac01a2290 con 0x7f4ac01040a0 2024-04-24T08:07:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.466+0000 7f4ac6800640 1 -- 172.21.15.28:0/2162889202 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4ac01a2730 con 0x7f4ac01040a0 2024-04-24T08:07:17.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.467+0000 7f4ac6800640 1 -- 172.21.15.28:0/2162889202 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4ac010b440 con 0x7f4ac01040a0 2024-04-24T08:07:17.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.470+0000 7f4abe000640 1 -- 172.21.15.28:0/2162889202 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f4ab4042030 con 0x7f4ac01040a0 2024-04-24T08:07:17.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.472+0000 7f4abe000640 1 --2- 172.21.15.28:0/2162889202 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4aa80a3bb0 0x7f4aa80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:17.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.472+0000 7f4abe000640 1 -- 172.21.15.28:0/2162889202 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f4ab40f0930 con 0x7f4ac01040a0 2024-04-24T08:07:17.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.472+0000 7f4ac4a00640 1 --2- 172.21.15.28:0/2162889202 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4aa80a3bb0 0x7f4aa80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:17.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.473+0000 7f4ac4a00640 1 --2- 172.21.15.28:0/2162889202 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4aa80a3bb0 0x7f4aa80a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4ac019b8f0 tx=0x7f4aac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:17.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.473+0000 7f4abe000640 1 -- 172.21.15.28:0/2162889202 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4ab40bd0c0 con 0x7f4ac01040a0 2024-04-24T08:07:17.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:17 smithi028 ceph-mon[27786]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:17.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:17 smithi028 ceph-mon[27786]: Deploying daemon mds.d on smithi028 2024-04-24T08:07:17.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:17 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.28:6834/1751430859,v1:172.21.15.28:6835/1751430859] up:boot 2024-04-24T08:07:17.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:17 smithi028 ceph-mon[27786]: fsmap 3 up:standby 2024-04-24T08:07:17.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:17 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-24T08:07:17.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:17 smithi117 ceph-mon[32438]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:17.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:17 smithi117 ceph-mon[32438]: Deploying daemon mds.d on smithi028 2024-04-24T08:07:17.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:17 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.28:6834/1751430859,v1:172.21.15.28:6835/1751430859] up:boot 2024-04-24T08:07:17.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:17 smithi117 ceph-mon[32438]: fsmap 3 up:standby 2024-04-24T08:07:17.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:17 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-24T08:07:17.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:17 smithi194 ceph-mon[32836]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:17.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:17 smithi194 ceph-mon[32836]: Deploying daemon mds.d on smithi028 2024-04-24T08:07:17.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:17 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.28:6834/1751430859,v1:172.21.15.28:6835/1751430859] up:boot 2024-04-24T08:07:17.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:17 smithi194 ceph-mon[32836]: fsmap 3 up:standby 2024-04-24T08:07:17.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:17 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-24T08:07:17.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.727+0000 7f4ac6800640 1 -- 172.21.15.28:0/2162889202 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4ac002d870 con 0x7f4ac01040a0 2024-04-24T08:07:17.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.728+0000 7f4abe000640 1 -- 172.21.15.28:0/2162889202 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7f4ab40c13c0 con 0x7f4ac01040a0 2024-04-24T08:07:17.729 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:17.729 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-04-24T08:07:17.729 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 4 2024-04-24T08:07:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.731+0000 7f4ac6800640 1 -- 172.21.15.28:0/2162889202 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4aa80a3bb0 msgr2=0x7f4aa80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.731+0000 7f4ac6800640 1 --2- 172.21.15.28:0/2162889202 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4aa80a3bb0 0x7f4aa80a6070 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f4ac019b8f0 tx=0x7f4aac005f90 comp rx=0 tx=0).stop 2024-04-24T08:07:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.731+0000 7f4ac6800640 1 -- 172.21.15.28:0/2162889202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ac01040a0 msgr2=0x7f4ac019a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:17.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.731+0000 7f4ac6800640 1 --2- 172.21.15.28:0/2162889202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ac01040a0 0x7f4ac019a3e0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f4ab4031440 tx=0x7f4ab4032ce0 comp rx=0 tx=0).stop 2024-04-24T08:07:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.731+0000 7f4ac6800640 1 -- 172.21.15.28:0/2162889202 shutdown_connections 2024-04-24T08:07:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.731+0000 7f4ac6800640 1 --2- 172.21.15.28:0/2162889202 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4aa80a3bb0 0x7f4aa80a6070 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.731+0000 7f4ac6800640 1 --2- 172.21.15.28:0/2162889202 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4ac0105c50 0x7f4ac01a1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.731+0000 7f4ac6800640 1 --2- 172.21.15.28:0/2162889202 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4ac0105290 0x7f4ac019a920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.731+0000 7f4ac6800640 1 --2- 172.21.15.28:0/2162889202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ac01040a0 0x7f4ac019a3e0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.731+0000 7f4ac6800640 1 -- 172.21.15.28:0/2162889202 >> 172.21.15.28:0/2162889202 conn(0x7f4ac00ff610 msgr2=0x7f4ac0108610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.731+0000 7f4ac6800640 1 -- 172.21.15.28:0/2162889202 shutdown_connections 2024-04-24T08:07:17.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:17.731+0000 7f4ac6800640 1 -- 172.21.15.28:0/2162889202 wait complete. 2024-04-24T08:07:18.460 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:18 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2162889202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:18.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:18 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2162889202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:18.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:18 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2162889202' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:19.287 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:19 smithi028 ceph-mon[27786]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:19.539 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 24 08:07:19 smithi028 systemd[1]: Starting Ceph mds.d for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:07:19.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:19 smithi117 ceph-mon[32438]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:19.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:19 smithi194 ceph-mon[32836]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:19.750 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:19.958 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 24 08:07:19 smithi028 podman[66022]: 2024-04-24 08:07:19.587490722 +0000 UTC m=+0.041676606 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:07:19.959 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 24 08:07:19 smithi028 podman[66022]: 2024-04-24 08:07:19.696285141 +0000 UTC m=+0.150471048 container create 9ef75adc945fc74cf88f4e78eb881427cfdced967a1536b2c81d62ca6d17b220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-d, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-24T08:07:19.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.967+0000 7f6748800640 1 -- 172.21.15.28:0/2459161381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6740105e10 msgr2=0x7f674010c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:19.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.967+0000 7f6748800640 1 --2- 172.21.15.28:0/2459161381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6740105e10 0x7f674010c6a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f673c009be0 tx=0x7f673c02a6c0 comp rx=0 tx=0).stop 2024-04-24T08:07:19.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.967+0000 7f6748800640 1 -- 172.21.15.28:0/2459161381 shutdown_connections 2024-04-24T08:07:19.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.967+0000 7f6748800640 1 --2- 172.21.15.28:0/2459161381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6740105e10 0x7f674010c6a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:19.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.967+0000 7f6748800640 1 --2- 172.21.15.28:0/2459161381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6740105450 0x7f67401058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:19.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.967+0000 7f6748800640 1 --2- 172.21.15.28:0/2459161381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6740104260 0x7f6740104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:19.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.967+0000 7f6748800640 1 -- 172.21.15.28:0/2459161381 >> 172.21.15.28:0/2459161381 conn(0x7f67400ff7f0 msgr2=0x7f6740101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:19.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.967+0000 7f6748800640 1 -- 172.21.15.28:0/2459161381 shutdown_connections 2024-04-24T08:07:19.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.968+0000 7f6748800640 1 -- 172.21.15.28:0/2459161381 wait complete. 2024-04-24T08:07:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.969+0000 7f6748800640 1 Processor -- start 2024-04-24T08:07:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.969+0000 7f6748800640 1 -- start start 2024-04-24T08:07:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.969+0000 7f6748800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6740104260 0x7f674019a510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.969+0000 7f6748800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6740105450 0x7f674019aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.969+0000 7f6748800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6740105e10 0x7f67401a1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.969+0000 7f6748800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f674010fef0 con 0x7f6740105e10 2024-04-24T08:07:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.969+0000 7f6748800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f674010fd70 con 0x7f6740104260 2024-04-24T08:07:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.969+0000 7f6748800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6740110070 con 0x7f6740105450 2024-04-24T08:07:19.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.970+0000 7f6745400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6740105450 0x7f674019aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:19.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.970+0000 7f6745e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6740104260 0x7f674019a510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:19.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.970+0000 7f6745e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6740104260 0x7f674019a510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:57710/0 (socket says 172.21.15.28:57710) 2024-04-24T08:07:19.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.970+0000 7f6745e00640 1 -- 172.21.15.28:0/3104456738 learned_addr learned my addr 172.21.15.28:0/3104456738 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:19.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.970+0000 7f6745400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6740105450 0x7f674019aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:41114/0 (socket says 172.21.15.28:41114) 2024-04-24T08:07:19.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.970+0000 7f6746800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6740105e10 0x7f67401a1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:19.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.970+0000 7f6745400640 1 -- 172.21.15.28:0/3104456738 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6740104260 msgr2=0x7f674019a510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:19.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.970+0000 7f6745400640 1 --2- 172.21.15.28:0/3104456738 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6740104260 0x7f674019a510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:19.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.970+0000 7f6745400640 1 -- 172.21.15.28:0/3104456738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6740105e10 msgr2=0x7f67401a1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:19.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.970+0000 7f6745400640 1 --2- 172.21.15.28:0/3104456738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6740105e10 0x7f67401a1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:19.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.970+0000 7f6745400640 1 -- 172.21.15.28:0/3104456738 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67401a2130 con 0x7f6740105450 2024-04-24T08:07:19.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.970+0000 7f6746800640 1 --2- 172.21.15.28:0/3104456738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6740105e10 0x7f67401a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:19.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.970+0000 7f6745e00640 1 --2- 172.21.15.28:0/3104456738 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6740104260 0x7f674019a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:19.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.971+0000 7f6745400640 1 --2- 172.21.15.28:0/3104456738 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6740105450 0x7f674019aa50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f673000d720 tx=0x7f673000dbe0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:19.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.971+0000 7f673aa00640 1 -- 172.21.15.28:0/3104456738 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6730017070 con 0x7f6740105450 2024-04-24T08:07:19.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.971+0000 7f6748800640 1 -- 172.21.15.28:0/3104456738 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67401a2420 con 0x7f6740105450 2024-04-24T08:07:19.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.971+0000 7f673aa00640 1 -- 172.21.15.28:0/3104456738 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f673000de30 con 0x7f6740105450 2024-04-24T08:07:19.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.971+0000 7f673aa00640 1 -- 172.21.15.28:0/3104456738 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67300047a0 con 0x7f6740105450 2024-04-24T08:07:19.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.971+0000 7f6748800640 1 -- 172.21.15.28:0/3104456738 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67401a2970 con 0x7f6740105450 2024-04-24T08:07:19.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.972+0000 7f6748800640 1 -- 172.21.15.28:0/3104456738 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f674010b600 con 0x7f6740105450 2024-04-24T08:07:19.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.974+0000 7f673aa00640 1 -- 172.21.15.28:0/3104456738 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f67300099d0 con 0x7f6740105450 2024-04-24T08:07:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.976+0000 7f673aa00640 1 --2- 172.21.15.28:0/3104456738 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f67280a3c30 0x7f67280a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.977+0000 7f673aa00640 1 -- 172.21.15.28:0/3104456738 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f67300cfba0 con 0x7f6740105450 2024-04-24T08:07:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.977+0000 7f6745e00640 1 --2- 172.21.15.28:0/3104456738 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f67280a3c30 0x7f67280a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:19.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.977+0000 7f6745e00640 1 --2- 172.21.15.28:0/3104456738 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f67280a3c30 0x7f67280a60f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f67340096c0 tx=0x7f6734009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:19.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:19.980+0000 7f673aa00640 1 -- 172.21.15.28:0/3104456738 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6730020020 con 0x7f6740105450 2024-04-24T08:07:20.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:20.254+0000 7f6748800640 1 -- 172.21.15.28:0/3104456738 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f6740062c00 con 0x7f6740105450 2024-04-24T08:07:20.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:20.255+0000 7f673aa00640 1 -- 172.21.15.28:0/3104456738 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7f673009c980 con 0x7f6740105450 2024-04-24T08:07:20.256 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:20.256 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-24T08:07:20.256 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 5 2024-04-24T08:07:20.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:20.258+0000 7f6748800640 1 -- 172.21.15.28:0/3104456738 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f67280a3c30 msgr2=0x7f67280a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:20.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:20.258+0000 7f6748800640 1 --2- 172.21.15.28:0/3104456738 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f67280a3c30 0x7f67280a60f0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f67340096c0 tx=0x7f6734009260 comp rx=0 tx=0).stop 2024-04-24T08:07:20.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:20.258+0000 7f6748800640 1 -- 172.21.15.28:0/3104456738 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6740105450 msgr2=0x7f674019aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:20.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:20.258+0000 7f6748800640 1 --2- 172.21.15.28:0/3104456738 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6740105450 0x7f674019aa50 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f673000d720 tx=0x7f673000dbe0 comp rx=0 tx=0).stop 2024-04-24T08:07:20.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:20.258+0000 7f6748800640 1 -- 172.21.15.28:0/3104456738 shutdown_connections 2024-04-24T08:07:20.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:20.258+0000 7f6748800640 1 --2- 172.21.15.28:0/3104456738 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f67280a3c30 0x7f67280a60f0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:20.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:20.258+0000 7f6748800640 1 --2- 172.21.15.28:0/3104456738 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6740105e10 0x7f67401a1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:20.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:20.258+0000 7f6748800640 1 --2- 172.21.15.28:0/3104456738 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6740105450 0x7f674019aa50 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:20.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:20.258+0000 7f6748800640 1 --2- 172.21.15.28:0/3104456738 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6740104260 0x7f674019a510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:20.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:20.258+0000 7f6748800640 1 -- 172.21.15.28:0/3104456738 >> 172.21.15.28:0/3104456738 conn(0x7f67400ff7f0 msgr2=0x7f67401087d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:20.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:20.258+0000 7f6748800640 1 -- 172.21.15.28:0/3104456738 shutdown_connections 2024-04-24T08:07:20.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:20.258+0000 7f6748800640 1 -- 172.21.15.28:0/3104456738 wait complete. 2024-04-24T08:07:20.277 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 24 08:07:19 smithi028 podman[66022]: 2024-04-24 08:07:19.958357687 +0000 UTC m=+0.412543647 container init 9ef75adc945fc74cf88f4e78eb881427cfdced967a1536b2c81d62ca6d17b220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-d, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-24T08:07:20.278 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 24 08:07:19 smithi028 podman[66022]: 2024-04-24 08:07:19.968292834 +0000 UTC m=+0.422478705 container start 9ef75adc945fc74cf88f4e78eb881427cfdced967a1536b2c81d62ca6d17b220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-d, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:07:20.278 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 24 08:07:20 smithi028 ceph-mds[66074]: -- 172.21.15.28:0/2053462137 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55bc02ddfa40 con 0x55bc02db2800 2024-04-24T08:07:20.278 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 24 08:07:20 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-d[66053]: starting mds.d at 2024-04-24T08:07:20.278 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 24 08:07:20 smithi028 bash[66022]: 9ef75adc945fc74cf88f4e78eb881427cfdced967a1536b2c81d62ca6d17b220 2024-04-24T08:07:20.278 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 24 08:07:20 smithi028 systemd[1]: Started Ceph mds.d for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T08:07:21.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:21 smithi194 ceph-mon[32836]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:21.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:21 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:21.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:21 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:21.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:21 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.28:6836/711950790,v1:172.21.15.28:6837/711950790] up:boot 2024-04-24T08:07:21.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:21 smithi194 ceph-mon[32836]: fsmap 4 up:standby 2024-04-24T08:07:21.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:21 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-24T08:07:21.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:21 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:21.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:21 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:21.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:21 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:21.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:21 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:21.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:21 smithi194 ceph-mon[32836]: Deploying daemon mds.e on smithi117 2024-04-24T08:07:21.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:21 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3104456738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:21.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:21 smithi028 ceph-mon[27786]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:21.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:21 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:21.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:21 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:21.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:21 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.28:6836/711950790,v1:172.21.15.28:6837/711950790] up:boot 2024-04-24T08:07:21.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:21 smithi028 ceph-mon[27786]: fsmap 4 up:standby 2024-04-24T08:07:21.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:21 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-24T08:07:21.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:21 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:21.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:21 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:21.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:21 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:21.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:21 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:21.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:21 smithi028 ceph-mon[27786]: Deploying daemon mds.e on smithi117 2024-04-24T08:07:21.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:21 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3104456738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:21 smithi117 ceph-mon[32438]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:21 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:21 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:21 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.28:6836/711950790,v1:172.21.15.28:6837/711950790] up:boot 2024-04-24T08:07:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:21 smithi117 ceph-mon[32438]: fsmap 4 up:standby 2024-04-24T08:07:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:21 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-24T08:07:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:21 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:21 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:21 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:21 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:21 smithi117 ceph-mon[32438]: Deploying daemon mds.e on smithi117 2024-04-24T08:07:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:21 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3104456738' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:22.280 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:22.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.510+0000 7f672e400640 1 -- 172.21.15.28:0/2211958219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6728102900 msgr2=0x7f6728104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:22.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.510+0000 7f672e400640 1 --2- 172.21.15.28:0/2211958219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6728102900 0x7f6728104cf0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f6718009ca0 tx=0x7f671802a8c0 comp rx=0 tx=0).stop 2024-04-24T08:07:22.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.510+0000 7f672e400640 1 -- 172.21.15.28:0/2211958219 shutdown_connections 2024-04-24T08:07:22.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.510+0000 7f672e400640 1 --2- 172.21.15.28:0/2211958219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6728107b60 0x7f6728109fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:22.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.510+0000 7f672e400640 1 --2- 172.21.15.28:0/2211958219 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6728105230 0x7f6728107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:22.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.510+0000 7f672e400640 1 --2- 172.21.15.28:0/2211958219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6728102900 0x7f6728104cf0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:22.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.510+0000 7f672e400640 1 -- 172.21.15.28:0/2211958219 >> 172.21.15.28:0/2211958219 conn(0x7f67280fc310 msgr2=0x7f67280fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:22.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.510+0000 7f672e400640 1 -- 172.21.15.28:0/2211958219 shutdown_connections 2024-04-24T08:07:22.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.510+0000 7f672e400640 1 -- 172.21.15.28:0/2211958219 wait complete. 2024-04-24T08:07:22.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.511+0000 7f672e400640 1 Processor -- start 2024-04-24T08:07:22.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.511+0000 7f672e400640 1 -- start start 2024-04-24T08:07:22.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.512+0000 7f672e400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6728102900 0x7f67281981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:22.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.512+0000 7f672e400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6728105230 0x7f6728198720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:22.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.512+0000 7f672e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6728107b60 0x7f672819f790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:22.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.512+0000 7f672e400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f672810db30 con 0x7f6728107b60 2024-04-24T08:07:22.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.512+0000 7f672e400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f672810d9b0 con 0x7f6728105230 2024-04-24T08:07:22.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.512+0000 7f672e400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f672810dcb0 con 0x7f6728102900 2024-04-24T08:07:22.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.512+0000 7f672d000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6728102900 0x7f67281981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:22.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.512+0000 7f672da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6728107b60 0x7f672819f790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:22.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.512+0000 7f6727e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6728105230 0x7f6728198720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:22.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.512+0000 7f672d000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6728102900 0x7f67281981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:41146/0 (socket says 172.21.15.28:41146) 2024-04-24T08:07:22.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.512+0000 7f6727e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6728105230 0x7f6728198720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:57736/0 (socket says 172.21.15.28:57736) 2024-04-24T08:07:22.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.512+0000 7f672d000640 1 -- 172.21.15.28:0/1464976854 learned_addr learned my addr 172.21.15.28:0/1464976854 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:22.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.512+0000 7f672da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6728107b60 0x7f672819f790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54176/0 (socket says 172.21.15.28:54176) 2024-04-24T08:07:22.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.513+0000 7f672d000640 1 -- 172.21.15.28:0/1464976854 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6728105230 msgr2=0x7f6728198720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:22.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.513+0000 7f672d000640 1 --2- 172.21.15.28:0/1464976854 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6728105230 0x7f6728198720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:22.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.513+0000 7f672d000640 1 -- 172.21.15.28:0/1464976854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6728107b60 msgr2=0x7f672819f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.513+0000 7f672d000640 1 --2- 172.21.15.28:0/1464976854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6728107b60 0x7f672819f790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.513+0000 7f672d000640 1 -- 172.21.15.28:0/1464976854 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f672819fe00 con 0x7f6728102900 2024-04-24T08:07:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.513+0000 7f672da00640 1 --2- 172.21.15.28:0/1464976854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6728107b60 0x7f672819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.513+0000 7f672d000640 1 --2- 172.21.15.28:0/1464976854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6728102900 0x7f67281981e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6718031a40 tx=0x7f6718031ee0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.513+0000 7f6725600640 1 -- 172.21.15.28:0/1464976854 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671802fa20 con 0x7f6728102900 2024-04-24T08:07:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.513+0000 7f6725600640 1 -- 172.21.15.28:0/1464976854 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f671802fbc0 con 0x7f6728102900 2024-04-24T08:07:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.513+0000 7f6725600640 1 -- 172.21.15.28:0/1464976854 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f671803a780 con 0x7f6728102900 2024-04-24T08:07:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.513+0000 7f672e400640 1 -- 172.21.15.28:0/1464976854 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67281a0090 con 0x7f6728102900 2024-04-24T08:07:22.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.514+0000 7f672e400640 1 -- 172.21.15.28:0/1464976854 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67281a0630 con 0x7f6728102900 2024-04-24T08:07:22.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.515+0000 7f672e400640 1 -- 172.21.15.28:0/1464976854 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f67280fde00 con 0x7f6728102900 2024-04-24T08:07:22.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.517+0000 7f6725600640 1 -- 172.21.15.28:0/1464976854 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f671803a920 con 0x7f6728102900 2024-04-24T08:07:22.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.518+0000 7f6725600640 1 --2- 172.21.15.28:0/1464976854 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f670c0a3c30 0x7f670c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:22.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.518+0000 7f6725600640 1 -- 172.21.15.28:0/1464976854 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f67180f1480 con 0x7f6728102900 2024-04-24T08:07:22.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.518+0000 7f6727e00640 1 --2- 172.21.15.28:0/1464976854 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f670c0a3c30 0x7f670c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:22.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.519+0000 7f6727e00640 1 --2- 172.21.15.28:0/1464976854 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f670c0a3c30 0x7f670c0a60f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f67281996f0 tx=0x7f6714005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:22.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.522+0000 7f6725600640 1 -- 172.21.15.28:0/1464976854 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f67180be260 con 0x7f6728102900 2024-04-24T08:07:22.621 INFO:journalctl@ceph.mds.e.smithi117.stdout:Apr 24 08:07:22 smithi117 systemd[1]: Starting Ceph mds.e for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:07:22.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.777+0000 7f672e400640 1 -- 172.21.15.28:0/1464976854 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f672802d620 con 0x7f6728102900 2024-04-24T08:07:22.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.778+0000 7f6725600640 1 -- 172.21.15.28:0/1464976854 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3871 (secure 0 0 0) 0x7f67180bdb60 con 0x7f6728102900 2024-04-24T08:07:22.778 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:22.778 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5}],"filesystems":[]} 2024-04-24T08:07:22.779 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 5 2024-04-24T08:07:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.780+0000 7f672e400640 1 -- 172.21.15.28:0/1464976854 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f670c0a3c30 msgr2=0x7f670c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.780+0000 7f672e400640 1 --2- 172.21.15.28:0/1464976854 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f670c0a3c30 0x7f670c0a60f0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f67281996f0 tx=0x7f6714005f90 comp rx=0 tx=0).stop 2024-04-24T08:07:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.780+0000 7f672e400640 1 -- 172.21.15.28:0/1464976854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6728102900 msgr2=0x7f67281981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.780+0000 7f672e400640 1 --2- 172.21.15.28:0/1464976854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6728102900 0x7f67281981e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f6718031a40 tx=0x7f6718031ee0 comp rx=0 tx=0).stop 2024-04-24T08:07:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.781+0000 7f672e400640 1 -- 172.21.15.28:0/1464976854 shutdown_connections 2024-04-24T08:07:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.781+0000 7f672e400640 1 --2- 172.21.15.28:0/1464976854 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f670c0a3c30 0x7f670c0a60f0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.781+0000 7f672e400640 1 --2- 172.21.15.28:0/1464976854 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6728107b60 0x7f672819f790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:22.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.781+0000 7f672e400640 1 --2- 172.21.15.28:0/1464976854 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6728105230 0x7f6728198720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:22.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.781+0000 7f672e400640 1 --2- 172.21.15.28:0/1464976854 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6728102900 0x7f67281981e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:22.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.781+0000 7f672e400640 1 -- 172.21.15.28:0/1464976854 >> 172.21.15.28:0/1464976854 conn(0x7f67280fc310 msgr2=0x7f6728103300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:22.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.781+0000 7f672e400640 1 -- 172.21.15.28:0/1464976854 shutdown_connections 2024-04-24T08:07:22.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:22.781+0000 7f672e400640 1 -- 172.21.15.28:0/1464976854 wait complete. 2024-04-24T08:07:22.876 INFO:journalctl@ceph.mds.e.smithi117.stdout:Apr 24 08:07:22 smithi117 podman[62242]: 2024-04-24 08:07:22.524471202 +0000 UTC m=+0.018272284 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:07:22.876 INFO:journalctl@ceph.mds.e.smithi117.stdout:Apr 24 08:07:22 smithi117 podman[62242]: 2024-04-24 08:07:22.639318747 +0000 UTC m=+0.133119807 container create 25e09601568a697f1dda8718d24a97f60ac361997a738a38f880a7fa2b48c0a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-e, GIT_CLEAN=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:07:22.877 INFO:journalctl@ceph.mds.e.smithi117.stdout:Apr 24 08:07:22 smithi117 podman[62242]: 2024-04-24 08:07:22.850482144 +0000 UTC m=+0.344283204 container init 25e09601568a697f1dda8718d24a97f60ac361997a738a38f880a7fa2b48c0a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-e, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True) 2024-04-24T08:07:22.877 INFO:journalctl@ceph.mds.e.smithi117.stdout:Apr 24 08:07:22 smithi117 podman[62242]: 2024-04-24 08:07:22.853164786 +0000 UTC m=+0.346965846 container start 25e09601568a697f1dda8718d24a97f60ac361997a738a38f880a7fa2b48c0a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-e, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-04-24T08:07:22.877 INFO:journalctl@ceph.mds.e.smithi117.stdout:Apr 24 08:07:22 smithi117 ceph-mds[62262]: -- 172.21.15.117:0/2809423273 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55c270009a40 con 0x55c26ffdc800 2024-04-24T08:07:23.127 INFO:journalctl@ceph.mds.e.smithi117.stdout:Apr 24 08:07:22 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-e[62258]: starting mds.e at 2024-04-24T08:07:23.128 INFO:journalctl@ceph.mds.e.smithi117.stdout:Apr 24 08:07:22 smithi117 bash[62242]: 25e09601568a697f1dda8718d24a97f60ac361997a738a38f880a7fa2b48c0a5 2024-04-24T08:07:23.128 INFO:journalctl@ceph.mds.e.smithi117.stdout:Apr 24 08:07:22 smithi117 systemd[1]: Started Ceph mds.e for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T08:07:23.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:23 smithi194 ceph-mon[32836]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:23.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:23 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1464976854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:23.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:23 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:23.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:23 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:23.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:23 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:23.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:23 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:23.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:23 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:23.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:23 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:23.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:23 smithi028 ceph-mon[27786]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:23.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:23 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1464976854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:23.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:23 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:23.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:23 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:23.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:23 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:23.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:23 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:23.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:23 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:23.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:23 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:23.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:23 smithi117 ceph-mon[32438]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:23.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:23 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1464976854' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:23.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:23 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:23.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:23 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:23.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:23 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:23.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:23 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:23.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:23 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:23.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:23 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:24.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:24 smithi028 ceph-mon[27786]: Deploying daemon mds.f on smithi194 2024-04-24T08:07:24.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:24 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.117:6834/2458524548,v1:172.21.15.117:6835/2458524548] up:boot 2024-04-24T08:07:24.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:24 smithi028 ceph-mon[27786]: fsmap 5 up:standby 2024-04-24T08:07:24.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:24 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T08:07:24.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:24 smithi194 ceph-mon[32836]: Deploying daemon mds.f on smithi194 2024-04-24T08:07:24.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:24 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.117:6834/2458524548,v1:172.21.15.117:6835/2458524548] up:boot 2024-04-24T08:07:24.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:24 smithi194 ceph-mon[32836]: fsmap 5 up:standby 2024-04-24T08:07:24.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:24 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T08:07:24.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:24 smithi117 ceph-mon[32438]: Deploying daemon mds.f on smithi194 2024-04-24T08:07:24.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:24 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.117:6834/2458524548,v1:172.21.15.117:6835/2458524548] up:boot 2024-04-24T08:07:24.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:24 smithi117 ceph-mon[32438]: fsmap 5 up:standby 2024-04-24T08:07:24.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:24 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T08:07:24.802 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:25.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.019+0000 7fa1d6e00640 1 -- 172.21.15.28:0/3751154301 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1d010a210 msgr2=0x7fa1d0110aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:25.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.019+0000 7fa1d6e00640 1 --2- 172.21.15.28:0/3751154301 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1d010a210 0x7fa1d0110aa0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8009ca0 tx=0x7fa1c802a8c0 comp rx=0 tx=0).stop 2024-04-24T08:07:25.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.019+0000 7fa1d6e00640 1 -- 172.21.15.28:0/3751154301 shutdown_connections 2024-04-24T08:07:25.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.019+0000 7fa1d6e00640 1 --2- 172.21.15.28:0/3751154301 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1d010a210 0x7fa1d0110aa0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:25.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.019+0000 7fa1d6e00640 1 --2- 172.21.15.28:0/3751154301 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1d0109850 0x7fa1d0109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:25.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.019+0000 7fa1d6e00640 1 --2- 172.21.15.28:0/3751154301 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d0108660 0x7fa1d0108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:25.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.019+0000 7fa1d6e00640 1 -- 172.21.15.28:0/3751154301 >> 172.21.15.28:0/3751154301 conn(0x7fa1d0077620 msgr2=0x7fa1d0079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.019+0000 7fa1d6e00640 1 -- 172.21.15.28:0/3751154301 shutdown_connections 2024-04-24T08:07:25.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.019+0000 7fa1d6e00640 1 -- 172.21.15.28:0/3751154301 wait complete. 2024-04-24T08:07:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.020+0000 7fa1d6e00640 1 Processor -- start 2024-04-24T08:07:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.020+0000 7fa1d6e00640 1 -- start start 2024-04-24T08:07:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.021+0000 7fa1d6e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1d0108660 0x7fa1d019e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.021+0000 7fa1d6e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d0109850 0x7fa1d019ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.021+0000 7fa1d6e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1d010a210 0x7fa1d01a5ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.021+0000 7fa1d6e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d0114330 con 0x7fa1d0109850 2024-04-24T08:07:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.021+0000 7fa1d6e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d01141b0 con 0x7fa1d010a210 2024-04-24T08:07:25.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.021+0000 7fa1d6e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa1d01144b0 con 0x7fa1d0108660 2024-04-24T08:07:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.021+0000 7fa1cfe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1d0108660 0x7fa1d019e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.021+0000 7fa1cfe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1d0108660 0x7fa1d019e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:41162/0 (socket says 172.21.15.28:41162) 2024-04-24T08:07:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.021+0000 7fa1cf400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d0109850 0x7fa1d019ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.021+0000 7fa1cfe00640 1 -- 172.21.15.28:0/4173792707 learned_addr learned my addr 172.21.15.28:0/4173792707 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.021+0000 7fa1d4e00640 1 --2- 172.21.15.28:0/4173792707 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1d010a210 0x7fa1d01a5ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.022+0000 7fa1cfe00640 1 -- 172.21.15.28:0/4173792707 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1d010a210 msgr2=0x7fa1d01a5ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.022+0000 7fa1cfe00640 1 --2- 172.21.15.28:0/4173792707 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1d010a210 0x7fa1d01a5ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.022+0000 7fa1cfe00640 1 -- 172.21.15.28:0/4173792707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d0109850 msgr2=0x7fa1d019ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.022+0000 7fa1cfe00640 1 --2- 172.21.15.28:0/4173792707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d0109850 0x7fa1d019ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.022+0000 7fa1cfe00640 1 -- 172.21.15.28:0/4173792707 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa1d01a6540 con 0x7fa1d0108660 2024-04-24T08:07:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.022+0000 7fa1cf400640 1 --2- 172.21.15.28:0/4173792707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d0109850 0x7fa1d019ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.022+0000 7fa1cfe00640 1 --2- 172.21.15.28:0/4173792707 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1d0108660 0x7fa1d019e920 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa1c000e9c0 tx=0x7fa1c000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:25.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.022+0000 7fa1ccc00640 1 -- 172.21.15.28:0/4173792707 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c000cd80 con 0x7fa1d0108660 2024-04-24T08:07:25.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.022+0000 7fa1ccc00640 1 -- 172.21.15.28:0/4173792707 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa1c00035f0 con 0x7fa1d0108660 2024-04-24T08:07:25.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.022+0000 7fa1ccc00640 1 -- 172.21.15.28:0/4173792707 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa1c0010690 con 0x7fa1d0108660 2024-04-24T08:07:25.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.022+0000 7fa1d6e00640 1 -- 172.21.15.28:0/4173792707 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa1d01a6830 con 0x7fa1d0108660 2024-04-24T08:07:25.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.022+0000 7fa1d6e00640 1 -- 172.21.15.28:0/4173792707 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa1d01a6d30 con 0x7fa1d0108660 2024-04-24T08:07:25.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.023+0000 7fa1d6e00640 1 -- 172.21.15.28:0/4173792707 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa1d010f920 con 0x7fa1d0108660 2024-04-24T08:07:25.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.026+0000 7fa1ccc00640 1 -- 172.21.15.28:0/4173792707 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fa1c00037a0 con 0x7fa1d0108660 2024-04-24T08:07:25.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.027+0000 7fa1ccc00640 1 --2- 172.21.15.28:0/4173792707 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa1b40a3c30 0x7fa1b40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:25.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.027+0000 7fa1ccc00640 1 -- 172.21.15.28:0/4173792707 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fa1c00d0400 con 0x7fa1d0108660 2024-04-24T08:07:25.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.028+0000 7fa1cf400640 1 --2- 172.21.15.28:0/4173792707 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa1b40a3c30 0x7fa1b40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:25.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.028+0000 7fa1cf400640 1 --2- 172.21.15.28:0/4173792707 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa1b40a3c30 0x7fa1b40a60f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa1d019fe30 tx=0x7fa1bc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:25.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.029+0000 7fa1ccc00640 1 -- 172.21.15.28:0/4173792707 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa1c009d270 con 0x7fa1d0108660 2024-04-24T08:07:25.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.280+0000 7fa1d6e00640 1 -- 172.21.15.28:0/4173792707 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa1d002d8d0 con 0x7fa1d0108660 2024-04-24T08:07:25.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.281+0000 7fa1ccc00640 1 -- 172.21.15.28:0/4173792707 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4699 (secure 0 0 0) 0x7fa1c009cb70 con 0x7fa1d0108660 2024-04-24T08:07:25.282 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:25.282 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-04-24T08:07:25.282 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 6 2024-04-24T08:07:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.284+0000 7fa1d6e00640 1 -- 172.21.15.28:0/4173792707 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa1b40a3c30 msgr2=0x7fa1b40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.284+0000 7fa1d6e00640 1 --2- 172.21.15.28:0/4173792707 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa1b40a3c30 0x7fa1b40a60f0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fa1d019fe30 tx=0x7fa1bc005f90 comp rx=0 tx=0).stop 2024-04-24T08:07:25.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.284+0000 7fa1d6e00640 1 -- 172.21.15.28:0/4173792707 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1d0108660 msgr2=0x7fa1d019e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.284+0000 7fa1d6e00640 1 --2- 172.21.15.28:0/4173792707 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1d0108660 0x7fa1d019e920 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fa1c000e9c0 tx=0x7fa1c000ee80 comp rx=0 tx=0).stop 2024-04-24T08:07:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.285+0000 7fa1d6e00640 1 -- 172.21.15.28:0/4173792707 shutdown_connections 2024-04-24T08:07:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.285+0000 7fa1d6e00640 1 --2- 172.21.15.28:0/4173792707 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa1b40a3c30 0x7fa1b40a60f0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:25.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.285+0000 7fa1d6e00640 1 --2- 172.21.15.28:0/4173792707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa1d0109850 0x7fa1d019ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.285+0000 7fa1d6e00640 1 --2- 172.21.15.28:0/4173792707 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1d010a210 0x7fa1d01a5ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.285+0000 7fa1d6e00640 1 --2- 172.21.15.28:0/4173792707 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa1d0108660 0x7fa1d019e920 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.285+0000 7fa1d6e00640 1 -- 172.21.15.28:0/4173792707 >> 172.21.15.28:0/4173792707 conn(0x7fa1d0077620 msgr2=0x7fa1d010caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.285+0000 7fa1d6e00640 1 -- 172.21.15.28:0/4173792707 shutdown_connections 2024-04-24T08:07:25.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:25.285+0000 7fa1d6e00640 1 -- 172.21.15.28:0/4173792707 wait complete. 2024-04-24T08:07:25.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:25 smithi194 ceph-mon[32836]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:25.441 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:07:25 smithi194 systemd[1]: Starting Ceph mds.f for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:07:25.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:25 smithi028 ceph-mon[27786]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:25.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:25 smithi117 ceph-mon[32438]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:25.738 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:07:25 smithi194 podman[63518]: 2024-04-24 08:07:25.367817017 +0000 UTC m=+0.017950651 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:07:25.738 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:07:25 smithi194 podman[63518]: 2024-04-24 08:07:25.477326939 +0000 UTC m=+0.127460636 container create fc34c6ee65bff0444d5b77c1569b94e0d224fb2703ec46aee328222273775ea9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-f, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:07:26.190 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:07:25 smithi194 podman[63518]: 2024-04-24 08:07:25.738424092 +0000 UTC m=+0.388557723 container init fc34c6ee65bff0444d5b77c1569b94e0d224fb2703ec46aee328222273775ea9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-f, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-04-24T08:07:26.191 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:07:25 smithi194 podman[63518]: 2024-04-24 08:07:25.741282972 +0000 UTC m=+0.391416611 container start fc34c6ee65bff0444d5b77c1569b94e0d224fb2703ec46aee328222273775ea9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-f, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-24T08:07:26.191 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:07:25 smithi194 ceph-mds[63537]: -- 172.21.15.194:0/696318532 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55e09be1ba40 con 0x55e09bdee800 2024-04-24T08:07:26.191 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:07:25 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-f[63533]: starting mds.f at 2024-04-24T08:07:26.191 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:07:25 smithi194 bash[63518]: fc34c6ee65bff0444d5b77c1569b94e0d224fb2703ec46aee328222273775ea9 2024-04-24T08:07:26.191 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:07:25 smithi194 systemd[1]: Started Ceph mds.f for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T08:07:26.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:26 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/4173792707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:26.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:26.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:26.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:26.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:26.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:26.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:26 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:26 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/4173792707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:26.563 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:26 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:26.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:26 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/4173792707' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:26.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:26.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:26.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:26.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:26.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:26.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:26 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:27.304 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:27.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.524+0000 7f8b8be00640 1 -- 172.21.15.28:0/3622242061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b8c105230 msgr2=0x7f8b8c107620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:27.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.524+0000 7f8b8be00640 1 --2- 172.21.15.28:0/3622242061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b8c105230 0x7f8b8c107620 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f8b78009900 tx=0x7f8b7802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:07:27.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.524+0000 7f8b8be00640 1 -- 172.21.15.28:0/3622242061 shutdown_connections 2024-04-24T08:07:27.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.524+0000 7f8b8be00640 1 --2- 172.21.15.28:0/3622242061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b8c105230 0x7f8b8c107620 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:27.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.524+0000 7f8b8be00640 1 --2- 172.21.15.28:0/3622242061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b8c107b60 0x7f8b8c109fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:27.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.525+0000 7f8b8be00640 1 --2- 172.21.15.28:0/3622242061 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b8c102900 0x7f8b8c104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:27.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.525+0000 7f8b8be00640 1 -- 172.21.15.28:0/3622242061 >> 172.21.15.28:0/3622242061 conn(0x7f8b8c0fc2d0 msgr2=0x7f8b8c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:27.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.525+0000 7f8b8be00640 1 -- 172.21.15.28:0/3622242061 shutdown_connections 2024-04-24T08:07:27.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.525+0000 7f8b8be00640 1 -- 172.21.15.28:0/3622242061 wait complete. 2024-04-24T08:07:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.526+0000 7f8b8be00640 1 Processor -- start 2024-04-24T08:07:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.526+0000 7f8b8be00640 1 -- start start 2024-04-24T08:07:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.526+0000 7f8b8be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b8c102900 0x7f8b8c1981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.526+0000 7f8b8be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b8c105230 0x7f8b8c198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.526+0000 7f8b8be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b8c107b60 0x7f8b8c19f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.526+0000 7f8b8be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8c10db40 con 0x7f8b8c102900 2024-04-24T08:07:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.526+0000 7f8b8be00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8c10d9c0 con 0x7f8b8c107b60 2024-04-24T08:07:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.526+0000 7f8b8be00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8b8c10dcc0 con 0x7f8b8c105230 2024-04-24T08:07:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.526+0000 7f8b8aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b8c102900 0x7f8b8c1981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:27.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.526+0000 7f8b8aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b8c102900 0x7f8b8c1981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54200/0 (socket says 172.21.15.28:54200) 2024-04-24T08:07:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.526+0000 7f8b8aa00640 1 -- 172.21.15.28:0/651740324 learned_addr learned my addr 172.21.15.28:0/651740324 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.526+0000 7f8b8a000640 1 --2- 172.21.15.28:0/651740324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b8c105230 0x7f8b8c198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.526+0000 7f8b8b400640 1 --2- 172.21.15.28:0/651740324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b8c107b60 0x7f8b8c19f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.527+0000 7f8b8aa00640 1 -- 172.21.15.28:0/651740324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b8c105230 msgr2=0x7f8b8c198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.527+0000 7f8b8aa00640 1 --2- 172.21.15.28:0/651740324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b8c105230 0x7f8b8c198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.527+0000 7f8b8aa00640 1 -- 172.21.15.28:0/651740324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b8c107b60 msgr2=0x7f8b8c19f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.527+0000 7f8b8aa00640 1 --2- 172.21.15.28:0/651740324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b8c107b60 0x7f8b8c19f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.527+0000 7f8b8aa00640 1 -- 172.21.15.28:0/651740324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b8c19fde0 con 0x7f8b8c102900 2024-04-24T08:07:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.527+0000 7f8b8aa00640 1 --2- 172.21.15.28:0/651740324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b8c102900 0x7f8b8c1981c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c009d50 tx=0x7f8b7c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.527+0000 7f8b83400640 1 -- 172.21.15.28:0/651740324 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c003b70 con 0x7f8b8c102900 2024-04-24T08:07:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.527+0000 7f8b83400640 1 -- 172.21.15.28:0/651740324 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b7c003d10 con 0x7f8b8c102900 2024-04-24T08:07:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.527+0000 7f8b8be00640 1 -- 172.21.15.28:0/651740324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b8c1a00d0 con 0x7f8b8c102900 2024-04-24T08:07:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.527+0000 7f8b83400640 1 -- 172.21.15.28:0/651740324 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b7c0108f0 con 0x7f8b8c102900 2024-04-24T08:07:27.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.528+0000 7f8b8be00640 1 -- 172.21.15.28:0/651740324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b8c1a0640 con 0x7f8b8c102900 2024-04-24T08:07:27.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.529+0000 7f8b8be00640 1 -- 172.21.15.28:0/651740324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8b8c0fddc0 con 0x7f8b8c102900 2024-04-24T08:07:27.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.531+0000 7f8b83400640 1 -- 172.21.15.28:0/651740324 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8b7c004000 con 0x7f8b8c102900 2024-04-24T08:07:27.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.532+0000 7f8b83400640 1 --2- 172.21.15.28:0/651740324 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8b700a3c30 0x7f8b700a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:27.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.532+0000 7f8b83400640 1 -- 172.21.15.28:0/651740324 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f8b7c0d05a0 con 0x7f8b8c102900 2024-04-24T08:07:27.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.532+0000 7f8b8a000640 1 --2- 172.21.15.28:0/651740324 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8b700a3c30 0x7f8b700a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:27.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.533+0000 7f8b8a000640 1 --2- 172.21.15.28:0/651740324 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8b700a3c30 0x7f8b700a60f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8b7802a9e0 tx=0x7f8b780315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:27.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.537+0000 7f8b83400640 1 -- 172.21.15.28:0/651740324 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8b7c09d380 con 0x7f8b8c102900 2024-04-24T08:07:27.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:27 smithi028 ceph-mon[27786]: Deploying daemon mds.c on smithi194 2024-04-24T08:07:27.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:27 smithi028 ceph-mon[27786]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:27.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:27 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.194:6834/356867408,v1:172.21.15.194:6835/356867408] up:boot 2024-04-24T08:07:27.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:27 smithi028 ceph-mon[27786]: fsmap 6 up:standby 2024-04-24T08:07:27.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:27 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T08:07:27.613 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:27 smithi194 ceph-mon[32836]: Deploying daemon mds.c on smithi194 2024-04-24T08:07:27.613 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:27 smithi194 ceph-mon[32836]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:27.613 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:27 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.194:6834/356867408,v1:172.21.15.194:6835/356867408] up:boot 2024-04-24T08:07:27.613 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:27 smithi194 ceph-mon[32836]: fsmap 6 up:standby 2024-04-24T08:07:27.613 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:27 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T08:07:27.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:27 smithi117 ceph-mon[32438]: Deploying daemon mds.c on smithi194 2024-04-24T08:07:27.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:27 smithi117 ceph-mon[32438]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:27.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:27 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.194:6834/356867408,v1:172.21.15.194:6835/356867408] up:boot 2024-04-24T08:07:27.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:27 smithi117 ceph-mon[32438]: fsmap 6 up:standby 2024-04-24T08:07:27.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:27 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T08:07:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.784+0000 7f8b8be00640 1 -- 172.21.15.28:0/651740324 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8b8c02d620 con 0x7f8b8c102900 2024-04-24T08:07:27.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.786+0000 7f8b83400640 1 -- 172.21.15.28:0/651740324 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5524 (secure 0 0 0) 0x7f8b7c09cc80 con 0x7f8b8c102900 2024-04-24T08:07:27.787 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:27.787 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-04-24T08:07:27.787 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 7 2024-04-24T08:07:27.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.788+0000 7f8b8be00640 1 -- 172.21.15.28:0/651740324 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8b700a3c30 msgr2=0x7f8b700a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:27.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.788+0000 7f8b8be00640 1 --2- 172.21.15.28:0/651740324 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8b700a3c30 0x7f8b700a60f0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f8b7802a9e0 tx=0x7f8b780315b0 comp rx=0 tx=0).stop 2024-04-24T08:07:27.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.789+0000 7f8b8be00640 1 -- 172.21.15.28:0/651740324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b8c102900 msgr2=0x7f8b8c1981c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:27.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.789+0000 7f8b8be00640 1 --2- 172.21.15.28:0/651740324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b8c102900 0x7f8b8c1981c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8b7c009d50 tx=0x7f8b7c00c6a0 comp rx=0 tx=0).stop 2024-04-24T08:07:27.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.789+0000 7f8b8be00640 1 -- 172.21.15.28:0/651740324 shutdown_connections 2024-04-24T08:07:27.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.789+0000 7f8b8be00640 1 --2- 172.21.15.28:0/651740324 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8b700a3c30 0x7f8b700a60f0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:27.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.789+0000 7f8b8be00640 1 --2- 172.21.15.28:0/651740324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b8c107b60 0x7f8b8c19f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:27.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.789+0000 7f8b8be00640 1 --2- 172.21.15.28:0/651740324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8b8c105230 0x7f8b8c198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:27.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.789+0000 7f8b8be00640 1 --2- 172.21.15.28:0/651740324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8b8c102900 0x7f8b8c1981c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:27.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.789+0000 7f8b8be00640 1 -- 172.21.15.28:0/651740324 >> 172.21.15.28:0/651740324 conn(0x7f8b8c0fc2d0 msgr2=0x7f8b8c103240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:27.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.789+0000 7f8b8be00640 1 -- 172.21.15.28:0/651740324 shutdown_connections 2024-04-24T08:07:27.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:27.789+0000 7f8b8be00640 1 -- 172.21.15.28:0/651740324 wait complete. 2024-04-24T08:07:28.190 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:07:28 smithi194 systemd[1]: Starting Ceph mds.c for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:07:28.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:28 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/651740324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:28.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:28 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/651740324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:28.677 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:28 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/651740324' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:28.677 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:07:28 smithi194 podman[63824]: 2024-04-24 08:07:28.209680969 +0000 UTC m=+0.018712564 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:07:28.677 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:07:28 smithi194 podman[63824]: 2024-04-24 08:07:28.307242697 +0000 UTC m=+0.116274264 container create d1d18f3a853e002baebe18e0f190d16a45c754c1529e0286930b2b1b036d04c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-c, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-24T08:07:28.678 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:07:28 smithi194 podman[63824]: 2024-04-24 08:07:28.676875001 +0000 UTC m=+0.485906575 container init d1d18f3a853e002baebe18e0f190d16a45c754c1529e0286930b2b1b036d04c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-c, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:07:28.940 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:07:28 smithi194 podman[63824]: 2024-04-24 08:07:28.679844862 +0000 UTC m=+0.488876438 container start d1d18f3a853e002baebe18e0f190d16a45c754c1529e0286930b2b1b036d04c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-c, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-24T08:07:28.940 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:07:28 smithi194 ceph-mds[63843]: -- 172.21.15.194:0/3582802956 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a5177a1a40 con 0x55a517774c00 2024-04-24T08:07:28.940 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:07:28 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-c[63839]: starting mds.c at 2024-04-24T08:07:28.940 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:07:28 smithi194 bash[63824]: d1d18f3a853e002baebe18e0f190d16a45c754c1529e0286930b2b1b036d04c7 2024-04-24T08:07:28.941 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:07:28 smithi194 systemd[1]: Started Ceph mds.c for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T08:07:29.808 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.026+0000 7fb293e00640 1 -- 172.21.15.28:0/302115755 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294105270 msgr2=0x7fb2941056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.026+0000 7fb293e00640 1 --2- 172.21.15.28:0/302115755 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294105270 0x7fb2941056f0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb27c009900 tx=0x7fb27c02a400 comp rx=0 tx=0).stop 2024-04-24T08:07:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.026+0000 7fb293e00640 1 -- 172.21.15.28:0/302115755 shutdown_connections 2024-04-24T08:07:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.026+0000 7fb293e00640 1 --2- 172.21.15.28:0/302115755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb294105c30 0x7fb29410c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.026+0000 7fb293e00640 1 --2- 172.21.15.28:0/302115755 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294105270 0x7fb2941056f0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.026+0000 7fb293e00640 1 --2- 172.21.15.28:0/302115755 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb294104080 0x7fb2941044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.026+0000 7fb293e00640 1 -- 172.21.15.28:0/302115755 >> 172.21.15.28:0/302115755 conn(0x7fb2940ff610 msgr2=0x7fb294101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.026+0000 7fb293e00640 1 -- 172.21.15.28:0/302115755 shutdown_connections 2024-04-24T08:07:30.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.026+0000 7fb293e00640 1 -- 172.21.15.28:0/302115755 wait complete. 2024-04-24T08:07:30.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.028+0000 7fb293e00640 1 Processor -- start 2024-04-24T08:07:30.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.028+0000 7fb293e00640 1 -- start start 2024-04-24T08:07:30.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.028+0000 7fb293e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb294104080 0x7fb29419a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:30.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.028+0000 7fb293e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294105270 0x7fb29419a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.029+0000 7fb293e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb294105c30 0x7fb2941a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.029+0000 7fb293e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb29410fd60 con 0x7fb294105c30 2024-04-24T08:07:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.029+0000 7fb293e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb29410fbe0 con 0x7fb294104080 2024-04-24T08:07:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.029+0000 7fb293e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb29410fee0 con 0x7fb294105270 2024-04-24T08:07:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.029+0000 7fb293400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb294105c30 0x7fb2941a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.029+0000 7fb292a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb294104080 0x7fb29419a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.029+0000 7fb293400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb294105c30 0x7fb2941a1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55700/0 (socket says 172.21.15.28:55700) 2024-04-24T08:07:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.029+0000 7fb293400640 1 -- 172.21.15.28:0/4001489586 learned_addr learned my addr 172.21.15.28:0/4001489586 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.029+0000 7fb28be00640 1 --2- 172.21.15.28:0/4001489586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294105270 0x7fb29419a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.029+0000 7fb293400640 1 -- 172.21.15.28:0/4001489586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294105270 msgr2=0x7fb29419a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.029+0000 7fb293400640 1 --2- 172.21.15.28:0/4001489586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294105270 0x7fb29419a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.029+0000 7fb293400640 1 -- 172.21.15.28:0/4001489586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb294104080 msgr2=0x7fb29419a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.029+0000 7fb293400640 1 --2- 172.21.15.28:0/4001489586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb294104080 0x7fb29419a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:30.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.029+0000 7fb293400640 1 -- 172.21.15.28:0/4001489586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2941a1fd0 con 0x7fb294105c30 2024-04-24T08:07:30.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.029+0000 7fb292a00640 1 --2- 172.21.15.28:0/4001489586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb294104080 0x7fb29419a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:30.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.029+0000 7fb28be00640 1 --2- 172.21.15.28:0/4001489586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294105270 0x7fb29419a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:30.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.030+0000 7fb293400640 1 --2- 172.21.15.28:0/4001489586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb294105c30 0x7fb2941a1960 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb28c00eb10 tx=0x7fb28c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:30.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.030+0000 7fb28b400640 1 -- 172.21.15.28:0/4001489586 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c004b20 con 0x7fb294105c30 2024-04-24T08:07:30.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.030+0000 7fb28b400640 1 -- 172.21.15.28:0/4001489586 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb28c004cc0 con 0x7fb294105c30 2024-04-24T08:07:30.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.030+0000 7fb293e00640 1 -- 172.21.15.28:0/4001489586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2941a22c0 con 0x7fb294105c30 2024-04-24T08:07:30.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.030+0000 7fb28b400640 1 -- 172.21.15.28:0/4001489586 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb28c010760 con 0x7fb294105c30 2024-04-24T08:07:30.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.030+0000 7fb293e00640 1 -- 172.21.15.28:0/4001489586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2941a27e0 con 0x7fb294105c30 2024-04-24T08:07:30.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.032+0000 7fb293e00640 1 -- 172.21.15.28:0/4001489586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb29410b340 con 0x7fb294105c30 2024-04-24T08:07:30.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.034+0000 7fb28b400640 1 -- 172.21.15.28:0/4001489586 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fb28c005020 con 0x7fb294105c30 2024-04-24T08:07:30.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.036+0000 7fb28b400640 1 --2- 172.21.15.28:0/4001489586 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2780a3bb0 0x7fb2780a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:30.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.037+0000 7fb28b400640 1 -- 172.21.15.28:0/4001489586 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fb28c0d0620 con 0x7fb294105c30 2024-04-24T08:07:30.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.037+0000 7fb292a00640 1 --2- 172.21.15.28:0/4001489586 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2780a3bb0 0x7fb2780a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:30.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.038+0000 7fb292a00640 1 --2- 172.21.15.28:0/4001489586 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2780a3bb0 0x7fb2780a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb284005fd0 tx=0x7fb28400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:30.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.041+0000 7fb28b400640 1 -- 172.21.15.28:0/4001489586 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb28c09d400 con 0x7fb294105c30 2024-04-24T08:07:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:29 smithi117 ceph-mon[32438]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:29 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.194:6836/3046177244,v1:172.21.15.194:6837/3046177244] up:boot 2024-04-24T08:07:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:29 smithi117 ceph-mon[32438]: fsmap 7 up:standby 2024-04-24T08:07:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:29 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:07:30.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:29 smithi194 ceph-mon[32836]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:30.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:30.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:30.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:30.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:30.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:30.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:30.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:29 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.194:6836/3046177244,v1:172.21.15.194:6837/3046177244] up:boot 2024-04-24T08:07:30.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:29 smithi194 ceph-mon[32836]: fsmap 7 up:standby 2024-04-24T08:07:30.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:29 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:07:30.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:29 smithi028 ceph-mon[27786]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:30.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:30.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:30.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:30.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:30.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:30.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:30.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:29 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.194:6836/3046177244,v1:172.21.15.194:6837/3046177244] up:boot 2024-04-24T08:07:30.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:29 smithi028 ceph-mon[27786]: fsmap 7 up:standby 2024-04-24T08:07:30.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:29 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:07:30.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.292+0000 7fb293e00640 1 -- 172.21.15.28:0/4001489586 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb29402d620 con 0x7fb294105c30 2024-04-24T08:07:30.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.294+0000 7fb28b400640 1 -- 172.21.15.28:0/4001489586 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6352 (secure 0 0 0) 0x7fb28c09cd00 con 0x7fb294105c30 2024-04-24T08:07:30.295 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:30.295 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-24T08:07:30.295 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 8 2024-04-24T08:07:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.296+0000 7fb293e00640 1 -- 172.21.15.28:0/4001489586 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2780a3bb0 msgr2=0x7fb2780a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.296+0000 7fb293e00640 1 --2- 172.21.15.28:0/4001489586 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2780a3bb0 0x7fb2780a6070 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fb284005fd0 tx=0x7fb28400b410 comp rx=0 tx=0).stop 2024-04-24T08:07:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.296+0000 7fb293e00640 1 -- 172.21.15.28:0/4001489586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb294105c30 msgr2=0x7fb2941a1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.296+0000 7fb293e00640 1 --2- 172.21.15.28:0/4001489586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb294105c30 0x7fb2941a1960 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb28c00eb10 tx=0x7fb28c00c2d0 comp rx=0 tx=0).stop 2024-04-24T08:07:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.296+0000 7fb293e00640 1 -- 172.21.15.28:0/4001489586 shutdown_connections 2024-04-24T08:07:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.296+0000 7fb293e00640 1 --2- 172.21.15.28:0/4001489586 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2780a3bb0 0x7fb2780a6070 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.296+0000 7fb293e00640 1 --2- 172.21.15.28:0/4001489586 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb294105c30 0x7fb2941a1960 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.296+0000 7fb293e00640 1 --2- 172.21.15.28:0/4001489586 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb294105270 0x7fb29419a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.296+0000 7fb293e00640 1 --2- 172.21.15.28:0/4001489586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb294104080 0x7fb29419a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.296+0000 7fb293e00640 1 -- 172.21.15.28:0/4001489586 >> 172.21.15.28:0/4001489586 conn(0x7fb2940ff610 msgr2=0x7fb294108510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.296+0000 7fb293e00640 1 -- 172.21.15.28:0/4001489586 shutdown_connections 2024-04-24T08:07:30.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:30.297+0000 7fb293e00640 1 -- 172.21.15.28:0/4001489586 wait complete. 2024-04-24T08:07:31.110 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:30 smithi194 ceph-mon[32836]: Deploying daemon mds.h on smithi117 2024-04-24T08:07:31.111 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:30 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/4001489586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:31.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:30 smithi117 ceph-mon[32438]: Deploying daemon mds.h on smithi117 2024-04-24T08:07:31.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:30 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/4001489586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:31.128 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:07:31 smithi117 systemd[1]: Starting Ceph mds.h for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:07:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:30 smithi028 ceph-mon[27786]: Deploying daemon mds.h on smithi117 2024-04-24T08:07:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:30 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/4001489586' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:31.496 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:07:31 smithi117 podman[62552]: 2024-04-24 08:07:31.141286829 +0000 UTC m=+0.017711139 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:07:31.497 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:07:31 smithi117 podman[62552]: 2024-04-24 08:07:31.260924556 +0000 UTC m=+0.137348888 container create 070fa8e2068bf39f9c159592fe0fa22919620cee3e670413a2513620284d1d8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-04-24T08:07:31.497 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:07:31 smithi117 podman[62552]: 2024-04-24 08:07:31.49693304 +0000 UTC m=+0.373357330 container init 070fa8e2068bf39f9c159592fe0fa22919620cee3e670413a2513620284d1d8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-h, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:07:31.855 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:07:31 smithi117 podman[62552]: 2024-04-24 08:07:31.500133693 +0000 UTC m=+0.376557976 container start 070fa8e2068bf39f9c159592fe0fa22919620cee3e670413a2513620284d1d8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, ceph=True) 2024-04-24T08:07:31.855 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:07:31 smithi117 ceph-mds[62571]: -- 172.21.15.117:0/888758575 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x558b646cda40 con 0x558b646a0800 2024-04-24T08:07:31.855 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:07:31 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-h[62567]: starting mds.h at 2024-04-24T08:07:31.855 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:07:31 smithi117 bash[62552]: 070fa8e2068bf39f9c159592fe0fa22919620cee3e670413a2513620284d1d8d 2024-04-24T08:07:31.855 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:07:31 smithi117 systemd[1]: Started Ceph mds.h for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T08:07:32.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:31 smithi117 ceph-mon[32438]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:32.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:31 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:32.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:31 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:32.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:31 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:32.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:31 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:32.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:31 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:32.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:31 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:32.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:31 smithi194 ceph-mon[32836]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:32.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:31 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:32.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:31 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:32.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:31 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:32.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:31 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:32.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:31 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:32.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:31 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:32.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:31 smithi028 ceph-mon[27786]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:32.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:31 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:32.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:31 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:32.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:31 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:32.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:31 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:32.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:31 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:32.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:31 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:32.318 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:32.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.541+0000 7ffa5a600640 1 -- 172.21.15.28:0/763146071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa5410a030 msgr2=0x7ffa541108c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:32.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.541+0000 7ffa5a600640 1 --2- 172.21.15.28:0/763146071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa5410a030 0x7ffa541108c0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c009860 tx=0x7ffa4c02a340 comp rx=0 tx=0).stop 2024-04-24T08:07:32.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.541+0000 7ffa5a600640 1 -- 172.21.15.28:0/763146071 shutdown_connections 2024-04-24T08:07:32.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.541+0000 7ffa5a600640 1 --2- 172.21.15.28:0/763146071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa5410a030 0x7ffa541108c0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:32.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.541+0000 7ffa5a600640 1 --2- 172.21.15.28:0/763146071 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa54109670 0x7ffa54109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:32.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.541+0000 7ffa5a600640 1 --2- 172.21.15.28:0/763146071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa54108480 0x7ffa541088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:32.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.541+0000 7ffa5a600640 1 -- 172.21.15.28:0/763146071 >> 172.21.15.28:0/763146071 conn(0x7ffa54077440 msgr2=0x7ffa54079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:32.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.541+0000 7ffa5a600640 1 -- 172.21.15.28:0/763146071 shutdown_connections 2024-04-24T08:07:32.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.541+0000 7ffa5a600640 1 -- 172.21.15.28:0/763146071 wait complete. 2024-04-24T08:07:32.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.542+0000 7ffa5a600640 1 Processor -- start 2024-04-24T08:07:32.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.543+0000 7ffa5a600640 1 -- start start 2024-04-24T08:07:32.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.543+0000 7ffa5a600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa54108480 0x7ffa5419e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:32.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.543+0000 7ffa5a600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa54109670 0x7ffa5419ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:32.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.543+0000 7ffa5a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5410a030 0x7ffa541a5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:32.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.543+0000 7ffa5a600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa541140e0 con 0x7ffa5410a030 2024-04-24T08:07:32.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.543+0000 7ffa5a600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ffa54113f60 con 0x7ffa54108480 2024-04-24T08:07:32.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.543+0000 7ffa5a600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ffa54114260 con 0x7ffa54109670 2024-04-24T08:07:32.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.544+0000 7ffa53e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa54109670 0x7ffa5419ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:32.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.544+0000 7ffa59200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa54108480 0x7ffa5419e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:32.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.544+0000 7ffa59c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5410a030 0x7ffa541a5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:32.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.544+0000 7ffa53e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa54109670 0x7ffa5419ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:34454/0 (socket says 172.21.15.28:34454) 2024-04-24T08:07:32.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.544+0000 7ffa59200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa54108480 0x7ffa5419e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:60166/0 (socket says 172.21.15.28:60166) 2024-04-24T08:07:32.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.544+0000 7ffa53e00640 1 -- 172.21.15.28:0/1273485160 learned_addr learned my addr 172.21.15.28:0/1273485160 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:32.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.544+0000 7ffa59200640 1 -- 172.21.15.28:0/1273485160 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa54109670 msgr2=0x7ffa5419ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:32.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.544+0000 7ffa59200640 1 --2- 172.21.15.28:0/1273485160 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa54109670 0x7ffa5419ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:32.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.544+0000 7ffa59200640 1 -- 172.21.15.28:0/1273485160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5410a030 msgr2=0x7ffa541a5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:32.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.544+0000 7ffa59200640 1 --2- 172.21.15.28:0/1273485160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5410a030 0x7ffa541a5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:32.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.544+0000 7ffa59200640 1 -- 172.21.15.28:0/1273485160 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa541a6340 con 0x7ffa54108480 2024-04-24T08:07:32.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.544+0000 7ffa59c00640 1 --2- 172.21.15.28:0/1273485160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5410a030 0x7ffa541a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:32.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.544+0000 7ffa53e00640 1 --2- 172.21.15.28:0/1273485160 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa54109670 0x7ffa5419ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:32.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.544+0000 7ffa59200640 1 --2- 172.21.15.28:0/1273485160 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa54108480 0x7ffa5419e720 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffa44005fd0 tx=0x7ffa4400ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:32.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.545+0000 7ffa51600640 1 -- 172.21.15.28:0/1273485160 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa44003970 con 0x7ffa54108480 2024-04-24T08:07:32.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.545+0000 7ffa51600640 1 -- 172.21.15.28:0/1273485160 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa44004030 con 0x7ffa54108480 2024-04-24T08:07:32.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.545+0000 7ffa51600640 1 -- 172.21.15.28:0/1273485160 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa440107f0 con 0x7ffa54108480 2024-04-24T08:07:32.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.545+0000 7ffa5a600640 1 -- 172.21.15.28:0/1273485160 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa541a6630 con 0x7ffa54108480 2024-04-24T08:07:32.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.545+0000 7ffa5a600640 1 -- 172.21.15.28:0/1273485160 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa541a6bd0 con 0x7ffa54108480 2024-04-24T08:07:32.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.546+0000 7ffa5a600640 1 -- 172.21.15.28:0/1273485160 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa5410f7a0 con 0x7ffa54108480 2024-04-24T08:07:32.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.548+0000 7ffa51600640 1 -- 172.21.15.28:0/1273485160 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ffa440041e0 con 0x7ffa54108480 2024-04-24T08:07:32.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.550+0000 7ffa51600640 1 --2- 172.21.15.28:0/1273485160 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffa380a3c30 0x7ffa380a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:32.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.550+0000 7ffa51600640 1 -- 172.21.15.28:0/1273485160 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7ffa440d0790 con 0x7ffa54108480 2024-04-24T08:07:32.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.550+0000 7ffa53e00640 1 --2- 172.21.15.28:0/1273485160 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffa380a3c30 0x7ffa380a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:32.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.551+0000 7ffa53e00640 1 --2- 172.21.15.28:0/1273485160 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffa380a3c30 0x7ffa380a60f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffa5419fc30 tx=0x7ffa40009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:32.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.554+0000 7ffa51600640 1 -- 172.21.15.28:0/1273485160 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa4409d570 con 0x7ffa54108480 2024-04-24T08:07:32.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.802+0000 7ffa5a600640 1 -- 172.21.15.28:0/1273485160 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ffa5402d620 con 0x7ffa54108480 2024-04-24T08:07:32.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.803+0000 7ffa51600640 1 -- 172.21.15.28:0/1273485160 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7177 (secure 0 0 0) 0x7ffa4409ce70 con 0x7ffa54108480 2024-04-24T08:07:32.805 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:32.805 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-04-24T08:07:32.805 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 9 2024-04-24T08:07:32.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.806+0000 7ffa5a600640 1 -- 172.21.15.28:0/1273485160 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffa380a3c30 msgr2=0x7ffa380a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:32.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.806+0000 7ffa5a600640 1 --2- 172.21.15.28:0/1273485160 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffa380a3c30 0x7ffa380a60f0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7ffa5419fc30 tx=0x7ffa40009260 comp rx=0 tx=0).stop 2024-04-24T08:07:32.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.806+0000 7ffa5a600640 1 -- 172.21.15.28:0/1273485160 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa54108480 msgr2=0x7ffa5419e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:32.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.806+0000 7ffa5a600640 1 --2- 172.21.15.28:0/1273485160 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa54108480 0x7ffa5419e720 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7ffa44005fd0 tx=0x7ffa4400ef90 comp rx=0 tx=0).stop 2024-04-24T08:07:32.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.806+0000 7ffa5a600640 1 -- 172.21.15.28:0/1273485160 shutdown_connections 2024-04-24T08:07:32.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.806+0000 7ffa5a600640 1 --2- 172.21.15.28:0/1273485160 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffa380a3c30 0x7ffa380a60f0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:32.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.806+0000 7ffa5a600640 1 --2- 172.21.15.28:0/1273485160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5410a030 0x7ffa541a5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:32.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.806+0000 7ffa5a600640 1 --2- 172.21.15.28:0/1273485160 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa54109670 0x7ffa5419ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:32.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.806+0000 7ffa5a600640 1 --2- 172.21.15.28:0/1273485160 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa54108480 0x7ffa5419e720 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:32.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.806+0000 7ffa5a600640 1 -- 172.21.15.28:0/1273485160 >> 172.21.15.28:0/1273485160 conn(0x7ffa54077440 msgr2=0x7ffa5410c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:32.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.806+0000 7ffa5a600640 1 -- 172.21.15.28:0/1273485160 shutdown_connections 2024-04-24T08:07:32.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:32.807+0000 7ffa5a600640 1 -- 172.21.15.28:0/1273485160 wait complete. 2024-04-24T08:07:32.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:32 smithi194 ceph-mon[32836]: Deploying daemon mds.i on smithi194 2024-04-24T08:07:32.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:32 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.117:6836/862115546,v1:172.21.15.117:6837/862115546] up:boot 2024-04-24T08:07:32.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:32 smithi194 ceph-mon[32836]: fsmap 8 up:standby 2024-04-24T08:07:32.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:32 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:07:32.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:32 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1273485160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:33.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:32 smithi117 ceph-mon[32438]: Deploying daemon mds.i on smithi194 2024-04-24T08:07:33.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:32 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.117:6836/862115546,v1:172.21.15.117:6837/862115546] up:boot 2024-04-24T08:07:33.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:32 smithi117 ceph-mon[32438]: fsmap 8 up:standby 2024-04-24T08:07:33.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:32 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:07:33.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:32 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1273485160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:33.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:32 smithi028 ceph-mon[27786]: Deploying daemon mds.i on smithi194 2024-04-24T08:07:33.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:32 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.117:6836/862115546,v1:172.21.15.117:6837/862115546] up:boot 2024-04-24T08:07:33.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:32 smithi028 ceph-mon[27786]: fsmap 8 up:standby 2024-04-24T08:07:33.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:32 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:07:33.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:32 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1273485160' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:34.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:33 smithi117 ceph-mon[32438]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:34.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:33 smithi194 ceph-mon[32836]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:34.191 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:07:34 smithi194 systemd[1]: Starting Ceph mds.i for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:07:34.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:33 smithi028 ceph-mon[27786]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:34.496 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:07:34 smithi194 podman[64145]: 2024-04-24 08:07:34.138358819 +0000 UTC m=+0.018042335 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:07:34.496 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:07:34 smithi194 podman[64145]: 2024-04-24 08:07:34.251891907 +0000 UTC m=+0.131575412 container create a5cb61bed1c11731badde0d125941a2f273e0b94b8b7daaacea463f87c0e6038 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:07:34.496 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:07:34 smithi194 podman[64145]: 2024-04-24 08:07:34.496227882 +0000 UTC m=+0.375911388 container init a5cb61bed1c11731badde0d125941a2f273e0b94b8b7daaacea463f87c0e6038 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-i, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424) 2024-04-24T08:07:34.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:34.940 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:07:34 smithi194 podman[64145]: 2024-04-24 08:07:34.498698567 +0000 UTC m=+0.378382076 container start a5cb61bed1c11731badde0d125941a2f273e0b94b8b7daaacea463f87c0e6038 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-i, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-04-24T08:07:34.940 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:07:34 smithi194 ceph-mds[64164]: -- 172.21.15.194:0/1879451972 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560310ad7a40 con 0x560310aaa800 2024-04-24T08:07:34.941 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:07:34 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-i[64160]: starting mds.i at 2024-04-24T08:07:34.941 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:07:34 smithi194 bash[64145]: a5cb61bed1c11731badde0d125941a2f273e0b94b8b7daaacea463f87c0e6038 2024-04-24T08:07:34.941 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:07:34 smithi194 systemd[1]: Started Ceph mds.i for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T08:07:35.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.017+0000 7f354aa00640 1 -- 172.21.15.28:0/769020192 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3544103c00 msgr2=0x7f354410a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.017+0000 7f354aa00640 1 --2- 172.21.15.28:0/769020192 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3544103c00 0x7f354410a490 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f353c009ca0 tx=0x7f353c02a8c0 comp rx=0 tx=0).stop 2024-04-24T08:07:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.017+0000 7f354aa00640 1 -- 172.21.15.28:0/769020192 shutdown_connections 2024-04-24T08:07:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.017+0000 7f354aa00640 1 --2- 172.21.15.28:0/769020192 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3544103240 0x7f35441036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.017+0000 7f354aa00640 1 --2- 172.21.15.28:0/769020192 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3544103c00 0x7f354410a490 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.017+0000 7f354aa00640 1 --2- 172.21.15.28:0/769020192 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3544102050 0x7f3544102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.017+0000 7f354aa00640 1 -- 172.21.15.28:0/769020192 >> 172.21.15.28:0/769020192 conn(0x7f35440fd600 msgr2=0x7f35440ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.017+0000 7f354aa00640 1 -- 172.21.15.28:0/769020192 shutdown_connections 2024-04-24T08:07:35.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.017+0000 7f354aa00640 1 -- 172.21.15.28:0/769020192 wait complete. 2024-04-24T08:07:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.018+0000 7f354aa00640 1 Processor -- start 2024-04-24T08:07:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.018+0000 7f354aa00640 1 -- start start 2024-04-24T08:07:35.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.019+0000 7f354aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3544102050 0x7f3544075db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.019+0000 7f354aa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3544103240 0x7f35440743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.019+0000 7f354aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3544103c00 0x7f3544074930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.019+0000 7f354aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f354410dd30 con 0x7f3544103c00 2024-04-24T08:07:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.019+0000 7f354aa00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f354410dbb0 con 0x7f3544103240 2024-04-24T08:07:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.019+0000 7f354aa00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f354410deb0 con 0x7f3544102050 2024-04-24T08:07:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.019+0000 7f3543e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3544102050 0x7f3544075db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.019+0000 7f3543e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3544102050 0x7f3544075db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:34468/0 (socket says 172.21.15.28:34468) 2024-04-24T08:07:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.019+0000 7f3543e00640 1 -- 172.21.15.28:0/2994865861 learned_addr learned my addr 172.21.15.28:0/2994865861 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.019+0000 7f3548a00640 1 --2- 172.21.15.28:0/2994865861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3544103c00 0x7f3544074930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.019+0000 7f3543400640 1 --2- 172.21.15.28:0/2994865861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3544103240 0x7f35440743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.019+0000 7f3543e00640 1 -- 172.21.15.28:0/2994865861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3544103240 msgr2=0x7f35440743f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.019+0000 7f3543e00640 1 --2- 172.21.15.28:0/2994865861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3544103240 0x7f35440743f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:35.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.019+0000 7f3543e00640 1 -- 172.21.15.28:0/2994865861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3544103c00 msgr2=0x7f3544074930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.019+0000 7f3543e00640 1 --2- 172.21.15.28:0/2994865861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3544103c00 0x7f3544074930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.019+0000 7f3543e00640 1 -- 172.21.15.28:0/2994865861 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35440751c0 con 0x7f3544102050 2024-04-24T08:07:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.019+0000 7f3548a00640 1 --2- 172.21.15.28:0/2994865861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3544103c00 0x7f3544074930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:07:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.020+0000 7f3543e00640 1 --2- 172.21.15.28:0/2994865861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3544102050 0x7f3544075db0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f353400e9c0 tx=0x7f353400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.020+0000 7f3540c00640 1 -- 172.21.15.28:0/2994865861 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f353400cd80 con 0x7f3544102050 2024-04-24T08:07:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.020+0000 7f3540c00640 1 -- 172.21.15.28:0/2994865861 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35340035f0 con 0x7f3544102050 2024-04-24T08:07:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.020+0000 7f354aa00640 1 -- 172.21.15.28:0/2994865861 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35441a23e0 con 0x7f3544102050 2024-04-24T08:07:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.020+0000 7f3540c00640 1 -- 172.21.15.28:0/2994865861 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3534010690 con 0x7f3544102050 2024-04-24T08:07:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.020+0000 7f354aa00640 1 -- 172.21.15.28:0/2994865861 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35441a28e0 con 0x7f3544102050 2024-04-24T08:07:35.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.022+0000 7f354aa00640 1 -- 172.21.15.28:0/2994865861 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3544109310 con 0x7f3544102050 2024-04-24T08:07:35.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.023+0000 7f3540c00640 1 -- 172.21.15.28:0/2994865861 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f35340037a0 con 0x7f3544102050 2024-04-24T08:07:35.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.025+0000 7f3540c00640 1 --2- 172.21.15.28:0/2994865861 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f35280a3bb0 0x7f35280a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:35.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.025+0000 7f3540c00640 1 -- 172.21.15.28:0/2994865861 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f35340d0470 con 0x7f3544102050 2024-04-24T08:07:35.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.025+0000 7f3543400640 1 --2- 172.21.15.28:0/2994865861 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f35280a3bb0 0x7f35280a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:35.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.026+0000 7f3543400640 1 --2- 172.21.15.28:0/2994865861 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f35280a3bb0 0x7f35280a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f35300095f0 tx=0x7f3530005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:35.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.029+0000 7f3540c00640 1 -- 172.21.15.28:0/2994865861 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f353409d250 con 0x7f3544102050 2024-04-24T08:07:35.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.302+0000 7f354aa00640 1 -- 172.21.15.28:0/2994865861 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f354402d620 con 0x7f3544102050 2024-04-24T08:07:35.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.303+0000 7f3540c00640 1 -- 172.21.15.28:0/2994865861 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8007 (secure 0 0 0) 0x7f353409cb50 con 0x7f3544102050 2024-04-24T08:07:35.304 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:35.305 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-24T08:07:35.305 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 10 2024-04-24T08:07:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.305+0000 7f354aa00640 1 -- 172.21.15.28:0/2994865861 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f35280a3bb0 msgr2=0x7f35280a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.305+0000 7f354aa00640 1 --2- 172.21.15.28:0/2994865861 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f35280a3bb0 0x7f35280a6070 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f35300095f0 tx=0x7f3530005f90 comp rx=0 tx=0).stop 2024-04-24T08:07:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.305+0000 7f354aa00640 1 -- 172.21.15.28:0/2994865861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3544102050 msgr2=0x7f3544075db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.305+0000 7f354aa00640 1 --2- 172.21.15.28:0/2994865861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3544102050 0x7f3544075db0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f353400e9c0 tx=0x7f353400ee80 comp rx=0 tx=0).stop 2024-04-24T08:07:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.306+0000 7f354aa00640 1 -- 172.21.15.28:0/2994865861 shutdown_connections 2024-04-24T08:07:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.306+0000 7f354aa00640 1 --2- 172.21.15.28:0/2994865861 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f35280a3bb0 0x7f35280a6070 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.306+0000 7f354aa00640 1 --2- 172.21.15.28:0/2994865861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3544103c00 0x7f3544074930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.306+0000 7f354aa00640 1 --2- 172.21.15.28:0/2994865861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3544103240 0x7f35440743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.306+0000 7f354aa00640 1 --2- 172.21.15.28:0/2994865861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3544102050 0x7f3544075db0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.306+0000 7f354aa00640 1 -- 172.21.15.28:0/2994865861 >> 172.21.15.28:0/2994865861 conn(0x7f35440fd600 msgr2=0x7f35441064e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:35.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.306+0000 7f354aa00640 1 -- 172.21.15.28:0/2994865861 shutdown_connections 2024-04-24T08:07:35.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:35.306+0000 7f354aa00640 1 -- 172.21.15.28:0/2994865861 wait complete. 2024-04-24T08:07:35.934 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:35 smithi028 ceph-mon[27786]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:35.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:35.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:35.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:35.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:35.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:35.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:35.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:35 smithi028 ceph-mon[27786]: Deploying daemon mds.g on smithi028 2024-04-24T08:07:35.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:35 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] up:boot 2024-04-24T08:07:35.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:35 smithi028 ceph-mon[27786]: fsmap 9 up:standby 2024-04-24T08:07:35.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:35 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:07:35.935 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:35 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2994865861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:35.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:35 smithi194 ceph-mon[32836]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:35.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:35.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:35.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:35.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:35.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:35.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:35.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:35 smithi194 ceph-mon[32836]: Deploying daemon mds.g on smithi028 2024-04-24T08:07:35.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:35 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] up:boot 2024-04-24T08:07:35.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:35 smithi194 ceph-mon[32836]: fsmap 9 up:standby 2024-04-24T08:07:35.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:35 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:07:35.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:35 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2994865861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:36.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:35 smithi117 ceph-mon[32438]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:36.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:36.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:36.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:36.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:36.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:36.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:36.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:35 smithi117 ceph-mon[32438]: Deploying daemon mds.g on smithi028 2024-04-24T08:07:36.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:35 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] up:boot 2024-04-24T08:07:36.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:35 smithi117 ceph-mon[32438]: fsmap 9 up:standby 2024-04-24T08:07:36.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:35 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:07:36.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:35 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2994865861' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:37.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:37.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.617+0000 7f189d800640 1 -- 172.21.15.28:0/3991632607 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f189810bdf0 msgr2=0x7f1898114c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:37.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.617+0000 7f189d800640 1 --2- 172.21.15.28:0/3991632607 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f189810bdf0 0x7f1898114c90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f1888009ca0 tx=0x7f188802a8c0 comp rx=0 tx=0).stop 2024-04-24T08:07:37.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.618+0000 7f189d800640 1 -- 172.21.15.28:0/3991632607 shutdown_connections 2024-04-24T08:07:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.618+0000 7f189d800640 1 --2- 172.21.15.28:0/3991632607 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f189810b970 0x7f1898112740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.618+0000 7f189d800640 1 --2- 172.21.15.28:0/3991632607 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f189810bdf0 0x7f1898114c90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.618+0000 7f189d800640 1 --2- 172.21.15.28:0/3991632607 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1898109000 0x7f189810b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.619+0000 7f189d800640 1 -- 172.21.15.28:0/3991632607 >> 172.21.15.28:0/3991632607 conn(0x7f189806eea0 msgr2=0x7f18980712e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:37.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.619+0000 7f189d800640 1 -- 172.21.15.28:0/3991632607 shutdown_connections 2024-04-24T08:07:37.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.619+0000 7f189d800640 1 -- 172.21.15.28:0/3991632607 wait complete. 2024-04-24T08:07:37.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.620+0000 7f189d800640 1 Processor -- start 2024-04-24T08:07:37.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.621+0000 7f189d800640 1 -- start start 2024-04-24T08:07:37.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.622+0000 7f189d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1898109000 0x7f1898119c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:37.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.622+0000 7f189d800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f189810b970 0x7f189811a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:37.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.622+0000 7f189d800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f189810bdf0 0x7f189811f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:37.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.623+0000 7f189d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f189810d640 con 0x7f1898109000 2024-04-24T08:07:37.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.623+0000 7f189d800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f189810d4c0 con 0x7f189810bdf0 2024-04-24T08:07:37.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.623+0000 7f189d800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f189810d7c0 con 0x7f189810b970 2024-04-24T08:07:37.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.623+0000 7f1896000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f189810b970 0x7f189811a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:37.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.623+0000 7f1896000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f189810b970 0x7f189811a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:34494/0 (socket says 172.21.15.28:34494) 2024-04-24T08:07:37.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.623+0000 7f1896000640 1 -- 172.21.15.28:0/423963204 learned_addr learned my addr 172.21.15.28:0/423963204 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:37.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.623+0000 7f1897400640 1 --2- 172.21.15.28:0/423963204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f189810bdf0 0x7f189811f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:37.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.623+0000 7f1896000640 1 -- 172.21.15.28:0/423963204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f189810bdf0 msgr2=0x7f189811f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:37.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.623+0000 7f1896000640 1 --2- 172.21.15.28:0/423963204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f189810bdf0 0x7f189811f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:37.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.623+0000 7f1896000640 1 -- 172.21.15.28:0/423963204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1898109000 msgr2=0x7f1898119c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:37.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.623+0000 7f1896000640 1 --2- 172.21.15.28:0/423963204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1898109000 0x7f1898119c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:37.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.623+0000 7f1896000640 1 -- 172.21.15.28:0/423963204 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f189811f7b0 con 0x7f189810b970 2024-04-24T08:07:37.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.624+0000 7f1896000640 1 --2- 172.21.15.28:0/423963204 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f189810b970 0x7f189811a1a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f189000e9c0 tx=0x7f189000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:37.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.624+0000 7f188f400640 1 -- 172.21.15.28:0/423963204 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189000cd80 con 0x7f189810b970 2024-04-24T08:07:37.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.624+0000 7f188f400640 1 -- 172.21.15.28:0/423963204 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18900035f0 con 0x7f189810b970 2024-04-24T08:07:37.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.624+0000 7f188f400640 1 -- 172.21.15.28:0/423963204 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1890010690 con 0x7f189810b970 2024-04-24T08:07:37.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.624+0000 7f189d800640 1 -- 172.21.15.28:0/423963204 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18981b78d0 con 0x7f189810b970 2024-04-24T08:07:37.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.624+0000 7f189d800640 1 -- 172.21.15.28:0/423963204 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18981b7ce0 con 0x7f189810b970 2024-04-24T08:07:37.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.626+0000 7f189d800640 1 -- 172.21.15.28:0/423963204 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1898068e20 con 0x7f189810b970 2024-04-24T08:07:37.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.628+0000 7f188f400640 1 -- 172.21.15.28:0/423963204 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f18900037a0 con 0x7f189810b970 2024-04-24T08:07:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.630+0000 7f188f400640 1 --2- 172.21.15.28:0/423963204 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f187c0a3c80 0x7f187c0a6140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.630+0000 7f188f400640 1 -- 172.21.15.28:0/423963204 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f18900d0460 con 0x7f189810b970 2024-04-24T08:07:37.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.630+0000 7f1896a00640 1 --2- 172.21.15.28:0/423963204 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f187c0a3c80 0x7f187c0a6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:37.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.631+0000 7f1896a00640 1 --2- 172.21.15.28:0/423963204 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f187c0a3c80 0x7f187c0a6140 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f18840058c0 tx=0x7f1884005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:37.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.634+0000 7f188f400640 1 -- 172.21.15.28:0/423963204 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f189009d240 con 0x7f189810b970 2024-04-24T08:07:37.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:37 smithi028 ceph-mon[27786]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:37.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.886+0000 7f189d800640 1 -- 172.21.15.28:0/423963204 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1898111730 con 0x7f189810b970 2024-04-24T08:07:37.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.887+0000 7f188f400640 1 -- 172.21.15.28:0/423963204 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8007 (secure 0 0 0) 0x7f189009cb40 con 0x7f189810b970 2024-04-24T08:07:37.888 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:37.889 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-04-24T08:07:37.889 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 10 2024-04-24T08:07:37.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.890+0000 7f189d800640 1 -- 172.21.15.28:0/423963204 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f187c0a3c80 msgr2=0x7f187c0a6140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:37.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.890+0000 7f189d800640 1 --2- 172.21.15.28:0/423963204 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f187c0a3c80 0x7f187c0a6140 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f18840058c0 tx=0x7f1884005850 comp rx=0 tx=0).stop 2024-04-24T08:07:37.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.890+0000 7f189d800640 1 -- 172.21.15.28:0/423963204 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f189810b970 msgr2=0x7f189811a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:37.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.890+0000 7f189d800640 1 --2- 172.21.15.28:0/423963204 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f189810b970 0x7f189811a1a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f189000e9c0 tx=0x7f189000ee80 comp rx=0 tx=0).stop 2024-04-24T08:07:37.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.890+0000 7f189d800640 1 -- 172.21.15.28:0/423963204 shutdown_connections 2024-04-24T08:07:37.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.890+0000 7f189d800640 1 --2- 172.21.15.28:0/423963204 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f187c0a3c80 0x7f187c0a6140 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:37.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.890+0000 7f189d800640 1 --2- 172.21.15.28:0/423963204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f189810bdf0 0x7f189811f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:37.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.890+0000 7f189d800640 1 --2- 172.21.15.28:0/423963204 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f189810b970 0x7f189811a1a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:37.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.890+0000 7f189d800640 1 --2- 172.21.15.28:0/423963204 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1898109000 0x7f1898119c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:37.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.890+0000 7f189d800640 1 -- 172.21.15.28:0/423963204 >> 172.21.15.28:0/423963204 conn(0x7f189806eea0 msgr2=0x7f1898071030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:37.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.890+0000 7f189d800640 1 -- 172.21.15.28:0/423963204 shutdown_connections 2024-04-24T08:07:37.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:37.890+0000 7f189d800640 1 -- 172.21.15.28:0/423963204 wait complete. 2024-04-24T08:07:37.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:37 smithi194 ceph-mon[32836]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:38.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:37 smithi117 ceph-mon[32438]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:38.805 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 24 08:07:38 smithi028 systemd[1]: Started Ceph mds.g for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T08:07:38.805 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:38 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/423963204' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:38.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:38 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/423963204' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:39.029 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:38 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/423963204' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:39.913 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:39.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:39 smithi194 ceph-mon[32836]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:39.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:39 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] up:boot 2024-04-24T08:07:39.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:39 smithi194 ceph-mon[32836]: fsmap 10 up:standby 2024-04-24T08:07:39.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:39 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:07:39.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:39 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:39.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:39 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:39.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:39 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:39.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:39 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:39.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:39 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:39.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:39 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:39.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:39 smithi194 ceph-mon[32836]: Deploying daemon mds.a on smithi028 2024-04-24T08:07:39.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:39 smithi028 ceph-mon[27786]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:39.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:39 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] up:boot 2024-04-24T08:07:39.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:39 smithi028 ceph-mon[27786]: fsmap 10 up:standby 2024-04-24T08:07:39.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:39 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:07:39.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:39 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:39.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:39 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:39.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:39 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:39.985 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:39 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:39.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:39 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:39.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:39 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:39.986 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:39 smithi028 ceph-mon[27786]: Deploying daemon mds.a on smithi028 2024-04-24T08:07:40.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.117+0000 7f7f6b200640 1 -- 172.21.15.28:0/3630630190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f641029a0 msgr2=0x7f7f64104d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:40.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.117+0000 7f7f6b200640 1 --2- 172.21.15.28:0/3630630190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f641029a0 0x7f7f64104d90 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7f58009900 tx=0x7f7f5802a410 comp rx=0 tx=0).stop 2024-04-24T08:07:40.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.117+0000 7f7f6b200640 1 -- 172.21.15.28:0/3630630190 shutdown_connections 2024-04-24T08:07:40.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.117+0000 7f7f6b200640 1 --2- 172.21.15.28:0/3630630190 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f64107c00 0x7f7f64109ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:40.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.117+0000 7f7f6b200640 1 --2- 172.21.15.28:0/3630630190 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f641052d0 0x7f7f641076c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:40.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.117+0000 7f7f6b200640 1 --2- 172.21.15.28:0/3630630190 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f641029a0 0x7f7f64104d90 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:40.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.117+0000 7f7f6b200640 1 -- 172.21.15.28:0/3630630190 >> 172.21.15.28:0/3630630190 conn(0x7f7f640fc2d0 msgr2=0x7f7f640fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.117+0000 7f7f6b200640 1 -- 172.21.15.28:0/3630630190 shutdown_connections 2024-04-24T08:07:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.117+0000 7f7f6b200640 1 -- 172.21.15.28:0/3630630190 wait complete. 2024-04-24T08:07:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.118+0000 7f7f6b200640 1 Processor -- start 2024-04-24T08:07:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.118+0000 7f7f6b200640 1 -- start start 2024-04-24T08:07:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.119+0000 7f7f6b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f641029a0 0x7f7f6419a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.119+0000 7f7f6b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f641052d0 0x7f7f6419a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.119+0000 7f7f6b200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f64107c00 0x7f7f641a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.119+0000 7f7f6b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6410db10 con 0x7f7f641029a0 2024-04-24T08:07:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.119+0000 7f7f6b200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6410d990 con 0x7f7f64107c00 2024-04-24T08:07:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.119+0000 7f7f6b200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6410dc90 con 0x7f7f641052d0 2024-04-24T08:07:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.119+0000 7f7f69e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f641029a0 0x7f7f6419a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.119+0000 7f7f69e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f641029a0 0x7f7f6419a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44384/0 (socket says 172.21.15.28:44384) 2024-04-24T08:07:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.119+0000 7f7f69e00640 1 -- 172.21.15.28:0/326170287 learned_addr learned my addr 172.21.15.28:0/326170287 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.119+0000 7f7f6a800640 1 --2- 172.21.15.28:0/326170287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f64107c00 0x7f7f641a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.119+0000 7f7f69400640 1 --2- 172.21.15.28:0/326170287 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f641052d0 0x7f7f6419a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.120+0000 7f7f69e00640 1 -- 172.21.15.28:0/326170287 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f641052d0 msgr2=0x7f7f6419a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.120+0000 7f7f69e00640 1 --2- 172.21.15.28:0/326170287 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f641052d0 0x7f7f6419a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.120+0000 7f7f69e00640 1 -- 172.21.15.28:0/326170287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f64107c00 msgr2=0x7f7f641a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.120+0000 7f7f69e00640 1 --2- 172.21.15.28:0/326170287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f64107c00 0x7f7f641a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.120+0000 7f7f69e00640 1 -- 172.21.15.28:0/326170287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f641a1fe0 con 0x7f7f641029a0 2024-04-24T08:07:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.120+0000 7f7f6a800640 1 --2- 172.21.15.28:0/326170287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f64107c00 0x7f7f641a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.120+0000 7f7f69e00640 1 --2- 172.21.15.28:0/326170287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f641029a0 0x7f7f6419a3c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7f58009230 tx=0x7f7f580042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.120+0000 7f7f5ea00640 1 -- 172.21.15.28:0/326170287 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5803f030 con 0x7f7f641029a0 2024-04-24T08:07:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.120+0000 7f7f5ea00640 1 -- 172.21.15.28:0/326170287 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f58003ce0 con 0x7f7f641029a0 2024-04-24T08:07:40.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.120+0000 7f7f6b200640 1 -- 172.21.15.28:0/326170287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f641a2270 con 0x7f7f641029a0 2024-04-24T08:07:40.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.120+0000 7f7f5ea00640 1 -- 172.21.15.28:0/326170287 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f58035070 con 0x7f7f641029a0 2024-04-24T08:07:40.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.120+0000 7f7f6b200640 1 -- 172.21.15.28:0/326170287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f641a27b0 con 0x7f7f641029a0 2024-04-24T08:07:40.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.121+0000 7f7f6b200640 1 -- 172.21.15.28:0/326170287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f641096f0 con 0x7f7f641029a0 2024-04-24T08:07:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.124+0000 7f7f5ea00640 1 -- 172.21.15.28:0/326170287 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7f58042030 con 0x7f7f641029a0 2024-04-24T08:07:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.125+0000 7f7f5ea00640 1 --2- 172.21.15.28:0/326170287 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7f4c0a3c30 0x7f7f4c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.125+0000 7f7f5ea00640 1 -- 172.21.15.28:0/326170287 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f7f580f07e0 con 0x7f7f641029a0 2024-04-24T08:07:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.125+0000 7f7f69400640 1 --2- 172.21.15.28:0/326170287 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7f4c0a3c30 0x7f7f4c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:40.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.126+0000 7f7f69400640 1 --2- 172.21.15.28:0/326170287 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7f4c0a3c30 0x7f7f4c0a60f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7f540095f0 tx=0x7f7f54005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:40.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:39 smithi117 ceph-mon[32438]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:40.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:39 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] up:boot 2024-04-24T08:07:40.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:39 smithi117 ceph-mon[32438]: fsmap 10 up:standby 2024-04-24T08:07:40.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:39 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:07:40.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:39 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:40.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:39 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:40.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:39 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:40.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:39 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:40.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:39 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:40.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:39 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:40.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:39 smithi117 ceph-mon[32438]: Deploying daemon mds.a on smithi028 2024-04-24T08:07:40.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.128+0000 7f7f5ea00640 1 -- 172.21.15.28:0/326170287 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7f580bcf70 con 0x7f7f641029a0 2024-04-24T08:07:40.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.393+0000 7f7f6b200640 1 -- 172.21.15.28:0/326170287 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7f6402d620 con 0x7f7f641029a0 2024-04-24T08:07:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.394+0000 7f7f5ea00640 1 -- 172.21.15.28:0/326170287 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8833 (secure 0 0 0) 0x7f7f580c1270 con 0x7f7f641029a0 2024-04-24T08:07:40.396 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:40.396 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-24T08:07:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 11 2024-04-24T08:07:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.397+0000 7f7f6b200640 1 -- 172.21.15.28:0/326170287 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7f4c0a3c30 msgr2=0x7f7f4c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.397+0000 7f7f6b200640 1 --2- 172.21.15.28:0/326170287 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7f4c0a3c30 0x7f7f4c0a60f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7f540095f0 tx=0x7f7f54005f90 comp rx=0 tx=0).stop 2024-04-24T08:07:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.397+0000 7f7f6b200640 1 -- 172.21.15.28:0/326170287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f641029a0 msgr2=0x7f7f6419a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.397+0000 7f7f6b200640 1 --2- 172.21.15.28:0/326170287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f641029a0 0x7f7f6419a3c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f7f58009230 tx=0x7f7f580042f0 comp rx=0 tx=0).stop 2024-04-24T08:07:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.397+0000 7f7f6b200640 1 -- 172.21.15.28:0/326170287 shutdown_connections 2024-04-24T08:07:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.397+0000 7f7f6b200640 1 --2- 172.21.15.28:0/326170287 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7f4c0a3c30 0x7f7f4c0a60f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.397+0000 7f7f6b200640 1 --2- 172.21.15.28:0/326170287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f64107c00 0x7f7f641a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.397+0000 7f7f6b200640 1 --2- 172.21.15.28:0/326170287 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7f641052d0 0x7f7f6419a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.397+0000 7f7f6b200640 1 --2- 172.21.15.28:0/326170287 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7f641029a0 0x7f7f6419a3c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.397+0000 7f7f6b200640 1 -- 172.21.15.28:0/326170287 >> 172.21.15.28:0/326170287 conn(0x7f7f640fc2d0 msgr2=0x7f7f64102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.397+0000 7f7f6b200640 1 -- 172.21.15.28:0/326170287 shutdown_connections 2024-04-24T08:07:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:40.397+0000 7f7f6b200640 1 -- 172.21.15.28:0/326170287 wait complete. 2024-04-24T08:07:40.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:40 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/326170287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:41.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:40 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/326170287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:41.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:40 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/326170287' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:41.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:41 smithi194 ceph-mon[32836]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:41.943 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:41 smithi028 ceph-mon[27786]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:42.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:41 smithi117 ceph-mon[32438]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:42.289 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 24 08:07:41 smithi028 systemd[1]: Starting Ceph mds.a for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:07:42.419 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:42.636 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 24 08:07:42 smithi028 podman[67008]: 2024-04-24 08:07:42.249971382 +0000 UTC m=+0.033649320 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:07:42.636 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 24 08:07:42 smithi028 podman[67008]: 2024-04-24 08:07:42.365868107 +0000 UTC m=+0.149546123 container create e9d1c0734f937c94376e5d3cefdfe5c1c5653bc950dc0136be151867107ffbd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-24T08:07:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.643+0000 7f5d7d200640 1 -- 172.21.15.28:0/957367519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d78104080 msgr2=0x7f5d781044a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.643+0000 7f5d7d200640 1 --2- 172.21.15.28:0/957367519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d78104080 0x7f5d781044a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f5d68009900 tx=0x7f5d6802a410 comp rx=0 tx=0).stop 2024-04-24T08:07:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.643+0000 7f5d7d200640 1 -- 172.21.15.28:0/957367519 shutdown_connections 2024-04-24T08:07:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.643+0000 7f5d7d200640 1 --2- 172.21.15.28:0/957367519 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d78105c30 0x7f5d7810c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.643+0000 7f5d7d200640 1 --2- 172.21.15.28:0/957367519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d78105270 0x7f5d781056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.643+0000 7f5d7d200640 1 --2- 172.21.15.28:0/957367519 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d78104080 0x7f5d781044a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.643+0000 7f5d7d200640 1 -- 172.21.15.28:0/957367519 >> 172.21.15.28:0/957367519 conn(0x7f5d780ff610 msgr2=0x7f5d78101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:42.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.644+0000 7f5d7d200640 1 -- 172.21.15.28:0/957367519 shutdown_connections 2024-04-24T08:07:42.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.644+0000 7f5d7d200640 1 -- 172.21.15.28:0/957367519 wait complete. 2024-04-24T08:07:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.645+0000 7f5d7d200640 1 Processor -- start 2024-04-24T08:07:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.645+0000 7f5d7d200640 1 -- start start 2024-04-24T08:07:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.645+0000 7f5d7d200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d78104080 0x7f5d7819a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.645+0000 7f5d7d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d78105270 0x7f5d7819a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.645+0000 7f5d7d200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d78105c30 0x7f5d781a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.645+0000 7f5d7d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7810fcd0 con 0x7f5d78105270 2024-04-24T08:07:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.645+0000 7f5d7d200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7810fb50 con 0x7f5d78105c30 2024-04-24T08:07:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.645+0000 7f5d7d200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5d7810fe50 con 0x7f5d78104080 2024-04-24T08:07:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.645+0000 7f5d76a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d78105270 0x7f5d7819a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:42.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.645+0000 7f5d77e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d78105c30 0x7f5d781a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:42.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.645+0000 7f5d77400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d78104080 0x7f5d7819a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:42.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.645+0000 7f5d77e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d78105c30 0x7f5d781a1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:42828/0 (socket says 172.21.15.28:42828) 2024-04-24T08:07:42.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.645+0000 7f5d77e00640 1 -- 172.21.15.28:0/256458219 learned_addr learned my addr 172.21.15.28:0/256458219 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:42.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.645+0000 7f5d77400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d78104080 0x7f5d7819a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:34554/0 (socket says 172.21.15.28:34554) 2024-04-24T08:07:42.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.646+0000 7f5d77e00640 1 -- 172.21.15.28:0/256458219 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d78104080 msgr2=0x7f5d7819a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:42.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.646+0000 7f5d77e00640 1 --2- 172.21.15.28:0/256458219 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d78104080 0x7f5d7819a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:42.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.646+0000 7f5d77e00640 1 -- 172.21.15.28:0/256458219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d78105270 msgr2=0x7f5d7819a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:42.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.646+0000 7f5d77e00640 1 --2- 172.21.15.28:0/256458219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d78105270 0x7f5d7819a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:42.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.646+0000 7f5d77e00640 1 -- 172.21.15.28:0/256458219 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5d781a1fd0 con 0x7f5d78105c30 2024-04-24T08:07:42.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.646+0000 7f5d77e00640 1 --2- 172.21.15.28:0/256458219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d78105c30 0x7f5d781a1960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5d70009fd0 tx=0x7f5d7000b260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:42.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.646+0000 7f5d6fe00640 1 -- 172.21.15.28:0/256458219 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d70017070 con 0x7f5d78105c30 2024-04-24T08:07:42.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.646+0000 7f5d6fe00640 1 -- 172.21.15.28:0/256458219 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5d70003e50 con 0x7f5d78105c30 2024-04-24T08:07:42.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.647+0000 7f5d7d200640 1 -- 172.21.15.28:0/256458219 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5d781a22c0 con 0x7f5d78105c30 2024-04-24T08:07:42.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.647+0000 7f5d7d200640 1 -- 172.21.15.28:0/256458219 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5d781a27c0 con 0x7f5d78105c30 2024-04-24T08:07:42.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.647+0000 7f5d6fe00640 1 -- 172.21.15.28:0/256458219 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5d700106c0 con 0x7f5d78105c30 2024-04-24T08:07:42.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.648+0000 7f5d7d200640 1 -- 172.21.15.28:0/256458219 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5d7810b3f0 con 0x7f5d78105c30 2024-04-24T08:07:42.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.650+0000 7f5d6fe00640 1 -- 172.21.15.28:0/256458219 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5d70003970 con 0x7f5d78105c30 2024-04-24T08:07:42.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.652+0000 7f5d6fe00640 1 --2- 172.21.15.28:0/256458219 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5d5c0a3bb0 0x7f5d5c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:42.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.652+0000 7f5d6fe00640 1 -- 172.21.15.28:0/256458219 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f5d700cfce0 con 0x7f5d78105c30 2024-04-24T08:07:42.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.652+0000 7f5d77400640 1 --2- 172.21.15.28:0/256458219 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5d5c0a3bb0 0x7f5d5c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:42.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.653+0000 7f5d77400640 1 --2- 172.21.15.28:0/256458219 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5d5c0a3bb0 0x7f5d5c0a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5d68005ab0 tx=0x7f5d68000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:42.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.654+0000 7f5d6fe00640 1 -- 172.21.15.28:0/256458219 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5d7009c470 con 0x7f5d78105c30 2024-04-24T08:07:42.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.949+0000 7f5d7d200640 1 -- 172.21.15.28:0/256458219 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f5d78062c00 con 0x7f5d78105c30 2024-04-24T08:07:42.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.950+0000 7f5d6fe00640 1 -- 172.21.15.28:0/256458219 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8833 (secure 0 0 0) 0x7f5d700a0770 con 0x7f5d78105c30 2024-04-24T08:07:42.952 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:42.952 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-24T08:07:42.952 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 11 2024-04-24T08:07:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.953+0000 7f5d7d200640 1 -- 172.21.15.28:0/256458219 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5d5c0a3bb0 msgr2=0x7f5d5c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.953+0000 7f5d7d200640 1 --2- 172.21.15.28:0/256458219 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5d5c0a3bb0 0x7f5d5c0a6070 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5d68005ab0 tx=0x7f5d68000bc0 comp rx=0 tx=0).stop 2024-04-24T08:07:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.953+0000 7f5d7d200640 1 -- 172.21.15.28:0/256458219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d78105c30 msgr2=0x7f5d781a1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.953+0000 7f5d7d200640 1 --2- 172.21.15.28:0/256458219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d78105c30 0x7f5d781a1960 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f5d70009fd0 tx=0x7f5d7000b260 comp rx=0 tx=0).stop 2024-04-24T08:07:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.953+0000 7f5d7d200640 1 -- 172.21.15.28:0/256458219 shutdown_connections 2024-04-24T08:07:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.953+0000 7f5d7d200640 1 --2- 172.21.15.28:0/256458219 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5d5c0a3bb0 0x7f5d5c0a6070 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.953+0000 7f5d7d200640 1 --2- 172.21.15.28:0/256458219 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5d78105270 0x7f5d7819a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.953+0000 7f5d7d200640 1 --2- 172.21.15.28:0/256458219 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d78105c30 0x7f5d781a1960 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.953+0000 7f5d7d200640 1 --2- 172.21.15.28:0/256458219 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5d78104080 0x7f5d7819a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.953+0000 7f5d7d200640 1 -- 172.21.15.28:0/256458219 >> 172.21.15.28:0/256458219 conn(0x7f5d780ff610 msgr2=0x7f5d781085c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.953+0000 7f5d7d200640 1 -- 172.21.15.28:0/256458219 shutdown_connections 2024-04-24T08:07:42.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:42.953+0000 7f5d7d200640 1 -- 172.21.15.28:0/256458219 wait complete. 2024-04-24T08:07:42.971 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 24 08:07:42 smithi028 podman[67008]: 2024-04-24 08:07:42.635557885 +0000 UTC m=+0.419235823 container init e9d1c0734f937c94376e5d3cefdfe5c1c5653bc950dc0136be151867107ffbd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-a, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:07:42.971 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 24 08:07:42 smithi028 podman[67008]: 2024-04-24 08:07:42.643358985 +0000 UTC m=+0.427036924 container start e9d1c0734f937c94376e5d3cefdfe5c1c5653bc950dc0136be151867107ffbd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-24T08:07:42.972 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 24 08:07:42 smithi028 ceph-mds[67057]: -- 172.21.15.28:0/3631900614 <== mon.0 v2:172.21.15.28:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564f21437a40 con 0x564f2140b000 2024-04-24T08:07:42.972 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 24 08:07:42 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-a[67040]: starting mds.a at 2024-04-24T08:07:42.972 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 24 08:07:42 smithi028 bash[67008]: e9d1c0734f937c94376e5d3cefdfe5c1c5653bc950dc0136be151867107ffbd6 2024-04-24T08:07:42.972 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 24 08:07:42 smithi028 systemd[1]: Started Ceph mds.a for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T08:07:43.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:43 smithi117 ceph-mon[32438]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:43.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:43 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:43.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:43 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:43.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:43 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:43.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:43 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:43.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:43 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:43.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:43 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:43.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:43 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/256458219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:43.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:43 smithi194 ceph-mon[32836]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:43.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:43 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:43.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:43 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:43.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:43 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:43.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:43 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:43.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:43 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:43.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:43 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:43.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:43 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/256458219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:44.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:43 smithi028 ceph-mon[27786]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:44.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:43 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:44.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:43 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:44.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:43 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:44.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:43 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-04-24T08:07:44.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:43 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-04-24T08:07:44.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:43 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:44.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:43 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/256458219' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:44.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:44 smithi117 ceph-mon[32438]: Deploying daemon mds.b on smithi117 2024-04-24T08:07:44.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:44 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.28:6840/666043045,v1:172.21.15.28:6841/666043045] up:boot 2024-04-24T08:07:44.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:44 smithi117 ceph-mon[32438]: fsmap 11 up:standby 2024-04-24T08:07:44.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:44 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T08:07:44.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:44 smithi194 ceph-mon[32836]: Deploying daemon mds.b on smithi117 2024-04-24T08:07:44.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:44 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.28:6840/666043045,v1:172.21.15.28:6841/666043045] up:boot 2024-04-24T08:07:44.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:44 smithi194 ceph-mon[32836]: fsmap 11 up:standby 2024-04-24T08:07:44.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:44 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T08:07:44.974 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:45.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:44 smithi028 ceph-mon[27786]: Deploying daemon mds.b on smithi117 2024-04-24T08:07:45.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:44 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.28:6840/666043045,v1:172.21.15.28:6841/666043045] up:boot 2024-04-24T08:07:45.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:44 smithi028 ceph-mon[27786]: fsmap 11 up:standby 2024-04-24T08:07:45.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:44 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T08:07:45.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.197+0000 7f0b42000640 1 -- 172.21.15.28:0/3076981776 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b3c108480 msgr2=0x7f0b3c1088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:45.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.197+0000 7f0b42000640 1 --2- 172.21.15.28:0/3076981776 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b3c108480 0x7f0b3c1088a0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f0b30009900 tx=0x7f0b3002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:07:45.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.198+0000 7f0b42000640 1 -- 172.21.15.28:0/3076981776 shutdown_connections 2024-04-24T08:07:45.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.198+0000 7f0b42000640 1 --2- 172.21.15.28:0/3076981776 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b3c109670 0x7f0b3c109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:45.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.198+0000 7f0b42000640 1 --2- 172.21.15.28:0/3076981776 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b3c10a030 0x7f0b3c1108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:45.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.198+0000 7f0b42000640 1 --2- 172.21.15.28:0/3076981776 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b3c108480 0x7f0b3c1088a0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:45.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.198+0000 7f0b42000640 1 -- 172.21.15.28:0/3076981776 >> 172.21.15.28:0/3076981776 conn(0x7f0b3c077440 msgr2=0x7f0b3c079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:45.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.198+0000 7f0b42000640 1 -- 172.21.15.28:0/3076981776 shutdown_connections 2024-04-24T08:07:45.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.198+0000 7f0b42000640 1 -- 172.21.15.28:0/3076981776 wait complete. 2024-04-24T08:07:45.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.199+0000 7f0b42000640 1 Processor -- start 2024-04-24T08:07:45.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.199+0000 7f0b42000640 1 -- start start 2024-04-24T08:07:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.200+0000 7f0b42000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b3c108480 0x7f0b3c19e7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.200+0000 7f0b42000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b3c109670 0x7f0b3c19ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.200+0000 7f0b42000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b3c10a030 0x7f0b3c1a5d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.200+0000 7f0b42000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c114160 con 0x7f0b3c109670 2024-04-24T08:07:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.200+0000 7f0b42000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c113fe0 con 0x7f0b3c108480 2024-04-24T08:07:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.200+0000 7f0b42000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0b3c1142e0 con 0x7f0b3c10a030 2024-04-24T08:07:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.200+0000 7f0b3be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b3c109670 0x7f0b3c19ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.200+0000 7f0b40c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b3c108480 0x7f0b3c19e7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.200+0000 7f0b3be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b3c109670 0x7f0b3c19ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44454/0 (socket says 172.21.15.28:44454) 2024-04-24T08:07:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.200+0000 7f0b40c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b3c108480 0x7f0b3c19e7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:42856/0 (socket says 172.21.15.28:42856) 2024-04-24T08:07:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.200+0000 7f0b3be00640 1 -- 172.21.15.28:0/4195084187 learned_addr learned my addr 172.21.15.28:0/4195084187 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:45.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.200+0000 7f0b41600640 1 --2- 172.21.15.28:0/4195084187 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b3c10a030 0x7f0b3c1a5d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.200+0000 7f0b40c00640 1 -- 172.21.15.28:0/4195084187 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b3c10a030 msgr2=0x7f0b3c1a5d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.200+0000 7f0b40c00640 1 --2- 172.21.15.28:0/4195084187 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b3c10a030 0x7f0b3c1a5d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.200+0000 7f0b40c00640 1 -- 172.21.15.28:0/4195084187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b3c109670 msgr2=0x7f0b3c19ed00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.200+0000 7f0b40c00640 1 --2- 172.21.15.28:0/4195084187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b3c109670 0x7f0b3c19ed00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.200+0000 7f0b40c00640 1 -- 172.21.15.28:0/4195084187 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0b3c1a63e0 con 0x7f0b3c108480 2024-04-24T08:07:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.201+0000 7f0b41600640 1 --2- 172.21.15.28:0/4195084187 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b3c10a030 0x7f0b3c1a5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.201+0000 7f0b3be00640 1 --2- 172.21.15.28:0/4195084187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b3c109670 0x7f0b3c19ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.201+0000 7f0b40c00640 1 --2- 172.21.15.28:0/4195084187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b3c108480 0x7f0b3c19e7c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0b30009230 tx=0x7f0b30032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.201+0000 7f0b39600640 1 -- 172.21.15.28:0/4195084187 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b3003f030 con 0x7f0b3c108480 2024-04-24T08:07:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.201+0000 7f0b42000640 1 -- 172.21.15.28:0/4195084187 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0b3c1a6670 con 0x7f0b3c108480 2024-04-24T08:07:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.201+0000 7f0b39600640 1 -- 172.21.15.28:0/4195084187 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0b30003ce0 con 0x7f0b3c108480 2024-04-24T08:07:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.201+0000 7f0b39600640 1 -- 172.21.15.28:0/4195084187 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0b30035070 con 0x7f0b3c108480 2024-04-24T08:07:45.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.201+0000 7f0b42000640 1 -- 172.21.15.28:0/4195084187 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0b3c1a6b80 con 0x7f0b3c108480 2024-04-24T08:07:45.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.202+0000 7f0b42000640 1 -- 172.21.15.28:0/4195084187 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0b3c10f7a0 con 0x7f0b3c108480 2024-04-24T08:07:45.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.204+0000 7f0b39600640 1 -- 172.21.15.28:0/4195084187 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f0b30042030 con 0x7f0b3c108480 2024-04-24T08:07:45.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.206+0000 7f0b39600640 1 --2- 172.21.15.28:0/4195084187 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0b240a3c30 0x7f0b240a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:45.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.206+0000 7f0b39600640 1 -- 172.21.15.28:0/4195084187 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f0b300f0590 con 0x7f0b3c108480 2024-04-24T08:07:45.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.206+0000 7f0b3be00640 1 --2- 172.21.15.28:0/4195084187 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0b240a3c30 0x7f0b240a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:45.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.207+0000 7f0b3be00640 1 --2- 172.21.15.28:0/4195084187 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0b240a3c30 0x7f0b240a60f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c0095f0 tx=0x7f0b2c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:45.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.209+0000 7f0b39600640 1 -- 172.21.15.28:0/4195084187 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0b300bcd20 con 0x7f0b3c108480 2024-04-24T08:07:45.301 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:07:45 smithi117 systemd[1]: Starting Ceph mds.b for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:07:45.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.460+0000 7f0b42000640 1 -- 172.21.15.28:0/4195084187 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0b3c02d620 con 0x7f0b3c108480 2024-04-24T08:07:45.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.479+0000 7f0b39600640 1 -- 172.21.15.28:0/4195084187 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9656 (secure 0 0 0) 0x7f0b300c1020 con 0x7f0b3c108480 2024-04-24T08:07:45.482 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:45.482 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-24T08:07:45.482 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 12 2024-04-24T08:07:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.482+0000 7f0b42000640 1 -- 172.21.15.28:0/4195084187 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0b240a3c30 msgr2=0x7f0b240a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.482+0000 7f0b42000640 1 --2- 172.21.15.28:0/4195084187 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0b240a3c30 0x7f0b240a60f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c0095f0 tx=0x7f0b2c005f90 comp rx=0 tx=0).stop 2024-04-24T08:07:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.482+0000 7f0b42000640 1 -- 172.21.15.28:0/4195084187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b3c108480 msgr2=0x7f0b3c19e7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.482+0000 7f0b42000640 1 --2- 172.21.15.28:0/4195084187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b3c108480 0x7f0b3c19e7c0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f0b30009230 tx=0x7f0b30032ce0 comp rx=0 tx=0).stop 2024-04-24T08:07:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.482+0000 7f0b42000640 1 -- 172.21.15.28:0/4195084187 shutdown_connections 2024-04-24T08:07:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.482+0000 7f0b42000640 1 --2- 172.21.15.28:0/4195084187 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0b240a3c30 0x7f0b240a60f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.482+0000 7f0b42000640 1 --2- 172.21.15.28:0/4195084187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0b3c109670 0x7f0b3c19ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.482+0000 7f0b42000640 1 --2- 172.21.15.28:0/4195084187 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0b3c10a030 0x7f0b3c1a5d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.482+0000 7f0b42000640 1 --2- 172.21.15.28:0/4195084187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b3c108480 0x7f0b3c19e7c0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:45.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.482+0000 7f0b42000640 1 -- 172.21.15.28:0/4195084187 >> 172.21.15.28:0/4195084187 conn(0x7f0b3c077440 msgr2=0x7f0b3c10c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.483+0000 7f0b42000640 1 -- 172.21.15.28:0/4195084187 shutdown_connections 2024-04-24T08:07:45.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:45.483+0000 7f0b42000640 1 -- 172.21.15.28:0/4195084187 wait complete. 2024-04-24T08:07:45.584 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:07:45 smithi117 podman[62876]: 2024-04-24 08:07:45.203585446 +0000 UTC m=+0.018205012 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:07:45.585 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:07:45 smithi117 podman[62876]: 2024-04-24 08:07:45.331413196 +0000 UTC m=+0.146032779 container create cc15683a7e2d9f76fadbc1d73558683f5e35f3cfac110a7210dd2492359aef2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-24T08:07:45.585 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:07:45 smithi117 podman[62876]: 2024-04-24 08:07:45.584466725 +0000 UTC m=+0.399086266 container init cc15683a7e2d9f76fadbc1d73558683f5e35f3cfac110a7210dd2492359aef2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-b, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-24T08:07:45.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:45 smithi028 ceph-mon[27786]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:45.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:45 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/4195084187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:45.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:45 smithi117 ceph-mon[32438]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:45.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:45 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/4195084187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:45.878 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:07:45 smithi117 podman[62876]: 2024-04-24 08:07:45.587172633 +0000 UTC m=+0.401792181 container start cc15683a7e2d9f76fadbc1d73558683f5e35f3cfac110a7210dd2492359aef2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-b, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-04-24T08:07:45.878 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:07:45 smithi117 ceph-mds[62895]: -- 172.21.15.117:0/1091748114 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561bc812ba40 con 0x561bc80fec00 2024-04-24T08:07:45.878 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:07:45 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-b[62891]: starting mds.b at 2024-04-24T08:07:45.878 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:07:45 smithi117 bash[62876]: cc15683a7e2d9f76fadbc1d73558683f5e35f3cfac110a7210dd2492359aef2a 2024-04-24T08:07:45.878 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:07:45 smithi117 systemd[1]: Started Ceph mds.b for 473aaf58-0210-11ef-bc93-c7b262605968. 2024-04-24T08:07:46.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:45 smithi194 ceph-mon[32836]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:46.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:45 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/4195084187' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:46.965 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:46 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] up:boot 2024-04-24T08:07:46.965 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:46 smithi117 ceph-mon[32438]: fsmap 12 up:standby 2024-04-24T08:07:46.965 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:46 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:07:46.965 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:46 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:46.965 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:46 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:46.965 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:46 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:46.965 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:46 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:46.965 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:46 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:46.966 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:46 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:07:47.002 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:46 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] up:boot 2024-04-24T08:07:47.002 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:46 smithi194 ceph-mon[32836]: fsmap 12 up:standby 2024-04-24T08:07:47.002 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:46 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:07:47.002 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:46 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:47.002 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:46 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:47.002 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:46 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:47.002 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:46 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:47.003 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:46 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:47.003 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:46 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:07:47.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:46 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] up:boot 2024-04-24T08:07:47.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:46 smithi028 ceph-mon[27786]: fsmap 12 up:standby 2024-04-24T08:07:47.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:46 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:07:47.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:46 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:47.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:46 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:47.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:46 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:47.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:46 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:47.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:46 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:47.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:46 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:07:47.506 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:47.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.714+0000 7f0693800640 1 -- 172.21.15.28:0/3074911295 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f068c0769c0 msgr2=0x7f068c076de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:47.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.714+0000 7f0693800640 1 --2- 172.21.15.28:0/3074911295 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f068c0769c0 0x7f068c076de0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f067c009ca0 tx=0x7f067c02a8b0 comp rx=0 tx=0).stop 2024-04-24T08:07:47.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.715+0000 7f0693800640 1 -- 172.21.15.28:0/3074911295 shutdown_connections 2024-04-24T08:07:47.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.715+0000 7f0693800640 1 --2- 172.21.15.28:0/3074911295 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c0785a0 0x7f068c078e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:47.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.715+0000 7f0693800640 1 --2- 172.21.15.28:0/3074911295 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f068c077c00 0x7f068c078060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:47.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.715+0000 7f0693800640 1 --2- 172.21.15.28:0/3074911295 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f068c0769c0 0x7f068c076de0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:47.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.715+0000 7f0693800640 1 -- 172.21.15.28:0/3074911295 >> 172.21.15.28:0/3074911295 conn(0x7f068c0ff7f0 msgr2=0x7f068c101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:47.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.715+0000 7f0693800640 1 -- 172.21.15.28:0/3074911295 shutdown_connections 2024-04-24T08:07:47.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.715+0000 7f0693800640 1 -- 172.21.15.28:0/3074911295 wait complete. 2024-04-24T08:07:47.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.716+0000 7f0693800640 1 Processor -- start 2024-04-24T08:07:47.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.716+0000 7f0693800640 1 -- start start 2024-04-24T08:07:47.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.716+0000 7f0693800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f068c0769c0 0x7f068c19e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:47.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.716+0000 7f0693800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f068c077c00 0x7f068c19eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:47.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.716+0000 7f0693800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c0785a0 0x7f068c1a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:47.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.716+0000 7f0693800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f068c114350 con 0x7f068c0785a0 2024-04-24T08:07:47.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.716+0000 7f0693800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f068c1141d0 con 0x7f068c0769c0 2024-04-24T08:07:47.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.716+0000 7f0693800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f068c1144d0 con 0x7f068c077c00 2024-04-24T08:07:47.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.716+0000 7f0690e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f068c0769c0 0x7f068c19e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:47.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.716+0000 7f0690e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f068c0769c0 0x7f068c19e970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:42868/0 (socket says 172.21.15.28:42868) 2024-04-24T08:07:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.716+0000 7f0690e00640 1 -- 172.21.15.28:0/3147962748 learned_addr learned my addr 172.21.15.28:0/3147962748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.716+0000 7f0691800640 1 --2- 172.21.15.28:0/3147962748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c0785a0 0x7f068c1a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.716+0000 7f068be00640 1 --2- 172.21.15.28:0/3147962748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f068c077c00 0x7f068c19eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.717+0000 7f0690e00640 1 -- 172.21.15.28:0/3147962748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f068c077c00 msgr2=0x7f068c19eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.717+0000 7f0690e00640 1 --2- 172.21.15.28:0/3147962748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f068c077c00 0x7f068c19eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.717+0000 7f0690e00640 1 -- 172.21.15.28:0/3147962748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c0785a0 msgr2=0x7f068c1a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.717+0000 7f0690e00640 1 --2- 172.21.15.28:0/3147962748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c0785a0 0x7f068c1a5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.717+0000 7f0690e00640 1 -- 172.21.15.28:0/3147962748 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f068c1a6590 con 0x7f068c0769c0 2024-04-24T08:07:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.717+0000 7f0691800640 1 --2- 172.21.15.28:0/3147962748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c0785a0 0x7f068c1a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:07:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.717+0000 7f068be00640 1 --2- 172.21.15.28:0/3147962748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f068c077c00 0x7f068c19eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:07:47.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.717+0000 7f0690e00640 1 --2- 172.21.15.28:0/3147962748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f068c0769c0 0x7f068c19e970 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f067c000f80 tx=0x7f067c003970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:47.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.717+0000 7f0689600640 1 -- 172.21.15.28:0/3147962748 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c004040 con 0x7f068c0769c0 2024-04-24T08:07:47.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.717+0000 7f0689600640 1 -- 172.21.15.28:0/3147962748 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f067c040070 con 0x7f068c0769c0 2024-04-24T08:07:47.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.717+0000 7f0693800640 1 -- 172.21.15.28:0/3147962748 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f068c1a6820 con 0x7f068c0769c0 2024-04-24T08:07:47.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.717+0000 7f0689600640 1 -- 172.21.15.28:0/3147962748 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f067c045c80 con 0x7f068c0769c0 2024-04-24T08:07:47.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.717+0000 7f0693800640 1 -- 172.21.15.28:0/3147962748 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f068c1a6d70 con 0x7f068c0769c0 2024-04-24T08:07:47.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.719+0000 7f0693800640 1 -- 172.21.15.28:0/3147962748 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f068c110bf0 con 0x7f068c0769c0 2024-04-24T08:07:47.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.720+0000 7f0689600640 1 -- 172.21.15.28:0/3147962748 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f067c045470 con 0x7f068c0769c0 2024-04-24T08:07:47.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.722+0000 7f0689600640 1 --2- 172.21.15.28:0/3147962748 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f06700a3bb0 0x7f06700a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:47.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.722+0000 7f0689600640 1 -- 172.21.15.28:0/3147962748 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f067c0f1260 con 0x7f068c0769c0 2024-04-24T08:07:47.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.722+0000 7f068be00640 1 --2- 172.21.15.28:0/3147962748 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f06700a3bb0 0x7f06700a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:47.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.723+0000 7f068be00640 1 --2- 172.21.15.28:0/3147962748 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f06700a3bb0 0x7f06700a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f068c19fe80 tx=0x7f0674005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:47.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.727+0000 7f0689600640 1 -- 172.21.15.28:0/3147962748 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f067c045e20 con 0x7f068c0769c0 2024-04-24T08:07:47.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.996+0000 7f0693800640 1 -- 172.21.15.28:0/3147962748 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f068c02d620 con 0x7f068c0769c0 2024-04-24T08:07:47.998 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:47.997+0000 7f0689600640 1 -- 172.21.15.28:0/3147962748 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10485 (secure 0 0 0) 0x7f067c0be040 con 0x7f068c0769c0 2024-04-24T08:07:47.998 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:47.999 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24523,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24535,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24556,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24572,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13},{"gid":24574,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11}],"filesystems":[]} 2024-04-24T08:07:47.999 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 13 2024-04-24T08:07:48.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.000+0000 7f0693800640 1 -- 172.21.15.28:0/3147962748 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f06700a3bb0 msgr2=0x7f06700a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:48.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.000+0000 7f0693800640 1 --2- 172.21.15.28:0/3147962748 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f06700a3bb0 0x7f06700a6070 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f068c19fe80 tx=0x7f0674005f90 comp rx=0 tx=0).stop 2024-04-24T08:07:48.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.000+0000 7f0693800640 1 -- 172.21.15.28:0/3147962748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f068c0769c0 msgr2=0x7f068c19e970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.000+0000 7f0693800640 1 --2- 172.21.15.28:0/3147962748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f068c0769c0 0x7f068c19e970 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f067c000f80 tx=0x7f067c003970 comp rx=0 tx=0).stop 2024-04-24T08:07:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.000+0000 7f0693800640 1 -- 172.21.15.28:0/3147962748 shutdown_connections 2024-04-24T08:07:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.000+0000 7f0693800640 1 --2- 172.21.15.28:0/3147962748 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f06700a3bb0 0x7f06700a6070 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.000+0000 7f0693800640 1 --2- 172.21.15.28:0/3147962748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f068c0785a0 0x7f068c1a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.000+0000 7f0693800640 1 --2- 172.21.15.28:0/3147962748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f068c077c00 0x7f068c19eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.000+0000 7f0693800640 1 --2- 172.21.15.28:0/3147962748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f068c0769c0 0x7f068c19e970 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.000+0000 7f0693800640 1 -- 172.21.15.28:0/3147962748 >> 172.21.15.28:0/3147962748 conn(0x7f068c0ff7f0 msgr2=0x7f068c1012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.000+0000 7f0693800640 1 -- 172.21.15.28:0/3147962748 shutdown_connections 2024-04-24T08:07:48.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.000+0000 7f0693800640 1 -- 172.21.15.28:0/3147962748 wait complete. 2024-04-24T08:07:48.021 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:07:48.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:47 smithi117 ceph-mon[32438]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:48.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:47 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:48.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:47 smithi194 ceph-mon[32836]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:48.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:47 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:48.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.209+0000 7fbdc6200640 1 -- 172.21.15.28:0/3597858849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdc0102ad0 msgr2=0x7fbdc0104ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:48.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.209+0000 7fbdc6200640 1 --2- 172.21.15.28:0/3597858849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdc0102ad0 0x7fbdc0104ec0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fbdb0009900 tx=0x7fbdb002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.210+0000 7fbdc6200640 1 -- 172.21.15.28:0/3597858849 shutdown_connections 2024-04-24T08:07:48.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.210+0000 7fbdc6200640 1 --2- 172.21.15.28:0/3597858849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc0107d30 0x7fbdc010a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.210+0000 7fbdc6200640 1 --2- 172.21.15.28:0/3597858849 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdc0105400 0x7fbdc01077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.210+0000 7fbdc6200640 1 --2- 172.21.15.28:0/3597858849 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdc0102ad0 0x7fbdc0104ec0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.210+0000 7fbdc6200640 1 -- 172.21.15.28:0/3597858849 >> 172.21.15.28:0/3597858849 conn(0x7fbdc00fc4e0 msgr2=0x7fbdc00fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:48.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.210+0000 7fbdc6200640 1 -- 172.21.15.28:0/3597858849 shutdown_connections 2024-04-24T08:07:48.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.210+0000 7fbdc6200640 1 -- 172.21.15.28:0/3597858849 wait complete. 2024-04-24T08:07:48.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.211+0000 7fbdc6200640 1 Processor -- start 2024-04-24T08:07:48.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.211+0000 7fbdc6200640 1 -- start start 2024-04-24T08:07:48.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.211+0000 7fbdc6200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdc0102ad0 0x7fbdc019a620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:48.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.211+0000 7fbdc6200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc0105400 0x7fbdc019ab60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:48.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.211+0000 7fbdc6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdc0107d30 0x7fbdc01a1b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:48.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.211+0000 7fbdc6200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc010dce0 con 0x7fbdc0107d30 2024-04-24T08:07:48.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.211+0000 7fbdc6200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc010db60 con 0x7fbdc0105400 2024-04-24T08:07:48.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.211+0000 7fbdc6200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbdc010de60 con 0x7fbdc0102ad0 2024-04-24T08:07:48.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.211+0000 7fbdbea00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc0105400 0x7fbdc019ab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:48.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.211+0000 7fbdbfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdc0107d30 0x7fbdc01a1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:48.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.211+0000 7fbdbf400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdc0102ad0 0x7fbdc019a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:48.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.211+0000 7fbdbea00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc0105400 0x7fbdc019ab60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:42886/0 (socket says 172.21.15.28:42886) 2024-04-24T08:07:48.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.211+0000 7fbdbea00640 1 -- 172.21.15.28:0/3112966410 learned_addr learned my addr 172.21.15.28:0/3112966410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:48.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.212+0000 7fbdbea00640 1 -- 172.21.15.28:0/3112966410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdc0102ad0 msgr2=0x7fbdc019a620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:48.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.212+0000 7fbdbea00640 1 --2- 172.21.15.28:0/3112966410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdc0102ad0 0x7fbdc019a620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.212+0000 7fbdbea00640 1 -- 172.21.15.28:0/3112966410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdc0107d30 msgr2=0x7fbdc01a1b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:48.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.212+0000 7fbdbea00640 1 --2- 172.21.15.28:0/3112966410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdc0107d30 0x7fbdc01a1b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.212+0000 7fbdbea00640 1 -- 172.21.15.28:0/3112966410 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdc01a21f0 con 0x7fbdc0105400 2024-04-24T08:07:48.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.212+0000 7fbdbea00640 1 --2- 172.21.15.28:0/3112966410 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc0105400 0x7fbdc019ab60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbdac00e9f0 tx=0x7fbdac00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:48.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.212+0000 7fbdb7e00640 1 -- 172.21.15.28:0/3112966410 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdac00ccc0 con 0x7fbdc0105400 2024-04-24T08:07:48.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.212+0000 7fbdc6200640 1 -- 172.21.15.28:0/3112966410 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdc01a24e0 con 0x7fbdc0105400 2024-04-24T08:07:48.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.212+0000 7fbdb7e00640 1 -- 172.21.15.28:0/3112966410 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdac005500 con 0x7fbdc0105400 2024-04-24T08:07:48.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.212+0000 7fbdb7e00640 1 -- 172.21.15.28:0/3112966410 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdac010690 con 0x7fbdc0105400 2024-04-24T08:07:48.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.213+0000 7fbdc6200640 1 -- 172.21.15.28:0/3112966410 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdc01a29e0 con 0x7fbdc0105400 2024-04-24T08:07:48.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.214+0000 7fbdc6200640 1 -- 172.21.15.28:0/3112966410 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbdc0109880 con 0x7fbdc0105400 2024-04-24T08:07:48.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.216+0000 7fbdb7e00640 1 -- 172.21.15.28:0/3112966410 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fbdac0046c0 con 0x7fbdc0105400 2024-04-24T08:07:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.218+0000 7fbdb7e00640 1 --2- 172.21.15.28:0/3112966410 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbda40a3bb0 0x7fbda40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.218+0000 7fbdb7e00640 1 -- 172.21.15.28:0/3112966410 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7fbdac0d0330 con 0x7fbdc0105400 2024-04-24T08:07:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.218+0000 7fbdbf400640 1 --2- 172.21.15.28:0/3112966410 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbda40a3bb0 0x7fbda40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:48.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.219+0000 7fbdbf400640 1 --2- 172.21.15.28:0/3112966410 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbda40a3bb0 0x7fbda40a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00314e0 tx=0x7fbdb0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:48.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.222+0000 7fbdb7e00640 1 -- 172.21.15.28:0/3112966410 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbdac09d110 con 0x7fbdc0105400 2024-04-24T08:07:48.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:47 smithi028 ceph-mon[27786]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:48.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:47 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:48.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.446+0000 7fbdc6200640 1 -- 172.21.15.28:0/3112966410 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbdc0062c00 con 0x7fbdc0105400 2024-04-24T08:07:48.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.447+0000 7fbdb7e00640 1 -- 172.21.15.28:0/3112966410 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) ==== 74+0+18444 (secure 0 0 0) 0x7fbdac09ca10 con 0x7fbdc0105400 2024-04-24T08:07:48.449 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:48.449 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":82,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:06:56.246184+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:07:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.449+0000 7fbdc6200640 1 -- 172.21.15.28:0/3112966410 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbda40a3bb0 msgr2=0x7fbda40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.449+0000 7fbdc6200640 1 --2- 172.21.15.28:0/3112966410 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbda40a3bb0 0x7fbda40a6070 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fbdb00314e0 tx=0x7fbdb0000bc0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.449+0000 7fbdc6200640 1 -- 172.21.15.28:0/3112966410 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc0105400 msgr2=0x7fbdc019ab60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.449+0000 7fbdc6200640 1 --2- 172.21.15.28:0/3112966410 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc0105400 0x7fbdc019ab60 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fbdac00e9f0 tx=0x7fbdac00edf0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.450+0000 7fbdc6200640 1 -- 172.21.15.28:0/3112966410 shutdown_connections 2024-04-24T08:07:48.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.450+0000 7fbdc6200640 1 --2- 172.21.15.28:0/3112966410 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbda40a3bb0 0x7fbda40a6070 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.450+0000 7fbdc6200640 1 --2- 172.21.15.28:0/3112966410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdc0107d30 0x7fbdc01a1b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.450+0000 7fbdc6200640 1 --2- 172.21.15.28:0/3112966410 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc0105400 0x7fbdc019ab60 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.450+0000 7fbdc6200640 1 --2- 172.21.15.28:0/3112966410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdc0102ad0 0x7fbdc019a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.450+0000 7fbdc6200640 1 -- 172.21.15.28:0/3112966410 >> 172.21.15.28:0/3112966410 conn(0x7fbdc00fc4e0 msgr2=0x7fbdc01011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:48.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.450+0000 7fbdc6200640 1 -- 172.21.15.28:0/3112966410 shutdown_connections 2024-04-24T08:07:48.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.450+0000 7fbdc6200640 1 -- 172.21.15.28:0/3112966410 wait complete. 2024-04-24T08:07:48.469 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:02:34.086371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:07:48.470 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:07:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.647+0000 7f17cac00640 1 -- 172.21.15.28:0/3742763799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c4107d20 msgr2=0x7f17c410a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.647+0000 7f17cac00640 1 --2- 172.21.15.28:0/3742763799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c4107d20 0x7f17c410a170 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f17bc009ca0 tx=0x7f17bc02a8c0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.647+0000 7f17cac00640 1 -- 172.21.15.28:0/3742763799 shutdown_connections 2024-04-24T08:07:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.647+0000 7f17cac00640 1 --2- 172.21.15.28:0/3742763799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c4107d20 0x7f17c410a170 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.647+0000 7f17cac00640 1 --2- 172.21.15.28:0/3742763799 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c41053f0 0x7f17c41077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.647+0000 7f17cac00640 1 --2- 172.21.15.28:0/3742763799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17c4102ac0 0x7f17c4104eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.647+0000 7f17cac00640 1 -- 172.21.15.28:0/3742763799 >> 172.21.15.28:0/3742763799 conn(0x7f17c40fc4b0 msgr2=0x7f17c40fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.647+0000 7f17cac00640 1 -- 172.21.15.28:0/3742763799 shutdown_connections 2024-04-24T08:07:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.647+0000 7f17cac00640 1 -- 172.21.15.28:0/3742763799 wait complete. 2024-04-24T08:07:48.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.648+0000 7f17cac00640 1 Processor -- start 2024-04-24T08:07:48.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.648+0000 7f17cac00640 1 -- start start 2024-04-24T08:07:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.648+0000 7f17cac00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17c4102ac0 0x7f17c41023e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.648+0000 7f17cac00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c41053f0 0x7f17c4100a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.648+0000 7f17cac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c4107d20 0x7f17c4100fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.648+0000 7f17cac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f17c410dd20 con 0x7f17c4107d20 2024-04-24T08:07:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.648+0000 7f17cac00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f17c410dba0 con 0x7f17c41053f0 2024-04-24T08:07:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.648+0000 7f17cac00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f17c410dea0 con 0x7f17c4102ac0 2024-04-24T08:07:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.649+0000 7f17c8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c4107d20 0x7f17c4100fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.649+0000 7f17c8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c4107d20 0x7f17c4100fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44508/0 (socket says 172.21.15.28:44508) 2024-04-24T08:07:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.649+0000 7f17c8c00640 1 -- 172.21.15.28:0/4196321186 learned_addr learned my addr 172.21.15.28:0/4196321186 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.649+0000 7f17c3e00640 1 --2- 172.21.15.28:0/4196321186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17c4102ac0 0x7f17c41023e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.649+0000 7f17c3400640 1 --2- 172.21.15.28:0/4196321186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c41053f0 0x7f17c4100a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.649+0000 7f17c8c00640 1 -- 172.21.15.28:0/4196321186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17c4102ac0 msgr2=0x7f17c41023e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.649+0000 7f17c8c00640 1 --2- 172.21.15.28:0/4196321186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17c4102ac0 0x7f17c41023e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.649+0000 7f17c8c00640 1 -- 172.21.15.28:0/4196321186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c41053f0 msgr2=0x7f17c4100a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.649+0000 7f17c8c00640 1 --2- 172.21.15.28:0/4196321186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c41053f0 0x7f17c4100a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.649+0000 7f17c8c00640 1 -- 172.21.15.28:0/4196321186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c4101870 con 0x7f17c4107d20 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.649+0000 7f17c3e00640 1 --2- 172.21.15.28:0/4196321186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17c4102ac0 0x7f17c41023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.650+0000 7f17c8c00640 1 --2- 172.21.15.28:0/4196321186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c4107d20 0x7f17c4100fb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f17bc02f540 tx=0x7f17bc031750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.650+0000 7f17c0c00640 1 -- 172.21.15.28:0/4196321186 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc003b70 con 0x7f17c4107d20 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.650+0000 7f17cac00640 1 -- 172.21.15.28:0/4196321186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17c41a2420 con 0x7f17c4107d20 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.650+0000 7f17c0c00640 1 -- 172.21.15.28:0/4196321186 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17bc003d10 con 0x7f17c4107d20 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.650+0000 7f17c0c00640 1 -- 172.21.15.28:0/4196321186 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17bc02fb50 con 0x7f17c4107d20 2024-04-24T08:07:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.650+0000 7f17cac00640 1 -- 172.21.15.28:0/4196321186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17c41a2940 con 0x7f17c4107d20 2024-04-24T08:07:48.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.651+0000 7f17cac00640 1 -- 172.21.15.28:0/4196321186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f17c40fdfa0 con 0x7f17c4107d20 2024-04-24T08:07:48.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.653+0000 7f17c0c00640 1 -- 172.21.15.28:0/4196321186 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f17bc02fcf0 con 0x7f17c4107d20 2024-04-24T08:07:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.657+0000 7f17c0c00640 1 --2- 172.21.15.28:0/4196321186 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f17a80a3bb0 0x7f17a80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.657+0000 7f17c0c00640 1 -- 172.21.15.28:0/4196321186 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f17bc0f1510 con 0x7f17c4107d20 2024-04-24T08:07:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.657+0000 7f17c3e00640 1 --2- 172.21.15.28:0/4196321186 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f17a80a3bb0 0x7f17a80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:48.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.657+0000 7f17c3e00640 1 --2- 172.21.15.28:0/4196321186 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f17a80a3bb0 0x7f17a80a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f17b4009fc0 tx=0x7f17b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:48.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.658+0000 7f17c0c00640 1 -- 172.21.15.28:0/4196321186 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f17bc0be2f0 con 0x7f17c4107d20 2024-04-24T08:07:48.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.913+0000 7f17cac00640 1 -- 172.21.15.28:0/4196321186 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f17c402d620 con 0x7f17c4107d20 2024-04-24T08:07:48.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.914+0000 7f17c0c00640 1 -- 172.21.15.28:0/4196321186 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) ==== 93+0+10 (secure 0 0 0) 0x7f17bc0bdbf0 con 0x7f17c4107d20 2024-04-24T08:07:48.914 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-24T08:07:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.916+0000 7f17cac00640 1 -- 172.21.15.28:0/4196321186 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f17a80a3bb0 msgr2=0x7f17a80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.916+0000 7f17cac00640 1 --2- 172.21.15.28:0/4196321186 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f17a80a3bb0 0x7f17a80a6070 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f17b4009fc0 tx=0x7f17b4005f90 comp rx=0 tx=0).stop 2024-04-24T08:07:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.916+0000 7f17cac00640 1 -- 172.21.15.28:0/4196321186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c4107d20 msgr2=0x7f17c4100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.916+0000 7f17cac00640 1 --2- 172.21.15.28:0/4196321186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c4107d20 0x7f17c4100fb0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f17bc02f540 tx=0x7f17bc031750 comp rx=0 tx=0).stop 2024-04-24T08:07:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.916+0000 7f17cac00640 1 -- 172.21.15.28:0/4196321186 shutdown_connections 2024-04-24T08:07:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.916+0000 7f17cac00640 1 --2- 172.21.15.28:0/4196321186 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f17a80a3bb0 0x7f17a80a6070 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.916+0000 7f17cac00640 1 --2- 172.21.15.28:0/4196321186 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f17c4107d20 0x7f17c4100fb0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.916+0000 7f17cac00640 1 --2- 172.21.15.28:0/4196321186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c41053f0 0x7f17c4100a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.916+0000 7f17cac00640 1 --2- 172.21.15.28:0/4196321186 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f17c4102ac0 0x7f17c41023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.916+0000 7f17cac00640 1 -- 172.21.15.28:0/4196321186 >> 172.21.15.28:0/4196321186 conn(0x7f17c40fc4b0 msgr2=0x7f17c41033a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.916+0000 7f17cac00640 1 -- 172.21.15.28:0/4196321186 shutdown_connections 2024-04-24T08:07:48.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:48.916+0000 7f17cac00640 1 -- 172.21.15.28:0/4196321186 wait complete. 2024-04-24T08:07:48.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-04-24T08:07:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.112+0000 7f83e6a00640 1 -- 172.21.15.28:0/1925270503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83e00769c0 msgr2=0x7f83e0076de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:49.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.112+0000 7f83e6a00640 1 --2- 172.21.15.28:0/1925270503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83e00769c0 0x7f83e0076de0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f83d0009ca0 tx=0x7f83d002a8c0 comp rx=0 tx=0).stop 2024-04-24T08:07:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.112+0000 7f83e6a00640 1 -- 172.21.15.28:0/1925270503 shutdown_connections 2024-04-24T08:07:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.112+0000 7f83e6a00640 1 --2- 172.21.15.28:0/1925270503 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83e00785a0 0x7f83e0078e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.112+0000 7f83e6a00640 1 --2- 172.21.15.28:0/1925270503 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83e0077c00 0x7f83e0078060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.112+0000 7f83e6a00640 1 --2- 172.21.15.28:0/1925270503 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83e00769c0 0x7f83e0076de0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.112+0000 7f83e6a00640 1 -- 172.21.15.28:0/1925270503 >> 172.21.15.28:0/1925270503 conn(0x7f83e00ff7f0 msgr2=0x7f83e0101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.112+0000 7f83e6a00640 1 -- 172.21.15.28:0/1925270503 shutdown_connections 2024-04-24T08:07:49.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.112+0000 7f83e6a00640 1 -- 172.21.15.28:0/1925270503 wait complete. 2024-04-24T08:07:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.113+0000 7f83e6a00640 1 Processor -- start 2024-04-24T08:07:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.113+0000 7f83e6a00640 1 -- start start 2024-04-24T08:07:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.114+0000 7f83e6a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83e00769c0 0x7f83e019e960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.114+0000 7f83e6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83e0077c00 0x7f83e019eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.114+0000 7f83e6a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83e00785a0 0x7f83e01a5f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.114+0000 7f83e6a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f83e0114300 con 0x7f83e0077c00 2024-04-24T08:07:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.114+0000 7f83e6a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f83e0114180 con 0x7f83e00769c0 2024-04-24T08:07:49.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.114+0000 7f83e6a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f83e0114480 con 0x7f83e00785a0 2024-04-24T08:07:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.114+0000 7f83d7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83e0077c00 0x7f83e019eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.114+0000 7f83d7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83e0077c00 0x7f83e019eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:44524/0 (socket says 172.21.15.28:44524) 2024-04-24T08:07:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.114+0000 7f83d7400640 1 -- 172.21.15.28:0/1494017540 learned_addr learned my addr 172.21.15.28:0/1494017540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.114+0000 7f83dfe00640 1 --2- 172.21.15.28:0/1494017540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83e00769c0 0x7f83e019e960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.114+0000 7f83e4a00640 1 --2- 172.21.15.28:0/1494017540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83e00785a0 0x7f83e01a5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.114+0000 7f83d7400640 1 -- 172.21.15.28:0/1494017540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83e00785a0 msgr2=0x7f83e01a5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.114+0000 7f83d7400640 1 --2- 172.21.15.28:0/1494017540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83e00785a0 0x7f83e01a5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.114+0000 7f83d7400640 1 -- 172.21.15.28:0/1494017540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83e00769c0 msgr2=0x7f83e019e960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.114+0000 7f83d7400640 1 --2- 172.21.15.28:0/1494017540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83e00769c0 0x7f83e019e960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.114+0000 7f83d7400640 1 -- 172.21.15.28:0/1494017540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f83e01a6580 con 0x7f83e0077c00 2024-04-24T08:07:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.115+0000 7f83d7400640 1 --2- 172.21.15.28:0/1494017540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83e0077c00 0x7f83e019eea0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f83c800e9f0 tx=0x7f83c800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.115+0000 7f83dd600640 1 -- 172.21.15.28:0/1494017540 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c800cd80 con 0x7f83e0077c00 2024-04-24T08:07:49.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.115+0000 7f83dd600640 1 -- 172.21.15.28:0/1494017540 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f83c8005500 con 0x7f83e0077c00 2024-04-24T08:07:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.115+0000 7f83e6a00640 1 -- 172.21.15.28:0/1494017540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f83e01a6870 con 0x7f83e0077c00 2024-04-24T08:07:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.115+0000 7f83dd600640 1 -- 172.21.15.28:0/1494017540 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f83c8010690 con 0x7f83e0077c00 2024-04-24T08:07:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.115+0000 7f83e6a00640 1 -- 172.21.15.28:0/1494017540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f83e01a6d70 con 0x7f83e0077c00 2024-04-24T08:07:49.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.116+0000 7f83e6a00640 1 -- 172.21.15.28:0/1494017540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f83e0110da0 con 0x7f83e0077c00 2024-04-24T08:07:49.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.119+0000 7f83dd600640 1 -- 172.21.15.28:0/1494017540 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f83c80046c0 con 0x7f83e0077c00 2024-04-24T08:07:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.120+0000 7f83dd600640 1 --2- 172.21.15.28:0/1494017540 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f83c40a3bb0 0x7f83c40a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:49.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.120+0000 7f83dd600640 1 -- 172.21.15.28:0/1494017540 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f83c80d0600 con 0x7f83e0077c00 2024-04-24T08:07:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.121+0000 7f83dfe00640 1 --2- 172.21.15.28:0/1494017540 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f83c40a3bb0 0x7f83c40a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:49.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.121+0000 7f83dfe00640 1 --2- 172.21.15.28:0/1494017540 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f83c40a3bb0 0x7f83c40a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f83d0006010 tx=0x7f83d0000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:49.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.122+0000 7f83dd600640 1 -- 172.21.15.28:0/1494017540 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f83c8053ff0 con 0x7f83e0077c00 2024-04-24T08:07:49.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:48 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:49.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:48 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:49.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:48 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:49.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:48 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3147962748' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:49.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:48 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3112966410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:07:49.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:48 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:49.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:48 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:49.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:48 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:49.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:48 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3147962748' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:49.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:48 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3112966410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:07:49.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:48 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:49.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:48 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:49.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:48 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:49.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:48 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3147962748' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:07:49.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:48 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3112966410' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:07:49.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.369+0000 7f83e6a00640 1 -- 172.21.15.28:0/1494017540 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7f83e002d870 con 0x7f83e0077c00 2024-04-24T08:07:49.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.370+0000 7f83dd600640 1 -- 172.21.15.28:0/1494017540 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) ==== 84+0+65 (secure 0 0 0) 0x7f83c809d3e0 con 0x7f83e0077c00 2024-04-24T08:07:49.370 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:49.370 INFO:teuthology.orchestra.run.smithi028.stdout:[ 2024-04-24T08:07:49.371 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:07:49.371 INFO:teuthology.orchestra.run.smithi028.stdout: "poolnum": 1, 2024-04-24T08:07:49.371 INFO:teuthology.orchestra.run.smithi028.stdout: "poolname": ".mgr" 2024-04-24T08:07:49.371 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:07:49.371 INFO:teuthology.orchestra.run.smithi028.stdout:] 2024-04-24T08:07:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.372+0000 7f83e6a00640 1 -- 172.21.15.28:0/1494017540 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f83c40a3bb0 msgr2=0x7f83c40a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.372+0000 7f83e6a00640 1 --2- 172.21.15.28:0/1494017540 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f83c40a3bb0 0x7f83c40a6070 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f83d0006010 tx=0x7f83d0000f40 comp rx=0 tx=0).stop 2024-04-24T08:07:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.372+0000 7f83e6a00640 1 -- 172.21.15.28:0/1494017540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83e0077c00 msgr2=0x7f83e019eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.372+0000 7f83e6a00640 1 --2- 172.21.15.28:0/1494017540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83e0077c00 0x7f83e019eea0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f83c800e9f0 tx=0x7f83c800edf0 comp rx=0 tx=0).stop 2024-04-24T08:07:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.372+0000 7f83e6a00640 1 -- 172.21.15.28:0/1494017540 shutdown_connections 2024-04-24T08:07:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.372+0000 7f83e6a00640 1 --2- 172.21.15.28:0/1494017540 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f83c40a3bb0 0x7f83c40a6070 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.372+0000 7f83e6a00640 1 --2- 172.21.15.28:0/1494017540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f83e0077c00 0x7f83e019eea0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.372+0000 7f83e6a00640 1 --2- 172.21.15.28:0/1494017540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f83e00785a0 0x7f83e01a5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.372+0000 7f83e6a00640 1 --2- 172.21.15.28:0/1494017540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f83e00769c0 0x7f83e019e960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.372+0000 7f83e6a00640 1 -- 172.21.15.28:0/1494017540 >> 172.21.15.28:0/1494017540 conn(0x7f83e00ff7f0 msgr2=0x7f83e0100f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:49.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.372+0000 7f83e6a00640 1 -- 172.21.15.28:0/1494017540 shutdown_connections 2024-04-24T08:07:49.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.372+0000 7f83e6a00640 1 -- 172.21.15.28:0/1494017540 wait complete. 2024-04-24T08:07:49.391 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-04-24T08:07:49.392 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-04-24T08:07:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.579+0000 7f16aa600640 1 -- 172.21.15.28:0/1875498610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16a4076040 msgr2=0x7f16a4074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.579+0000 7f16aa600640 1 --2- 172.21.15.28:0/1875498610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16a4076040 0x7f16a4074490 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f1698009900 tx=0x7f169802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:07:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.579+0000 7f16aa600640 1 -- 172.21.15.28:0/1875498610 shutdown_connections 2024-04-24T08:07:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.579+0000 7f16aa600640 1 --2- 172.21.15.28:0/1875498610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4074a60 0x7f16a4074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.579+0000 7f16aa600640 1 --2- 172.21.15.28:0/1875498610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16a4109dd0 0x7f16a410c260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.579+0000 7f16aa600640 1 --2- 172.21.15.28:0/1875498610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16a4076040 0x7f16a4074490 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.579+0000 7f16aa600640 1 -- 172.21.15.28:0/1875498610 >> 172.21.15.28:0/1875498610 conn(0x7f16a40fd4a0 msgr2=0x7f16a40ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.579+0000 7f16aa600640 1 -- 172.21.15.28:0/1875498610 shutdown_connections 2024-04-24T08:07:49.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.579+0000 7f16aa600640 1 -- 172.21.15.28:0/1875498610 wait complete. 2024-04-24T08:07:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.580+0000 7f16aa600640 1 Processor -- start 2024-04-24T08:07:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.580+0000 7f16aa600640 1 -- start start 2024-04-24T08:07:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.581+0000 7f16aa600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16a4074a60 0x7f16a419a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.581+0000 7f16aa600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16a4076040 0x7f16a419a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.581+0000 7f16aa600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4109dd0 0x7f16a41a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.581+0000 7f16aa600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16a410fde0 con 0x7f16a4109dd0 2024-04-24T08:07:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.581+0000 7f16aa600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f16a410fc60 con 0x7f16a4074a60 2024-04-24T08:07:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.581+0000 7f16aa600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f16a410ff60 con 0x7f16a4076040 2024-04-24T08:07:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.581+0000 7f16a3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16a4076040 0x7f16a419a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.581+0000 7f16a3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16a4076040 0x7f16a419a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:34688/0 (socket says 172.21.15.28:34688) 2024-04-24T08:07:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.581+0000 7f16a3e00640 1 -- 172.21.15.28:0/2520562604 learned_addr learned my addr 172.21.15.28:0/2520562604 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.581+0000 7f16a9200640 1 --2- 172.21.15.28:0/2520562604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16a4074a60 0x7f16a419a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.581+0000 7f16a9c00640 1 --2- 172.21.15.28:0/2520562604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4109dd0 0x7f16a41a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.581+0000 7f16a3e00640 1 -- 172.21.15.28:0/2520562604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16a4074a60 msgr2=0x7f16a419a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.581+0000 7f16a3e00640 1 --2- 172.21.15.28:0/2520562604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16a4074a60 0x7f16a419a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.582+0000 7f16a3e00640 1 -- 172.21.15.28:0/2520562604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4109dd0 msgr2=0x7f16a41a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:49.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.582+0000 7f16a3e00640 1 --2- 172.21.15.28:0/2520562604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4109dd0 0x7f16a41a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.582+0000 7f16a3e00640 1 -- 172.21.15.28:0/2520562604 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16a41a2010 con 0x7f16a4076040 2024-04-24T08:07:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.582+0000 7f16a9200640 1 --2- 172.21.15.28:0/2520562604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16a4074a60 0x7f16a419a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.582+0000 7f16a3e00640 1 --2- 172.21.15.28:0/2520562604 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16a4076040 0x7f16a419a930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f169000e9f0 tx=0x7f169000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.582+0000 7f16a1600640 1 -- 172.21.15.28:0/2520562604 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f169000cd80 con 0x7f16a4076040 2024-04-24T08:07:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.582+0000 7f16aa600640 1 -- 172.21.15.28:0/2520562604 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16a41a2300 con 0x7f16a4076040 2024-04-24T08:07:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.582+0000 7f16a1600640 1 -- 172.21.15.28:0/2520562604 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1690010910 con 0x7f16a4076040 2024-04-24T08:07:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.582+0000 7f16a1600640 1 -- 172.21.15.28:0/2520562604 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16900188f0 con 0x7f16a4076040 2024-04-24T08:07:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.582+0000 7f16aa600640 1 -- 172.21.15.28:0/2520562604 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16a41a28a0 con 0x7f16a4076040 2024-04-24T08:07:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.583+0000 7f16aa600640 1 -- 172.21.15.28:0/2520562604 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16a4108b00 con 0x7f16a4076040 2024-04-24T08:07:49.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.586+0000 7f16a1600640 1 -- 172.21.15.28:0/2520562604 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f1690010ac0 con 0x7f16a4076040 2024-04-24T08:07:49.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.587+0000 7f16a1600640 1 --2- 172.21.15.28:0/2520562604 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f168c0a3c30 0x7f168c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:49.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.588+0000 7f16a1600640 1 -- 172.21.15.28:0/2520562604 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7127+0+0 (secure 0 0 0) 0x7f16900d0300 con 0x7f16a4076040 2024-04-24T08:07:49.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.588+0000 7f16a9200640 1 --2- 172.21.15.28:0/2520562604 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f168c0a3c30 0x7f168c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:49.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.589+0000 7f16a9200640 1 --2- 172.21.15.28:0/2520562604 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f168c0a3c30 0x7f168c0a60f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1698031440 tx=0x7f16980315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:49.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.592+0000 7f16a1600640 1 -- 172.21.15.28:0/2520562604 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f169009ca90 con 0x7f16a4076040 2024-04-24T08:07:49.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:49.801+0000 7f16aa600640 1 -- 172.21.15.28:0/2520562604 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7f16a402d620 con 0x7f16a4076040 2024-04-24T08:07:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.019+0000 7f16a1600640 1 -- 172.21.15.28:0/2520562604 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) ==== 138+0+0 (secure 0 0 0) 0x7f16900a0d90 con 0x7f16a4076040 2024-04-24T08:07:50.019 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' created 2024-04-24T08:07:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.024+0000 7f16aa600640 1 -- 172.21.15.28:0/2520562604 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f168c0a3c30 msgr2=0x7f168c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.024+0000 7f16aa600640 1 --2- 172.21.15.28:0/2520562604 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f168c0a3c30 0x7f168c0a60f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f1698031440 tx=0x7f16980315b0 comp rx=0 tx=0).stop 2024-04-24T08:07:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.024+0000 7f16aa600640 1 -- 172.21.15.28:0/2520562604 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16a4076040 msgr2=0x7f16a419a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.024+0000 7f16aa600640 1 --2- 172.21.15.28:0/2520562604 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16a4076040 0x7f16a419a930 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f169000e9f0 tx=0x7f169000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:07:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.024+0000 7f16aa600640 1 -- 172.21.15.28:0/2520562604 shutdown_connections 2024-04-24T08:07:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.024+0000 7f16aa600640 1 --2- 172.21.15.28:0/2520562604 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f168c0a3c30 0x7f168c0a60f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.024+0000 7f16aa600640 1 --2- 172.21.15.28:0/2520562604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16a4109dd0 0x7f16a41a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.024+0000 7f16aa600640 1 --2- 172.21.15.28:0/2520562604 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16a4076040 0x7f16a419a930 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.024+0000 7f16aa600640 1 --2- 172.21.15.28:0/2520562604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16a4074a60 0x7f16a419a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:50.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.024+0000 7f16aa600640 1 -- 172.21.15.28:0/2520562604 >> 172.21.15.28:0/2520562604 conn(0x7f16a40fd4a0 msgr2=0x7f16a4105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.025+0000 7f16aa600640 1 -- 172.21.15.28:0/2520562604 shutdown_connections 2024-04-24T08:07:50.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.025+0000 7f16aa600640 1 -- 172.21.15.28:0/2520562604 wait complete. 2024-04-24T08:07:50.048 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-04-24T08:07:50.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:49 smithi117 ceph-mon[32438]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:50.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:49 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/4196321186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:07:50.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:49 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:50.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:49 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:50.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:49 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:50.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:49 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:07:50.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:49 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:50.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:49 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:07:50.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:49 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1494017540' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-24T08:07:50.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:49 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2520562604' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-24T08:07:50.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:49 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-24T08:07:50.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:49 smithi194 ceph-mon[32836]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:50.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:49 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/4196321186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:07:50.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:49 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:50.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:49 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:50.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:49 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:50.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:49 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:07:50.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:49 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:50.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:49 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:07:50.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:49 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1494017540' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-24T08:07:50.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:49 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2520562604' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-24T08:07:50.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:49 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-24T08:07:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.238+0000 7f909f000640 1 -- 172.21.15.28:0/2631499388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9098104280 msgr2=0x7f90981046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.238+0000 7f909f000640 1 --2- 172.21.15.28:0/2631499388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9098104280 0x7f90981046a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f9088009900 tx=0x7f908802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:07:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.238+0000 7f909f000640 1 -- 172.21.15.28:0/2631499388 shutdown_connections 2024-04-24T08:07:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.239+0000 7f909f000640 1 --2- 172.21.15.28:0/2631499388 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9098105e30 0x7f909810c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.239+0000 7f909f000640 1 --2- 172.21.15.28:0/2631499388 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9098105470 0x7f90981058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.239+0000 7f909f000640 1 --2- 172.21.15.28:0/2631499388 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9098104280 0x7f90981046a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.239+0000 7f909f000640 1 -- 172.21.15.28:0/2631499388 >> 172.21.15.28:0/2631499388 conn(0x7f90980ff7f0 msgr2=0x7f9098101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.239+0000 7f909f000640 1 -- 172.21.15.28:0/2631499388 shutdown_connections 2024-04-24T08:07:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.239+0000 7f909f000640 1 -- 172.21.15.28:0/2631499388 wait complete. 2024-04-24T08:07:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.240+0000 7f909f000640 1 Processor -- start 2024-04-24T08:07:50.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.240+0000 7f909f000640 1 -- start start 2024-04-24T08:07:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.240+0000 7f909f000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9098104280 0x7f909819a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:50.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.241+0000 7f909f000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9098105470 0x7f909819aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.241+0000 7f909f000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9098105e30 0x7f90981a1b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.241+0000 7f909f000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f909810ff60 con 0x7f9098105470 2024-04-24T08:07:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.241+0000 7f909f000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f909810fde0 con 0x7f9098104280 2024-04-24T08:07:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.241+0000 7f909f000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f90981100e0 con 0x7f9098105e30 2024-04-24T08:07:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.241+0000 7f9097e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9098104280 0x7f909819a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.241+0000 7f9097e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9098104280 0x7f909819a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:57532/0 (socket says 172.21.15.28:57532) 2024-04-24T08:07:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.241+0000 7f9097400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9098105470 0x7f909819aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.241+0000 7f9097e00640 1 -- 172.21.15.28:0/3466283874 learned_addr learned my addr 172.21.15.28:0/3466283874 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.241+0000 7f909d000640 1 --2- 172.21.15.28:0/3466283874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9098105e30 0x7f90981a1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.241+0000 7f9097e00640 1 -- 172.21.15.28:0/3466283874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9098105e30 msgr2=0x7f90981a1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.241+0000 7f9097e00640 1 --2- 172.21.15.28:0/3466283874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9098105e30 0x7f90981a1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.241+0000 7f9097e00640 1 -- 172.21.15.28:0/3466283874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9098105470 msgr2=0x7f909819aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.241+0000 7f9097e00640 1 --2- 172.21.15.28:0/3466283874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9098105470 0x7f909819aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.241+0000 7f9097e00640 1 -- 172.21.15.28:0/3466283874 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f90981a21d0 con 0x7f9098104280 2024-04-24T08:07:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.241+0000 7f909d000640 1 --2- 172.21.15.28:0/3466283874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9098105e30 0x7f90981a1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.242+0000 7f9097e00640 1 --2- 172.21.15.28:0/3466283874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9098104280 0x7f909819a5b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9088031440 tx=0x7f9088032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.242+0000 7f9094c00640 1 -- 172.21.15.28:0/3466283874 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f908803f030 con 0x7f9098104280 2024-04-24T08:07:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.242+0000 7f9094c00640 1 -- 172.21.15.28:0/3466283874 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9088003ce0 con 0x7f9098104280 2024-04-24T08:07:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.242+0000 7f9094c00640 1 -- 172.21.15.28:0/3466283874 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9088035070 con 0x7f9098104280 2024-04-24T08:07:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.242+0000 7f909f000640 1 -- 172.21.15.28:0/3466283874 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f90981a2460 con 0x7f9098104280 2024-04-24T08:07:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.242+0000 7f909f000640 1 -- 172.21.15.28:0/3466283874 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90981a2950 con 0x7f9098104280 2024-04-24T08:07:50.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.244+0000 7f909f000640 1 -- 172.21.15.28:0/3466283874 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f909810b5a0 con 0x7f9098104280 2024-04-24T08:07:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.246+0000 7f9094c00640 1 -- 172.21.15.28:0/3466283874 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9088042030 con 0x7f9098104280 2024-04-24T08:07:50.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.248+0000 7f9094c00640 1 --2- 172.21.15.28:0/3466283874 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f907c0a3bb0 0x7f907c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:50.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.248+0000 7f9094c00640 1 -- 172.21.15.28:0/3466283874 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7497+0+0 (secure 0 0 0) 0x7f90880f0220 con 0x7f9098104280 2024-04-24T08:07:50.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.248+0000 7f9097400640 1 --2- 172.21.15.28:0/3466283874 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f907c0a3bb0 0x7f907c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:50.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.249+0000 7f9097400640 1 --2- 172.21.15.28:0/3466283874 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f907c0a3bb0 0x7f907c0a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f909819bac0 tx=0x7f9084005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:50.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.253+0000 7f9094c00640 1 -- 172.21.15.28:0/3466283874 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f90880bc840 con 0x7f9098104280 2024-04-24T08:07:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:49 smithi028 ceph-mon[27786]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:49 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/4196321186' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:07:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:07:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:49 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:07:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:49 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1494017540' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-04-24T08:07:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:49 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2520562604' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-24T08:07:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:49 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-04-24T08:07:50.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:50.479+0000 7f909f000640 1 -- 172.21.15.28:0/3466283874 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7f909802d620 con 0x7f9098104280 2024-04-24T08:07:51.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.041+0000 7f9094c00640 1 -- 172.21.15.28:0/3466283874 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) ==== 170+0+0 (secure 0 0 0) 0x7f90880c0b40 con 0x7f9098104280 2024-04-24T08:07:51.043 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' created 2024-04-24T08:07:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.048+0000 7f909f000640 1 -- 172.21.15.28:0/3466283874 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f907c0a3bb0 msgr2=0x7f907c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.048+0000 7f909f000640 1 --2- 172.21.15.28:0/3466283874 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f907c0a3bb0 0x7f907c0a6070 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f909819bac0 tx=0x7f9084005f90 comp rx=0 tx=0).stop 2024-04-24T08:07:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.048+0000 7f909f000640 1 -- 172.21.15.28:0/3466283874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9098104280 msgr2=0x7f909819a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.048+0000 7f909f000640 1 --2- 172.21.15.28:0/3466283874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9098104280 0x7f909819a5b0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f9088031440 tx=0x7f9088032ce0 comp rx=0 tx=0).stop 2024-04-24T08:07:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.049+0000 7f909f000640 1 -- 172.21.15.28:0/3466283874 shutdown_connections 2024-04-24T08:07:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.049+0000 7f909f000640 1 --2- 172.21.15.28:0/3466283874 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f907c0a3bb0 0x7f907c0a6070 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:51.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.049+0000 7f909f000640 1 --2- 172.21.15.28:0/3466283874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9098105470 0x7f909819aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.049+0000 7f909f000640 1 --2- 172.21.15.28:0/3466283874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9098105e30 0x7f90981a1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.049+0000 7f909f000640 1 --2- 172.21.15.28:0/3466283874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9098104280 0x7f909819a5b0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.049+0000 7f909f000640 1 -- 172.21.15.28:0/3466283874 >> 172.21.15.28:0/3466283874 conn(0x7f90980ff7f0 msgr2=0x7f9098108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.049+0000 7f909f000640 1 -- 172.21.15.28:0/3466283874 shutdown_connections 2024-04-24T08:07:51.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.049+0000 7f909f000640 1 -- 172.21.15.28:0/3466283874 wait complete. 2024-04-24T08:07:51.085 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-04-24T08:07:51.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:51 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-24T08:07:51.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:51 smithi028 ceph-mon[27786]: osdmap e83: 12 total, 12 up, 12 in 2024-04-24T08:07:51.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:51.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:51 smithi028 ceph-mon[27786]: pgmap v313: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:51.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:51 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3466283874' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-24T08:07:51.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:51 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-24T08:07:51.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:51.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:51.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:51.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.299+0000 7f57ea000640 1 -- 172.21.15.28:0/940912129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57e4103230 msgr2=0x7f57e41036b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:51.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.299+0000 7f57ea000640 1 --2- 172.21.15.28:0/940912129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57e4103230 0x7f57e41036b0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f57d0009860 tx=0x7f57d002a340 comp rx=0 tx=0).stop 2024-04-24T08:07:51.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.299+0000 7f57ea000640 1 -- 172.21.15.28:0/940912129 shutdown_connections 2024-04-24T08:07:51.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.299+0000 7f57ea000640 1 --2- 172.21.15.28:0/940912129 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57e4103bf0 0x7f57e410a480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:51.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.299+0000 7f57ea000640 1 --2- 172.21.15.28:0/940912129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57e4103230 0x7f57e41036b0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:51.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.299+0000 7f57ea000640 1 --2- 172.21.15.28:0/940912129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57e4102040 0x7f57e4102460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:51.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.299+0000 7f57ea000640 1 -- 172.21.15.28:0/940912129 >> 172.21.15.28:0/940912129 conn(0x7f57e40fd5b0 msgr2=0x7f57e40ff9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:51.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.299+0000 7f57ea000640 1 -- 172.21.15.28:0/940912129 shutdown_connections 2024-04-24T08:07:51.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.299+0000 7f57ea000640 1 -- 172.21.15.28:0/940912129 wait complete. 2024-04-24T08:07:51.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.300+0000 7f57ea000640 1 Processor -- start 2024-04-24T08:07:51.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.301+0000 7f57ea000640 1 -- start start 2024-04-24T08:07:51.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.301+0000 7f57ea000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57e4102040 0x7f57e419a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:51.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.301+0000 7f57ea000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57e4103230 0x7f57e419ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:51.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.301+0000 7f57ea000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57e4103bf0 0x7f57e41a1ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:51.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.301+0000 7f57ea000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57e410dcd0 con 0x7f57e4102040 2024-04-24T08:07:51.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.301+0000 7f57ea000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f57e410db50 con 0x7f57e4103230 2024-04-24T08:07:51.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.301+0000 7f57ea000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f57e410de50 con 0x7f57e4103bf0 2024-04-24T08:07:51.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.301+0000 7f57e3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57e4102040 0x7f57e419a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:51.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.301+0000 7f57e3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57e4102040 0x7f57e419a5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37140/0 (socket says 172.21.15.28:37140) 2024-04-24T08:07:51.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.301+0000 7f57e2a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57e4103230 0x7f57e419ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:51.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.301+0000 7f57e3400640 1 -- 172.21.15.28:0/3890778998 learned_addr learned my addr 172.21.15.28:0/3890778998 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:51.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.301+0000 7f57e3e00640 1 --2- 172.21.15.28:0/3890778998 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57e4103bf0 0x7f57e41a1ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:51.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.302+0000 7f57e2a00640 1 -- 172.21.15.28:0/3890778998 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57e4103bf0 msgr2=0x7f57e41a1ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:51.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.302+0000 7f57e2a00640 1 --2- 172.21.15.28:0/3890778998 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57e4103bf0 0x7f57e41a1ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.302+0000 7f57e2a00640 1 -- 172.21.15.28:0/3890778998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57e4102040 msgr2=0x7f57e419a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.302+0000 7f57e2a00640 1 --2- 172.21.15.28:0/3890778998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57e4102040 0x7f57e419a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.302+0000 7f57e2a00640 1 -- 172.21.15.28:0/3890778998 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57e41a2150 con 0x7f57e4103230 2024-04-24T08:07:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.302+0000 7f57e2a00640 1 --2- 172.21.15.28:0/3890778998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57e4103230 0x7f57e419ab10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f57d0035040 tx=0x7f57d0033d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.302+0000 7f57dbe00640 1 -- 172.21.15.28:0/3890778998 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d0037040 con 0x7f57e4103230 2024-04-24T08:07:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.302+0000 7f57ea000640 1 -- 172.21.15.28:0/3890778998 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57e41a23e0 con 0x7f57e4103230 2024-04-24T08:07:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.302+0000 7f57dbe00640 1 -- 172.21.15.28:0/3890778998 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57d0030850 con 0x7f57e4103230 2024-04-24T08:07:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.302+0000 7f57dbe00640 1 -- 172.21.15.28:0/3890778998 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57d0004920 con 0x7f57e4103230 2024-04-24T08:07:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.302+0000 7f57ea000640 1 -- 172.21.15.28:0/3890778998 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57e41a28a0 con 0x7f57e4103230 2024-04-24T08:07:51.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.304+0000 7f57ea000640 1 -- 172.21.15.28:0/3890778998 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57e41a2bc0 con 0x7f57e4103230 2024-04-24T08:07:51.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.306+0000 7f57dbe00640 1 -- 172.21.15.28:0/3890778998 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f57d0004050 con 0x7f57e4103230 2024-04-24T08:07:51.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.307+0000 7f57dbe00640 1 --2- 172.21.15.28:0/3890778998 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f57c80a3bb0 0x7f57c80a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:51.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.307+0000 7f57dbe00640 1 -- 172.21.15.28:0/3890778998 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7879+0+0 (secure 0 0 0) 0x7f57d00eff50 con 0x7f57e4103230 2024-04-24T08:07:51.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.308+0000 7f57e3400640 1 --2- 172.21.15.28:0/3890778998 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f57c80a3bb0 0x7f57c80a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:51.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.308+0000 7f57e3400640 1 --2- 172.21.15.28:0/3890778998 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f57c80a3bb0 0x7f57c80a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f57d4009c60 tx=0x7f57d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:51.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.311+0000 7f57dbe00640 1 -- 172.21.15.28:0/3890778998 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f57d0049020 con 0x7f57e4103230 2024-04-24T08:07:51.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:51 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-24T08:07:51.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:51 smithi117 ceph-mon[32438]: osdmap e83: 12 total, 12 up, 12 in 2024-04-24T08:07:51.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:51.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:51 smithi117 ceph-mon[32438]: pgmap v313: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:51.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:51 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3466283874' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-24T08:07:51.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:51 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-24T08:07:51.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:51.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:51.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:51.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:51 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-04-24T08:07:51.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:51 smithi194 ceph-mon[32836]: osdmap e83: 12 total, 12 up, 12 in 2024-04-24T08:07:51.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:51.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:51 smithi194 ceph-mon[32836]: pgmap v313: 33 pgs: 32 unknown, 1 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:51.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:51 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3466283874' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-24T08:07:51.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:51 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-04-24T08:07:51.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:51.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:51.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:51.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:51.582+0000 7f57ea000640 1 -- 172.21.15.28:0/3890778998 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7f57e41a2d90 con 0x7f57e4103230 2024-04-24T08:07:52.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.076+0000 7f57dbe00640 1 -- 172.21.15.28:0/3890778998 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7f57d00bca40 con 0x7f57e4103230 2024-04-24T08:07:52.077 INFO:teuthology.orchestra.run.smithi028.stderr:new fs with metadata pool 2 and data pool 3 2024-04-24T08:07:52.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.083+0000 7f57ea000640 1 -- 172.21.15.28:0/3890778998 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f57c80a3bb0 msgr2=0x7f57c80a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:52.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.083+0000 7f57ea000640 1 --2- 172.21.15.28:0/3890778998 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f57c80a3bb0 0x7f57c80a6070 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f57d4009c60 tx=0x7f57d4009260 comp rx=0 tx=0).stop 2024-04-24T08:07:52.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.083+0000 7f57ea000640 1 -- 172.21.15.28:0/3890778998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57e4103230 msgr2=0x7f57e419ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:52.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.083+0000 7f57ea000640 1 --2- 172.21.15.28:0/3890778998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57e4103230 0x7f57e419ab10 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f57d0035040 tx=0x7f57d0033d90 comp rx=0 tx=0).stop 2024-04-24T08:07:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.084+0000 7f57ea000640 1 -- 172.21.15.28:0/3890778998 shutdown_connections 2024-04-24T08:07:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.084+0000 7f57ea000640 1 --2- 172.21.15.28:0/3890778998 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f57c80a3bb0 0x7f57c80a6070 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.084+0000 7f57ea000640 1 --2- 172.21.15.28:0/3890778998 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57e4103bf0 0x7f57e41a1ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.084+0000 7f57ea000640 1 --2- 172.21.15.28:0/3890778998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57e4103230 0x7f57e419ab10 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.084+0000 7f57ea000640 1 --2- 172.21.15.28:0/3890778998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57e4102040 0x7f57e419a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.084+0000 7f57ea000640 1 -- 172.21.15.28:0/3890778998 >> 172.21.15.28:0/3890778998 conn(0x7f57e40fd5b0 msgr2=0x7f57e4074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:52.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.084+0000 7f57ea000640 1 -- 172.21.15.28:0/3890778998 shutdown_connections 2024-04-24T08:07:52.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.084+0000 7f57ea000640 1 -- 172.21.15.28:0/3890778998 wait complete. 2024-04-24T08:07:52.108 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:07:52.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.287+0000 7f983e600640 1 -- 172.21.15.28:0/989711513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9838104260 msgr2=0x7f9838104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:52.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.287+0000 7f983e600640 1 --2- 172.21.15.28:0/989711513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9838104260 0x7f9838104680 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f9828009900 tx=0x7f982802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.287+0000 7f983e600640 1 -- 172.21.15.28:0/989711513 shutdown_connections 2024-04-24T08:07:52.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.287+0000 7f983e600640 1 --2- 172.21.15.28:0/989711513 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9838105e10 0x7f983810c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.287+0000 7f983e600640 1 --2- 172.21.15.28:0/989711513 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9838105450 0x7f98381058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.287+0000 7f983e600640 1 --2- 172.21.15.28:0/989711513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9838104260 0x7f9838104680 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.287+0000 7f983e600640 1 -- 172.21.15.28:0/989711513 >> 172.21.15.28:0/989711513 conn(0x7f98380ff7f0 msgr2=0x7f9838101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:52.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.287+0000 7f983e600640 1 -- 172.21.15.28:0/989711513 shutdown_connections 2024-04-24T08:07:52.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.287+0000 7f983e600640 1 -- 172.21.15.28:0/989711513 wait complete. 2024-04-24T08:07:52.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:52 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-24T08:07:52.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:52 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:52.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:52 smithi028 ceph-mon[27786]: osdmap e84: 12 total, 12 up, 12 in 2024-04-24T08:07:52.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:52 smithi028 ceph-mon[27786]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-24T08:07:52.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:52 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3890778998' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-24T08:07:52.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:52 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-24T08:07:52.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:52 smithi028 ceph-mon[27786]: daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-24T08:07:52.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.288+0000 7f983e600640 1 Processor -- start 2024-04-24T08:07:52.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.288+0000 7f983e600640 1 -- start start 2024-04-24T08:07:52.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.289+0000 7f983e600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9838104260 0x7f983819e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:52.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.289+0000 7f983e600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9838105450 0x7f983819ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:52.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.289+0000 7f983e600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9838105e10 0x7f98381a5fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:52.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.289+0000 7f983e600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f983810ff40 con 0x7f9838105e10 2024-04-24T08:07:52.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.289+0000 7f983e600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f983810fdc0 con 0x7f9838105450 2024-04-24T08:07:52.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.289+0000 7f983e600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f98381100c0 con 0x7f9838104260 2024-04-24T08:07:52.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.289+0000 7f9837400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9838104260 0x7f983819e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:52.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.289+0000 7f9837400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9838104260 0x7f983819e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:52370/0 (socket says 172.21.15.28:52370) 2024-04-24T08:07:52.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.289+0000 7f9837400640 1 -- 172.21.15.28:0/2896612429 learned_addr learned my addr 172.21.15.28:0/2896612429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:52.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.289+0000 7f9837e00640 1 --2- 172.21.15.28:0/2896612429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9838105e10 0x7f98381a5fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:52.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.289+0000 7f982fe00640 1 --2- 172.21.15.28:0/2896612429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9838105450 0x7f983819ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.290+0000 7f9837400640 1 -- 172.21.15.28:0/2896612429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9838105450 msgr2=0x7f983819ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.290+0000 7f9837400640 1 --2- 172.21.15.28:0/2896612429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9838105450 0x7f983819ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.290+0000 7f9837400640 1 -- 172.21.15.28:0/2896612429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9838105e10 msgr2=0x7f98381a5fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.290+0000 7f9837400640 1 --2- 172.21.15.28:0/2896612429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9838105e10 0x7f98381a5fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.290+0000 7f9837400640 1 -- 172.21.15.28:0/2896612429 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98381a6610 con 0x7f9838104260 2024-04-24T08:07:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.290+0000 7f9837e00640 1 --2- 172.21.15.28:0/2896612429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9838105e10 0x7f98381a5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.290+0000 7f9837400640 1 --2- 172.21.15.28:0/2896612429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9838104260 0x7f983819e9f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9828031440 tx=0x7f9828032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.290+0000 7f9834c00640 1 -- 172.21.15.28:0/2896612429 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f982803f030 con 0x7f9838104260 2024-04-24T08:07:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.290+0000 7f983e600640 1 -- 172.21.15.28:0/2896612429 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98381a68a0 con 0x7f9838104260 2024-04-24T08:07:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.290+0000 7f9834c00640 1 -- 172.21.15.28:0/2896612429 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9828003ce0 con 0x7f9838104260 2024-04-24T08:07:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.290+0000 7f9834c00640 1 -- 172.21.15.28:0/2896612429 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9828035070 con 0x7f9838104260 2024-04-24T08:07:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.290+0000 7f983e600640 1 -- 172.21.15.28:0/2896612429 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98381a6de0 con 0x7f9838104260 2024-04-24T08:07:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.292+0000 7f983e600640 1 -- 172.21.15.28:0/2896612429 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98381a6fe0 con 0x7f9838104260 2024-04-24T08:07:52.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.293+0000 7f9834c00640 1 -- 172.21.15.28:0/2896612429 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9828042030 con 0x7f9838104260 2024-04-24T08:07:52.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.295+0000 7f9834c00640 1 --2- 172.21.15.28:0/2896612429 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f981c0a3c30 0x7f981c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:52.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.295+0000 7f9834c00640 1 -- 172.21.15.28:0/2896612429 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f98280f0750 con 0x7f9838104260 2024-04-24T08:07:52.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.295+0000 7f982fe00640 1 --2- 172.21.15.28:0/2896612429 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f981c0a3c30 0x7f981c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:52.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.296+0000 7f982fe00640 1 --2- 172.21.15.28:0/2896612429 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f981c0a3c30 0x7f981c0a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f983819ff00 tx=0x7f9820005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:52.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.299+0000 7f9834c00640 1 -- 172.21.15.28:0/2896612429 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f98280bcb90 con 0x7f9838104260 2024-04-24T08:07:52.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:52 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-24T08:07:52.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:52 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:52.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:52 smithi117 ceph-mon[32438]: osdmap e84: 12 total, 12 up, 12 in 2024-04-24T08:07:52.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:52 smithi117 ceph-mon[32438]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-24T08:07:52.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:52 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3890778998' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-24T08:07:52.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:52 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-24T08:07:52.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:52 smithi117 ceph-mon[32438]: daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-24T08:07:52.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:52 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-04-24T08:07:52.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:52 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:52.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:52 smithi194 ceph-mon[32836]: osdmap e84: 12 total, 12 up, 12 in 2024-04-24T08:07:52.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:52 smithi194 ceph-mon[32836]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-04-24T08:07:52.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:52 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3890778998' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-24T08:07:52.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:52 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-04-24T08:07:52.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:52 smithi194 ceph-mon[32836]: daemon mds.g assigned to filesystem cephfs as rank 0 2024-04-24T08:07:52.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.532+0000 7f983e600640 1 -- 172.21.15.28:0/2896612429 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f98381a72d0 con 0x7f9838104260 2024-04-24T08:07:52.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.534+0000 7f9834c00640 1 -- 172.21.15.28:0/2896612429 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22220 (secure 0 0 0) 0x7f98280c0e90 con 0x7f9838104260 2024-04-24T08:07:52.535 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:52.536 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":85,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:07:51.585715+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:07:52.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.537+0000 7f983e600640 1 -- 172.21.15.28:0/2896612429 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f981c0a3c30 msgr2=0x7f981c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:52.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.537+0000 7f983e600640 1 --2- 172.21.15.28:0/2896612429 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f981c0a3c30 0x7f981c0a60f0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f983819ff00 tx=0x7f9820005f90 comp rx=0 tx=0).stop 2024-04-24T08:07:52.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.537+0000 7f983e600640 1 -- 172.21.15.28:0/2896612429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9838104260 msgr2=0x7f983819e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:52.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.537+0000 7f983e600640 1 --2- 172.21.15.28:0/2896612429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9838104260 0x7f983819e9f0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f9828031440 tx=0x7f9828032ce0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.537+0000 7f983e600640 1 -- 172.21.15.28:0/2896612429 shutdown_connections 2024-04-24T08:07:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.537+0000 7f983e600640 1 --2- 172.21.15.28:0/2896612429 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f981c0a3c30 0x7f981c0a60f0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.537+0000 7f983e600640 1 --2- 172.21.15.28:0/2896612429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9838105e10 0x7f98381a5fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.537+0000 7f983e600640 1 --2- 172.21.15.28:0/2896612429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9838105450 0x7f983819ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.538+0000 7f983e600640 1 --2- 172.21.15.28:0/2896612429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9838104260 0x7f983819e9f0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.538+0000 7f983e600640 1 -- 172.21.15.28:0/2896612429 >> 172.21.15.28:0/2896612429 conn(0x7f98380ff7f0 msgr2=0x7f9838078450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.538+0000 7f983e600640 1 -- 172.21.15.28:0/2896612429 shutdown_connections 2024-04-24T08:07:52.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.538+0000 7f983e600640 1 -- 172.21.15.28:0/2896612429 wait complete. 2024-04-24T08:07:52.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:07:52.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.766+0000 7f9495a00640 1 -- 172.21.15.28:0/331836828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490103980 msgr2=0x7f949010a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.766+0000 7f9495a00640 1 --2- 172.21.15.28:0/331836828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490103980 0x7f949010a2a0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f9488009ca0 tx=0x7f948802a8c0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.766+0000 7f9495a00640 1 -- 172.21.15.28:0/331836828 shutdown_connections 2024-04-24T08:07:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.766+0000 7f9495a00640 1 --2- 172.21.15.28:0/331836828 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490103980 0x7f949010a2a0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.766+0000 7f9495a00640 1 --2- 172.21.15.28:0/331836828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9490102fc0 0x7f9490103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.766+0000 7f9495a00640 1 --2- 172.21.15.28:0/331836828 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9490101e70 0x7f9490102290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.766+0000 7f9495a00640 1 -- 172.21.15.28:0/331836828 >> 172.21.15.28:0/331836828 conn(0x7f94900fd3e0 msgr2=0x7f94900ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.766+0000 7f9495a00640 1 -- 172.21.15.28:0/331836828 shutdown_connections 2024-04-24T08:07:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.766+0000 7f9495a00640 1 -- 172.21.15.28:0/331836828 wait complete. 2024-04-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.767+0000 7f9495a00640 1 Processor -- start 2024-04-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.767+0000 7f9495a00640 1 -- start start 2024-04-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.767+0000 7f9495a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490101e70 0x7f949019a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.767+0000 7f9495a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9490102fc0 0x7f949019a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.767+0000 7f9495a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9490103980 0x7f94901a18b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:52.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.767+0000 7f9495a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f949010db40 con 0x7f9490101e70 2024-04-24T08:07:52.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.767+0000 7f9495a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f949010d9c0 con 0x7f9490103980 2024-04-24T08:07:52.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.767+0000 7f9495a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f949010dcc0 con 0x7f9490102fc0 2024-04-24T08:07:52.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.768+0000 7f948fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490101e70 0x7f949019a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:52.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.768+0000 7f948fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490101e70 0x7f949019a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37168/0 (socket says 172.21.15.28:37168) 2024-04-24T08:07:52.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.768+0000 7f9495000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9490103980 0x7f94901a18b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:52.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.768+0000 7f948fe00640 1 -- 172.21.15.28:0/4141580576 learned_addr learned my addr 172.21.15.28:0/4141580576 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:52.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.768+0000 7f948f400640 1 --2- 172.21.15.28:0/4141580576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9490102fc0 0x7f949019a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:52.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.768+0000 7f948fe00640 1 -- 172.21.15.28:0/4141580576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9490102fc0 msgr2=0x7f949019a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:52.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.768+0000 7f948fe00640 1 --2- 172.21.15.28:0/4141580576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9490102fc0 0x7f949019a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.768+0000 7f948fe00640 1 -- 172.21.15.28:0/4141580576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9490103980 msgr2=0x7f94901a18b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.768+0000 7f948fe00640 1 --2- 172.21.15.28:0/4141580576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9490103980 0x7f94901a18b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.768+0000 7f948fe00640 1 -- 172.21.15.28:0/4141580576 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f94901a1f20 con 0x7f9490101e70 2024-04-24T08:07:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.768+0000 7f948fe00640 1 --2- 172.21.15.28:0/4141580576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490101e70 0x7f949019a300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f948000e9f0 tx=0x7f948000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.769+0000 7f948cc00640 1 -- 172.21.15.28:0/4141580576 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948000cd80 con 0x7f9490101e70 2024-04-24T08:07:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.769+0000 7f948cc00640 1 -- 172.21.15.28:0/4141580576 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9480010910 con 0x7f9490101e70 2024-04-24T08:07:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.769+0000 7f948cc00640 1 -- 172.21.15.28:0/4141580576 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480018990 con 0x7f9490101e70 2024-04-24T08:07:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.769+0000 7f9495a00640 1 -- 172.21.15.28:0/4141580576 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94901a2210 con 0x7f9490101e70 2024-04-24T08:07:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.769+0000 7f9495a00640 1 -- 172.21.15.28:0/4141580576 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94901a2710 con 0x7f9490101e70 2024-04-24T08:07:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.770+0000 7f9495a00640 1 -- 172.21.15.28:0/4141580576 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f94901a2970 con 0x7f9490101e70 2024-04-24T08:07:52.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.772+0000 7f948cc00640 1 -- 172.21.15.28:0/4141580576 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f9480010ac0 con 0x7f9490101e70 2024-04-24T08:07:52.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.774+0000 7f948cc00640 1 --2- 172.21.15.28:0/4141580576 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f94740a3bb0 0x7f94740a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:52.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.774+0000 7f948cc00640 1 -- 172.21.15.28:0/4141580576 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 7979+0+0 (secure 0 0 0) 0x7f94800d0670 con 0x7f9490101e70 2024-04-24T08:07:52.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.774+0000 7f948f400640 1 --2- 172.21.15.28:0/4141580576 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f94740a3bb0 0x7f94740a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.775+0000 7f948f400640 1 --2- 172.21.15.28:0/4141580576 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f94740a3bb0 0x7f94740a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f949019b810 tx=0x7f947c00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.778+0000 7f948cc00640 1 -- 172.21.15.28:0/4141580576 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f948009cab0 con 0x7f9490101e70 2024-04-24T08:07:52.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.979+0000 7f9495a00640 1 -- 172.21.15.28:0/4141580576 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f94901a3120 con 0x7f9490101e70 2024-04-24T08:07:52.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.982+0000 7f948cc00640 1 -- 172.21.15.28:0/4141580576 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) ==== 74+0+22220 (secure 0 0 0) 0x7f94800a0db0 con 0x7f9490101e70 2024-04-24T08:07:52.983 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:07:52.983 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":85,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:07:51.585715+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:07:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.984+0000 7f9495a00640 1 -- 172.21.15.28:0/4141580576 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f94740a3bb0 msgr2=0x7f94740a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.984+0000 7f9495a00640 1 --2- 172.21.15.28:0/4141580576 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f94740a3bb0 0x7f94740a6070 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f949019b810 tx=0x7f947c00b410 comp rx=0 tx=0).stop 2024-04-24T08:07:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.984+0000 7f9495a00640 1 -- 172.21.15.28:0/4141580576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490101e70 msgr2=0x7f949019a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.984+0000 7f9495a00640 1 --2- 172.21.15.28:0/4141580576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490101e70 0x7f949019a300 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f948000e9f0 tx=0x7f948000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.984+0000 7f9495a00640 1 -- 172.21.15.28:0/4141580576 shutdown_connections 2024-04-24T08:07:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.984+0000 7f9495a00640 1 --2- 172.21.15.28:0/4141580576 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f94740a3bb0 0x7f94740a6070 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.984+0000 7f9495a00640 1 --2- 172.21.15.28:0/4141580576 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9490103980 0x7f94901a18b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.984+0000 7f9495a00640 1 --2- 172.21.15.28:0/4141580576 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9490102fc0 0x7f949019a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.984+0000 7f9495a00640 1 --2- 172.21.15.28:0/4141580576 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490101e70 0x7f949019a300 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.984+0000 7f9495a00640 1 -- 172.21.15.28:0/4141580576 >> 172.21.15.28:0/4141580576 conn(0x7f94900fd3e0 msgr2=0x7f9490074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:52.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.985+0000 7f9495a00640 1 -- 172.21.15.28:0/4141580576 shutdown_connections 2024-04-24T08:07:52.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:52.985+0000 7f9495a00640 1 -- 172.21.15.28:0/4141580576 wait complete. 2024-04-24T08:07:53.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-04-24T08:07:53.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.211+0000 7f06af200640 1 -- 172.21.15.28:0/3615416897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06b0104140 msgr2=0x7f06b0104560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.211+0000 7f06af200640 1 --2- 172.21.15.28:0/3615416897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06b0104140 0x7f06b0104560 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f06a0009900 tx=0x7f06a002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:07:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.211+0000 7f06af200640 1 -- 172.21.15.28:0/3615416897 shutdown_connections 2024-04-24T08:07:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.211+0000 7f06af200640 1 --2- 172.21.15.28:0/3615416897 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b0105cf0 0x7f06b010c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.211+0000 7f06af200640 1 --2- 172.21.15.28:0/3615416897 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06b0105330 0x7f06b01057b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.211+0000 7f06af200640 1 --2- 172.21.15.28:0/3615416897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06b0104140 0x7f06b0104560 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.211+0000 7f06af200640 1 -- 172.21.15.28:0/3615416897 >> 172.21.15.28:0/3615416897 conn(0x7f06b00ff6b0 msgr2=0x7f06b0101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.211+0000 7f06af200640 1 -- 172.21.15.28:0/3615416897 shutdown_connections 2024-04-24T08:07:53.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.211+0000 7f06af200640 1 -- 172.21.15.28:0/3615416897 wait complete. 2024-04-24T08:07:53.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.212+0000 7f06af200640 1 Processor -- start 2024-04-24T08:07:53.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.213+0000 7f06af200640 1 -- start start 2024-04-24T08:07:53.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.213+0000 7f06af200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06b0104140 0x7f06b019a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:53.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.213+0000 7f06af200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b0105330 0x7f06b019a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:53.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.213+0000 7f06af200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06b0105cf0 0x7f06b01a19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:53.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.213+0000 7f06af200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f06b010fe20 con 0x7f06b0105330 2024-04-24T08:07:53.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.213+0000 7f06af200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f06b010fca0 con 0x7f06b0105cf0 2024-04-24T08:07:53.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.213+0000 7f06af200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f06b010ffa0 con 0x7f06b0104140 2024-04-24T08:07:53.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.213+0000 7f06ad400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b0105330 0x7f06b019a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.213+0000 7f06ade00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06b0104140 0x7f06b019a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.213+0000 7f06ad400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b0105330 0x7f06b019a980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37194/0 (socket says 172.21.15.28:37194) 2024-04-24T08:07:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.213+0000 7f06ad400640 1 -- 172.21.15.28:0/679589662 learned_addr learned my addr 172.21.15.28:0/679589662 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.213+0000 7f06ae800640 1 --2- 172.21.15.28:0/679589662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06b0105cf0 0x7f06b01a19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.214+0000 7f06ad400640 1 -- 172.21.15.28:0/679589662 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06b0104140 msgr2=0x7f06b019a440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.214+0000 7f06ad400640 1 --2- 172.21.15.28:0/679589662 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06b0104140 0x7f06b019a440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.214+0000 7f06ad400640 1 -- 172.21.15.28:0/679589662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06b0105cf0 msgr2=0x7f06b01a19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.214+0000 7f06ad400640 1 --2- 172.21.15.28:0/679589662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06b0105cf0 0x7f06b01a19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:53.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.214+0000 7f06ad400640 1 -- 172.21.15.28:0/679589662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06b01a2060 con 0x7f06b0105330 2024-04-24T08:07:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.214+0000 7f06ad400640 1 --2- 172.21.15.28:0/679589662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b0105330 0x7f06b019a980 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f069c00e9f0 tx=0x7f069c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.214+0000 7f06a6a00640 1 -- 172.21.15.28:0/679589662 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c00cd80 con 0x7f06b0105330 2024-04-24T08:07:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.214+0000 7f06af200640 1 -- 172.21.15.28:0/679589662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06b01a2350 con 0x7f06b0105330 2024-04-24T08:07:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.215+0000 7f06a6a00640 1 -- 172.21.15.28:0/679589662 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f069c005500 con 0x7f06b0105330 2024-04-24T08:07:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.215+0000 7f06a6a00640 1 -- 172.21.15.28:0/679589662 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f069c010690 con 0x7f06b0105330 2024-04-24T08:07:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.215+0000 7f06af200640 1 -- 172.21.15.28:0/679589662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06b01a28a0 con 0x7f06b0105330 2024-04-24T08:07:53.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.216+0000 7f06af200640 1 -- 172.21.15.28:0/679589662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06b010b460 con 0x7f06b0105330 2024-04-24T08:07:53.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.222+0000 7f06a6a00640 1 -- 172.21.15.28:0/679589662 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f069c0046c0 con 0x7f06b0105330 2024-04-24T08:07:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.224+0000 7f06a6a00640 1 --2- 172.21.15.28:0/679589662 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f06940a3bb0 0x7f06940a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.224+0000 7f06a6a00640 1 -- 172.21.15.28:0/679589662 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f069c0d0950 con 0x7f06b0105330 2024-04-24T08:07:53.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.224+0000 7f06ade00640 1 --2- 172.21.15.28:0/679589662 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f06940a3bb0 0x7f06940a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:53.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.225+0000 7f06ade00640 1 --2- 172.21.15.28:0/679589662 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f06940a3bb0 0x7f06940a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f06a002a9e0 tx=0x7f06a00315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:53.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.227+0000 7f06a6a00640 1 -- 172.21.15.28:0/679589662 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f069c053ff0 con 0x7f06b0105330 2024-04-24T08:07:53.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:53 smithi028 ceph-mon[27786]: osdmap e85: 12 total, 12 up, 12 in 2024-04-24T08:07:53.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:53 smithi028 ceph-mon[27786]: pgmap v316: 97 pgs: 12 creating+peering, 81 unknown, 4 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:53.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:53 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-24T08:07:53.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:53 smithi028 ceph-mon[27786]: fsmap cephfs:1 {0=g=up:creating} 11 up:standby 2024-04-24T08:07:53.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:53 smithi028 ceph-mon[27786]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-24T08:07:53.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:53 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:53.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:53 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:53.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:53 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:53.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:53 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:07:53.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:53 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:53.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:53 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2896612429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:07:53.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:53 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/4141580576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:07:53.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:53 smithi117 ceph-mon[32438]: osdmap e85: 12 total, 12 up, 12 in 2024-04-24T08:07:53.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:53 smithi117 ceph-mon[32438]: pgmap v316: 97 pgs: 12 creating+peering, 81 unknown, 4 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:53.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:53 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-24T08:07:53.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:53 smithi117 ceph-mon[32438]: fsmap cephfs:1 {0=g=up:creating} 11 up:standby 2024-04-24T08:07:53.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:53 smithi117 ceph-mon[32438]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-24T08:07:53.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:53.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:53.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:53.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:07:53.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:53.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:53 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2896612429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:07:53.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:53 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/4141580576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:07:53.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:53 smithi194 ceph-mon[32836]: osdmap e85: 12 total, 12 up, 12 in 2024-04-24T08:07:53.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:53 smithi194 ceph-mon[32836]: pgmap v316: 97 pgs: 12 creating+peering, 81 unknown, 4 active+clean; 577 KiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:07:53.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:53 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-04-24T08:07:53.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:53 smithi194 ceph-mon[32836]: fsmap cephfs:1 {0=g=up:creating} 11 up:standby 2024-04-24T08:07:53.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:53 smithi194 ceph-mon[32836]: daemon mds.g is now active in filesystem cephfs as rank 0 2024-04-24T08:07:53.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:53 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:53.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:53 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:53.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:53 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:07:53.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:53 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:07:53.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:53 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:07:53.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:53 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2896612429' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:07:53.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:53 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/4141580576' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:07:53.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:53.476+0000 7f06af200640 1 -- 172.21.15.28:0/679589662 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7f06b002d620 con 0x7f06b0105330 2024-04-24T08:07:54.039 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 24 08:07:53 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-g[66667]: 2024-04-24T08:07:53.669+0000 7fb141c00700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-04-24T08:07:54.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.083+0000 7f06a6a00640 1 -- 172.21.15.28:0/679589662 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) ==== 118+0+0 (secure 0 0 0) 0x7f069c09d3e0 con 0x7f06b0105330 2024-04-24T08:07:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.086+0000 7f06af200640 1 -- 172.21.15.28:0/679589662 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f06940a3bb0 msgr2=0x7f06940a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.086+0000 7f06af200640 1 --2- 172.21.15.28:0/679589662 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f06940a3bb0 0x7f06940a6070 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f06a002a9e0 tx=0x7f06a00315b0 comp rx=0 tx=0).stop 2024-04-24T08:07:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.086+0000 7f06af200640 1 -- 172.21.15.28:0/679589662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b0105330 msgr2=0x7f06b019a980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.086+0000 7f06af200640 1 --2- 172.21.15.28:0/679589662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b0105330 0x7f06b019a980 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f069c00e9f0 tx=0x7f069c00edf0 comp rx=0 tx=0).stop 2024-04-24T08:07:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.086+0000 7f06af200640 1 -- 172.21.15.28:0/679589662 shutdown_connections 2024-04-24T08:07:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.086+0000 7f06af200640 1 --2- 172.21.15.28:0/679589662 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f06940a3bb0 0x7f06940a6070 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:54.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.086+0000 7f06af200640 1 --2- 172.21.15.28:0/679589662 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f06b0105330 0x7f06b019a980 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.086+0000 7f06af200640 1 --2- 172.21.15.28:0/679589662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06b0105cf0 0x7f06b01a19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.086+0000 7f06af200640 1 --2- 172.21.15.28:0/679589662 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f06b0104140 0x7f06b019a440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.086+0000 7f06af200640 1 -- 172.21.15.28:0/679589662 >> 172.21.15.28:0/679589662 conn(0x7f06b00ff6b0 msgr2=0x7f06b0108630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.087+0000 7f06af200640 1 -- 172.21.15.28:0/679589662 shutdown_connections 2024-04-24T08:07:54.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.087+0000 7f06af200640 1 -- 172.21.15.28:0/679589662 wait complete. 2024-04-24T08:07:54.113 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--size 25000000000'}} 2024-04-24T08:07:54.113 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-04-24T08:07:54.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.327+0000 7f970d200640 1 -- 172.21.15.28:0/1739147224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9708102990 msgr2=0x7f9708104d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:54.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.327+0000 7f970d200640 1 --2- 172.21.15.28:0/1739147224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9708102990 0x7f9708104d80 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f96f8009900 tx=0x7f96f802a410 comp rx=0 tx=0).stop 2024-04-24T08:07:54.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.328+0000 7f970d200640 1 -- 172.21.15.28:0/1739147224 shutdown_connections 2024-04-24T08:07:54.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.328+0000 7f970d200640 1 --2- 172.21.15.28:0/1739147224 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9708107bf0 0x7f970810a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:54.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.328+0000 7f970d200640 1 --2- 172.21.15.28:0/1739147224 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97081052c0 0x7f97081076b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:54.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.328+0000 7f970d200640 1 --2- 172.21.15.28:0/1739147224 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9708102990 0x7f9708104d80 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:54.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.328+0000 7f970d200640 1 -- 172.21.15.28:0/1739147224 >> 172.21.15.28:0/1739147224 conn(0x7f97080fc3a0 msgr2=0x7f97080fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:54.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.328+0000 7f970d200640 1 -- 172.21.15.28:0/1739147224 shutdown_connections 2024-04-24T08:07:54.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.328+0000 7f970d200640 1 -- 172.21.15.28:0/1739147224 wait complete. 2024-04-24T08:07:54.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.329+0000 7f970d200640 1 Processor -- start 2024-04-24T08:07:54.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.329+0000 7f970d200640 1 -- start start 2024-04-24T08:07:54.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.330+0000 7f970d200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9708102990 0x7f970819a440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:54.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.330+0000 7f970d200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97081052c0 0x7f970819a980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:54.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.330+0000 7f970d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9708107bf0 0x7f97081a19f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:54.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.330+0000 7f970d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f970810dbf0 con 0x7f9708107bf0 2024-04-24T08:07:54.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.330+0000 7f970d200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f970810da70 con 0x7f97081052c0 2024-04-24T08:07:54.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.330+0000 7f970d200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f970810dd70 con 0x7f9708102990 2024-04-24T08:07:54.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.330+0000 7f9707400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9708102990 0x7f970819a440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:54.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.330+0000 7f9707400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9708102990 0x7f970819a440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:52426/0 (socket says 172.21.15.28:52426) 2024-04-24T08:07:54.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.330+0000 7f9707400640 1 -- 172.21.15.28:0/2198823505 learned_addr learned my addr 172.21.15.28:0/2198823505 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:54.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.330+0000 7f96ffe00640 1 --2- 172.21.15.28:0/2198823505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97081052c0 0x7f970819a980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:54.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.330+0000 7f9707e00640 1 --2- 172.21.15.28:0/2198823505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9708107bf0 0x7f97081a19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:54.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.330+0000 7f9707400640 1 -- 172.21.15.28:0/2198823505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97081052c0 msgr2=0x7f970819a980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:54.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.330+0000 7f9707400640 1 --2- 172.21.15.28:0/2198823505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97081052c0 0x7f970819a980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:54.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.330+0000 7f9707400640 1 -- 172.21.15.28:0/2198823505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9708107bf0 msgr2=0x7f97081a19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:54.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.330+0000 7f9707400640 1 --2- 172.21.15.28:0/2198823505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9708107bf0 0x7f97081a19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:54.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.330+0000 7f9707400640 1 -- 172.21.15.28:0/2198823505 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f97081a2060 con 0x7f9708102990 2024-04-24T08:07:54.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.330+0000 7f9707e00640 1 --2- 172.21.15.28:0/2198823505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9708107bf0 0x7f97081a19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:07:54.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.331+0000 7f9707400640 1 --2- 172.21.15.28:0/2198823505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9708102990 0x7f970819a440 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f96f8031440 tx=0x7f96f8032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:54.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.331+0000 7f9704c00640 1 -- 172.21.15.28:0/2198823505 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f803f030 con 0x7f9708102990 2024-04-24T08:07:54.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.331+0000 7f9704c00640 1 -- 172.21.15.28:0/2198823505 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96f8003ce0 con 0x7f9708102990 2024-04-24T08:07:54.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.331+0000 7f970d200640 1 -- 172.21.15.28:0/2198823505 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f97081a22f0 con 0x7f9708102990 2024-04-24T08:07:54.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.331+0000 7f9704c00640 1 -- 172.21.15.28:0/2198823505 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f8035070 con 0x7f9708102990 2024-04-24T08:07:54.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.331+0000 7f970d200640 1 -- 172.21.15.28:0/2198823505 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97081a2830 con 0x7f9708102990 2024-04-24T08:07:54.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.333+0000 7f970d200640 1 -- 172.21.15.28:0/2198823505 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9708109740 con 0x7f9708102990 2024-04-24T08:07:54.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.334+0000 7f9704c00640 1 -- 172.21.15.28:0/2198823505 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f96f8042030 con 0x7f9708102990 2024-04-24T08:07:54.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.336+0000 7f9704c00640 1 --2- 172.21.15.28:0/2198823505 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f96ec0a3c30 0x7f96ec0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:54.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.336+0000 7f9704c00640 1 -- 172.21.15.28:0/2198823505 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f96f80f0810 con 0x7f9708102990 2024-04-24T08:07:54.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.336+0000 7f96ffe00640 1 --2- 172.21.15.28:0/2198823505 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f96ec0a3c30 0x7f96ec0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:54.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.337+0000 7f96ffe00640 1 --2- 172.21.15.28:0/2198823505 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f96ec0a3c30 0x7f96ec0a60f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f970819b950 tx=0x7f96f0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:54.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.340+0000 7f9704c00640 1 -- 172.21.15.28:0/2198823505 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f96f80bd2a0 con 0x7f9708102990 2024-04-24T08:07:54.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:54 smithi117 ceph-mon[32438]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-24T08:07:54.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:54 smithi117 ceph-mon[32438]: Cluster is now healthy 2024-04-24T08:07:54.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:54 smithi117 ceph-mon[32438]: osdmap e86: 12 total, 12 up, 12 in 2024-04-24T08:07:54.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:54 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] up:active 2024-04-24T08:07:54.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:54 smithi117 ceph-mon[32438]: fsmap cephfs:1 {0=g=up:active} 11 up:standby 2024-04-24T08:07:54.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:54 smithi117 ceph-mon[32438]: fsmap cephfs:1 {0=g=up:active} 11 up:standby 2024-04-24T08:07:54.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:54 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/679589662' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-24T08:07:54.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:54 smithi194 ceph-mon[32836]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-24T08:07:54.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:54 smithi194 ceph-mon[32836]: Cluster is now healthy 2024-04-24T08:07:54.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:54 smithi194 ceph-mon[32836]: osdmap e86: 12 total, 12 up, 12 in 2024-04-24T08:07:54.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:54 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] up:active 2024-04-24T08:07:54.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:54 smithi194 ceph-mon[32836]: fsmap cephfs:1 {0=g=up:active} 11 up:standby 2024-04-24T08:07:54.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:54 smithi194 ceph-mon[32836]: fsmap cephfs:1 {0=g=up:active} 11 up:standby 2024-04-24T08:07:54.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:54 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/679589662' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-24T08:07:54.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:54 smithi028 ceph-mon[27786]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-04-24T08:07:54.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:54 smithi028 ceph-mon[27786]: Cluster is now healthy 2024-04-24T08:07:54.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:54 smithi028 ceph-mon[27786]: osdmap e86: 12 total, 12 up, 12 in 2024-04-24T08:07:54.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:54 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] up:active 2024-04-24T08:07:54.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:54 smithi028 ceph-mon[27786]: fsmap cephfs:1 {0=g=up:active} 11 up:standby 2024-04-24T08:07:54.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:54 smithi028 ceph-mon[27786]: fsmap cephfs:1 {0=g=up:active} 11 up:standby 2024-04-24T08:07:54.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:54 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/679589662' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-04-24T08:07:54.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:54.593+0000 7f970d200640 1 -- 172.21.15.28:0/2198823505 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f9708062c00 con 0x7f9708102990 2024-04-24T08:07:55.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.099+0000 7f9704c00640 1 -- 172.21.15.28:0/2198823505 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v18) ==== 105+0+0 (secure 0 0 0) 0x7f96f80bcba0 con 0x7f9708102990 2024-04-24T08:07:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.104+0000 7f970d200640 1 -- 172.21.15.28:0/2198823505 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f96ec0a3c30 msgr2=0x7f96ec0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.104+0000 7f970d200640 1 --2- 172.21.15.28:0/2198823505 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f96ec0a3c30 0x7f96ec0a60f0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f970819b950 tx=0x7f96f0005f90 comp rx=0 tx=0).stop 2024-04-24T08:07:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.104+0000 7f970d200640 1 -- 172.21.15.28:0/2198823505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9708102990 msgr2=0x7f970819a440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.104+0000 7f970d200640 1 --2- 172.21.15.28:0/2198823505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9708102990 0x7f970819a440 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f96f8031440 tx=0x7f96f8032ce0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.104+0000 7f970d200640 1 -- 172.21.15.28:0/2198823505 shutdown_connections 2024-04-24T08:07:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.104+0000 7f970d200640 1 --2- 172.21.15.28:0/2198823505 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f96ec0a3c30 0x7f96ec0a60f0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.104+0000 7f970d200640 1 --2- 172.21.15.28:0/2198823505 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9708107bf0 0x7f97081a19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.104+0000 7f970d200640 1 --2- 172.21.15.28:0/2198823505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97081052c0 0x7f970819a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.104+0000 7f970d200640 1 --2- 172.21.15.28:0/2198823505 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9708102990 0x7f970819a440 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.104+0000 7f970d200640 1 -- 172.21.15.28:0/2198823505 >> 172.21.15.28:0/2198823505 conn(0x7f97080fc3a0 msgr2=0x7f9708102080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.105+0000 7f970d200640 1 -- 172.21.15.28:0/2198823505 shutdown_connections 2024-04-24T08:07:55.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.105+0000 7f970d200640 1 -- 172.21.15.28:0/2198823505 wait complete. 2024-04-24T08:07:55.133 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-04-24T08:07:55.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.337+0000 7f897e200640 1 -- 172.21.15.28:0/4264335947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8978109dd0 msgr2=0x7f897810c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:55.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.337+0000 7f897e200640 1 --2- 172.21.15.28:0/4264335947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8978109dd0 0x7f897810c260 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f8970009ca0 tx=0x7f897002a8c0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.337+0000 7f897e200640 1 -- 172.21.15.28:0/4264335947 shutdown_connections 2024-04-24T08:07:55.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.337+0000 7f897e200640 1 --2- 172.21.15.28:0/4264335947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8978109dd0 0x7f897810c260 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.337+0000 7f897e200640 1 --2- 172.21.15.28:0/4264335947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8978074a60 0x7f8978074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.337+0000 7f897e200640 1 --2- 172.21.15.28:0/4264335947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8978076040 0x7f8978074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.337+0000 7f897e200640 1 -- 172.21.15.28:0/4264335947 >> 172.21.15.28:0/4264335947 conn(0x7f89780fd4a0 msgr2=0x7f89780ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:55.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.337+0000 7f897e200640 1 -- 172.21.15.28:0/4264335947 shutdown_connections 2024-04-24T08:07:55.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.337+0000 7f897e200640 1 -- 172.21.15.28:0/4264335947 wait complete. 2024-04-24T08:07:55.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.338+0000 7f897e200640 1 Processor -- start 2024-04-24T08:07:55.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.339+0000 7f897e200640 1 -- start start 2024-04-24T08:07:55.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.339+0000 7f897e200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8978074a60 0x7f897819a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:55.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.339+0000 7f897e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8978076040 0x7f897819a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:55.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.339+0000 7f897e200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8978109dd0 0x7f89781a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:55.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.339+0000 7f897e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f897810fde0 con 0x7f8978076040 2024-04-24T08:07:55.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.339+0000 7f897e200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f897810fc60 con 0x7f8978074a60 2024-04-24T08:07:55.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.339+0000 7f897e200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f897810ff60 con 0x7f8978109dd0 2024-04-24T08:07:55.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.339+0000 7f8977e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8978076040 0x7f897819a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:55.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.339+0000 7f8977e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8978076040 0x7f897819a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37234/0 (socket says 172.21.15.28:37234) 2024-04-24T08:07:55.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.339+0000 7f8977e00640 1 -- 172.21.15.28:0/2753707017 learned_addr learned my addr 172.21.15.28:0/2753707017 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:55.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.339+0000 7f897ce00640 1 --2- 172.21.15.28:0/2753707017 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8978074a60 0x7f897819a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:55.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.339+0000 7f897d800640 1 --2- 172.21.15.28:0/2753707017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8978109dd0 0x7f89781a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:55.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.340+0000 7f8977e00640 1 -- 172.21.15.28:0/2753707017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8978109dd0 msgr2=0x7f89781a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:55.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.340+0000 7f8977e00640 1 --2- 172.21.15.28:0/2753707017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8978109dd0 0x7f89781a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.340+0000 7f8977e00640 1 -- 172.21.15.28:0/2753707017 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8978074a60 msgr2=0x7f897819a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:55.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.340+0000 7f8977e00640 1 --2- 172.21.15.28:0/2753707017 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8978074a60 0x7f897819a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:55.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.340+0000 7f8977e00640 1 -- 172.21.15.28:0/2753707017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89781a1fd0 con 0x7f8978076040 2024-04-24T08:07:55.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.340+0000 7f8977e00640 1 --2- 172.21.15.28:0/2753707017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8978076040 0x7f897819a8f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f896800e9f0 tx=0x7f896800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:55.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.340+0000 7f8975600640 1 -- 172.21.15.28:0/2753707017 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f896800cd80 con 0x7f8978076040 2024-04-24T08:07:55.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.340+0000 7f8975600640 1 -- 172.21.15.28:0/2753707017 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8968010910 con 0x7f8978076040 2024-04-24T08:07:55.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.340+0000 7f897e200640 1 -- 172.21.15.28:0/2753707017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89781a22c0 con 0x7f8978076040 2024-04-24T08:07:55.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.340+0000 7f8975600640 1 -- 172.21.15.28:0/2753707017 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8968018990 con 0x7f8978076040 2024-04-24T08:07:55.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.341+0000 7f897e200640 1 -- 172.21.15.28:0/2753707017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89781a2810 con 0x7f8978076040 2024-04-24T08:07:55.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.342+0000 7f897e200640 1 -- 172.21.15.28:0/2753707017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8978108b00 con 0x7f8978076040 2024-04-24T08:07:55.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.344+0000 7f8975600640 1 -- 172.21.15.28:0/2753707017 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f8968010ac0 con 0x7f8978076040 2024-04-24T08:07:55.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.345+0000 7f8975600640 1 --2- 172.21.15.28:0/2753707017 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f89600a3bb0 0x7f89600a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:55.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.346+0000 7f8975600640 1 -- 172.21.15.28:0/2753707017 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f89680d0670 con 0x7f8978076040 2024-04-24T08:07:55.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.346+0000 7f897ce00640 1 --2- 172.21.15.28:0/2753707017 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f89600a3bb0 0x7f89600a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:55.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.346+0000 7f897ce00640 1 --2- 172.21.15.28:0/2753707017 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f89600a3bb0 0x7f89600a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f896c009bd0 tx=0x7f896c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:55.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.350+0000 7f8975600640 1 -- 172.21.15.28:0/2753707017 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f896809cab0 con 0x7f8978076040 2024-04-24T08:07:55.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:55 smithi117 ceph-mon[32438]: pgmap v318: 97 pgs: 12 creating+peering, 40 unknown, 45 active+clean; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-24T08:07:55.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:55 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/679589662' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-24T08:07:55.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:55 smithi117 ceph-mon[32438]: fsmap cephfs:1 {0=g=up:active} 11 up:standby 2024-04-24T08:07:55.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:55 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2198823505' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-24T08:07:55.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:55 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-24T08:07:55.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:55 smithi194 ceph-mon[32836]: pgmap v318: 97 pgs: 12 creating+peering, 40 unknown, 45 active+clean; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-24T08:07:55.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:55 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/679589662' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-24T08:07:55.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:55 smithi194 ceph-mon[32836]: fsmap cephfs:1 {0=g=up:active} 11 up:standby 2024-04-24T08:07:55.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:55 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2198823505' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-24T08:07:55.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:55 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-24T08:07:55.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:55 smithi028 ceph-mon[27786]: pgmap v318: 97 pgs: 12 creating+peering, 40 unknown, 45 active+clean; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 4 op/s 2024-04-24T08:07:55.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:55 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/679589662' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-04-24T08:07:55.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:55 smithi028 ceph-mon[27786]: fsmap cephfs:1 {0=g=up:active} 11 up:standby 2024-04-24T08:07:55.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:55 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2198823505' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-24T08:07:55.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:55 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-04-24T08:07:55.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:55.600+0000 7f897e200640 1 -- 172.21.15.28:0/2753707017 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f897802d620 con 0x7f8978076040 2024-04-24T08:07:56.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.114+0000 7f8975600640 1 -- 172.21.15.28:0/2753707017 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) ==== 121+0+0 (secure 0 0 0) 0x7f89680a0db0 con 0x7f8978076040 2024-04-24T08:07:56.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.118+0000 7f897e200640 1 -- 172.21.15.28:0/2753707017 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f89600a3bb0 msgr2=0x7f89600a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.118+0000 7f897e200640 1 --2- 172.21.15.28:0/2753707017 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f89600a3bb0 0x7f89600a6070 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f896c009bd0 tx=0x7f896c005f90 comp rx=0 tx=0).stop 2024-04-24T08:07:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.118+0000 7f897e200640 1 -- 172.21.15.28:0/2753707017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8978076040 msgr2=0x7f897819a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.118+0000 7f897e200640 1 --2- 172.21.15.28:0/2753707017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8978076040 0x7f897819a8f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f896800e9f0 tx=0x7f896800eeb0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.118+0000 7f897e200640 1 -- 172.21.15.28:0/2753707017 shutdown_connections 2024-04-24T08:07:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.118+0000 7f897e200640 1 --2- 172.21.15.28:0/2753707017 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f89600a3bb0 0x7f89600a6070 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.118+0000 7f897e200640 1 --2- 172.21.15.28:0/2753707017 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8978076040 0x7f897819a8f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.118+0000 7f897e200640 1 --2- 172.21.15.28:0/2753707017 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8978109dd0 0x7f89781a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.118+0000 7f897e200640 1 --2- 172.21.15.28:0/2753707017 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8978074a60 0x7f897819a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.118+0000 7f897e200640 1 -- 172.21.15.28:0/2753707017 >> 172.21.15.28:0/2753707017 conn(0x7f89780fd4a0 msgr2=0x7f8978105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.118+0000 7f897e200640 1 -- 172.21.15.28:0/2753707017 shutdown_connections 2024-04-24T08:07:56.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.119+0000 7f897e200640 1 -- 172.21.15.28:0/2753707017 wait complete. 2024-04-24T08:07:56.153 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-04-24T08:07:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.343+0000 7fcef9e00640 1 -- 172.21.15.28:0/3074338484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4100780 msgr2=0x7fcef4100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.343+0000 7fcef9e00640 1 --2- 172.21.15.28:0/3074338484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4100780 0x7fcef4100ba0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fcee8009900 tx=0x7fcee802a410 comp rx=0 tx=0).stop 2024-04-24T08:07:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.343+0000 7fcef9e00640 1 -- 172.21.15.28:0/3074338484 shutdown_connections 2024-04-24T08:07:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.343+0000 7fcef9e00640 1 --2- 172.21.15.28:0/3074338484 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcef4101a80 0x7fcef4101f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.343+0000 7fcef9e00640 1 --2- 172.21.15.28:0/3074338484 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcef41010e0 0x7fcef4101540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.343+0000 7fcef9e00640 1 --2- 172.21.15.28:0/3074338484 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4100780 0x7fcef4100ba0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.343+0000 7fcef9e00640 1 -- 172.21.15.28:0/3074338484 >> 172.21.15.28:0/3074338484 conn(0x7fcef40fc3a0 msgr2=0x7fcef40fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:56.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.343+0000 7fcef9e00640 1 -- 172.21.15.28:0/3074338484 shutdown_connections 2024-04-24T08:07:56.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.343+0000 7fcef9e00640 1 -- 172.21.15.28:0/3074338484 wait complete. 2024-04-24T08:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.344+0000 7fcef9e00640 1 Processor -- start 2024-04-24T08:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.344+0000 7fcef9e00640 1 -- start start 2024-04-24T08:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.345+0000 7fcef9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcef4100780 0x7fcef419a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.345+0000 7fcef9e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcef41010e0 0x7fcef419a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.345+0000 7fcef9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4101a80 0x7fcef41a19b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.345+0000 7fcef9e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcef410fe00 con 0x7fcef4101a80 2024-04-24T08:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.345+0000 7fcef9e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcef410fc80 con 0x7fcef41010e0 2024-04-24T08:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.345+0000 7fcef9e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcef410ff80 con 0x7fcef4100780 2024-04-24T08:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.345+0000 7fcef9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4101a80 0x7fcef41a19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.345+0000 7fcef9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4101a80 0x7fcef41a19b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37256/0 (socket says 172.21.15.28:37256) 2024-04-24T08:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.345+0000 7fcef9400640 1 -- 172.21.15.28:0/279509587 learned_addr learned my addr 172.21.15.28:0/279509587 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:56.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.345+0000 7fcef8a00640 1 --2- 172.21.15.28:0/279509587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcef4100780 0x7fcef419a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.345+0000 7fcef3e00640 1 --2- 172.21.15.28:0/279509587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcef41010e0 0x7fcef419a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.345+0000 7fcef9400640 1 -- 172.21.15.28:0/279509587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcef4100780 msgr2=0x7fcef419a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.345+0000 7fcef9400640 1 --2- 172.21.15.28:0/279509587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcef4100780 0x7fcef419a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.345+0000 7fcef9400640 1 -- 172.21.15.28:0/279509587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcef41010e0 msgr2=0x7fcef419a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.345+0000 7fcef9400640 1 --2- 172.21.15.28:0/279509587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcef41010e0 0x7fcef419a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.345+0000 7fcef9400640 1 -- 172.21.15.28:0/279509587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcef41a2020 con 0x7fcef4101a80 2024-04-24T08:07:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.346+0000 7fcef9400640 1 --2- 172.21.15.28:0/279509587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4101a80 0x7fcef41a19b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fceec00eb10 tx=0x7fceec00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.346+0000 7fcef1600640 1 -- 172.21.15.28:0/279509587 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec004b20 con 0x7fcef4101a80 2024-04-24T08:07:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.346+0000 7fcef1600640 1 -- 172.21.15.28:0/279509587 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fceec004cc0 con 0x7fcef4101a80 2024-04-24T08:07:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.346+0000 7fcef1600640 1 -- 172.21.15.28:0/279509587 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceec010760 con 0x7fcef4101a80 2024-04-24T08:07:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.346+0000 7fcef9e00640 1 -- 172.21.15.28:0/279509587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcef41a2310 con 0x7fcef4101a80 2024-04-24T08:07:56.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.346+0000 7fcef9e00640 1 -- 172.21.15.28:0/279509587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcef41a2810 con 0x7fcef4101a80 2024-04-24T08:07:56.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.347+0000 7fcef9e00640 1 -- 172.21.15.28:0/279509587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcef410b4d0 con 0x7fcef4101a80 2024-04-24T08:07:56.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.351+0000 7fcef1600640 1 -- 172.21.15.28:0/279509587 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7fceec005000 con 0x7fcef4101a80 2024-04-24T08:07:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.353+0000 7fcef1600640 1 --2- 172.21.15.28:0/279509587 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fcedc0a3c30 0x7fcedc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.353+0000 7fcef1600640 1 -- 172.21.15.28:0/279509587 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fceec0d0b10 con 0x7fcef4101a80 2024-04-24T08:07:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.353+0000 7fcef8a00640 1 --2- 172.21.15.28:0/279509587 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fcedc0a3c30 0x7fcedc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:56.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.353+0000 7fcef1600640 1 -- 172.21.15.28:0/279509587 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fceec053ff0 con 0x7fcef4101a80 2024-04-24T08:07:56.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.354+0000 7fcef8a00640 1 --2- 172.21.15.28:0/279509587 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fcedc0a3c30 0x7fcedc0a60f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcee8031440 tx=0x7fcee80315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:56.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:56 smithi117 ceph-mon[32438]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-24T08:07:56.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:56 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-24T08:07:56.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:56 smithi117 ceph-mon[32438]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-24T08:07:56.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:56 smithi117 ceph-mon[32438]: fsmap cephfs:1 {0=g=up:active} 11 up:standby 2024-04-24T08:07:56.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:56 smithi117 ceph-mon[32438]: fsmap cephfs:2 {0=g=up:active,1=b=up:creating} 10 up:standby 2024-04-24T08:07:56.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:56 smithi117 ceph-mon[32438]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-24T08:07:56.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:56 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2753707017' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-24T08:07:56.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:56 smithi194 ceph-mon[32836]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-24T08:07:56.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:56 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-24T08:07:56.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:56 smithi194 ceph-mon[32836]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-24T08:07:56.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:56 smithi194 ceph-mon[32836]: fsmap cephfs:1 {0=g=up:active} 11 up:standby 2024-04-24T08:07:56.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:56 smithi194 ceph-mon[32836]: fsmap cephfs:2 {0=g=up:active,1=b=up:creating} 10 up:standby 2024-04-24T08:07:56.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:56 smithi194 ceph-mon[32836]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-24T08:07:56.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:56 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2753707017' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-24T08:07:56.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:56 smithi028 ceph-mon[27786]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-04-24T08:07:56.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:56 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-04-24T08:07:56.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:56 smithi028 ceph-mon[27786]: daemon mds.b assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-04-24T08:07:56.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:56 smithi028 ceph-mon[27786]: fsmap cephfs:1 {0=g=up:active} 11 up:standby 2024-04-24T08:07:56.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:56 smithi028 ceph-mon[27786]: fsmap cephfs:2 {0=g=up:active,1=b=up:creating} 10 up:standby 2024-04-24T08:07:56.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:56 smithi028 ceph-mon[27786]: daemon mds.b is now active in filesystem cephfs as rank 1 2024-04-24T08:07:56.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:56 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2753707017' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-04-24T08:07:56.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:56.605+0000 7fcef9e00640 1 -- 172.21.15.28:0/279509587 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7fcef402d620 con 0x7fcef4101a80 2024-04-24T08:07:57.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.126+0000 7fcef1600640 1 -- 172.21.15.28:0/279509587 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) ==== 115+0+0 (secure 0 0 0) 0x7fceec09d5a0 con 0x7fcef4101a80 2024-04-24T08:07:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.129+0000 7fcef9e00640 1 -- 172.21.15.28:0/279509587 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fcedc0a3c30 msgr2=0x7fcedc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.130+0000 7fcef9e00640 1 --2- 172.21.15.28:0/279509587 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fcedc0a3c30 0x7fcedc0a60f0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fcee8031440 tx=0x7fcee80315b0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.130+0000 7fcef9e00640 1 -- 172.21.15.28:0/279509587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4101a80 msgr2=0x7fcef41a19b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.130+0000 7fcef9e00640 1 --2- 172.21.15.28:0/279509587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4101a80 0x7fcef41a19b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fceec00eb10 tx=0x7fceec00c2d0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.130+0000 7fcef9e00640 1 -- 172.21.15.28:0/279509587 shutdown_connections 2024-04-24T08:07:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.130+0000 7fcef9e00640 1 --2- 172.21.15.28:0/279509587 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fcedc0a3c30 0x7fcedc0a60f0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.130+0000 7fcef9e00640 1 --2- 172.21.15.28:0/279509587 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcef4101a80 0x7fcef41a19b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.130+0000 7fcef9e00640 1 --2- 172.21.15.28:0/279509587 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcef41010e0 0x7fcef419a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.130+0000 7fcef9e00640 1 --2- 172.21.15.28:0/279509587 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcef4100780 0x7fcef419a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.130+0000 7fcef9e00640 1 -- 172.21.15.28:0/279509587 >> 172.21.15.28:0/279509587 conn(0x7fcef40fc3a0 msgr2=0x7fcef40fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:57.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.130+0000 7fcef9e00640 1 -- 172.21.15.28:0/279509587 shutdown_connections 2024-04-24T08:07:57.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.130+0000 7fcef9e00640 1 -- 172.21.15.28:0/279509587 wait complete. 2024-04-24T08:07:57.151 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--size 25000000000'} subvols for filesystem 'cephfs' 2024-04-24T08:07:57.151 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-04-24T08:07:57.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.351+0000 7f770b200640 1 -- 172.21.15.28:0/2005053544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c0749e0 msgr2=0x7f770c074e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.351+0000 7f770b200640 1 --2- 172.21.15.28:0/2005053544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c0749e0 0x7f770c074e40 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f76f8009900 tx=0x7f76f802a480 comp rx=0 tx=0).stop 2024-04-24T08:07:57.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.351+0000 7f770b200640 1 -- 172.21.15.28:0/2005053544 shutdown_connections 2024-04-24T08:07:57.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.351+0000 7f770b200640 1 --2- 172.21.15.28:0/2005053544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c0749e0 0x7f770c074e40 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.351+0000 7f770b200640 1 --2- 172.21.15.28:0/2005053544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f770c109de0 0x7f770c10c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.351+0000 7f770b200640 1 --2- 172.21.15.28:0/2005053544 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f770c075fc0 0x7f770c074410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.351+0000 7f770b200640 1 -- 172.21.15.28:0/2005053544 >> 172.21.15.28:0/2005053544 conn(0x7f770c0fd4a0 msgr2=0x7f770c0ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:57.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.351+0000 7f770b200640 1 -- 172.21.15.28:0/2005053544 shutdown_connections 2024-04-24T08:07:57.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.351+0000 7f770b200640 1 -- 172.21.15.28:0/2005053544 wait complete. 2024-04-24T08:07:57.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.352+0000 7f770b200640 1 Processor -- start 2024-04-24T08:07:57.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.352+0000 7f770b200640 1 -- start start 2024-04-24T08:07:57.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.352+0000 7f770b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f770c0749e0 0x7f770c19a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.352+0000 7f770b200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f770c075fc0 0x7f770c19a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.352+0000 7f770b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c109de0 0x7f770c1a19e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.352+0000 7f770b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f770c10fe20 con 0x7f770c109de0 2024-04-24T08:07:57.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.352+0000 7f770b200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f770c10fca0 con 0x7f770c075fc0 2024-04-24T08:07:57.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.352+0000 7f770b200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f770c10ffa0 con 0x7f770c0749e0 2024-04-24T08:07:57.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.353+0000 7f770a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c109de0 0x7f770c1a19e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.353+0000 7f770a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c109de0 0x7f770c1a19e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37270/0 (socket says 172.21.15.28:37270) 2024-04-24T08:07:57.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.353+0000 7f770a800640 1 -- 172.21.15.28:0/3521700686 learned_addr learned my addr 172.21.15.28:0/3521700686 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:57.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.353+0000 7f7709e00640 1 --2- 172.21.15.28:0/3521700686 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f770c0749e0 0x7f770c19a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.353+0000 7f7709400640 1 --2- 172.21.15.28:0/3521700686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f770c075fc0 0x7f770c19a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.353+0000 7f770a800640 1 -- 172.21.15.28:0/3521700686 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f770c0749e0 msgr2=0x7f770c19a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.353+0000 7f770a800640 1 --2- 172.21.15.28:0/3521700686 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f770c0749e0 0x7f770c19a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.353+0000 7f770a800640 1 -- 172.21.15.28:0/3521700686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f770c075fc0 msgr2=0x7f770c19a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.353+0000 7f770a800640 1 --2- 172.21.15.28:0/3521700686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f770c075fc0 0x7f770c19a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.353+0000 7f770a800640 1 -- 172.21.15.28:0/3521700686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f770c1a2050 con 0x7f770c109de0 2024-04-24T08:07:57.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.353+0000 7f770a800640 1 --2- 172.21.15.28:0/3521700686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c109de0 0x7f770c1a19e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f770400efd0 tx=0x7f770400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:57.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.353+0000 7f7702a00640 1 -- 172.21.15.28:0/3521700686 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7704003b70 con 0x7f770c109de0 2024-04-24T08:07:57.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.354+0000 7f7702a00640 1 -- 172.21.15.28:0/3521700686 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7704003d10 con 0x7f770c109de0 2024-04-24T08:07:57.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.354+0000 7f770b200640 1 -- 172.21.15.28:0/3521700686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f770c1a2340 con 0x7f770c109de0 2024-04-24T08:07:57.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.354+0000 7f7702a00640 1 -- 172.21.15.28:0/3521700686 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f77040109f0 con 0x7f770c109de0 2024-04-24T08:07:57.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.354+0000 7f770b200640 1 -- 172.21.15.28:0/3521700686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f770c1a2860 con 0x7f770c109de0 2024-04-24T08:07:57.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.355+0000 7f770b200640 1 -- 172.21.15.28:0/3521700686 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f770c108ba0 con 0x7f770c109de0 2024-04-24T08:07:57.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.357+0000 7f7702a00640 1 -- 172.21.15.28:0/3521700686 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f7704004020 con 0x7f770c109de0 2024-04-24T08:07:57.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.359+0000 7f7702a00640 1 --2- 172.21.15.28:0/3521700686 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f76f00a3c30 0x7f76f00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.359+0000 7f7702a00640 1 -- 172.21.15.28:0/3521700686 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f77040d06f0 con 0x7f770c109de0 2024-04-24T08:07:57.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.359+0000 7f7709e00640 1 --2- 172.21.15.28:0/3521700686 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f76f00a3c30 0x7f76f00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.360+0000 7f7709e00640 1 --2- 172.21.15.28:0/3521700686 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f76f00a3c30 0x7f76f00a60f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009bd0 tx=0x7f76fc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:57.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.364+0000 7f7702a00640 1 -- 172.21.15.28:0/3521700686 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f770409d180 con 0x7f770c109de0 2024-04-24T08:07:57.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:57 smithi117 ceph-mon[32438]: pgmap v319: 97 pgs: 2 creating+peering, 95 active+clean; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 8 op/s 2024-04-24T08:07:57.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:57 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2753707017' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-24T08:07:57.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:57 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] up:active 2024-04-24T08:07:57.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:57 smithi117 ceph-mon[32438]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-24T08:07:57.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:57 smithi117 ceph-mon[32438]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-24T08:07:57.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:57 smithi117 ceph-mon[32438]: Cluster is now healthy 2024-04-24T08:07:57.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:57 smithi117 ceph-mon[32438]: fsmap cephfs:2 {0=g=up:active,1=b=up:active} 10 up:standby 2024-04-24T08:07:57.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:57 smithi117 ceph-mon[32438]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2024-04-24T08:07:57.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:57 smithi117 ceph-mon[32438]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-24T08:07:57.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:57 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/279509587' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-24T08:07:57.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:57 smithi194 ceph-mon[32836]: pgmap v319: 97 pgs: 2 creating+peering, 95 active+clean; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 8 op/s 2024-04-24T08:07:57.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:57 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2753707017' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-24T08:07:57.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:57 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] up:active 2024-04-24T08:07:57.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:57 smithi194 ceph-mon[32836]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-24T08:07:57.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:57 smithi194 ceph-mon[32836]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-24T08:07:57.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:57 smithi194 ceph-mon[32836]: Cluster is now healthy 2024-04-24T08:07:57.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:57 smithi194 ceph-mon[32836]: fsmap cephfs:2 {0=g=up:active,1=b=up:active} 10 up:standby 2024-04-24T08:07:57.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:57 smithi194 ceph-mon[32836]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2024-04-24T08:07:57.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:57 smithi194 ceph-mon[32836]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-24T08:07:57.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:57 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/279509587' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-24T08:07:57.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:57 smithi028 ceph-mon[27786]: pgmap v319: 97 pgs: 2 creating+peering, 95 active+clean; 578 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s wr, 8 op/s 2024-04-24T08:07:57.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:57 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2753707017' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-04-24T08:07:57.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:57 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] up:active 2024-04-24T08:07:57.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:57 smithi028 ceph-mon[27786]: daemon mds.i assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-04-24T08:07:57.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:57 smithi028 ceph-mon[27786]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-04-24T08:07:57.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:57 smithi028 ceph-mon[27786]: Cluster is now healthy 2024-04-24T08:07:57.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:57 smithi028 ceph-mon[27786]: fsmap cephfs:2 {0=g=up:active,1=b=up:active} 10 up:standby 2024-04-24T08:07:57.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:57 smithi028 ceph-mon[27786]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:creating} 9 up:standby 2024-04-24T08:07:57.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:57 smithi028 ceph-mon[27786]: daemon mds.i is now active in filesystem cephfs as rank 2 2024-04-24T08:07:57.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:57 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/279509587' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-04-24T08:07:57.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.593+0000 7f770b200640 1 -- 172.21.15.28:0/3521700686 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f770c062440 con 0x7f76f00a3c30 2024-04-24T08:07:57.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.626+0000 7f7702a00640 1 -- 172.21.15.28:0/3521700686 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f770c062440 con 0x7f76f00a3c30 2024-04-24T08:07:57.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.629+0000 7f770b200640 1 -- 172.21.15.28:0/3521700686 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f76f00a3c30 msgr2=0x7f76f00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.629+0000 7f770b200640 1 --2- 172.21.15.28:0/3521700686 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f76f00a3c30 0x7f76f00a60f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f76fc009bd0 tx=0x7f76fc005f90 comp rx=0 tx=0).stop 2024-04-24T08:07:57.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.629+0000 7f770b200640 1 -- 172.21.15.28:0/3521700686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c109de0 msgr2=0x7f770c1a19e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.629+0000 7f770b200640 1 --2- 172.21.15.28:0/3521700686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c109de0 0x7f770c1a19e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f770400efd0 tx=0x7f770400c6a0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.629+0000 7f770b200640 1 -- 172.21.15.28:0/3521700686 shutdown_connections 2024-04-24T08:07:57.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.629+0000 7f770b200640 1 --2- 172.21.15.28:0/3521700686 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f76f00a3c30 0x7f76f00a60f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.629+0000 7f770b200640 1 --2- 172.21.15.28:0/3521700686 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f770c109de0 0x7f770c1a19e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.629+0000 7f770b200640 1 --2- 172.21.15.28:0/3521700686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f770c075fc0 0x7f770c19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.629+0000 7f770b200640 1 --2- 172.21.15.28:0/3521700686 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f770c0749e0 0x7f770c19a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.629+0000 7f770b200640 1 -- 172.21.15.28:0/3521700686 >> 172.21.15.28:0/3521700686 conn(0x7f770c0fd4a0 msgr2=0x7f770c105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:57.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.629+0000 7f770b200640 1 -- 172.21.15.28:0/3521700686 shutdown_connections 2024-04-24T08:07:57.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.629+0000 7f770b200640 1 -- 172.21.15.28:0/3521700686 wait complete. 2024-04-24T08:07:57.647 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-04-24T08:07:57.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.840+0000 7f3a31800640 1 -- 172.21.15.28:0/3413402752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a2c102920 msgr2=0x7f3a2c109240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.840+0000 7f3a31800640 1 --2- 172.21.15.28:0/3413402752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a2c102920 0x7f3a2c109240 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3a24009900 tx=0x7f3a2402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.840+0000 7f3a31800640 1 -- 172.21.15.28:0/3413402752 shutdown_connections 2024-04-24T08:07:57.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.840+0000 7f3a31800640 1 --2- 172.21.15.28:0/3413402752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a2c102920 0x7f3a2c109240 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.840+0000 7f3a31800640 1 --2- 172.21.15.28:0/3413402752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a2c101f60 0x7f3a2c1023e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.840+0000 7f3a31800640 1 --2- 172.21.15.28:0/3413402752 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a2c100e10 0x7f3a2c101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.840+0000 7f3a31800640 1 -- 172.21.15.28:0/3413402752 >> 172.21.15.28:0/3413402752 conn(0x7f3a2c0fc3a0 msgr2=0x7f3a2c0fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:57.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.840+0000 7f3a31800640 1 -- 172.21.15.28:0/3413402752 shutdown_connections 2024-04-24T08:07:57.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.840+0000 7f3a31800640 1 -- 172.21.15.28:0/3413402752 wait complete. 2024-04-24T08:07:57.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.841+0000 7f3a31800640 1 Processor -- start 2024-04-24T08:07:57.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.841+0000 7f3a31800640 1 -- start start 2024-04-24T08:07:57.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.842+0000 7f3a31800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a2c100e10 0x7f3a2c1981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.842+0000 7f3a31800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a2c101f60 0x7f3a2c1986f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.842+0000 7f3a31800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a2c102920 0x7f3a2c19f760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.842+0000 7f3a31800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c10cb00 con 0x7f3a2c100e10 2024-04-24T08:07:57.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.842+0000 7f3a31800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c10c980 con 0x7f3a2c102920 2024-04-24T08:07:57.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.842+0000 7f3a31800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3a2c10cc80 con 0x7f3a2c101f60 2024-04-24T08:07:57.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.842+0000 7f3a2be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a2c100e10 0x7f3a2c1981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.842+0000 7f3a2be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a2c100e10 0x7f3a2c1981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37310/0 (socket says 172.21.15.28:37310) 2024-04-24T08:07:57.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.842+0000 7f3a2be00640 1 -- 172.21.15.28:0/4165167230 learned_addr learned my addr 172.21.15.28:0/4165167230 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:57.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.842+0000 7f3a30e00640 1 --2- 172.21.15.28:0/4165167230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a2c102920 0x7f3a2c19f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.842+0000 7f3a2b400640 1 --2- 172.21.15.28:0/4165167230 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a2c101f60 0x7f3a2c1986f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.843+0000 7f3a2be00640 1 -- 172.21.15.28:0/4165167230 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a2c101f60 msgr2=0x7f3a2c1986f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.843+0000 7f3a2be00640 1 --2- 172.21.15.28:0/4165167230 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a2c101f60 0x7f3a2c1986f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.843+0000 7f3a2be00640 1 -- 172.21.15.28:0/4165167230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a2c102920 msgr2=0x7f3a2c19f760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:57.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.843+0000 7f3a2be00640 1 --2- 172.21.15.28:0/4165167230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a2c102920 0x7f3a2c19f760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:57.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.843+0000 7f3a2be00640 1 -- 172.21.15.28:0/4165167230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3a2c19fdd0 con 0x7f3a2c100e10 2024-04-24T08:07:57.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.843+0000 7f3a30e00640 1 --2- 172.21.15.28:0/4165167230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a2c102920 0x7f3a2c19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:07:57.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.843+0000 7f3a2be00640 1 --2- 172.21.15.28:0/4165167230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a2c100e10 0x7f3a2c1981b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c00eb90 tx=0x7f3a1c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:57.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.843+0000 7f3a28c00640 1 -- 172.21.15.28:0/4165167230 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c004d20 con 0x7f3a2c100e10 2024-04-24T08:07:57.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.843+0000 7f3a28c00640 1 -- 172.21.15.28:0/4165167230 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3a1c00ee90 con 0x7f3a2c100e10 2024-04-24T08:07:57.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.843+0000 7f3a28c00640 1 -- 172.21.15.28:0/4165167230 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3a1c010830 con 0x7f3a2c100e10 2024-04-24T08:07:57.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.843+0000 7f3a31800640 1 -- 172.21.15.28:0/4165167230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3a2c1a00c0 con 0x7f3a2c100e10 2024-04-24T08:07:57.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.843+0000 7f3a31800640 1 -- 172.21.15.28:0/4165167230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3a2c1a0610 con 0x7f3a2c100e10 2024-04-24T08:07:57.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.844+0000 7f3a31800640 1 -- 172.21.15.28:0/4165167230 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3a2c1a08d0 con 0x7f3a2c100e10 2024-04-24T08:07:57.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.847+0000 7f3a28c00640 1 -- 172.21.15.28:0/4165167230 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f3a1c005000 con 0x7f3a2c100e10 2024-04-24T08:07:57.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.848+0000 7f3a28c00640 1 --2- 172.21.15.28:0/4165167230 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3a100a3c30 0x7f3a100a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:57.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.848+0000 7f3a28c00640 1 -- 172.21.15.28:0/4165167230 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f3a1c0d0bc0 con 0x7f3a2c100e10 2024-04-24T08:07:57.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.848+0000 7f3a2b400640 1 --2- 172.21.15.28:0/4165167230 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3a100a3c30 0x7f3a100a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:57.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.849+0000 7f3a2b400640 1 --2- 172.21.15.28:0/4165167230 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3a100a3c30 0x7f3a100a60f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3a140095f0 tx=0x7f3a14005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:57.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:57.851+0000 7f3a28c00640 1 -- 172.21.15.28:0/4165167230 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3a1c053ff0 con 0x7f3a2c100e10 2024-04-24T08:07:58.039 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:07:57 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:07:57.601+0000 7f38d5e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:07:58.039 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:07:57 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:07:57.601+0000 7f38d5e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:07:58.039 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:07:57 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:07:57.601+0000 7f38d5e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:07:58.039 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:07:57 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:07:57.601+0000 7f38d5e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:07:58.039 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:07:57 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:07:57.601+0000 7f38d5e00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:07:58.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.082+0000 7f3a31800640 1 -- 172.21.15.28:0/4165167230 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f3a2c0fe4b0 con 0x7f3a100a3c30 2024-04-24T08:07:58.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.124+0000 7f3a28c00640 1 -- 172.21.15.28:0/4165167230 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f3a2c0fe4b0 con 0x7f3a100a3c30 2024-04-24T08:07:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.126+0000 7f3a31800640 1 -- 172.21.15.28:0/4165167230 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3a100a3c30 msgr2=0x7f3a100a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.126+0000 7f3a31800640 1 --2- 172.21.15.28:0/4165167230 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3a100a3c30 0x7f3a100a60f0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f3a140095f0 tx=0x7f3a14005f90 comp rx=0 tx=0).stop 2024-04-24T08:07:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.126+0000 7f3a31800640 1 -- 172.21.15.28:0/4165167230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a2c100e10 msgr2=0x7f3a2c1981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.126+0000 7f3a31800640 1 --2- 172.21.15.28:0/4165167230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a2c100e10 0x7f3a2c1981b0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3a1c00eb90 tx=0x7f3a1c00c6a0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.126+0000 7f3a31800640 1 -- 172.21.15.28:0/4165167230 shutdown_connections 2024-04-24T08:07:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.126+0000 7f3a31800640 1 --2- 172.21.15.28:0/4165167230 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3a100a3c30 0x7f3a100a60f0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.126+0000 7f3a31800640 1 --2- 172.21.15.28:0/4165167230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a2c102920 0x7f3a2c19f760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.126+0000 7f3a31800640 1 --2- 172.21.15.28:0/4165167230 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3a2c101f60 0x7f3a2c1986f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.126+0000 7f3a31800640 1 --2- 172.21.15.28:0/4165167230 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3a2c100e10 0x7f3a2c1981b0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.126+0000 7f3a31800640 1 -- 172.21.15.28:0/4165167230 >> 172.21.15.28:0/4165167230 conn(0x7f3a2c0fc3a0 msgr2=0x7f3a2c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.127+0000 7f3a31800640 1 -- 172.21.15.28:0/4165167230 shutdown_connections 2024-04-24T08:07:58.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.127+0000 7f3a31800640 1 -- 172.21.15.28:0/4165167230 wait complete. 2024-04-24T08:07:58.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-04-24T08:07:58.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.356+0000 7f5a46400640 1 -- 172.21.15.28:0/484492319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a40102ac0 msgr2=0x7f5a40104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.356+0000 7f5a46400640 1 --2- 172.21.15.28:0/484492319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a40102ac0 0x7f5a40104eb0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f5a30009900 tx=0x7f5a3002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.356+0000 7f5a46400640 1 -- 172.21.15.28:0/484492319 shutdown_connections 2024-04-24T08:07:58.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.356+0000 7f5a46400640 1 --2- 172.21.15.28:0/484492319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a40107d20 0x7f5a4010a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.356+0000 7f5a46400640 1 --2- 172.21.15.28:0/484492319 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5a401053f0 0x7f5a401077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.356+0000 7f5a46400640 1 --2- 172.21.15.28:0/484492319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a40102ac0 0x7f5a40104eb0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.356+0000 7f5a46400640 1 -- 172.21.15.28:0/484492319 >> 172.21.15.28:0/484492319 conn(0x7f5a400fc4b0 msgr2=0x7f5a400fe8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:58.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.357+0000 7f5a46400640 1 -- 172.21.15.28:0/484492319 shutdown_connections 2024-04-24T08:07:58.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.357+0000 7f5a46400640 1 -- 172.21.15.28:0/484492319 wait complete. 2024-04-24T08:07:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.357+0000 7f5a46400640 1 Processor -- start 2024-04-24T08:07:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.358+0000 7f5a46400640 1 -- start start 2024-04-24T08:07:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.358+0000 7f5a46400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a40102ac0 0x7f5a4019a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.358+0000 7f5a46400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a401053f0 0x7f5a4019ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.358+0000 7f5a46400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5a40107d20 0x7f5a401a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.358+0000 7f5a46400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4010dd20 con 0x7f5a401053f0 2024-04-24T08:07:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.358+0000 7f5a46400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4010dba0 con 0x7f5a40102ac0 2024-04-24T08:07:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.358+0000 7f5a46400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5a4010dea0 con 0x7f5a40107d20 2024-04-24T08:07:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.358+0000 7f5a3ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a401053f0 0x7f5a4019ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.358+0000 7f5a3f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a40102ac0 0x7f5a4019a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.358+0000 7f5a3ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a401053f0 0x7f5a4019ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37318/0 (socket says 172.21.15.28:37318) 2024-04-24T08:07:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.358+0000 7f5a3ea00640 1 -- 172.21.15.28:0/2621088214 learned_addr learned my addr 172.21.15.28:0/2621088214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.359+0000 7f5a3fe00640 1 --2- 172.21.15.28:0/2621088214 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5a40107d20 0x7f5a401a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.359+0000 7f5a3ea00640 1 -- 172.21.15.28:0/2621088214 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5a40107d20 msgr2=0x7f5a401a1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.359+0000 7f5a3ea00640 1 --2- 172.21.15.28:0/2621088214 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5a40107d20 0x7f5a401a1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.359+0000 7f5a3ea00640 1 -- 172.21.15.28:0/2621088214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a40102ac0 msgr2=0x7f5a4019a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.359+0000 7f5a3ea00640 1 --2- 172.21.15.28:0/2621088214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a40102ac0 0x7f5a4019a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.359+0000 7f5a3ea00640 1 -- 172.21.15.28:0/2621088214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a401a2210 con 0x7f5a401053f0 2024-04-24T08:07:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.359+0000 7f5a3f400640 1 --2- 172.21.15.28:0/2621088214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a40102ac0 0x7f5a4019a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.359+0000 7f5a3ea00640 1 --2- 172.21.15.28:0/2621088214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a401053f0 0x7f5a4019ab30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c00e9f0 tx=0x7f5a2c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.359+0000 7f5a37e00640 1 -- 172.21.15.28:0/2621088214 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c00ccc0 con 0x7f5a401053f0 2024-04-24T08:07:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.360+0000 7f5a37e00640 1 -- 172.21.15.28:0/2621088214 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a2c005500 con 0x7f5a401053f0 2024-04-24T08:07:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.360+0000 7f5a46400640 1 -- 172.21.15.28:0/2621088214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a401a2500 con 0x7f5a401053f0 2024-04-24T08:07:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.360+0000 7f5a37e00640 1 -- 172.21.15.28:0/2621088214 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a2c010690 con 0x7f5a401053f0 2024-04-24T08:07:58.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.360+0000 7f5a46400640 1 -- 172.21.15.28:0/2621088214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a401a2a00 con 0x7f5a401053f0 2024-04-24T08:07:58.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.361+0000 7f5a46400640 1 -- 172.21.15.28:0/2621088214 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a40109870 con 0x7f5a401053f0 2024-04-24T08:07:58.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.363+0000 7f5a37e00640 1 -- 172.21.15.28:0/2621088214 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7f5a2c0046c0 con 0x7f5a401053f0 2024-04-24T08:07:58.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.365+0000 7f5a37e00640 1 --2- 172.21.15.28:0/2621088214 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5a240a3bb0 0x7f5a240a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.365+0000 7f5a37e00640 1 -- 172.21.15.28:0/2621088214 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f5a2c0d0690 con 0x7f5a401053f0 2024-04-24T08:07:58.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.365+0000 7f5a3f400640 1 --2- 172.21.15.28:0/2621088214 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5a240a3bb0 0x7f5a240a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.366+0000 7f5a3f400640 1 --2- 172.21.15.28:0/2621088214 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5a240a3bb0 0x7f5a240a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5a30031440 tx=0x7f5a300315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:58.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.369+0000 7f5a37e00640 1 -- 172.21.15.28:0/2621088214 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5a2c09d120 con 0x7f5a401053f0 2024-04-24T08:07:58.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:58 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/279509587' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-24T08:07:58.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:58 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] up:active 2024-04-24T08:07:58.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:58 smithi194 ceph-mon[32836]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2024-04-24T08:07:58.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:58 smithi194 ceph-mon[32836]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:07:58.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:58 smithi194 ceph-mon[32836]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:07:58.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:58 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/279509587' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-24T08:07:58.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:58 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] up:active 2024-04-24T08:07:58.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:58 smithi028 ceph-mon[27786]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2024-04-24T08:07:58.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:58 smithi028 ceph-mon[27786]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:07:58.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:58 smithi028 ceph-mon[27786]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:07:58.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.608+0000 7f5a46400640 1 -- 172.21.15.28:0/2621088214 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f5a40062440 con 0x7f5a240a3bb0 2024-04-24T08:07:58.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.616+0000 7f5a37e00640 1 -- 172.21.15.28:0/2621088214 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f5a40062440 con 0x7f5a240a3bb0 2024-04-24T08:07:58.616 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_0/15b3bbe4-0d2e-47bd-a64e-5d83e067145b 2024-04-24T08:07:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.618+0000 7f5a46400640 1 -- 172.21.15.28:0/2621088214 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5a240a3bb0 msgr2=0x7f5a240a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.618+0000 7f5a46400640 1 --2- 172.21.15.28:0/2621088214 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5a240a3bb0 0x7f5a240a6070 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f5a30031440 tx=0x7f5a300315b0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.618+0000 7f5a46400640 1 -- 172.21.15.28:0/2621088214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a401053f0 msgr2=0x7f5a4019ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.618+0000 7f5a46400640 1 --2- 172.21.15.28:0/2621088214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a401053f0 0x7f5a4019ab30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f5a2c00e9f0 tx=0x7f5a2c00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.619+0000 7f5a46400640 1 -- 172.21.15.28:0/2621088214 shutdown_connections 2024-04-24T08:07:58.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.619+0000 7f5a46400640 1 --2- 172.21.15.28:0/2621088214 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5a240a3bb0 0x7f5a240a6070 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.619+0000 7f5a46400640 1 --2- 172.21.15.28:0/2621088214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5a401053f0 0x7f5a4019ab30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.619+0000 7f5a46400640 1 --2- 172.21.15.28:0/2621088214 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5a40107d20 0x7f5a401a1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.619+0000 7f5a46400640 1 --2- 172.21.15.28:0/2621088214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a40102ac0 0x7f5a4019a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.619+0000 7f5a46400640 1 -- 172.21.15.28:0/2621088214 >> 172.21.15.28:0/2621088214 conn(0x7f5a400fc4b0 msgr2=0x7f5a401021f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.619+0000 7f5a46400640 1 -- 172.21.15.28:0/2621088214 shutdown_connections 2024-04-24T08:07:58.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.619+0000 7f5a46400640 1 -- 172.21.15.28:0/2621088214 wait complete. 2024-04-24T08:07:58.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:58 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/279509587' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-04-24T08:07:58.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:58 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] up:active 2024-04-24T08:07:58.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:58 smithi117 ceph-mon[32438]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:active} 9 up:standby 2024-04-24T08:07:58.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:58 smithi117 ceph-mon[32438]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:07:58.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:58 smithi117 ceph-mon[32438]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:07:58.637 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-04-24T08:07:58.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.851+0000 7ffa61200640 1 -- 172.21.15.28:0/244337612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5c101340 msgr2=0x7ffa5c1017c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.851+0000 7ffa61200640 1 --2- 172.21.15.28:0/244337612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5c101340 0x7ffa5c1017c0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ffa48009900 tx=0x7ffa4802a400 comp rx=0 tx=0).stop 2024-04-24T08:07:58.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.851+0000 7ffa61200640 1 -- 172.21.15.28:0/244337612 shutdown_connections 2024-04-24T08:07:58.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.851+0000 7ffa61200640 1 --2- 172.21.15.28:0/244337612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5c101340 0x7ffa5c1017c0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.851+0000 7ffa61200640 1 --2- 172.21.15.28:0/244337612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa5c101d00 0x7ffa5c108590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.851+0000 7ffa61200640 1 --2- 172.21.15.28:0/244337612 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa5c100150 0x7ffa5c100570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.851+0000 7ffa61200640 1 -- 172.21.15.28:0/244337612 >> 172.21.15.28:0/244337612 conn(0x7ffa5c0fb6c0 msgr2=0x7ffa5c0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:58.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.851+0000 7ffa61200640 1 -- 172.21.15.28:0/244337612 shutdown_connections 2024-04-24T08:07:58.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.851+0000 7ffa61200640 1 -- 172.21.15.28:0/244337612 wait complete. 2024-04-24T08:07:58.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.852+0000 7ffa61200640 1 Processor -- start 2024-04-24T08:07:58.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.852+0000 7ffa61200640 1 -- start start 2024-04-24T08:07:58.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.852+0000 7ffa61200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa5c100150 0x7ffa5c19a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.852+0000 7ffa61200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5c101340 0x7ffa5c19a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.852+0000 7ffa61200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa5c101d00 0x7ffa5c1a1a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.852+0000 7ffa61200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5c10b9a0 con 0x7ffa5c101340 2024-04-24T08:07:58.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.852+0000 7ffa61200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5c10b820 con 0x7ffa5c100150 2024-04-24T08:07:58.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.853+0000 7ffa61200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ffa5c10bb20 con 0x7ffa5c101d00 2024-04-24T08:07:58.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.853+0000 7ffa5aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5c101340 0x7ffa5c19a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.853+0000 7ffa5be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa5c101d00 0x7ffa5c1a1a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.853+0000 7ffa5aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5c101340 0x7ffa5c19a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37342/0 (socket says 172.21.15.28:37342) 2024-04-24T08:07:58.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.853+0000 7ffa5aa00640 1 -- 172.21.15.28:0/2165435005 learned_addr learned my addr 172.21.15.28:0/2165435005 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:58.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.853+0000 7ffa5b400640 1 --2- 172.21.15.28:0/2165435005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa5c100150 0x7ffa5c19a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.853+0000 7ffa5be00640 1 -- 172.21.15.28:0/2165435005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa5c100150 msgr2=0x7ffa5c19a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.853+0000 7ffa5be00640 1 --2- 172.21.15.28:0/2165435005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa5c100150 0x7ffa5c19a4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.853+0000 7ffa5be00640 1 -- 172.21.15.28:0/2165435005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5c101340 msgr2=0x7ffa5c19a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:58.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.853+0000 7ffa5be00640 1 --2- 172.21.15.28:0/2165435005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5c101340 0x7ffa5c19a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.853+0000 7ffa5be00640 1 -- 172.21.15.28:0/2165435005 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa5c1a20c0 con 0x7ffa5c101d00 2024-04-24T08:07:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.853+0000 7ffa5aa00640 1 --2- 172.21.15.28:0/2165435005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5c101340 0x7ffa5c19a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.854+0000 7ffa5be00640 1 --2- 172.21.15.28:0/2165435005 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa5c101d00 0x7ffa5c1a1a50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ffa54005fd0 tx=0x7ffa5400aac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.854+0000 7ffa53e00640 1 -- 172.21.15.28:0/2165435005 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa54018020 con 0x7ffa5c101d00 2024-04-24T08:07:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.854+0000 7ffa61200640 1 -- 172.21.15.28:0/2165435005 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa5c1a23b0 con 0x7ffa5c101d00 2024-04-24T08:07:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.854+0000 7ffa53e00640 1 -- 172.21.15.28:0/2165435005 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa54004d20 con 0x7ffa5c101d00 2024-04-24T08:07:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.854+0000 7ffa53e00640 1 -- 172.21.15.28:0/2165435005 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa54007ac0 con 0x7ffa5c101d00 2024-04-24T08:07:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.854+0000 7ffa61200640 1 -- 172.21.15.28:0/2165435005 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa5c1a28d0 con 0x7ffa5c101d00 2024-04-24T08:07:58.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.855+0000 7ffa61200640 1 -- 172.21.15.28:0/2165435005 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa5c1074a0 con 0x7ffa5c101d00 2024-04-24T08:07:58.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.857+0000 7ffa53e00640 1 -- 172.21.15.28:0/2165435005 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 16) ==== 138646+0+0 (secure 0 0 0) 0x7ffa54007c60 con 0x7ffa5c101d00 2024-04-24T08:07:58.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.859+0000 7ffa53e00640 1 --2- 172.21.15.28:0/2165435005 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffa400a3c30 0x7ffa400a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:58.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.859+0000 7ffa53e00640 1 -- 172.21.15.28:0/2165435005 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7ffa540d00d0 con 0x7ffa5c101d00 2024-04-24T08:07:58.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.859+0000 7ffa5b400640 1 --2- 172.21.15.28:0/2165435005 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffa400a3c30 0x7ffa400a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:58.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.859+0000 7ffa5b400640 1 --2- 172.21.15.28:0/2165435005 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffa400a3c30 0x7ffa400a60f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c1011a0 tx=0x7ffa4c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:58.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:58.861+0000 7ffa53e00640 1 -- 172.21.15.28:0/2165435005 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa54022020 con 0x7ffa5c101d00 2024-04-24T08:07:59.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.103+0000 7ffa61200640 1 -- 172.21.15.28:0/2165435005 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7ffa5c0624d0 con 0x7ffa400a3c30 2024-04-24T08:07:59.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.154+0000 7ffa53e00640 1 -- 172.21.15.28:0/2165435005 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7ffa5c0624d0 con 0x7ffa400a3c30 2024-04-24T08:07:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.157+0000 7ffa61200640 1 -- 172.21.15.28:0/2165435005 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffa400a3c30 msgr2=0x7ffa400a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.157+0000 7ffa61200640 1 --2- 172.21.15.28:0/2165435005 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffa400a3c30 0x7ffa400a60f0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ffa5c1011a0 tx=0x7ffa4c00b390 comp rx=0 tx=0).stop 2024-04-24T08:07:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.157+0000 7ffa61200640 1 -- 172.21.15.28:0/2165435005 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa5c101d00 msgr2=0x7ffa5c1a1a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.157+0000 7ffa61200640 1 --2- 172.21.15.28:0/2165435005 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa5c101d00 0x7ffa5c1a1a50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7ffa54005fd0 tx=0x7ffa5400aac0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.157+0000 7ffa61200640 1 -- 172.21.15.28:0/2165435005 shutdown_connections 2024-04-24T08:07:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.157+0000 7ffa61200640 1 --2- 172.21.15.28:0/2165435005 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ffa400a3c30 0x7ffa400a60f0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.157+0000 7ffa61200640 1 --2- 172.21.15.28:0/2165435005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa5c101340 0x7ffa5c19a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.157+0000 7ffa61200640 1 --2- 172.21.15.28:0/2165435005 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa5c101d00 0x7ffa5c1a1a50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.157+0000 7ffa61200640 1 --2- 172.21.15.28:0/2165435005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa5c100150 0x7ffa5c19a4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.158+0000 7ffa61200640 1 -- 172.21.15.28:0/2165435005 >> 172.21.15.28:0/2165435005 conn(0x7ffa5c0fb6c0 msgr2=0x7ffa5c104670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:59.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.158+0000 7ffa61200640 1 -- 172.21.15.28:0/2165435005 shutdown_connections 2024-04-24T08:07:59.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.158+0000 7ffa61200640 1 -- 172.21.15.28:0/2165435005 wait complete. 2024-04-24T08:07:59.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-04-24T08:07:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.395+0000 7f14f3800640 1 -- 172.21.15.28:0/3169411523 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14ec10a210 msgr2=0x7f14ec110aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.395+0000 7f14f3800640 1 --2- 172.21.15.28:0/3169411523 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14ec10a210 0x7f14ec110aa0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f14e4009ca0 tx=0x7f14e402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.395+0000 7f14f3800640 1 -- 172.21.15.28:0/3169411523 shutdown_connections 2024-04-24T08:07:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.395+0000 7f14f3800640 1 --2- 172.21.15.28:0/3169411523 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14ec109850 0x7f14ec109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.395+0000 7f14f3800640 1 --2- 172.21.15.28:0/3169411523 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14ec10a210 0x7f14ec110aa0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.395+0000 7f14f3800640 1 --2- 172.21.15.28:0/3169411523 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14ec108660 0x7f14ec108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.395+0000 7f14f3800640 1 -- 172.21.15.28:0/3169411523 >> 172.21.15.28:0/3169411523 conn(0x7f14ec077600 msgr2=0x7f14ec079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:59.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.395+0000 7f14f3800640 1 -- 172.21.15.28:0/3169411523 shutdown_connections 2024-04-24T08:07:59.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.395+0000 7f14f3800640 1 -- 172.21.15.28:0/3169411523 wait complete. 2024-04-24T08:07:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.396+0000 7f14f3800640 1 Processor -- start 2024-04-24T08:07:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.397+0000 7f14f3800640 1 -- start start 2024-04-24T08:07:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.397+0000 7f14f3800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14ec108660 0x7f14ec19e910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.397+0000 7f14f3800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14ec109850 0x7f14ec19ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.397+0000 7f14f3800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14ec10a210 0x7f14ec1a5ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.397+0000 7f14f3800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14ec114330 con 0x7f14ec10a210 2024-04-24T08:07:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.397+0000 7f14f3800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f14ec1141b0 con 0x7f14ec108660 2024-04-24T08:07:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.397+0000 7f14f3800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f14ec1144b0 con 0x7f14ec109850 2024-04-24T08:07:59.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.397+0000 7f14f0e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14ec108660 0x7f14ec19e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:59.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.397+0000 7f14ebe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14ec109850 0x7f14ec19ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:59.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.397+0000 7f14ebe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14ec109850 0x7f14ec19ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:52566/0 (socket says 172.21.15.28:52566) 2024-04-24T08:07:59.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.397+0000 7f14ebe00640 1 -- 172.21.15.28:0/964247534 learned_addr learned my addr 172.21.15.28:0/964247534 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:59.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.397+0000 7f14f0e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14ec108660 0x7f14ec19e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:57780/0 (socket says 172.21.15.28:57780) 2024-04-24T08:07:59.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.397+0000 7f14f1800640 1 --2- 172.21.15.28:0/964247534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14ec10a210 0x7f14ec1a5ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:59.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.398+0000 7f14ebe00640 1 -- 172.21.15.28:0/964247534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14ec108660 msgr2=0x7f14ec19e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:59.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.398+0000 7f14ebe00640 1 --2- 172.21.15.28:0/964247534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14ec108660 0x7f14ec19e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.398+0000 7f14ebe00640 1 -- 172.21.15.28:0/964247534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14ec10a210 msgr2=0x7f14ec1a5ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:59.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.398+0000 7f14ebe00640 1 --2- 172.21.15.28:0/964247534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14ec10a210 0x7f14ec1a5ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.398+0000 7f14ebe00640 1 -- 172.21.15.28:0/964247534 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f14ec1a6530 con 0x7f14ec109850 2024-04-24T08:07:59.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.398+0000 7f14f0e00640 1 --2- 172.21.15.28:0/964247534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14ec108660 0x7f14ec19e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:07:59.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.398+0000 7f14ebe00640 1 --2- 172.21.15.28:0/964247534 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14ec109850 0x7f14ec19ee50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f14d400e9f0 tx=0x7f14d400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:59.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.398+0000 7f14e9600640 1 -- 172.21.15.28:0/964247534 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d400cd80 con 0x7f14ec109850 2024-04-24T08:07:59.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.398+0000 7f14e9600640 1 -- 172.21.15.28:0/964247534 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f14d4010910 con 0x7f14ec109850 2024-04-24T08:07:59.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.398+0000 7f14f3800640 1 -- 172.21.15.28:0/964247534 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14ec1a6820 con 0x7f14ec109850 2024-04-24T08:07:59.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.398+0000 7f14e9600640 1 -- 172.21.15.28:0/964247534 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14d4018990 con 0x7f14ec109850 2024-04-24T08:07:59.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.399+0000 7f14f3800640 1 -- 172.21.15.28:0/964247534 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14ec1a6d70 con 0x7f14ec109850 2024-04-24T08:07:59.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.400+0000 7f14f3800640 1 -- 172.21.15.28:0/964247534 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f14ec10f980 con 0x7f14ec109850 2024-04-24T08:07:59.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.402+0000 7f14e9600640 1 -- 172.21.15.28:0/964247534 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f14d4010ac0 con 0x7f14ec109850 2024-04-24T08:07:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.403+0000 7f14e9600640 1 --2- 172.21.15.28:0/964247534 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f14d00a3c40 0x7f14d00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.403+0000 7f14e9600640 1 -- 172.21.15.28:0/964247534 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f14d40d09f0 con 0x7f14ec109850 2024-04-24T08:07:59.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.403+0000 7f14f0e00640 1 --2- 172.21.15.28:0/964247534 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f14d00a3c40 0x7f14d00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:59.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.404+0000 7f14f0e00640 1 --2- 172.21.15.28:0/964247534 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f14d00a3c40 0x7f14d00a6100 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f14ec1096b0 tx=0x7f14dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:59.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.405+0000 7f14e9600640 1 -- 172.21.15.28:0/964247534 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f14d409ccd0 con 0x7f14ec109850 2024-04-24T08:07:59.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:59 smithi194 ceph-mon[32836]: pgmap v320: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 582 B/s rd, 2.4 KiB/s wr, 12 op/s 2024-04-24T08:07:59.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:59 smithi194 ceph-mon[32836]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:07:59.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:07:59 smithi194 ceph-mon[32836]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:07:59.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:59 smithi028 ceph-mon[27786]: pgmap v320: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 582 B/s rd, 2.4 KiB/s wr, 12 op/s 2024-04-24T08:07:59.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:59 smithi028 ceph-mon[27786]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:07:59.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:07:59 smithi028 ceph-mon[27786]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:07:59.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:59 smithi117 ceph-mon[32438]: pgmap v320: 97 pgs: 97 active+clean; 579 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 582 B/s rd, 2.4 KiB/s wr, 12 op/s 2024-04-24T08:07:59.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:59 smithi117 ceph-mon[32438]: from='client.14568 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:07:59.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:07:59 smithi117 ceph-mon[32438]: from='client.14574 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:07:59.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.643+0000 7f14f3800640 1 -- 172.21.15.28:0/964247534 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f14ec062440 con 0x7f14d00a3c40 2024-04-24T08:07:59.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.655+0000 7f14e9600640 1 -- 172.21.15.28:0/964247534 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f14ec062440 con 0x7f14d00a3c40 2024-04-24T08:07:59.655 INFO:teuthology.orchestra.run.smithi028.stdout:/volumes/qa/sv_1/30aae337-e674-40f1-ba6b-0c04150e17c4 2024-04-24T08:07:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.657+0000 7f14f3800640 1 -- 172.21.15.28:0/964247534 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f14d00a3c40 msgr2=0x7f14d00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.657+0000 7f14f3800640 1 --2- 172.21.15.28:0/964247534 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f14d00a3c40 0x7f14d00a6100 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f14ec1096b0 tx=0x7f14dc005f90 comp rx=0 tx=0).stop 2024-04-24T08:07:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.658+0000 7f14f3800640 1 -- 172.21.15.28:0/964247534 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14ec109850 msgr2=0x7f14ec19ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:59.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.658+0000 7f14f3800640 1 --2- 172.21.15.28:0/964247534 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14ec109850 0x7f14ec19ee50 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f14d400e9f0 tx=0x7f14d400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.658+0000 7f14f3800640 1 -- 172.21.15.28:0/964247534 shutdown_connections 2024-04-24T08:07:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.658+0000 7f14f3800640 1 --2- 172.21.15.28:0/964247534 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f14d00a3c40 0x7f14d00a6100 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.658+0000 7f14f3800640 1 --2- 172.21.15.28:0/964247534 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f14ec10a210 0x7f14ec1a5ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.658+0000 7f14f3800640 1 --2- 172.21.15.28:0/964247534 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f14ec109850 0x7f14ec19ee50 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.658+0000 7f14f3800640 1 --2- 172.21.15.28:0/964247534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14ec108660 0x7f14ec19e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.658+0000 7f14f3800640 1 -- 172.21.15.28:0/964247534 >> 172.21.15.28:0/964247534 conn(0x7f14ec077600 msgr2=0x7f14ec10cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.658+0000 7f14f3800640 1 -- 172.21.15.28:0/964247534 shutdown_connections 2024-04-24T08:07:59.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.658+0000 7f14f3800640 1 -- 172.21.15.28:0/964247534 wait complete. 2024-04-24T08:07:59.678 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:07:59.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.891+0000 7f3f73200640 1 -- 172.21.15.28:0/436788996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f74109670 msgr2=0x7f3f74109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:59.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.891+0000 7f3f73200640 1 --2- 172.21.15.28:0/436788996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f74109670 0x7f3f74109af0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c009900 tx=0x7f3f5c02a410 comp rx=0 tx=0).stop 2024-04-24T08:07:59.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.891+0000 7f3f73200640 1 -- 172.21.15.28:0/436788996 shutdown_connections 2024-04-24T08:07:59.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.891+0000 7f3f73200640 1 --2- 172.21.15.28:0/436788996 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f7410a030 0x7f3f741108c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.891+0000 7f3f73200640 1 --2- 172.21.15.28:0/436788996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f74109670 0x7f3f74109af0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.892+0000 7f3f73200640 1 --2- 172.21.15.28:0/436788996 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f74108480 0x7f3f741088a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.892+0000 7f3f73200640 1 -- 172.21.15.28:0/436788996 >> 172.21.15.28:0/436788996 conn(0x7f3f74077440 msgr2=0x7f3f74079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:07:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.892+0000 7f3f73200640 1 -- 172.21.15.28:0/436788996 shutdown_connections 2024-04-24T08:07:59.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.892+0000 7f3f73200640 1 -- 172.21.15.28:0/436788996 wait complete. 2024-04-24T08:07:59.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.893+0000 7f3f73200640 1 Processor -- start 2024-04-24T08:07:59.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.893+0000 7f3f73200640 1 -- start start 2024-04-24T08:07:59.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.893+0000 7f3f73200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74108480 0x7f3f7419e780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:59.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.893+0000 7f3f73200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f74109670 0x7f3f7419ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:59.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.893+0000 7f3f73200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f7410a030 0x7f3f741a5d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:59.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.893+0000 7f3f73200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f74114160 con 0x7f3f74108480 2024-04-24T08:07:59.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.893+0000 7f3f73200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3f74113fe0 con 0x7f3f74109670 2024-04-24T08:07:59.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.893+0000 7f3f73200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3f741142e0 con 0x7f3f7410a030 2024-04-24T08:07:59.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.894+0000 7f3f71e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74108480 0x7f3f7419e780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:59.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.894+0000 7f3f71e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74108480 0x7f3f7419e780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56698/0 (socket says 172.21.15.28:56698) 2024-04-24T08:07:59.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.894+0000 7f3f69400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f74109670 0x7f3f7419ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:59.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.894+0000 7f3f71e00640 1 -- 172.21.15.28:0/3312739288 learned_addr learned my addr 172.21.15.28:0/3312739288 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:07:59.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.894+0000 7f3f72800640 1 --2- 172.21.15.28:0/3312739288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f7410a030 0x7f3f741a5d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:59.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.894+0000 7f3f71e00640 1 -- 172.21.15.28:0/3312739288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f7410a030 msgr2=0x7f3f741a5d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:59.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.894+0000 7f3f71e00640 1 --2- 172.21.15.28:0/3312739288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f7410a030 0x7f3f741a5d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.894+0000 7f3f71e00640 1 -- 172.21.15.28:0/3312739288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f74109670 msgr2=0x7f3f7419ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:07:59.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.894+0000 7f3f71e00640 1 --2- 172.21.15.28:0/3312739288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f74109670 0x7f3f7419ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:07:59.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.894+0000 7f3f71e00640 1 -- 172.21.15.28:0/3312739288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f741a63a0 con 0x7f3f74108480 2024-04-24T08:07:59.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.894+0000 7f3f72800640 1 --2- 172.21.15.28:0/3312739288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f7410a030 0x7f3f741a5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:07:59.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.895+0000 7f3f71e00640 1 --2- 172.21.15.28:0/3312739288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74108480 0x7f3f7419e780 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400e9c0 tx=0x7f3f6400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:59.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.895+0000 7f3f6b400640 1 -- 172.21.15.28:0/3312739288 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f6400cd80 con 0x7f3f74108480 2024-04-24T08:07:59.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.895+0000 7f3f6b400640 1 -- 172.21.15.28:0/3312739288 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f640035f0 con 0x7f3f74108480 2024-04-24T08:07:59.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.895+0000 7f3f6b400640 1 -- 172.21.15.28:0/3312739288 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f64010690 con 0x7f3f74108480 2024-04-24T08:07:59.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.895+0000 7f3f73200640 1 -- 172.21.15.28:0/3312739288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f741a6690 con 0x7f3f74108480 2024-04-24T08:07:59.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.895+0000 7f3f73200640 1 -- 172.21.15.28:0/3312739288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f741a6b60 con 0x7f3f74108480 2024-04-24T08:07:59.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.897+0000 7f3f73200640 1 -- 172.21.15.28:0/3312739288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f7410f7a0 con 0x7f3f74108480 2024-04-24T08:07:59.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.899+0000 7f3f6b400640 1 -- 172.21.15.28:0/3312739288 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3f640037a0 con 0x7f3f74108480 2024-04-24T08:07:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.900+0000 7f3f6b400640 1 --2- 172.21.15.28:0/3312739288 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3f580a3cc0 0x7f3f580a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:07:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.900+0000 7f3f6b400640 1 -- 172.21.15.28:0/3312739288 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f3f640d0870 con 0x7f3f74108480 2024-04-24T08:07:59.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.900+0000 7f3f69400640 1 --2- 172.21.15.28:0/3312739288 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3f580a3cc0 0x7f3f580a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:07:59.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.901+0000 7f3f69400640 1 --2- 172.21.15.28:0/3312739288 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3f580a3cc0 0x7f3f580a6180 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c031440 tx=0x7f3f5c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:07:59.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:07:59.905+0000 7f3f6b400640 1 -- 172.21.15.28:0/3312739288 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f6409d1a0 con 0x7f3f74108480 2024-04-24T08:08:00.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.157+0000 7f3f73200640 1 -- 172.21.15.28:0/3312739288 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3f7402d8d0 con 0x7f3f74108480 2024-04-24T08:08:00.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.158+0000 7f3f6b400640 1 -- 172.21.15.28:0/3312739288 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11871 (secure 0 0 0) 0x7f3f6409caa0 con 0x7f3f74108480 2024-04-24T08:08:00.159 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:08:00.159 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:07:57.133790+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:08:00.159 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 23 2024-04-24T08:08:00.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.161+0000 7f3f73200640 1 -- 172.21.15.28:0/3312739288 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3f580a3cc0 msgr2=0x7f3f580a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:00.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.161+0000 7f3f73200640 1 --2- 172.21.15.28:0/3312739288 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3f580a3cc0 0x7f3f580a6180 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f3f5c031440 tx=0x7f3f5c0315b0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.161+0000 7f3f73200640 1 -- 172.21.15.28:0/3312739288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74108480 msgr2=0x7f3f7419e780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:00.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.161+0000 7f3f73200640 1 --2- 172.21.15.28:0/3312739288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74108480 0x7f3f7419e780 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f3f6400e9c0 tx=0x7f3f6400ee80 comp rx=0 tx=0).stop 2024-04-24T08:08:00.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.161+0000 7f3f73200640 1 -- 172.21.15.28:0/3312739288 shutdown_connections 2024-04-24T08:08:00.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.161+0000 7f3f73200640 1 --2- 172.21.15.28:0/3312739288 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3f580a3cc0 0x7f3f580a6180 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.161+0000 7f3f73200640 1 --2- 172.21.15.28:0/3312739288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f7410a030 0x7f3f741a5d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.161+0000 7f3f73200640 1 --2- 172.21.15.28:0/3312739288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f74109670 0x7f3f7419ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.161+0000 7f3f73200640 1 --2- 172.21.15.28:0/3312739288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f74108480 0x7f3f7419e780 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.161+0000 7f3f73200640 1 -- 172.21.15.28:0/3312739288 >> 172.21.15.28:0/3312739288 conn(0x7f3f74077440 msgr2=0x7f3f7410c970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:00.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.161+0000 7f3f73200640 1 -- 172.21.15.28:0/3312739288 shutdown_connections 2024-04-24T08:08:00.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.161+0000 7f3f73200640 1 -- 172.21.15.28:0/3312739288 wait complete. 2024-04-24T08:08:00.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:08:00.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.394+0000 7fdc74a00640 1 -- 172.21.15.28:0/681022437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc70104280 msgr2=0x7fdc701046a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:00.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.394+0000 7fdc74a00640 1 --2- 172.21.15.28:0/681022437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc70104280 0x7fdc701046a0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fdc68009ca0 tx=0x7fdc6802a8c0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.394+0000 7fdc74a00640 1 -- 172.21.15.28:0/681022437 shutdown_connections 2024-04-24T08:08:00.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.394+0000 7fdc74a00640 1 --2- 172.21.15.28:0/681022437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc70105e30 0x7fdc7010c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.394+0000 7fdc74a00640 1 --2- 172.21.15.28:0/681022437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc70105470 0x7fdc701058f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.394+0000 7fdc74a00640 1 --2- 172.21.15.28:0/681022437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc70104280 0x7fdc701046a0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.394+0000 7fdc74a00640 1 -- 172.21.15.28:0/681022437 >> 172.21.15.28:0/681022437 conn(0x7fdc700ff7f0 msgr2=0x7fdc70101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:00.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.394+0000 7fdc74a00640 1 -- 172.21.15.28:0/681022437 shutdown_connections 2024-04-24T08:08:00.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.394+0000 7fdc74a00640 1 -- 172.21.15.28:0/681022437 wait complete. 2024-04-24T08:08:00.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.395+0000 7fdc74a00640 1 Processor -- start 2024-04-24T08:08:00.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.395+0000 7fdc74a00640 1 -- start start 2024-04-24T08:08:00.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.396+0000 7fdc74a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc70104280 0x7fdc7019a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:00.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.396+0000 7fdc74a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc70105470 0x7fdc7019ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:00.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.396+0000 7fdc74a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc70105e30 0x7fdc701a1b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:00.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.396+0000 7fdc74a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7010ff60 con 0x7fdc70105470 2024-04-24T08:08:00.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.396+0000 7fdc74a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdc7010fde0 con 0x7fdc70105e30 2024-04-24T08:08:00.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.396+0000 7fdc74a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdc701100e0 con 0x7fdc70104280 2024-04-24T08:08:00.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.396+0000 7fdc6d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc70105470 0x7fdc7019ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:00.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.396+0000 7fdc6d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc70105470 0x7fdc7019ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56706/0 (socket says 172.21.15.28:56706) 2024-04-24T08:08:00.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.396+0000 7fdc6d400640 1 -- 172.21.15.28:0/3672844364 learned_addr learned my addr 172.21.15.28:0/3672844364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:00.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.396+0000 7fdc6de00640 1 --2- 172.21.15.28:0/3672844364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc70104280 0x7fdc7019a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.396+0000 7fdc6e800640 1 --2- 172.21.15.28:0/3672844364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc70105e30 0x7fdc701a1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.397+0000 7fdc6e800640 1 -- 172.21.15.28:0/3672844364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc70104280 msgr2=0x7fdc7019a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.397+0000 7fdc6e800640 1 --2- 172.21.15.28:0/3672844364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc70104280 0x7fdc7019a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.397+0000 7fdc6e800640 1 -- 172.21.15.28:0/3672844364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc70105470 msgr2=0x7fdc7019ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.397+0000 7fdc6e800640 1 --2- 172.21.15.28:0/3672844364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc70105470 0x7fdc7019ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.397+0000 7fdc6e800640 1 -- 172.21.15.28:0/3672844364 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc701a2200 con 0x7fdc70105e30 2024-04-24T08:08:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.397+0000 7fdc6d400640 1 --2- 172.21.15.28:0/3672844364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc70105470 0x7fdc7019ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.397+0000 7fdc6e800640 1 --2- 172.21.15.28:0/3672844364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc70105e30 0x7fdc701a1b90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000e970 tx=0x7fdc6000ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.397+0000 7fdc66a00640 1 -- 172.21.15.28:0/3672844364 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6000ccc0 con 0x7fdc70105e30 2024-04-24T08:08:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.397+0000 7fdc66a00640 1 -- 172.21.15.28:0/3672844364 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc600035f0 con 0x7fdc70105e30 2024-04-24T08:08:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.397+0000 7fdc74a00640 1 -- 172.21.15.28:0/3672844364 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc701a24f0 con 0x7fdc70105e30 2024-04-24T08:08:00.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.397+0000 7fdc66a00640 1 -- 172.21.15.28:0/3672844364 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc6000ce60 con 0x7fdc70105e30 2024-04-24T08:08:00.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.397+0000 7fdc74a00640 1 -- 172.21.15.28:0/3672844364 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc701a29f0 con 0x7fdc70105e30 2024-04-24T08:08:00.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.398+0000 7fdc74a00640 1 -- 172.21.15.28:0/3672844364 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc7010b5a0 con 0x7fdc70105e30 2024-04-24T08:08:00.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.400+0000 7fdc66a00640 1 -- 172.21.15.28:0/3672844364 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fdc600037a0 con 0x7fdc70105e30 2024-04-24T08:08:00.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.402+0000 7fdc66a00640 1 --2- 172.21.15.28:0/3672844364 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fdc540a3c40 0x7fdc540a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:00.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.402+0000 7fdc66a00640 1 -- 172.21.15.28:0/3672844364 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fdc600d0700 con 0x7fdc70105e30 2024-04-24T08:08:00.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.402+0000 7fdc6de00640 1 --2- 172.21.15.28:0/3672844364 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fdc540a3c40 0x7fdc540a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:00.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.403+0000 7fdc6de00640 1 --2- 172.21.15.28:0/3672844364 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fdc540a3c40 0x7fdc540a6100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdc68000f80 tx=0x7fdc68005f30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:00.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.406+0000 7fdc66a00640 1 -- 172.21.15.28:0/3672844364 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc6009d030 con 0x7fdc70105e30 2024-04-24T08:08:00.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:00 smithi194 ceph-mon[32836]: from='client.24709 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:08:00.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:00 smithi194 ceph-mon[32836]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-24T08:08:00.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:00 smithi194 ceph-mon[32836]: from='client.24715 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:08:00.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:00 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3312739288' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:08:00.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:00 smithi028 ceph-mon[27786]: from='client.24709 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:08:00.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:00 smithi028 ceph-mon[27786]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-24T08:08:00.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:00 smithi028 ceph-mon[27786]: from='client.24715 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:08:00.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:00 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3312739288' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:08:00.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.612+0000 7fdc74a00640 1 -- 172.21.15.28:0/3672844364 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdc70062c00 con 0x7fdc70105e30 2024-04-24T08:08:00.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.614+0000 7fdc66a00640 1 -- 172.21.15.28:0/3672844364 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22207 (secure 0 0 0) 0x7fdc6009c930 con 0x7fdc70105e30 2024-04-24T08:08:00.615 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:08:00.616 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":86,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:07:53.034032+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:00.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.617+0000 7fdc74a00640 1 -- 172.21.15.28:0/3672844364 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fdc540a3c40 msgr2=0x7fdc540a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:00.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.617+0000 7fdc74a00640 1 --2- 172.21.15.28:0/3672844364 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fdc540a3c40 0x7fdc540a6100 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7fdc68000f80 tx=0x7fdc68005f30 comp rx=0 tx=0).stop 2024-04-24T08:08:00.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.617+0000 7fdc74a00640 1 -- 172.21.15.28:0/3672844364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc70105e30 msgr2=0x7fdc701a1b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:00.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.617+0000 7fdc74a00640 1 --2- 172.21.15.28:0/3672844364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc70105e30 0x7fdc701a1b90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fdc6000e970 tx=0x7fdc6000ee30 comp rx=0 tx=0).stop 2024-04-24T08:08:00.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.617+0000 7fdc74a00640 1 -- 172.21.15.28:0/3672844364 shutdown_connections 2024-04-24T08:08:00.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.617+0000 7fdc74a00640 1 --2- 172.21.15.28:0/3672844364 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fdc540a3c40 0x7fdc540a6100 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.617+0000 7fdc74a00640 1 --2- 172.21.15.28:0/3672844364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc70105470 0x7fdc7019ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.617+0000 7fdc74a00640 1 --2- 172.21.15.28:0/3672844364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc70105e30 0x7fdc701a1b90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.617+0000 7fdc74a00640 1 --2- 172.21.15.28:0/3672844364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc70104280 0x7fdc7019a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.617+0000 7fdc74a00640 1 -- 172.21.15.28:0/3672844364 >> 172.21.15.28:0/3672844364 conn(0x7fdc700ff7f0 msgr2=0x7fdc70108770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:00.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.618+0000 7fdc74a00640 1 -- 172.21.15.28:0/3672844364 shutdown_connections 2024-04-24T08:08:00.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.618+0000 7fdc74a00640 1 -- 172.21.15.28:0/3672844364 wait complete. 2024-04-24T08:08:00.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:00 smithi117 ceph-mon[32438]: from='client.24709 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:08:00.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:00 smithi117 ceph-mon[32438]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-04-24T08:08:00.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:00 smithi117 ceph-mon[32438]: from='client.24715 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:08:00.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:00 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3312739288' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:08:00.639 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-04-24T08:08:00.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:08:00.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.856+0000 7f067ae00640 1 -- 172.21.15.28:0/3838548798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0674102900 msgr2=0x7f0674104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:00.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.856+0000 7f067ae00640 1 --2- 172.21.15.28:0/3838548798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0674102900 0x7f0674104cf0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f0668009900 tx=0x7f066802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.856+0000 7f067ae00640 1 -- 172.21.15.28:0/3838548798 shutdown_connections 2024-04-24T08:08:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.856+0000 7f067ae00640 1 --2- 172.21.15.28:0/3838548798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0674107b60 0x7f0674109fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.856+0000 7f067ae00640 1 --2- 172.21.15.28:0/3838548798 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0674105230 0x7f0674107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.856+0000 7f067ae00640 1 --2- 172.21.15.28:0/3838548798 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0674102900 0x7f0674104cf0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.856+0000 7f067ae00640 1 -- 172.21.15.28:0/3838548798 >> 172.21.15.28:0/3838548798 conn(0x7f06740fc310 msgr2=0x7f06740fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.856+0000 7f067ae00640 1 -- 172.21.15.28:0/3838548798 shutdown_connections 2024-04-24T08:08:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.856+0000 7f067ae00640 1 -- 172.21.15.28:0/3838548798 wait complete. 2024-04-24T08:08:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.857+0000 7f067ae00640 1 Processor -- start 2024-04-24T08:08:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.857+0000 7f067ae00640 1 -- start start 2024-04-24T08:08:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.857+0000 7f067ae00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0674102900 0x7f067419a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.857+0000 7f067ae00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0674105230 0x7f067419a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.857+0000 7f067ae00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0674107b60 0x7f06741a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.857+0000 7f067ae00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f067410db80 con 0x7f0674105230 2024-04-24T08:08:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.857+0000 7f067ae00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f067410da00 con 0x7f0674102900 2024-04-24T08:08:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.857+0000 7f067ae00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f067410dd00 con 0x7f0674107b60 2024-04-24T08:08:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.858+0000 7f0679000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0674105230 0x7f067419a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.858+0000 7f0679a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0674102900 0x7f067419a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.858+0000 7f0679000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0674105230 0x7f067419a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56722/0 (socket says 172.21.15.28:56722) 2024-04-24T08:08:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.858+0000 7f0679000640 1 -- 172.21.15.28:0/3264443353 learned_addr learned my addr 172.21.15.28:0/3264443353 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.858+0000 7f067a400640 1 --2- 172.21.15.28:0/3264443353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0674107b60 0x7f06741a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:00.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.858+0000 7f0679000640 1 -- 172.21.15.28:0/3264443353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0674107b60 msgr2=0x7f06741a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.858+0000 7f0679000640 1 --2- 172.21.15.28:0/3264443353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0674107b60 0x7f06741a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.858+0000 7f0679000640 1 -- 172.21.15.28:0/3264443353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0674102900 msgr2=0x7f067419a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.858+0000 7f0679000640 1 --2- 172.21.15.28:0/3264443353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0674102900 0x7f067419a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.858+0000 7f0679000640 1 -- 172.21.15.28:0/3264443353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06741a2010 con 0x7f0674105230 2024-04-24T08:08:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.858+0000 7f067a400640 1 --2- 172.21.15.28:0/3264443353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0674107b60 0x7f06741a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:08:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.858+0000 7f0679000640 1 --2- 172.21.15.28:0/3264443353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0674105230 0x7f067419a930 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f066000e9f0 tx=0x7f066000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.859+0000 7f0672000640 1 -- 172.21.15.28:0/3264443353 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f066000ccc0 con 0x7f0674105230 2024-04-24T08:08:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.859+0000 7f0672000640 1 -- 172.21.15.28:0/3264443353 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0660005500 con 0x7f0674105230 2024-04-24T08:08:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.859+0000 7f0672000640 1 -- 172.21.15.28:0/3264443353 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0660010690 con 0x7f0674105230 2024-04-24T08:08:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.859+0000 7f067ae00640 1 -- 172.21.15.28:0/3264443353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06741a2300 con 0x7f0674105230 2024-04-24T08:08:00.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.859+0000 7f067ae00640 1 -- 172.21.15.28:0/3264443353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06741a2850 con 0x7f0674105230 2024-04-24T08:08:00.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.860+0000 7f067ae00640 1 -- 172.21.15.28:0/3264443353 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f06741096b0 con 0x7f0674105230 2024-04-24T08:08:00.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.862+0000 7f0672000640 1 -- 172.21.15.28:0/3264443353 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f06600046c0 con 0x7f0674105230 2024-04-24T08:08:00.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.863+0000 7f0672000640 1 --2- 172.21.15.28:0/3264443353 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f065c0a3cc0 0x7f065c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:00.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.864+0000 7f0672000640 1 -- 172.21.15.28:0/3264443353 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f06600d0860 con 0x7f0674105230 2024-04-24T08:08:00.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.864+0000 7f0679a00640 1 --2- 172.21.15.28:0/3264443353 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f065c0a3cc0 0x7f065c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:00.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.864+0000 7f0679a00640 1 --2- 172.21.15.28:0/3264443353 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f065c0a3cc0 0x7f065c0a6180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0668031440 tx=0x7f06680315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:00.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:00.868+0000 7f0672000640 1 -- 172.21.15.28:0/3264443353 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f066009d190 con 0x7f0674105230 2024-04-24T08:08:01.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.068+0000 7f067ae00640 1 -- 172.21.15.28:0/3264443353 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0674062440 con 0x7f065c0a3cc0 2024-04-24T08:08:01.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.084+0000 7f0672000640 1 -- 172.21.15.28:0/3264443353 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258853 (secure 0 0 0) 0x7f0674062440 con 0x7f065c0a3cc0 2024-04-24T08:08:01.090 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:08:01.092 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-24T08:08:01.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.090+0000 7f067ae00640 1 -- 172.21.15.28:0/3264443353 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f065c0a3cc0 msgr2=0x7f065c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:01.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.090+0000 7f067ae00640 1 --2- 172.21.15.28:0/3264443353 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f065c0a3cc0 0x7f065c0a6180 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f0668031440 tx=0x7f06680315b0 comp rx=0 tx=0).stop 2024-04-24T08:08:01.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.090+0000 7f067ae00640 1 -- 172.21.15.28:0/3264443353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0674105230 msgr2=0x7f067419a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:01.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.090+0000 7f067ae00640 1 --2- 172.21.15.28:0/3264443353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0674105230 0x7f067419a930 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f066000e9f0 tx=0x7f066000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:08:01.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.090+0000 7f067ae00640 1 -- 172.21.15.28:0/3264443353 shutdown_connections 2024-04-24T08:08:01.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.090+0000 7f067ae00640 1 --2- 172.21.15.28:0/3264443353 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f065c0a3cc0 0x7f065c0a6180 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:01.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.090+0000 7f067ae00640 1 --2- 172.21.15.28:0/3264443353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0674105230 0x7f067419a930 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:01.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.090+0000 7f067ae00640 1 --2- 172.21.15.28:0/3264443353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0674107b60 0x7f06741a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:01.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.090+0000 7f067ae00640 1 --2- 172.21.15.28:0/3264443353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0674102900 0x7f067419a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:01.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.090+0000 7f067ae00640 1 -- 172.21.15.28:0/3264443353 >> 172.21.15.28:0/3264443353 conn(0x7f06740fc310 msgr2=0x7f0674102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:01.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.090+0000 7f067ae00640 1 -- 172.21.15.28:0/3264443353 shutdown_connections 2024-04-24T08:08:01.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.091+0000 7f067ae00640 1 -- 172.21.15.28:0/3264443353 wait complete. 2024-04-24T08:08:01.111 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-04-24T08:08:00.060821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596260,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":161,"num_read_kb":139,"num_write":297,"num_write_kb":2970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":284,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360408,"kb_used_data":11236,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581416,"statfs":{"total":1151940427776,"available":1151571369984,"internally_reserved":0,"allocated":11505664,"data_stored":7142257,"data_compressed":108294,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19140,"internal_metadata":357414204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5892,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":25,"num_read_kb":20,"num_write":116,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.026435"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884280+0000","last_change":"2024-04-24T08:07:52.074319+0000","last_active":"2024-04-24T08:07:54.884280+0000","last_peered":"2024-04-24T08:07:54.884280+0000","last_clean":"2024-04-24T08:07:54.884280+0000","last_became_active":"2024-04-24T08:07:52.073923+0000","last_became_peered":"2024-04-24T08:07:52.073923+0000","last_unstale":"2024-04-24T08:07:54.884280+0000","last_undegraded":"2024-04-24T08:07:54.884280+0000","last_fullsized":"2024-04-24T08:07:54.884280+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:29.325904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884996+0000","last_change":"2024-04-24T08:07:52.068316+0000","last_active":"2024-04-24T08:07:54.884996+0000","last_peered":"2024-04-24T08:07:54.884996+0000","last_clean":"2024-04-24T08:07:54.884996+0000","last_became_active":"2024-04-24T08:07:52.067961+0000","last_became_peered":"2024-04-24T08:07:52.067961+0000","last_unstale":"2024-04-24T08:07:54.884996+0000","last_undegraded":"2024-04-24T08:07:54.884996+0000","last_fullsized":"2024-04-24T08:07:54.884996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:35:14.079348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884658+0000","last_change":"2024-04-24T08:07:52.066870+0000","last_active":"2024-04-24T08:07:54.884658+0000","last_peered":"2024-04-24T08:07:54.884658+0000","last_clean":"2024-04-24T08:07:54.884658+0000","last_became_active":"2024-04-24T08:07:52.066481+0000","last_became_peered":"2024-04-24T08:07:52.066481+0000","last_unstale":"2024-04-24T08:07:54.884658+0000","last_undegraded":"2024-04-24T08:07:54.884658+0000","last_fullsized":"2024-04-24T08:07:54.884658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:25:45.526667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.906627+0000","last_change":"2024-04-24T08:07:52.061548+0000","last_active":"2024-04-24T08:07:53.906627+0000","last_peered":"2024-04-24T08:07:53.906627+0000","last_clean":"2024-04-24T08:07:53.906627+0000","last_became_active":"2024-04-24T08:07:52.060519+0000","last_became_peered":"2024-04-24T08:07:52.060519+0000","last_unstale":"2024-04-24T08:07:53.906627+0000","last_undegraded":"2024-04-24T08:07:53.906627+0000","last_fullsized":"2024-04-24T08:07:53.906627+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:47:20.952035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.088523+0000","last_change":"2024-04-24T08:07:52.063805+0000","last_active":"2024-04-24T08:07:53.088523+0000","last_peered":"2024-04-24T08:07:53.088523+0000","last_clean":"2024-04-24T08:07:53.088523+0000","last_became_active":"2024-04-24T08:07:52.063414+0000","last_became_peered":"2024-04-24T08:07:52.063414+0000","last_unstale":"2024-04-24T08:07:53.088523+0000","last_undegraded":"2024-04-24T08:07:53.088523+0000","last_fullsized":"2024-04-24T08:07:53.088523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:06:24.465035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884525+0000","last_change":"2024-04-24T08:07:52.050152+0000","last_active":"2024-04-24T08:07:54.884525+0000","last_peered":"2024-04-24T08:07:54.884525+0000","last_clean":"2024-04-24T08:07:54.884525+0000","last_became_active":"2024-04-24T08:07:52.049805+0000","last_became_peered":"2024-04-24T08:07:52.049805+0000","last_unstale":"2024-04-24T08:07:54.884525+0000","last_undegraded":"2024-04-24T08:07:54.884525+0000","last_fullsized":"2024-04-24T08:07:54.884525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:47:34.644263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884128+0000","last_change":"2024-04-24T08:07:52.055726+0000","last_active":"2024-04-24T08:07:54.884128+0000","last_peered":"2024-04-24T08:07:54.884128+0000","last_clean":"2024-04-24T08:07:54.884128+0000","last_became_active":"2024-04-24T08:07:52.054563+0000","last_became_peered":"2024-04-24T08:07:52.054563+0000","last_unstale":"2024-04-24T08:07:54.884128+0000","last_undegraded":"2024-04-24T08:07:54.884128+0000","last_fullsized":"2024-04-24T08:07:54.884128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:34:57.430232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.043631+0000","last_change":"2024-04-24T08:07:52.053672+0000","last_active":"2024-04-24T08:07:53.043631+0000","last_peered":"2024-04-24T08:07:53.043631+0000","last_clean":"2024-04-24T08:07:53.043631+0000","last_became_active":"2024-04-24T08:07:52.053150+0000","last_became_peered":"2024-04-24T08:07:52.053150+0000","last_unstale":"2024-04-24T08:07:53.043631+0000","last_undegraded":"2024-04-24T08:07:53.043631+0000","last_fullsized":"2024-04-24T08:07:53.043631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:00:12.272555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.874420+0000","last_change":"2024-04-24T08:07:52.054648+0000","last_active":"2024-04-24T08:07:53.874420+0000","last_peered":"2024-04-24T08:07:53.874420+0000","last_clean":"2024-04-24T08:07:53.874420+0000","last_became_active":"2024-04-24T08:07:52.054301+0000","last_became_peered":"2024-04-24T08:07:52.054301+0000","last_unstale":"2024-04-24T08:07:53.874420+0000","last_undegraded":"2024-04-24T08:07:53.874420+0000","last_fullsized":"2024-04-24T08:07:53.874420+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:52:20.533855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884997+0000","last_change":"2024-04-24T08:07:52.052994+0000","last_active":"2024-04-24T08:07:54.884997+0000","last_peered":"2024-04-24T08:07:54.884997+0000","last_clean":"2024-04-24T08:07:54.884997+0000","last_became_active":"2024-04-24T08:07:52.052641+0000","last_became_peered":"2024-04-24T08:07:52.052641+0000","last_unstale":"2024-04-24T08:07:54.884997+0000","last_undegraded":"2024-04-24T08:07:54.884997+0000","last_fullsized":"2024-04-24T08:07:54.884997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:02.665622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.883970+0000","last_change":"2024-04-24T08:07:52.055901+0000","last_active":"2024-04-24T08:07:54.883970+0000","last_peered":"2024-04-24T08:07:54.883970+0000","last_clean":"2024-04-24T08:07:54.883970+0000","last_became_active":"2024-04-24T08:07:52.055277+0000","last_became_peered":"2024-04-24T08:07:52.055277+0000","last_unstale":"2024-04-24T08:07:54.883970+0000","last_undegraded":"2024-04-24T08:07:54.883970+0000","last_fullsized":"2024-04-24T08:07:54.883970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:46.950578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.906605+0000","last_change":"2024-04-24T08:07:52.060329+0000","last_active":"2024-04-24T08:07:53.906605+0000","last_peered":"2024-04-24T08:07:53.906605+0000","last_clean":"2024-04-24T08:07:53.906605+0000","last_became_active":"2024-04-24T08:07:52.059914+0000","last_became_peered":"2024-04-24T08:07:52.059914+0000","last_unstale":"2024-04-24T08:07:53.906605+0000","last_undegraded":"2024-04-24T08:07:53.906605+0000","last_fullsized":"2024-04-24T08:07:53.906605+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:15:20.263916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.144525+0000","last_change":"2024-04-24T08:07:52.062108+0000","last_active":"2024-04-24T08:07:53.144525+0000","last_peered":"2024-04-24T08:07:53.144525+0000","last_clean":"2024-04-24T08:07:53.144525+0000","last_became_active":"2024-04-24T08:07:52.061665+0000","last_became_peered":"2024-04-24T08:07:52.061665+0000","last_unstale":"2024-04-24T08:07:53.144525+0000","last_undegraded":"2024-04-24T08:07:53.144525+0000","last_fullsized":"2024-04-24T08:07:53.144525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:55:58.071807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884654+0000","last_change":"2024-04-24T08:07:52.063033+0000","last_active":"2024-04-24T08:07:54.884654+0000","last_peered":"2024-04-24T08:07:54.884654+0000","last_clean":"2024-04-24T08:07:54.884654+0000","last_became_active":"2024-04-24T08:07:52.062610+0000","last_became_peered":"2024-04-24T08:07:52.062610+0000","last_unstale":"2024-04-24T08:07:54.884654+0000","last_undegraded":"2024-04-24T08:07:54.884654+0000","last_fullsized":"2024-04-24T08:07:54.884654+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:59:35.906720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884363+0000","last_change":"2024-04-24T08:07:52.064657+0000","last_active":"2024-04-24T08:07:54.884363+0000","last_peered":"2024-04-24T08:07:54.884363+0000","last_clean":"2024-04-24T08:07:54.884363+0000","last_became_active":"2024-04-24T08:07:52.064188+0000","last_became_peered":"2024-04-24T08:07:52.064188+0000","last_unstale":"2024-04-24T08:07:54.884363+0000","last_undegraded":"2024-04-24T08:07:54.884363+0000","last_fullsized":"2024-04-24T08:07:54.884363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:56:32.139420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.874066+0000","last_change":"2024-04-24T08:07:52.067227+0000","last_active":"2024-04-24T08:07:53.874066+0000","last_peered":"2024-04-24T08:07:53.874066+0000","last_clean":"2024-04-24T08:07:53.874066+0000","last_became_active":"2024-04-24T08:07:52.066854+0000","last_became_peered":"2024-04-24T08:07:52.066854+0000","last_unstale":"2024-04-24T08:07:53.874066+0000","last_undegraded":"2024-04-24T08:07:53.874066+0000","last_fullsized":"2024-04-24T08:07:53.874066+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:25:17.580223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884564+0000","last_change":"2024-04-24T08:07:52.074127+0000","last_active":"2024-04-24T08:07:54.884564+0000","last_peered":"2024-04-24T08:07:54.884564+0000","last_clean":"2024-04-24T08:07:54.884564+0000","last_became_active":"2024-04-24T08:07:52.073806+0000","last_became_peered":"2024-04-24T08:07:52.073806+0000","last_unstale":"2024-04-24T08:07:54.884564+0000","last_undegraded":"2024-04-24T08:07:54.884564+0000","last_fullsized":"2024-04-24T08:07:54.884564+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:25:22.068997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.043190+0000","last_change":"2024-04-24T08:07:52.051239+0000","last_active":"2024-04-24T08:07:53.043190+0000","last_peered":"2024-04-24T08:07:53.043190+0000","last_clean":"2024-04-24T08:07:53.043190+0000","last_became_active":"2024-04-24T08:07:52.049232+0000","last_became_peered":"2024-04-24T08:07:52.049232+0000","last_unstale":"2024-04-24T08:07:53.043190+0000","last_undegraded":"2024-04-24T08:07:53.043190+0000","last_fullsized":"2024-04-24T08:07:53.043190+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:07:39.768537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.042466+0000","last_change":"2024-04-24T08:07:52.062119+0000","last_active":"2024-04-24T08:07:53.042466+0000","last_peered":"2024-04-24T08:07:53.042466+0000","last_clean":"2024-04-24T08:07:53.042466+0000","last_became_active":"2024-04-24T08:07:52.061781+0000","last_became_peered":"2024-04-24T08:07:52.061781+0000","last_unstale":"2024-04-24T08:07:53.042466+0000","last_undegraded":"2024-04-24T08:07:53.042466+0000","last_fullsized":"2024-04-24T08:07:53.042466+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:49:41.595368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:56.143066+0000","last_change":"2024-04-24T08:07:51.052411+0000","last_active":"2024-04-24T08:07:56.143066+0000","last_peered":"2024-04-24T08:07:56.143066+0000","last_clean":"2024-04-24T08:07:56.143066+0000","last_became_active":"2024-04-24T08:07:51.051704+0000","last_became_peered":"2024-04-24T08:07:51.051704+0000","last_unstale":"2024-04-24T08:07:56.143066+0000","last_undegraded":"2024-04-24T08:07:56.143066+0000","last_fullsized":"2024-04-24T08:07:56.143066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:34:23.001338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.044550+0000","last_change":"2024-04-24T08:07:52.056508+0000","last_active":"2024-04-24T08:07:53.044550+0000","last_peered":"2024-04-24T08:07:53.044550+0000","last_clean":"2024-04-24T08:07:53.044550+0000","last_became_active":"2024-04-24T08:07:52.056158+0000","last_became_peered":"2024-04-24T08:07:52.056158+0000","last_unstale":"2024-04-24T08:07:53.044550+0000","last_undegraded":"2024-04-24T08:07:53.044550+0000","last_fullsized":"2024-04-24T08:07:53.044550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:52.267984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.044719+0000","last_change":"2024-04-24T08:07:51.080403+0000","last_active":"2024-04-24T08:07:53.044719+0000","last_peered":"2024-04-24T08:07:53.044719+0000","last_clean":"2024-04-24T08:07:53.044719+0000","last_became_active":"2024-04-24T08:07:51.079158+0000","last_became_peered":"2024-04-24T08:07:51.079158+0000","last_unstale":"2024-04-24T08:07:53.044719+0000","last_undegraded":"2024-04-24T08:07:53.044719+0000","last_fullsized":"2024-04-24T08:07:53.044719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:49:59.491850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.885525+0000","last_change":"2024-04-24T08:07:52.066963+0000","last_active":"2024-04-24T08:07:54.885525+0000","last_peered":"2024-04-24T08:07:54.885525+0000","last_clean":"2024-04-24T08:07:54.885525+0000","last_became_active":"2024-04-24T08:07:52.065475+0000","last_became_peered":"2024-04-24T08:07:52.065475+0000","last_unstale":"2024-04-24T08:07:54.885525+0000","last_undegraded":"2024-04-24T08:07:54.885525+0000","last_fullsized":"2024-04-24T08:07:54.885525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:28:34.220200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.088843+0000","last_change":"2024-04-24T08:07:51.042772+0000","last_active":"2024-04-24T08:07:53.088843+0000","last_peered":"2024-04-24T08:07:53.088843+0000","last_clean":"2024-04-24T08:07:53.088843+0000","last_became_active":"2024-04-24T08:07:51.042128+0000","last_became_peered":"2024-04-24T08:07:51.042128+0000","last_unstale":"2024-04-24T08:07:53.088843+0000","last_undegraded":"2024-04-24T08:07:53.088843+0000","last_fullsized":"2024-04-24T08:07:53.088843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:10:29.207991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.874067+0000","last_change":"2024-04-24T08:07:52.062411+0000","last_active":"2024-04-24T08:07:53.874067+0000","last_peered":"2024-04-24T08:07:53.874067+0000","last_clean":"2024-04-24T08:07:53.874067+0000","last_became_active":"2024-04-24T08:07:52.062037+0000","last_became_peered":"2024-04-24T08:07:52.062037+0000","last_unstale":"2024-04-24T08:07:53.874067+0000","last_undegraded":"2024-04-24T08:07:53.874067+0000","last_fullsized":"2024-04-24T08:07:53.874067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:16:17.463874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.088581+0000","last_change":"2024-04-24T08:07:51.046046+0000","last_active":"2024-04-24T08:07:53.088581+0000","last_peered":"2024-04-24T08:07:53.088581+0000","last_clean":"2024-04-24T08:07:53.088581+0000","last_became_active":"2024-04-24T08:07:51.045633+0000","last_became_peered":"2024-04-24T08:07:51.045633+0000","last_unstale":"2024-04-24T08:07:53.088581+0000","last_undegraded":"2024-04-24T08:07:53.088581+0000","last_fullsized":"2024-04-24T08:07:53.088581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:18:20.529234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884911+0000","last_change":"2024-04-24T08:07:52.061167+0000","last_active":"2024-04-24T08:07:54.884911+0000","last_peered":"2024-04-24T08:07:54.884911+0000","last_clean":"2024-04-24T08:07:54.884911+0000","last_became_active":"2024-04-24T08:07:52.060841+0000","last_became_peered":"2024-04-24T08:07:52.060841+0000","last_unstale":"2024-04-24T08:07:54.884911+0000","last_undegraded":"2024-04-24T08:07:54.884911+0000","last_fullsized":"2024-04-24T08:07:54.884911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:04:27.048760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:58.146198+0000","last_change":"2024-04-24T08:07:51.044274+0000","last_active":"2024-04-24T08:07:58.146198+0000","last_peered":"2024-04-24T08:07:58.146198+0000","last_clean":"2024-04-24T08:07:58.146198+0000","last_became_active":"2024-04-24T08:07:51.043938+0000","last_became_peered":"2024-04-24T08:07:51.043938+0000","last_unstale":"2024-04-24T08:07:58.146198+0000","last_undegraded":"2024-04-24T08:07:58.146198+0000","last_fullsized":"2024-04-24T08:07:58.146198+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:05:38.349981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.088535+0000","last_change":"2024-04-24T08:07:52.055777+0000","last_active":"2024-04-24T08:07:53.088535+0000","last_peered":"2024-04-24T08:07:53.088535+0000","last_clean":"2024-04-24T08:07:53.088535+0000","last_became_active":"2024-04-24T08:07:52.055415+0000","last_became_peered":"2024-04-24T08:07:52.055415+0000","last_unstale":"2024-04-24T08:07:53.088535+0000","last_undegraded":"2024-04-24T08:07:53.088535+0000","last_fullsized":"2024-04-24T08:07:53.088535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:18:31.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:56.142317+0000","last_change":"2024-04-24T08:07:51.072513+0000","last_active":"2024-04-24T08:07:56.142317+0000","last_peered":"2024-04-24T08:07:56.142317+0000","last_clean":"2024-04-24T08:07:56.142317+0000","last_became_active":"2024-04-24T08:07:51.072249+0000","last_became_peered":"2024-04-24T08:07:51.072249+0000","last_unstale":"2024-04-24T08:07:56.142317+0000","last_undegraded":"2024-04-24T08:07:56.142317+0000","last_fullsized":"2024-04-24T08:07:56.142317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:45:07.112617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884811+0000","last_change":"2024-04-24T08:07:52.060290+0000","last_active":"2024-04-24T08:07:54.884811+0000","last_peered":"2024-04-24T08:07:54.884811+0000","last_clean":"2024-04-24T08:07:54.884811+0000","last_became_active":"2024-04-24T08:07:52.059961+0000","last_became_peered":"2024-04-24T08:07:52.059961+0000","last_unstale":"2024-04-24T08:07:54.884811+0000","last_undegraded":"2024-04-24T08:07:54.884811+0000","last_fullsized":"2024-04-24T08:07:54.884811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:10:56.885967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:56.143225+0000","last_change":"2024-04-24T08:07:51.070901+0000","last_active":"2024-04-24T08:07:56.143225+0000","last_peered":"2024-04-24T08:07:56.143225+0000","last_clean":"2024-04-24T08:07:56.143225+0000","last_became_active":"2024-04-24T08:07:51.067178+0000","last_became_peered":"2024-04-24T08:07:51.067178+0000","last_unstale":"2024-04-24T08:07:56.143225+0000","last_undegraded":"2024-04-24T08:07:56.143225+0000","last_fullsized":"2024-04-24T08:07:56.143225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:24:26.807786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.043555+0000","last_change":"2024-04-24T08:07:52.079493+0000","last_active":"2024-04-24T08:07:53.043555+0000","last_peered":"2024-04-24T08:07:53.043555+0000","last_clean":"2024-04-24T08:07:53.043555+0000","last_became_active":"2024-04-24T08:07:52.077266+0000","last_became_peered":"2024-04-24T08:07:52.077266+0000","last_unstale":"2024-04-24T08:07:53.043555+0000","last_undegraded":"2024-04-24T08:07:53.043555+0000","last_fullsized":"2024-04-24T08:07:53.043555+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:40:45.660146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884222+0000","last_change":"2024-04-24T08:07:51.045963+0000","last_active":"2024-04-24T08:07:54.884222+0000","last_peered":"2024-04-24T08:07:54.884222+0000","last_clean":"2024-04-24T08:07:54.884222+0000","last_became_active":"2024-04-24T08:07:51.045641+0000","last_became_peered":"2024-04-24T08:07:51.045641+0000","last_unstale":"2024-04-24T08:07:54.884222+0000","last_undegraded":"2024-04-24T08:07:54.884222+0000","last_fullsized":"2024-04-24T08:07:54.884222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:07:59.646155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.042378+0000","last_change":"2024-04-24T08:07:52.057280+0000","last_active":"2024-04-24T08:07:53.042378+0000","last_peered":"2024-04-24T08:07:53.042378+0000","last_clean":"2024-04-24T08:07:53.042378+0000","last_became_active":"2024-04-24T08:07:52.054944+0000","last_became_peered":"2024-04-24T08:07:52.054944+0000","last_unstale":"2024-04-24T08:07:53.042378+0000","last_undegraded":"2024-04-24T08:07:53.042378+0000","last_fullsized":"2024-04-24T08:07:53.042378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:22:45.911472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.116385+0000","last_change":"2024-04-24T08:07:51.048898+0000","last_active":"2024-04-24T08:07:55.116385+0000","last_peered":"2024-04-24T08:07:55.116385+0000","last_clean":"2024-04-24T08:07:55.116385+0000","last_became_active":"2024-04-24T08:07:51.048585+0000","last_became_peered":"2024-04-24T08:07:51.048585+0000","last_unstale":"2024-04-24T08:07:55.116385+0000","last_undegraded":"2024-04-24T08:07:55.116385+0000","last_fullsized":"2024-04-24T08:07:55.116385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:40:30.413541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.874364+0000","last_change":"2024-04-24T08:07:52.048093+0000","last_active":"2024-04-24T08:07:53.874364+0000","last_peered":"2024-04-24T08:07:53.874364+0000","last_clean":"2024-04-24T08:07:53.874364+0000","last_became_active":"2024-04-24T08:07:52.047727+0000","last_became_peered":"2024-04-24T08:07:52.047727+0000","last_unstale":"2024-04-24T08:07:53.874364+0000","last_undegraded":"2024-04-24T08:07:53.874364+0000","last_fullsized":"2024-04-24T08:07:53.874364+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:57:26.248355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.116643+0000","last_change":"2024-04-24T08:07:51.047810+0000","last_active":"2024-04-24T08:07:55.116643+0000","last_peered":"2024-04-24T08:07:55.116643+0000","last_clean":"2024-04-24T08:07:55.116643+0000","last_became_active":"2024-04-24T08:07:51.047501+0000","last_became_peered":"2024-04-24T08:07:51.047501+0000","last_unstale":"2024-04-24T08:07:55.116643+0000","last_undegraded":"2024-04-24T08:07:55.116643+0000","last_fullsized":"2024-04-24T08:07:55.116643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:02:27.585202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.885516+0000","last_change":"2024-04-24T08:07:52.063782+0000","last_active":"2024-04-24T08:07:54.885516+0000","last_peered":"2024-04-24T08:07:54.885516+0000","last_clean":"2024-04-24T08:07:54.885516+0000","last_became_active":"2024-04-24T08:07:52.063056+0000","last_became_peered":"2024-04-24T08:07:52.063056+0000","last_unstale":"2024-04-24T08:07:54.885516+0000","last_undegraded":"2024-04-24T08:07:54.885516+0000","last_fullsized":"2024-04-24T08:07:54.885516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:41:30.966787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.113012+0000","last_change":"2024-04-24T08:07:51.076139+0000","last_active":"2024-04-24T08:07:55.113012+0000","last_peered":"2024-04-24T08:07:55.113012+0000","last_clean":"2024-04-24T08:07:55.113012+0000","last_became_active":"2024-04-24T08:07:51.075842+0000","last_became_peered":"2024-04-24T08:07:51.075842+0000","last_unstale":"2024-04-24T08:07:55.113012+0000","last_undegraded":"2024-04-24T08:07:55.113012+0000","last_fullsized":"2024-04-24T08:07:55.113012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:46:56.741726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.874272+0000","last_change":"2024-04-24T08:07:52.054058+0000","last_active":"2024-04-24T08:07:53.874272+0000","last_peered":"2024-04-24T08:07:53.874272+0000","last_clean":"2024-04-24T08:07:53.874272+0000","last_became_active":"2024-04-24T08:07:52.053142+0000","last_became_peered":"2024-04-24T08:07:52.053142+0000","last_unstale":"2024-04-24T08:07:53.874272+0000","last_undegraded":"2024-04-24T08:07:53.874272+0000","last_fullsized":"2024-04-24T08:07:53.874272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:44:43.554555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:57.144251+0000","last_change":"2024-04-24T08:07:51.048079+0000","last_active":"2024-04-24T08:07:57.144251+0000","last_peered":"2024-04-24T08:07:57.144251+0000","last_clean":"2024-04-24T08:07:57.144251+0000","last_became_active":"2024-04-24T08:07:51.047584+0000","last_became_peered":"2024-04-24T08:07:51.047584+0000","last_unstale":"2024-04-24T08:07:57.144251+0000","last_undegraded":"2024-04-24T08:07:57.144251+0000","last_fullsized":"2024-04-24T08:07:57.144251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:09:05.554984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.088770+0000","last_change":"2024-04-24T08:07:52.055894+0000","last_active":"2024-04-24T08:07:53.088770+0000","last_peered":"2024-04-24T08:07:53.088770+0000","last_clean":"2024-04-24T08:07:53.088770+0000","last_became_active":"2024-04-24T08:07:52.055548+0000","last_became_peered":"2024-04-24T08:07:52.055548+0000","last_unstale":"2024-04-24T08:07:53.088770+0000","last_undegraded":"2024-04-24T08:07:53.088770+0000","last_fullsized":"2024-04-24T08:07:53.088770+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:33:22.609865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:56.139953+0000","last_change":"2024-04-24T08:07:51.073196+0000","last_active":"2024-04-24T08:07:56.139953+0000","last_peered":"2024-04-24T08:07:56.139953+0000","last_clean":"2024-04-24T08:07:56.139953+0000","last_became_active":"2024-04-24T08:07:51.072893+0000","last_became_peered":"2024-04-24T08:07:51.072893+0000","last_unstale":"2024-04-24T08:07:56.139953+0000","last_undegraded":"2024-04-24T08:07:56.139953+0000","last_fullsized":"2024-04-24T08:07:56.139953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:18:07.826148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.144228+0000","last_change":"2024-04-24T08:07:52.069950+0000","last_active":"2024-04-24T08:07:53.144228+0000","last_peered":"2024-04-24T08:07:53.144228+0000","last_clean":"2024-04-24T08:07:53.144228+0000","last_became_active":"2024-04-24T08:07:52.069515+0000","last_became_peered":"2024-04-24T08:07:52.069515+0000","last_unstale":"2024-04-24T08:07:53.144228+0000","last_undegraded":"2024-04-24T08:07:53.144228+0000","last_fullsized":"2024-04-24T08:07:53.144228+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:52:27.530541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.114769+0000","last_change":"2024-04-24T08:07:51.067062+0000","last_active":"2024-04-24T08:07:55.114769+0000","last_peered":"2024-04-24T08:07:55.114769+0000","last_clean":"2024-04-24T08:07:55.114769+0000","last_became_active":"2024-04-24T08:07:51.066712+0000","last_became_peered":"2024-04-24T08:07:51.066712+0000","last_unstale":"2024-04-24T08:07:55.114769+0000","last_undegraded":"2024-04-24T08:07:55.114769+0000","last_fullsized":"2024-04-24T08:07:55.114769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:11:52.974470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884335+0000","last_change":"2024-04-24T08:07:52.064521+0000","last_active":"2024-04-24T08:07:54.884335+0000","last_peered":"2024-04-24T08:07:54.884335+0000","last_clean":"2024-04-24T08:07:54.884335+0000","last_became_active":"2024-04-24T08:07:52.064184+0000","last_became_peered":"2024-04-24T08:07:52.064184+0000","last_unstale":"2024-04-24T08:07:54.884335+0000","last_undegraded":"2024-04-24T08:07:54.884335+0000","last_fullsized":"2024-04-24T08:07:54.884335+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:00:27.294441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:58.148164+0000","last_change":"2024-04-24T08:07:51.043577+0000","last_active":"2024-04-24T08:07:58.148164+0000","last_peered":"2024-04-24T08:07:58.148164+0000","last_clean":"2024-04-24T08:07:58.148164+0000","last_became_active":"2024-04-24T08:07:51.043215+0000","last_became_peered":"2024-04-24T08:07:51.043215+0000","last_unstale":"2024-04-24T08:07:58.148164+0000","last_undegraded":"2024-04-24T08:07:58.148164+0000","last_fullsized":"2024-04-24T08:07:58.148164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:25:09.979236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.883646+0000","last_change":"2024-04-24T08:07:52.057633+0000","last_active":"2024-04-24T08:07:54.883646+0000","last_peered":"2024-04-24T08:07:54.883646+0000","last_clean":"2024-04-24T08:07:54.883646+0000","last_became_active":"2024-04-24T08:07:52.057309+0000","last_became_peered":"2024-04-24T08:07:52.057309+0000","last_unstale":"2024-04-24T08:07:54.883646+0000","last_undegraded":"2024-04-24T08:07:54.883646+0000","last_fullsized":"2024-04-24T08:07:54.883646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:49:56.045030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'10","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.119424+0000","last_change":"2024-04-24T08:07:51.058957+0000","last_active":"2024-04-24T08:07:55.119424+0000","last_peered":"2024-04-24T08:07:55.119424+0000","last_clean":"2024-04-24T08:07:55.119424+0000","last_became_active":"2024-04-24T08:07:51.058654+0000","last_became_peered":"2024-04-24T08:07:51.058654+0000","last_unstale":"2024-04-24T08:07:55.119424+0000","last_undegraded":"2024-04-24T08:07:55.119424+0000","last_fullsized":"2024-04-24T08:07:55.119424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:13.354590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884392+0000","last_change":"2024-04-24T08:07:52.051052+0000","last_active":"2024-04-24T08:07:54.884392+0000","last_peered":"2024-04-24T08:07:54.884392+0000","last_clean":"2024-04-24T08:07:54.884392+0000","last_became_active":"2024-04-24T08:07:52.050685+0000","last_became_peered":"2024-04-24T08:07:52.050685+0000","last_unstale":"2024-04-24T08:07:54.884392+0000","last_undegraded":"2024-04-24T08:07:54.884392+0000","last_fullsized":"2024-04-24T08:07:54.884392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:34.491653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.088809+0000","last_change":"2024-04-24T08:07:51.041088+0000","last_active":"2024-04-24T08:07:53.088809+0000","last_peered":"2024-04-24T08:07:53.088809+0000","last_clean":"2024-04-24T08:07:53.088809+0000","last_became_active":"2024-04-24T08:07:51.040752+0000","last_became_peered":"2024-04-24T08:07:51.040752+0000","last_unstale":"2024-04-24T08:07:53.088809+0000","last_undegraded":"2024-04-24T08:07:53.088809+0000","last_fullsized":"2024-04-24T08:07:53.088809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:48:23.405705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884323+0000","last_change":"2024-04-24T08:07:52.058016+0000","last_active":"2024-04-24T08:07:54.884323+0000","last_peered":"2024-04-24T08:07:54.884323+0000","last_clean":"2024-04-24T08:07:54.884323+0000","last_became_active":"2024-04-24T08:07:52.057621+0000","last_became_peered":"2024-04-24T08:07:52.057621+0000","last_unstale":"2024-04-24T08:07:54.884323+0000","last_undegraded":"2024-04-24T08:07:54.884323+0000","last_fullsized":"2024-04-24T08:07:54.884323+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:57:40.651658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.042997+0000","last_change":"2024-04-24T08:07:51.075583+0000","last_active":"2024-04-24T08:07:53.042997+0000","last_peered":"2024-04-24T08:07:53.042997+0000","last_clean":"2024-04-24T08:07:53.042997+0000","last_became_active":"2024-04-24T08:07:51.075111+0000","last_became_peered":"2024-04-24T08:07:51.075111+0000","last_unstale":"2024-04-24T08:07:53.042997+0000","last_undegraded":"2024-04-24T08:07:53.042997+0000","last_fullsized":"2024-04-24T08:07:53.042997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:56:00.820057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884938+0000","last_change":"2024-04-24T08:07:52.063232+0000","last_active":"2024-04-24T08:07:54.884938+0000","last_peered":"2024-04-24T08:07:54.884938+0000","last_clean":"2024-04-24T08:07:54.884938+0000","last_became_active":"2024-04-24T08:07:52.062868+0000","last_became_peered":"2024-04-24T08:07:52.062868+0000","last_unstale":"2024-04-24T08:07:54.884938+0000","last_undegraded":"2024-04-24T08:07:54.884938+0000","last_fullsized":"2024-04-24T08:07:54.884938+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:00:40.333389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.883681+0000","last_change":"2024-04-24T08:07:51.070647+0000","last_active":"2024-04-24T08:07:54.883681+0000","last_peered":"2024-04-24T08:07:54.883681+0000","last_clean":"2024-04-24T08:07:54.883681+0000","last_became_active":"2024-04-24T08:07:51.070298+0000","last_became_peered":"2024-04-24T08:07:51.070298+0000","last_unstale":"2024-04-24T08:07:54.883681+0000","last_undegraded":"2024-04-24T08:07:54.883681+0000","last_fullsized":"2024-04-24T08:07:54.883681+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:56:41.706796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884779+0000","last_change":"2024-04-24T08:07:52.073417+0000","last_active":"2024-04-24T08:07:54.884779+0000","last_peered":"2024-04-24T08:07:54.884779+0000","last_clean":"2024-04-24T08:07:54.884779+0000","last_became_active":"2024-04-24T08:07:52.073051+0000","last_became_peered":"2024-04-24T08:07:52.073051+0000","last_unstale":"2024-04-24T08:07:54.884779+0000","last_undegraded":"2024-04-24T08:07:54.884779+0000","last_fullsized":"2024-04-24T08:07:54.884779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:54:58.338279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884334+0000","last_change":"2024-04-24T08:07:52.060497+0000","last_active":"2024-04-24T08:07:54.884334+0000","last_peered":"2024-04-24T08:07:54.884334+0000","last_clean":"2024-04-24T08:07:54.884334+0000","last_became_active":"2024-04-24T08:07:52.059914+0000","last_became_peered":"2024-04-24T08:07:52.059914+0000","last_unstale":"2024-04-24T08:07:54.884334+0000","last_undegraded":"2024-04-24T08:07:54.884334+0000","last_fullsized":"2024-04-24T08:07:54.884334+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:18:04.371830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:57.144050+0000","last_change":"2024-04-24T08:07:51.076937+0000","last_active":"2024-04-24T08:07:57.144050+0000","last_peered":"2024-04-24T08:07:57.144050+0000","last_clean":"2024-04-24T08:07:57.144050+0000","last_became_active":"2024-04-24T08:07:51.076621+0000","last_became_peered":"2024-04-24T08:07:51.076621+0000","last_unstale":"2024-04-24T08:07:57.144050+0000","last_undegraded":"2024-04-24T08:07:57.144050+0000","last_fullsized":"2024-04-24T08:07:57.144050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:09.966994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.044070+0000","last_change":"2024-04-24T08:07:52.054150+0000","last_active":"2024-04-24T08:07:53.044070+0000","last_peered":"2024-04-24T08:07:53.044070+0000","last_clean":"2024-04-24T08:07:53.044070+0000","last_became_active":"2024-04-24T08:07:52.053817+0000","last_became_peered":"2024-04-24T08:07:52.053817+0000","last_unstale":"2024-04-24T08:07:53.044070+0000","last_undegraded":"2024-04-24T08:07:53.044070+0000","last_fullsized":"2024-04-24T08:07:53.044070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:46:16.570004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:57.141379+0000","last_change":"2024-04-24T08:07:51.045431+0000","last_active":"2024-04-24T08:07:57.141379+0000","last_peered":"2024-04-24T08:07:57.141379+0000","last_clean":"2024-04-24T08:07:57.141379+0000","last_became_active":"2024-04-24T08:07:51.045112+0000","last_became_peered":"2024-04-24T08:07:51.045112+0000","last_unstale":"2024-04-24T08:07:57.141379+0000","last_undegraded":"2024-04-24T08:07:57.141379+0000","last_fullsized":"2024-04-24T08:07:57.141379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:52:05.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":347,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.883583+0000","last_change":"2024-04-24T08:06:31.583310+0000","last_active":"2024-04-24T08:07:54.883583+0000","last_peered":"2024-04-24T08:07:54.883583+0000","last_clean":"2024-04-24T08:07:54.883583+0000","last_became_active":"2024-04-24T08:06:31.582735+0000","last_became_peered":"2024-04-24T08:06:31.582735+0000","last_unstale":"2024-04-24T08:07:54.883583+0000","last_undegraded":"2024-04-24T08:07:54.883583+0000","last_fullsized":"2024-04-24T08:07:54.883583+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:02:34.411025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:02:34.411025+0000","last_clean_scrub_stamp":"2024-04-24T08:02:34.411025+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:48.816092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.143981+0000","last_change":"2024-04-24T08:07:52.063961+0000","last_active":"2024-04-24T08:07:53.143981+0000","last_peered":"2024-04-24T08:07:53.143981+0000","last_clean":"2024-04-24T08:07:53.143981+0000","last_became_active":"2024-04-24T08:07:52.063608+0000","last_became_peered":"2024-04-24T08:07:52.063608+0000","last_unstale":"2024-04-24T08:07:53.143981+0000","last_undegraded":"2024-04-24T08:07:53.143981+0000","last_fullsized":"2024-04-24T08:07:53.143981+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:40:50.233349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.883848+0000","last_change":"2024-04-24T08:07:51.045626+0000","last_active":"2024-04-24T08:07:54.883848+0000","last_peered":"2024-04-24T08:07:54.883848+0000","last_clean":"2024-04-24T08:07:54.883848+0000","last_became_active":"2024-04-24T08:07:51.045293+0000","last_became_peered":"2024-04-24T08:07:51.045293+0000","last_unstale":"2024-04-24T08:07:54.883848+0000","last_undegraded":"2024-04-24T08:07:54.883848+0000","last_fullsized":"2024-04-24T08:07:54.883848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:13:04.601331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.088774+0000","last_change":"2024-04-24T08:07:51.079228+0000","last_active":"2024-04-24T08:07:53.088774+0000","last_peered":"2024-04-24T08:07:53.088774+0000","last_clean":"2024-04-24T08:07:53.088774+0000","last_became_active":"2024-04-24T08:07:51.078932+0000","last_became_peered":"2024-04-24T08:07:51.078932+0000","last_unstale":"2024-04-24T08:07:53.088774+0000","last_undegraded":"2024-04-24T08:07:53.088774+0000","last_fullsized":"2024-04-24T08:07:53.088774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:58:32.024133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.144627+0000","last_change":"2024-04-24T08:07:52.065943+0000","last_active":"2024-04-24T08:07:53.144627+0000","last_peered":"2024-04-24T08:07:53.144627+0000","last_clean":"2024-04-24T08:07:53.144627+0000","last_became_active":"2024-04-24T08:07:52.062810+0000","last_became_peered":"2024-04-24T08:07:52.062810+0000","last_unstale":"2024-04-24T08:07:53.144627+0000","last_undegraded":"2024-04-24T08:07:53.144627+0000","last_fullsized":"2024-04-24T08:07:53.144627+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:40:08.803652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:56.143145+0000","last_change":"2024-04-24T08:07:51.071118+0000","last_active":"2024-04-24T08:07:56.143145+0000","last_peered":"2024-04-24T08:07:56.143145+0000","last_clean":"2024-04-24T08:07:56.143145+0000","last_became_active":"2024-04-24T08:07:51.070678+0000","last_became_peered":"2024-04-24T08:07:51.070678+0000","last_unstale":"2024-04-24T08:07:56.143145+0000","last_undegraded":"2024-04-24T08:07:56.143145+0000","last_fullsized":"2024-04-24T08:07:56.143145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:57:44.212247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.044818+0000","last_change":"2024-04-24T08:07:52.058764+0000","last_active":"2024-04-24T08:07:53.044818+0000","last_peered":"2024-04-24T08:07:53.044818+0000","last_clean":"2024-04-24T08:07:53.044818+0000","last_became_active":"2024-04-24T08:07:52.058336+0000","last_became_peered":"2024-04-24T08:07:52.058336+0000","last_unstale":"2024-04-24T08:07:53.044818+0000","last_undegraded":"2024-04-24T08:07:53.044818+0000","last_fullsized":"2024-04-24T08:07:53.044818+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:05:37.245521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.114282+0000","last_change":"2024-04-24T08:07:51.067154+0000","last_active":"2024-04-24T08:07:55.114282+0000","last_peered":"2024-04-24T08:07:55.114282+0000","last_clean":"2024-04-24T08:07:55.114282+0000","last_became_active":"2024-04-24T08:07:51.066769+0000","last_became_peered":"2024-04-24T08:07:51.066769+0000","last_unstale":"2024-04-24T08:07:55.114282+0000","last_undegraded":"2024-04-24T08:07:55.114282+0000","last_fullsized":"2024-04-24T08:07:55.114282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:39:25.908189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.885189+0000","last_change":"2024-04-24T08:07:52.072134+0000","last_active":"2024-04-24T08:07:54.885189+0000","last_peered":"2024-04-24T08:07:54.885189+0000","last_clean":"2024-04-24T08:07:54.885189+0000","last_became_active":"2024-04-24T08:07:52.071703+0000","last_became_peered":"2024-04-24T08:07:52.071703+0000","last_unstale":"2024-04-24T08:07:54.885189+0000","last_undegraded":"2024-04-24T08:07:54.885189+0000","last_fullsized":"2024-04-24T08:07:54.885189+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:59.829988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.118999+0000","last_change":"2024-04-24T08:07:51.059151+0000","last_active":"2024-04-24T08:07:55.118999+0000","last_peered":"2024-04-24T08:07:55.118999+0000","last_clean":"2024-04-24T08:07:55.118999+0000","last_became_active":"2024-04-24T08:07:51.058872+0000","last_became_peered":"2024-04-24T08:07:51.058872+0000","last_unstale":"2024-04-24T08:07:55.118999+0000","last_undegraded":"2024-04-24T08:07:55.118999+0000","last_fullsized":"2024-04-24T08:07:55.118999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:36.023595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.088429+0000","last_change":"2024-04-24T08:07:52.056919+0000","last_active":"2024-04-24T08:07:53.088429+0000","last_peered":"2024-04-24T08:07:53.088429+0000","last_clean":"2024-04-24T08:07:53.088429+0000","last_became_active":"2024-04-24T08:07:52.056573+0000","last_became_peered":"2024-04-24T08:07:52.056573+0000","last_unstale":"2024-04-24T08:07:53.088429+0000","last_undegraded":"2024-04-24T08:07:53.088429+0000","last_fullsized":"2024-04-24T08:07:53.088429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:58:17.939465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.121925+0000","last_change":"2024-04-24T08:07:51.056325+0000","last_active":"2024-04-24T08:07:55.121925+0000","last_peered":"2024-04-24T08:07:55.121925+0000","last_clean":"2024-04-24T08:07:55.121925+0000","last_became_active":"2024-04-24T08:07:51.055862+0000","last_became_peered":"2024-04-24T08:07:51.055862+0000","last_unstale":"2024-04-24T08:07:55.121925+0000","last_undegraded":"2024-04-24T08:07:55.121925+0000","last_fullsized":"2024-04-24T08:07:55.121925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:11:44.487530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.883971+0000","last_change":"2024-04-24T08:07:52.065570+0000","last_active":"2024-04-24T08:07:54.883971+0000","last_peered":"2024-04-24T08:07:54.883971+0000","last_clean":"2024-04-24T08:07:54.883971+0000","last_became_active":"2024-04-24T08:07:52.065196+0000","last_became_peered":"2024-04-24T08:07:52.065196+0000","last_unstale":"2024-04-24T08:07:54.883971+0000","last_undegraded":"2024-04-24T08:07:54.883971+0000","last_fullsized":"2024-04-24T08:07:54.883971+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:48:09.474159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.423562+0000","last_change":"2024-04-24T08:07:51.067187+0000","last_active":"2024-04-24T08:07:54.423562+0000","last_peered":"2024-04-24T08:07:54.423562+0000","last_clean":"2024-04-24T08:07:54.423562+0000","last_became_active":"2024-04-24T08:07:51.066808+0000","last_became_peered":"2024-04-24T08:07:51.066808+0000","last_unstale":"2024-04-24T08:07:54.423562+0000","last_undegraded":"2024-04-24T08:07:54.423562+0000","last_fullsized":"2024-04-24T08:07:54.423562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:32:11.852191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.044285+0000","last_change":"2024-04-24T08:07:52.078426+0000","last_active":"2024-04-24T08:07:53.044285+0000","last_peered":"2024-04-24T08:07:53.044285+0000","last_clean":"2024-04-24T08:07:53.044285+0000","last_became_active":"2024-04-24T08:07:52.078076+0000","last_became_peered":"2024-04-24T08:07:52.078076+0000","last_unstale":"2024-04-24T08:07:53.044285+0000","last_undegraded":"2024-04-24T08:07:53.044285+0000","last_fullsized":"2024-04-24T08:07:53.044285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:55:29.885688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.117614+0000","last_change":"2024-04-24T08:07:51.042601+0000","last_active":"2024-04-24T08:07:55.117614+0000","last_peered":"2024-04-24T08:07:55.117614+0000","last_clean":"2024-04-24T08:07:55.117614+0000","last_became_active":"2024-04-24T08:07:51.042252+0000","last_became_peered":"2024-04-24T08:07:51.042252+0000","last_unstale":"2024-04-24T08:07:55.117614+0000","last_undegraded":"2024-04-24T08:07:55.117614+0000","last_fullsized":"2024-04-24T08:07:55.117614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:11:44.640303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884194+0000","last_change":"2024-04-24T08:07:52.063142+0000","last_active":"2024-04-24T08:07:54.884194+0000","last_peered":"2024-04-24T08:07:54.884194+0000","last_clean":"2024-04-24T08:07:54.884194+0000","last_became_active":"2024-04-24T08:07:52.062263+0000","last_became_peered":"2024-04-24T08:07:52.062263+0000","last_unstale":"2024-04-24T08:07:54.884194+0000","last_undegraded":"2024-04-24T08:07:54.884194+0000","last_fullsized":"2024-04-24T08:07:54.884194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:38.879380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:56.141666+0000","last_change":"2024-04-24T08:07:51.075154+0000","last_active":"2024-04-24T08:07:56.141666+0000","last_peered":"2024-04-24T08:07:56.141666+0000","last_clean":"2024-04-24T08:07:56.141666+0000","last_became_active":"2024-04-24T08:07:51.074866+0000","last_became_peered":"2024-04-24T08:07:51.074866+0000","last_unstale":"2024-04-24T08:07:56.141666+0000","last_undegraded":"2024-04-24T08:07:56.141666+0000","last_fullsized":"2024-04-24T08:07:56.141666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:58:14.810693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884178+0000","last_change":"2024-04-24T08:07:52.055958+0000","last_active":"2024-04-24T08:07:54.884178+0000","last_peered":"2024-04-24T08:07:54.884178+0000","last_clean":"2024-04-24T08:07:54.884178+0000","last_became_active":"2024-04-24T08:07:52.055598+0000","last_became_peered":"2024-04-24T08:07:52.055598+0000","last_unstale":"2024-04-24T08:07:54.884178+0000","last_undegraded":"2024-04-24T08:07:54.884178+0000","last_fullsized":"2024-04-24T08:07:54.884178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:12:22.452576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884830+0000","last_change":"2024-04-24T08:07:51.073234+0000","last_active":"2024-04-24T08:07:54.884830+0000","last_peered":"2024-04-24T08:07:54.884830+0000","last_clean":"2024-04-24T08:07:54.884830+0000","last_became_active":"2024-04-24T08:07:51.072887+0000","last_became_peered":"2024-04-24T08:07:51.072887+0000","last_unstale":"2024-04-24T08:07:54.884830+0000","last_undegraded":"2024-04-24T08:07:54.884830+0000","last_fullsized":"2024-04-24T08:07:54.884830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:28:15.908859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884070+0000","last_change":"2024-04-24T08:07:52.061188+0000","last_active":"2024-04-24T08:07:54.884070+0000","last_peered":"2024-04-24T08:07:54.884070+0000","last_clean":"2024-04-24T08:07:54.884070+0000","last_became_active":"2024-04-24T08:07:52.059616+0000","last_became_peered":"2024-04-24T08:07:52.059616+0000","last_unstale":"2024-04-24T08:07:54.884070+0000","last_undegraded":"2024-04-24T08:07:54.884070+0000","last_fullsized":"2024-04-24T08:07:54.884070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:43.546091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.144824+0000","last_change":"2024-04-24T08:07:51.078300+0000","last_active":"2024-04-24T08:07:53.144824+0000","last_peered":"2024-04-24T08:07:53.144824+0000","last_clean":"2024-04-24T08:07:53.144824+0000","last_became_active":"2024-04-24T08:07:51.076026+0000","last_became_peered":"2024-04-24T08:07:51.076026+0000","last_unstale":"2024-04-24T08:07:53.144824+0000","last_undegraded":"2024-04-24T08:07:53.144824+0000","last_fullsized":"2024-04-24T08:07:53.144824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:45:31.494213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884271+0000","last_change":"2024-04-24T08:07:52.056148+0000","last_active":"2024-04-24T08:07:54.884271+0000","last_peered":"2024-04-24T08:07:54.884271+0000","last_clean":"2024-04-24T08:07:54.884271+0000","last_became_active":"2024-04-24T08:07:52.055768+0000","last_became_peered":"2024-04-24T08:07:52.055768+0000","last_unstale":"2024-04-24T08:07:54.884271+0000","last_undegraded":"2024-04-24T08:07:54.884271+0000","last_fullsized":"2024-04-24T08:07:54.884271+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:52:48.503479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884068+0000","last_change":"2024-04-24T08:07:52.056611+0000","last_active":"2024-04-24T08:07:54.884068+0000","last_peered":"2024-04-24T08:07:54.884068+0000","last_clean":"2024-04-24T08:07:54.884068+0000","last_became_active":"2024-04-24T08:07:52.055788+0000","last_became_peered":"2024-04-24T08:07:52.055788+0000","last_unstale":"2024-04-24T08:07:54.884068+0000","last_undegraded":"2024-04-24T08:07:54.884068+0000","last_fullsized":"2024-04-24T08:07:54.884068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:21:28.997588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.042281+0000","last_change":"2024-04-24T08:07:52.059952+0000","last_active":"2024-04-24T08:07:53.042281+0000","last_peered":"2024-04-24T08:07:53.042281+0000","last_clean":"2024-04-24T08:07:53.042281+0000","last_became_active":"2024-04-24T08:07:52.059612+0000","last_became_peered":"2024-04-24T08:07:52.059612+0000","last_unstale":"2024-04-24T08:07:53.042281+0000","last_undegraded":"2024-04-24T08:07:53.042281+0000","last_fullsized":"2024-04-24T08:07:53.042281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:19:09.989429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.042226+0000","last_change":"2024-04-24T08:07:52.070899+0000","last_active":"2024-04-24T08:07:53.042226+0000","last_peered":"2024-04-24T08:07:53.042226+0000","last_clean":"2024-04-24T08:07:53.042226+0000","last_became_active":"2024-04-24T08:07:52.070556+0000","last_became_peered":"2024-04-24T08:07:52.070556+0000","last_unstale":"2024-04-24T08:07:53.042226+0000","last_undegraded":"2024-04-24T08:07:53.042226+0000","last_fullsized":"2024-04-24T08:07:53.042226+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:11.381122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.885498+0000","last_change":"2024-04-24T08:07:52.068524+0000","last_active":"2024-04-24T08:07:54.885498+0000","last_peered":"2024-04-24T08:07:54.885498+0000","last_clean":"2024-04-24T08:07:54.885498+0000","last_became_active":"2024-04-24T08:07:52.068173+0000","last_became_peered":"2024-04-24T08:07:52.068173+0000","last_unstale":"2024-04-24T08:07:54.885498+0000","last_undegraded":"2024-04-24T08:07:54.885498+0000","last_fullsized":"2024-04-24T08:07:54.885498+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:25:08.390138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.144715+0000","last_change":"2024-04-24T08:07:52.078604+0000","last_active":"2024-04-24T08:07:53.144715+0000","last_peered":"2024-04-24T08:07:53.144715+0000","last_clean":"2024-04-24T08:07:53.144715+0000","last_became_active":"2024-04-24T08:07:52.078122+0000","last_became_peered":"2024-04-24T08:07:52.078122+0000","last_unstale":"2024-04-24T08:07:53.144715+0000","last_undegraded":"2024-04-24T08:07:53.144715+0000","last_fullsized":"2024-04-24T08:07:53.144715+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:24:19.062821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884398+0000","last_change":"2024-04-24T08:07:52.055986+0000","last_active":"2024-04-24T08:07:54.884398+0000","last_peered":"2024-04-24T08:07:54.884398+0000","last_clean":"2024-04-24T08:07:54.884398+0000","last_became_active":"2024-04-24T08:07:52.055609+0000","last_became_peered":"2024-04-24T08:07:52.055609+0000","last_unstale":"2024-04-24T08:07:54.884398+0000","last_undegraded":"2024-04-24T08:07:54.884398+0000","last_fullsized":"2024-04-24T08:07:54.884398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:38:04.731204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.042646+0000","last_change":"2024-04-24T08:07:52.055605+0000","last_active":"2024-04-24T08:07:53.042646+0000","last_peered":"2024-04-24T08:07:53.042646+0000","last_clean":"2024-04-24T08:07:53.042646+0000","last_became_active":"2024-04-24T08:07:52.055268+0000","last_became_peered":"2024-04-24T08:07:52.055268+0000","last_unstale":"2024-04-24T08:07:53.042646+0000","last_undegraded":"2024-04-24T08:07:53.042646+0000","last_fullsized":"2024-04-24T08:07:53.042646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:50:29.197484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.044650+0000","last_change":"2024-04-24T08:07:52.064590+0000","last_active":"2024-04-24T08:07:53.044650+0000","last_peered":"2024-04-24T08:07:53.044650+0000","last_clean":"2024-04-24T08:07:53.044650+0000","last_became_active":"2024-04-24T08:07:52.064113+0000","last_became_peered":"2024-04-24T08:07:52.064113+0000","last_unstale":"2024-04-24T08:07:53.044650+0000","last_undegraded":"2024-04-24T08:07:53.044650+0000","last_fullsized":"2024-04-24T08:07:53.044650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:36:10.926145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884318+0000","last_change":"2024-04-24T08:07:52.053757+0000","last_active":"2024-04-24T08:07:54.884318+0000","last_peered":"2024-04-24T08:07:54.884318+0000","last_clean":"2024-04-24T08:07:54.884318+0000","last_became_active":"2024-04-24T08:07:52.053380+0000","last_became_peered":"2024-04-24T08:07:52.053380+0000","last_unstale":"2024-04-24T08:07:54.884318+0000","last_undegraded":"2024-04-24T08:07:54.884318+0000","last_fullsized":"2024-04-24T08:07:54.884318+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:45:27.284230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.043732+0000","last_change":"2024-04-24T08:07:52.068308+0000","last_active":"2024-04-24T08:07:53.043732+0000","last_peered":"2024-04-24T08:07:53.043732+0000","last_clean":"2024-04-24T08:07:53.043732+0000","last_became_active":"2024-04-24T08:07:52.067786+0000","last_became_peered":"2024-04-24T08:07:52.067786+0000","last_unstale":"2024-04-24T08:07:53.043732+0000","last_undegraded":"2024-04-24T08:07:53.043732+0000","last_fullsized":"2024-04-24T08:07:53.043732+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:26:09.349120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.874142+0000","last_change":"2024-04-24T08:07:52.053089+0000","last_active":"2024-04-24T08:07:53.874142+0000","last_peered":"2024-04-24T08:07:53.874142+0000","last_clean":"2024-04-24T08:07:53.874142+0000","last_became_active":"2024-04-24T08:07:52.052753+0000","last_became_peered":"2024-04-24T08:07:52.052753+0000","last_unstale":"2024-04-24T08:07:53.874142+0000","last_undegraded":"2024-04-24T08:07:53.874142+0000","last_fullsized":"2024-04-24T08:07:53.874142+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:02:14.583922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.144314+0000","last_change":"2024-04-24T08:07:52.076633+0000","last_active":"2024-04-24T08:07:53.144314+0000","last_peered":"2024-04-24T08:07:53.144314+0000","last_clean":"2024-04-24T08:07:53.144314+0000","last_became_active":"2024-04-24T08:07:52.076280+0000","last_became_peered":"2024-04-24T08:07:52.076280+0000","last_unstale":"2024-04-24T08:07:53.144314+0000","last_undegraded":"2024-04-24T08:07:53.144314+0000","last_fullsized":"2024-04-24T08:07:53.144314+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:29:59.823772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884007+0000","last_change":"2024-04-24T08:07:52.053017+0000","last_active":"2024-04-24T08:07:54.884007+0000","last_peered":"2024-04-24T08:07:54.884007+0000","last_clean":"2024-04-24T08:07:54.884007+0000","last_became_active":"2024-04-24T08:07:52.052325+0000","last_became_peered":"2024-04-24T08:07:52.052325+0000","last_unstale":"2024-04-24T08:07:54.884007+0000","last_undegraded":"2024-04-24T08:07:54.884007+0000","last_fullsized":"2024-04-24T08:07:54.884007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:05:58.161360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":213304,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5892,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":25,"num_read_kb":20,"num_write":116,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":52138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350991,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36244,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708908,"statfs":{"total":95995035648,"available":95957921792,"internally_reserved":0,"allocated":925696,"data_stored":444311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612627,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1037869,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776153,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":445095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037791,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":444937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":1035883,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495660,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":444401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":937984,"data_stored":445923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738445,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":445789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411395,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":471562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182460,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":444595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":445346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757298,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":1036546,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":39934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:08:01.116 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:08:01.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.369+0000 7f13cbe00640 1 -- 172.21.15.28:0/498203843 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13cc104080 msgr2=0x7f13cc1044a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:01.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.369+0000 7f13cbe00640 1 --2- 172.21.15.28:0/498203843 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13cc104080 0x7f13cc1044a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f13bc009900 tx=0x7f13bc02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:08:01.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.369+0000 7f13cbe00640 1 -- 172.21.15.28:0/498203843 shutdown_connections 2024-04-24T08:08:01.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.369+0000 7f13cbe00640 1 --2- 172.21.15.28:0/498203843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13cc105270 0x7f13cc1056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:01.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.369+0000 7f13cbe00640 1 --2- 172.21.15.28:0/498203843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13cc105c30 0x7f13cc10c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:01.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.369+0000 7f13cbe00640 1 --2- 172.21.15.28:0/498203843 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13cc104080 0x7f13cc1044a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:01.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.369+0000 7f13cbe00640 1 -- 172.21.15.28:0/498203843 >> 172.21.15.28:0/498203843 conn(0x7f13cc0ff610 msgr2=0x7f13cc101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:01.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.369+0000 7f13cbe00640 1 -- 172.21.15.28:0/498203843 shutdown_connections 2024-04-24T08:08:01.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.369+0000 7f13cbe00640 1 -- 172.21.15.28:0/498203843 wait complete. 2024-04-24T08:08:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.370+0000 7f13cbe00640 1 Processor -- start 2024-04-24T08:08:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.370+0000 7f13cbe00640 1 -- start start 2024-04-24T08:08:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.370+0000 7f13cbe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13cc104080 0x7f13cc19a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.370+0000 7f13cbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13cc105270 0x7f13cc19a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:01.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.370+0000 7f13cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13cc105c30 0x7f13cc1a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.370+0000 7f13cbe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13cc10fd60 con 0x7f13cc105c30 2024-04-24T08:08:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.370+0000 7f13cbe00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f13cc10fbe0 con 0x7f13cc104080 2024-04-24T08:08:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.370+0000 7f13cbe00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f13cc10fee0 con 0x7f13cc105270 2024-04-24T08:08:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.371+0000 7f13caa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13cc104080 0x7f13cc19a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.371+0000 7f13cb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13cc105c30 0x7f13cc1a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.371+0000 7f13caa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13cc104080 0x7f13cc19a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41500/0 (socket says 172.21.15.28:41500) 2024-04-24T08:08:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.371+0000 7f13caa00640 1 -- 172.21.15.28:0/3287351091 learned_addr learned my addr 172.21.15.28:0/3287351091 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.371+0000 7f13ca000640 1 --2- 172.21.15.28:0/3287351091 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13cc105270 0x7f13cc19a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.371+0000 7f13cb400640 1 -- 172.21.15.28:0/3287351091 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13cc105270 msgr2=0x7f13cc19a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.371+0000 7f13cb400640 1 --2- 172.21.15.28:0/3287351091 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13cc105270 0x7f13cc19a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.371+0000 7f13cb400640 1 -- 172.21.15.28:0/3287351091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13cc104080 msgr2=0x7f13cc19a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:01.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.371+0000 7f13cb400640 1 --2- 172.21.15.28:0/3287351091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13cc104080 0x7f13cc19a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:01.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.371+0000 7f13cb400640 1 -- 172.21.15.28:0/3287351091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13cc1a1fd0 con 0x7f13cc105c30 2024-04-24T08:08:01.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.371+0000 7f13caa00640 1 --2- 172.21.15.28:0/3287351091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13cc104080 0x7f13cc19a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:01.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.371+0000 7f13cb400640 1 --2- 172.21.15.28:0/3287351091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13cc105c30 0x7f13cc1a1960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f13c400e970 tx=0x7f13c400ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:01.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.372+0000 7f13c3400640 1 -- 172.21.15.28:0/3287351091 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c400ccc0 con 0x7f13cc105c30 2024-04-24T08:08:01.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.372+0000 7f13c3400640 1 -- 172.21.15.28:0/3287351091 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13c40035f0 con 0x7f13cc105c30 2024-04-24T08:08:01.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.372+0000 7f13cbe00640 1 -- 172.21.15.28:0/3287351091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13cc1a22c0 con 0x7f13cc105c30 2024-04-24T08:08:01.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.372+0000 7f13c3400640 1 -- 172.21.15.28:0/3287351091 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13c400ce60 con 0x7f13cc105c30 2024-04-24T08:08:01.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.372+0000 7f13cbe00640 1 -- 172.21.15.28:0/3287351091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13cc1a27c0 con 0x7f13cc105c30 2024-04-24T08:08:01.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.373+0000 7f13cbe00640 1 -- 172.21.15.28:0/3287351091 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f13cc10b3a0 con 0x7f13cc105c30 2024-04-24T08:08:01.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.375+0000 7f13c3400640 1 -- 172.21.15.28:0/3287351091 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f13c40037a0 con 0x7f13cc105c30 2024-04-24T08:08:01.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.377+0000 7f13c3400640 1 --2- 172.21.15.28:0/3287351091 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13b00a3c40 0x7f13b00a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:01.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.377+0000 7f13c3400640 1 -- 172.21.15.28:0/3287351091 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f13c40d0700 con 0x7f13cc105c30 2024-04-24T08:08:01.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.377+0000 7f13caa00640 1 --2- 172.21.15.28:0/3287351091 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13b00a3c40 0x7f13b00a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:01.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.378+0000 7f13caa00640 1 --2- 172.21.15.28:0/3287351091 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13b00a3c40 0x7f13b00a6100 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f13bc000c00 tx=0x7f13bc0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:01.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.381+0000 7f13c3400640 1 -- 172.21.15.28:0/3287351091 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f13c409d030 con 0x7f13cc105c30 2024-04-24T08:08:01.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:01 smithi194 ceph-mon[32836]: pgmap v321: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.0 KiB/s wr, 17 op/s 2024-04-24T08:08:01.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:01 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3672844364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:08:01.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:01 smithi028 ceph-mon[27786]: pgmap v321: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.0 KiB/s wr, 17 op/s 2024-04-24T08:08:01.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:01 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3672844364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:08:01.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.581+0000 7f13cbe00640 1 -- 172.21.15.28:0/3287351091 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f13cc062440 con 0x7f13b00a3c40 2024-04-24T08:08:01.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.591+0000 7f13c3400640 1 -- 172.21.15.28:0/3287351091 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258853 (secure 0 0 0) 0x7f13cc062440 con 0x7f13b00a3c40 2024-04-24T08:08:01.598 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:08:01.600 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-24T08:08:01.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.598+0000 7f13cbe00640 1 -- 172.21.15.28:0/3287351091 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13b00a3c40 msgr2=0x7f13b00a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:01.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.598+0000 7f13cbe00640 1 --2- 172.21.15.28:0/3287351091 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13b00a3c40 0x7f13b00a6100 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f13bc000c00 tx=0x7f13bc0315b0 comp rx=0 tx=0).stop 2024-04-24T08:08:01.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.598+0000 7f13cbe00640 1 -- 172.21.15.28:0/3287351091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13cc105c30 msgr2=0x7f13cc1a1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:01.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.598+0000 7f13cbe00640 1 --2- 172.21.15.28:0/3287351091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13cc105c30 0x7f13cc1a1960 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f13c400e970 tx=0x7f13c400ee30 comp rx=0 tx=0).stop 2024-04-24T08:08:01.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.599+0000 7f13cbe00640 1 -- 172.21.15.28:0/3287351091 shutdown_connections 2024-04-24T08:08:01.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.599+0000 7f13cbe00640 1 --2- 172.21.15.28:0/3287351091 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13b00a3c40 0x7f13b00a6100 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:01.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.599+0000 7f13cbe00640 1 --2- 172.21.15.28:0/3287351091 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13cc105c30 0x7f13cc1a1960 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:01.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.599+0000 7f13cbe00640 1 --2- 172.21.15.28:0/3287351091 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13cc105270 0x7f13cc19a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.599+0000 7f13cbe00640 1 --2- 172.21.15.28:0/3287351091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13cc104080 0x7f13cc19a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.599+0000 7f13cbe00640 1 -- 172.21.15.28:0/3287351091 >> 172.21.15.28:0/3287351091 conn(0x7f13cc0ff610 msgr2=0x7f13cc108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.599+0000 7f13cbe00640 1 -- 172.21.15.28:0/3287351091 shutdown_connections 2024-04-24T08:08:01.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:01.599+0000 7f13cbe00640 1 -- 172.21.15.28:0/3287351091 wait complete. 2024-04-24T08:08:01.620 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":321,"stamp":"2024-04-24T08:08:00.060821+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596260,"num_objects":56,"num_object_clones":0,"num_object_copies":168,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":56,"num_whiteouts":0,"num_read":161,"num_read_kb":139,"num_write":297,"num_write_kb":2970,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":237,"ondisk_log_size":237,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":284,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360408,"kb_used_data":11236,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124581416,"statfs":{"total":1151940427776,"available":1151571369984,"internally_reserved":0,"allocated":11505664,"data_stored":7142257,"data_compressed":108294,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19140,"internal_metadata":357414204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":10,"apply_latency_ms":10,"commit_latency_ns":10000000,"apply_latency_ns":10000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5892,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":25,"num_read_kb":20,"num_write":116,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.026435"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884280+0000","last_change":"2024-04-24T08:07:52.074319+0000","last_active":"2024-04-24T08:07:54.884280+0000","last_peered":"2024-04-24T08:07:54.884280+0000","last_clean":"2024-04-24T08:07:54.884280+0000","last_became_active":"2024-04-24T08:07:52.073923+0000","last_became_peered":"2024-04-24T08:07:52.073923+0000","last_unstale":"2024-04-24T08:07:54.884280+0000","last_undegraded":"2024-04-24T08:07:54.884280+0000","last_fullsized":"2024-04-24T08:07:54.884280+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:29.325904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884996+0000","last_change":"2024-04-24T08:07:52.068316+0000","last_active":"2024-04-24T08:07:54.884996+0000","last_peered":"2024-04-24T08:07:54.884996+0000","last_clean":"2024-04-24T08:07:54.884996+0000","last_became_active":"2024-04-24T08:07:52.067961+0000","last_became_peered":"2024-04-24T08:07:52.067961+0000","last_unstale":"2024-04-24T08:07:54.884996+0000","last_undegraded":"2024-04-24T08:07:54.884996+0000","last_fullsized":"2024-04-24T08:07:54.884996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:35:14.079348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884658+0000","last_change":"2024-04-24T08:07:52.066870+0000","last_active":"2024-04-24T08:07:54.884658+0000","last_peered":"2024-04-24T08:07:54.884658+0000","last_clean":"2024-04-24T08:07:54.884658+0000","last_became_active":"2024-04-24T08:07:52.066481+0000","last_became_peered":"2024-04-24T08:07:52.066481+0000","last_unstale":"2024-04-24T08:07:54.884658+0000","last_undegraded":"2024-04-24T08:07:54.884658+0000","last_fullsized":"2024-04-24T08:07:54.884658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:25:45.526667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.906627+0000","last_change":"2024-04-24T08:07:52.061548+0000","last_active":"2024-04-24T08:07:53.906627+0000","last_peered":"2024-04-24T08:07:53.906627+0000","last_clean":"2024-04-24T08:07:53.906627+0000","last_became_active":"2024-04-24T08:07:52.060519+0000","last_became_peered":"2024-04-24T08:07:52.060519+0000","last_unstale":"2024-04-24T08:07:53.906627+0000","last_undegraded":"2024-04-24T08:07:53.906627+0000","last_fullsized":"2024-04-24T08:07:53.906627+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:47:20.952035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.088523+0000","last_change":"2024-04-24T08:07:52.063805+0000","last_active":"2024-04-24T08:07:53.088523+0000","last_peered":"2024-04-24T08:07:53.088523+0000","last_clean":"2024-04-24T08:07:53.088523+0000","last_became_active":"2024-04-24T08:07:52.063414+0000","last_became_peered":"2024-04-24T08:07:52.063414+0000","last_unstale":"2024-04-24T08:07:53.088523+0000","last_undegraded":"2024-04-24T08:07:53.088523+0000","last_fullsized":"2024-04-24T08:07:53.088523+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:06:24.465035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884525+0000","last_change":"2024-04-24T08:07:52.050152+0000","last_active":"2024-04-24T08:07:54.884525+0000","last_peered":"2024-04-24T08:07:54.884525+0000","last_clean":"2024-04-24T08:07:54.884525+0000","last_became_active":"2024-04-24T08:07:52.049805+0000","last_became_peered":"2024-04-24T08:07:52.049805+0000","last_unstale":"2024-04-24T08:07:54.884525+0000","last_undegraded":"2024-04-24T08:07:54.884525+0000","last_fullsized":"2024-04-24T08:07:54.884525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:47:34.644263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884128+0000","last_change":"2024-04-24T08:07:52.055726+0000","last_active":"2024-04-24T08:07:54.884128+0000","last_peered":"2024-04-24T08:07:54.884128+0000","last_clean":"2024-04-24T08:07:54.884128+0000","last_became_active":"2024-04-24T08:07:52.054563+0000","last_became_peered":"2024-04-24T08:07:52.054563+0000","last_unstale":"2024-04-24T08:07:54.884128+0000","last_undegraded":"2024-04-24T08:07:54.884128+0000","last_fullsized":"2024-04-24T08:07:54.884128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:34:57.430232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.043631+0000","last_change":"2024-04-24T08:07:52.053672+0000","last_active":"2024-04-24T08:07:53.043631+0000","last_peered":"2024-04-24T08:07:53.043631+0000","last_clean":"2024-04-24T08:07:53.043631+0000","last_became_active":"2024-04-24T08:07:52.053150+0000","last_became_peered":"2024-04-24T08:07:52.053150+0000","last_unstale":"2024-04-24T08:07:53.043631+0000","last_undegraded":"2024-04-24T08:07:53.043631+0000","last_fullsized":"2024-04-24T08:07:53.043631+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:00:12.272555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.874420+0000","last_change":"2024-04-24T08:07:52.054648+0000","last_active":"2024-04-24T08:07:53.874420+0000","last_peered":"2024-04-24T08:07:53.874420+0000","last_clean":"2024-04-24T08:07:53.874420+0000","last_became_active":"2024-04-24T08:07:52.054301+0000","last_became_peered":"2024-04-24T08:07:52.054301+0000","last_unstale":"2024-04-24T08:07:53.874420+0000","last_undegraded":"2024-04-24T08:07:53.874420+0000","last_fullsized":"2024-04-24T08:07:53.874420+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:52:20.533855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884997+0000","last_change":"2024-04-24T08:07:52.052994+0000","last_active":"2024-04-24T08:07:54.884997+0000","last_peered":"2024-04-24T08:07:54.884997+0000","last_clean":"2024-04-24T08:07:54.884997+0000","last_became_active":"2024-04-24T08:07:52.052641+0000","last_became_peered":"2024-04-24T08:07:52.052641+0000","last_unstale":"2024-04-24T08:07:54.884997+0000","last_undegraded":"2024-04-24T08:07:54.884997+0000","last_fullsized":"2024-04-24T08:07:54.884997+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:02.665622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.883970+0000","last_change":"2024-04-24T08:07:52.055901+0000","last_active":"2024-04-24T08:07:54.883970+0000","last_peered":"2024-04-24T08:07:54.883970+0000","last_clean":"2024-04-24T08:07:54.883970+0000","last_became_active":"2024-04-24T08:07:52.055277+0000","last_became_peered":"2024-04-24T08:07:52.055277+0000","last_unstale":"2024-04-24T08:07:54.883970+0000","last_undegraded":"2024-04-24T08:07:54.883970+0000","last_fullsized":"2024-04-24T08:07:54.883970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:46.950578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.906605+0000","last_change":"2024-04-24T08:07:52.060329+0000","last_active":"2024-04-24T08:07:53.906605+0000","last_peered":"2024-04-24T08:07:53.906605+0000","last_clean":"2024-04-24T08:07:53.906605+0000","last_became_active":"2024-04-24T08:07:52.059914+0000","last_became_peered":"2024-04-24T08:07:52.059914+0000","last_unstale":"2024-04-24T08:07:53.906605+0000","last_undegraded":"2024-04-24T08:07:53.906605+0000","last_fullsized":"2024-04-24T08:07:53.906605+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:15:20.263916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.144525+0000","last_change":"2024-04-24T08:07:52.062108+0000","last_active":"2024-04-24T08:07:53.144525+0000","last_peered":"2024-04-24T08:07:53.144525+0000","last_clean":"2024-04-24T08:07:53.144525+0000","last_became_active":"2024-04-24T08:07:52.061665+0000","last_became_peered":"2024-04-24T08:07:52.061665+0000","last_unstale":"2024-04-24T08:07:53.144525+0000","last_undegraded":"2024-04-24T08:07:53.144525+0000","last_fullsized":"2024-04-24T08:07:53.144525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:55:58.071807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884654+0000","last_change":"2024-04-24T08:07:52.063033+0000","last_active":"2024-04-24T08:07:54.884654+0000","last_peered":"2024-04-24T08:07:54.884654+0000","last_clean":"2024-04-24T08:07:54.884654+0000","last_became_active":"2024-04-24T08:07:52.062610+0000","last_became_peered":"2024-04-24T08:07:52.062610+0000","last_unstale":"2024-04-24T08:07:54.884654+0000","last_undegraded":"2024-04-24T08:07:54.884654+0000","last_fullsized":"2024-04-24T08:07:54.884654+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:59:35.906720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884363+0000","last_change":"2024-04-24T08:07:52.064657+0000","last_active":"2024-04-24T08:07:54.884363+0000","last_peered":"2024-04-24T08:07:54.884363+0000","last_clean":"2024-04-24T08:07:54.884363+0000","last_became_active":"2024-04-24T08:07:52.064188+0000","last_became_peered":"2024-04-24T08:07:52.064188+0000","last_unstale":"2024-04-24T08:07:54.884363+0000","last_undegraded":"2024-04-24T08:07:54.884363+0000","last_fullsized":"2024-04-24T08:07:54.884363+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:56:32.139420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.874066+0000","last_change":"2024-04-24T08:07:52.067227+0000","last_active":"2024-04-24T08:07:53.874066+0000","last_peered":"2024-04-24T08:07:53.874066+0000","last_clean":"2024-04-24T08:07:53.874066+0000","last_became_active":"2024-04-24T08:07:52.066854+0000","last_became_peered":"2024-04-24T08:07:52.066854+0000","last_unstale":"2024-04-24T08:07:53.874066+0000","last_undegraded":"2024-04-24T08:07:53.874066+0000","last_fullsized":"2024-04-24T08:07:53.874066+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:25:17.580223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884564+0000","last_change":"2024-04-24T08:07:52.074127+0000","last_active":"2024-04-24T08:07:54.884564+0000","last_peered":"2024-04-24T08:07:54.884564+0000","last_clean":"2024-04-24T08:07:54.884564+0000","last_became_active":"2024-04-24T08:07:52.073806+0000","last_became_peered":"2024-04-24T08:07:52.073806+0000","last_unstale":"2024-04-24T08:07:54.884564+0000","last_undegraded":"2024-04-24T08:07:54.884564+0000","last_fullsized":"2024-04-24T08:07:54.884564+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:25:22.068997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.043190+0000","last_change":"2024-04-24T08:07:52.051239+0000","last_active":"2024-04-24T08:07:53.043190+0000","last_peered":"2024-04-24T08:07:53.043190+0000","last_clean":"2024-04-24T08:07:53.043190+0000","last_became_active":"2024-04-24T08:07:52.049232+0000","last_became_peered":"2024-04-24T08:07:52.049232+0000","last_unstale":"2024-04-24T08:07:53.043190+0000","last_undegraded":"2024-04-24T08:07:53.043190+0000","last_fullsized":"2024-04-24T08:07:53.043190+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:07:39.768537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.042466+0000","last_change":"2024-04-24T08:07:52.062119+0000","last_active":"2024-04-24T08:07:53.042466+0000","last_peered":"2024-04-24T08:07:53.042466+0000","last_clean":"2024-04-24T08:07:53.042466+0000","last_became_active":"2024-04-24T08:07:52.061781+0000","last_became_peered":"2024-04-24T08:07:52.061781+0000","last_unstale":"2024-04-24T08:07:53.042466+0000","last_undegraded":"2024-04-24T08:07:53.042466+0000","last_fullsized":"2024-04-24T08:07:53.042466+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:49:41.595368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:56.143066+0000","last_change":"2024-04-24T08:07:51.052411+0000","last_active":"2024-04-24T08:07:56.143066+0000","last_peered":"2024-04-24T08:07:56.143066+0000","last_clean":"2024-04-24T08:07:56.143066+0000","last_became_active":"2024-04-24T08:07:51.051704+0000","last_became_peered":"2024-04-24T08:07:51.051704+0000","last_unstale":"2024-04-24T08:07:56.143066+0000","last_undegraded":"2024-04-24T08:07:56.143066+0000","last_fullsized":"2024-04-24T08:07:56.143066+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:34:23.001338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.044550+0000","last_change":"2024-04-24T08:07:52.056508+0000","last_active":"2024-04-24T08:07:53.044550+0000","last_peered":"2024-04-24T08:07:53.044550+0000","last_clean":"2024-04-24T08:07:53.044550+0000","last_became_active":"2024-04-24T08:07:52.056158+0000","last_became_peered":"2024-04-24T08:07:52.056158+0000","last_unstale":"2024-04-24T08:07:53.044550+0000","last_undegraded":"2024-04-24T08:07:53.044550+0000","last_fullsized":"2024-04-24T08:07:53.044550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:52.267984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.044719+0000","last_change":"2024-04-24T08:07:51.080403+0000","last_active":"2024-04-24T08:07:53.044719+0000","last_peered":"2024-04-24T08:07:53.044719+0000","last_clean":"2024-04-24T08:07:53.044719+0000","last_became_active":"2024-04-24T08:07:51.079158+0000","last_became_peered":"2024-04-24T08:07:51.079158+0000","last_unstale":"2024-04-24T08:07:53.044719+0000","last_undegraded":"2024-04-24T08:07:53.044719+0000","last_fullsized":"2024-04-24T08:07:53.044719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:49:59.491850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.885525+0000","last_change":"2024-04-24T08:07:52.066963+0000","last_active":"2024-04-24T08:07:54.885525+0000","last_peered":"2024-04-24T08:07:54.885525+0000","last_clean":"2024-04-24T08:07:54.885525+0000","last_became_active":"2024-04-24T08:07:52.065475+0000","last_became_peered":"2024-04-24T08:07:52.065475+0000","last_unstale":"2024-04-24T08:07:54.885525+0000","last_undegraded":"2024-04-24T08:07:54.885525+0000","last_fullsized":"2024-04-24T08:07:54.885525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:28:34.220200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.088843+0000","last_change":"2024-04-24T08:07:51.042772+0000","last_active":"2024-04-24T08:07:53.088843+0000","last_peered":"2024-04-24T08:07:53.088843+0000","last_clean":"2024-04-24T08:07:53.088843+0000","last_became_active":"2024-04-24T08:07:51.042128+0000","last_became_peered":"2024-04-24T08:07:51.042128+0000","last_unstale":"2024-04-24T08:07:53.088843+0000","last_undegraded":"2024-04-24T08:07:53.088843+0000","last_fullsized":"2024-04-24T08:07:53.088843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:10:29.207991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.874067+0000","last_change":"2024-04-24T08:07:52.062411+0000","last_active":"2024-04-24T08:07:53.874067+0000","last_peered":"2024-04-24T08:07:53.874067+0000","last_clean":"2024-04-24T08:07:53.874067+0000","last_became_active":"2024-04-24T08:07:52.062037+0000","last_became_peered":"2024-04-24T08:07:52.062037+0000","last_unstale":"2024-04-24T08:07:53.874067+0000","last_undegraded":"2024-04-24T08:07:53.874067+0000","last_fullsized":"2024-04-24T08:07:53.874067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:16:17.463874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.088581+0000","last_change":"2024-04-24T08:07:51.046046+0000","last_active":"2024-04-24T08:07:53.088581+0000","last_peered":"2024-04-24T08:07:53.088581+0000","last_clean":"2024-04-24T08:07:53.088581+0000","last_became_active":"2024-04-24T08:07:51.045633+0000","last_became_peered":"2024-04-24T08:07:51.045633+0000","last_unstale":"2024-04-24T08:07:53.088581+0000","last_undegraded":"2024-04-24T08:07:53.088581+0000","last_fullsized":"2024-04-24T08:07:53.088581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:18:20.529234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884911+0000","last_change":"2024-04-24T08:07:52.061167+0000","last_active":"2024-04-24T08:07:54.884911+0000","last_peered":"2024-04-24T08:07:54.884911+0000","last_clean":"2024-04-24T08:07:54.884911+0000","last_became_active":"2024-04-24T08:07:52.060841+0000","last_became_peered":"2024-04-24T08:07:52.060841+0000","last_unstale":"2024-04-24T08:07:54.884911+0000","last_undegraded":"2024-04-24T08:07:54.884911+0000","last_fullsized":"2024-04-24T08:07:54.884911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:04:27.048760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:58.146198+0000","last_change":"2024-04-24T08:07:51.044274+0000","last_active":"2024-04-24T08:07:58.146198+0000","last_peered":"2024-04-24T08:07:58.146198+0000","last_clean":"2024-04-24T08:07:58.146198+0000","last_became_active":"2024-04-24T08:07:51.043938+0000","last_became_peered":"2024-04-24T08:07:51.043938+0000","last_unstale":"2024-04-24T08:07:58.146198+0000","last_undegraded":"2024-04-24T08:07:58.146198+0000","last_fullsized":"2024-04-24T08:07:58.146198+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:05:38.349981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":5,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.088535+0000","last_change":"2024-04-24T08:07:52.055777+0000","last_active":"2024-04-24T08:07:53.088535+0000","last_peered":"2024-04-24T08:07:53.088535+0000","last_clean":"2024-04-24T08:07:53.088535+0000","last_became_active":"2024-04-24T08:07:52.055415+0000","last_became_peered":"2024-04-24T08:07:52.055415+0000","last_unstale":"2024-04-24T08:07:53.088535+0000","last_undegraded":"2024-04-24T08:07:53.088535+0000","last_fullsized":"2024-04-24T08:07:53.088535+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:18:31.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:56.142317+0000","last_change":"2024-04-24T08:07:51.072513+0000","last_active":"2024-04-24T08:07:56.142317+0000","last_peered":"2024-04-24T08:07:56.142317+0000","last_clean":"2024-04-24T08:07:56.142317+0000","last_became_active":"2024-04-24T08:07:51.072249+0000","last_became_peered":"2024-04-24T08:07:51.072249+0000","last_unstale":"2024-04-24T08:07:56.142317+0000","last_undegraded":"2024-04-24T08:07:56.142317+0000","last_fullsized":"2024-04-24T08:07:56.142317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:45:07.112617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884811+0000","last_change":"2024-04-24T08:07:52.060290+0000","last_active":"2024-04-24T08:07:54.884811+0000","last_peered":"2024-04-24T08:07:54.884811+0000","last_clean":"2024-04-24T08:07:54.884811+0000","last_became_active":"2024-04-24T08:07:52.059961+0000","last_became_peered":"2024-04-24T08:07:52.059961+0000","last_unstale":"2024-04-24T08:07:54.884811+0000","last_undegraded":"2024-04-24T08:07:54.884811+0000","last_fullsized":"2024-04-24T08:07:54.884811+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:10:56.885967+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:56.143225+0000","last_change":"2024-04-24T08:07:51.070901+0000","last_active":"2024-04-24T08:07:56.143225+0000","last_peered":"2024-04-24T08:07:56.143225+0000","last_clean":"2024-04-24T08:07:56.143225+0000","last_became_active":"2024-04-24T08:07:51.067178+0000","last_became_peered":"2024-04-24T08:07:51.067178+0000","last_unstale":"2024-04-24T08:07:56.143225+0000","last_undegraded":"2024-04-24T08:07:56.143225+0000","last_fullsized":"2024-04-24T08:07:56.143225+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:24:26.807786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.043555+0000","last_change":"2024-04-24T08:07:52.079493+0000","last_active":"2024-04-24T08:07:53.043555+0000","last_peered":"2024-04-24T08:07:53.043555+0000","last_clean":"2024-04-24T08:07:53.043555+0000","last_became_active":"2024-04-24T08:07:52.077266+0000","last_became_peered":"2024-04-24T08:07:52.077266+0000","last_unstale":"2024-04-24T08:07:53.043555+0000","last_undegraded":"2024-04-24T08:07:53.043555+0000","last_fullsized":"2024-04-24T08:07:53.043555+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:40:45.660146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884222+0000","last_change":"2024-04-24T08:07:51.045963+0000","last_active":"2024-04-24T08:07:54.884222+0000","last_peered":"2024-04-24T08:07:54.884222+0000","last_clean":"2024-04-24T08:07:54.884222+0000","last_became_active":"2024-04-24T08:07:51.045641+0000","last_became_peered":"2024-04-24T08:07:51.045641+0000","last_unstale":"2024-04-24T08:07:54.884222+0000","last_undegraded":"2024-04-24T08:07:54.884222+0000","last_fullsized":"2024-04-24T08:07:54.884222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:07:59.646155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.042378+0000","last_change":"2024-04-24T08:07:52.057280+0000","last_active":"2024-04-24T08:07:53.042378+0000","last_peered":"2024-04-24T08:07:53.042378+0000","last_clean":"2024-04-24T08:07:53.042378+0000","last_became_active":"2024-04-24T08:07:52.054944+0000","last_became_peered":"2024-04-24T08:07:52.054944+0000","last_unstale":"2024-04-24T08:07:53.042378+0000","last_undegraded":"2024-04-24T08:07:53.042378+0000","last_fullsized":"2024-04-24T08:07:53.042378+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:22:45.911472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.116385+0000","last_change":"2024-04-24T08:07:51.048898+0000","last_active":"2024-04-24T08:07:55.116385+0000","last_peered":"2024-04-24T08:07:55.116385+0000","last_clean":"2024-04-24T08:07:55.116385+0000","last_became_active":"2024-04-24T08:07:51.048585+0000","last_became_peered":"2024-04-24T08:07:51.048585+0000","last_unstale":"2024-04-24T08:07:55.116385+0000","last_undegraded":"2024-04-24T08:07:55.116385+0000","last_fullsized":"2024-04-24T08:07:55.116385+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:40:30.413541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.874364+0000","last_change":"2024-04-24T08:07:52.048093+0000","last_active":"2024-04-24T08:07:53.874364+0000","last_peered":"2024-04-24T08:07:53.874364+0000","last_clean":"2024-04-24T08:07:53.874364+0000","last_became_active":"2024-04-24T08:07:52.047727+0000","last_became_peered":"2024-04-24T08:07:52.047727+0000","last_unstale":"2024-04-24T08:07:53.874364+0000","last_undegraded":"2024-04-24T08:07:53.874364+0000","last_fullsized":"2024-04-24T08:07:53.874364+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:57:26.248355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.116643+0000","last_change":"2024-04-24T08:07:51.047810+0000","last_active":"2024-04-24T08:07:55.116643+0000","last_peered":"2024-04-24T08:07:55.116643+0000","last_clean":"2024-04-24T08:07:55.116643+0000","last_became_active":"2024-04-24T08:07:51.047501+0000","last_became_peered":"2024-04-24T08:07:51.047501+0000","last_unstale":"2024-04-24T08:07:55.116643+0000","last_undegraded":"2024-04-24T08:07:55.116643+0000","last_fullsized":"2024-04-24T08:07:55.116643+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:02:27.585202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.885516+0000","last_change":"2024-04-24T08:07:52.063782+0000","last_active":"2024-04-24T08:07:54.885516+0000","last_peered":"2024-04-24T08:07:54.885516+0000","last_clean":"2024-04-24T08:07:54.885516+0000","last_became_active":"2024-04-24T08:07:52.063056+0000","last_became_peered":"2024-04-24T08:07:52.063056+0000","last_unstale":"2024-04-24T08:07:54.885516+0000","last_undegraded":"2024-04-24T08:07:54.885516+0000","last_fullsized":"2024-04-24T08:07:54.885516+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:41:30.966787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.113012+0000","last_change":"2024-04-24T08:07:51.076139+0000","last_active":"2024-04-24T08:07:55.113012+0000","last_peered":"2024-04-24T08:07:55.113012+0000","last_clean":"2024-04-24T08:07:55.113012+0000","last_became_active":"2024-04-24T08:07:51.075842+0000","last_became_peered":"2024-04-24T08:07:51.075842+0000","last_unstale":"2024-04-24T08:07:55.113012+0000","last_undegraded":"2024-04-24T08:07:55.113012+0000","last_fullsized":"2024-04-24T08:07:55.113012+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:46:56.741726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.874272+0000","last_change":"2024-04-24T08:07:52.054058+0000","last_active":"2024-04-24T08:07:53.874272+0000","last_peered":"2024-04-24T08:07:53.874272+0000","last_clean":"2024-04-24T08:07:53.874272+0000","last_became_active":"2024-04-24T08:07:52.053142+0000","last_became_peered":"2024-04-24T08:07:52.053142+0000","last_unstale":"2024-04-24T08:07:53.874272+0000","last_undegraded":"2024-04-24T08:07:53.874272+0000","last_fullsized":"2024-04-24T08:07:53.874272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:44:43.554555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'8","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:57.144251+0000","last_change":"2024-04-24T08:07:51.048079+0000","last_active":"2024-04-24T08:07:57.144251+0000","last_peered":"2024-04-24T08:07:57.144251+0000","last_clean":"2024-04-24T08:07:57.144251+0000","last_became_active":"2024-04-24T08:07:51.047584+0000","last_became_peered":"2024-04-24T08:07:51.047584+0000","last_unstale":"2024-04-24T08:07:57.144251+0000","last_undegraded":"2024-04-24T08:07:57.144251+0000","last_fullsized":"2024-04-24T08:07:57.144251+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:09:05.554984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1612,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":3,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.088770+0000","last_change":"2024-04-24T08:07:52.055894+0000","last_active":"2024-04-24T08:07:53.088770+0000","last_peered":"2024-04-24T08:07:53.088770+0000","last_clean":"2024-04-24T08:07:53.088770+0000","last_became_active":"2024-04-24T08:07:52.055548+0000","last_became_peered":"2024-04-24T08:07:52.055548+0000","last_unstale":"2024-04-24T08:07:53.088770+0000","last_undegraded":"2024-04-24T08:07:53.088770+0000","last_fullsized":"2024-04-24T08:07:53.088770+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:33:22.609865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:56.139953+0000","last_change":"2024-04-24T08:07:51.073196+0000","last_active":"2024-04-24T08:07:56.139953+0000","last_peered":"2024-04-24T08:07:56.139953+0000","last_clean":"2024-04-24T08:07:56.139953+0000","last_became_active":"2024-04-24T08:07:51.072893+0000","last_became_peered":"2024-04-24T08:07:51.072893+0000","last_unstale":"2024-04-24T08:07:56.139953+0000","last_undegraded":"2024-04-24T08:07:56.139953+0000","last_fullsized":"2024-04-24T08:07:56.139953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:18:07.826148+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.144228+0000","last_change":"2024-04-24T08:07:52.069950+0000","last_active":"2024-04-24T08:07:53.144228+0000","last_peered":"2024-04-24T08:07:53.144228+0000","last_clean":"2024-04-24T08:07:53.144228+0000","last_became_active":"2024-04-24T08:07:52.069515+0000","last_became_peered":"2024-04-24T08:07:52.069515+0000","last_unstale":"2024-04-24T08:07:53.144228+0000","last_undegraded":"2024-04-24T08:07:53.144228+0000","last_fullsized":"2024-04-24T08:07:53.144228+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:52:27.530541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.114769+0000","last_change":"2024-04-24T08:07:51.067062+0000","last_active":"2024-04-24T08:07:55.114769+0000","last_peered":"2024-04-24T08:07:55.114769+0000","last_clean":"2024-04-24T08:07:55.114769+0000","last_became_active":"2024-04-24T08:07:51.066712+0000","last_became_peered":"2024-04-24T08:07:51.066712+0000","last_unstale":"2024-04-24T08:07:55.114769+0000","last_undegraded":"2024-04-24T08:07:55.114769+0000","last_fullsized":"2024-04-24T08:07:55.114769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:11:52.974470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884335+0000","last_change":"2024-04-24T08:07:52.064521+0000","last_active":"2024-04-24T08:07:54.884335+0000","last_peered":"2024-04-24T08:07:54.884335+0000","last_clean":"2024-04-24T08:07:54.884335+0000","last_became_active":"2024-04-24T08:07:52.064184+0000","last_became_peered":"2024-04-24T08:07:52.064184+0000","last_unstale":"2024-04-24T08:07:54.884335+0000","last_undegraded":"2024-04-24T08:07:54.884335+0000","last_fullsized":"2024-04-24T08:07:54.884335+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:00:27.294441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:58.148164+0000","last_change":"2024-04-24T08:07:51.043577+0000","last_active":"2024-04-24T08:07:58.148164+0000","last_peered":"2024-04-24T08:07:58.148164+0000","last_clean":"2024-04-24T08:07:58.148164+0000","last_became_active":"2024-04-24T08:07:51.043215+0000","last_became_peered":"2024-04-24T08:07:51.043215+0000","last_unstale":"2024-04-24T08:07:58.148164+0000","last_undegraded":"2024-04-24T08:07:58.148164+0000","last_fullsized":"2024-04-24T08:07:58.148164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:25:09.979236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":6,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.883646+0000","last_change":"2024-04-24T08:07:52.057633+0000","last_active":"2024-04-24T08:07:54.883646+0000","last_peered":"2024-04-24T08:07:54.883646+0000","last_clean":"2024-04-24T08:07:54.883646+0000","last_became_active":"2024-04-24T08:07:52.057309+0000","last_became_peered":"2024-04-24T08:07:52.057309+0000","last_unstale":"2024-04-24T08:07:54.883646+0000","last_undegraded":"2024-04-24T08:07:54.883646+0000","last_fullsized":"2024-04-24T08:07:54.883646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:49:56.045030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'10","reported_seq":25,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.119424+0000","last_change":"2024-04-24T08:07:51.058957+0000","last_active":"2024-04-24T08:07:55.119424+0000","last_peered":"2024-04-24T08:07:55.119424+0000","last_clean":"2024-04-24T08:07:55.119424+0000","last_became_active":"2024-04-24T08:07:51.058654+0000","last_became_peered":"2024-04-24T08:07:51.058654+0000","last_unstale":"2024-04-24T08:07:55.119424+0000","last_undegraded":"2024-04-24T08:07:55.119424+0000","last_fullsized":"2024-04-24T08:07:55.119424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:13.354590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":5,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884392+0000","last_change":"2024-04-24T08:07:52.051052+0000","last_active":"2024-04-24T08:07:54.884392+0000","last_peered":"2024-04-24T08:07:54.884392+0000","last_clean":"2024-04-24T08:07:54.884392+0000","last_became_active":"2024-04-24T08:07:52.050685+0000","last_became_peered":"2024-04-24T08:07:52.050685+0000","last_unstale":"2024-04-24T08:07:54.884392+0000","last_undegraded":"2024-04-24T08:07:54.884392+0000","last_fullsized":"2024-04-24T08:07:54.884392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:34.491653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.088809+0000","last_change":"2024-04-24T08:07:51.041088+0000","last_active":"2024-04-24T08:07:53.088809+0000","last_peered":"2024-04-24T08:07:53.088809+0000","last_clean":"2024-04-24T08:07:53.088809+0000","last_became_active":"2024-04-24T08:07:51.040752+0000","last_became_peered":"2024-04-24T08:07:51.040752+0000","last_unstale":"2024-04-24T08:07:53.088809+0000","last_undegraded":"2024-04-24T08:07:53.088809+0000","last_fullsized":"2024-04-24T08:07:53.088809+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:48:23.405705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884323+0000","last_change":"2024-04-24T08:07:52.058016+0000","last_active":"2024-04-24T08:07:54.884323+0000","last_peered":"2024-04-24T08:07:54.884323+0000","last_clean":"2024-04-24T08:07:54.884323+0000","last_became_active":"2024-04-24T08:07:52.057621+0000","last_became_peered":"2024-04-24T08:07:52.057621+0000","last_unstale":"2024-04-24T08:07:54.884323+0000","last_undegraded":"2024-04-24T08:07:54.884323+0000","last_fullsized":"2024-04-24T08:07:54.884323+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:57:40.651658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.042997+0000","last_change":"2024-04-24T08:07:51.075583+0000","last_active":"2024-04-24T08:07:53.042997+0000","last_peered":"2024-04-24T08:07:53.042997+0000","last_clean":"2024-04-24T08:07:53.042997+0000","last_became_active":"2024-04-24T08:07:51.075111+0000","last_became_peered":"2024-04-24T08:07:51.075111+0000","last_unstale":"2024-04-24T08:07:53.042997+0000","last_undegraded":"2024-04-24T08:07:53.042997+0000","last_fullsized":"2024-04-24T08:07:53.042997+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:56:00.820057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884938+0000","last_change":"2024-04-24T08:07:52.063232+0000","last_active":"2024-04-24T08:07:54.884938+0000","last_peered":"2024-04-24T08:07:54.884938+0000","last_clean":"2024-04-24T08:07:54.884938+0000","last_became_active":"2024-04-24T08:07:52.062868+0000","last_became_peered":"2024-04-24T08:07:52.062868+0000","last_unstale":"2024-04-24T08:07:54.884938+0000","last_undegraded":"2024-04-24T08:07:54.884938+0000","last_fullsized":"2024-04-24T08:07:54.884938+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:00:40.333389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":15,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.883681+0000","last_change":"2024-04-24T08:07:51.070647+0000","last_active":"2024-04-24T08:07:54.883681+0000","last_peered":"2024-04-24T08:07:54.883681+0000","last_clean":"2024-04-24T08:07:54.883681+0000","last_became_active":"2024-04-24T08:07:51.070298+0000","last_became_peered":"2024-04-24T08:07:51.070298+0000","last_unstale":"2024-04-24T08:07:54.883681+0000","last_undegraded":"2024-04-24T08:07:54.883681+0000","last_fullsized":"2024-04-24T08:07:54.883681+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:56:41.706796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884779+0000","last_change":"2024-04-24T08:07:52.073417+0000","last_active":"2024-04-24T08:07:54.884779+0000","last_peered":"2024-04-24T08:07:54.884779+0000","last_clean":"2024-04-24T08:07:54.884779+0000","last_became_active":"2024-04-24T08:07:52.073051+0000","last_became_peered":"2024-04-24T08:07:52.073051+0000","last_unstale":"2024-04-24T08:07:54.884779+0000","last_undegraded":"2024-04-24T08:07:54.884779+0000","last_fullsized":"2024-04-24T08:07:54.884779+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:54:58.338279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884334+0000","last_change":"2024-04-24T08:07:52.060497+0000","last_active":"2024-04-24T08:07:54.884334+0000","last_peered":"2024-04-24T08:07:54.884334+0000","last_clean":"2024-04-24T08:07:54.884334+0000","last_became_active":"2024-04-24T08:07:52.059914+0000","last_became_peered":"2024-04-24T08:07:52.059914+0000","last_unstale":"2024-04-24T08:07:54.884334+0000","last_undegraded":"2024-04-24T08:07:54.884334+0000","last_fullsized":"2024-04-24T08:07:54.884334+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:18:04.371830+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:57.144050+0000","last_change":"2024-04-24T08:07:51.076937+0000","last_active":"2024-04-24T08:07:57.144050+0000","last_peered":"2024-04-24T08:07:57.144050+0000","last_clean":"2024-04-24T08:07:57.144050+0000","last_became_active":"2024-04-24T08:07:51.076621+0000","last_became_peered":"2024-04-24T08:07:51.076621+0000","last_unstale":"2024-04-24T08:07:57.144050+0000","last_undegraded":"2024-04-24T08:07:57.144050+0000","last_fullsized":"2024-04-24T08:07:57.144050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:09.966994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.044070+0000","last_change":"2024-04-24T08:07:52.054150+0000","last_active":"2024-04-24T08:07:53.044070+0000","last_peered":"2024-04-24T08:07:53.044070+0000","last_clean":"2024-04-24T08:07:53.044070+0000","last_became_active":"2024-04-24T08:07:52.053817+0000","last_became_peered":"2024-04-24T08:07:52.053817+0000","last_unstale":"2024-04-24T08:07:53.044070+0000","last_undegraded":"2024-04-24T08:07:53.044070+0000","last_fullsized":"2024-04-24T08:07:53.044070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:46:16.570004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:57.141379+0000","last_change":"2024-04-24T08:07:51.045431+0000","last_active":"2024-04-24T08:07:57.141379+0000","last_peered":"2024-04-24T08:07:57.141379+0000","last_clean":"2024-04-24T08:07:57.141379+0000","last_became_active":"2024-04-24T08:07:51.045112+0000","last_became_peered":"2024-04-24T08:07:51.045112+0000","last_unstale":"2024-04-24T08:07:57.141379+0000","last_undegraded":"2024-04-24T08:07:57.141379+0000","last_fullsized":"2024-04-24T08:07:57.141379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:52:05.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":347,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.883583+0000","last_change":"2024-04-24T08:06:31.583310+0000","last_active":"2024-04-24T08:07:54.883583+0000","last_peered":"2024-04-24T08:07:54.883583+0000","last_clean":"2024-04-24T08:07:54.883583+0000","last_became_active":"2024-04-24T08:06:31.582735+0000","last_became_peered":"2024-04-24T08:06:31.582735+0000","last_unstale":"2024-04-24T08:07:54.883583+0000","last_undegraded":"2024-04-24T08:07:54.883583+0000","last_fullsized":"2024-04-24T08:07:54.883583+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:02:34.411025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:02:34.411025+0000","last_clean_scrub_stamp":"2024-04-24T08:02:34.411025+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:48.816092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.143981+0000","last_change":"2024-04-24T08:07:52.063961+0000","last_active":"2024-04-24T08:07:53.143981+0000","last_peered":"2024-04-24T08:07:53.143981+0000","last_clean":"2024-04-24T08:07:53.143981+0000","last_became_active":"2024-04-24T08:07:52.063608+0000","last_became_peered":"2024-04-24T08:07:52.063608+0000","last_unstale":"2024-04-24T08:07:53.143981+0000","last_undegraded":"2024-04-24T08:07:53.143981+0000","last_fullsized":"2024-04-24T08:07:53.143981+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:40:50.233349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.883848+0000","last_change":"2024-04-24T08:07:51.045626+0000","last_active":"2024-04-24T08:07:54.883848+0000","last_peered":"2024-04-24T08:07:54.883848+0000","last_clean":"2024-04-24T08:07:54.883848+0000","last_became_active":"2024-04-24T08:07:51.045293+0000","last_became_peered":"2024-04-24T08:07:51.045293+0000","last_unstale":"2024-04-24T08:07:54.883848+0000","last_undegraded":"2024-04-24T08:07:54.883848+0000","last_fullsized":"2024-04-24T08:07:54.883848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:13:04.601331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.088774+0000","last_change":"2024-04-24T08:07:51.079228+0000","last_active":"2024-04-24T08:07:53.088774+0000","last_peered":"2024-04-24T08:07:53.088774+0000","last_clean":"2024-04-24T08:07:53.088774+0000","last_became_active":"2024-04-24T08:07:51.078932+0000","last_became_peered":"2024-04-24T08:07:51.078932+0000","last_unstale":"2024-04-24T08:07:53.088774+0000","last_undegraded":"2024-04-24T08:07:53.088774+0000","last_fullsized":"2024-04-24T08:07:53.088774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:58:32.024133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.144627+0000","last_change":"2024-04-24T08:07:52.065943+0000","last_active":"2024-04-24T08:07:53.144627+0000","last_peered":"2024-04-24T08:07:53.144627+0000","last_clean":"2024-04-24T08:07:53.144627+0000","last_became_active":"2024-04-24T08:07:52.062810+0000","last_became_peered":"2024-04-24T08:07:52.062810+0000","last_unstale":"2024-04-24T08:07:53.144627+0000","last_undegraded":"2024-04-24T08:07:53.144627+0000","last_fullsized":"2024-04-24T08:07:53.144627+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:40:08.803652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:56.143145+0000","last_change":"2024-04-24T08:07:51.071118+0000","last_active":"2024-04-24T08:07:56.143145+0000","last_peered":"2024-04-24T08:07:56.143145+0000","last_clean":"2024-04-24T08:07:56.143145+0000","last_became_active":"2024-04-24T08:07:51.070678+0000","last_became_peered":"2024-04-24T08:07:51.070678+0000","last_unstale":"2024-04-24T08:07:56.143145+0000","last_undegraded":"2024-04-24T08:07:56.143145+0000","last_fullsized":"2024-04-24T08:07:56.143145+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:57:44.212247+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.044818+0000","last_change":"2024-04-24T08:07:52.058764+0000","last_active":"2024-04-24T08:07:53.044818+0000","last_peered":"2024-04-24T08:07:53.044818+0000","last_clean":"2024-04-24T08:07:53.044818+0000","last_became_active":"2024-04-24T08:07:52.058336+0000","last_became_peered":"2024-04-24T08:07:52.058336+0000","last_unstale":"2024-04-24T08:07:53.044818+0000","last_undegraded":"2024-04-24T08:07:53.044818+0000","last_fullsized":"2024-04-24T08:07:53.044818+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:05:37.245521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.114282+0000","last_change":"2024-04-24T08:07:51.067154+0000","last_active":"2024-04-24T08:07:55.114282+0000","last_peered":"2024-04-24T08:07:55.114282+0000","last_clean":"2024-04-24T08:07:55.114282+0000","last_became_active":"2024-04-24T08:07:51.066769+0000","last_became_peered":"2024-04-24T08:07:51.066769+0000","last_unstale":"2024-04-24T08:07:55.114282+0000","last_undegraded":"2024-04-24T08:07:55.114282+0000","last_fullsized":"2024-04-24T08:07:55.114282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:39:25.908189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.885189+0000","last_change":"2024-04-24T08:07:52.072134+0000","last_active":"2024-04-24T08:07:54.885189+0000","last_peered":"2024-04-24T08:07:54.885189+0000","last_clean":"2024-04-24T08:07:54.885189+0000","last_became_active":"2024-04-24T08:07:52.071703+0000","last_became_peered":"2024-04-24T08:07:52.071703+0000","last_unstale":"2024-04-24T08:07:54.885189+0000","last_undegraded":"2024-04-24T08:07:54.885189+0000","last_fullsized":"2024-04-24T08:07:54.885189+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:59.829988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.118999+0000","last_change":"2024-04-24T08:07:51.059151+0000","last_active":"2024-04-24T08:07:55.118999+0000","last_peered":"2024-04-24T08:07:55.118999+0000","last_clean":"2024-04-24T08:07:55.118999+0000","last_became_active":"2024-04-24T08:07:51.058872+0000","last_became_peered":"2024-04-24T08:07:51.058872+0000","last_unstale":"2024-04-24T08:07:55.118999+0000","last_undegraded":"2024-04-24T08:07:55.118999+0000","last_fullsized":"2024-04-24T08:07:55.118999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:36.023595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.088429+0000","last_change":"2024-04-24T08:07:52.056919+0000","last_active":"2024-04-24T08:07:53.088429+0000","last_peered":"2024-04-24T08:07:53.088429+0000","last_clean":"2024-04-24T08:07:53.088429+0000","last_became_active":"2024-04-24T08:07:52.056573+0000","last_became_peered":"2024-04-24T08:07:52.056573+0000","last_unstale":"2024-04-24T08:07:53.088429+0000","last_undegraded":"2024-04-24T08:07:53.088429+0000","last_fullsized":"2024-04-24T08:07:53.088429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:58:17.939465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.121925+0000","last_change":"2024-04-24T08:07:51.056325+0000","last_active":"2024-04-24T08:07:55.121925+0000","last_peered":"2024-04-24T08:07:55.121925+0000","last_clean":"2024-04-24T08:07:55.121925+0000","last_became_active":"2024-04-24T08:07:51.055862+0000","last_became_peered":"2024-04-24T08:07:51.055862+0000","last_unstale":"2024-04-24T08:07:55.121925+0000","last_undegraded":"2024-04-24T08:07:55.121925+0000","last_fullsized":"2024-04-24T08:07:55.121925+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:11:44.487530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.883971+0000","last_change":"2024-04-24T08:07:52.065570+0000","last_active":"2024-04-24T08:07:54.883971+0000","last_peered":"2024-04-24T08:07:54.883971+0000","last_clean":"2024-04-24T08:07:54.883971+0000","last_became_active":"2024-04-24T08:07:52.065196+0000","last_became_peered":"2024-04-24T08:07:52.065196+0000","last_unstale":"2024-04-24T08:07:54.883971+0000","last_undegraded":"2024-04-24T08:07:54.883971+0000","last_fullsized":"2024-04-24T08:07:54.883971+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:48:09.474159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.423562+0000","last_change":"2024-04-24T08:07:51.067187+0000","last_active":"2024-04-24T08:07:54.423562+0000","last_peered":"2024-04-24T08:07:54.423562+0000","last_clean":"2024-04-24T08:07:54.423562+0000","last_became_active":"2024-04-24T08:07:51.066808+0000","last_became_peered":"2024-04-24T08:07:51.066808+0000","last_unstale":"2024-04-24T08:07:54.423562+0000","last_undegraded":"2024-04-24T08:07:54.423562+0000","last_fullsized":"2024-04-24T08:07:54.423562+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:32:11.852191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.044285+0000","last_change":"2024-04-24T08:07:52.078426+0000","last_active":"2024-04-24T08:07:53.044285+0000","last_peered":"2024-04-24T08:07:53.044285+0000","last_clean":"2024-04-24T08:07:53.044285+0000","last_became_active":"2024-04-24T08:07:52.078076+0000","last_became_peered":"2024-04-24T08:07:52.078076+0000","last_unstale":"2024-04-24T08:07:53.044285+0000","last_undegraded":"2024-04-24T08:07:53.044285+0000","last_fullsized":"2024-04-24T08:07:53.044285+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:55:29.885688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:55.117614+0000","last_change":"2024-04-24T08:07:51.042601+0000","last_active":"2024-04-24T08:07:55.117614+0000","last_peered":"2024-04-24T08:07:55.117614+0000","last_clean":"2024-04-24T08:07:55.117614+0000","last_became_active":"2024-04-24T08:07:51.042252+0000","last_became_peered":"2024-04-24T08:07:51.042252+0000","last_unstale":"2024-04-24T08:07:55.117614+0000","last_undegraded":"2024-04-24T08:07:55.117614+0000","last_fullsized":"2024-04-24T08:07:55.117614+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:11:44.640303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884194+0000","last_change":"2024-04-24T08:07:52.063142+0000","last_active":"2024-04-24T08:07:54.884194+0000","last_peered":"2024-04-24T08:07:54.884194+0000","last_clean":"2024-04-24T08:07:54.884194+0000","last_became_active":"2024-04-24T08:07:52.062263+0000","last_became_peered":"2024-04-24T08:07:52.062263+0000","last_unstale":"2024-04-24T08:07:54.884194+0000","last_undegraded":"2024-04-24T08:07:54.884194+0000","last_fullsized":"2024-04-24T08:07:54.884194+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:38.879380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:56.141666+0000","last_change":"2024-04-24T08:07:51.075154+0000","last_active":"2024-04-24T08:07:56.141666+0000","last_peered":"2024-04-24T08:07:56.141666+0000","last_clean":"2024-04-24T08:07:56.141666+0000","last_became_active":"2024-04-24T08:07:51.074866+0000","last_became_peered":"2024-04-24T08:07:51.074866+0000","last_unstale":"2024-04-24T08:07:56.141666+0000","last_undegraded":"2024-04-24T08:07:56.141666+0000","last_fullsized":"2024-04-24T08:07:56.141666+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:58:14.810693+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884178+0000","last_change":"2024-04-24T08:07:52.055958+0000","last_active":"2024-04-24T08:07:54.884178+0000","last_peered":"2024-04-24T08:07:54.884178+0000","last_clean":"2024-04-24T08:07:54.884178+0000","last_became_active":"2024-04-24T08:07:52.055598+0000","last_became_peered":"2024-04-24T08:07:52.055598+0000","last_unstale":"2024-04-24T08:07:54.884178+0000","last_undegraded":"2024-04-24T08:07:54.884178+0000","last_fullsized":"2024-04-24T08:07:54.884178+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:12:22.452576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884830+0000","last_change":"2024-04-24T08:07:51.073234+0000","last_active":"2024-04-24T08:07:54.884830+0000","last_peered":"2024-04-24T08:07:54.884830+0000","last_clean":"2024-04-24T08:07:54.884830+0000","last_became_active":"2024-04-24T08:07:51.072887+0000","last_became_peered":"2024-04-24T08:07:51.072887+0000","last_unstale":"2024-04-24T08:07:54.884830+0000","last_undegraded":"2024-04-24T08:07:54.884830+0000","last_fullsized":"2024-04-24T08:07:54.884830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:28:15.908859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884070+0000","last_change":"2024-04-24T08:07:52.061188+0000","last_active":"2024-04-24T08:07:54.884070+0000","last_peered":"2024-04-24T08:07:54.884070+0000","last_clean":"2024-04-24T08:07:54.884070+0000","last_became_active":"2024-04-24T08:07:52.059616+0000","last_became_peered":"2024-04-24T08:07:52.059616+0000","last_unstale":"2024-04-24T08:07:54.884070+0000","last_undegraded":"2024-04-24T08:07:54.884070+0000","last_fullsized":"2024-04-24T08:07:54.884070+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:43.546091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.144824+0000","last_change":"2024-04-24T08:07:51.078300+0000","last_active":"2024-04-24T08:07:53.144824+0000","last_peered":"2024-04-24T08:07:53.144824+0000","last_clean":"2024-04-24T08:07:53.144824+0000","last_became_active":"2024-04-24T08:07:51.076026+0000","last_became_peered":"2024-04-24T08:07:51.076026+0000","last_unstale":"2024-04-24T08:07:53.144824+0000","last_undegraded":"2024-04-24T08:07:53.144824+0000","last_fullsized":"2024-04-24T08:07:53.144824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:45:31.494213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884271+0000","last_change":"2024-04-24T08:07:52.056148+0000","last_active":"2024-04-24T08:07:54.884271+0000","last_peered":"2024-04-24T08:07:54.884271+0000","last_clean":"2024-04-24T08:07:54.884271+0000","last_became_active":"2024-04-24T08:07:52.055768+0000","last_became_peered":"2024-04-24T08:07:52.055768+0000","last_unstale":"2024-04-24T08:07:54.884271+0000","last_undegraded":"2024-04-24T08:07:54.884271+0000","last_fullsized":"2024-04-24T08:07:54.884271+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:52:48.503479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884068+0000","last_change":"2024-04-24T08:07:52.056611+0000","last_active":"2024-04-24T08:07:54.884068+0000","last_peered":"2024-04-24T08:07:54.884068+0000","last_clean":"2024-04-24T08:07:54.884068+0000","last_became_active":"2024-04-24T08:07:52.055788+0000","last_became_peered":"2024-04-24T08:07:52.055788+0000","last_unstale":"2024-04-24T08:07:54.884068+0000","last_undegraded":"2024-04-24T08:07:54.884068+0000","last_fullsized":"2024-04-24T08:07:54.884068+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:21:28.997588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.042281+0000","last_change":"2024-04-24T08:07:52.059952+0000","last_active":"2024-04-24T08:07:53.042281+0000","last_peered":"2024-04-24T08:07:53.042281+0000","last_clean":"2024-04-24T08:07:53.042281+0000","last_became_active":"2024-04-24T08:07:52.059612+0000","last_became_peered":"2024-04-24T08:07:52.059612+0000","last_unstale":"2024-04-24T08:07:53.042281+0000","last_undegraded":"2024-04-24T08:07:53.042281+0000","last_fullsized":"2024-04-24T08:07:53.042281+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:19:09.989429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.042226+0000","last_change":"2024-04-24T08:07:52.070899+0000","last_active":"2024-04-24T08:07:53.042226+0000","last_peered":"2024-04-24T08:07:53.042226+0000","last_clean":"2024-04-24T08:07:53.042226+0000","last_became_active":"2024-04-24T08:07:52.070556+0000","last_became_peered":"2024-04-24T08:07:52.070556+0000","last_unstale":"2024-04-24T08:07:53.042226+0000","last_undegraded":"2024-04-24T08:07:53.042226+0000","last_fullsized":"2024-04-24T08:07:53.042226+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:11.381122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.885498+0000","last_change":"2024-04-24T08:07:52.068524+0000","last_active":"2024-04-24T08:07:54.885498+0000","last_peered":"2024-04-24T08:07:54.885498+0000","last_clean":"2024-04-24T08:07:54.885498+0000","last_became_active":"2024-04-24T08:07:52.068173+0000","last_became_peered":"2024-04-24T08:07:52.068173+0000","last_unstale":"2024-04-24T08:07:54.885498+0000","last_undegraded":"2024-04-24T08:07:54.885498+0000","last_fullsized":"2024-04-24T08:07:54.885498+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:25:08.390138+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.144715+0000","last_change":"2024-04-24T08:07:52.078604+0000","last_active":"2024-04-24T08:07:53.144715+0000","last_peered":"2024-04-24T08:07:53.144715+0000","last_clean":"2024-04-24T08:07:53.144715+0000","last_became_active":"2024-04-24T08:07:52.078122+0000","last_became_peered":"2024-04-24T08:07:52.078122+0000","last_unstale":"2024-04-24T08:07:53.144715+0000","last_undegraded":"2024-04-24T08:07:53.144715+0000","last_fullsized":"2024-04-24T08:07:53.144715+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:24:19.062821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884398+0000","last_change":"2024-04-24T08:07:52.055986+0000","last_active":"2024-04-24T08:07:54.884398+0000","last_peered":"2024-04-24T08:07:54.884398+0000","last_clean":"2024-04-24T08:07:54.884398+0000","last_became_active":"2024-04-24T08:07:52.055609+0000","last_became_peered":"2024-04-24T08:07:52.055609+0000","last_unstale":"2024-04-24T08:07:54.884398+0000","last_undegraded":"2024-04-24T08:07:54.884398+0000","last_fullsized":"2024-04-24T08:07:54.884398+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:38:04.731204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.042646+0000","last_change":"2024-04-24T08:07:52.055605+0000","last_active":"2024-04-24T08:07:53.042646+0000","last_peered":"2024-04-24T08:07:53.042646+0000","last_clean":"2024-04-24T08:07:53.042646+0000","last_became_active":"2024-04-24T08:07:52.055268+0000","last_became_peered":"2024-04-24T08:07:52.055268+0000","last_unstale":"2024-04-24T08:07:53.042646+0000","last_undegraded":"2024-04-24T08:07:53.042646+0000","last_fullsized":"2024-04-24T08:07:53.042646+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:50:29.197484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.044650+0000","last_change":"2024-04-24T08:07:52.064590+0000","last_active":"2024-04-24T08:07:53.044650+0000","last_peered":"2024-04-24T08:07:53.044650+0000","last_clean":"2024-04-24T08:07:53.044650+0000","last_became_active":"2024-04-24T08:07:52.064113+0000","last_became_peered":"2024-04-24T08:07:52.064113+0000","last_unstale":"2024-04-24T08:07:53.044650+0000","last_undegraded":"2024-04-24T08:07:53.044650+0000","last_fullsized":"2024-04-24T08:07:53.044650+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:36:10.926145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884318+0000","last_change":"2024-04-24T08:07:52.053757+0000","last_active":"2024-04-24T08:07:54.884318+0000","last_peered":"2024-04-24T08:07:54.884318+0000","last_clean":"2024-04-24T08:07:54.884318+0000","last_became_active":"2024-04-24T08:07:52.053380+0000","last_became_peered":"2024-04-24T08:07:52.053380+0000","last_unstale":"2024-04-24T08:07:54.884318+0000","last_undegraded":"2024-04-24T08:07:54.884318+0000","last_fullsized":"2024-04-24T08:07:54.884318+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:45:27.284230+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.043732+0000","last_change":"2024-04-24T08:07:52.068308+0000","last_active":"2024-04-24T08:07:53.043732+0000","last_peered":"2024-04-24T08:07:53.043732+0000","last_clean":"2024-04-24T08:07:53.043732+0000","last_became_active":"2024-04-24T08:07:52.067786+0000","last_became_peered":"2024-04-24T08:07:52.067786+0000","last_unstale":"2024-04-24T08:07:53.043732+0000","last_undegraded":"2024-04-24T08:07:53.043732+0000","last_fullsized":"2024-04-24T08:07:53.043732+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:26:09.349120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.874142+0000","last_change":"2024-04-24T08:07:52.053089+0000","last_active":"2024-04-24T08:07:53.874142+0000","last_peered":"2024-04-24T08:07:53.874142+0000","last_clean":"2024-04-24T08:07:53.874142+0000","last_became_active":"2024-04-24T08:07:52.052753+0000","last_became_peered":"2024-04-24T08:07:52.052753+0000","last_unstale":"2024-04-24T08:07:53.874142+0000","last_undegraded":"2024-04-24T08:07:53.874142+0000","last_fullsized":"2024-04-24T08:07:53.874142+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:02:14.583922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:53.144314+0000","last_change":"2024-04-24T08:07:52.076633+0000","last_active":"2024-04-24T08:07:53.144314+0000","last_peered":"2024-04-24T08:07:53.144314+0000","last_clean":"2024-04-24T08:07:53.144314+0000","last_became_active":"2024-04-24T08:07:52.076280+0000","last_became_peered":"2024-04-24T08:07:52.076280+0000","last_unstale":"2024-04-24T08:07:53.144314+0000","last_undegraded":"2024-04-24T08:07:53.144314+0000","last_fullsized":"2024-04-24T08:07:53.144314+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:29:59.823772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-04-24T08:07:54.884007+0000","last_change":"2024-04-24T08:07:52.053017+0000","last_active":"2024-04-24T08:07:54.884007+0000","last_peered":"2024-04-24T08:07:54.884007+0000","last_clean":"2024-04-24T08:07:54.884007+0000","last_became_active":"2024-04-24T08:07:52.052325+0000","last_became_peered":"2024-04-24T08:07:52.052325+0000","last_unstale":"2024-04-24T08:07:54.884007+0000","last_undegraded":"2024-04-24T08:07:54.884007+0000","last_fullsized":"2024-04-24T08:07:54.884007+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:05:58.161360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":213304,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":5892,"num_objects":54,"num_object_clones":0,"num_object_copies":162,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":54,"num_whiteouts":0,"num_read":25,"num_read_kb":20,"num_write":116,"num_write_kb":32,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":36,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":266240,"data_stored":52138,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350991,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36244,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708908,"statfs":{"total":95995035648,"available":95957921792,"internally_reserved":0,"allocated":925696,"data_stored":444311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612627,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1019904,"data_stored":1037869,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776153,"num_pgs":10,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27940,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717212,"statfs":{"total":95995035648,"available":95966425088,"internally_reserved":0,"allocated":942080,"data_stored":445095,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037791,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":444937,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332133,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":987136,"data_stored":1035883,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495660,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27924,"kb_used_data":908,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717228,"statfs":{"total":95995035648,"available":95966441472,"internally_reserved":0,"allocated":929792,"data_stored":444401,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574919,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":937984,"data_stored":445923,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738445,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":932,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":954368,"data_stored":445789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411395,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":471562,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182460,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27936,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717216,"statfs":{"total":95995035648,"available":95966429184,"internally_reserved":0,"allocated":942080,"data_stored":444595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":937984,"data_stored":445346,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757298,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":1003520,"data_stored":1036546,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":39934,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1035,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1204,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3190,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:08:01.624 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-04-24T08:08:01.624 INFO:tasks.cephadm:Setting up client nodes... 2024-04-24T08:08:01.625 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-04-24T08:08:01.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:01 smithi117 ceph-mon[32438]: pgmap v321: 97 pgs: 97 active+clean; 582 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4.0 KiB/s wr, 17 op/s 2024-04-24T08:08:01.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:01 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3672844364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:08:02.126 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:02.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:02 smithi194 ceph-mon[32836]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:08:02.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:02 smithi194 ceph-mon[32836]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:08:02.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:02 smithi028 ceph-mon[27786]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:08:02.493 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:02 smithi028 ceph-mon[27786]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:08:02.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:02 smithi117 ceph-mon[32438]: from='client.14604 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:08:02.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:02 smithi117 ceph-mon[32438]: from='client.14610 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:08:03.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:03 smithi194 ceph-mon[32836]: pgmap v322: 97 pgs: 97 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4.4 KiB/s wr, 16 op/s 2024-04-24T08:08:03.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:03 smithi028 ceph-mon[27786]: pgmap v322: 97 pgs: 97 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4.4 KiB/s wr, 16 op/s 2024-04-24T08:08:03.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:03 smithi117 ceph-mon[32438]: pgmap v322: 97 pgs: 97 active+clean; 583 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4.4 KiB/s wr, 16 op/s 2024-04-24T08:08:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.623+0000 7f3841400700 1 -- 172.21.15.28:0/3828588304 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f383c102770 msgr2=0x7f383c102bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.623+0000 7f3841400700 1 --2- 172.21.15.28:0/3828588304 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f383c102770 0x7f383c102bf0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f383400b0a0 tx=0x7f383400b3b0 comp rx=0 tx=0).stop 2024-04-24T08:08:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.623+0000 7f3841400700 1 -- 172.21.15.28:0/3828588304 shutdown_connections 2024-04-24T08:08:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.623+0000 7f3841400700 1 --2- 172.21.15.28:0/3828588304 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f383c102770 0x7f383c102bf0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.623+0000 7f3841400700 1 --2- 172.21.15.28:0/3828588304 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383c101450 0x7f383c101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.623+0000 7f3841400700 1 --2- 172.21.15.28:0/3828588304 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f383c101db0 0x7f383c102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.623+0000 7f3841400700 1 -- 172.21.15.28:0/3828588304 >> 172.21.15.28:0/3828588304 conn(0x7f383c0fd010 msgr2=0x7f383c0ff450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.623+0000 7f3841400700 1 -- 172.21.15.28:0/3828588304 shutdown_connections 2024-04-24T08:08:04.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.623+0000 7f3841400700 1 -- 172.21.15.28:0/3828588304 wait complete. 2024-04-24T08:08:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.624+0000 7f3841400700 1 Processor -- start 2024-04-24T08:08:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.624+0000 7f3841400700 1 -- start start 2024-04-24T08:08:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.625+0000 7f3841400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383c101450 0x7f383c19ae60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.625+0000 7f3841400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f383c101db0 0x7f383c19b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.625+0000 7f3841400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f383c102770 0x7f383c1a23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:04.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.625+0000 7f3841400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f383c110970 con 0x7f383c101450 2024-04-24T08:08:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.625+0000 7f3841400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f383c1107f0 con 0x7f383c102770 2024-04-24T08:08:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.625+0000 7f3841400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f383c110af0 con 0x7f383c101db0 2024-04-24T08:08:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.625+0000 7f3839e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f383c101db0 0x7f383c19b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.625+0000 7f383a800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383c101450 0x7f383c19ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.625+0000 7f3839e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f383c101db0 0x7f383c19b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:46518/0 (socket says 172.21.15.28:46518) 2024-04-24T08:08:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.625+0000 7f3839e00700 1 -- 172.21.15.28:0/1566030664 learned_addr learned my addr 172.21.15.28:0/1566030664 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.625+0000 7f383b200700 1 --2- 172.21.15.28:0/1566030664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f383c102770 0x7f383c1a23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.626+0000 7f3839e00700 1 -- 172.21.15.28:0/1566030664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f383c102770 msgr2=0x7f383c1a23d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.626+0000 7f3839e00700 1 --2- 172.21.15.28:0/1566030664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f383c102770 0x7f383c1a23d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.626+0000 7f3839e00700 1 -- 172.21.15.28:0/1566030664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383c101450 msgr2=0x7f383c19ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.626+0000 7f3839e00700 1 --2- 172.21.15.28:0/1566030664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383c101450 0x7f383c19ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.626+0000 7f3839e00700 1 -- 172.21.15.28:0/1566030664 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f383c1a2a70 con 0x7f383c101db0 2024-04-24T08:08:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.626+0000 7f383a800700 1 --2- 172.21.15.28:0/1566030664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383c101450 0x7f383c19ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:04.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.626+0000 7f3839e00700 1 --2- 172.21.15.28:0/1566030664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f383c101db0 0x7f383c19b3a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f382800e9d0 tx=0x7f382800ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:04.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.626+0000 7f3833400700 1 -- 172.21.15.28:0/1566030664 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f382800cbd0 con 0x7f383c101db0 2024-04-24T08:08:04.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.626+0000 7f3833400700 1 -- 172.21.15.28:0/1566030664 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f382800ee70 con 0x7f383c101db0 2024-04-24T08:08:04.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.626+0000 7f3841400700 1 -- 172.21.15.28:0/1566030664 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f383c1a2d60 con 0x7f383c101db0 2024-04-24T08:08:04.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.626+0000 7f3833400700 1 -- 172.21.15.28:0/1566030664 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3828018660 con 0x7f383c101db0 2024-04-24T08:08:04.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.626+0000 7f3841400700 1 -- 172.21.15.28:0/1566030664 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f383c1a3260 con 0x7f383c101db0 2024-04-24T08:08:04.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.628+0000 7f3841400700 1 -- 172.21.15.28:0/1566030664 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f383c02dfe0 con 0x7f383c101db0 2024-04-24T08:08:04.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.630+0000 7f3833400700 1 -- 172.21.15.28:0/1566030664 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3828010430 con 0x7f383c101db0 2024-04-24T08:08:04.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.631+0000 7f3833400700 1 --2- 172.21.15.28:0/1566030664 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f38200a3e00 0x7f38200a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:04.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.631+0000 7f3833400700 1 -- 172.21.15.28:0/1566030664 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f38280d05e0 con 0x7f383c101db0 2024-04-24T08:08:04.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.631+0000 7f383a800700 1 --2- 172.21.15.28:0/1566030664 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f38200a3e00 0x7f38200a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:04.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.632+0000 7f383a800700 1 --2- 172.21.15.28:0/1566030664 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f38200a3e00 0x7f38200a62c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f382c009680 tx=0x7f382c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:04.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.634+0000 7f3833400700 1 -- 172.21.15.28:0/1566030664 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f382809cf10 con 0x7f383c101db0 2024-04-24T08:08:04.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.955+0000 7f3841400700 1 -- 172.21.15.28:0/1566030664 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f383c02e1e0 con 0x7f383c101db0 2024-04-24T08:08:04.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.964+0000 7f3833400700 1 -- 172.21.15.28:0/1566030664 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f382800cd70 con 0x7f383c101db0 2024-04-24T08:08:04.965 INFO:teuthology.orchestra.run.smithi028.stdout:[client.0] 2024-04-24T08:08:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: key = AQDkvShmrBIiORAAbKWyWQK2iGPxPRJ55XF9+g== 2024-04-24T08:08:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.967+0000 7f3841400700 1 -- 172.21.15.28:0/1566030664 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f38200a3e00 msgr2=0x7f38200a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.967+0000 7f3841400700 1 --2- 172.21.15.28:0/1566030664 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f38200a3e00 0x7f38200a62c0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f382c009680 tx=0x7f382c009260 comp rx=0 tx=0).stop 2024-04-24T08:08:04.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.967+0000 7f3841400700 1 -- 172.21.15.28:0/1566030664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f383c101db0 msgr2=0x7f383c19b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.967+0000 7f3841400700 1 --2- 172.21.15.28:0/1566030664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f383c101db0 0x7f383c19b3a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f382800e9d0 tx=0x7f382800ece0 comp rx=0 tx=0).stop 2024-04-24T08:08:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.967+0000 7f3841400700 1 -- 172.21.15.28:0/1566030664 shutdown_connections 2024-04-24T08:08:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.967+0000 7f3841400700 1 --2- 172.21.15.28:0/1566030664 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f38200a3e00 0x7f38200a62c0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.967+0000 7f3841400700 1 --2- 172.21.15.28:0/1566030664 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f383c102770 0x7f383c1a23d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.967+0000 7f3841400700 1 --2- 172.21.15.28:0/1566030664 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f383c101450 0x7f383c19ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.967+0000 7f3841400700 1 --2- 172.21.15.28:0/1566030664 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f383c101db0 0x7f383c19b3a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.967+0000 7f3841400700 1 -- 172.21.15.28:0/1566030664 >> 172.21.15.28:0/1566030664 conn(0x7f383c0fd010 msgr2=0x7f383c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.967+0000 7f3841400700 1 -- 172.21.15.28:0/1566030664 shutdown_connections 2024-04-24T08:08:04.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:04.967+0000 7f3841400700 1 -- 172.21.15.28:0/1566030664 wait complete. 2024-04-24T08:08:05.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:05 smithi028 ceph-mon[27786]: pgmap v323: 97 pgs: 97 active+clean; 618 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.3 KiB/s wr, 18 op/s 2024-04-24T08:08:05.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:05 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1566030664' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T08:08:05.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:05 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T08:08:05.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:05 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-24T08:08:05.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:05 smithi194 ceph-mon[32836]: pgmap v323: 97 pgs: 97 active+clean; 618 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.3 KiB/s wr, 18 op/s 2024-04-24T08:08:05.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:05 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1566030664' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T08:08:05.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:05 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T08:08:05.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:05 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-24T08:08:05.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:05 smithi117 ceph-mon[32438]: pgmap v323: 97 pgs: 97 active+clean; 618 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 7.3 KiB/s wr, 18 op/s 2024-04-24T08:08:05.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:05 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1566030664' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T08:08:05.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:05 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-04-24T08:08:05.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:05 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-04-24T08:08:05.928 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T08:08:05.928 DEBUG:teuthology.orchestra.run.smithi028:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-04-24T08:08:05.929 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-04-24T08:08:06.048 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-04-24T08:08:06.049 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-04-24T08:08:06.049 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph mgr dump --format=json 2024-04-24T08:08:06.543 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:07.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:07 smithi028 ceph-mon[27786]: pgmap v324: 97 pgs: 97 active+clean; 618 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 6.6 KiB/s wr, 16 op/s 2024-04-24T08:08:07.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:07 smithi117 ceph-mon[32438]: pgmap v324: 97 pgs: 97 active+clean; 618 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 6.6 KiB/s wr, 16 op/s 2024-04-24T08:08:07.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:07 smithi194 ceph-mon[32836]: pgmap v324: 97 pgs: 97 active+clean; 618 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 6.4 KiB/s rd, 6.6 KiB/s wr, 16 op/s 2024-04-24T08:08:08.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.936+0000 7efd9a800700 1 -- 172.21.15.28:0/1420176806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd94102b40 msgr2=0x7efd94102fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:08.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.936+0000 7efd9a800700 1 --2- 172.21.15.28:0/1420176806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd94102b40 0x7efd94102fc0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7efd7c009990 tx=0x7efd7c009ca0 comp rx=0 tx=0).stop 2024-04-24T08:08:08.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.936+0000 7efd9a800700 1 -- 172.21.15.28:0/1420176806 shutdown_connections 2024-04-24T08:08:08.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.936+0000 7efd9a800700 1 --2- 172.21.15.28:0/1420176806 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd94103500 0x7efd9410f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:08.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.936+0000 7efd9a800700 1 --2- 172.21.15.28:0/1420176806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd94108b90 0x7efd94108f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:08.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.936+0000 7efd9a800700 1 --2- 172.21.15.28:0/1420176806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd94102b40 0x7efd94102fc0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:08.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.936+0000 7efd9a800700 1 -- 172.21.15.28:0/1420176806 >> 172.21.15.28:0/1420176806 conn(0x7efd94079c50 msgr2=0x7efd9407a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:08.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.936+0000 7efd9a800700 1 -- 172.21.15.28:0/1420176806 shutdown_connections 2024-04-24T08:08:08.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.936+0000 7efd9a800700 1 -- 172.21.15.28:0/1420176806 wait complete. 2024-04-24T08:08:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.937+0000 7efd9a800700 1 Processor -- start 2024-04-24T08:08:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.938+0000 7efd9a800700 1 -- start start 2024-04-24T08:08:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.938+0000 7efd9a800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd94102b40 0x7efd9419b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.938+0000 7efd9a800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd94103500 0x7efd9419ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:08.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.938+0000 7efd9a800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd94108b90 0x7efd9419fe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.938+0000 7efd9a800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd94112e00 con 0x7efd94102b40 2024-04-24T08:08:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.938+0000 7efd9a800700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efd94112c80 con 0x7efd94108b90 2024-04-24T08:08:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.938+0000 7efd9a800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efd94112f80 con 0x7efd94103500 2024-04-24T08:08:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.939+0000 7efd93400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd94102b40 0x7efd9419b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.939+0000 7efd93400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd94102b40 0x7efd9419b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56770/0 (socket says 172.21.15.28:56770) 2024-04-24T08:08:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.939+0000 7efd8aa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd94103500 0x7efd9419ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.939+0000 7efd93400700 1 -- 172.21.15.28:0/3388621030 learned_addr learned my addr 172.21.15.28:0/3388621030 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.939+0000 7efd93e00700 1 --2- 172.21.15.28:0/3388621030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd94108b90 0x7efd9419fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.939+0000 7efd8aa00700 1 -- 172.21.15.28:0/3388621030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd94108b90 msgr2=0x7efd9419fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.939+0000 7efd8aa00700 1 --2- 172.21.15.28:0/3388621030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd94108b90 0x7efd9419fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.939+0000 7efd8aa00700 1 -- 172.21.15.28:0/3388621030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd94102b40 msgr2=0x7efd9419b550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.939+0000 7efd8aa00700 1 --2- 172.21.15.28:0/3388621030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd94102b40 0x7efd9419b550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:08.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.939+0000 7efd8aa00700 1 -- 172.21.15.28:0/3388621030 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd941a0560 con 0x7efd94103500 2024-04-24T08:08:08.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.939+0000 7efd8aa00700 1 --2- 172.21.15.28:0/3388621030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd94103500 0x7efd9419ba90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efd7c006010 tx=0x7efd7c00bab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:08.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.940+0000 7efd90c00700 1 -- 172.21.15.28:0/3388621030 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c027070 con 0x7efd94103500 2024-04-24T08:08:08.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.940+0000 7efd90c00700 1 -- 172.21.15.28:0/3388621030 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd7c0056d0 con 0x7efd94103500 2024-04-24T08:08:08.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.940+0000 7efd9a800700 1 -- 172.21.15.28:0/3388621030 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd941a07f0 con 0x7efd94103500 2024-04-24T08:08:08.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.940+0000 7efd90c00700 1 -- 172.21.15.28:0/3388621030 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd7c022d90 con 0x7efd94103500 2024-04-24T08:08:08.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.940+0000 7efd9a800700 1 -- 172.21.15.28:0/3388621030 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd941a80d0 con 0x7efd94103500 2024-04-24T08:08:08.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.941+0000 7efd9a800700 1 -- 172.21.15.28:0/3388621030 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd940697e0 con 0x7efd94103500 2024-04-24T08:08:08.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.943+0000 7efd90c00700 1 -- 172.21.15.28:0/3388621030 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7efd7c02a070 con 0x7efd94103500 2024-04-24T08:08:08.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.944+0000 7efd90c00700 1 --2- 172.21.15.28:0/3388621030 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd780a3cd0 0x7efd780a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:08.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.945+0000 7efd90c00700 1 -- 172.21.15.28:0/3388621030 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7efd7c0d13a0 con 0x7efd94103500 2024-04-24T08:08:08.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.945+0000 7efd93400700 1 --2- 172.21.15.28:0/3388621030 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd780a3cd0 0x7efd780a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:08.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.945+0000 7efd93400700 1 --2- 172.21.15.28:0/3388621030 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd780a3cd0 0x7efd780a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efd84009c10 tx=0x7efd8400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:08.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:08.949+0000 7efd90c00700 1 -- 172.21.15.28:0/3388621030 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efd7c05eff0 con 0x7efd94103500 2024-04-24T08:08:09.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:09.260+0000 7efd9a800700 1 -- 172.21.15.28:0/3388621030 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7efd941a82a0 con 0x7efd94103500 2024-04-24T08:08:09.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:09.266+0000 7efd90c00700 1 -- 172.21.15.28:0/3388621030 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+264994 (secure 0 0 0) 0x7efd7c013070 con 0x7efd94103500 2024-04-24T08:08:09.269 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:08:09.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:09.272+0000 7efd9a800700 1 -- 172.21.15.28:0/3388621030 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd780a3cd0 msgr2=0x7efd780a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:09.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:09.272+0000 7efd9a800700 1 --2- 172.21.15.28:0/3388621030 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd780a3cd0 0x7efd780a6190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7efd84009c10 tx=0x7efd8400b040 comp rx=0 tx=0).stop 2024-04-24T08:08:09.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:09.272+0000 7efd9a800700 1 -- 172.21.15.28:0/3388621030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd94103500 msgr2=0x7efd9419ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:09.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:09.272+0000 7efd9a800700 1 --2- 172.21.15.28:0/3388621030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd94103500 0x7efd9419ba90 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7efd7c006010 tx=0x7efd7c00bab0 comp rx=0 tx=0).stop 2024-04-24T08:08:09.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:09.273+0000 7efd9a800700 1 -- 172.21.15.28:0/3388621030 shutdown_connections 2024-04-24T08:08:09.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:09.273+0000 7efd9a800700 1 --2- 172.21.15.28:0/3388621030 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd780a3cd0 0x7efd780a6190 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:09.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:09.273+0000 7efd9a800700 1 --2- 172.21.15.28:0/3388621030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd94108b90 0x7efd9419fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:09.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:09.273+0000 7efd9a800700 1 --2- 172.21.15.28:0/3388621030 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd94102b40 0x7efd9419b550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:09.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:09.273+0000 7efd9a800700 1 --2- 172.21.15.28:0/3388621030 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd94103500 0x7efd9419ba90 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:09.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:09.273+0000 7efd9a800700 1 -- 172.21.15.28:0/3388621030 >> 172.21.15.28:0/3388621030 conn(0x7efd94079c50 msgr2=0x7efd941015c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:09.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:09.273+0000 7efd9a800700 1 -- 172.21.15.28:0/3388621030 shutdown_connections 2024-04-24T08:08:09.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:09.273+0000 7efd9a800700 1 -- 172.21.15.28:0/3388621030 wait complete. 2024-04-24T08:08:09.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:09 smithi028 ceph-mon[27786]: pgmap v325: 97 pgs: 97 active+clean; 630 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.1 KiB/s wr, 15 op/s 2024-04-24T08:08:09.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:09 smithi117 ceph-mon[32438]: pgmap v325: 97 pgs: 97 active+clean; 630 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.1 KiB/s wr, 15 op/s 2024-04-24T08:08:09.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:09 smithi194 ceph-mon[32836]: pgmap v325: 97 pgs: 97 active+clean; 630 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.2 KiB/s rd, 7.1 KiB/s wr, 15 op/s 2024-04-24T08:08:10.076 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6800","nonce":361125225},{"type":"v1","addr":"172.21.15.28:6801","nonce":361125225}]},"active_addr":"172.21.15.28:6801/361125225","active_change":"2024-04-24T07:59:49.915993+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MANAGED_BY_CLUSTERS":{"name":"MANAGED_BY_CLUSTERS","type":"str","level":"advanced","flags":0,"default_value":"[]","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.28:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":3758227512}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":2660639233}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":1819419617}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":874707228}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.28:0","nonce":2117396069}]}]} 2024-04-24T08:08:10.084 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-04-24T08:08:10.084 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-04-24T08:08:10.084 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-24T08:08:10.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:10 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3388621030' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T08:08:10.564 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:10.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:10 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3388621030' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T08:08:10.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:10 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3388621030' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-04-24T08:08:11.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:11 smithi028 ceph-mon[27786]: pgmap v326: 97 pgs: 97 active+clean; 630 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.8 KiB/s wr, 13 op/s 2024-04-24T08:08:11.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:11 smithi117 ceph-mon[32438]: pgmap v326: 97 pgs: 97 active+clean; 630 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.8 KiB/s wr, 13 op/s 2024-04-24T08:08:11.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:11 smithi194 ceph-mon[32836]: pgmap v326: 97 pgs: 97 active+clean; 630 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 6.8 KiB/s wr, 13 op/s 2024-04-24T08:08:12.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.918+0000 7f141bc00700 1 -- 172.21.15.28:0/2075996436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f141410e460 msgr2=0x7f1414116030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:12.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.918+0000 7f141bc00700 1 --2- 172.21.15.28:0/2075996436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f141410e460 0x7f1414116030 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f140c00b0d0 tx=0x7f140c00b3e0 comp rx=0 tx=0).stop 2024-04-24T08:08:12.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.918+0000 7f141bc00700 1 -- 172.21.15.28:0/2075996436 shutdown_connections 2024-04-24T08:08:12.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.918+0000 7f141bc00700 1 --2- 172.21.15.28:0/2075996436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f141410e460 0x7f1414116030 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:12.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.918+0000 7f141bc00700 1 --2- 172.21.15.28:0/2075996436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1414109380 0x7f1414109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:12.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.918+0000 7f141bc00700 1 --2- 172.21.15.28:0/2075996436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1414109d30 0x7f141410dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:12.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.918+0000 7f141bc00700 1 -- 172.21.15.28:0/2075996436 >> 172.21.15.28:0/2075996436 conn(0x7f14140762c0 msgr2=0x7f1414107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:12.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.918+0000 7f141bc00700 1 -- 172.21.15.28:0/2075996436 shutdown_connections 2024-04-24T08:08:12.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.918+0000 7f141bc00700 1 -- 172.21.15.28:0/2075996436 wait complete. 2024-04-24T08:08:12.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.919+0000 7f141bc00700 1 Processor -- start 2024-04-24T08:08:12.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.919+0000 7f141bc00700 1 -- start start 2024-04-24T08:08:12.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.920+0000 7f141bc00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1414109380 0x7f14141a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.920+0000 7f141bc00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1414109d30 0x7f14141a5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.920+0000 7f141bc00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f141410e460 0x7f141419f680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.920+0000 7f141bc00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f14141194a0 con 0x7f1414109380 2024-04-24T08:08:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.920+0000 7f141bc00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1414119320 con 0x7f1414109d30 2024-04-24T08:08:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.920+0000 7f141bc00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1414119620 con 0x7f141410e460 2024-04-24T08:08:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.920+0000 7f1419200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1414109380 0x7f14141a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.920+0000 7f1419200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1414109380 0x7f14141a55b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45670/0 (socket says 172.21.15.28:45670) 2024-04-24T08:08:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.920+0000 7f1419200700 1 -- 172.21.15.28:0/2836925041 learned_addr learned my addr 172.21.15.28:0/2836925041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.920+0000 7f1413e00700 1 --2- 172.21.15.28:0/2836925041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1414109d30 0x7f14141a5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.920+0000 7f1419c00700 1 --2- 172.21.15.28:0/2836925041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f141410e460 0x7f141419f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.921+0000 7f1419200700 1 -- 172.21.15.28:0/2836925041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f141410e460 msgr2=0x7f141419f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:12.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.921+0000 7f1419200700 1 --2- 172.21.15.28:0/2836925041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f141410e460 0x7f141419f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:12.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.921+0000 7f1419200700 1 -- 172.21.15.28:0/2836925041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1414109d30 msgr2=0x7f14141a5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:12.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.921+0000 7f1419200700 1 --2- 172.21.15.28:0/2836925041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1414109d30 0x7f14141a5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:12.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.921+0000 7f1419200700 1 -- 172.21.15.28:0/2836925041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141419ff60 con 0x7f1414109380 2024-04-24T08:08:12.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.921+0000 7f1419200700 1 --2- 172.21.15.28:0/2836925041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1414109380 0x7f14141a55b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f140800e9a0 tx=0x7f140800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:12.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.921+0000 7f1411600700 1 -- 172.21.15.28:0/2836925041 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f140800ca10 con 0x7f1414109380 2024-04-24T08:08:12.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.921+0000 7f1411600700 1 -- 172.21.15.28:0/2836925041 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f140800cbb0 con 0x7f1414109380 2024-04-24T08:08:12.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.921+0000 7f1411600700 1 -- 172.21.15.28:0/2836925041 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1408018680 con 0x7f1414109380 2024-04-24T08:08:12.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.921+0000 7f141bc00700 1 -- 172.21.15.28:0/2836925041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f14141a0250 con 0x7f1414109380 2024-04-24T08:08:12.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.922+0000 7f141bc00700 1 -- 172.21.15.28:0/2836925041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14141ac530 con 0x7f1414109380 2024-04-24T08:08:12.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.923+0000 7f141bc00700 1 -- 172.21.15.28:0/2836925041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f141410a860 con 0x7f1414109380 2024-04-24T08:08:12.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.925+0000 7f1411600700 1 -- 172.21.15.28:0/2836925041 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f1408010430 con 0x7f1414109380 2024-04-24T08:08:12.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.926+0000 7f1411600700 1 --2- 172.21.15.28:0/2836925041 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13fc0a3cd0 0x7f13fc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:12.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.926+0000 7f1411600700 1 -- 172.21.15.28:0/2836925041 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f14080d06e0 con 0x7f1414109380 2024-04-24T08:08:12.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.926+0000 7f1413e00700 1 --2- 172.21.15.28:0/2836925041 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13fc0a3cd0 0x7f13fc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:12.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.927+0000 7f1413e00700 1 --2- 172.21.15.28:0/2836925041 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13fc0a3cd0 0x7f13fc0a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f14141a0e40 tx=0x7f1400009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:12.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:12.931+0000 7f1411600700 1 -- 172.21.15.28:0/2836925041 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f140809d010 con 0x7f1414109380 2024-04-24T08:08:13.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:13.163+0000 7f141bc00700 1 -- 172.21.15.28:0/2836925041 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f14141ac700 con 0x7f1414109380 2024-04-24T08:08:13.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:13.165+0000 7f1411600700 1 -- 172.21.15.28:0/2836925041 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22207 (secure 0 0 0) 0x7f140809c910 con 0x7f1414109380 2024-04-24T08:08:13.167 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:08:13.168 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":86,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:07:53.034032+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:13.168+0000 7f141bc00700 1 -- 172.21.15.28:0/2836925041 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13fc0a3cd0 msgr2=0x7f13fc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:13.168+0000 7f141bc00700 1 --2- 172.21.15.28:0/2836925041 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13fc0a3cd0 0x7f13fc0a6190 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f14141a0e40 tx=0x7f1400009260 comp rx=0 tx=0).stop 2024-04-24T08:08:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:13.168+0000 7f141bc00700 1 -- 172.21.15.28:0/2836925041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1414109380 msgr2=0x7f14141a55b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:13.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:13.168+0000 7f141bc00700 1 --2- 172.21.15.28:0/2836925041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1414109380 0x7f14141a55b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f140800e9a0 tx=0x7f140800ecb0 comp rx=0 tx=0).stop 2024-04-24T08:08:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:13.169+0000 7f141bc00700 1 -- 172.21.15.28:0/2836925041 shutdown_connections 2024-04-24T08:08:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:13.169+0000 7f141bc00700 1 --2- 172.21.15.28:0/2836925041 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13fc0a3cd0 0x7f13fc0a6190 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:13.169+0000 7f141bc00700 1 --2- 172.21.15.28:0/2836925041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f141410e460 0x7f141419f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:13.169+0000 7f141bc00700 1 --2- 172.21.15.28:0/2836925041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1414109380 0x7f14141a55b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:13.169+0000 7f141bc00700 1 --2- 172.21.15.28:0/2836925041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1414109d30 0x7f14141a5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:13.169+0000 7f141bc00700 1 -- 172.21.15.28:0/2836925041 >> 172.21.15.28:0/2836925041 conn(0x7f14140762c0 msgr2=0x7f14140767e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:13.169+0000 7f141bc00700 1 -- 172.21.15.28:0/2836925041 shutdown_connections 2024-04-24T08:08:13.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:13.169+0000 7f141bc00700 1 -- 172.21.15.28:0/2836925041 wait complete. 2024-04-24T08:08:13.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:13 smithi194 ceph-mon[32836]: pgmap v327: 97 pgs: 97 active+clean; 630 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 5.6 KiB/s wr, 9 op/s 2024-04-24T08:08:13.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:13 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2836925041' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:08:13.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:13 smithi028 ceph-mon[27786]: pgmap v327: 97 pgs: 97 active+clean; 630 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 5.6 KiB/s wr, 9 op/s 2024-04-24T08:08:13.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:13 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2836925041' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:08:13.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:13 smithi117 ceph-mon[32438]: pgmap v327: 97 pgs: 97 active+clean; 630 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 5.6 KiB/s wr, 9 op/s 2024-04-24T08:08:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:13 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2836925041' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:08:14.006 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-04-24T08:08:14.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd dump --format=json 2024-04-24T08:08:14.476 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:15.519 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:15 smithi028 ceph-mon[27786]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.3 KiB/s wr, 10 op/s 2024-04-24T08:08:15.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:15 smithi117 ceph-mon[32438]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.3 KiB/s wr, 10 op/s 2024-04-24T08:08:15.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:15 smithi194 ceph-mon[32836]: pgmap v328: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 8.4 KiB/s rd, 5.3 KiB/s wr, 10 op/s 2024-04-24T08:08:16.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.743+0000 7f3c7e200700 1 -- 172.21.15.28:0/3680540340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c78104f70 msgr2=0x7f3c78105350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.743+0000 7f3c7e200700 1 --2- 172.21.15.28:0/3680540340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c78104f70 0x7f3c78105350 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f3c680098f0 tx=0x7f3c68009c00 comp rx=0 tx=0).stop 2024-04-24T08:08:16.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.743+0000 7f3c7e200700 1 -- 172.21.15.28:0/3680540340 shutdown_connections 2024-04-24T08:08:16.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.743+0000 7f3c7e200700 1 --2- 172.21.15.28:0/3680540340 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c7810a050 0x7f3c78111c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.743+0000 7f3c7e200700 1 --2- 172.21.15.28:0/3680540340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c78104f70 0x7f3c78105350 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.743+0000 7f3c7e200700 1 --2- 172.21.15.28:0/3680540340 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c78105920 0x7f3c78109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.743+0000 7f3c7e200700 1 -- 172.21.15.28:0/3680540340 >> 172.21.15.28:0/3680540340 conn(0x7f3c78100800 msgr2=0x7f3c78102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:16.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.743+0000 7f3c7e200700 1 -- 172.21.15.28:0/3680540340 shutdown_connections 2024-04-24T08:08:16.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.743+0000 7f3c7e200700 1 -- 172.21.15.28:0/3680540340 wait complete. 2024-04-24T08:08:16.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.744+0000 7f3c7e200700 1 Processor -- start 2024-04-24T08:08:16.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.744+0000 7f3c7e200700 1 -- start start 2024-04-24T08:08:16.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.745+0000 7f3c7e200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c78104f70 0x7f3c7819b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.745+0000 7f3c7e200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c78105920 0x7f3c7819ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.745+0000 7f3c7e200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c7810a050 0x7f3c7819fe00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.745+0000 7f3c7e200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3c78115080 con 0x7f3c78104f70 2024-04-24T08:08:16.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.745+0000 7f3c7e200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3c78114f00 con 0x7f3c78105920 2024-04-24T08:08:16.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.745+0000 7f3c7e200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3c78115200 con 0x7f3c7810a050 2024-04-24T08:08:16.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.745+0000 7f3c77400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c78104f70 0x7f3c7819b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.745+0000 7f3c77400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c78104f70 0x7f3c7819b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:45692/0 (socket says 172.21.15.28:45692) 2024-04-24T08:08:16.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.745+0000 7f3c77400700 1 -- 172.21.15.28:0/3081844880 learned_addr learned my addr 172.21.15.28:0/3081844880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:16.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.745+0000 7f3c6ea00700 1 --2- 172.21.15.28:0/3081844880 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c78105920 0x7f3c7819ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.745+0000 7f3c77e00700 1 --2- 172.21.15.28:0/3081844880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c7810a050 0x7f3c7819fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.745+0000 7f3c77400700 1 -- 172.21.15.28:0/3081844880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c7810a050 msgr2=0x7f3c7819fe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.746+0000 7f3c77400700 1 --2- 172.21.15.28:0/3081844880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c7810a050 0x7f3c7819fe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.746+0000 7f3c77400700 1 -- 172.21.15.28:0/3081844880 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c78105920 msgr2=0x7f3c7819ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.746+0000 7f3c77400700 1 --2- 172.21.15.28:0/3081844880 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c78105920 0x7f3c7819ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.746+0000 7f3c77400700 1 -- 172.21.15.28:0/3081844880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c781a0540 con 0x7f3c78104f70 2024-04-24T08:08:16.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.746+0000 7f3c6ea00700 1 --2- 172.21.15.28:0/3081844880 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c78105920 0x7f3c7819ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:16.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.746+0000 7f3c77400700 1 --2- 172.21.15.28:0/3081844880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c78104f70 0x7f3c7819b530 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3c68009fd0 tx=0x7f3c68005040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:16.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.746+0000 7f3c74c00700 1 -- 172.21.15.28:0/3081844880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c68025070 con 0x7f3c78104f70 2024-04-24T08:08:16.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.746+0000 7f3c74c00700 1 -- 172.21.15.28:0/3081844880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c680054f0 con 0x7f3c78104f70 2024-04-24T08:08:16.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.746+0000 7f3c7e200700 1 -- 172.21.15.28:0/3081844880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c781a07d0 con 0x7f3c78104f70 2024-04-24T08:08:16.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.746+0000 7f3c74c00700 1 -- 172.21.15.28:0/3081844880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c68004cc0 con 0x7f3c78104f70 2024-04-24T08:08:16.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.747+0000 7f3c7e200700 1 -- 172.21.15.28:0/3081844880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c781a80b0 con 0x7f3c78104f70 2024-04-24T08:08:16.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.748+0000 7f3c7e200700 1 -- 172.21.15.28:0/3081844880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c78106450 con 0x7f3c78104f70 2024-04-24T08:08:16.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.750+0000 7f3c74c00700 1 -- 172.21.15.28:0/3081844880 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3c68028070 con 0x7f3c78104f70 2024-04-24T08:08:16.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.751+0000 7f3c74c00700 1 --2- 172.21.15.28:0/3081844880 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3c5c0a3cd0 0x7f3c5c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:16.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.751+0000 7f3c74c00700 1 -- 172.21.15.28:0/3081844880 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f3c680d0c60 con 0x7f3c78104f70 2024-04-24T08:08:16.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.752+0000 7f3c6ea00700 1 --2- 172.21.15.28:0/3081844880 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3c5c0a3cd0 0x7f3c5c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:16.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.752+0000 7f3c6ea00700 1 --2- 172.21.15.28:0/3081844880 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3c5c0a3cd0 0x7f3c5c0a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3c78101c20 tx=0x7f3c6000b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:16.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.756+0000 7f3c74c00700 1 -- 172.21.15.28:0/3081844880 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3c6805cff0 con 0x7f3c78104f70 2024-04-24T08:08:16.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.987+0000 7f3c7e200700 1 -- 172.21.15.28:0/3081844880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f3c7802dfe0 con 0x7f3c78104f70 2024-04-24T08:08:16.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.991+0000 7f3c74c00700 1 -- 172.21.15.28:0/3081844880 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) ==== 74+0+22207 (secure 0 0 0) 0x7f3c68013030 con 0x7f3c78104f70 2024-04-24T08:08:16.993 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:08:16.993 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":86,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:07:53.034032+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:08:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.994+0000 7f3c7e200700 1 -- 172.21.15.28:0/3081844880 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3c5c0a3cd0 msgr2=0x7f3c5c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.994+0000 7f3c7e200700 1 --2- 172.21.15.28:0/3081844880 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3c5c0a3cd0 0x7f3c5c0a6190 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f3c78101c20 tx=0x7f3c6000b040 comp rx=0 tx=0).stop 2024-04-24T08:08:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.994+0000 7f3c7e200700 1 -- 172.21.15.28:0/3081844880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c78104f70 msgr2=0x7f3c7819b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:16.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.995+0000 7f3c7e200700 1 --2- 172.21.15.28:0/3081844880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c78104f70 0x7f3c7819b530 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f3c68009fd0 tx=0x7f3c68005040 comp rx=0 tx=0).stop 2024-04-24T08:08:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.995+0000 7f3c7e200700 1 -- 172.21.15.28:0/3081844880 shutdown_connections 2024-04-24T08:08:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.995+0000 7f3c7e200700 1 --2- 172.21.15.28:0/3081844880 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3c5c0a3cd0 0x7f3c5c0a6190 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.995+0000 7f3c7e200700 1 --2- 172.21.15.28:0/3081844880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3c7810a050 0x7f3c7819fe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.995+0000 7f3c7e200700 1 --2- 172.21.15.28:0/3081844880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3c78104f70 0x7f3c7819b530 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.995+0000 7f3c7e200700 1 --2- 172.21.15.28:0/3081844880 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c78105920 0x7f3c7819ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.995+0000 7f3c7e200700 1 -- 172.21.15.28:0/3081844880 >> 172.21.15.28:0/3081844880 conn(0x7f3c78100800 msgr2=0x7f3c781091e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.995+0000 7f3c7e200700 1 -- 172.21.15.28:0/3081844880 shutdown_connections 2024-04-24T08:08:16.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:16.995+0000 7f3c7e200700 1 -- 172.21.15.28:0/3081844880 wait complete. 2024-04-24T08:08:17.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:17 smithi028 ceph-mon[27786]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-04-24T08:08:17.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:17 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3081844880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:08:17.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:17 smithi117 ceph-mon[32438]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-04-24T08:08:17.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:17 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3081844880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:08:17.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:17 smithi194 ceph-mon[32836]: pgmap v329: 97 pgs: 97 active+clean; 637 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 2.0 KiB/s wr, 6 op/s 2024-04-24T08:08:17.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:17 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3081844880' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:08:17.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-04-24T08:08:17.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-04-24T08:08:17.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-04-24T08:08:17.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-04-24T08:08:17.835 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-04-24T08:08:17.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-04-24T08:08:17.836 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-04-24T08:08:17.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-04-24T08:08:17.837 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-04-24T08:08:17.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-04-24T08:08:17.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-04-24T08:08:17.838 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-04-24T08:08:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:18.915 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:18.921 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:18.930 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:18.940 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:18.948 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:18.963 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:18.967 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:19.409 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:19 smithi028 ceph-mon[27786]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-24T08:08:19.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:19 smithi117 ceph-mon[32438]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-24T08:08:19.640 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:19.649 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:19.651 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:19.653 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:19.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:19 smithi194 ceph-mon[32836]: pgmap v330: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-04-24T08:08:21.375 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:21 smithi028 ceph-mon[27786]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-24T08:08:21.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:21 smithi117 ceph-mon[32438]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-24T08:08:21.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:21 smithi194 ceph-mon[32836]: pgmap v331: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-24T08:08:23.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:23 smithi028 ceph-mon[27786]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-24T08:08:23.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:23 smithi117 ceph-mon[32438]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-24T08:08:23.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:23 smithi194 ceph-mon[32836]: pgmap v332: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 938 B/s wr, 5 op/s 2024-04-24T08:08:25.501 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:25 smithi028 ceph-mon[27786]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-24T08:08:25.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:25 smithi117 ceph-mon[32438]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-24T08:08:25.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:25 smithi194 ceph-mon[32836]: pgmap v333: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 938 B/s wr, 7 op/s 2024-04-24T08:08:27.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:27 smithi028 ceph-mon[27786]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:08:27.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:27 smithi117 ceph-mon[32438]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:08:27.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:27 smithi194 ceph-mon[32836]: pgmap v334: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:08:29.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:29 smithi028 ceph-mon[27786]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-24T08:08:29.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:29 smithi117 ceph-mon[32438]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-24T08:08:29.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:29 smithi194 ceph-mon[32836]: pgmap v335: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-24T08:08:31.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:31 smithi028 ceph-mon[27786]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:08:31.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:31 smithi117 ceph-mon[32438]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:08:31.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:31 smithi194 ceph-mon[32836]: pgmap v336: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:08:33.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:33 smithi028 ceph-mon[27786]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:08:33.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.676+0000 7fe15d000700 1 -- 172.21.15.28:0/218978612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1580757b0 msgr2=0x7fe1581134e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.676+0000 7fe15d000700 1 --2- 172.21.15.28:0/218978612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1580757b0 0x7fe1581134e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fe144009990 tx=0x7fe144009ca0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.676+0000 7fe15d000700 1 -- 172.21.15.28:0/218978612 shutdown_connections 2024-04-24T08:08:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.676+0000 7fe15d000700 1 --2- 172.21.15.28:0/218978612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe158113a20 0x7fe158115e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.676+0000 7fe15d000700 1 --2- 172.21.15.28:0/218978612 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe158074e90 0x7fe158075270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.676+0000 7fe15d000700 1 --2- 172.21.15.28:0/218978612 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1580757b0 0x7fe1581134e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.676+0000 7fe15d000700 1 -- 172.21.15.28:0/218978612 >> 172.21.15.28:0/218978612 conn(0x7fe1580fe500 msgr2=0x7fe158100920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.676+0000 7fe15d000700 1 -- 172.21.15.28:0/218978612 shutdown_connections 2024-04-24T08:08:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.677+0000 7fe15d000700 1 -- 172.21.15.28:0/218978612 wait complete. 2024-04-24T08:08:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.678+0000 7fe15d000700 1 Processor -- start 2024-04-24T08:08:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.678+0000 7fe15d000700 1 -- start start 2024-04-24T08:08:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.678+0000 7fe15d000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe158074e90 0x7fe1581a5410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:33.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.678+0000 7fe15d000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1580757b0 0x7fe1581a5950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.678+0000 7fe15d000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe158113a20 0x7fe15819f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.678+0000 7fe15d000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe158119350 con 0x7fe158113a20 2024-04-24T08:08:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.678+0000 7fe15d000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe1581191d0 con 0x7fe1580757b0 2024-04-24T08:08:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.678+0000 7fe15d000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe1581194d0 con 0x7fe158074e90 2024-04-24T08:08:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.678+0000 7fe157e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe158113a20 0x7fe15819f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.678+0000 7fe156a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1580757b0 0x7fe1581a5950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.678+0000 7fe157e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe158113a20 0x7fe15819f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50540/0 (socket says 172.21.15.28:50540) 2024-04-24T08:08:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.678+0000 7fe156a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1580757b0 0x7fe1581a5950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:43314/0 (socket says 172.21.15.28:43314) 2024-04-24T08:08:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.678+0000 7fe157e00700 1 -- 172.21.15.28:0/2366648999 learned_addr learned my addr 172.21.15.28:0/2366648999 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.679+0000 7fe157400700 1 --2- 172.21.15.28:0/2366648999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe158074e90 0x7fe1581a5410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.679+0000 7fe156a00700 1 -- 172.21.15.28:0/2366648999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe158074e90 msgr2=0x7fe1581a5410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.679+0000 7fe156a00700 1 --2- 172.21.15.28:0/2366648999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe158074e90 0x7fe1581a5410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.679+0000 7fe156a00700 1 -- 172.21.15.28:0/2366648999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe158113a20 msgr2=0x7fe15819f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.679+0000 7fe156a00700 1 --2- 172.21.15.28:0/2366648999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe158113a20 0x7fe15819f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.679+0000 7fe156a00700 1 -- 172.21.15.28:0/2366648999 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe15819fd30 con 0x7fe1580757b0 2024-04-24T08:08:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.679+0000 7fe157400700 1 --2- 172.21.15.28:0/2366648999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe158074e90 0x7fe1581a5410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:08:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.679+0000 7fe157e00700 1 --2- 172.21.15.28:0/2366648999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe158113a20 0x7fe15819f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.679+0000 7fe156a00700 1 --2- 172.21.15.28:0/2366648999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1580757b0 0x7fe1581a5950 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe1440059d0 tx=0x7fe14400bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.680+0000 7fe14fe00700 1 -- 172.21.15.28:0/2366648999 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe144027070 con 0x7fe1580757b0 2024-04-24T08:08:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.680+0000 7fe15d000700 1 -- 172.21.15.28:0/2366648999 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe15819ffc0 con 0x7fe1580757b0 2024-04-24T08:08:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.680+0000 7fe14fe00700 1 -- 172.21.15.28:0/2366648999 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe144003d70 con 0x7fe1580757b0 2024-04-24T08:08:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.680+0000 7fe14fe00700 1 -- 172.21.15.28:0/2366648999 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe14400f460 con 0x7fe1580757b0 2024-04-24T08:08:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.680+0000 7fe15d000700 1 -- 172.21.15.28:0/2366648999 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1581ac2e0 con 0x7fe1580757b0 2024-04-24T08:08:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.680+0000 7fe15d000700 1 -- 172.21.15.28:0/2366648999 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fe158063280 con 0x7fe1580757b0 2024-04-24T08:08:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.683+0000 7fe14fe00700 1 -- 172.21.15.28:0/2366648999 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe14402a070 con 0x7fe1580757b0 2024-04-24T08:08:33.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.684+0000 7fe14fe00700 1 --2- 172.21.15.28:0/2366648999 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe13c0a3cd0 0x7fe13c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:33.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.685+0000 7fe14fe00700 1 -- 172.21.15.28:0/2366648999 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fe1440d1db0 con 0x7fe1580757b0 2024-04-24T08:08:33.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.685+0000 7fe157400700 1 --2- 172.21.15.28:0/2366648999 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe13c0a3cd0 0x7fe13c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:33.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.685+0000 7fe14fe00700 1 --2- 172.21.15.28:0/2366648999 >> [v2:172.21.15.194:6808/1399608260,v1:172.21.15.194:6809/1399608260] conn(0x7fe13c0ad4b0 0x7fe13c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:33.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.685+0000 7fe14fe00700 1 -- 172.21.15.28:0/2366648999 --> [v2:172.21.15.194:6808/1399608260,v1:172.21.15.194:6809/1399608260] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fe144010ea0 con 0x7fe13c0ad4b0 2024-04-24T08:08:33.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.685+0000 7fe14fe00700 1 -- 172.21.15.28:0/2366648999 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fe1440d2160 con 0x7fe1580757b0 2024-04-24T08:08:33.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.685+0000 7fe157e00700 1 --2- 172.21.15.28:0/2366648999 >> [v2:172.21.15.194:6808/1399608260,v1:172.21.15.194:6809/1399608260] conn(0x7fe13c0ad4b0 0x7fe13c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:33.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.685+0000 7fe157400700 1 --2- 172.21.15.28:0/2366648999 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe13c0a3cd0 0x7fe13c0a6190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe148009cb0 tx=0x7fe148009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:33.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.686+0000 7fe157e00700 1 --2- 172.21.15.28:0/2366648999 >> [v2:172.21.15.194:6808/1399608260,v1:172.21.15.194:6809/1399608260] conn(0x7fe13c0ad4b0 0x7fe13c0af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:33.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.687+0000 7fe14fe00700 1 -- 172.21.15.28:0/2366648999 <== osd.5 v2:172.21.15.194:6808/1399608260 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fe144010ea0 con 0x7fe13c0ad4b0 2024-04-24T08:08:33.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.705+0000 7fe15d000700 1 -- 172.21.15.28:0/2366648999 --> [v2:172.21.15.194:6808/1399608260,v1:172.21.15.194:6809/1399608260] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fe158063680 con 0x7fe13c0ad4b0 2024-04-24T08:08:33.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.706+0000 7fe14fe00700 1 -- 172.21.15.28:0/2366648999 <== osd.5 v2:172.21.15.194:6808/1399608260 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fe158063680 con 0x7fe13c0ad4b0 2024-04-24T08:08:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.707+0000 7fe15d000700 1 -- 172.21.15.28:0/2366648999 >> [v2:172.21.15.194:6808/1399608260,v1:172.21.15.194:6809/1399608260] conn(0x7fe13c0ad4b0 msgr2=0x7fe13c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.707+0000 7fe15d000700 1 --2- 172.21.15.28:0/2366648999 >> [v2:172.21.15.194:6808/1399608260,v1:172.21.15.194:6809/1399608260] conn(0x7fe13c0ad4b0 0x7fe13c0af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.707+0000 7fe15d000700 1 -- 172.21.15.28:0/2366648999 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe13c0a3cd0 msgr2=0x7fe13c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.707+0000 7fe15d000700 1 --2- 172.21.15.28:0/2366648999 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe13c0a3cd0 0x7fe13c0a6190 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fe148009cb0 tx=0x7fe148009260 comp rx=0 tx=0).stop 2024-04-24T08:08:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.707+0000 7fe15d000700 1 -- 172.21.15.28:0/2366648999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1580757b0 msgr2=0x7fe1581a5950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.707+0000 7fe15d000700 1 --2- 172.21.15.28:0/2366648999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1580757b0 0x7fe1581a5950 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fe1440059d0 tx=0x7fe14400bab0 comp rx=0 tx=0).stop 2024-04-24T08:08:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.707+0000 7fe157e00700 1 -- 172.21.15.28:0/2366648999 reap_dead start 2024-04-24T08:08:33.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.707+0000 7fe15d000700 1 -- 172.21.15.28:0/2366648999 shutdown_connections 2024-04-24T08:08:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.707+0000 7fe15d000700 1 -- 172.21.15.28:0/2366648999 >> 172.21.15.28:0/2366648999 conn(0x7fe1580fe500 msgr2=0x7fe1580ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.707+0000 7fe15d000700 1 -- 172.21.15.28:0/2366648999 shutdown_connections 2024-04-24T08:08:33.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.707+0000 7fe15d000700 1 -- 172.21.15.28:0/2366648999 wait complete. 2024-04-24T08:08:33.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:33 smithi117 ceph-mon[32438]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:08:33.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:33 smithi194 ceph-mon[32836]: pgmap v337: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:08:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.069+0000 7fa331e00700 1 -- 172.21.15.28:0/1837943345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa32c06ba30 msgr2=0x7fa32c10f9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.069+0000 7fa331e00700 1 --2- 172.21.15.28:0/1837943345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa32c06ba30 0x7fa32c10f9a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa32400b0a0 tx=0x7fa32400b3b0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.069+0000 7fa331e00700 1 -- 172.21.15.28:0/1837943345 shutdown_connections 2024-04-24T08:08:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.069+0000 7fa331e00700 1 --2- 172.21.15.28:0/1837943345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa32c06ba30 0x7fa32c10f9a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.069+0000 7fa331e00700 1 --2- 172.21.15.28:0/1837943345 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa32c108b60 0x7fa32c108f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.069+0000 7fa331e00700 1 --2- 172.21.15.28:0/1837943345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa32c06b090 0x7fa32c06b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.069+0000 7fa331e00700 1 -- 172.21.15.28:0/1837943345 >> 172.21.15.28:0/1837943345 conn(0x7fa32c078960 msgr2=0x7fa32c078d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.069+0000 7fa331e00700 1 -- 172.21.15.28:0/1837943345 shutdown_connections 2024-04-24T08:08:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.069+0000 7fa331e00700 1 -- 172.21.15.28:0/1837943345 wait complete. 2024-04-24T08:08:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.070+0000 7fa331e00700 1 Processor -- start 2024-04-24T08:08:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.070+0000 7fa331e00700 1 -- start start 2024-04-24T08:08:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.071+0000 7fa331e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa32c06b090 0x7fa32c1992f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.071+0000 7fa331e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa32c06ba30 0x7fa32c199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.071+0000 7fa331e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa32c108b60 0x7fa32c19db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.071+0000 7fa331e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c112dc0 con 0x7fa32c108b60 2024-04-24T08:08:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.071+0000 7fa331e00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c112c40 con 0x7fa32c06ba30 2024-04-24T08:08:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.071+0000 7fa331e00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa32c112f40 con 0x7fa32c06b090 2024-04-24T08:08:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.071+0000 7fa32aa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa32c06b090 0x7fa32c1992f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.071+0000 7fa32aa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa32c06b090 0x7fa32c1992f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:35350/0 (socket says 172.21.15.28:35350) 2024-04-24T08:08:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.071+0000 7fa32aa00700 1 -- 172.21.15.28:0/1710401155 learned_addr learned my addr 172.21.15.28:0/1710401155 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.071+0000 7fa32b400700 1 --2- 172.21.15.28:0/1710401155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa32c108b60 0x7fa32c19db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.071+0000 7fa32a000700 1 --2- 172.21.15.28:0/1710401155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa32c06ba30 0x7fa32c199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.072+0000 7fa32aa00700 1 -- 172.21.15.28:0/1710401155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa32c06ba30 msgr2=0x7fa32c199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.072+0000 7fa32aa00700 1 --2- 172.21.15.28:0/1710401155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa32c06ba30 0x7fa32c199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.072+0000 7fa32aa00700 1 -- 172.21.15.28:0/1710401155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa32c108b60 msgr2=0x7fa32c19db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.072+0000 7fa32aa00700 1 --2- 172.21.15.28:0/1710401155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa32c108b60 0x7fa32c19db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.072+0000 7fa32aa00700 1 -- 172.21.15.28:0/1710401155 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa32c19e2b0 con 0x7fa32c06b090 2024-04-24T08:08:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.072+0000 7fa32b400700 1 --2- 172.21.15.28:0/1710401155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa32c108b60 0x7fa32c19db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:08:34.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.072+0000 7fa32aa00700 1 --2- 172.21.15.28:0/1710401155 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa32c06b090 0x7fa32c1992f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00e9c0 tx=0x7fa31c00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:34.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.072+0000 7fa330a00700 1 -- 172.21.15.28:0/1710401155 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31c00cbd0 con 0x7fa32c06b090 2024-04-24T08:08:34.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.072+0000 7fa330a00700 1 -- 172.21.15.28:0/1710401155 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa31c00cd70 con 0x7fa32c06b090 2024-04-24T08:08:34.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.072+0000 7fa331e00700 1 -- 172.21.15.28:0/1710401155 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa32c19e5a0 con 0x7fa32c06b090 2024-04-24T08:08:34.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.072+0000 7fa330a00700 1 -- 172.21.15.28:0/1710401155 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31c010430 con 0x7fa32c06b090 2024-04-24T08:08:34.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.072+0000 7fa331e00700 1 -- 172.21.15.28:0/1710401155 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa32c1a5e20 con 0x7fa32c06b090 2024-04-24T08:08:34.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.073+0000 7fa331e00700 1 -- 172.21.15.28:0/1710401155 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa32c063200 con 0x7fa32c06b090 2024-04-24T08:08:34.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.076+0000 7fa330a00700 1 -- 172.21.15.28:0/1710401155 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa31c004750 con 0x7fa32c06b090 2024-04-24T08:08:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.077+0000 7fa330a00700 1 --2- 172.21.15.28:0/1710401155 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa3100a3cd0 0x7fa3100a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.077+0000 7fa330a00700 1 -- 172.21.15.28:0/1710401155 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fa31c0d0a40 con 0x7fa32c06b090 2024-04-24T08:08:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.077+0000 7fa32a000700 1 --2- 172.21.15.28:0/1710401155 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa3100a3cd0 0x7fa3100a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.078+0000 7fa330a00700 1 --2- 172.21.15.28:0/1710401155 >> [v2:172.21.15.117:6824/2345072654,v1:172.21.15.117:6825/2345072654] conn(0x7fa3100ad4b0 0x7fa3100af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.078+0000 7fa330a00700 1 -- 172.21.15.28:0/1710401155 --> [v2:172.21.15.117:6824/2345072654,v1:172.21.15.117:6825/2345072654] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa31c019e70 con 0x7fa3100ad4b0 2024-04-24T08:08:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.078+0000 7fa330a00700 1 -- 172.21.15.28:0/1710401155 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fa31c0d0df0 con 0x7fa32c06b090 2024-04-24T08:08:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.078+0000 7fa32a000700 1 --2- 172.21.15.28:0/1710401155 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa3100a3cd0 0x7fa3100a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa318009680 tx=0x7fa318009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.078+0000 7fa32b400700 1 --2- 172.21.15.28:0/1710401155 >> [v2:172.21.15.117:6824/2345072654,v1:172.21.15.117:6825/2345072654] conn(0x7fa3100ad4b0 0x7fa3100af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.079+0000 7fa32b400700 1 --2- 172.21.15.28:0/1710401155 >> [v2:172.21.15.117:6824/2345072654,v1:172.21.15.117:6825/2345072654] conn(0x7fa3100ad4b0 0x7fa3100af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.080+0000 7fa330a00700 1 -- 172.21.15.28:0/1710401155 <== osd.10 v2:172.21.15.117:6824/2345072654 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa31c019e70 con 0x7fa3100ad4b0 2024-04-24T08:08:34.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.098+0000 7fa331e00700 1 -- 172.21.15.28:0/1710401155 --> [v2:172.21.15.117:6824/2345072654,v1:172.21.15.117:6825/2345072654] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa32c063680 con 0x7fa3100ad4b0 2024-04-24T08:08:34.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.099+0000 7fa330a00700 1 -- 172.21.15.28:0/1710401155 <== osd.10 v2:172.21.15.117:6824/2345072654 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa32c063680 con 0x7fa3100ad4b0 2024-04-24T08:08:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.100+0000 7fa331e00700 1 -- 172.21.15.28:0/1710401155 >> [v2:172.21.15.117:6824/2345072654,v1:172.21.15.117:6825/2345072654] conn(0x7fa3100ad4b0 msgr2=0x7fa3100af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.100+0000 7fa331e00700 1 --2- 172.21.15.28:0/1710401155 >> [v2:172.21.15.117:6824/2345072654,v1:172.21.15.117:6825/2345072654] conn(0x7fa3100ad4b0 0x7fa3100af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.100+0000 7fa331e00700 1 -- 172.21.15.28:0/1710401155 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa3100a3cd0 msgr2=0x7fa3100a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.100+0000 7fa331e00700 1 --2- 172.21.15.28:0/1710401155 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa3100a3cd0 0x7fa3100a6190 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fa318009680 tx=0x7fa318009260 comp rx=0 tx=0).stop 2024-04-24T08:08:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.100+0000 7fa331e00700 1 -- 172.21.15.28:0/1710401155 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa32c06b090 msgr2=0x7fa32c1992f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.100+0000 7fa331e00700 1 --2- 172.21.15.28:0/1710401155 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa32c06b090 0x7fa32c1992f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7fa31c00e9c0 tx=0x7fa31c00ecd0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.100+0000 7fa32b400700 1 -- 172.21.15.28:0/1710401155 reap_dead start 2024-04-24T08:08:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.100+0000 7fa331e00700 1 -- 172.21.15.28:0/1710401155 shutdown_connections 2024-04-24T08:08:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.100+0000 7fa331e00700 1 -- 172.21.15.28:0/1710401155 >> 172.21.15.28:0/1710401155 conn(0x7fa32c078960 msgr2=0x7fa32c0ffad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.100+0000 7fa331e00700 1 -- 172.21.15.28:0/1710401155 shutdown_connections 2024-04-24T08:08:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.100+0000 7fa331e00700 1 -- 172.21.15.28:0/1710401155 wait complete. 2024-04-24T08:08:34.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.293+0000 7ff44d200700 1 -- 172.21.15.28:0/1861486840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff44806a800 msgr2=0x7ff44806ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.293+0000 7ff44d200700 1 --2- 172.21.15.28:0/1861486840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff44806a800 0x7ff44806ac60 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7ff434009990 tx=0x7ff434009ca0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.293+0000 7ff44d200700 1 -- 172.21.15.28:0/1861486840 shutdown_connections 2024-04-24T08:08:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.293+0000 7ff44d200700 1 --2- 172.21.15.28:0/1861486840 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff44806b1a0 0x7ff44810f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.293+0000 7ff44d200700 1 --2- 172.21.15.28:0/1861486840 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff448108b70 0x7ff448108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.293+0000 7ff44d200700 1 --2- 172.21.15.28:0/1861486840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff44806a800 0x7ff44806ac60 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.293+0000 7ff44d200700 1 -- 172.21.15.28:0/1861486840 >> 172.21.15.28:0/1861486840 conn(0x7ff4480789a0 msgr2=0x7ff448078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.293+0000 7ff44d200700 1 -- 172.21.15.28:0/1861486840 shutdown_connections 2024-04-24T08:08:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.293+0000 7ff44d200700 1 -- 172.21.15.28:0/1861486840 wait complete. 2024-04-24T08:08:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.294+0000 7ff44d200700 1 Processor -- start 2024-04-24T08:08:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.295+0000 7ff44d200700 1 -- start start 2024-04-24T08:08:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.295+0000 7ff44d200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff44806a800 0x7ff448101ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.295+0000 7ff44d200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff44806b1a0 0x7ff448102000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.295+0000 7ff44d200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff448108b70 0x7ff448102540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.295+0000 7ff44d200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff448112e20 con 0x7ff44806a800 2024-04-24T08:08:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.295+0000 7ff44d200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff448112ca0 con 0x7ff448108b70 2024-04-24T08:08:34.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.295+0000 7ff44d200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff448112fa0 con 0x7ff44806b1a0 2024-04-24T08:08:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.295+0000 7ff445e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff44806a800 0x7ff448101ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.296+0000 7ff445e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff44806a800 0x7ff448101ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50560/0 (socket says 172.21.15.28:50560) 2024-04-24T08:08:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.296+0000 7ff445e00700 1 -- 172.21.15.28:0/62932520 learned_addr learned my addr 172.21.15.28:0/62932520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.296+0000 7ff446800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff448108b70 0x7ff448102540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.296+0000 7ff445400700 1 --2- 172.21.15.28:0/62932520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff44806b1a0 0x7ff448102000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.296+0000 7ff445e00700 1 -- 172.21.15.28:0/62932520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff44806b1a0 msgr2=0x7ff448102000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.296+0000 7ff445e00700 1 --2- 172.21.15.28:0/62932520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff44806b1a0 0x7ff448102000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.296+0000 7ff445e00700 1 -- 172.21.15.28:0/62932520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff448108b70 msgr2=0x7ff448102540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.296+0000 7ff445e00700 1 --2- 172.21.15.28:0/62932520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff448108b70 0x7ff448102540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.296+0000 7ff445e00700 1 -- 172.21.15.28:0/62932520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4481abd70 con 0x7ff44806a800 2024-04-24T08:08:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.296+0000 7ff445e00700 1 --2- 172.21.15.28:0/62932520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff44806a800 0x7ff448101ac0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff43800e9c0 tx=0x7ff43800ecd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.297+0000 7ff43ea00700 1 -- 172.21.15.28:0/62932520 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff43800cbd0 con 0x7ff44806a800 2024-04-24T08:08:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.297+0000 7ff43ea00700 1 -- 172.21.15.28:0/62932520 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff43800ee60 con 0x7ff44806a800 2024-04-24T08:08:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.297+0000 7ff44d200700 1 -- 172.21.15.28:0/62932520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4481abfa0 con 0x7ff44806a800 2024-04-24T08:08:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.297+0000 7ff43ea00700 1 -- 172.21.15.28:0/62932520 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff438005630 con 0x7ff44806a800 2024-04-24T08:08:34.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.297+0000 7ff44d200700 1 -- 172.21.15.28:0/62932520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4481ac400 con 0x7ff44806a800 2024-04-24T08:08:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.298+0000 7ff44d200700 1 -- 172.21.15.28:0/62932520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff448063200 con 0x7ff44806a800 2024-04-24T08:08:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.300+0000 7ff43ea00700 1 -- 172.21.15.28:0/62932520 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7ff438004750 con 0x7ff44806a800 2024-04-24T08:08:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.301+0000 7ff43ea00700 1 --2- 172.21.15.28:0/62932520 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ff42c0a3d50 0x7ff42c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.302+0000 7ff43ea00700 1 -- 172.21.15.28:0/62932520 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7ff4380d0af0 con 0x7ff44806a800 2024-04-24T08:08:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.302+0000 7ff445400700 1 --2- 172.21.15.28:0/62932520 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ff42c0a3d50 0x7ff42c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.302+0000 7ff43ea00700 1 --2- 172.21.15.28:0/62932520 >> [v2:172.21.15.194:6816/301057688,v1:172.21.15.194:6817/301057688] conn(0x7ff42c0ad530 0x7ff42c0af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.302+0000 7ff43ea00700 1 -- 172.21.15.28:0/62932520 --> [v2:172.21.15.194:6816/301057688,v1:172.21.15.194:6817/301057688] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff42c0b0000 con 0x7ff42c0ad530 2024-04-24T08:08:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.302+0000 7ff43ea00700 1 -- 172.21.15.28:0/62932520 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7ff4380d0ea0 con 0x7ff44806a800 2024-04-24T08:08:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.302+0000 7ff446800700 1 --2- 172.21.15.28:0/62932520 >> [v2:172.21.15.194:6816/301057688,v1:172.21.15.194:6817/301057688] conn(0x7ff42c0ad530 0x7ff42c0af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.302+0000 7ff445400700 1 --2- 172.21.15.28:0/62932520 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ff42c0a3d50 0x7ff42c0a6210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff448103310 tx=0x7ff43401a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.303+0000 7ff446800700 1 --2- 172.21.15.28:0/62932520 >> [v2:172.21.15.194:6816/301057688,v1:172.21.15.194:6817/301057688] conn(0x7ff42c0ad530 0x7ff42c0af950 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:34.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.304+0000 7ff43ea00700 1 -- 172.21.15.28:0/62932520 <== osd.8 v2:172.21.15.194:6816/301057688 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff42c0b0000 con 0x7ff42c0ad530 2024-04-24T08:08:34.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.327+0000 7ff44d200700 1 -- 172.21.15.28:0/62932520 --> [v2:172.21.15.194:6816/301057688,v1:172.21.15.194:6817/301057688] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff448063600 con 0x7ff42c0ad530 2024-04-24T08:08:34.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.327+0000 7ff43ea00700 1 -- 172.21.15.28:0/62932520 <== osd.8 v2:172.21.15.194:6816/301057688 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff448063600 con 0x7ff42c0ad530 2024-04-24T08:08:34.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.328+0000 7ff44d200700 1 -- 172.21.15.28:0/62932520 >> [v2:172.21.15.194:6816/301057688,v1:172.21.15.194:6817/301057688] conn(0x7ff42c0ad530 msgr2=0x7ff42c0af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.328+0000 7ff44d200700 1 --2- 172.21.15.28:0/62932520 >> [v2:172.21.15.194:6816/301057688,v1:172.21.15.194:6817/301057688] conn(0x7ff42c0ad530 0x7ff42c0af950 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.328+0000 7ff44d200700 1 -- 172.21.15.28:0/62932520 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ff42c0a3d50 msgr2=0x7ff42c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.328+0000 7ff44d200700 1 --2- 172.21.15.28:0/62932520 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ff42c0a3d50 0x7ff42c0a6210 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7ff448103310 tx=0x7ff43401a040 comp rx=0 tx=0).stop 2024-04-24T08:08:34.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.328+0000 7ff44d200700 1 -- 172.21.15.28:0/62932520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff44806a800 msgr2=0x7ff448101ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.328+0000 7ff44d200700 1 --2- 172.21.15.28:0/62932520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff44806a800 0x7ff448101ac0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ff43800e9c0 tx=0x7ff43800ecd0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.328+0000 7ff446800700 1 -- 172.21.15.28:0/62932520 reap_dead start 2024-04-24T08:08:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.328+0000 7ff44d200700 1 -- 172.21.15.28:0/62932520 shutdown_connections 2024-04-24T08:08:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.328+0000 7ff44d200700 1 -- 172.21.15.28:0/62932520 >> 172.21.15.28:0/62932520 conn(0x7ff4480789a0 msgr2=0x7ff4480ffb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.328+0000 7ff44d200700 1 -- 172.21.15.28:0/62932520 shutdown_connections 2024-04-24T08:08:34.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:34.328+0000 7ff44d200700 1 -- 172.21.15.28:0/62932520 wait complete. 2024-04-24T08:08:34.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.376+0000 7f3322a00700 1 -- 172.21.15.28:0/1218477583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f331c06a800 msgr2=0x7f331c06abe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.376+0000 7f3322a00700 1 --2- 172.21.15.28:0/1218477583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f331c06a800 0x7f331c06abe0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f330c0098f0 tx=0x7f330c009c00 comp rx=0 tx=0).stop 2024-04-24T08:08:34.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.376+0000 7f3322a00700 1 -- 172.21.15.28:0/1218477583 shutdown_connections 2024-04-24T08:08:34.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.376+0000 7f3322a00700 1 --2- 172.21.15.28:0/1218477583 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f331c10f5a0 0x7f331c111990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.376+0000 7f3322a00700 1 --2- 172.21.15.28:0/1218477583 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f331c06a800 0x7f331c06abe0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.376+0000 7f3322a00700 1 --2- 172.21.15.28:0/1218477583 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f331c06b120 0x7f331c10f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.376+0000 7f3322a00700 1 -- 172.21.15.28:0/1218477583 >> 172.21.15.28:0/1218477583 conn(0x7f331c0789a0 msgr2=0x7f331c078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:34.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.376+0000 7f3322a00700 1 -- 172.21.15.28:0/1218477583 shutdown_connections 2024-04-24T08:08:34.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.377+0000 7f3322a00700 1 -- 172.21.15.28:0/1218477583 wait complete. 2024-04-24T08:08:34.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.378+0000 7f3322a00700 1 Processor -- start 2024-04-24T08:08:34.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.378+0000 7f3322a00700 1 -- start start 2024-04-24T08:08:34.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.378+0000 7f3322a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f331c06a800 0x7f331c19b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.378+0000 7f3322a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f331c06b120 0x7f331c19bb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.378+0000 7f3322a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f331c10f5a0 0x7f331c19fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.378+0000 7f3322a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f331c115070 con 0x7f331c10f5a0 2024-04-24T08:08:34.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.378+0000 7f3322a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f331c114ef0 con 0x7f331c06b120 2024-04-24T08:08:34.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.378+0000 7f3322a00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f331c1151f0 con 0x7f331c06a800 2024-04-24T08:08:34.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.379+0000 7f331be00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f331c06a800 0x7f331c19b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.379+0000 7f331b400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f331c06b120 0x7f331c19bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.379+0000 7f3320a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f331c10f5a0 0x7f331c19fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.379+0000 7f331be00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f331c06a800 0x7f331c19b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:35316/0 (socket says 172.21.15.28:35316) 2024-04-24T08:08:34.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.379+0000 7f331be00700 1 -- 172.21.15.28:0/2531732255 learned_addr learned my addr 172.21.15.28:0/2531732255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:34.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.379+0000 7f331b400700 1 -- 172.21.15.28:0/2531732255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f331c06a800 msgr2=0x7f331c19b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.379+0000 7f331b400700 1 --2- 172.21.15.28:0/2531732255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f331c06a800 0x7f331c19b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.379+0000 7f331b400700 1 -- 172.21.15.28:0/2531732255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f331c10f5a0 msgr2=0x7f331c19fe40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.379+0000 7f331b400700 1 --2- 172.21.15.28:0/2531732255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f331c10f5a0 0x7f331c19fe40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.379+0000 7f331b400700 1 -- 172.21.15.28:0/2531732255 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f331c1a0580 con 0x7f331c06b120 2024-04-24T08:08:34.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.379+0000 7f331be00700 1 --2- 172.21.15.28:0/2531732255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f331c06a800 0x7f331c19b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:34.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.379+0000 7f3320a00700 1 --2- 172.21.15.28:0/2531732255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f331c10f5a0 0x7f331c19fe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:34.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.379+0000 7f331b400700 1 --2- 172.21.15.28:0/2531732255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f331c06b120 0x7f331c19bb00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f330800e9c0 tx=0x7f330800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:34.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.380+0000 7f3318c00700 1 -- 172.21.15.28:0/2531732255 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f330800c9c0 con 0x7f331c06b120 2024-04-24T08:08:34.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.380+0000 7f3318c00700 1 -- 172.21.15.28:0/2531732255 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f330800cb60 con 0x7f331c06b120 2024-04-24T08:08:34.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.380+0000 7f3318c00700 1 -- 172.21.15.28:0/2531732255 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3308018680 con 0x7f331c06b120 2024-04-24T08:08:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.380+0000 7f3322a00700 1 -- 172.21.15.28:0/2531732255 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f331c1a0870 con 0x7f331c06b120 2024-04-24T08:08:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.380+0000 7f3322a00700 1 -- 172.21.15.28:0/2531732255 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f331c1a80f0 con 0x7f331c06b120 2024-04-24T08:08:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.381+0000 7f3322a00700 1 -- 172.21.15.28:0/2531732255 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f331c063240 con 0x7f331c06b120 2024-04-24T08:08:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.383+0000 7f3318c00700 1 -- 172.21.15.28:0/2531732255 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f330800cd10 con 0x7f331c06b120 2024-04-24T08:08:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.384+0000 7f3318c00700 1 --2- 172.21.15.28:0/2531732255 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f33000a3d50 0x7f33000a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.384+0000 7f3318c00700 1 -- 172.21.15.28:0/2531732255 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f33080d0b10 con 0x7f331c06b120 2024-04-24T08:08:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.385+0000 7f331be00700 1 --2- 172.21.15.28:0/2531732255 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f33000a3d50 0x7f33000a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.385+0000 7f3318c00700 1 --2- 172.21.15.28:0/2531732255 >> [v2:172.21.15.28:6802/1414246664,v1:172.21.15.28:6803/1414246664] conn(0x7f33000ad530 0x7f33000af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.385+0000 7f3318c00700 1 -- 172.21.15.28:0/2531732255 --> [v2:172.21.15.28:6802/1414246664,v1:172.21.15.28:6803/1414246664] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f330800be50 con 0x7f33000ad530 2024-04-24T08:08:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.385+0000 7f3318c00700 1 -- 172.21.15.28:0/2531732255 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f33080d0ec0 con 0x7f331c06b120 2024-04-24T08:08:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.385+0000 7f3320a00700 1 --2- 172.21.15.28:0/2531732255 >> [v2:172.21.15.28:6802/1414246664,v1:172.21.15.28:6803/1414246664] conn(0x7f33000ad530 0x7f33000af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.385+0000 7f331be00700 1 --2- 172.21.15.28:0/2531732255 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f33000a3d50 0x7f33000a6210 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f330c00c010 tx=0x7f330c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:34.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.386+0000 7f3320a00700 1 --2- 172.21.15.28:0/2531732255 >> [v2:172.21.15.28:6802/1414246664,v1:172.21.15.28:6803/1414246664] conn(0x7f33000ad530 0x7f33000af950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:34.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.386+0000 7f3318c00700 1 -- 172.21.15.28:0/2531732255 <== osd.0 v2:172.21.15.28:6802/1414246664 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f330800be50 con 0x7f33000ad530 2024-04-24T08:08:34.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.406+0000 7f3322a00700 1 -- 172.21.15.28:0/2531732255 --> [v2:172.21.15.28:6802/1414246664,v1:172.21.15.28:6803/1414246664] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f331c063670 con 0x7f33000ad530 2024-04-24T08:08:34.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.407+0000 7f3318c00700 1 -- 172.21.15.28:0/2531732255 <== osd.0 v2:172.21.15.28:6802/1414246664 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f331c063670 con 0x7f33000ad530 2024-04-24T08:08:34.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.407+0000 7f3322a00700 1 -- 172.21.15.28:0/2531732255 >> [v2:172.21.15.28:6802/1414246664,v1:172.21.15.28:6803/1414246664] conn(0x7f33000ad530 msgr2=0x7f33000af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.407+0000 7f3322a00700 1 --2- 172.21.15.28:0/2531732255 >> [v2:172.21.15.28:6802/1414246664,v1:172.21.15.28:6803/1414246664] conn(0x7f33000ad530 0x7f33000af950 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.408+0000 7f3322a00700 1 -- 172.21.15.28:0/2531732255 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f33000a3d50 msgr2=0x7f33000a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.408+0000 7f3322a00700 1 --2- 172.21.15.28:0/2531732255 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f33000a3d50 0x7f33000a6210 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f330c00c010 tx=0x7f330c019040 comp rx=0 tx=0).stop 2024-04-24T08:08:34.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.408+0000 7f3322a00700 1 -- 172.21.15.28:0/2531732255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f331c06b120 msgr2=0x7f331c19bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:34.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.408+0000 7f3322a00700 1 --2- 172.21.15.28:0/2531732255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f331c06b120 0x7f331c19bb00 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f330800e9c0 tx=0x7f330800ecd0 comp rx=0 tx=0).stop 2024-04-24T08:08:34.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.408+0000 7f3320a00700 1 -- 172.21.15.28:0/2531732255 reap_dead start 2024-04-24T08:08:34.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.408+0000 7f3322a00700 1 -- 172.21.15.28:0/2531732255 shutdown_connections 2024-04-24T08:08:34.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.408+0000 7f3322a00700 1 -- 172.21.15.28:0/2531732255 >> 172.21.15.28:0/2531732255 conn(0x7f331c0789a0 msgr2=0x7f331c110a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:34.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.408+0000 7f3322a00700 1 -- 172.21.15.28:0/2531732255 shutdown_connections 2024-04-24T08:08:34.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:32.408+0000 7f3322a00700 1 -- 172.21.15.28:0/2531732255 wait complete. 2024-04-24T08:08:35.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:35 smithi028 ceph-mon[27786]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:08:35.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:35 smithi117 ceph-mon[32438]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:08:35.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:35 smithi194 ceph-mon[32836]: pgmap v338: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:08:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.863+0000 7f5e5fc00700 1 -- 172.21.15.28:0/3600648661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e58113cc0 msgr2=0x7f5e581160b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.863+0000 7f5e5fc00700 1 --2- 172.21.15.28:0/3600648661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e58113cc0 0x7f5e581160b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5e5000b0a0 tx=0x7f5e5000b3b0 comp rx=0 tx=0).stop 2024-04-24T08:08:37.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.863+0000 7f5e5fc00700 1 -- 172.21.15.28:0/3600648661 shutdown_connections 2024-04-24T08:08:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.863+0000 7f5e5fc00700 1 --2- 172.21.15.28:0/3600648661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e58113cc0 0x7f5e581160b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.863+0000 7f5e5fc00700 1 --2- 172.21.15.28:0/3600648661 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e58074d20 0x7f5e58075100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.863+0000 7f5e5fc00700 1 --2- 172.21.15.28:0/3600648661 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58075640 0x7f5e58113780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.863+0000 7f5e5fc00700 1 -- 172.21.15.28:0/3600648661 >> 172.21.15.28:0/3600648661 conn(0x7f5e58079c50 msgr2=0x7f5e5807a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.863+0000 7f5e5fc00700 1 -- 172.21.15.28:0/3600648661 shutdown_connections 2024-04-24T08:08:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.863+0000 7f5e5fc00700 1 -- 172.21.15.28:0/3600648661 wait complete. 2024-04-24T08:08:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.864+0000 7f5e5fc00700 1 Processor -- start 2024-04-24T08:08:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.864+0000 7f5e5fc00700 1 -- start start 2024-04-24T08:08:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.865+0000 7f5e5fc00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e58074d20 0x7f5e581a5610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.865+0000 7f5e5fc00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e58075640 0x7f5e581a5b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.865+0000 7f5e5fc00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58113cc0 0x7f5e5819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:37.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.865+0000 7f5e5fc00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e58119440 con 0x7f5e58113cc0 2024-04-24T08:08:37.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.865+0000 7f5e5fc00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5e581192c0 con 0x7f5e58074d20 2024-04-24T08:08:37.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.865+0000 7f5e5fc00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5e581195c0 con 0x7f5e58075640 2024-04-24T08:08:37.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.865+0000 7f5e5d200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e58074d20 0x7f5e581a5610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:37.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.865+0000 7f5e5d200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e58074d20 0x7f5e581a5610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:43408/0 (socket says 172.21.15.28:43408) 2024-04-24T08:08:37.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.865+0000 7f5e5dc00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58113cc0 0x7f5e5819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:37.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.865+0000 7f5e5d200700 1 -- 172.21.15.28:0/203601491 learned_addr learned my addr 172.21.15.28:0/203601491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:37.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.865+0000 7f5e4fe00700 1 --2- 172.21.15.28:0/203601491 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e58075640 0x7f5e581a5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:37.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.866+0000 7f5e4fe00700 1 -- 172.21.15.28:0/203601491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e58074d20 msgr2=0x7f5e581a5610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:37.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.866+0000 7f5e4fe00700 1 --2- 172.21.15.28:0/203601491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e58074d20 0x7f5e581a5610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:37.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.866+0000 7f5e4fe00700 1 -- 172.21.15.28:0/203601491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58113cc0 msgr2=0x7f5e5819f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:37.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.866+0000 7f5e4fe00700 1 --2- 172.21.15.28:0/203601491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58113cc0 0x7f5e5819f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:37.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.866+0000 7f5e4fe00700 1 -- 172.21.15.28:0/203601491 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e5819ffc0 con 0x7f5e58075640 2024-04-24T08:08:37.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.866+0000 7f5e5d200700 1 --2- 172.21.15.28:0/203601491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e58074d20 0x7f5e581a5610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:37.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.866+0000 7f5e5dc00700 1 --2- 172.21.15.28:0/203601491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58113cc0 0x7f5e5819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:37.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.866+0000 7f5e4fe00700 1 --2- 172.21.15.28:0/203601491 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e58075640 0x7f5e581a5b50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5e4000e9f0 tx=0x7f5e4000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.866+0000 7f5e56000700 1 -- 172.21.15.28:0/203601491 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4000c9c0 con 0x7f5e58075640 2024-04-24T08:08:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.866+0000 7f5e5fc00700 1 -- 172.21.15.28:0/203601491 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e581a02b0 con 0x7f5e58075640 2024-04-24T08:08:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.866+0000 7f5e56000700 1 -- 172.21.15.28:0/203601491 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e4000cb60 con 0x7f5e58075640 2024-04-24T08:08:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.866+0000 7f5e56000700 1 -- 172.21.15.28:0/203601491 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e40018680 con 0x7f5e58075640 2024-04-24T08:08:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.866+0000 7f5e5fc00700 1 -- 172.21.15.28:0/203601491 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e581ac590 con 0x7f5e58075640 2024-04-24T08:08:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.867+0000 7f5e5fc00700 1 -- 172.21.15.28:0/203601491 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f5e581134a0 con 0x7f5e58075640 2024-04-24T08:08:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.870+0000 7f5e56000700 1 -- 172.21.15.28:0/203601491 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f5e40010430 con 0x7f5e58075640 2024-04-24T08:08:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.871+0000 7f5e56000700 1 --2- 172.21.15.28:0/203601491 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5e3c0a3cd0 0x7f5e3c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.871+0000 7f5e56000700 1 -- 172.21.15.28:0/203601491 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f5e400d09d0 con 0x7f5e58075640 2024-04-24T08:08:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.871+0000 7f5e5d200700 1 --2- 172.21.15.28:0/203601491 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5e3c0a3cd0 0x7f5e3c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.872+0000 7f5e56000700 1 --2- 172.21.15.28:0/203601491 >> [v2:172.21.15.194:6800/65787773,v1:172.21.15.194:6801/65787773] conn(0x7f5e3c0ad4b0 0x7f5e3c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.872+0000 7f5e56000700 1 -- 172.21.15.28:0/203601491 --> [v2:172.21.15.194:6800/65787773,v1:172.21.15.194:6801/65787773] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f5e40019ec0 con 0x7f5e3c0ad4b0 2024-04-24T08:08:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.872+0000 7f5e56000700 1 -- 172.21.15.28:0/203601491 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f5e400d0d80 con 0x7f5e58075640 2024-04-24T08:08:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.872+0000 7f5e5d200700 1 --2- 172.21.15.28:0/203601491 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5e3c0a3cd0 0x7f5e3c0a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5e48011590 tx=0x7f5e48009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:37.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.872+0000 7f5e5dc00700 1 --2- 172.21.15.28:0/203601491 >> [v2:172.21.15.194:6800/65787773,v1:172.21.15.194:6801/65787773] conn(0x7f5e3c0ad4b0 0x7f5e3c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.873+0000 7f5e5dc00700 1 --2- 172.21.15.28:0/203601491 >> [v2:172.21.15.194:6800/65787773,v1:172.21.15.194:6801/65787773] conn(0x7f5e3c0ad4b0 0x7f5e3c0af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.874+0000 7f5e56000700 1 -- 172.21.15.28:0/203601491 <== osd.2 v2:172.21.15.194:6800/65787773 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f5e40019ec0 con 0x7f5e3c0ad4b0 2024-04-24T08:08:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.892+0000 7f5e5fc00700 1 -- 172.21.15.28:0/203601491 --> [v2:172.21.15.194:6800/65787773,v1:172.21.15.194:6801/65787773] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f5e58076410 con 0x7f5e3c0ad4b0 2024-04-24T08:08:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.893+0000 7f5e56000700 1 -- 172.21.15.28:0/203601491 <== osd.2 v2:172.21.15.194:6800/65787773 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f5e58076410 con 0x7f5e3c0ad4b0 2024-04-24T08:08:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.894+0000 7f5e5fc00700 1 -- 172.21.15.28:0/203601491 >> [v2:172.21.15.194:6800/65787773,v1:172.21.15.194:6801/65787773] conn(0x7f5e3c0ad4b0 msgr2=0x7f5e3c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.894+0000 7f5e5fc00700 1 --2- 172.21.15.28:0/203601491 >> [v2:172.21.15.194:6800/65787773,v1:172.21.15.194:6801/65787773] conn(0x7f5e3c0ad4b0 0x7f5e3c0af8d0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.894+0000 7f5e5fc00700 1 -- 172.21.15.28:0/203601491 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5e3c0a3cd0 msgr2=0x7f5e3c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.894+0000 7f5e5fc00700 1 --2- 172.21.15.28:0/203601491 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5e3c0a3cd0 0x7f5e3c0a6190 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f5e48011590 tx=0x7f5e48009260 comp rx=0 tx=0).stop 2024-04-24T08:08:37.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.894+0000 7f5e5fc00700 1 -- 172.21.15.28:0/203601491 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e58075640 msgr2=0x7f5e581a5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:37.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.894+0000 7f5e5fc00700 1 --2- 172.21.15.28:0/203601491 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e58075640 0x7f5e581a5b50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f5e4000e9f0 tx=0x7f5e4000ed00 comp rx=0 tx=0).stop 2024-04-24T08:08:37.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.894+0000 7f5e5dc00700 1 -- 172.21.15.28:0/203601491 reap_dead start 2024-04-24T08:08:37.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.894+0000 7f5e5fc00700 1 -- 172.21.15.28:0/203601491 shutdown_connections 2024-04-24T08:08:37.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.894+0000 7f5e5fc00700 1 -- 172.21.15.28:0/203601491 >> 172.21.15.28:0/203601491 conn(0x7f5e58079c50 msgr2=0x7f5e58112a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:37.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.894+0000 7f5e5fc00700 1 -- 172.21.15.28:0/203601491 shutdown_connections 2024-04-24T08:08:37.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.894+0000 7f5e5fc00700 1 -- 172.21.15.28:0/203601491 wait complete. 2024-04-24T08:08:37.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:37 smithi028 ceph-mon[27786]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:08:37.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:37 smithi117 ceph-mon[32438]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:08:37.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:37 smithi194 ceph-mon[32836]: pgmap v339: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:08:38.254 INFO:teuthology.orchestra.run.smithi028.stdout:163208757305 2024-04-24T08:08:38.254 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-04-24T08:08:38.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.560+0000 7f4a95400700 1 -- 172.21.15.28:0/203496493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a90108b70 msgr2=0x7f4a90108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:38.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.560+0000 7f4a95400700 1 --2- 172.21.15.28:0/203496493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a90108b70 0x7f4a90108f50 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f4a80009990 tx=0x7f4a80009ca0 comp rx=0 tx=0).stop 2024-04-24T08:08:38.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.560+0000 7f4a95400700 1 -- 172.21.15.28:0/203496493 shutdown_connections 2024-04-24T08:08:38.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.560+0000 7f4a95400700 1 --2- 172.21.15.28:0/203496493 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a9006b1a0 0x7f4a9010f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:38.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.560+0000 7f4a95400700 1 --2- 172.21.15.28:0/203496493 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a90108b70 0x7f4a90108f50 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:38.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.560+0000 7f4a95400700 1 --2- 172.21.15.28:0/203496493 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a9006a800 0x7f4a9006ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:38.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.560+0000 7f4a95400700 1 -- 172.21.15.28:0/203496493 >> 172.21.15.28:0/203496493 conn(0x7f4a900789a0 msgr2=0x7f4a90078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:38.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.560+0000 7f4a95400700 1 -- 172.21.15.28:0/203496493 shutdown_connections 2024-04-24T08:08:38.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.560+0000 7f4a95400700 1 -- 172.21.15.28:0/203496493 wait complete. 2024-04-24T08:08:38.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.561+0000 7f4a95400700 1 Processor -- start 2024-04-24T08:08:38.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.561+0000 7f4a95400700 1 -- start start 2024-04-24T08:08:38.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.562+0000 7f4a95400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a9006a800 0x7f4a9019b590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:38.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.562+0000 7f4a95400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9006b1a0 0x7f4a9019bad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:38.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.562+0000 7f4a95400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a90108b70 0x7f4a9019fe10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:38.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.562+0000 7f4a95400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4a90112e00 con 0x7f4a9006b1a0 2024-04-24T08:08:38.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.562+0000 7f4a95400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4a90112c80 con 0x7f4a90108b70 2024-04-24T08:08:38.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.562+0000 7f4a95400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4a90112f80 con 0x7f4a9006a800 2024-04-24T08:08:38.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.562+0000 7f4a8de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9006b1a0 0x7f4a9019bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:38.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.562+0000 7f4a8de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9006b1a0 0x7f4a9019bad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50576/0 (socket says 172.21.15.28:50576) 2024-04-24T08:08:38.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.562+0000 7f4a8de00700 1 -- 172.21.15.28:0/769607520 learned_addr learned my addr 172.21.15.28:0/769607520 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:38.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.562+0000 7f4a8e800700 1 --2- 172.21.15.28:0/769607520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a9006a800 0x7f4a9019b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:38.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.562+0000 7f4a8f200700 1 --2- 172.21.15.28:0/769607520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a90108b70 0x7f4a9019fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:38.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.562+0000 7f4a8de00700 1 -- 172.21.15.28:0/769607520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a9006a800 msgr2=0x7f4a9019b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:38.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.562+0000 7f4a8de00700 1 --2- 172.21.15.28:0/769607520 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4a9006a800 0x7f4a9019b590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:38.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.563+0000 7f4a8de00700 1 -- 172.21.15.28:0/769607520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a90108b70 msgr2=0x7f4a9019fe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:38.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.563+0000 7f4a8de00700 1 --2- 172.21.15.28:0/769607520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4a90108b70 0x7f4a9019fe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:38.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.563+0000 7f4a8de00700 1 -- 172.21.15.28:0/769607520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a901a0550 con 0x7f4a9006b1a0 2024-04-24T08:08:38.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.563+0000 7f4a8de00700 1 --2- 172.21.15.28:0/769607520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9006b1a0 0x7f4a9019bad0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00d950 tx=0x7f4a7c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:38.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.563+0000 7f4a87400700 1 -- 172.21.15.28:0/769607520 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c016070 con 0x7f4a9006b1a0 2024-04-24T08:08:38.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.563+0000 7f4a95400700 1 -- 172.21.15.28:0/769607520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a901a0840 con 0x7f4a9006b1a0 2024-04-24T08:08:38.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.563+0000 7f4a87400700 1 -- 172.21.15.28:0/769607520 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a7c00fd70 con 0x7f4a9006b1a0 2024-04-24T08:08:38.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.563+0000 7f4a87400700 1 -- 172.21.15.28:0/769607520 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a7c00e610 con 0x7f4a9006b1a0 2024-04-24T08:08:38.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.563+0000 7f4a95400700 1 -- 172.21.15.28:0/769607520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a901a80c0 con 0x7f4a9006b1a0 2024-04-24T08:08:38.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.565+0000 7f4a95400700 1 -- 172.21.15.28:0/769607520 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4a90063200 con 0x7f4a9006b1a0 2024-04-24T08:08:38.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.567+0000 7f4a87400700 1 -- 172.21.15.28:0/769607520 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f4a7c019070 con 0x7f4a9006b1a0 2024-04-24T08:08:38.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.568+0000 7f4a87400700 1 --2- 172.21.15.28:0/769607520 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4a740a3d50 0x7f4a740a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:38.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.568+0000 7f4a87400700 1 -- 172.21.15.28:0/769607520 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f4a7c0cfef0 con 0x7f4a9006b1a0 2024-04-24T08:08:38.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.568+0000 7f4a8e800700 1 --2- 172.21.15.28:0/769607520 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4a740a3d50 0x7f4a740a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:38.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.569+0000 7f4a87400700 1 --2- 172.21.15.28:0/769607520 >> [v2:172.21.15.117:6816/1472212690,v1:172.21.15.117:6817/1472212690] conn(0x7f4a740ad530 0x7f4a740af950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:38.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.569+0000 7f4a87400700 1 -- 172.21.15.28:0/769607520 --> [v2:172.21.15.117:6816/1472212690,v1:172.21.15.117:6817/1472212690] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4a740b0000 con 0x7f4a740ad530 2024-04-24T08:08:38.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.569+0000 7f4a87400700 1 -- 172.21.15.28:0/769607520 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f4a7c0d2510 con 0x7f4a9006b1a0 2024-04-24T08:08:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.569+0000 7f4a8f200700 1 --2- 172.21.15.28:0/769607520 >> [v2:172.21.15.117:6816/1472212690,v1:172.21.15.117:6817/1472212690] conn(0x7f4a740ad530 0x7f4a740af950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.569+0000 7f4a8e800700 1 --2- 172.21.15.28:0/769607520 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4a740a3d50 0x7f4a740a6210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4a80006010 tx=0x7f4a8000b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.570+0000 7f4a8f200700 1 --2- 172.21.15.28:0/769607520 >> [v2:172.21.15.117:6816/1472212690,v1:172.21.15.117:6817/1472212690] conn(0x7f4a740ad530 0x7f4a740af950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.571+0000 7f4a87400700 1 -- 172.21.15.28:0/769607520 <== osd.7 v2:172.21.15.117:6816/1472212690 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4a740b0000 con 0x7f4a740ad530 2024-04-24T08:08:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.591+0000 7f4a95400700 1 -- 172.21.15.28:0/769607520 --> [v2:172.21.15.117:6816/1472212690,v1:172.21.15.117:6817/1472212690] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4a90063630 con 0x7f4a740ad530 2024-04-24T08:08:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.592+0000 7f4a87400700 1 -- 172.21.15.28:0/769607520 <== osd.7 v2:172.21.15.117:6816/1472212690 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4a90063630 con 0x7f4a740ad530 2024-04-24T08:08:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.593+0000 7f4a95400700 1 -- 172.21.15.28:0/769607520 >> [v2:172.21.15.117:6816/1472212690,v1:172.21.15.117:6817/1472212690] conn(0x7f4a740ad530 msgr2=0x7f4a740af950 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.593+0000 7f4a95400700 1 --2- 172.21.15.28:0/769607520 >> [v2:172.21.15.117:6816/1472212690,v1:172.21.15.117:6817/1472212690] conn(0x7f4a740ad530 0x7f4a740af950 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.593+0000 7f4a95400700 1 -- 172.21.15.28:0/769607520 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4a740a3d50 msgr2=0x7f4a740a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.593+0000 7f4a95400700 1 --2- 172.21.15.28:0/769607520 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4a740a3d50 0x7f4a740a6210 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f4a80006010 tx=0x7f4a8000b260 comp rx=0 tx=0).stop 2024-04-24T08:08:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.593+0000 7f4a95400700 1 -- 172.21.15.28:0/769607520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9006b1a0 msgr2=0x7f4a9019bad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.593+0000 7f4a95400700 1 --2- 172.21.15.28:0/769607520 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4a9006b1a0 0x7f4a9019bad0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f4a7c00d950 tx=0x7f4a7c00dc60 comp rx=0 tx=0).stop 2024-04-24T08:08:38.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.593+0000 7f4a8f200700 1 -- 172.21.15.28:0/769607520 reap_dead start 2024-04-24T08:08:38.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.593+0000 7f4a95400700 1 -- 172.21.15.28:0/769607520 shutdown_connections 2024-04-24T08:08:38.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.593+0000 7f4a95400700 1 -- 172.21.15.28:0/769607520 >> 172.21.15.28:0/769607520 conn(0x7f4a900789a0 msgr2=0x7f4a9010dee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:38.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.594+0000 7f4a95400700 1 -- 172.21.15.28:0/769607520 shutdown_connections 2024-04-24T08:08:38.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:36.594+0000 7f4a95400700 1 -- 172.21.15.28:0/769607520 wait complete. 2024-04-24T08:08:39.387 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:39 smithi028 ceph-mon[27786]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:08:39.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:39 smithi117 ceph-mon[32438]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:08:39.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:39 smithi194 ceph-mon[32836]: pgmap v340: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:08:39.721 INFO:teuthology.orchestra.run.smithi028.stdout:77309411403 2024-04-24T08:08:39.722 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-04-24T08:08:39.843 INFO:teuthology.orchestra.run.smithi028.stdout:257698037799 2024-04-24T08:08:39.844 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-04-24T08:08:40.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.270+0000 7f8e65800700 1 -- 172.21.15.28:0/2841304970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e60108b70 msgr2=0x7f8e60108f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.270+0000 7f8e65800700 1 --2- 172.21.15.28:0/2841304970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e60108b70 0x7f8e60108f50 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8e50009990 tx=0x7f8e50009ca0 comp rx=0 tx=0).stop 2024-04-24T08:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.270+0000 7f8e65800700 1 -- 172.21.15.28:0/2841304970 shutdown_connections 2024-04-24T08:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.270+0000 7f8e65800700 1 --2- 172.21.15.28:0/2841304970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e6006b1a0 0x7f8e6010f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.270+0000 7f8e65800700 1 --2- 172.21.15.28:0/2841304970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e60108b70 0x7f8e60108f50 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.270+0000 7f8e65800700 1 --2- 172.21.15.28:0/2841304970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e6006a800 0x7f8e6006ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.270+0000 7f8e65800700 1 -- 172.21.15.28:0/2841304970 >> 172.21.15.28:0/2841304970 conn(0x7f8e600789a0 msgr2=0x7f8e60078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.270+0000 7f8e65800700 1 -- 172.21.15.28:0/2841304970 shutdown_connections 2024-04-24T08:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.270+0000 7f8e65800700 1 -- 172.21.15.28:0/2841304970 wait complete. 2024-04-24T08:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.271+0000 7f8e65800700 1 Processor -- start 2024-04-24T08:08:40.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.271+0000 7f8e65800700 1 -- start start 2024-04-24T08:08:40.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.271+0000 7f8e65800700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e6006a800 0x7f8e60199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:40.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.271+0000 7f8e65800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e6006b1a0 0x7f8e60199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:40.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.271+0000 7f8e65800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e60108b70 0x7f8e6019dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:40.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.272+0000 7f8e65800700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e60112e20 con 0x7f8e6006b1a0 2024-04-24T08:08:40.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.272+0000 7f8e65800700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8e60112ca0 con 0x7f8e6006a800 2024-04-24T08:08:40.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.272+0000 7f8e65800700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8e60112fa0 con 0x7f8e60108b70 2024-04-24T08:08:40.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.272+0000 7f8e57e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e6006b1a0 0x7f8e60199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:40.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.272+0000 7f8e57e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e6006b1a0 0x7f8e60199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50646/0 (socket says 172.21.15.28:50646) 2024-04-24T08:08:40.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.272+0000 7f8e57e00700 1 -- 172.21.15.28:0/4293980905 learned_addr learned my addr 172.21.15.28:0/4293980905 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:40.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.272+0000 7f8e5ea00700 1 --2- 172.21.15.28:0/4293980905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e6006a800 0x7f8e60199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:40.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.272+0000 7f8e5f400700 1 --2- 172.21.15.28:0/4293980905 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e60108b70 0x7f8e6019dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:40.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.272+0000 7f8e57e00700 1 -- 172.21.15.28:0/4293980905 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e60108b70 msgr2=0x7f8e6019dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:40.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.272+0000 7f8e57e00700 1 --2- 172.21.15.28:0/4293980905 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e60108b70 0x7f8e6019dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:40.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.272+0000 7f8e57e00700 1 -- 172.21.15.28:0/4293980905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e6006a800 msgr2=0x7f8e60199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:40.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.272+0000 7f8e57e00700 1 --2- 172.21.15.28:0/4293980905 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e6006a800 0x7f8e60199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:40.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.272+0000 7f8e57e00700 1 -- 172.21.15.28:0/4293980905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e6019e2f0 con 0x7f8e6006b1a0 2024-04-24T08:08:40.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.272+0000 7f8e5f400700 1 --2- 172.21.15.28:0/4293980905 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e60108b70 0x7f8e6019dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:08:40.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.273+0000 7f8e57e00700 1 --2- 172.21.15.28:0/4293980905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e6006b1a0 0x7f8e60199870 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800eb30 tx=0x7f8e4800ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:40.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.273+0000 7f8e57400700 1 -- 172.21.15.28:0/4293980905 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e4800cc90 con 0x7f8e6006b1a0 2024-04-24T08:08:40.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.273+0000 7f8e57400700 1 -- 172.21.15.28:0/4293980905 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e48004500 con 0x7f8e6006b1a0 2024-04-24T08:08:40.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.273+0000 7f8e57400700 1 -- 172.21.15.28:0/4293980905 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e48010430 con 0x7f8e6006b1a0 2024-04-24T08:08:40.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.273+0000 7f8e65800700 1 -- 172.21.15.28:0/4293980905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e6019e5e0 con 0x7f8e6006b1a0 2024-04-24T08:08:40.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.273+0000 7f8e65800700 1 -- 172.21.15.28:0/4293980905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e601a5e60 con 0x7f8e6006b1a0 2024-04-24T08:08:40.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.274+0000 7f8e65800700 1 -- 172.21.15.28:0/4293980905 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f8e60063240 con 0x7f8e6006b1a0 2024-04-24T08:08:40.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.276+0000 7f8e57400700 1 -- 172.21.15.28:0/4293980905 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8e48003f70 con 0x7f8e6006b1a0 2024-04-24T08:08:40.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.278+0000 7f8e57400700 1 --2- 172.21.15.28:0/4293980905 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8e440a3cd0 0x7f8e440a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:40.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.278+0000 7f8e57400700 1 -- 172.21.15.28:0/4293980905 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f8e480d09e0 con 0x7f8e6006b1a0 2024-04-24T08:08:40.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.278+0000 7f8e5ea00700 1 --2- 172.21.15.28:0/4293980905 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8e440a3cd0 0x7f8e440a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:40.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.278+0000 7f8e57400700 1 --2- 172.21.15.28:0/4293980905 >> [v2:172.21.15.117:6800/3269221814,v1:172.21.15.117:6801/3269221814] conn(0x7f8e440ad4b0 0x7f8e440af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:40.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.278+0000 7f8e57400700 1 -- 172.21.15.28:0/4293980905 --> [v2:172.21.15.117:6800/3269221814,v1:172.21.15.117:6801/3269221814] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f8e48023e50 con 0x7f8e440ad4b0 2024-04-24T08:08:40.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.278+0000 7f8e57400700 1 -- 172.21.15.28:0/4293980905 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f8e480d0ec0 con 0x7f8e6006b1a0 2024-04-24T08:08:40.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.279+0000 7f8e5f400700 1 --2- 172.21.15.28:0/4293980905 >> [v2:172.21.15.117:6800/3269221814,v1:172.21.15.117:6801/3269221814] conn(0x7f8e440ad4b0 0x7f8e440af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:40.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.279+0000 7f8e5ea00700 1 --2- 172.21.15.28:0/4293980905 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8e440a3cd0 0x7f8e440a6190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000b2e0 tx=0x7f8e50005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:40.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.279+0000 7f8e5f400700 1 --2- 172.21.15.28:0/4293980905 >> [v2:172.21.15.117:6800/3269221814,v1:172.21.15.117:6801/3269221814] conn(0x7f8e440ad4b0 0x7f8e440af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:40.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.281+0000 7f8e57400700 1 -- 172.21.15.28:0/4293980905 <== osd.1 v2:172.21.15.117:6800/3269221814 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f8e48023e50 con 0x7f8e440ad4b0 2024-04-24T08:08:40.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.299+0000 7f8e65800700 1 -- 172.21.15.28:0/4293980905 --> [v2:172.21.15.117:6800/3269221814,v1:172.21.15.117:6801/3269221814] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f8e600636c0 con 0x7f8e440ad4b0 2024-04-24T08:08:40.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.300+0000 7f8e57400700 1 -- 172.21.15.28:0/4293980905 <== osd.1 v2:172.21.15.117:6800/3269221814 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f8e600636c0 con 0x7f8e440ad4b0 2024-04-24T08:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.300+0000 7f8e65800700 1 -- 172.21.15.28:0/4293980905 >> [v2:172.21.15.117:6800/3269221814,v1:172.21.15.117:6801/3269221814] conn(0x7f8e440ad4b0 msgr2=0x7f8e440af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.300+0000 7f8e65800700 1 --2- 172.21.15.28:0/4293980905 >> [v2:172.21.15.117:6800/3269221814,v1:172.21.15.117:6801/3269221814] conn(0x7f8e440ad4b0 0x7f8e440af8d0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.300+0000 7f8e65800700 1 -- 172.21.15.28:0/4293980905 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8e440a3cd0 msgr2=0x7f8e440a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.300+0000 7f8e65800700 1 --2- 172.21.15.28:0/4293980905 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f8e440a3cd0 0x7f8e440a6190 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8e5000b2e0 tx=0x7f8e50005fd0 comp rx=0 tx=0).stop 2024-04-24T08:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.300+0000 7f8e65800700 1 -- 172.21.15.28:0/4293980905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e6006b1a0 msgr2=0x7f8e60199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.300+0000 7f8e65800700 1 --2- 172.21.15.28:0/4293980905 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e6006b1a0 0x7f8e60199870 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f8e4800eb30 tx=0x7f8e4800ee40 comp rx=0 tx=0).stop 2024-04-24T08:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.300+0000 7f8e5f400700 1 -- 172.21.15.28:0/4293980905 reap_dead start 2024-04-24T08:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.301+0000 7f8e65800700 1 -- 172.21.15.28:0/4293980905 shutdown_connections 2024-04-24T08:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.301+0000 7f8e65800700 1 -- 172.21.15.28:0/4293980905 >> 172.21.15.28:0/4293980905 conn(0x7f8e600789a0 msgr2=0x7f8e600ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.301+0000 7f8e65800700 1 -- 172.21.15.28:0/4293980905 shutdown_connections 2024-04-24T08:08:40.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.301+0000 7f8e65800700 1 -- 172.21.15.28:0/4293980905 wait complete. 2024-04-24T08:08:41.088 INFO:teuthology.orchestra.run.smithi028.stdout:313532612635 2024-04-24T08:08:41.088 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-04-24T08:08:41.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:41 smithi028 ceph-mon[27786]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:08:41.386 INFO:teuthology.orchestra.run.smithi028.stdout:34359738452 2024-04-24T08:08:41.386 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-04-24T08:08:41.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:41 smithi117 ceph-mon[32438]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:08:41.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:41 smithi194 ceph-mon[32836]: pgmap v341: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:08:41.856 INFO:teuthology.orchestra.run.smithi028.stdout:219043332142 2024-04-24T08:08:41.857 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-04-24T08:08:41.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.031+0000 7effbde00700 1 -- 172.21.15.28:0/3100308400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effb8074d20 msgr2=0x7effb8075100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:41.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.031+0000 7effbde00700 1 --2- 172.21.15.28:0/3100308400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effb8074d20 0x7effb8075100 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7effa8009990 tx=0x7effa8009ca0 comp rx=0 tx=0).stop 2024-04-24T08:08:41.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.031+0000 7effbde00700 1 -- 172.21.15.28:0/3100308400 shutdown_connections 2024-04-24T08:08:41.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.031+0000 7effbde00700 1 --2- 172.21.15.28:0/3100308400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb810f520 0x7effb8111910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:41.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.031+0000 7effbde00700 1 --2- 172.21.15.28:0/3100308400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effb8074d20 0x7effb8075100 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:41.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.031+0000 7effbde00700 1 --2- 172.21.15.28:0/3100308400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effb8075640 0x7effb810efe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.031+0000 7effbde00700 1 -- 172.21.15.28:0/3100308400 >> 172.21.15.28:0/3100308400 conn(0x7effb8079bc0 msgr2=0x7effb8079fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.031+0000 7effbde00700 1 -- 172.21.15.28:0/3100308400 shutdown_connections 2024-04-24T08:08:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.031+0000 7effbde00700 1 -- 172.21.15.28:0/3100308400 wait complete. 2024-04-24T08:08:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.032+0000 7effbde00700 1 Processor -- start 2024-04-24T08:08:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.033+0000 7effbde00700 1 -- start start 2024-04-24T08:08:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.033+0000 7effbde00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb8074d20 0x7effb819b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.033+0000 7effbde00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effb8075640 0x7effb819b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.033+0000 7effbde00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effb810f520 0x7effb819fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.033+0000 7effbde00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7effb81150a0 con 0x7effb8074d20 2024-04-24T08:08:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.033+0000 7effbde00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7effb8114f20 con 0x7effb810f520 2024-04-24T08:08:41.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.033+0000 7effbde00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7effb8115220 con 0x7effb8075640 2024-04-24T08:08:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.033+0000 7effb6a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb8074d20 0x7effb819b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.033+0000 7effb6a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb8074d20 0x7effb819b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50624/0 (socket says 172.21.15.28:50624) 2024-04-24T08:08:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.033+0000 7effb6a00700 1 -- 172.21.15.28:0/4192609679 learned_addr learned my addr 172.21.15.28:0/4192609679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.034+0000 7effb6000700 1 --2- 172.21.15.28:0/4192609679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effb8075640 0x7effb819b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.034+0000 7effb7400700 1 --2- 172.21.15.28:0/4192609679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effb810f520 0x7effb819fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.034+0000 7effb6a00700 1 -- 172.21.15.28:0/4192609679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effb8075640 msgr2=0x7effb819b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.034+0000 7effb6a00700 1 --2- 172.21.15.28:0/4192609679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effb8075640 0x7effb819b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.034+0000 7effb6a00700 1 -- 172.21.15.28:0/4192609679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effb810f520 msgr2=0x7effb819fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.034+0000 7effb6a00700 1 --2- 172.21.15.28:0/4192609679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effb810f520 0x7effb819fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.034+0000 7effb6a00700 1 -- 172.21.15.28:0/4192609679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb81a04c0 con 0x7effb8074d20 2024-04-24T08:08:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.034+0000 7effb7400700 1 --2- 172.21.15.28:0/4192609679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effb810f520 0x7effb819fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.034+0000 7effb6a00700 1 --2- 172.21.15.28:0/4192609679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb8074d20 0x7effb819b4b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7effa8005910 tx=0x7effa800bab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:41.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.035+0000 7effbca00700 1 -- 172.21.15.28:0/4192609679 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa8027070 con 0x7effb8074d20 2024-04-24T08:08:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.035+0000 7effbde00700 1 -- 172.21.15.28:0/4192609679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effb81a0750 con 0x7effb8074d20 2024-04-24T08:08:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.035+0000 7effbca00700 1 -- 172.21.15.28:0/4192609679 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effa8004650 con 0x7effb8074d20 2024-04-24T08:08:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.035+0000 7effbca00700 1 -- 172.21.15.28:0/4192609679 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa800f460 con 0x7effb8074d20 2024-04-24T08:08:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.035+0000 7effbde00700 1 -- 172.21.15.28:0/4192609679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effb81a8030 con 0x7effb8074d20 2024-04-24T08:08:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.036+0000 7effbde00700 1 -- 172.21.15.28:0/4192609679 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7effb8100630 con 0x7effb8074d20 2024-04-24T08:08:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.038+0000 7effbca00700 1 -- 172.21.15.28:0/4192609679 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7effa802a070 con 0x7effb8074d20 2024-04-24T08:08:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.039+0000 7effbca00700 1 --2- 172.21.15.28:0/4192609679 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7eff9c0a3cd0 0x7eff9c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.040+0000 7effbca00700 1 -- 172.21.15.28:0/4192609679 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7effa80d1ab0 con 0x7effb8074d20 2024-04-24T08:08:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.040+0000 7effb6000700 1 --2- 172.21.15.28:0/4192609679 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7eff9c0a3cd0 0x7eff9c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.040+0000 7effbca00700 1 --2- 172.21.15.28:0/4192609679 >> [v2:172.21.15.194:6824/275366773,v1:172.21.15.194:6825/275366773] conn(0x7eff9c0ad4b0 0x7eff9c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.040+0000 7effbca00700 1 -- 172.21.15.28:0/4192609679 --> [v2:172.21.15.194:6824/275366773,v1:172.21.15.194:6825/275366773] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7effa8019e80 con 0x7eff9c0ad4b0 2024-04-24T08:08:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.040+0000 7effbca00700 1 -- 172.21.15.28:0/4192609679 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7effa80d1e60 con 0x7effb8074d20 2024-04-24T08:08:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.040+0000 7effb7400700 1 --2- 172.21.15.28:0/4192609679 >> [v2:172.21.15.194:6824/275366773,v1:172.21.15.194:6825/275366773] conn(0x7eff9c0ad4b0 0x7eff9c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:41.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.040+0000 7effb6000700 1 --2- 172.21.15.28:0/4192609679 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7eff9c0a3cd0 0x7eff9c0a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7effb81029b0 tx=0x7effa4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.041+0000 7effb7400700 1 --2- 172.21.15.28:0/4192609679 >> [v2:172.21.15.194:6824/275366773,v1:172.21.15.194:6825/275366773] conn(0x7eff9c0ad4b0 0x7eff9c0af8d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.042+0000 7effbca00700 1 -- 172.21.15.28:0/4192609679 <== osd.11 v2:172.21.15.194:6824/275366773 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7effa8019e80 con 0x7eff9c0ad4b0 2024-04-24T08:08:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.062+0000 7effbde00700 1 -- 172.21.15.28:0/4192609679 --> [v2:172.21.15.194:6824/275366773,v1:172.21.15.194:6825/275366773] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7effb8076410 con 0x7eff9c0ad4b0 2024-04-24T08:08:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.062+0000 7effbca00700 1 -- 172.21.15.28:0/4192609679 <== osd.11 v2:172.21.15.194:6824/275366773 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7effb8076410 con 0x7eff9c0ad4b0 2024-04-24T08:08:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.063+0000 7effbde00700 1 -- 172.21.15.28:0/4192609679 >> [v2:172.21.15.194:6824/275366773,v1:172.21.15.194:6825/275366773] conn(0x7eff9c0ad4b0 msgr2=0x7eff9c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.063+0000 7effbde00700 1 --2- 172.21.15.28:0/4192609679 >> [v2:172.21.15.194:6824/275366773,v1:172.21.15.194:6825/275366773] conn(0x7eff9c0ad4b0 0x7eff9c0af8d0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.063+0000 7effbde00700 1 -- 172.21.15.28:0/4192609679 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7eff9c0a3cd0 msgr2=0x7eff9c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.063+0000 7effbde00700 1 --2- 172.21.15.28:0/4192609679 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7eff9c0a3cd0 0x7eff9c0a6190 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7effb81029b0 tx=0x7effa4009260 comp rx=0 tx=0).stop 2024-04-24T08:08:41.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.063+0000 7effbde00700 1 -- 172.21.15.28:0/4192609679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb8074d20 msgr2=0x7effb819b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.063+0000 7effbde00700 1 --2- 172.21.15.28:0/4192609679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effb8074d20 0x7effb819b4b0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7effa8005910 tx=0x7effa800bab0 comp rx=0 tx=0).stop 2024-04-24T08:08:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.063+0000 7effb7400700 1 -- 172.21.15.28:0/4192609679 reap_dead start 2024-04-24T08:08:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.063+0000 7effbde00700 1 -- 172.21.15.28:0/4192609679 shutdown_connections 2024-04-24T08:08:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.063+0000 7effbde00700 1 -- 172.21.15.28:0/4192609679 >> 172.21.15.28:0/4192609679 conn(0x7effb8079bc0 msgr2=0x7effb80ffc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.063+0000 7effbde00700 1 -- 172.21.15.28:0/4192609679 shutdown_connections 2024-04-24T08:08:41.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:39.064+0000 7effbde00700 1 -- 172.21.15.28:0/4192609679 wait complete. 2024-04-24T08:08:43.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.349+0000 7fc3e1c00700 1 -- 172.21.15.28:0/2131744762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3dc10a020 msgr2=0x7fc3dc111bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:43.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.349+0000 7fc3e1c00700 1 --2- 172.21.15.28:0/2131744762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3dc10a020 0x7fc3dc111bf0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7fc3d400b0d0 tx=0x7fc3d400b3e0 comp rx=0 tx=0).stop 2024-04-24T08:08:43.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.349+0000 7fc3e1c00700 1 -- 172.21.15.28:0/2131744762 shutdown_connections 2024-04-24T08:08:43.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.349+0000 7fc3e1c00700 1 --2- 172.21.15.28:0/2131744762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3dc10a020 0x7fc3dc111bf0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:43.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.349+0000 7fc3e1c00700 1 --2- 172.21.15.28:0/2131744762 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3dc104f40 0x7fc3dc105320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:43.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.349+0000 7fc3e1c00700 1 --2- 172.21.15.28:0/2131744762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3dc1058f0 0x7fc3dc109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:43.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.349+0000 7fc3e1c00700 1 -- 172.21.15.28:0/2131744762 >> 172.21.15.28:0/2131744762 conn(0x7fc3dc1007d0 msgr2=0x7fc3dc102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:43.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.350+0000 7fc3e1c00700 1 -- 172.21.15.28:0/2131744762 shutdown_connections 2024-04-24T08:08:43.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.350+0000 7fc3e1c00700 1 -- 172.21.15.28:0/2131744762 wait complete. 2024-04-24T08:08:43.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.351+0000 7fc3e1c00700 1 Processor -- start 2024-04-24T08:08:43.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.351+0000 7fc3e1c00700 1 -- start start 2024-04-24T08:08:43.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.352+0000 7fc3e1c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3dc104f40 0x7fc3dc19b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:43.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.352+0000 7fc3e1c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3dc1058f0 0x7fc3dc19b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:43.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.352+0000 7fc3e1c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3dc10a020 0x7fc3dc19fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:43.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.352+0000 7fc3e1c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc115080 con 0x7fc3dc1058f0 2024-04-24T08:08:43.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.352+0000 7fc3e1c00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc114f00 con 0x7fc3dc10a020 2024-04-24T08:08:43.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.352+0000 7fc3e1c00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc3dc115200 con 0x7fc3dc104f40 2024-04-24T08:08:43.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.352+0000 7fc3daa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3dc104f40 0x7fc3dc19b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:43.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.352+0000 7fc3daa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3dc104f40 0x7fc3dc19b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:50582/0 (socket says 172.21.15.28:50582) 2024-04-24T08:08:43.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.352+0000 7fc3daa00700 1 -- 172.21.15.28:0/847192172 learned_addr learned my addr 172.21.15.28:0/847192172 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:43.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.352+0000 7fc3da000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3dc1058f0 0x7fc3dc19b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:43.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.352+0000 7fc3db400700 1 --2- 172.21.15.28:0/847192172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3dc10a020 0x7fc3dc19fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:43.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.353+0000 7fc3daa00700 1 -- 172.21.15.28:0/847192172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3dc10a020 msgr2=0x7fc3dc19fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:43.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.353+0000 7fc3daa00700 1 --2- 172.21.15.28:0/847192172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3dc10a020 0x7fc3dc19fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:43.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.353+0000 7fc3daa00700 1 -- 172.21.15.28:0/847192172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3dc1058f0 msgr2=0x7fc3dc19b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:43.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.353+0000 7fc3daa00700 1 --2- 172.21.15.28:0/847192172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3dc1058f0 0x7fc3dc19b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:43.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.353+0000 7fc3daa00700 1 -- 172.21.15.28:0/847192172 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3dc1a04c0 con 0x7fc3dc104f40 2024-04-24T08:08:43.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.353+0000 7fc3da000700 1 --2- 172.21.15.28:0/847192172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3dc1058f0 0x7fc3dc19b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:43.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.353+0000 7fc3daa00700 1 --2- 172.21.15.28:0/847192172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3dc104f40 0x7fc3dc19b4b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc00e9c0 tx=0x7fc3cc00ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:43.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.353+0000 7fc3d3400700 1 -- 172.21.15.28:0/847192172 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc00cbd0 con 0x7fc3dc104f40 2024-04-24T08:08:43.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.353+0000 7fc3d3400700 1 -- 172.21.15.28:0/847192172 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3cc00ee60 con 0x7fc3dc104f40 2024-04-24T08:08:43.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.353+0000 7fc3d3400700 1 -- 172.21.15.28:0/847192172 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3cc005630 con 0x7fc3dc104f40 2024-04-24T08:08:43.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.353+0000 7fc3e1c00700 1 -- 172.21.15.28:0/847192172 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3dc1a07b0 con 0x7fc3dc104f40 2024-04-24T08:08:43.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.354+0000 7fc3e1c00700 1 -- 172.21.15.28:0/847192172 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3dc1a8030 con 0x7fc3dc104f40 2024-04-24T08:08:43.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.354+0000 7fc3e1c00700 1 -- 172.21.15.28:0/847192172 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fc3dc063280 con 0x7fc3dc104f40 2024-04-24T08:08:43.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.356+0000 7fc3d3400700 1 -- 172.21.15.28:0/847192172 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc3cc004750 con 0x7fc3dc104f40 2024-04-24T08:08:43.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.358+0000 7fc3d3400700 1 --2- 172.21.15.28:0/847192172 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc3c00a3cd0 0x7fc3c00a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:43.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.358+0000 7fc3d3400700 1 -- 172.21.15.28:0/847192172 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fc3cc0d0770 con 0x7fc3dc104f40 2024-04-24T08:08:43.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.358+0000 7fc3da000700 1 --2- 172.21.15.28:0/847192172 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc3c00a3cd0 0x7fc3c00a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:43.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.358+0000 7fc3d3400700 1 --2- 172.21.15.28:0/847192172 >> [v2:172.21.15.28:6818/1356466279,v1:172.21.15.28:6819/1356466279] conn(0x7fc3c00ad510 0x7fc3c00af930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:43.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.358+0000 7fc3d3400700 1 -- 172.21.15.28:0/847192172 --> [v2:172.21.15.28:6818/1356466279,v1:172.21.15.28:6819/1356466279] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fc3cc017e50 con 0x7fc3c00ad510 2024-04-24T08:08:43.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.358+0000 7fc3d3400700 1 -- 172.21.15.28:0/847192172 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fc3cc0d7050 con 0x7fc3dc104f40 2024-04-24T08:08:43.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.359+0000 7fc3db400700 1 --2- 172.21.15.28:0/847192172 >> [v2:172.21.15.28:6818/1356466279,v1:172.21.15.28:6819/1356466279] conn(0x7fc3c00ad510 0x7fc3c00af930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:43.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.359+0000 7fc3da000700 1 --2- 172.21.15.28:0/847192172 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc3c00a3cd0 0x7fc3c00a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8009680 tx=0x7fc3c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:43.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.359+0000 7fc3db400700 1 --2- 172.21.15.28:0/847192172 >> [v2:172.21.15.28:6818/1356466279,v1:172.21.15.28:6819/1356466279] conn(0x7fc3c00ad510 0x7fc3c00af930 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:43.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.360+0000 7fc3d3400700 1 -- 172.21.15.28:0/847192172 <== osd.6 v2:172.21.15.28:6818/1356466279 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fc3cc017e50 con 0x7fc3c00ad510 2024-04-24T08:08:43.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.379+0000 7fc3e1c00700 1 -- 172.21.15.28:0/847192172 --> [v2:172.21.15.28:6818/1356466279,v1:172.21.15.28:6819/1356466279] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fc3dc063700 con 0x7fc3c00ad510 2024-04-24T08:08:43.380 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:43 smithi028 ceph-mon[27786]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:08:43.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.379+0000 7fc3d3400700 1 -- 172.21.15.28:0/847192172 <== osd.6 v2:172.21.15.28:6818/1356466279 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fc3dc063700 con 0x7fc3c00ad510 2024-04-24T08:08:43.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.380+0000 7fc3e1c00700 1 -- 172.21.15.28:0/847192172 >> [v2:172.21.15.28:6818/1356466279,v1:172.21.15.28:6819/1356466279] conn(0x7fc3c00ad510 msgr2=0x7fc3c00af930 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:43.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.380+0000 7fc3e1c00700 1 --2- 172.21.15.28:0/847192172 >> [v2:172.21.15.28:6818/1356466279,v1:172.21.15.28:6819/1356466279] conn(0x7fc3c00ad510 0x7fc3c00af930 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:43.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.380+0000 7fc3e1c00700 1 -- 172.21.15.28:0/847192172 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc3c00a3cd0 msgr2=0x7fc3c00a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.380+0000 7fc3e1c00700 1 --2- 172.21.15.28:0/847192172 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc3c00a3cd0 0x7fc3c00a6190 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fc3c8009680 tx=0x7fc3c8009260 comp rx=0 tx=0).stop 2024-04-24T08:08:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.380+0000 7fc3e1c00700 1 -- 172.21.15.28:0/847192172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3dc104f40 msgr2=0x7fc3dc19b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.380+0000 7fc3e1c00700 1 --2- 172.21.15.28:0/847192172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3dc104f40 0x7fc3dc19b4b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc3cc00e9c0 tx=0x7fc3cc00ecd0 comp rx=0 tx=0).stop 2024-04-24T08:08:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.380+0000 7fc3db400700 1 -- 172.21.15.28:0/847192172 reap_dead start 2024-04-24T08:08:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.380+0000 7fc3e1c00700 1 -- 172.21.15.28:0/847192172 shutdown_connections 2024-04-24T08:08:43.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.380+0000 7fc3e1c00700 1 -- 172.21.15.28:0/847192172 >> 172.21.15.28:0/847192172 conn(0x7fc3dc1007d0 msgr2=0x7fc3dc1011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:43.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.381+0000 7fc3e1c00700 1 -- 172.21.15.28:0/847192172 shutdown_connections 2024-04-24T08:08:43.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.381+0000 7fc3e1c00700 1 -- 172.21.15.28:0/847192172 wait complete. 2024-04-24T08:08:43.614 INFO:teuthology.orchestra.run.smithi028.stdout:347892351000 2024-04-24T08:08:43.615 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-04-24T08:08:43.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:43 smithi117 ceph-mon[32438]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:08:43.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:43 smithi194 ceph-mon[32836]: pgmap v342: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:08:43.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.815+0000 7f7ac2400700 1 -- 172.21.15.28:0/1363038077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc102a20 msgr2=0x7f7abc102ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:43.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.815+0000 7f7ac2400700 1 --2- 172.21.15.28:0/1363038077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc102a20 0x7f7abc102ea0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009990 tx=0x7f7aa8009ca0 comp rx=0 tx=0).stop 2024-04-24T08:08:43.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.815+0000 7f7ac2400700 1 -- 172.21.15.28:0/1363038077 shutdown_connections 2024-04-24T08:08:43.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.815+0000 7f7ac2400700 1 --2- 172.21.15.28:0/1363038077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7abc1033e0 0x7f7abc10f8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:43.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.815+0000 7f7ac2400700 1 --2- 172.21.15.28:0/1363038077 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7abc108a20 0x7f7abc108e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:43.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.815+0000 7f7ac2400700 1 --2- 172.21.15.28:0/1363038077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc102a20 0x7f7abc102ea0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.815+0000 7f7ac2400700 1 -- 172.21.15.28:0/1363038077 >> 172.21.15.28:0/1363038077 conn(0x7f7abc0fe570 msgr2=0x7f7abc100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.815+0000 7f7ac2400700 1 -- 172.21.15.28:0/1363038077 shutdown_connections 2024-04-24T08:08:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.815+0000 7f7ac2400700 1 -- 172.21.15.28:0/1363038077 wait complete. 2024-04-24T08:08:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.816+0000 7f7ac2400700 1 Processor -- start 2024-04-24T08:08:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.816+0000 7f7ac2400700 1 -- start start 2024-04-24T08:08:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.816+0000 7f7ac2400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc102a20 0x7f7abc074df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.817+0000 7f7ac2400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7abc1033e0 0x7f7abc075330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.817+0000 7f7ac2400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7abc108a20 0x7f7abc075870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.817+0000 7f7ac2400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7abc112e00 con 0x7f7abc102a20 2024-04-24T08:08:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.817+0000 7f7ac2400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7abc112c80 con 0x7f7abc108a20 2024-04-24T08:08:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.817+0000 7f7ac2400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7abc112f80 con 0x7f7abc1033e0 2024-04-24T08:08:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.817+0000 7f7ac1000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc102a20 0x7f7abc074df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.817+0000 7f7abbe00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7abc1033e0 0x7f7abc075330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.817+0000 7f7ac1000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc102a20 0x7f7abc074df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56990/0 (socket says 172.21.15.28:56990) 2024-04-24T08:08:43.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.817+0000 7f7ac1000700 1 -- 172.21.15.28:0/3773577595 learned_addr learned my addr 172.21.15.28:0/3773577595 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.817+0000 7f7ac1a00700 1 --2- 172.21.15.28:0/3773577595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7abc108a20 0x7f7abc075870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.818+0000 7f7ac1000700 1 -- 172.21.15.28:0/3773577595 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7abc1033e0 msgr2=0x7f7abc075330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.818+0000 7f7ac1000700 1 --2- 172.21.15.28:0/3773577595 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7abc1033e0 0x7f7abc075330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.818+0000 7f7ac1000700 1 -- 172.21.15.28:0/3773577595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7abc108a20 msgr2=0x7f7abc075870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.818+0000 7f7ac1000700 1 --2- 172.21.15.28:0/3773577595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7abc108a20 0x7f7abc075870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.818+0000 7f7ac1000700 1 -- 172.21.15.28:0/3773577595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7abc1a7720 con 0x7f7abc102a20 2024-04-24T08:08:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.818+0000 7f7ac1a00700 1 --2- 172.21.15.28:0/3773577595 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7abc108a20 0x7f7abc075870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.818+0000 7f7ac1000700 1 --2- 172.21.15.28:0/3773577595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc102a20 0x7f7abc074df0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7ab000e9a0 tx=0x7f7ab000ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.818+0000 7f7ab9600700 1 -- 172.21.15.28:0/3773577595 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab000ca10 con 0x7f7abc102a20 2024-04-24T08:08:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.818+0000 7f7ab9600700 1 -- 172.21.15.28:0/3773577595 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ab000cbb0 con 0x7f7abc102a20 2024-04-24T08:08:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.819+0000 7f7ac2400700 1 -- 172.21.15.28:0/3773577595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7abc1a7a10 con 0x7f7abc102a20 2024-04-24T08:08:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.819+0000 7f7ab9600700 1 -- 172.21.15.28:0/3773577595 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ab0018680 con 0x7f7abc102a20 2024-04-24T08:08:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.819+0000 7f7ac2400700 1 -- 172.21.15.28:0/3773577595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7abc1a7f50 con 0x7f7abc102a20 2024-04-24T08:08:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.820+0000 7f7ac2400700 1 -- 172.21.15.28:0/3773577595 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f7abc063180 con 0x7f7abc102a20 2024-04-24T08:08:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.824+0000 7f7ab9600700 1 -- 172.21.15.28:0/3773577595 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f7ab0010430 con 0x7f7abc102a20 2024-04-24T08:08:43.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.825+0000 7f7ab9600700 1 --2- 172.21.15.28:0/3773577595 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7aa40a3cd0 0x7f7aa40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:43.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.825+0000 7f7ab9600700 1 -- 172.21.15.28:0/3773577595 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f7ab00d0e60 con 0x7f7abc102a20 2024-04-24T08:08:43.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.825+0000 7f7abbe00700 1 --2- 172.21.15.28:0/3773577595 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7aa40a3cd0 0x7f7aa40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:43.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.826+0000 7f7ab9600700 1 --2- 172.21.15.28:0/3773577595 >> [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688] conn(0x7f7aa40ad4b0 0x7f7aa40af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:43.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.826+0000 7f7ab9600700 1 -- 172.21.15.28:0/3773577595 --> [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f7ab0027e50 con 0x7f7aa40ad4b0 2024-04-24T08:08:43.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.826+0000 7f7ab9600700 1 -- 172.21.15.28:0/3773577595 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f7ab00d1210 con 0x7f7abc102a20 2024-04-24T08:08:43.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.826+0000 7f7ac1a00700 1 --2- 172.21.15.28:0/3773577595 >> [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688] conn(0x7f7aa40ad4b0 0x7f7aa40af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:43.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.826+0000 7f7abbe00700 1 --2- 172.21.15.28:0/3773577595 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7aa40a3cd0 0x7f7aa40a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7abc100b70 tx=0x7f7aa8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:43.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.827+0000 7f7ac1a00700 1 --2- 172.21.15.28:0/3773577595 >> [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688] conn(0x7f7aa40ad4b0 0x7f7aa40af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.828+0000 7f7ab9600700 1 -- 172.21.15.28:0/3773577595 <== osd.9 v2:172.21.15.28:6826/2537306688 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f7ab0027e50 con 0x7f7aa40ad4b0 2024-04-24T08:08:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.846+0000 7f7ac2400700 1 -- 172.21.15.28:0/3773577595 --> [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f7abc063600 con 0x7f7aa40ad4b0 2024-04-24T08:08:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.847+0000 7f7ab9600700 1 -- 172.21.15.28:0/3773577595 <== osd.9 v2:172.21.15.28:6826/2537306688 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f7abc063600 con 0x7f7aa40ad4b0 2024-04-24T08:08:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.848+0000 7f7ac2400700 1 -- 172.21.15.28:0/3773577595 >> [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688] conn(0x7f7aa40ad4b0 msgr2=0x7f7aa40af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.848+0000 7f7ac2400700 1 --2- 172.21.15.28:0/3773577595 >> [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688] conn(0x7f7aa40ad4b0 0x7f7aa40af8d0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.848+0000 7f7ac2400700 1 -- 172.21.15.28:0/3773577595 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7aa40a3cd0 msgr2=0x7f7aa40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.848+0000 7f7ac2400700 1 --2- 172.21.15.28:0/3773577595 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7aa40a3cd0 0x7f7aa40a6190 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7abc100b70 tx=0x7f7aa8005fd0 comp rx=0 tx=0).stop 2024-04-24T08:08:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.848+0000 7f7ac2400700 1 -- 172.21.15.28:0/3773577595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc102a20 msgr2=0x7f7abc074df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.848+0000 7f7ac2400700 1 --2- 172.21.15.28:0/3773577595 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7abc102a20 0x7f7abc074df0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f7ab000e9a0 tx=0x7f7ab000ecb0 comp rx=0 tx=0).stop 2024-04-24T08:08:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.848+0000 7f7ac1a00700 1 -- 172.21.15.28:0/3773577595 reap_dead start 2024-04-24T08:08:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.848+0000 7f7ac2400700 1 -- 172.21.15.28:0/3773577595 shutdown_connections 2024-04-24T08:08:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.848+0000 7f7ac2400700 1 -- 172.21.15.28:0/3773577595 >> 172.21.15.28:0/3773577595 conn(0x7f7abc0fe570 msgr2=0x7f7abc0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.849+0000 7f7ac2400700 1 -- 172.21.15.28:0/3773577595 shutdown_connections 2024-04-24T08:08:43.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:43.849+0000 7f7ac2400700 1 -- 172.21.15.28:0/3773577595 wait complete. 2024-04-24T08:08:44.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.147+0000 7fbdf5a00700 1 -- 172.21.15.28:0/1684939358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdf010f2e0 msgr2=0x7fbdf01116d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.147+0000 7fbdf5a00700 1 --2- 172.21.15.28:0/1684939358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdf010f2e0 0x7fbdf01116d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbde800b0a0 tx=0x7fbde800b3b0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.148+0000 7fbdf5a00700 1 -- 172.21.15.28:0/1684939358 shutdown_connections 2024-04-24T08:08:44.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.148+0000 7fbdf5a00700 1 --2- 172.21.15.28:0/1684939358 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdf010f2e0 0x7fbdf01116d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.148+0000 7fbdf5a00700 1 --2- 172.21.15.28:0/1684939358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdf0074e10 0x7fbdf00751f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.148+0000 7fbdf5a00700 1 --2- 172.21.15.28:0/1684939358 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdf0075730 0x7fbdf010eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.148+0000 7fbdf5a00700 1 -- 172.21.15.28:0/1684939358 >> 172.21.15.28:0/1684939358 conn(0x7fbdf00fe570 msgr2=0x7fbdf0100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:44.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.148+0000 7fbdf5a00700 1 -- 172.21.15.28:0/1684939358 shutdown_connections 2024-04-24T08:08:44.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.148+0000 7fbdf5a00700 1 -- 172.21.15.28:0/1684939358 wait complete. 2024-04-24T08:08:44.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.149+0000 7fbdf5a00700 1 Processor -- start 2024-04-24T08:08:44.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.149+0000 7fbdf5a00700 1 -- start start 2024-04-24T08:08:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.149+0000 7fbdf5a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdf0074e10 0x7fbdf019b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.149+0000 7fbdf5a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdf0075730 0x7fbdf019b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.149+0000 7fbdf5a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdf010f2e0 0x7fbdf019fc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.149+0000 7fbdf5a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf0114f80 con 0x7fbdf010f2e0 2024-04-24T08:08:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.149+0000 7fbdf5a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf0114e00 con 0x7fbdf0074e10 2024-04-24T08:08:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.149+0000 7fbdf5a00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbdf0115100 con 0x7fbdf0075730 2024-04-24T08:08:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.150+0000 7fbdf5000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdf010f2e0 0x7fbdf019fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.150+0000 7fbdefe00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdf0074e10 0x7fbdf019b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.150+0000 7fbdef400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdf0075730 0x7fbdf019b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.150+0000 7fbdefe00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdf0074e10 0x7fbdf019b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:44184/0 (socket says 172.21.15.28:44184) 2024-04-24T08:08:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.150+0000 7fbdef400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdf0075730 0x7fbdf019b8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:50608/0 (socket says 172.21.15.28:50608) 2024-04-24T08:08:44.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.150+0000 7fbdf5000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdf010f2e0 0x7fbdf019fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57016/0 (socket says 172.21.15.28:57016) 2024-04-24T08:08:44.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.150+0000 7fbdefe00700 1 -- 172.21.15.28:0/232474513 learned_addr learned my addr 172.21.15.28:0/232474513 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:44.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.150+0000 7fbdef400700 1 -- 172.21.15.28:0/232474513 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdf0074e10 msgr2=0x7fbdf019b370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.150+0000 7fbdef400700 1 --2- 172.21.15.28:0/232474513 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdf0074e10 0x7fbdf019b370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.150+0000 7fbdef400700 1 -- 172.21.15.28:0/232474513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdf010f2e0 msgr2=0x7fbdf019fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.150+0000 7fbdef400700 1 --2- 172.21.15.28:0/232474513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdf010f2e0 0x7fbdf019fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.150+0000 7fbdef400700 1 -- 172.21.15.28:0/232474513 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbdf01a0380 con 0x7fbdf0075730 2024-04-24T08:08:44.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.150+0000 7fbdf5000700 1 --2- 172.21.15.28:0/232474513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbdf010f2e0 0x7fbdf019fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:44.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.151+0000 7fbdef400700 1 --2- 172.21.15.28:0/232474513 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdf0075730 0x7fbdf019b8b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbddc00e9f0 tx=0x7fbddc00ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:44.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.151+0000 7fbdecc00700 1 -- 172.21.15.28:0/232474513 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbddc00cb70 con 0x7fbdf0075730 2024-04-24T08:08:44.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.151+0000 7fbdecc00700 1 -- 172.21.15.28:0/232474513 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbddc00ee90 con 0x7fbdf0075730 2024-04-24T08:08:44.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.151+0000 7fbdecc00700 1 -- 172.21.15.28:0/232474513 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbddc003dc0 con 0x7fbdf0075730 2024-04-24T08:08:44.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.151+0000 7fbdf5a00700 1 -- 172.21.15.28:0/232474513 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbdf01a0670 con 0x7fbdf0075730 2024-04-24T08:08:44.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.151+0000 7fbdf5a00700 1 -- 172.21.15.28:0/232474513 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbdf01a7ef0 con 0x7fbdf0075730 2024-04-24T08:08:44.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.152+0000 7fbdf5a00700 1 -- 172.21.15.28:0/232474513 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fbdf0100500 con 0x7fbdf0075730 2024-04-24T08:08:44.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.155+0000 7fbdecc00700 1 -- 172.21.15.28:0/232474513 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fbddc003f60 con 0x7fbdf0075730 2024-04-24T08:08:44.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.156+0000 7fbdecc00700 1 --2- 172.21.15.28:0/232474513 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbdd40a3cd0 0x7fbdd40a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.156+0000 7fbdecc00700 1 -- 172.21.15.28:0/232474513 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7fbddc0d0850 con 0x7fbdf0075730 2024-04-24T08:08:44.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.157+0000 7fbdefe00700 1 --2- 172.21.15.28:0/232474513 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbdd40a3cd0 0x7fbdd40a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.157+0000 7fbdecc00700 1 --2- 172.21.15.28:0/232474513 >> [v2:172.21.15.117:6808/1649446681,v1:172.21.15.117:6809/1649446681] conn(0x7fbdd40ad4b0 0x7fbdd40af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.157+0000 7fbdecc00700 1 -- 172.21.15.28:0/232474513 --> [v2:172.21.15.117:6808/1649446681,v1:172.21.15.117:6809/1649446681] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fbddc0090b0 con 0x7fbdd40ad4b0 2024-04-24T08:08:44.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.157+0000 7fbdecc00700 1 -- 172.21.15.28:0/232474513 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7fbddc0d0c00 con 0x7fbdf0075730 2024-04-24T08:08:44.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.157+0000 7fbdf5000700 1 --2- 172.21.15.28:0/232474513 >> [v2:172.21.15.117:6808/1649446681,v1:172.21.15.117:6809/1649446681] conn(0x7fbdd40ad4b0 0x7fbdd40af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.157+0000 7fbdefe00700 1 --2- 172.21.15.28:0/232474513 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbdd40a3cd0 0x7fbdd40a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009d60 tx=0x7fbde0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:44.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.158+0000 7fbdf5000700 1 --2- 172.21.15.28:0/232474513 >> [v2:172.21.15.117:6808/1649446681,v1:172.21.15.117:6809/1649446681] conn(0x7fbdd40ad4b0 0x7fbdd40af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:44.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.159+0000 7fbdecc00700 1 -- 172.21.15.28:0/232474513 <== osd.4 v2:172.21.15.117:6808/1649446681 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fbddc0090b0 con 0x7fbdd40ad4b0 2024-04-24T08:08:44.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.177+0000 7fbdf5a00700 1 -- 172.21.15.28:0/232474513 --> [v2:172.21.15.117:6808/1649446681,v1:172.21.15.117:6809/1649446681] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fbdf0076550 con 0x7fbdd40ad4b0 2024-04-24T08:08:44.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.177+0000 7fbdecc00700 1 -- 172.21.15.28:0/232474513 <== osd.4 v2:172.21.15.117:6808/1649446681 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fbdf0076550 con 0x7fbdd40ad4b0 2024-04-24T08:08:44.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.178+0000 7fbdf5a00700 1 -- 172.21.15.28:0/232474513 >> [v2:172.21.15.117:6808/1649446681,v1:172.21.15.117:6809/1649446681] conn(0x7fbdd40ad4b0 msgr2=0x7fbdd40af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.178+0000 7fbdf5a00700 1 --2- 172.21.15.28:0/232474513 >> [v2:172.21.15.117:6808/1649446681,v1:172.21.15.117:6809/1649446681] conn(0x7fbdd40ad4b0 0x7fbdd40af8d0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.178+0000 7fbdf5a00700 1 -- 172.21.15.28:0/232474513 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbdd40a3cd0 msgr2=0x7fbdd40a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.178+0000 7fbdf5a00700 1 --2- 172.21.15.28:0/232474513 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbdd40a3cd0 0x7fbdd40a6190 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009d60 tx=0x7fbde0009260 comp rx=0 tx=0).stop 2024-04-24T08:08:44.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.178+0000 7fbdf5a00700 1 -- 172.21.15.28:0/232474513 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdf0075730 msgr2=0x7fbdf019b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.178+0000 7fbdf5a00700 1 --2- 172.21.15.28:0/232474513 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbdf0075730 0x7fbdf019b8b0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fbddc00e9f0 tx=0x7fbddc00ed00 comp rx=0 tx=0).stop 2024-04-24T08:08:44.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.178+0000 7fbdf5000700 1 -- 172.21.15.28:0/232474513 reap_dead start 2024-04-24T08:08:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.178+0000 7fbdf5a00700 1 -- 172.21.15.28:0/232474513 shutdown_connections 2024-04-24T08:08:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.178+0000 7fbdf5a00700 1 -- 172.21.15.28:0/232474513 >> 172.21.15.28:0/232474513 conn(0x7fbdf00fe570 msgr2=0x7fbdf01114e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.178+0000 7fbdf5a00700 1 -- 172.21.15.28:0/232474513 shutdown_connections 2024-04-24T08:08:44.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.178+0000 7fbdf5a00700 1 -- 172.21.15.28:0/232474513 wait complete. 2024-04-24T08:08:44.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.846+0000 7f067be00700 1 -- 172.21.15.28:0/1100726513 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0674104f90 msgr2=0x7f0674105370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.846+0000 7f067be00700 1 --2- 172.21.15.28:0/1100726513 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0674104f90 0x7f0674105370 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0668009990 tx=0x7f0668009ca0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.846+0000 7f067be00700 1 -- 172.21.15.28:0/1100726513 shutdown_connections 2024-04-24T08:08:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.846+0000 7f067be00700 1 --2- 172.21.15.28:0/1100726513 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f067410a070 0x7f0674111c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.846+0000 7f067be00700 1 --2- 172.21.15.28:0/1100726513 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0674104f90 0x7f0674105370 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.846+0000 7f067be00700 1 --2- 172.21.15.28:0/1100726513 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0674105940 0x7f0674109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.846+0000 7f067be00700 1 -- 172.21.15.28:0/1100726513 >> 172.21.15.28:0/1100726513 conn(0x7f0674100820 msgr2=0x7f0674102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.846+0000 7f067be00700 1 -- 172.21.15.28:0/1100726513 shutdown_connections 2024-04-24T08:08:44.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.846+0000 7f067be00700 1 -- 172.21.15.28:0/1100726513 wait complete. 2024-04-24T08:08:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.847+0000 7f067be00700 1 Processor -- start 2024-04-24T08:08:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.847+0000 7f067be00700 1 -- start start 2024-04-24T08:08:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.848+0000 7f067be00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0674104f90 0x7f067419b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.848+0000 7f067be00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0674105940 0x7f067419ba00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.848+0000 7f067be00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067410a070 0x7f067419fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.848+0000 7f067be00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0674115050 con 0x7f067410a070 2024-04-24T08:08:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.848+0000 7f067be00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0674114ed0 con 0x7f0674105940 2024-04-24T08:08:44.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.848+0000 7f067be00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f06741151d0 con 0x7f0674104f90 2024-04-24T08:08:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.848+0000 7f0678a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0674105940 0x7f067419ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.848+0000 7f0679e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067410a070 0x7f067419fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.848+0000 7f0678a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0674105940 0x7f067419ba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:44210/0 (socket says 172.21.15.28:44210) 2024-04-24T08:08:44.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.848+0000 7f0678a00700 1 -- 172.21.15.28:0/2512342408 learned_addr learned my addr 172.21.15.28:0/2512342408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:44.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.848+0000 7f0679400700 1 --2- 172.21.15.28:0/2512342408 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0674104f90 0x7f067419b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.848+0000 7f0678a00700 1 -- 172.21.15.28:0/2512342408 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0674104f90 msgr2=0x7f067419b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.848+0000 7f0678a00700 1 --2- 172.21.15.28:0/2512342408 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0674104f90 0x7f067419b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.848+0000 7f0678a00700 1 -- 172.21.15.28:0/2512342408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067410a070 msgr2=0x7f067419fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.848+0000 7f0678a00700 1 --2- 172.21.15.28:0/2512342408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f067410a070 0x7f067419fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.848+0000 7f0678a00700 1 -- 172.21.15.28:0/2512342408 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f06741a04d0 con 0x7f0674105940 2024-04-24T08:08:44.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.849+0000 7f0678a00700 1 --2- 172.21.15.28:0/2512342408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0674105940 0x7f067419ba00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f066000e970 tx=0x7f066000ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:44.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.849+0000 7f0672000700 1 -- 172.21.15.28:0/2512342408 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0660004a30 con 0x7f0674105940 2024-04-24T08:08:44.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.849+0000 7f0672000700 1 -- 172.21.15.28:0/2512342408 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f066000ccc0 con 0x7f0674105940 2024-04-24T08:08:44.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.849+0000 7f0672000700 1 -- 172.21.15.28:0/2512342408 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0660005650 con 0x7f0674105940 2024-04-24T08:08:44.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.849+0000 7f067be00700 1 -- 172.21.15.28:0/2512342408 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f06741a07c0 con 0x7f0674105940 2024-04-24T08:08:44.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.849+0000 7f067be00700 1 -- 172.21.15.28:0/2512342408 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f06741a8040 con 0x7f0674105940 2024-04-24T08:08:44.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.850+0000 7f067be00700 1 -- 172.21.15.28:0/2512342408 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0674063280 con 0x7f0674105940 2024-04-24T08:08:44.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.852+0000 7f0672000700 1 -- 172.21.15.28:0/2512342408 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f0660004bd0 con 0x7f0674105940 2024-04-24T08:08:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.854+0000 7f0672000700 1 --2- 172.21.15.28:0/2512342408 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f065c0a3cd0 0x7f065c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.854+0000 7f0672000700 1 -- 172.21.15.28:0/2512342408 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(86..86 src has 1..86) ==== 7979+0+0 (secure 0 0 0) 0x7f06600d0af0 con 0x7f0674105940 2024-04-24T08:08:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.854+0000 7f0679400700 1 --2- 172.21.15.28:0/2512342408 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f065c0a3cd0 0x7f065c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.854+0000 7f0672000700 1 --2- 172.21.15.28:0/2512342408 >> [v2:172.21.15.28:6810/3962695585,v1:172.21.15.28:6811/3962695585] conn(0x7f065c0ad4b0 0x7f065c0af8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:44.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.854+0000 7f0672000700 1 -- 172.21.15.28:0/2512342408 --> [v2:172.21.15.28:6810/3962695585,v1:172.21.15.28:6811/3962695585] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f065c0aff80 con 0x7f065c0ad4b0 2024-04-24T08:08:44.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.854+0000 7f0672000700 1 -- 172.21.15.28:0/2512342408 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) ==== 24+0+0 (secure 0 0 0) 0x7f06600d0ea0 con 0x7f0674105940 2024-04-24T08:08:44.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.854+0000 7f0679e00700 1 --2- 172.21.15.28:0/2512342408 >> [v2:172.21.15.28:6810/3962695585,v1:172.21.15.28:6811/3962695585] conn(0x7f065c0ad4b0 0x7f065c0af8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:44.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.855+0000 7f0679400700 1 --2- 172.21.15.28:0/2512342408 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f065c0a3cd0 0x7f065c0a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f06680098e0 tx=0x7f0668005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:44.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.855+0000 7f0679e00700 1 --2- 172.21.15.28:0/2512342408 >> [v2:172.21.15.28:6810/3962695585,v1:172.21.15.28:6811/3962695585] conn(0x7f065c0ad4b0 0x7f065c0af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:44.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.856+0000 7f0672000700 1 -- 172.21.15.28:0/2512342408 <== osd.3 v2:172.21.15.28:6810/3962695585 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f065c0aff80 con 0x7f065c0ad4b0 2024-04-24T08:08:44.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.874+0000 7f067be00700 1 -- 172.21.15.28:0/2512342408 --> [v2:172.21.15.28:6810/3962695585,v1:172.21.15.28:6811/3962695585] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0674063700 con 0x7f065c0ad4b0 2024-04-24T08:08:44.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.875+0000 7f0672000700 1 -- 172.21.15.28:0/2512342408 <== osd.3 v2:172.21.15.28:6810/3962695585 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f0674063700 con 0x7f065c0ad4b0 2024-04-24T08:08:44.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.875+0000 7f067be00700 1 -- 172.21.15.28:0/2512342408 >> [v2:172.21.15.28:6810/3962695585,v1:172.21.15.28:6811/3962695585] conn(0x7f065c0ad4b0 msgr2=0x7f065c0af8d0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.875+0000 7f067be00700 1 --2- 172.21.15.28:0/2512342408 >> [v2:172.21.15.28:6810/3962695585,v1:172.21.15.28:6811/3962695585] conn(0x7f065c0ad4b0 0x7f065c0af8d0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:44.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.876+0000 7f067be00700 1 -- 172.21.15.28:0/2512342408 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f065c0a3cd0 msgr2=0x7f065c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.876+0000 7f067be00700 1 --2- 172.21.15.28:0/2512342408 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f065c0a3cd0 0x7f065c0a6190 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f06680098e0 tx=0x7f0668005960 comp rx=0 tx=0).stop 2024-04-24T08:08:44.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.876+0000 7f067be00700 1 -- 172.21.15.28:0/2512342408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0674105940 msgr2=0x7f067419ba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:44.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.876+0000 7f067be00700 1 --2- 172.21.15.28:0/2512342408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0674105940 0x7f067419ba00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f066000e970 tx=0x7f066000ec80 comp rx=0 tx=0).stop 2024-04-24T08:08:44.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.876+0000 7f0679e00700 1 -- 172.21.15.28:0/2512342408 reap_dead start 2024-04-24T08:08:44.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.876+0000 7f067be00700 1 -- 172.21.15.28:0/2512342408 shutdown_connections 2024-04-24T08:08:44.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.876+0000 7f067be00700 1 -- 172.21.15.28:0/2512342408 >> 172.21.15.28:0/2512342408 conn(0x7f0674100820 msgr2=0x7f0674102610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:44.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.876+0000 7f067be00700 1 -- 172.21.15.28:0/2512342408 shutdown_connections 2024-04-24T08:08:44.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:44.876+0000 7f067be00700 1 -- 172.21.15.28:0/2512342408 wait complete. 2024-04-24T08:08:45.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:44 smithi194 ceph-mon[32836]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:08:45.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:44 smithi028 ceph-mon[27786]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:08:45.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:44 smithi117 ceph-mon[32438]: pgmap v343: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:08:45.718 INFO:teuthology.orchestra.run.smithi028.stdout:141733920833 2024-04-24T08:08:45.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-04-24T08:08:46.336 INFO:teuthology.orchestra.run.smithi028.stdout:55834574928 2024-04-24T08:08:46.337 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-04-24T08:08:47.405 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:47 smithi028 ceph-mon[27786]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:08:47.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:47 smithi194 ceph-mon[32836]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:08:47.458 INFO:teuthology.orchestra.run.smithi028.stdout:292057776163 2024-04-24T08:08:47.458 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-04-24T08:08:47.579 INFO:teuthology.orchestra.run.smithi028.stdout:197568495669 2024-04-24T08:08:47.579 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-04-24T08:08:47.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:47 smithi117 ceph-mon[32438]: pgmap v344: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:08:47.900 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:47.927 INFO:teuthology.orchestra.run.smithi028.stdout:107374182470 2024-04-24T08:08:47.927 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-04-24T08:08:48.364 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:48.372 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:48.375 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:48.375 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:49.285 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:49 smithi028 ceph-mon[27786]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-24T08:08:49.420 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:49.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:49 smithi194 ceph-mon[32836]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-24T08:08:49.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:49 smithi117 ceph-mon[32438]: pgmap v345: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-04-24T08:08:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-24T08:08:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-04-24T08:08:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-24T08:08:50.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-24T08:08:50.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-04-24T08:08:50.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-24T08:08:50.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2024-04-24T08:08:50.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-04-24T08:08:50.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-04-24T08:08:50.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 6]}]: dispatch 2024-04-24T08:08:50.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:50 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-24T08:08:50.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:50 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-04-24T08:08:50.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:50 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-24T08:08:50.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:50 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-24T08:08:50.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:50 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-04-24T08:08:50.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:50 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-24T08:08:50.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:50 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2024-04-24T08:08:50.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:50 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-04-24T08:08:50.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:50 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-04-24T08:08:50.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:50 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 6]}]: dispatch 2024-04-24T08:08:50.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:50 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]: dispatch 2024-04-24T08:08:50.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:50 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]: dispatch 2024-04-24T08:08:50.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:50 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]: dispatch 2024-04-24T08:08:50.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:50 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-04-24T08:08:50.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:50 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]: dispatch 2024-04-24T08:08:50.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:50 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-04-24T08:08:50.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:50 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]: dispatch 2024-04-24T08:08:50.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:50 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]: dispatch 2024-04-24T08:08:50.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:50 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]: dispatch 2024-04-24T08:08:50.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:50 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 6]}]: dispatch 2024-04-24T08:08:51.249 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:51.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:51 smithi194 ceph-mon[32836]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:08:51.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-24T08:08:51.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-04-24T08:08:51.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-24T08:08:51.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-24T08:08:51.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-04-24T08:08:51.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-24T08:08:51.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2024-04-24T08:08:51.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-04-24T08:08:51.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-04-24T08:08:51.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 6]}]': finished 2024-04-24T08:08:51.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:51 smithi194 ceph-mon[32836]: osdmap e87: 12 total, 12 up, 12 in 2024-04-24T08:08:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:51 smithi028 ceph-mon[27786]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:08:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-24T08:08:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-04-24T08:08:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-24T08:08:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-24T08:08:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-04-24T08:08:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-24T08:08:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2024-04-24T08:08:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-04-24T08:08:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-04-24T08:08:51.442 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 6]}]': finished 2024-04-24T08:08:51.443 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:51 smithi028 ceph-mon[27786]: osdmap e87: 12 total, 12 up, 12 in 2024-04-24T08:08:51.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:51 smithi117 ceph-mon[32438]: pgmap v346: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:08:51.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.6", "id": [8, 11]}]': finished 2024-04-24T08:08:51.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.c", "id": [2, 4]}]': finished 2024-04-24T08:08:51.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.13", "id": [2, 11]}]': finished 2024-04-24T08:08:51.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-04-24T08:08:51.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [4, 8]}]': finished 2024-04-24T08:08:51.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-04-24T08:08:51.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 7]}]': finished 2024-04-24T08:08:51.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.21", "id": [5, 9]}]': finished 2024-04-24T08:08:51.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.23", "id": [4, 8]}]': finished 2024-04-24T08:08:51.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.28", "id": [3, 6]}]': finished 2024-04-24T08:08:51.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:51 smithi117 ceph-mon[32438]: osdmap e87: 12 total, 12 up, 12 in 2024-04-24T08:08:52.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:52 smithi194 ceph-mon[32836]: osdmap e88: 12 total, 12 up, 12 in 2024-04-24T08:08:52.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:52 smithi028 ceph-mon[27786]: osdmap e88: 12 total, 12 up, 12 in 2024-04-24T08:08:52.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:52 smithi117 ceph-mon[32438]: osdmap e88: 12 total, 12 up, 12 in 2024-04-24T08:08:53.507 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:53 smithi194 ceph-mon[32836]: pgmap v349: 97 pgs: 3 peering, 94 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T08:08:53.508 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:53 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:08:53.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:53 smithi028 ceph-mon[27786]: pgmap v349: 97 pgs: 3 peering, 94 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T08:08:53.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:53 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:08:53.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:53 smithi117 ceph-mon[32438]: pgmap v349: 97 pgs: 3 peering, 94 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T08:08:53.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:53 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:08:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.308+0000 7fd24d000700 1 -- 172.21.15.28:0/3749646971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd248109d30 msgr2=0x7fd24810dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.308+0000 7fd24d000700 1 --2- 172.21.15.28:0/3749646971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd248109d30 0x7fd24810dd80 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd234009990 tx=0x7fd234009ca0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.308+0000 7fd24d000700 1 -- 172.21.15.28:0/3749646971 shutdown_connections 2024-04-24T08:08:53.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.308+0000 7fd24d000700 1 --2- 172.21.15.28:0/3749646971 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd24810e460 0x7fd248116030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.308+0000 7fd24d000700 1 --2- 172.21.15.28:0/3749646971 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd248109380 0x7fd248109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.308+0000 7fd24d000700 1 --2- 172.21.15.28:0/3749646971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd248109d30 0x7fd24810dd80 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.308+0000 7fd24d000700 1 -- 172.21.15.28:0/3749646971 >> 172.21.15.28:0/3749646971 conn(0x7fd2480762c0 msgr2=0x7fd248107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.308+0000 7fd24d000700 1 -- 172.21.15.28:0/3749646971 shutdown_connections 2024-04-24T08:08:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.308+0000 7fd24d000700 1 -- 172.21.15.28:0/3749646971 wait complete. 2024-04-24T08:08:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.310+0000 7fd24d000700 1 Processor -- start 2024-04-24T08:08:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.310+0000 7fd24d000700 1 -- start start 2024-04-24T08:08:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.311+0000 7fd24d000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd248109380 0x7fd2481a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.311+0000 7fd24d000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd248109d30 0x7fd2481a5b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.311+0000 7fd24d000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd24810e460 0x7fd24819f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.311+0000 7fd24d000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd248119420 con 0x7fd24810e460 2024-04-24T08:08:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.311+0000 7fd24d000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd2481192a0 con 0x7fd248109380 2024-04-24T08:08:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.311+0000 7fd24d000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd2481195a0 con 0x7fd248109d30 2024-04-24T08:08:53.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.311+0000 7fd245e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd248109380 0x7fd2481a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.311+0000 7fd245e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd248109380 0x7fd2481a55c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:55794/0 (socket says 172.21.15.28:55794) 2024-04-24T08:08:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.311+0000 7fd245e00700 1 -- 172.21.15.28:0/2949073690 learned_addr learned my addr 172.21.15.28:0/2949073690 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.311+0000 7fd245400700 1 --2- 172.21.15.28:0/2949073690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd248109d30 0x7fd2481a5b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.312+0000 7fd246800700 1 --2- 172.21.15.28:0/2949073690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd24810e460 0x7fd24819f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.312+0000 7fd245e00700 1 -- 172.21.15.28:0/2949073690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd248109d30 msgr2=0x7fd2481a5b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.312+0000 7fd245e00700 1 --2- 172.21.15.28:0/2949073690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd248109d30 0x7fd2481a5b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.312+0000 7fd245e00700 1 -- 172.21.15.28:0/2949073690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd24810e460 msgr2=0x7fd24819f6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.312+0000 7fd245e00700 1 --2- 172.21.15.28:0/2949073690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd24810e460 0x7fd24819f6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.312+0000 7fd245e00700 1 -- 172.21.15.28:0/2949073690 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd24819ffc0 con 0x7fd248109380 2024-04-24T08:08:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.312+0000 7fd245400700 1 --2- 172.21.15.28:0/2949073690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd248109d30 0x7fd2481a5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:08:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.312+0000 7fd245e00700 1 --2- 172.21.15.28:0/2949073690 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd248109380 0x7fd2481a55c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd23800e9a0 tx=0x7fd23800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:53.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.312+0000 7fd23ea00700 1 -- 172.21.15.28:0/2949073690 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd23800cbd0 con 0x7fd248109380 2024-04-24T08:08:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.312+0000 7fd24d000700 1 -- 172.21.15.28:0/2949073690 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2481a02b0 con 0x7fd248109380 2024-04-24T08:08:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.312+0000 7fd23ea00700 1 -- 172.21.15.28:0/2949073690 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd23800cd70 con 0x7fd248109380 2024-04-24T08:08:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.312+0000 7fd23ea00700 1 -- 172.21.15.28:0/2949073690 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd238018750 con 0x7fd248109380 2024-04-24T08:08:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.313+0000 7fd24d000700 1 -- 172.21.15.28:0/2949073690 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2481ac540 con 0x7fd248109380 2024-04-24T08:08:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.314+0000 7fd24d000700 1 -- 172.21.15.28:0/2949073690 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd24810a860 con 0x7fd248109380 2024-04-24T08:08:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.316+0000 7fd23ea00700 1 -- 172.21.15.28:0/2949073690 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd238010430 con 0x7fd248109380 2024-04-24T08:08:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.317+0000 7fd23ea00700 1 --2- 172.21.15.28:0/2949073690 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd22c0a3cd0 0x7fd22c0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.318+0000 7fd23ea00700 1 -- 172.21.15.28:0/2949073690 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fd2380d0890 con 0x7fd248109380 2024-04-24T08:08:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.318+0000 7fd245400700 1 --2- 172.21.15.28:0/2949073690 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd22c0a3cd0 0x7fd22c0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.318+0000 7fd245400700 1 --2- 172.21.15.28:0/2949073690 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd22c0a3cd0 0x7fd22c0a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd23400b2e0 tx=0x7fd234005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.324+0000 7fd23ea00700 1 -- 172.21.15.28:0/2949073690 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd23809d090 con 0x7fd248109380 2024-04-24T08:08:53.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.564+0000 7fd24d000700 1 -- 172.21.15.28:0/2949073690 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7fd2481a0bc0 con 0x7fd248109380 2024-04-24T08:08:53.795 INFO:teuthology.orchestra.run.smithi028.stdout:77309411406 2024-04-24T08:08:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.565+0000 7fd23ea00700 1 -- 172.21.15.28:0/2949073690 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7fd23809c990 con 0x7fd248109380 2024-04-24T08:08:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.568+0000 7fd24d000700 1 -- 172.21.15.28:0/2949073690 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd22c0a3cd0 msgr2=0x7fd22c0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.568+0000 7fd24d000700 1 --2- 172.21.15.28:0/2949073690 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd22c0a3cd0 0x7fd22c0a6190 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fd23400b2e0 tx=0x7fd234005960 comp rx=0 tx=0).stop 2024-04-24T08:08:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.568+0000 7fd24d000700 1 -- 172.21.15.28:0/2949073690 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd248109380 msgr2=0x7fd2481a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.568+0000 7fd24d000700 1 --2- 172.21.15.28:0/2949073690 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd248109380 0x7fd2481a55c0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fd23800e9a0 tx=0x7fd23800ecb0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.568+0000 7fd24d000700 1 -- 172.21.15.28:0/2949073690 shutdown_connections 2024-04-24T08:08:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.568+0000 7fd24d000700 1 --2- 172.21.15.28:0/2949073690 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd22c0a3cd0 0x7fd22c0a6190 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.568+0000 7fd24d000700 1 --2- 172.21.15.28:0/2949073690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd24810e460 0x7fd24819f6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.568+0000 7fd24d000700 1 --2- 172.21.15.28:0/2949073690 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd248109380 0x7fd2481a55c0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.568+0000 7fd24d000700 1 --2- 172.21.15.28:0/2949073690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd248109d30 0x7fd2481a5b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.568+0000 7fd24d000700 1 -- 172.21.15.28:0/2949073690 >> 172.21.15.28:0/2949073690 conn(0x7fd2480762c0 msgr2=0x7fd248076880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.568+0000 7fd24d000700 1 -- 172.21.15.28:0/2949073690 shutdown_connections 2024-04-24T08:08:53.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:53.568+0000 7fd24d000700 1 -- 172.21.15.28:0/2949073690 wait complete. 2024-04-24T08:08:54.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:54 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2949073690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-24T08:08:54.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:54 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:08:54.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:54 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:08:54.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:54 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:08:54.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:54 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2949073690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-24T08:08:54.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:54 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:08:54.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:54 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:08:54.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:54 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:08:54.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:54 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2949073690' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-04-24T08:08:54.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:54 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:08:54.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:54 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:08:54.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:54 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:08:54.848 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:55.081 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:55.083 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:55.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:55 smithi028 ceph-mon[27786]: pgmap v350: 97 pgs: 3 peering, 94 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 72 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T08:08:55.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:55 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:08:55.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:55 smithi117 ceph-mon[32438]: pgmap v350: 97 pgs: 3 peering, 94 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 72 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T08:08:55.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:55 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:08:55.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:55 smithi194 ceph-mon[32836]: pgmap v350: 97 pgs: 3 peering, 94 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s; 72 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T08:08:55.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:55 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:08:56.426 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:57.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:57 smithi028 ceph-mon[27786]: pgmap v351: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 76 B/s, 2 keys/s, 0 objects/s recovering 2024-04-24T08:08:57.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:57 smithi117 ceph-mon[32438]: pgmap v351: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 76 B/s, 2 keys/s, 0 objects/s recovering 2024-04-24T08:08:57.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:57 smithi194 ceph-mon[32836]: pgmap v351: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s; 76 B/s, 2 keys/s, 0 objects/s recovering 2024-04-24T08:08:58.156 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:08:59.078 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411403 got 77309411406 for osd.2 2024-04-24T08:08:59.079 DEBUG:teuthology.parallel:result is None 2024-04-24T08:08:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.295+0000 7f76df000700 1 -- 172.21.15.28:0/3280699602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d8108b90 msgr2=0x7f76d8108f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.295+0000 7f76df000700 1 --2- 172.21.15.28:0/3280699602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d8108b90 0x7f76d8108f70 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009990 tx=0x7f76c8009ca0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.295+0000 7f76df000700 1 -- 172.21.15.28:0/3280699602 shutdown_connections 2024-04-24T08:08:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.295+0000 7f76df000700 1 --2- 172.21.15.28:0/3280699602 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76d8103500 0x7f76d810f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.295+0000 7f76df000700 1 --2- 172.21.15.28:0/3280699602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d8108b90 0x7f76d8108f70 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.295+0000 7f76df000700 1 --2- 172.21.15.28:0/3280699602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76d8102b40 0x7f76d8102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.296+0000 7f76df000700 1 -- 172.21.15.28:0/3280699602 >> 172.21.15.28:0/3280699602 conn(0x7f76d8079c50 msgr2=0x7f76d807a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.296+0000 7f76df000700 1 -- 172.21.15.28:0/3280699602 shutdown_connections 2024-04-24T08:08:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.296+0000 7f76df000700 1 -- 172.21.15.28:0/3280699602 wait complete. 2024-04-24T08:08:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.297+0000 7f76df000700 1 Processor -- start 2024-04-24T08:08:59.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.297+0000 7f76df000700 1 -- start start 2024-04-24T08:08:59.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.298+0000 7f76df000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76d8102b40 0x7f76d81992c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.298+0000 7f76df000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d8103500 0x7f76d8199800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.298+0000 7f76df000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76d8108b90 0x7f76d819db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.298+0000 7f76df000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f76d8112e50 con 0x7f76d8102b40 2024-04-24T08:08:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.298+0000 7f76df000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f76d8112cd0 con 0x7f76d8103500 2024-04-24T08:08:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.298+0000 7f76df000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f76d8112fd0 con 0x7f76d8108b90 2024-04-24T08:08:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.298+0000 7f76d7400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d8103500 0x7f76d8199800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.298+0000 7f76d7e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76d8102b40 0x7f76d81992c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.298+0000 7f76d7400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d8103500 0x7f76d8199800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:55826/0 (socket says 172.21.15.28:55826) 2024-04-24T08:08:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.298+0000 7f76d7400700 1 -- 172.21.15.28:0/15832140 learned_addr learned my addr 172.21.15.28:0/15832140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:08:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.298+0000 7f76dd000700 1 --2- 172.21.15.28:0/15832140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76d8108b90 0x7f76d819db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.298+0000 7f76d7400700 1 -- 172.21.15.28:0/15832140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76d8108b90 msgr2=0x7f76d819db90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.298+0000 7f76d7400700 1 --2- 172.21.15.28:0/15832140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76d8108b90 0x7f76d819db90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.298+0000 7f76d7400700 1 -- 172.21.15.28:0/15832140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76d8102b40 msgr2=0x7f76d81992c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.298+0000 7f76d7400700 1 --2- 172.21.15.28:0/15832140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76d8102b40 0x7f76d81992c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.298+0000 7f76d7400700 1 -- 172.21.15.28:0/15832140 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d819e2d0 con 0x7f76d8103500 2024-04-24T08:08:59.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.299+0000 7f76d7400700 1 --2- 172.21.15.28:0/15832140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d8103500 0x7f76d8199800 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f76c400e9a0 tx=0x7f76c400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:59.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.299+0000 7f76d4c00700 1 -- 172.21.15.28:0/15832140 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c400ca10 con 0x7f76d8103500 2024-04-24T08:08:59.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.299+0000 7f76d4c00700 1 -- 172.21.15.28:0/15832140 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76c400cbb0 con 0x7f76d8103500 2024-04-24T08:08:59.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.299+0000 7f76df000700 1 -- 172.21.15.28:0/15832140 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d819e5c0 con 0x7f76d8103500 2024-04-24T08:08:59.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.299+0000 7f76d4c00700 1 -- 172.21.15.28:0/15832140 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c4018680 con 0x7f76d8103500 2024-04-24T08:08:59.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.299+0000 7f76df000700 1 -- 172.21.15.28:0/15832140 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d81a5e40 con 0x7f76d8103500 2024-04-24T08:08:59.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.300+0000 7f76df000700 1 -- 172.21.15.28:0/15832140 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76d80697e0 con 0x7f76d8103500 2024-04-24T08:08:59.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.302+0000 7f76d4c00700 1 -- 172.21.15.28:0/15832140 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f76c4010430 con 0x7f76d8103500 2024-04-24T08:08:59.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.304+0000 7f76d4c00700 1 --2- 172.21.15.28:0/15832140 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f76bc0a3d50 0x7f76bc0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:08:59.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.304+0000 7f76d4c00700 1 -- 172.21.15.28:0/15832140 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f76c40d0790 con 0x7f76d8103500 2024-04-24T08:08:59.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.304+0000 7f76d7e00700 1 --2- 172.21.15.28:0/15832140 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f76bc0a3d50 0x7f76bc0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:08:59.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.305+0000 7f76d7e00700 1 --2- 172.21.15.28:0/15832140 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f76bc0a3d50 0x7f76bc0a6210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f76c8006010 tx=0x7f76c800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:08:59.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.309+0000 7f76d4c00700 1 -- 172.21.15.28:0/15832140 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f76c409cf90 con 0x7f76d8103500 2024-04-24T08:08:59.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:08:59 smithi028 ceph-mon[27786]: pgmap v352: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 76 B/s, 2 keys/s, 0 objects/s recovering 2024-04-24T08:08:59.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.541+0000 7f76df000700 1 -- 172.21.15.28:0/15832140 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f76d802dfe0 con 0x7f76d8103500 2024-04-24T08:08:59.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.541+0000 7f76d4c00700 1 -- 172.21.15.28:0/15832140 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f76c4085e20 con 0x7f76d8103500 2024-04-24T08:08:59.543 INFO:teuthology.orchestra.run.smithi028.stdout:55834574932 2024-04-24T08:08:59.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.544+0000 7f76df000700 1 -- 172.21.15.28:0/15832140 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f76bc0a3d50 msgr2=0x7f76bc0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.544+0000 7f76df000700 1 --2- 172.21.15.28:0/15832140 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f76bc0a3d50 0x7f76bc0a6210 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f76c8006010 tx=0x7f76c800b260 comp rx=0 tx=0).stop 2024-04-24T08:08:59.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.544+0000 7f76df000700 1 -- 172.21.15.28:0/15832140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d8103500 msgr2=0x7f76d8199800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:08:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.544+0000 7f76df000700 1 --2- 172.21.15.28:0/15832140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d8103500 0x7f76d8199800 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f76c400e9a0 tx=0x7f76c400ecb0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.545+0000 7f76df000700 1 -- 172.21.15.28:0/15832140 shutdown_connections 2024-04-24T08:08:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.545+0000 7f76df000700 1 --2- 172.21.15.28:0/15832140 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f76bc0a3d50 0x7f76bc0a6210 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.545+0000 7f76df000700 1 --2- 172.21.15.28:0/15832140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76d8108b90 0x7f76d819db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.545+0000 7f76df000700 1 --2- 172.21.15.28:0/15832140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f76d8102b40 0x7f76d81992c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.545+0000 7f76df000700 1 --2- 172.21.15.28:0/15832140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76d8103500 0x7f76d8199800 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:08:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.545+0000 7f76df000700 1 -- 172.21.15.28:0/15832140 >> 172.21.15.28:0/15832140 conn(0x7f76d8079c50 msgr2=0x7f76d8101800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:08:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.545+0000 7f76df000700 1 -- 172.21.15.28:0/15832140 shutdown_connections 2024-04-24T08:08:59.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:08:59.545+0000 7f76df000700 1 -- 172.21.15.28:0/15832140 wait complete. 2024-04-24T08:08:59.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:08:59 smithi117 ceph-mon[32438]: pgmap v352: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 76 B/s, 2 keys/s, 0 objects/s recovering 2024-04-24T08:08:59.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:08:59 smithi194 ceph-mon[32836]: pgmap v352: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s; 76 B/s, 2 keys/s, 0 objects/s recovering 2024-04-24T08:09:00.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:00 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/15832140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-24T08:09:00.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:00 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/15832140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-24T08:09:00.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:00 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/15832140' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-04-24T08:09:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:01 smithi028 ceph-mon[27786]: pgmap v353: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 62 B/s, 1 keys/s, 0 objects/s recovering 2024-04-24T08:09:01.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:01 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3966432420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-24T08:09:01.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.781+0000 7f2be3a00700 1 -- 172.21.15.28:0/2642416335 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2bdc075640 msgr2=0x7f2bdc10efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:01.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.781+0000 7f2be3a00700 1 --2- 172.21.15.28:0/2642416335 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2bdc075640 0x7f2bdc10efb0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f2bc40098f0 tx=0x7f2bc4009c00 comp rx=0 tx=0).stop 2024-04-24T08:09:01.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.782+0000 7f2be3a00700 1 -- 172.21.15.28:0/2642416335 shutdown_connections 2024-04-24T08:09:01.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.782+0000 7f2be3a00700 1 --2- 172.21.15.28:0/2642416335 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc10f4f0 0x7f2bdc1118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:01.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.782+0000 7f2be3a00700 1 --2- 172.21.15.28:0/2642416335 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bdc074d20 0x7f2bdc075100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:01.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.782+0000 7f2be3a00700 1 --2- 172.21.15.28:0/2642416335 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2bdc075640 0x7f2bdc10efb0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.782+0000 7f2be3a00700 1 -- 172.21.15.28:0/2642416335 >> 172.21.15.28:0/2642416335 conn(0x7f2bdc079c50 msgr2=0x7f2bdc07a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.782+0000 7f2be3a00700 1 -- 172.21.15.28:0/2642416335 shutdown_connections 2024-04-24T08:09:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.782+0000 7f2be3a00700 1 -- 172.21.15.28:0/2642416335 wait complete. 2024-04-24T08:09:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.783+0000 7f2be3a00700 1 Processor -- start 2024-04-24T08:09:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.783+0000 7f2be3a00700 1 -- start start 2024-04-24T08:09:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.783+0000 7f2be3a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2bdc074d20 0x7f2bdc19b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.783+0000 7f2be3a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc075640 0x7f2bdc19ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.783+0000 7f2be3a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bdc10f4f0 0x7f2bdc19fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.783+0000 7f2be3a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2bdc115020 con 0x7f2bdc075640 2024-04-24T08:09:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.783+0000 7f2be3a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2bdc114ea0 con 0x7f2bdc10f4f0 2024-04-24T08:09:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.783+0000 7f2be3a00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2bdc1151a0 con 0x7f2bdc074d20 2024-04-24T08:09:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.784+0000 7f2bd3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc075640 0x7f2bdc19ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:01.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.784+0000 7f2bd3e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc075640 0x7f2bdc19ba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55242/0 (socket says 172.21.15.28:55242) 2024-04-24T08:09:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.784+0000 7f2be1000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2bdc074d20 0x7f2bdc19b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.784+0000 7f2bd3e00700 1 -- 172.21.15.28:0/3966432420 learned_addr learned my addr 172.21.15.28:0/3966432420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.784+0000 7f2be1a00700 1 --2- 172.21.15.28:0/3966432420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bdc10f4f0 0x7f2bdc19fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.784+0000 7f2bd3e00700 1 -- 172.21.15.28:0/3966432420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2bdc074d20 msgr2=0x7f2bdc19b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.784+0000 7f2bd3e00700 1 --2- 172.21.15.28:0/3966432420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2bdc074d20 0x7f2bdc19b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.784+0000 7f2bd3e00700 1 -- 172.21.15.28:0/3966432420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bdc10f4f0 msgr2=0x7f2bdc19fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.784+0000 7f2bd3e00700 1 --2- 172.21.15.28:0/3966432420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bdc10f4f0 0x7f2bdc19fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.784+0000 7f2bd3e00700 1 -- 172.21.15.28:0/3966432420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2bdc1a04f0 con 0x7f2bdc075640 2024-04-24T08:09:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.785+0000 7f2bd3e00700 1 --2- 172.21.15.28:0/3966432420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc075640 0x7f2bdc19ba70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4005950 tx=0x7f2bc400fb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.785+0000 7f2bda000700 1 -- 172.21.15.28:0/3966432420 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc40051f0 con 0x7f2bdc075640 2024-04-24T08:09:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.785+0000 7f2be3a00700 1 -- 172.21.15.28:0/3966432420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2bdc1a0780 con 0x7f2bdc075640 2024-04-24T08:09:01.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.785+0000 7f2bda000700 1 -- 172.21.15.28:0/3966432420 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2bc400b560 con 0x7f2bdc075640 2024-04-24T08:09:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.785+0000 7f2bda000700 1 -- 172.21.15.28:0/3966432420 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2bc4003a40 con 0x7f2bdc075640 2024-04-24T08:09:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.785+0000 7f2be3a00700 1 -- 172.21.15.28:0/3966432420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2bdc1a8060 con 0x7f2bdc075640 2024-04-24T08:09:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.786+0000 7f2be3a00700 1 -- 172.21.15.28:0/3966432420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2bdc0697e0 con 0x7f2bdc075640 2024-04-24T08:09:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.788+0000 7f2bda000700 1 -- 172.21.15.28:0/3966432420 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f2bc400ba80 con 0x7f2bdc075640 2024-04-24T08:09:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.789+0000 7f2bda000700 1 --2- 172.21.15.28:0/3966432420 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2bbc0a3cd0 0x7f2bbc0a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.790+0000 7f2bda000700 1 -- 172.21.15.28:0/3966432420 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f2bc40d1590 con 0x7f2bdc075640 2024-04-24T08:09:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.790+0000 7f2be1000700 1 --2- 172.21.15.28:0/3966432420 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2bbc0a3cd0 0x7f2bbc0a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.791+0000 7f2be1000700 1 --2- 172.21.15.28:0/3966432420 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2bbc0a3cd0 0x7f2bbc0a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc005fd0 tx=0x7f2bcc009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:01.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:00.795+0000 7f2bda000700 1 -- 172.21.15.28:0/3966432420 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2bc4013030 con 0x7f2bdc075640 2024-04-24T08:09:01.509 INFO:teuthology.orchestra.run.smithi028.stdout:257698037804 2024-04-24T08:09:01.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:01.027+0000 7f2be3a00700 1 -- 172.21.15.28:0/3966432420 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7f2bdc02dfe0 con 0x7f2bdc075640 2024-04-24T08:09:01.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:01.028+0000 7f2bda000700 1 -- 172.21.15.28:0/3966432420 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2bc4021380 con 0x7f2bdc075640 2024-04-24T08:09:01.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:01.031+0000 7f2be3a00700 1 -- 172.21.15.28:0/3966432420 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2bbc0a3cd0 msgr2=0x7f2bbc0a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:01.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:01.031+0000 7f2be3a00700 1 --2- 172.21.15.28:0/3966432420 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2bbc0a3cd0 0x7f2bbc0a6190 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f2bcc005fd0 tx=0x7f2bcc009310 comp rx=0 tx=0).stop 2024-04-24T08:09:01.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:01.031+0000 7f2be3a00700 1 -- 172.21.15.28:0/3966432420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc075640 msgr2=0x7f2bdc19ba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:01.031+0000 7f2be3a00700 1 --2- 172.21.15.28:0/3966432420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc075640 0x7f2bdc19ba70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2bc4005950 tx=0x7f2bc400fb20 comp rx=0 tx=0).stop 2024-04-24T08:09:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:01.031+0000 7f2be3a00700 1 -- 172.21.15.28:0/3966432420 shutdown_connections 2024-04-24T08:09:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:01.031+0000 7f2be3a00700 1 --2- 172.21.15.28:0/3966432420 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2bbc0a3cd0 0x7f2bbc0a6190 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:01.031+0000 7f2be3a00700 1 --2- 172.21.15.28:0/3966432420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2bdc10f4f0 0x7f2bdc19fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:01.031+0000 7f2be3a00700 1 --2- 172.21.15.28:0/3966432420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2bdc074d20 0x7f2bdc19b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:01.032+0000 7f2be3a00700 1 --2- 172.21.15.28:0/3966432420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2bdc075640 0x7f2bdc19ba70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:01.032+0000 7f2be3a00700 1 -- 172.21.15.28:0/3966432420 >> 172.21.15.28:0/3966432420 conn(0x7f2bdc079c50 msgr2=0x7f2bdc0ffd00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:01.032+0000 7f2be3a00700 1 -- 172.21.15.28:0/3966432420 shutdown_connections 2024-04-24T08:09:01.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:01.032+0000 7f2be3a00700 1 -- 172.21.15.28:0/3966432420 wait complete. 2024-04-24T08:09:01.591 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:01 smithi194 ceph-mon[32836]: pgmap v353: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 62 B/s, 1 keys/s, 0 objects/s recovering 2024-04-24T08:09:01.592 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:01 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3966432420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-24T08:09:01.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:01 smithi117 ceph-mon[32438]: pgmap v353: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s; 62 B/s, 1 keys/s, 0 objects/s recovering 2024-04-24T08:09:01.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:01 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3966432420' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-04-24T08:09:03.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.209+0000 7fe0d1a00700 1 -- 172.21.15.28:0/422765320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0cc107ab0 msgr2=0x7fe0cc107e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:03.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.209+0000 7fe0d1a00700 1 --2- 172.21.15.28:0/422765320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0cc107ab0 0x7fe0cc107e90 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc0098f0 tx=0x7fe0bc009c00 comp rx=0 tx=0).stop 2024-04-24T08:09:03.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.209+0000 7fe0d1a00700 1 -- 172.21.15.28:0/422765320 shutdown_connections 2024-04-24T08:09:03.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.209+0000 7fe0d1a00700 1 --2- 172.21.15.28:0/422765320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0cc103540 0x7fe0cc10fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.209+0000 7fe0d1a00700 1 --2- 172.21.15.28:0/422765320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0cc107ab0 0x7fe0cc107e90 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.209+0000 7fe0d1a00700 1 --2- 172.21.15.28:0/422765320 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0cc102ba0 0x7fe0cc103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.209+0000 7fe0d1a00700 1 -- 172.21.15.28:0/422765320 >> 172.21.15.28:0/422765320 conn(0x7fe0cc079c50 msgr2=0x7fe0cc07a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:03.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.210+0000 7fe0d1a00700 1 -- 172.21.15.28:0/422765320 shutdown_connections 2024-04-24T08:09:03.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.210+0000 7fe0d1a00700 1 -- 172.21.15.28:0/422765320 wait complete. 2024-04-24T08:09:03.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.211+0000 7fe0d1a00700 1 Processor -- start 2024-04-24T08:09:03.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.211+0000 7fe0d1a00700 1 -- start start 2024-04-24T08:09:03.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.211+0000 7fe0d1a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0cc102ba0 0x7fe0cc074df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:03.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.211+0000 7fe0d1a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0cc103540 0x7fe0cc075330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:03.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.211+0000 7fe0d1a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0cc107ab0 0x7fe0cc075870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:03.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.211+0000 7fe0d1a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc112e80 con 0x7fe0cc102ba0 2024-04-24T08:09:03.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.211+0000 7fe0d1a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc112d00 con 0x7fe0cc103540 2024-04-24T08:09:03.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.211+0000 7fe0d1a00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe0cc113000 con 0x7fe0cc107ab0 2024-04-24T08:09:03.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.212+0000 7fe0cb400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0cc107ab0 0x7fe0cc075870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:03.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.212+0000 7fe0ca000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0cc103540 0x7fe0cc075330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:03.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.212+0000 7fe0ca000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0cc103540 0x7fe0cc075330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41886/0 (socket says 172.21.15.28:41886) 2024-04-24T08:09:03.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.212+0000 7fe0cb400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0cc107ab0 0x7fe0cc075870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:49346/0 (socket says 172.21.15.28:49346) 2024-04-24T08:09:03.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.212+0000 7fe0ca000700 1 -- 172.21.15.28:0/591488094 learned_addr learned my addr 172.21.15.28:0/591488094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:03.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.212+0000 7fe0caa00700 1 --2- 172.21.15.28:0/591488094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0cc102ba0 0x7fe0cc074df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:03.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.212+0000 7fe0cb400700 1 -- 172.21.15.28:0/591488094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0cc103540 msgr2=0x7fe0cc075330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:03.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.212+0000 7fe0cb400700 1 --2- 172.21.15.28:0/591488094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0cc103540 0x7fe0cc075330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.212+0000 7fe0cb400700 1 -- 172.21.15.28:0/591488094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0cc102ba0 msgr2=0x7fe0cc074df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:03.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.212+0000 7fe0cb400700 1 --2- 172.21.15.28:0/591488094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0cc102ba0 0x7fe0cc074df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.212+0000 7fe0cb400700 1 -- 172.21.15.28:0/591488094 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0cc1a78a0 con 0x7fe0cc107ab0 2024-04-24T08:09:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.212+0000 7fe0caa00700 1 --2- 172.21.15.28:0/591488094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0cc102ba0 0x7fe0cc074df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:09:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.213+0000 7fe0cb400700 1 --2- 172.21.15.28:0/591488094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0cc107ab0 0x7fe0cc075870 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe0c400efd0 tx=0x7fe0c400c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.213+0000 7fe0c3400700 1 -- 172.21.15.28:0/591488094 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4018020 con 0x7fe0cc107ab0 2024-04-24T08:09:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.213+0000 7fe0c3400700 1 -- 172.21.15.28:0/591488094 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0c4003a00 con 0x7fe0cc107ab0 2024-04-24T08:09:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.213+0000 7fe0d1a00700 1 -- 172.21.15.28:0/591488094 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0cc1a7b90 con 0x7fe0cc107ab0 2024-04-24T08:09:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.213+0000 7fe0c3400700 1 -- 172.21.15.28:0/591488094 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0c4004710 con 0x7fe0cc107ab0 2024-04-24T08:09:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.213+0000 7fe0d1a00700 1 -- 172.21.15.28:0/591488094 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0cc1a80d0 con 0x7fe0cc107ab0 2024-04-24T08:09:03.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.214+0000 7fe0d1a00700 1 -- 172.21.15.28:0/591488094 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe0cc0697e0 con 0x7fe0cc107ab0 2024-04-24T08:09:03.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.216+0000 7fe0c3400700 1 -- 172.21.15.28:0/591488094 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fe0c4012050 con 0x7fe0cc107ab0 2024-04-24T08:09:03.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.218+0000 7fe0c3400700 1 --2- 172.21.15.28:0/591488094 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe0b00a3d50 0x7fe0b00a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:03.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.218+0000 7fe0c3400700 1 -- 172.21.15.28:0/591488094 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe0c40d00b0 con 0x7fe0cc107ab0 2024-04-24T08:09:03.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.218+0000 7fe0caa00700 1 --2- 172.21.15.28:0/591488094 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe0b00a3d50 0x7fe0b00a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:03.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.219+0000 7fe0caa00700 1 --2- 172.21.15.28:0/591488094 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe0b00a3d50 0x7fe0b00a6210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc009810 tx=0x7fe0bc019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:03.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.222+0000 7fe0c3400700 1 -- 172.21.15.28:0/591488094 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe0c4022020 con 0x7fe0cc107ab0 2024-04-24T08:09:03.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.457+0000 7fe0d1a00700 1 -- 172.21.15.28:0/591488094 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7fe0cc10b0e0 con 0x7fe0cc107ab0 2024-04-24T08:09:03.459 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:03 smithi028 ceph-mon[27786]: pgmap v354: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 56 B/s, 1 keys/s, 0 objects/s recovering 2024-04-24T08:09:03.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.458+0000 7fe0c3400700 1 -- 172.21.15.28:0/591488094 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fe0c4010040 con 0x7fe0cc107ab0 2024-04-24T08:09:03.459 INFO:teuthology.orchestra.run.smithi028.stdout:141733920836 2024-04-24T08:09:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.461+0000 7fe0d1a00700 1 -- 172.21.15.28:0/591488094 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe0b00a3d50 msgr2=0x7fe0b00a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.461+0000 7fe0d1a00700 1 --2- 172.21.15.28:0/591488094 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe0b00a3d50 0x7fe0b00a6210 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fe0bc009810 tx=0x7fe0bc019040 comp rx=0 tx=0).stop 2024-04-24T08:09:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.461+0000 7fe0d1a00700 1 -- 172.21.15.28:0/591488094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0cc107ab0 msgr2=0x7fe0cc075870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.461+0000 7fe0d1a00700 1 --2- 172.21.15.28:0/591488094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0cc107ab0 0x7fe0cc075870 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fe0c400efd0 tx=0x7fe0c400c3a0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.461+0000 7fe0d1a00700 1 -- 172.21.15.28:0/591488094 shutdown_connections 2024-04-24T08:09:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.461+0000 7fe0d1a00700 1 --2- 172.21.15.28:0/591488094 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe0b00a3d50 0x7fe0b00a6210 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.461+0000 7fe0d1a00700 1 --2- 172.21.15.28:0/591488094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe0cc107ab0 0x7fe0cc075870 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.461+0000 7fe0d1a00700 1 --2- 172.21.15.28:0/591488094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe0cc102ba0 0x7fe0cc074df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.461+0000 7fe0d1a00700 1 --2- 172.21.15.28:0/591488094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0cc103540 0x7fe0cc075330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.461+0000 7fe0d1a00700 1 -- 172.21.15.28:0/591488094 >> 172.21.15.28:0/591488094 conn(0x7fe0cc079c50 msgr2=0x7fe0cc100d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.462+0000 7fe0d1a00700 1 -- 172.21.15.28:0/591488094 shutdown_connections 2024-04-24T08:09:03.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:03.462+0000 7fe0d1a00700 1 -- 172.21.15.28:0/591488094 wait complete. 2024-04-24T08:09:03.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:03 smithi117 ceph-mon[32438]: pgmap v354: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 56 B/s, 1 keys/s, 0 objects/s recovering 2024-04-24T08:09:03.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:03 smithi194 ceph-mon[32836]: pgmap v354: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 56 B/s, 1 keys/s, 0 objects/s recovering 2024-04-24T08:09:04.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:04 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/591488094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-24T08:09:04.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:04 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/591488094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-24T08:09:04.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:04 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/591488094' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-04-24T08:09:04.709 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037799 got 257698037804 for osd.8 2024-04-24T08:09:04.709 DEBUG:teuthology.parallel:result is None 2024-04-24T08:09:05.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:05 smithi028 ceph-mon[27786]: pgmap v355: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 51 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T08:09:05.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:05 smithi117 ceph-mon[32438]: pgmap v355: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 51 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T08:09:05.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:05 smithi194 ceph-mon[32836]: pgmap v355: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s; 51 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T08:09:06.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.225+0000 7fa617e00700 1 -- 172.21.15.28:0/3466182092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa61006b1a0 msgr2=0x7fa61010f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.225+0000 7fa617e00700 1 --2- 172.21.15.28:0/3466182092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa61006b1a0 0x7fa61010f9b0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fa608009f90 tx=0x7fa60800b330 comp rx=0 tx=0).stop 2024-04-24T08:09:06.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.225+0000 7fa617e00700 1 -- 172.21.15.28:0/3466182092 shutdown_connections 2024-04-24T08:09:06.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.225+0000 7fa617e00700 1 --2- 172.21.15.28:0/3466182092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa61006b1a0 0x7fa61010f9b0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.225+0000 7fa617e00700 1 --2- 172.21.15.28:0/3466182092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa610108b70 0x7fa610108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.225+0000 7fa617e00700 1 --2- 172.21.15.28:0/3466182092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa61006a800 0x7fa61006ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.225+0000 7fa617e00700 1 -- 172.21.15.28:0/3466182092 >> 172.21.15.28:0/3466182092 conn(0x7fa6100789a0 msgr2=0x7fa610078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:06.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.225+0000 7fa617e00700 1 -- 172.21.15.28:0/3466182092 shutdown_connections 2024-04-24T08:09:06.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.225+0000 7fa617e00700 1 -- 172.21.15.28:0/3466182092 wait complete. 2024-04-24T08:09:06.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.226+0000 7fa617e00700 1 Processor -- start 2024-04-24T08:09:06.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.226+0000 7fa617e00700 1 -- start start 2024-04-24T08:09:06.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.227+0000 7fa617e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa61006a800 0x7fa61019b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.227+0000 7fa617e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa61006b1a0 0x7fa61019ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.227+0000 7fa617e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa610108b70 0x7fa61019fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.227+0000 7fa617e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa610112e20 con 0x7fa61006b1a0 2024-04-24T08:09:06.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.227+0000 7fa617e00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa610112ca0 con 0x7fa610108b70 2024-04-24T08:09:06.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.227+0000 7fa617e00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa610112fa0 con 0x7fa61006a800 2024-04-24T08:09:06.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.227+0000 7fa615400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa61006a800 0x7fa61019b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.227+0000 7fa614a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa61006b1a0 0x7fa61019ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.227+0000 7fa615400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa61006a800 0x7fa61019b530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:49354/0 (socket says 172.21.15.28:49354) 2024-04-24T08:09:06.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.227+0000 7fa615400700 1 -- 172.21.15.28:0/4131912436 learned_addr learned my addr 172.21.15.28:0/4131912436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:06.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.227+0000 7fa615e00700 1 --2- 172.21.15.28:0/4131912436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa610108b70 0x7fa61019fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.228+0000 7fa615400700 1 -- 172.21.15.28:0/4131912436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa610108b70 msgr2=0x7fa61019fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.228+0000 7fa615400700 1 --2- 172.21.15.28:0/4131912436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa610108b70 0x7fa61019fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.228+0000 7fa615400700 1 -- 172.21.15.28:0/4131912436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa61006b1a0 msgr2=0x7fa61019ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.228+0000 7fa615400700 1 --2- 172.21.15.28:0/4131912436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa61006b1a0 0x7fa61019ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.228+0000 7fa615400700 1 -- 172.21.15.28:0/4131912436 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6101a04f0 con 0x7fa61006a800 2024-04-24T08:09:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.228+0000 7fa614a00700 1 --2- 172.21.15.28:0/4131912436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa61006b1a0 0x7fa61019ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.228+0000 7fa615400700 1 --2- 172.21.15.28:0/4131912436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa61006a800 0x7fa61019b530 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa60400ec10 tx=0x7fa60400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.228+0000 7fa60e000700 1 -- 172.21.15.28:0/4131912436 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa60400cc90 con 0x7fa61006a800 2024-04-24T08:09:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.228+0000 7fa60e000700 1 -- 172.21.15.28:0/4131912436 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa60400ce30 con 0x7fa61006a800 2024-04-24T08:09:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.228+0000 7fa60e000700 1 -- 172.21.15.28:0/4131912436 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa604010690 con 0x7fa61006a800 2024-04-24T08:09:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.228+0000 7fa617e00700 1 -- 172.21.15.28:0/4131912436 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6101a07e0 con 0x7fa61006a800 2024-04-24T08:09:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.229+0000 7fa617e00700 1 -- 172.21.15.28:0/4131912436 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6101a8060 con 0x7fa61006a800 2024-04-24T08:09:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.230+0000 7fa617e00700 1 -- 172.21.15.28:0/4131912436 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6100697e0 con 0x7fa61006a800 2024-04-24T08:09:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.232+0000 7fa60e000700 1 -- 172.21.15.28:0/4131912436 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fa604003680 con 0x7fa61006a800 2024-04-24T08:09:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.233+0000 7fa60e000700 1 --2- 172.21.15.28:0/4131912436 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa5f80a3d50 0x7fa5f80a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.233+0000 7fa60e000700 1 -- 172.21.15.28:0/4131912436 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fa6040d09d0 con 0x7fa61006a800 2024-04-24T08:09:06.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.233+0000 7fa614a00700 1 --2- 172.21.15.28:0/4131912436 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa5f80a3d50 0x7fa5f80a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.234+0000 7fa614a00700 1 --2- 172.21.15.28:0/4131912436 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa5f80a3d50 0x7fa5f80a6210 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa610078fe0 tx=0x7fa5fc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:06.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.235+0000 7fa60e000700 1 -- 172.21.15.28:0/4131912436 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa60409d1d0 con 0x7fa61006a800 2024-04-24T08:09:06.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.333+0000 7f9d4b200700 1 -- 172.21.15.28:0/2220099183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4c103c30 msgr2=0x7f9d4c104010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.333+0000 7f9d4b200700 1 --2- 172.21.15.28:0/2220099183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4c103c30 0x7f9d4c104010 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c009990 tx=0x7f9d3c009ca0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.333+0000 7f9d4b200700 1 -- 172.21.15.28:0/2220099183 shutdown_connections 2024-04-24T08:09:06.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.333+0000 7f9d4b200700 1 --2- 172.21.15.28:0/2220099183 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d4c10d060 0x7f9d4c10f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.333+0000 7f9d4b200700 1 --2- 172.21.15.28:0/2220099183 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4c103c30 0x7f9d4c104010 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.333+0000 7f9d4b200700 1 --2- 172.21.15.28:0/2220099183 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d4c104550 0x7f9d4c10ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.333+0000 7f9d4b200700 1 -- 172.21.15.28:0/2220099183 >> 172.21.15.28:0/2220099183 conn(0x7f9d4c0fd480 msgr2=0x7f9d4c0ff8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.333+0000 7f9d4b200700 1 -- 172.21.15.28:0/2220099183 shutdown_connections 2024-04-24T08:09:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.333+0000 7f9d4b200700 1 -- 172.21.15.28:0/2220099183 wait complete. 2024-04-24T08:09:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.335+0000 7f9d4b200700 1 Processor -- start 2024-04-24T08:09:06.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.335+0000 7f9d4b200700 1 -- start start 2024-04-24T08:09:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.335+0000 7f9d4b200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d4c103c30 0x7f9d4c101890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.335+0000 7f9d4b200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d4c104550 0x7f9d4c101dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.335+0000 7f9d4b200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4c10d060 0x7f9d4c102310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.335+0000 7f9d4b200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c112dc0 con 0x7f9d4c10d060 2024-04-24T08:09:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.335+0000 7f9d4b200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c112c40 con 0x7f9d4c103c30 2024-04-24T08:09:06.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.335+0000 7f9d4b200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c112f40 con 0x7f9d4c104550 2024-04-24T08:09:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.336+0000 7f9d49e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d4c103c30 0x7f9d4c101890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.336+0000 7f9d49e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d4c103c30 0x7f9d4c101890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41914/0 (socket says 172.21.15.28:41914) 2024-04-24T08:09:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.336+0000 7f9d49e00700 1 -- 172.21.15.28:0/448457018 learned_addr learned my addr 172.21.15.28:0/448457018 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.336+0000 7f9d4a800700 1 --2- 172.21.15.28:0/448457018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4c10d060 0x7f9d4c102310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.336+0000 7f9d49e00700 1 -- 172.21.15.28:0/448457018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d4c104550 msgr2=0x7f9d4c101dd0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T08:09:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.336+0000 7f9d49e00700 1 --2- 172.21.15.28:0/448457018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d4c104550 0x7f9d4c101dd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.336+0000 7f9d49e00700 1 -- 172.21.15.28:0/448457018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4c10d060 msgr2=0x7f9d4c102310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.336+0000 7f9d49e00700 1 --2- 172.21.15.28:0/448457018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4c10d060 0x7f9d4c102310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.336+0000 7f9d49e00700 1 -- 172.21.15.28:0/448457018 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4c1a7750 con 0x7f9d4c103c30 2024-04-24T08:09:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.336+0000 7f9d49e00700 1 --2- 172.21.15.28:0/448457018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d4c103c30 0x7f9d4c101890 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c000c00 tx=0x7f9d3c00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.337+0000 7f9d42a00700 1 -- 172.21.15.28:0/448457018 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c003890 con 0x7f9d4c103c30 2024-04-24T08:09:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.337+0000 7f9d42a00700 1 -- 172.21.15.28:0/448457018 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d3c004830 con 0x7f9d4c103c30 2024-04-24T08:09:06.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.337+0000 7f9d4b200700 1 -- 172.21.15.28:0/448457018 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d4c1a79e0 con 0x7f9d4c103c30 2024-04-24T08:09:06.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.337+0000 7f9d42a00700 1 -- 172.21.15.28:0/448457018 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3c0054b0 con 0x7f9d4c103c30 2024-04-24T08:09:06.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.337+0000 7f9d4b200700 1 -- 172.21.15.28:0/448457018 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d4c1a7e60 con 0x7f9d4c103c30 2024-04-24T08:09:06.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.338+0000 7f9d4b200700 1 -- 172.21.15.28:0/448457018 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d4c0697e0 con 0x7f9d4c103c30 2024-04-24T08:09:06.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.341+0000 7f9d42a00700 1 -- 172.21.15.28:0/448457018 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f9d3c009d10 con 0x7f9d4c103c30 2024-04-24T08:09:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.343+0000 7f9d42a00700 1 --2- 172.21.15.28:0/448457018 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9d300a3cd0 0x7f9d300a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.343+0000 7f9d42a00700 1 -- 172.21.15.28:0/448457018 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f9d3c0d1250 con 0x7f9d4c103c30 2024-04-24T08:09:06.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.343+0000 7f9d49400700 1 --2- 172.21.15.28:0/448457018 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9d300a3cd0 0x7f9d300a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:06.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.344+0000 7f9d49400700 1 --2- 172.21.15.28:0/448457018 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9d300a3cd0 0x7f9d300a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c101020 tx=0x7f9d38009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:06.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.347+0000 7f9d42a00700 1 -- 172.21.15.28:0/448457018 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d3c013070 con 0x7f9d4c103c30 2024-04-24T08:09:06.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.477+0000 7fa617e00700 1 -- 172.21.15.28:0/4131912436 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7fa61002dfe0 con 0x7fa61006a800 2024-04-24T08:09:06.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.478+0000 7fa60e000700 1 -- 172.21.15.28:0/4131912436 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa60409cad0 con 0x7fa61006a800 2024-04-24T08:09:06.478 INFO:teuthology.orchestra.run.smithi028.stdout:219043332148 2024-04-24T08:09:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.481+0000 7fa617e00700 1 -- 172.21.15.28:0/4131912436 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa5f80a3d50 msgr2=0x7fa5f80a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.481+0000 7fa617e00700 1 --2- 172.21.15.28:0/4131912436 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa5f80a3d50 0x7fa5f80a6210 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa610078fe0 tx=0x7fa5fc009260 comp rx=0 tx=0).stop 2024-04-24T08:09:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.481+0000 7fa617e00700 1 -- 172.21.15.28:0/4131912436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa61006a800 msgr2=0x7fa61019b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.481+0000 7fa617e00700 1 --2- 172.21.15.28:0/4131912436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa61006a800 0x7fa61019b530 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fa60400ec10 tx=0x7fa60400ef20 comp rx=0 tx=0).stop 2024-04-24T08:09:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.482+0000 7fa617e00700 1 -- 172.21.15.28:0/4131912436 shutdown_connections 2024-04-24T08:09:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.482+0000 7fa617e00700 1 --2- 172.21.15.28:0/4131912436 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa5f80a3d50 0x7fa5f80a6210 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.482+0000 7fa617e00700 1 --2- 172.21.15.28:0/4131912436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa610108b70 0x7fa61019fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.482+0000 7fa617e00700 1 --2- 172.21.15.28:0/4131912436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa61006a800 0x7fa61019b530 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.482+0000 7fa617e00700 1 --2- 172.21.15.28:0/4131912436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa61006b1a0 0x7fa61019ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.482+0000 7fa617e00700 1 -- 172.21.15.28:0/4131912436 >> 172.21.15.28:0/4131912436 conn(0x7fa6100789a0 msgr2=0x7fa61010dff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.482+0000 7fa617e00700 1 -- 172.21.15.28:0/4131912436 shutdown_connections 2024-04-24T08:09:06.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.482+0000 7fa617e00700 1 -- 172.21.15.28:0/4131912436 wait complete. 2024-04-24T08:09:06.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.589+0000 7f9d4b200700 1 -- 172.21.15.28:0/448457018 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7f9d4c02dfe0 con 0x7f9d4c103c30 2024-04-24T08:09:06.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.590+0000 7f9d42a00700 1 -- 172.21.15.28:0/448457018 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f9d3c09dae0 con 0x7f9d4c103c30 2024-04-24T08:09:06.590 INFO:teuthology.orchestra.run.smithi028.stdout:197568495674 2024-04-24T08:09:06.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.593+0000 7f9d4b200700 1 -- 172.21.15.28:0/448457018 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9d300a3cd0 msgr2=0x7f9d300a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.593+0000 7f9d4b200700 1 --2- 172.21.15.28:0/448457018 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9d300a3cd0 0x7f9d300a6190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c101020 tx=0x7f9d38009260 comp rx=0 tx=0).stop 2024-04-24T08:09:06.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.593+0000 7f9d4b200700 1 -- 172.21.15.28:0/448457018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d4c103c30 msgr2=0x7f9d4c101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.593+0000 7f9d4b200700 1 --2- 172.21.15.28:0/448457018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d4c103c30 0x7f9d4c101890 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c000c00 tx=0x7f9d3c00bab0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.593+0000 7f9d4b200700 1 -- 172.21.15.28:0/448457018 shutdown_connections 2024-04-24T08:09:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.593+0000 7f9d4b200700 1 --2- 172.21.15.28:0/448457018 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f9d300a3cd0 0x7f9d300a6190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.593+0000 7f9d4b200700 1 --2- 172.21.15.28:0/448457018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4c10d060 0x7f9d4c102310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.593+0000 7f9d4b200700 1 --2- 172.21.15.28:0/448457018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d4c103c30 0x7f9d4c101890 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.593+0000 7f9d4b200700 1 --2- 172.21.15.28:0/448457018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d4c104550 0x7f9d4c101dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.593+0000 7f9d4b200700 1 -- 172.21.15.28:0/448457018 >> 172.21.15.28:0/448457018 conn(0x7f9d4c0fd480 msgr2=0x7f9d4c0fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.593+0000 7f9d4b200700 1 -- 172.21.15.28:0/448457018 shutdown_connections 2024-04-24T08:09:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.593+0000 7f9d4b200700 1 -- 172.21.15.28:0/448457018 wait complete. 2024-04-24T08:09:07.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.983+0000 7f39bd400700 1 -- 172.21.15.28:0/595506533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b8109380 msgr2=0x7f39b8109760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:07.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.983+0000 7f39bd400700 1 --2- 172.21.15.28:0/595506533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b8109380 0x7f39b8109760 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f39a80099c0 tx=0x7f39a8009cd0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.983+0000 7f39bd400700 1 -- 172.21.15.28:0/595506533 shutdown_connections 2024-04-24T08:09:07.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.983+0000 7f39bd400700 1 --2- 172.21.15.28:0/595506533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39b810e460 0x7f39b8116030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.983+0000 7f39bd400700 1 --2- 172.21.15.28:0/595506533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b8109380 0x7f39b8109760 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.983+0000 7f39bd400700 1 --2- 172.21.15.28:0/595506533 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f39b8109d30 0x7f39b810dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.983+0000 7f39bd400700 1 -- 172.21.15.28:0/595506533 >> 172.21.15.28:0/595506533 conn(0x7f39b80762c0 msgr2=0x7f39b8107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:07.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.983+0000 7f39bd400700 1 -- 172.21.15.28:0/595506533 shutdown_connections 2024-04-24T08:09:07.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.983+0000 7f39bd400700 1 -- 172.21.15.28:0/595506533 wait complete. 2024-04-24T08:09:07.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.984+0000 7f39bd400700 1 Processor -- start 2024-04-24T08:09:07.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.985+0000 7f39bd400700 1 -- start start 2024-04-24T08:09:07.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.985+0000 7f39bd400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39b8109380 0x7f39b81a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.985+0000 7f39bd400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b8109d30 0x7f39b81a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.985+0000 7f39bd400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f39b810e460 0x7f39b819f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.985+0000 7f39bd400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f39b8119490 con 0x7f39b8109d30 2024-04-24T08:09:07.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.985+0000 7f39bd400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f39b8119310 con 0x7f39b8109380 2024-04-24T08:09:07.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.985+0000 7f39bd400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f39b8119610 con 0x7f39b810e460 2024-04-24T08:09:07.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.985+0000 7f39b5e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b8109d30 0x7f39b81a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:07.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.985+0000 7f39b5e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b8109d30 0x7f39b81a5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55332/0 (socket says 172.21.15.28:55332) 2024-04-24T08:09:07.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.985+0000 7f39b5e00700 1 -- 172.21.15.28:0/387962176 learned_addr learned my addr 172.21.15.28:0/387962176 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:07.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.985+0000 7f39b6800700 1 --2- 172.21.15.28:0/387962176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39b8109380 0x7f39b81a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:07.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.985+0000 7f39b7200700 1 --2- 172.21.15.28:0/387962176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f39b810e460 0x7f39b819f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:07.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.986+0000 7f39b5e00700 1 -- 172.21.15.28:0/387962176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f39b810e460 msgr2=0x7f39b819f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:07.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.986+0000 7f39b5e00700 1 --2- 172.21.15.28:0/387962176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f39b810e460 0x7f39b819f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.986+0000 7f39b5e00700 1 -- 172.21.15.28:0/387962176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39b8109380 msgr2=0x7f39b81a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:07.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.986+0000 7f39b5e00700 1 --2- 172.21.15.28:0/387962176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39b8109380 0x7f39b81a5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.986+0000 7f39b5e00700 1 -- 172.21.15.28:0/387962176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39b81a0010 con 0x7f39b8109d30 2024-04-24T08:09:07.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.986+0000 7f39b6800700 1 --2- 172.21.15.28:0/387962176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39b8109380 0x7f39b81a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:07.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.986+0000 7f39b5e00700 1 --2- 172.21.15.28:0/387962176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b8109d30 0x7f39b81a5ba0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f39a400e970 tx=0x7f39a400ec80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:07.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.986+0000 7f39af400700 1 -- 172.21.15.28:0/387962176 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a4004a30 con 0x7f39b8109d30 2024-04-24T08:09:07.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.986+0000 7f39af400700 1 -- 172.21.15.28:0/387962176 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39a400cca0 con 0x7f39b8109d30 2024-04-24T08:09:07.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.986+0000 7f39bd400700 1 -- 172.21.15.28:0/387962176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39b81a0300 con 0x7f39b8109d30 2024-04-24T08:09:07.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.986+0000 7f39af400700 1 -- 172.21.15.28:0/387962176 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39a4005680 con 0x7f39b8109d30 2024-04-24T08:09:07.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.987+0000 7f39bd400700 1 -- 172.21.15.28:0/387962176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39b81ac5e0 con 0x7f39b8109d30 2024-04-24T08:09:07.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.988+0000 7f39bd400700 1 -- 172.21.15.28:0/387962176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39b810a860 con 0x7f39b8109d30 2024-04-24T08:09:07.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.990+0000 7f39af400700 1 -- 172.21.15.28:0/387962176 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f39a400c7c0 con 0x7f39b8109d30 2024-04-24T08:09:07.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.991+0000 7f39af400700 1 --2- 172.21.15.28:0/387962176 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f399c0a3d50 0x7f399c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:07.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.991+0000 7f39af400700 1 -- 172.21.15.28:0/387962176 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f39a40d0730 con 0x7f39b8109d30 2024-04-24T08:09:07.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.991+0000 7f39b6800700 1 --2- 172.21.15.28:0/387962176 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f399c0a3d50 0x7f399c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:07.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.992+0000 7f39b6800700 1 --2- 172.21.15.28:0/387962176 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f399c0a3d50 0x7f399c0a6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f39a8000c00 tx=0x7f39a8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:07.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:06.996+0000 7f39af400700 1 -- 172.21.15.28:0/387962176 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f39a409cf30 con 0x7f39b8109d30 2024-04-24T08:09:07.339 INFO:teuthology.orchestra.run.smithi028.stdout:313532612641 2024-04-24T08:09:07.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:07.236+0000 7f39bd400700 1 -- 172.21.15.28:0/387962176 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f39b802dfe0 con 0x7f39b8109d30 2024-04-24T08:09:07.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:07.237+0000 7f39af400700 1 -- 172.21.15.28:0/387962176 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f39a409c830 con 0x7f39b8109d30 2024-04-24T08:09:07.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:07.240+0000 7f39bd400700 1 -- 172.21.15.28:0/387962176 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f399c0a3d50 msgr2=0x7f399c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:07.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:07.240+0000 7f39bd400700 1 --2- 172.21.15.28:0/387962176 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f399c0a3d50 0x7f399c0a6210 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f39a8000c00 tx=0x7f39a8005fb0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:07.240+0000 7f39bd400700 1 -- 172.21.15.28:0/387962176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b8109d30 msgr2=0x7f39b81a5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:07.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:07.240+0000 7f39bd400700 1 --2- 172.21.15.28:0/387962176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b8109d30 0x7f39b81a5ba0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f39a400e970 tx=0x7f39a400ec80 comp rx=0 tx=0).stop 2024-04-24T08:09:07.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:07.240+0000 7f39bd400700 1 -- 172.21.15.28:0/387962176 shutdown_connections 2024-04-24T08:09:07.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:07.240+0000 7f39bd400700 1 --2- 172.21.15.28:0/387962176 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f399c0a3d50 0x7f399c0a6210 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:07.240+0000 7f39bd400700 1 --2- 172.21.15.28:0/387962176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f39b810e460 0x7f39b819f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:07.240+0000 7f39bd400700 1 --2- 172.21.15.28:0/387962176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f39b8109380 0x7f39b81a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:07.240+0000 7f39bd400700 1 --2- 172.21.15.28:0/387962176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f39b8109d30 0x7f39b81a5ba0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:07.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:07.240+0000 7f39bd400700 1 -- 172.21.15.28:0/387962176 >> 172.21.15.28:0/387962176 conn(0x7f39b80762c0 msgr2=0x7f39b80768e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:07.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:07.240+0000 7f39bd400700 1 -- 172.21.15.28:0/387962176 shutdown_connections 2024-04-24T08:09:07.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:07.240+0000 7f39bd400700 1 -- 172.21.15.28:0/387962176 wait complete. 2024-04-24T08:09:07.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:07 smithi028 ceph-mon[27786]: pgmap v356: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T08:09:07.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:07 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/4131912436' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-24T08:09:07.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:07 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/448457018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-24T08:09:07.532 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:07 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/387962176' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-24T08:09:07.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:07 smithi117 ceph-mon[32438]: pgmap v356: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T08:09:07.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:07 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/4131912436' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-24T08:09:07.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:07 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/448457018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-24T08:09:07.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:07 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/387962176' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-24T08:09:07.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:07 smithi194 ceph-mon[32836]: pgmap v356: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 2 B/s, 0 keys/s, 0 objects/s recovering 2024-04-24T08:09:07.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:07 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/4131912436' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-04-24T08:09:07.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:07 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/448457018' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-04-24T08:09:07.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:07 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/387962176' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-04-24T08:09:09.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.289+0000 7fc4b5000700 1 -- 172.21.15.28:0/192552931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b006bdb0 msgr2=0x7fc4b0107790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.289+0000 7fc4b5000700 1 --2- 172.21.15.28:0/192552931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b006bdb0 0x7fc4b0107790 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fc4980099c0 tx=0x7fc498009cd0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.289+0000 7fc4b5000700 1 -- 172.21.15.28:0/192552931 shutdown_connections 2024-04-24T08:09:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.289+0000 7fc4b5000700 1 --2- 172.21.15.28:0/192552931 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4b0107ec0 0x7fc4b010fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.289+0000 7fc4b5000700 1 --2- 172.21.15.28:0/192552931 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4b006b400 0x7fc4b006b7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.289+0000 7fc4b5000700 1 --2- 172.21.15.28:0/192552931 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b006bdb0 0x7fc4b0107790 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.289+0000 7fc4b5000700 1 -- 172.21.15.28:0/192552931 >> 172.21.15.28:0/192552931 conn(0x7fc4b0078750 msgr2=0x7fc4b0078b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.289+0000 7fc4b5000700 1 -- 172.21.15.28:0/192552931 shutdown_connections 2024-04-24T08:09:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.289+0000 7fc4b5000700 1 -- 172.21.15.28:0/192552931 wait complete. 2024-04-24T08:09:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.290+0000 7fc4b5000700 1 Processor -- start 2024-04-24T08:09:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.291+0000 7fc4b5000700 1 -- start start 2024-04-24T08:09:09.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.291+0000 7fc4b5000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4b006b400 0x7fc4b019b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.291+0000 7fc4b5000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4b006bdb0 0x7fc4b019bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.291+0000 7fc4b5000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b0107ec0 0x7fc4b019fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.291+0000 7fc4b5000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b0112ea0 con 0x7fc4b0107ec0 2024-04-24T08:09:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.291+0000 7fc4b5000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b0112d20 con 0x7fc4b006b400 2024-04-24T08:09:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.291+0000 7fc4b5000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b0113020 con 0x7fc4b006bdb0 2024-04-24T08:09:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.291+0000 7fc4ae800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b0107ec0 0x7fc4b019fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.291+0000 7fc4a7e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4b006bdb0 0x7fc4b019bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.291+0000 7fc4ae800700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b0107ec0 0x7fc4b019fdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55364/0 (socket says 172.21.15.28:55364) 2024-04-24T08:09:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.291+0000 7fc4ae800700 1 -- 172.21.15.28:0/1433241556 learned_addr learned my addr 172.21.15.28:0/1433241556 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.291+0000 7fc4a7e00700 1 --2- 172.21.15.28:0/1433241556 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4b006bdb0 0x7fc4b019bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:49398/0 (socket says 172.21.15.28:49398) 2024-04-24T08:09:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.291+0000 7fc4ade00700 1 --2- 172.21.15.28:0/1433241556 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4b006b400 0x7fc4b019b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.292+0000 7fc4ae800700 1 -- 172.21.15.28:0/1433241556 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4b006bdb0 msgr2=0x7fc4b019bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.292+0000 7fc4ae800700 1 --2- 172.21.15.28:0/1433241556 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4b006bdb0 0x7fc4b019bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.292+0000 7fc4ae800700 1 -- 172.21.15.28:0/1433241556 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4b006b400 msgr2=0x7fc4b019b570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.292+0000 7fc4ae800700 1 --2- 172.21.15.28:0/1433241556 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4b006b400 0x7fc4b019b570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.292+0000 7fc4ae800700 1 -- 172.21.15.28:0/1433241556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b01a0530 con 0x7fc4b0107ec0 2024-04-24T08:09:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.292+0000 7fc4a7e00700 1 --2- 172.21.15.28:0/1433241556 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4b006bdb0 0x7fc4b019bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.292+0000 7fc4ae800700 1 --2- 172.21.15.28:0/1433241556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b0107ec0 0x7fc4b019fdf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc4a800ef30 tx=0x7fc4a800c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.292+0000 7fc4a6a00700 1 -- 172.21.15.28:0/1433241556 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a80092e0 con 0x7fc4b0107ec0 2024-04-24T08:09:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.293+0000 7fc4b5000700 1 -- 172.21.15.28:0/1433241556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4b01a07c0 con 0x7fc4b0107ec0 2024-04-24T08:09:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.292+0000 7fc4a6a00700 1 -- 172.21.15.28:0/1433241556 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a8004bb0 con 0x7fc4b0107ec0 2024-04-24T08:09:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.293+0000 7fc4a6a00700 1 -- 172.21.15.28:0/1433241556 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a8004280 con 0x7fc4b0107ec0 2024-04-24T08:09:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.293+0000 7fc4b5000700 1 -- 172.21.15.28:0/1433241556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4b01a80a0 con 0x7fc4b0107ec0 2024-04-24T08:09:09.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.294+0000 7fc4b5000700 1 -- 172.21.15.28:0/1433241556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4b00697e0 con 0x7fc4b0107ec0 2024-04-24T08:09:09.056 INFO:teuthology.orchestra.run.smithi028.stdout:163208757312 2024-04-24T08:09:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.296+0000 7fc4a6a00700 1 -- 172.21.15.28:0/1433241556 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fc4a801b030 con 0x7fc4b0107ec0 2024-04-24T08:09:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.297+0000 7fc4a6a00700 1 --2- 172.21.15.28:0/1433241556 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc4940a3d50 0x7fc4940a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.298+0000 7fc4a6a00700 1 -- 172.21.15.28:0/1433241556 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fc4a80cffc0 con 0x7fc4b0107ec0 2024-04-24T08:09:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.298+0000 7fc4ade00700 1 --2- 172.21.15.28:0/1433241556 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc4940a3d50 0x7fc4940a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.298+0000 7fc4ade00700 1 --2- 172.21.15.28:0/1433241556 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc4940a3d50 0x7fc4940a6210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0009c60 tx=0x7fc4a0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.302+0000 7fc4a6a00700 1 -- 172.21.15.28:0/1433241556 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc4a8022020 con 0x7fc4b0107ec0 2024-04-24T08:09:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.558+0000 7fc4b5000700 1 -- 172.21.15.28:0/1433241556 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7fc4b01a8270 con 0x7fc4b0107ec0 2024-04-24T08:09:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.559+0000 7fc4a6a00700 1 -- 172.21.15.28:0/1433241556 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fc4a8010040 con 0x7fc4b0107ec0 2024-04-24T08:09:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.562+0000 7fc4b5000700 1 -- 172.21.15.28:0/1433241556 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc4940a3d50 msgr2=0x7fc4940a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.562+0000 7fc4b5000700 1 --2- 172.21.15.28:0/1433241556 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc4940a3d50 0x7fc4940a6210 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fc4a0009c60 tx=0x7fc4a0009260 comp rx=0 tx=0).stop 2024-04-24T08:09:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.562+0000 7fc4b5000700 1 -- 172.21.15.28:0/1433241556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b0107ec0 msgr2=0x7fc4b019fdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:09.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.562+0000 7fc4b5000700 1 --2- 172.21.15.28:0/1433241556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b0107ec0 0x7fc4b019fdf0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fc4a800ef30 tx=0x7fc4a800c3a0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.562+0000 7fc4b5000700 1 -- 172.21.15.28:0/1433241556 shutdown_connections 2024-04-24T08:09:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.562+0000 7fc4b5000700 1 --2- 172.21.15.28:0/1433241556 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc4940a3d50 0x7fc4940a6210 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.562+0000 7fc4b5000700 1 --2- 172.21.15.28:0/1433241556 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b0107ec0 0x7fc4b019fdf0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.562+0000 7fc4b5000700 1 --2- 172.21.15.28:0/1433241556 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4b006b400 0x7fc4b019b570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.562+0000 7fc4b5000700 1 --2- 172.21.15.28:0/1433241556 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4b006bdb0 0x7fc4b019bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.562+0000 7fc4b5000700 1 -- 172.21.15.28:0/1433241556 >> 172.21.15.28:0/1433241556 conn(0x7fc4b0078750 msgr2=0x7fc4b0101520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.563+0000 7fc4b5000700 1 -- 172.21.15.28:0/1433241556 shutdown_connections 2024-04-24T08:09:09.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:08.563+0000 7fc4b5000700 1 -- 172.21.15.28:0/1433241556 wait complete. 2024-04-24T08:09:09.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:09 smithi028 ceph-mon[27786]: pgmap v357: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:09:09.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:09 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1433241556' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-24T08:09:09.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:09 smithi117 ceph-mon[32438]: pgmap v357: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:09:09.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:09 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1433241556' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-24T08:09:09.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:09 smithi194 ceph-mon[32836]: pgmap v357: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:09:09.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:09 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1433241556' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-04-24T08:09:10.017 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332142 got 219043332148 for osd.7 2024-04-24T08:09:10.017 DEBUG:teuthology.parallel:result is None 2024-04-24T08:09:10.308 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574928 got 55834574932 for osd.1 2024-04-24T08:09:10.309 DEBUG:teuthology.parallel:result is None 2024-04-24T08:09:10.436 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612635 got 313532612641 for osd.10 2024-04-24T08:09:10.436 DEBUG:teuthology.parallel:result is None 2024-04-24T08:09:11.299 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:11 smithi028 ceph-mon[27786]: pgmap v358: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:09:11.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:11 smithi117 ceph-mon[32438]: pgmap v358: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:09:11.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:11 smithi194 ceph-mon[32836]: pgmap v358: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:09:12.218 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920833 got 141733920836 for osd.4 2024-04-24T08:09:12.219 DEBUG:teuthology.parallel:result is None 2024-04-24T08:09:12.456 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:12 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1653125359' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-24T08:09:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.918+0000 7f1351e00700 1 -- 172.21.15.28:0/591599629 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f134c101c70 msgr2=0x7f134c111870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.918+0000 7f1351e00700 1 --2- 172.21.15.28:0/591599629 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f134c101c70 0x7f134c111870 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f1344009f90 tx=0x7f134400b330 comp rx=0 tx=0).stop 2024-04-24T08:09:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.918+0000 7f1351e00700 1 -- 172.21.15.28:0/591599629 shutdown_connections 2024-04-24T08:09:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.918+0000 7f1351e00700 1 --2- 172.21.15.28:0/591599629 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f134c101c70 0x7f134c111870 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.918+0000 7f1351e00700 1 --2- 172.21.15.28:0/591599629 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c103a30 0x7f134c103e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.918+0000 7f1351e00700 1 --2- 172.21.15.28:0/591599629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f134c1010e0 0x7f134c101540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.918+0000 7f1351e00700 1 -- 172.21.15.28:0/591599629 >> 172.21.15.28:0/591599629 conn(0x7f134c078090 msgr2=0x7f134c0ff410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.918+0000 7f1351e00700 1 -- 172.21.15.28:0/591599629 shutdown_connections 2024-04-24T08:09:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.918+0000 7f1351e00700 1 -- 172.21.15.28:0/591599629 wait complete. 2024-04-24T08:09:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.920+0000 7f1351e00700 1 Processor -- start 2024-04-24T08:09:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.920+0000 7f1351e00700 1 -- start start 2024-04-24T08:09:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.920+0000 7f1351e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f134c1010e0 0x7f134c19b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.921+0000 7f1351e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f134c101c70 0x7f134c19b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.921+0000 7f1351e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c103a30 0x7f134c19fd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.921+0000 7f1351e00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f134c115010 con 0x7f134c103a30 2024-04-24T08:09:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.921+0000 7f1351e00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f134c114e90 con 0x7f134c1010e0 2024-04-24T08:09:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.921+0000 7f1351e00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f134c115190 con 0x7f134c101c70 2024-04-24T08:09:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.921+0000 7f134aa00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f134c1010e0 0x7f134c19b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.921+0000 7f134aa00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f134c1010e0 0x7f134c19b4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:39146/0 (socket says 172.21.15.28:39146) 2024-04-24T08:09:12.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.921+0000 7f134aa00700 1 -- 172.21.15.28:0/1653125359 learned_addr learned my addr 172.21.15.28:0/1653125359 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.921+0000 7f134b400700 1 --2- 172.21.15.28:0/1653125359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c103a30 0x7f134c19fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.921+0000 7f134a000700 1 --2- 172.21.15.28:0/1653125359 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f134c101c70 0x7f134c19b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.921+0000 7f134aa00700 1 -- 172.21.15.28:0/1653125359 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f134c101c70 msgr2=0x7f134c19b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.921+0000 7f134aa00700 1 --2- 172.21.15.28:0/1653125359 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f134c101c70 0x7f134c19b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.921+0000 7f134aa00700 1 -- 172.21.15.28:0/1653125359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c103a30 msgr2=0x7f134c19fd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.921+0000 7f134aa00700 1 --2- 172.21.15.28:0/1653125359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c103a30 0x7f134c19fd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.921+0000 7f134aa00700 1 -- 172.21.15.28:0/1653125359 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f134c1a0460 con 0x7f134c1010e0 2024-04-24T08:09:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.921+0000 7f134b400700 1 --2- 172.21.15.28:0/1653125359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c103a30 0x7f134c19fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.922+0000 7f134aa00700 1 --2- 172.21.15.28:0/1653125359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f134c1010e0 0x7f134c19b4a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f133c00e9a0 tx=0x7f133c00ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.922+0000 7f1350a00700 1 -- 172.21.15.28:0/1653125359 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c00c9c0 con 0x7f134c1010e0 2024-04-24T08:09:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.922+0000 7f1350a00700 1 -- 172.21.15.28:0/1653125359 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f133c00cb60 con 0x7f134c1010e0 2024-04-24T08:09:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.922+0000 7f1351e00700 1 -- 172.21.15.28:0/1653125359 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f134c1a0750 con 0x7f134c1010e0 2024-04-24T08:09:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.922+0000 7f1350a00700 1 -- 172.21.15.28:0/1653125359 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f133c018680 con 0x7f134c1010e0 2024-04-24T08:09:12.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.922+0000 7f1351e00700 1 -- 172.21.15.28:0/1653125359 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f134c1a7fd0 con 0x7f134c1010e0 2024-04-24T08:09:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.924+0000 7f1351e00700 1 -- 172.21.15.28:0/1653125359 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f134c1020f0 con 0x7f134c1010e0 2024-04-24T08:09:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.925+0000 7f1350a00700 1 -- 172.21.15.28:0/1653125359 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f133c00cd10 con 0x7f134c1010e0 2024-04-24T08:09:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.927+0000 7f1350a00700 1 --2- 172.21.15.28:0/1653125359 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13300a3cd0 0x7f13300a6190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.927+0000 7f1350a00700 1 -- 172.21.15.28:0/1653125359 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f133c0d0710 con 0x7f134c1010e0 2024-04-24T08:09:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.927+0000 7f134a000700 1 --2- 172.21.15.28:0/1653125359 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13300a3cd0 0x7f13300a6190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.928+0000 7f134a000700 1 --2- 172.21.15.28:0/1653125359 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13300a3cd0 0x7f13300a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f134c100650 tx=0x7f1338009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:11.931+0000 7f1350a00700 1 -- 172.21.15.28:0/1653125359 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f133c09cf10 con 0x7f134c1010e0 2024-04-24T08:09:12.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.172+0000 7f1351e00700 1 -- 172.21.15.28:0/1653125359 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f134c02dfe0 con 0x7f134c1010e0 2024-04-24T08:09:12.501 INFO:teuthology.orchestra.run.smithi028.stdout:34359738460 2024-04-24T08:09:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.172+0000 7f1350a00700 1 -- 172.21.15.28:0/1653125359 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f133c09c810 con 0x7f134c1010e0 2024-04-24T08:09:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.175+0000 7f1351e00700 1 -- 172.21.15.28:0/1653125359 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13300a3cd0 msgr2=0x7f13300a6190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.175+0000 7f1351e00700 1 --2- 172.21.15.28:0/1653125359 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13300a3cd0 0x7f13300a6190 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f134c100650 tx=0x7f1338009310 comp rx=0 tx=0).stop 2024-04-24T08:09:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.175+0000 7f1351e00700 1 -- 172.21.15.28:0/1653125359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f134c1010e0 msgr2=0x7f134c19b4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.175+0000 7f1351e00700 1 --2- 172.21.15.28:0/1653125359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f134c1010e0 0x7f134c19b4a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f133c00e9a0 tx=0x7f133c00ecb0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.175+0000 7f1351e00700 1 -- 172.21.15.28:0/1653125359 shutdown_connections 2024-04-24T08:09:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.175+0000 7f1351e00700 1 --2- 172.21.15.28:0/1653125359 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f13300a3cd0 0x7f13300a6190 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.175+0000 7f1351e00700 1 --2- 172.21.15.28:0/1653125359 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f134c103a30 0x7f134c19fd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.175+0000 7f1351e00700 1 --2- 172.21.15.28:0/1653125359 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f134c1010e0 0x7f134c19b4a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.175+0000 7f1351e00700 1 --2- 172.21.15.28:0/1653125359 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f134c101c70 0x7f134c19b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.175+0000 7f1351e00700 1 -- 172.21.15.28:0/1653125359 >> 172.21.15.28:0/1653125359 conn(0x7f134c078090 msgr2=0x7f134c0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.175+0000 7f1351e00700 1 -- 172.21.15.28:0/1653125359 shutdown_connections 2024-04-24T08:09:12.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.176+0000 7f1351e00700 1 -- 172.21.15.28:0/1653125359 wait complete. 2024-04-24T08:09:12.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:12 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1653125359' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-24T08:09:12.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:12 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1653125359' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-04-24T08:09:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.970+0000 7fd451a00700 1 -- 172.21.15.28:0/3284855847 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd44c1019f0 msgr2=0x7fd44c101e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.970+0000 7fd451a00700 1 --2- 172.21.15.28:0/3284855847 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd44c1019f0 0x7fd44c101e70 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd4380098f0 tx=0x7fd438009c00 comp rx=0 tx=0).stop 2024-04-24T08:09:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.970+0000 7fd451a00700 1 -- 172.21.15.28:0/3284855847 shutdown_connections 2024-04-24T08:09:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.970+0000 7fd451a00700 1 --2- 172.21.15.28:0/3284855847 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd44c1023b0 0x7fd44c10e8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.970+0000 7fd451a00700 1 --2- 172.21.15.28:0/3284855847 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd44c107a40 0x7fd44c107e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.970+0000 7fd451a00700 1 --2- 172.21.15.28:0/3284855847 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd44c1019f0 0x7fd44c101e70 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.970+0000 7fd451a00700 1 -- 172.21.15.28:0/3284855847 >> 172.21.15.28:0/3284855847 conn(0x7fd44c0789a0 msgr2=0x7fd44c078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.970+0000 7fd451a00700 1 -- 172.21.15.28:0/3284855847 shutdown_connections 2024-04-24T08:09:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.970+0000 7fd451a00700 1 -- 172.21.15.28:0/3284855847 wait complete. 2024-04-24T08:09:12.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.971+0000 7fd451a00700 1 Processor -- start 2024-04-24T08:09:12.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.972+0000 7fd451a00700 1 -- start start 2024-04-24T08:09:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.972+0000 7fd451a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd44c1019f0 0x7fd44c19b520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.972+0000 7fd451a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd44c1023b0 0x7fd44c19ba60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.972+0000 7fd451a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd44c107a40 0x7fd44c19fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.972+0000 7fd451a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c111d40 con 0x7fd44c1023b0 2024-04-24T08:09:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.972+0000 7fd451a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c111bc0 con 0x7fd44c1019f0 2024-04-24T08:09:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.972+0000 7fd451a00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd44c111ec0 con 0x7fd44c107a40 2024-04-24T08:09:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.972+0000 7fd44a000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd44c1023b0 0x7fd44c19ba60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.972+0000 7fd44a000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd44c1023b0 0x7fd44c19ba60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33830/0 (socket says 172.21.15.28:33830) 2024-04-24T08:09:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.972+0000 7fd44a000700 1 -- 172.21.15.28:0/697964898 learned_addr learned my addr 172.21.15.28:0/697964898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.972+0000 7fd44aa00700 1 --2- 172.21.15.28:0/697964898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd44c1019f0 0x7fd44c19b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.972+0000 7fd44b400700 1 --2- 172.21.15.28:0/697964898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd44c107a40 0x7fd44c19fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.973+0000 7fd44a000700 1 -- 172.21.15.28:0/697964898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd44c107a40 msgr2=0x7fd44c19fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.973+0000 7fd44a000700 1 --2- 172.21.15.28:0/697964898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd44c107a40 0x7fd44c19fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.973+0000 7fd44a000700 1 -- 172.21.15.28:0/697964898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd44c1019f0 msgr2=0x7fd44c19b520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:12.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.973+0000 7fd44a000700 1 --2- 172.21.15.28:0/697964898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd44c1019f0 0x7fd44c19b520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:12.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.973+0000 7fd44a000700 1 -- 172.21.15.28:0/697964898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd44c1a0530 con 0x7fd44c1023b0 2024-04-24T08:09:12.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.973+0000 7fd44a000700 1 --2- 172.21.15.28:0/697964898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd44c1023b0 0x7fd44c19ba60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd438005950 tx=0x7fd43800fb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:12.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.973+0000 7fd443400700 1 -- 172.21.15.28:0/697964898 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd4380051f0 con 0x7fd44c1023b0 2024-04-24T08:09:12.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.973+0000 7fd443400700 1 -- 172.21.15.28:0/697964898 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd43800b560 con 0x7fd44c1023b0 2024-04-24T08:09:12.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.973+0000 7fd443400700 1 -- 172.21.15.28:0/697964898 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd438003a40 con 0x7fd44c1023b0 2024-04-24T08:09:12.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.973+0000 7fd451a00700 1 -- 172.21.15.28:0/697964898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd44c1a07c0 con 0x7fd44c1023b0 2024-04-24T08:09:12.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.974+0000 7fd451a00700 1 -- 172.21.15.28:0/697964898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd44c1a80a0 con 0x7fd44c1023b0 2024-04-24T08:09:12.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.975+0000 7fd451a00700 1 -- 172.21.15.28:0/697964898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd44c0697e0 con 0x7fd44c1023b0 2024-04-24T08:09:12.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.977+0000 7fd443400700 1 -- 172.21.15.28:0/697964898 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7fd43800ba80 con 0x7fd44c1023b0 2024-04-24T08:09:12.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.979+0000 7fd443400700 1 --2- 172.21.15.28:0/697964898 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd4300a3d50 0x7fd4300a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:12.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.979+0000 7fd443400700 1 -- 172.21.15.28:0/697964898 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fd4380d1530 con 0x7fd44c1023b0 2024-04-24T08:09:12.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.980+0000 7fd44aa00700 1 --2- 172.21.15.28:0/697964898 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd4300a3d50 0x7fd4300a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:12.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.980+0000 7fd44aa00700 1 --2- 172.21.15.28:0/697964898 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd4300a3d50 0x7fd4300a6210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd43c005fd0 tx=0x7fd43c009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:12.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:12.984+0000 7fd443400700 1 -- 172.21.15.28:0/697964898 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd438013030 con 0x7fd44c1023b0 2024-04-24T08:09:13.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.219+0000 7fd451a00700 1 -- 172.21.15.28:0/697964898 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7fd44c1a8270 con 0x7fd44c1023b0 2024-04-24T08:09:13.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.220+0000 7fd443400700 1 -- 172.21.15.28:0/697964898 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7fd438021380 con 0x7fd44c1023b0 2024-04-24T08:09:13.220 INFO:teuthology.orchestra.run.smithi028.stdout:347892351007 2024-04-24T08:09:13.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.223+0000 7fd451a00700 1 -- 172.21.15.28:0/697964898 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd4300a3d50 msgr2=0x7fd4300a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:13.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.223+0000 7fd451a00700 1 --2- 172.21.15.28:0/697964898 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd4300a3d50 0x7fd4300a6210 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd43c005fd0 tx=0x7fd43c009310 comp rx=0 tx=0).stop 2024-04-24T08:09:13.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.223+0000 7fd451a00700 1 -- 172.21.15.28:0/697964898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd44c1023b0 msgr2=0x7fd44c19ba60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:13.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.223+0000 7fd451a00700 1 --2- 172.21.15.28:0/697964898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd44c1023b0 0x7fd44c19ba60 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fd438005950 tx=0x7fd43800fb20 comp rx=0 tx=0).stop 2024-04-24T08:09:13.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.223+0000 7fd451a00700 1 -- 172.21.15.28:0/697964898 shutdown_connections 2024-04-24T08:09:13.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.223+0000 7fd451a00700 1 --2- 172.21.15.28:0/697964898 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd4300a3d50 0x7fd4300a6210 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.223+0000 7fd451a00700 1 --2- 172.21.15.28:0/697964898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd44c107a40 0x7fd44c19fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.223+0000 7fd451a00700 1 --2- 172.21.15.28:0/697964898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd44c1019f0 0x7fd44c19b520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.223+0000 7fd451a00700 1 --2- 172.21.15.28:0/697964898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd44c1023b0 0x7fd44c19ba60 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:13.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.223+0000 7fd451a00700 1 -- 172.21.15.28:0/697964898 >> 172.21.15.28:0/697964898 conn(0x7fd44c0789a0 msgr2=0x7fd44c06a7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:13.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.223+0000 7fd451a00700 1 -- 172.21.15.28:0/697964898 shutdown_connections 2024-04-24T08:09:13.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.224+0000 7fd451a00700 1 -- 172.21.15.28:0/697964898 wait complete. 2024-04-24T08:09:13.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:13 smithi028 ceph-mon[27786]: pgmap v359: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:09:13.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:13 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/697964898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-24T08:09:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:13 smithi117 ceph-mon[32438]: pgmap v359: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:09:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:13 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/697964898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-24T08:09:13.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:13 smithi194 ceph-mon[32836]: pgmap v359: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:09:13.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:13 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/697964898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-04-24T08:09:14.156 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892351000 got 347892351007 for osd.11 2024-04-24T08:09:14.156 DEBUG:teuthology.parallel:result is None 2024-04-24T08:09:14.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.984+0000 7f399ec00700 1 -- 172.21.15.28:0/2244637803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3998104f70 msgr2=0x7f3998105350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.984+0000 7f399ec00700 1 --2- 172.21.15.28:0/2244637803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3998104f70 0x7f3998105350 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f3988009990 tx=0x7f3988009ca0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.984+0000 7f399ec00700 1 -- 172.21.15.28:0/2244637803 shutdown_connections 2024-04-24T08:09:14.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.984+0000 7f399ec00700 1 --2- 172.21.15.28:0/2244637803 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f399810a050 0x7f3998111c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.984+0000 7f399ec00700 1 --2- 172.21.15.28:0/2244637803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3998104f70 0x7f3998105350 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.984+0000 7f399ec00700 1 --2- 172.21.15.28:0/2244637803 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3998105920 0x7f3998109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.985+0000 7f399ec00700 1 -- 172.21.15.28:0/2244637803 >> 172.21.15.28:0/2244637803 conn(0x7f3998100800 msgr2=0x7f3998102c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:14.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.985+0000 7f399ec00700 1 -- 172.21.15.28:0/2244637803 shutdown_connections 2024-04-24T08:09:14.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.985+0000 7f399ec00700 1 -- 172.21.15.28:0/2244637803 wait complete. 2024-04-24T08:09:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.987+0000 7f399ec00700 1 Processor -- start 2024-04-24T08:09:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.987+0000 7f399ec00700 1 -- start start 2024-04-24T08:09:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.988+0000 7f399ec00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3998104f70 0x7f399819b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.988+0000 7f399ec00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3998105920 0x7f399819b970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.988+0000 7f399ec00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f399810a050 0x7f399819fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.988+0000 7f399ec00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3998115040 con 0x7f3998104f70 2024-04-24T08:09:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.988+0000 7f399ec00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3998114ec0 con 0x7f3998105920 2024-04-24T08:09:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.988+0000 7f399ec00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f39981151c0 con 0x7f399810a050 2024-04-24T08:09:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.988+0000 7f3997400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3998105920 0x7f399819b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.988+0000 7f3997e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3998104f70 0x7f399819b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.988+0000 7f3997400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3998105920 0x7f399819b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:39192/0 (socket says 172.21.15.28:39192) 2024-04-24T08:09:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.988+0000 7f399cc00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f399810a050 0x7f399819fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.988+0000 7f3997400700 1 -- 172.21.15.28:0/1698473385 learned_addr learned my addr 172.21.15.28:0/1698473385 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.989+0000 7f3997400700 1 -- 172.21.15.28:0/1698473385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f399810a050 msgr2=0x7f399819fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.989+0000 7f3997400700 1 --2- 172.21.15.28:0/1698473385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f399810a050 0x7f399819fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.989+0000 7f3997400700 1 -- 172.21.15.28:0/1698473385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3998104f70 msgr2=0x7f399819b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.989+0000 7f3997400700 1 --2- 172.21.15.28:0/1698473385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3998104f70 0x7f399819b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.989+0000 7f3997400700 1 -- 172.21.15.28:0/1698473385 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f39981a03f0 con 0x7f3998105920 2024-04-24T08:09:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.989+0000 7f3997400700 1 --2- 172.21.15.28:0/1698473385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3998105920 0x7f399819b970 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f398400e970 tx=0x7f398400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.989+0000 7f3994c00700 1 -- 172.21.15.28:0/1698473385 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3984004a30 con 0x7f3998105920 2024-04-24T08:09:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.989+0000 7f3994c00700 1 -- 172.21.15.28:0/1698473385 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f398400ccc0 con 0x7f3998105920 2024-04-24T08:09:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.989+0000 7f399ec00700 1 -- 172.21.15.28:0/1698473385 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39981a06e0 con 0x7f3998105920 2024-04-24T08:09:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.989+0000 7f3994c00700 1 -- 172.21.15.28:0/1698473385 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3984005650 con 0x7f3998105920 2024-04-24T08:09:14.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.990+0000 7f399ec00700 1 -- 172.21.15.28:0/1698473385 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39981a7f60 con 0x7f3998105920 2024-04-24T08:09:14.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.991+0000 7f399ec00700 1 -- 172.21.15.28:0/1698473385 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f39980697e0 con 0x7f3998105920 2024-04-24T08:09:14.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.993+0000 7f3994c00700 1 -- 172.21.15.28:0/1698473385 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f3984004bd0 con 0x7f3998105920 2024-04-24T08:09:14.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.994+0000 7f3994c00700 1 --2- 172.21.15.28:0/1698473385 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f397c0a3d50 0x7f397c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:14.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.994+0000 7f3994c00700 1 -- 172.21.15.28:0/1698473385 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f39840d0780 con 0x7f3998105920 2024-04-24T08:09:14.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.994+0000 7f3997e00700 1 --2- 172.21.15.28:0/1698473385 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f397c0a3d50 0x7f397c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:14.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.995+0000 7f3997e00700 1 --2- 172.21.15.28:0/1698473385 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f397c0a3d50 0x7f397c0a6210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f39880098e0 tx=0x7f3988005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:14.272 INFO:teuthology.orchestra.run.smithi028.stdout:107374182475 2024-04-24T08:09:14.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:13.999+0000 7f3994c00700 1 -- 172.21.15.28:0/1698473385 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f398409cf80 con 0x7f3998105920 2024-04-24T08:09:14.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:14.240+0000 7f399ec00700 1 -- 172.21.15.28:0/1698473385 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f399802dfe0 con 0x7f3998105920 2024-04-24T08:09:14.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:14.241+0000 7f3994c00700 1 -- 172.21.15.28:0/1698473385 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f398409c880 con 0x7f3998105920 2024-04-24T08:09:14.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:14.244+0000 7f399ec00700 1 -- 172.21.15.28:0/1698473385 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f397c0a3d50 msgr2=0x7f397c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:14.244+0000 7f399ec00700 1 --2- 172.21.15.28:0/1698473385 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f397c0a3d50 0x7f397c0a6210 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f39880098e0 tx=0x7f3988005960 comp rx=0 tx=0).stop 2024-04-24T08:09:14.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:14.244+0000 7f399ec00700 1 -- 172.21.15.28:0/1698473385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3998105920 msgr2=0x7f399819b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:14.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:14.244+0000 7f399ec00700 1 --2- 172.21.15.28:0/1698473385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3998105920 0x7f399819b970 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f398400e970 tx=0x7f398400ec80 comp rx=0 tx=0).stop 2024-04-24T08:09:14.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:14.244+0000 7f399ec00700 1 -- 172.21.15.28:0/1698473385 shutdown_connections 2024-04-24T08:09:14.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:14.244+0000 7f399ec00700 1 --2- 172.21.15.28:0/1698473385 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f397c0a3d50 0x7f397c0a6210 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:14.244+0000 7f399ec00700 1 --2- 172.21.15.28:0/1698473385 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f399810a050 0x7f399819fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:14.244+0000 7f399ec00700 1 --2- 172.21.15.28:0/1698473385 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3998104f70 0x7f399819b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:14.244+0000 7f399ec00700 1 --2- 172.21.15.28:0/1698473385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3998105920 0x7f399819b970 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:14.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:14.244+0000 7f399ec00700 1 -- 172.21.15.28:0/1698473385 >> 172.21.15.28:0/1698473385 conn(0x7f3998100800 msgr2=0x7f3998102470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:14.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:14.245+0000 7f399ec00700 1 -- 172.21.15.28:0/1698473385 shutdown_connections 2024-04-24T08:09:14.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:14.245+0000 7f399ec00700 1 -- 172.21.15.28:0/1698473385 wait complete. 2024-04-24T08:09:14.421 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757305 got 163208757312 for osd.5 2024-04-24T08:09:14.421 DEBUG:teuthology.parallel:result is None 2024-04-24T08:09:14.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:14 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1698473385' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-24T08:09:14.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:14 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1698473385' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-24T08:09:14.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:14 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1698473385' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-04-24T08:09:15.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:15 smithi028 ceph-mon[27786]: pgmap v360: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:09:15.526 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182470 got 107374182475 for osd.3 2024-04-24T08:09:15.527 DEBUG:teuthology.parallel:result is None 2024-04-24T08:09:15.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:15 smithi117 ceph-mon[32438]: pgmap v360: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:09:15.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:15 smithi194 ceph-mon[32836]: pgmap v360: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:09:15.869 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495669 got 197568495674 for osd.6 2024-04-24T08:09:15.869 DEBUG:teuthology.parallel:result is None 2024-04-24T08:09:16.455 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738452 got 34359738460 for osd.0 2024-04-24T08:09:16.455 DEBUG:teuthology.parallel:result is None 2024-04-24T08:09:16.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.750+0000 7f873ee00700 1 -- 172.21.15.28:0/3004474346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8738107ab0 msgr2=0x7f8738107e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:16.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.750+0000 7f873ee00700 1 --2- 172.21.15.28:0/3004474346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8738107ab0 0x7f8738107e90 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f8728009990 tx=0x7f8728009ca0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.750+0000 7f873ee00700 1 -- 172.21.15.28:0/3004474346 shutdown_connections 2024-04-24T08:09:16.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.750+0000 7f873ee00700 1 --2- 172.21.15.28:0/3004474346 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8738103540 0x7f873810fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.750+0000 7f873ee00700 1 --2- 172.21.15.28:0/3004474346 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8738107ab0 0x7f8738107e90 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.750+0000 7f873ee00700 1 --2- 172.21.15.28:0/3004474346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8738102ba0 0x7f8738103000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.750+0000 7f873ee00700 1 -- 172.21.15.28:0/3004474346 >> 172.21.15.28:0/3004474346 conn(0x7f8738079c50 msgr2=0x7f873807a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:16.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.750+0000 7f873ee00700 1 -- 172.21.15.28:0/3004474346 shutdown_connections 2024-04-24T08:09:16.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.750+0000 7f873ee00700 1 -- 172.21.15.28:0/3004474346 wait complete. 2024-04-24T08:09:16.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.752+0000 7f873ee00700 1 Processor -- start 2024-04-24T08:09:16.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.752+0000 7f873ee00700 1 -- start start 2024-04-24T08:09:16.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.752+0000 7f873ee00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8738102ba0 0x7f8738199330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.752+0000 7f873ee00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8738103540 0x7f8738199870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.752+0000 7f873ee00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8738107ab0 0x7f873819dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.753+0000 7f873ee00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8738112e80 con 0x7f8738107ab0 2024-04-24T08:09:16.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.753+0000 7f873ee00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8738112d00 con 0x7f8738103540 2024-04-24T08:09:16.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.753+0000 7f873ee00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8738113000 con 0x7f8738102ba0 2024-04-24T08:09:16.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.753+0000 7f8737400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8738103540 0x7f8738199870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.753+0000 7f8737e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8738102ba0 0x7f8738199330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.753+0000 7f873ce00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8738107ab0 0x7f873819dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.753+0000 7f8737400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8738103540 0x7f8738199870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:39206/0 (socket says 172.21.15.28:39206) 2024-04-24T08:09:16.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.753+0000 7f8737400700 1 -- 172.21.15.28:0/4228880149 learned_addr learned my addr 172.21.15.28:0/4228880149 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:16.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.753+0000 7f8737400700 1 -- 172.21.15.28:0/4228880149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8738102ba0 msgr2=0x7f8738199330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:16.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.753+0000 7f8737400700 1 --2- 172.21.15.28:0/4228880149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8738102ba0 0x7f8738199330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.753+0000 7f8737400700 1 -- 172.21.15.28:0/4228880149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8738107ab0 msgr2=0x7f873819dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:16.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.753+0000 7f8737400700 1 --2- 172.21.15.28:0/4228880149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8738107ab0 0x7f873819dbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:16.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.753+0000 7f8737400700 1 -- 172.21.15.28:0/4228880149 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f873819e2f0 con 0x7f8738103540 2024-04-24T08:09:16.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.754+0000 7f8737400700 1 --2- 172.21.15.28:0/4228880149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8738103540 0x7f8738199870 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f872400e9a0 tx=0x7f872400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:16.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.754+0000 7f8734c00700 1 -- 172.21.15.28:0/4228880149 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f872400ca10 con 0x7f8738103540 2024-04-24T08:09:16.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.754+0000 7f8734c00700 1 -- 172.21.15.28:0/4228880149 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f872400cbb0 con 0x7f8738103540 2024-04-24T08:09:16.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.754+0000 7f873ee00700 1 -- 172.21.15.28:0/4228880149 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f873819e5e0 con 0x7f8738103540 2024-04-24T08:09:16.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.754+0000 7f8734c00700 1 -- 172.21.15.28:0/4228880149 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8724018680 con 0x7f8738103540 2024-04-24T08:09:16.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.754+0000 7f873ee00700 1 -- 172.21.15.28:0/4228880149 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f87381a5e60 con 0x7f8738103540 2024-04-24T08:09:16.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.755+0000 7f873ee00700 1 -- 172.21.15.28:0/4228880149 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f87380697e0 con 0x7f8738103540 2024-04-24T08:09:16.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.757+0000 7f8734c00700 1 -- 172.21.15.28:0/4228880149 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 17) ==== 138697+0+0 (secure 0 0 0) 0x7f8724010430 con 0x7f8738103540 2024-04-24T08:09:16.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.759+0000 7f8734c00700 1 --2- 172.21.15.28:0/4228880149 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f871c0a3d50 0x7f871c0a6210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:16.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.759+0000 7f8734c00700 1 -- 172.21.15.28:0/4228880149 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f87240d0790 con 0x7f8738103540 2024-04-24T08:09:16.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.759+0000 7f8737e00700 1 --2- 172.21.15.28:0/4228880149 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f871c0a3d50 0x7f871c0a6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:16.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.760+0000 7f8737e00700 1 --2- 172.21.15.28:0/4228880149 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f871c0a3d50 0x7f871c0a6210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8728006010 tx=0x7f872800b260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:16.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:16.763+0000 7f8734c00700 1 -- 172.21.15.28:0/4228880149 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f872409cf90 con 0x7f8738103540 2024-04-24T08:09:17.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:17.002+0000 7f873ee00700 1 -- 172.21.15.28:0/4228880149 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f873802dfe0 con 0x7f8738103540 2024-04-24T08:09:17.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:17.003+0000 7f8734c00700 1 -- 172.21.15.28:0/4228880149 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f8724085e20 con 0x7f8738103540 2024-04-24T08:09:17.005 INFO:teuthology.orchestra.run.smithi028.stdout:292057776169 2024-04-24T08:09:17.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:17.006+0000 7f873ee00700 1 -- 172.21.15.28:0/4228880149 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f871c0a3d50 msgr2=0x7f871c0a6210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:17.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:17.006+0000 7f873ee00700 1 --2- 172.21.15.28:0/4228880149 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f871c0a3d50 0x7f871c0a6210 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f8728006010 tx=0x7f872800b260 comp rx=0 tx=0).stop 2024-04-24T08:09:17.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:17.006+0000 7f873ee00700 1 -- 172.21.15.28:0/4228880149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8738103540 msgr2=0x7f8738199870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:17.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:17.006+0000 7f873ee00700 1 --2- 172.21.15.28:0/4228880149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8738103540 0x7f8738199870 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f872400e9a0 tx=0x7f872400ecb0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:17.006+0000 7f873ee00700 1 -- 172.21.15.28:0/4228880149 shutdown_connections 2024-04-24T08:09:17.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:17.006+0000 7f873ee00700 1 --2- 172.21.15.28:0/4228880149 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f871c0a3d50 0x7f871c0a6210 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:17.006+0000 7f873ee00700 1 --2- 172.21.15.28:0/4228880149 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8738107ab0 0x7f873819dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:17.006+0000 7f873ee00700 1 --2- 172.21.15.28:0/4228880149 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8738102ba0 0x7f8738199330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:17.006+0000 7f873ee00700 1 --2- 172.21.15.28:0/4228880149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8738103540 0x7f8738199870 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:17.006+0000 7f873ee00700 1 -- 172.21.15.28:0/4228880149 >> 172.21.15.28:0/4228880149 conn(0x7f8738079c50 msgr2=0x7f8738100c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:17.007+0000 7f873ee00700 1 -- 172.21.15.28:0/4228880149 shutdown_connections 2024-04-24T08:09:17.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:17.007+0000 7f873ee00700 1 -- 172.21.15.28:0/4228880149 wait complete. 2024-04-24T08:09:17.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:17 smithi028 ceph-mon[27786]: pgmap v361: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:09:17.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:17 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/4228880149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-24T08:09:17.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:17 smithi117 ceph-mon[32438]: pgmap v361: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:09:17.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:17 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/4228880149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-24T08:09:17.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:17 smithi194 ceph-mon[32836]: pgmap v361: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:09:17.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:17 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/4228880149' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-04-24T08:09:17.917 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776163 got 292057776169 for osd.9 2024-04-24T08:09:17.917 DEBUG:teuthology.parallel:result is None 2024-04-24T08:09:17.917 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-04-24T08:09:17.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-24T08:09:18.391 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:09:19.576 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:19 smithi028 ceph-mon[27786]: pgmap v362: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:09:19.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:19 smithi117 ceph-mon[32438]: pgmap v362: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:09:19.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:19 smithi194 ceph-mon[32836]: pgmap v362: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:09:20.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.846+0000 7fe497a00700 1 -- 172.21.15.28:0/1034496687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe490075640 msgr2=0x7fe49010efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:20.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.846+0000 7fe497a00700 1 --2- 172.21.15.28:0/1034496687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe490075640 0x7fe49010efb0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009990 tx=0x7fe47c009ca0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.847+0000 7fe497a00700 1 -- 172.21.15.28:0/1034496687 shutdown_connections 2024-04-24T08:09:20.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.847+0000 7fe497a00700 1 --2- 172.21.15.28:0/1034496687 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe49010f4f0 0x7fe4901118e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.847+0000 7fe497a00700 1 --2- 172.21.15.28:0/1034496687 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe490074d20 0x7fe490075100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.847+0000 7fe497a00700 1 --2- 172.21.15.28:0/1034496687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe490075640 0x7fe49010efb0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.847+0000 7fe497a00700 1 -- 172.21.15.28:0/1034496687 >> 172.21.15.28:0/1034496687 conn(0x7fe490079c50 msgr2=0x7fe49007a060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:20.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.847+0000 7fe497a00700 1 -- 172.21.15.28:0/1034496687 shutdown_connections 2024-04-24T08:09:20.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.847+0000 7fe497a00700 1 -- 172.21.15.28:0/1034496687 wait complete. 2024-04-24T08:09:20.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.848+0000 7fe497a00700 1 Processor -- start 2024-04-24T08:09:20.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.848+0000 7fe497a00700 1 -- start start 2024-04-24T08:09:20.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.849+0000 7fe497a00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe490074d20 0x7fe49019b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:20.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.849+0000 7fe497a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe490075640 0x7fe49019ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:20.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.849+0000 7fe497a00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe49010f4f0 0x7fe49019fde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:20.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.849+0000 7fe497a00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe490115060 con 0x7fe49010f4f0 2024-04-24T08:09:20.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.849+0000 7fe497a00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe490114ee0 con 0x7fe490075640 2024-04-24T08:09:20.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.849+0000 7fe497a00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe4901151e0 con 0x7fe490074d20 2024-04-24T08:09:20.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.849+0000 7fe48fe00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe490075640 0x7fe49019ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:20.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.849+0000 7fe48fe00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe490075640 0x7fe49019ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:48030/0 (socket says 172.21.15.28:48030) 2024-04-24T08:09:20.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.849+0000 7fe48fe00700 1 -- 172.21.15.28:0/2006342906 learned_addr learned my addr 172.21.15.28:0/2006342906 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:20.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.849+0000 7fe495000700 1 --2- 172.21.15.28:0/2006342906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe490074d20 0x7fe49019b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:20.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.849+0000 7fe495a00700 1 --2- 172.21.15.28:0/2006342906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe49010f4f0 0x7fe49019fde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:20.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.850+0000 7fe48fe00700 1 -- 172.21.15.28:0/2006342906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe490074d20 msgr2=0x7fe49019b510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:20.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.850+0000 7fe48fe00700 1 --2- 172.21.15.28:0/2006342906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe490074d20 0x7fe49019b510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.850+0000 7fe48fe00700 1 -- 172.21.15.28:0/2006342906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe49010f4f0 msgr2=0x7fe49019fde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:20.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.850+0000 7fe48fe00700 1 --2- 172.21.15.28:0/2006342906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe49010f4f0 0x7fe49019fde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:20.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.850+0000 7fe48fe00700 1 -- 172.21.15.28:0/2006342906 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4901a0520 con 0x7fe490075640 2024-04-24T08:09:20.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.850+0000 7fe48fe00700 1 --2- 172.21.15.28:0/2006342906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe490075640 0x7fe49019ba50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe47c000c00 tx=0x7fe47c00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:20.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.850+0000 7fe48d600700 1 -- 172.21.15.28:0/2006342906 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c026070 con 0x7fe490075640 2024-04-24T08:09:20.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.850+0000 7fe48d600700 1 -- 172.21.15.28:0/2006342906 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe47c003d70 con 0x7fe490075640 2024-04-24T08:09:20.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.850+0000 7fe48d600700 1 -- 172.21.15.28:0/2006342906 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe47c00fe20 con 0x7fe490075640 2024-04-24T08:09:20.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.850+0000 7fe497a00700 1 -- 172.21.15.28:0/2006342906 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4901a07b0 con 0x7fe490075640 2024-04-24T08:09:20.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.851+0000 7fe497a00700 1 -- 172.21.15.28:0/2006342906 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4901a8090 con 0x7fe490075640 2024-04-24T08:09:20.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.852+0000 7fe497a00700 1 -- 172.21.15.28:0/2006342906 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe4900697e0 con 0x7fe490075640 2024-04-24T08:09:20.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.855+0000 7fe48d600700 1 -- 172.21.15.28:0/2006342906 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe47c029070 con 0x7fe490075640 2024-04-24T08:09:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.857+0000 7fe48d600700 1 --2- 172.21.15.28:0/2006342906 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe4780a3c40 0x7fe4780a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.857+0000 7fe48d600700 1 -- 172.21.15.28:0/2006342906 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe47c0d1160 con 0x7fe490075640 2024-04-24T08:09:20.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.857+0000 7fe495000700 1 --2- 172.21.15.28:0/2006342906 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe4780a3c40 0x7fe4780a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:20.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.858+0000 7fe495000700 1 --2- 172.21.15.28:0/2006342906 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe4780a3c40 0x7fe4780a6100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe484009c60 tx=0x7fe484009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:20.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:20.861+0000 7fe48d600700 1 -- 172.21.15.28:0/2006342906 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe47c05dff0 con 0x7fe490075640 2024-04-24T08:09:21.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:21.092+0000 7fe497a00700 1 -- 172.21.15.28:0/2006342906 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fe49019c870 con 0x7fe4780a3c40 2024-04-24T08:09:21.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:21.109+0000 7fe48d600700 1 -- 172.21.15.28:0/2006342906 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258932 (secure 0 0 0) 0x7fe49019c870 con 0x7fe4780a3c40 2024-04-24T08:09:21.116 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:09:21.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:21.116+0000 7fe497a00700 1 -- 172.21.15.28:0/2006342906 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe4780a3c40 msgr2=0x7fe4780a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:21.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:21.116+0000 7fe497a00700 1 --2- 172.21.15.28:0/2006342906 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe4780a3c40 0x7fe4780a6100 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe484009c60 tx=0x7fe484009260 comp rx=0 tx=0).stop 2024-04-24T08:09:21.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:21.116+0000 7fe497a00700 1 -- 172.21.15.28:0/2006342906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe490075640 msgr2=0x7fe49019ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:21.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:21.116+0000 7fe497a00700 1 --2- 172.21.15.28:0/2006342906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe490075640 0x7fe49019ba50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fe47c000c00 tx=0x7fe47c00bab0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:21.116+0000 7fe497a00700 1 -- 172.21.15.28:0/2006342906 shutdown_connections 2024-04-24T08:09:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:21.116+0000 7fe497a00700 1 --2- 172.21.15.28:0/2006342906 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe4780a3c40 0x7fe4780a6100 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:21.116+0000 7fe497a00700 1 --2- 172.21.15.28:0/2006342906 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe49010f4f0 0x7fe49019fde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:21.116+0000 7fe497a00700 1 --2- 172.21.15.28:0/2006342906 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe490074d20 0x7fe49019b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:21.116+0000 7fe497a00700 1 --2- 172.21.15.28:0/2006342906 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe490075640 0x7fe49019ba50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:21.116+0000 7fe497a00700 1 -- 172.21.15.28:0/2006342906 >> 172.21.15.28:0/2006342906 conn(0x7fe490079c50 msgr2=0x7fe4900ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:21.116+0000 7fe497a00700 1 -- 172.21.15.28:0/2006342906 shutdown_connections 2024-04-24T08:09:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:21.116+0000 7fe497a00700 1 -- 172.21.15.28:0/2006342906 wait complete. 2024-04-24T08:09:21.119 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-24T08:09:21.404 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:21 smithi028 ceph-mon[27786]: pgmap v363: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:09:21.404 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:21 smithi028 ceph-mon[27786]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-24T08:09:21.404 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:21 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:09:21.404 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:21 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:09:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:21 smithi117 ceph-mon[32438]: pgmap v363: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:09:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:21 smithi117 ceph-mon[32438]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-24T08:09:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:21 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:09:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:21 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:09:21.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:21 smithi194 ceph-mon[32836]: pgmap v363: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:09:21.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:21 smithi194 ceph-mon[32836]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-04-24T08:09:21.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:21 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:09:21.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:21 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:09:22.073 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":363,"stamp":"2024-04-24T08:09:20.095957+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":653874,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":651,"num_read_kb":442,"num_write":343,"num_write_kb":3048,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1181352,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":276,"ondisk_log_size":276,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360944,"kb_used_data":11712,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580880,"statfs":{"total":1151940427776,"available":1151570821120,"internally_reserved":0,"allocated":11993088,"data_stored":7500762,"data_compressed":177906,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":19140,"internal_metadata":357414204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":75,"num_read_kb":35,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005668"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.166980+0000","last_change":"2024-04-24T08:07:52.074319+0000","last_active":"2024-04-24T08:08:51.166980+0000","last_peered":"2024-04-24T08:08:51.166980+0000","last_clean":"2024-04-24T08:08:51.166980+0000","last_became_active":"2024-04-24T08:07:52.073923+0000","last_became_peered":"2024-04-24T08:07:52.073923+0000","last_unstale":"2024-04-24T08:08:51.166980+0000","last_undegraded":"2024-04-24T08:08:51.166980+0000","last_fullsized":"2024-04-24T08:08:51.166980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:29.325904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.163288+0000","last_change":"2024-04-24T08:07:52.068316+0000","last_active":"2024-04-24T08:08:51.163288+0000","last_peered":"2024-04-24T08:08:51.163288+0000","last_clean":"2024-04-24T08:08:51.163288+0000","last_became_active":"2024-04-24T08:07:52.067961+0000","last_became_peered":"2024-04-24T08:07:52.067961+0000","last_unstale":"2024-04-24T08:08:51.163288+0000","last_undegraded":"2024-04-24T08:08:51.163288+0000","last_fullsized":"2024-04-24T08:08:51.163288+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:35:14.079348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.165619+0000","last_change":"2024-04-24T08:07:52.066870+0000","last_active":"2024-04-24T08:08:51.165619+0000","last_peered":"2024-04-24T08:08:51.165619+0000","last_clean":"2024-04-24T08:08:51.165619+0000","last_became_active":"2024-04-24T08:07:52.066481+0000","last_became_peered":"2024-04-24T08:07:52.066481+0000","last_unstale":"2024-04-24T08:08:51.165619+0000","last_undegraded":"2024-04-24T08:08:51.165619+0000","last_fullsized":"2024-04-24T08:08:51.165619+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:25:45.526667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.168801+0000","last_change":"2024-04-24T08:07:52.061548+0000","last_active":"2024-04-24T08:08:51.168801+0000","last_peered":"2024-04-24T08:08:51.168801+0000","last_clean":"2024-04-24T08:08:51.168801+0000","last_became_active":"2024-04-24T08:07:52.060519+0000","last_became_peered":"2024-04-24T08:07:52.060519+0000","last_unstale":"2024-04-24T08:08:51.168801+0000","last_undegraded":"2024-04-24T08:08:51.168801+0000","last_fullsized":"2024-04-24T08:08:51.168801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:47:20.952035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.172050+0000","last_change":"2024-04-24T08:07:52.063805+0000","last_active":"2024-04-24T08:08:51.172050+0000","last_peered":"2024-04-24T08:08:51.172050+0000","last_clean":"2024-04-24T08:08:51.172050+0000","last_became_active":"2024-04-24T08:07:52.063414+0000","last_became_peered":"2024-04-24T08:07:52.063414+0000","last_unstale":"2024-04-24T08:08:51.172050+0000","last_undegraded":"2024-04-24T08:08:51.172050+0000","last_fullsized":"2024-04-24T08:08:51.172050+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:06:24.465035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.164236+0000","last_change":"2024-04-24T08:07:52.050152+0000","last_active":"2024-04-24T08:08:51.164236+0000","last_peered":"2024-04-24T08:08:51.164236+0000","last_clean":"2024-04-24T08:08:51.164236+0000","last_became_active":"2024-04-24T08:07:52.049805+0000","last_became_peered":"2024-04-24T08:07:52.049805+0000","last_unstale":"2024-04-24T08:08:51.164236+0000","last_undegraded":"2024-04-24T08:08:51.164236+0000","last_fullsized":"2024-04-24T08:08:51.164236+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:47:34.644263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.585368+0000","last_change":"2024-04-24T08:07:52.055726+0000","last_active":"2024-04-24T08:08:51.585368+0000","last_peered":"2024-04-24T08:08:51.585368+0000","last_clean":"2024-04-24T08:08:51.585368+0000","last_became_active":"2024-04-24T08:07:52.054563+0000","last_became_peered":"2024-04-24T08:07:52.054563+0000","last_unstale":"2024-04-24T08:08:51.585368+0000","last_undegraded":"2024-04-24T08:08:51.585368+0000","last_fullsized":"2024-04-24T08:08:51.585368+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:34:57.430232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.174787+0000","last_change":"2024-04-24T08:07:52.053672+0000","last_active":"2024-04-24T08:08:51.174787+0000","last_peered":"2024-04-24T08:08:51.174787+0000","last_clean":"2024-04-24T08:08:51.174787+0000","last_became_active":"2024-04-24T08:07:52.053150+0000","last_became_peered":"2024-04-24T08:07:52.053150+0000","last_unstale":"2024-04-24T08:08:51.174787+0000","last_undegraded":"2024-04-24T08:08:51.174787+0000","last_fullsized":"2024-04-24T08:08:51.174787+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:00:12.272555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.168943+0000","last_change":"2024-04-24T08:07:52.054648+0000","last_active":"2024-04-24T08:08:51.168943+0000","last_peered":"2024-04-24T08:08:51.168943+0000","last_clean":"2024-04-24T08:08:51.168943+0000","last_became_active":"2024-04-24T08:07:52.054301+0000","last_became_peered":"2024-04-24T08:07:52.054301+0000","last_unstale":"2024-04-24T08:08:51.168943+0000","last_undegraded":"2024-04-24T08:08:51.168943+0000","last_fullsized":"2024-04-24T08:08:51.168943+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:52:20.533855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.164558+0000","last_change":"2024-04-24T08:07:52.052994+0000","last_active":"2024-04-24T08:08:51.164558+0000","last_peered":"2024-04-24T08:08:51.164558+0000","last_clean":"2024-04-24T08:08:51.164558+0000","last_became_active":"2024-04-24T08:07:52.052641+0000","last_became_peered":"2024-04-24T08:07:52.052641+0000","last_unstale":"2024-04-24T08:08:51.164558+0000","last_undegraded":"2024-04-24T08:08:51.164558+0000","last_fullsized":"2024-04-24T08:08:51.164558+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:02.665622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.585635+0000","last_change":"2024-04-24T08:07:52.055901+0000","last_active":"2024-04-24T08:08:51.585635+0000","last_peered":"2024-04-24T08:08:51.585635+0000","last_clean":"2024-04-24T08:08:51.585635+0000","last_became_active":"2024-04-24T08:07:52.055277+0000","last_became_peered":"2024-04-24T08:07:52.055277+0000","last_unstale":"2024-04-24T08:08:51.585635+0000","last_undegraded":"2024-04-24T08:08:51.585635+0000","last_fullsized":"2024-04-24T08:08:51.585635+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:46.950578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.168645+0000","last_change":"2024-04-24T08:07:52.060329+0000","last_active":"2024-04-24T08:08:51.168645+0000","last_peered":"2024-04-24T08:08:51.168645+0000","last_clean":"2024-04-24T08:08:51.168645+0000","last_became_active":"2024-04-24T08:07:52.059914+0000","last_became_peered":"2024-04-24T08:07:52.059914+0000","last_unstale":"2024-04-24T08:08:51.168645+0000","last_undegraded":"2024-04-24T08:08:51.168645+0000","last_fullsized":"2024-04-24T08:08:51.168645+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:15:20.263916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.249229+0000","last_change":"2024-04-24T08:07:52.062108+0000","last_active":"2024-04-24T08:08:51.249229+0000","last_peered":"2024-04-24T08:08:51.249229+0000","last_clean":"2024-04-24T08:08:51.249229+0000","last_became_active":"2024-04-24T08:07:52.061665+0000","last_became_peered":"2024-04-24T08:07:52.061665+0000","last_unstale":"2024-04-24T08:08:51.249229+0000","last_undegraded":"2024-04-24T08:08:51.249229+0000","last_fullsized":"2024-04-24T08:08:51.249229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:55:58.071807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.164310+0000","last_change":"2024-04-24T08:07:52.063033+0000","last_active":"2024-04-24T08:08:51.164310+0000","last_peered":"2024-04-24T08:08:51.164310+0000","last_clean":"2024-04-24T08:08:51.164310+0000","last_became_active":"2024-04-24T08:07:52.062610+0000","last_became_peered":"2024-04-24T08:07:52.062610+0000","last_unstale":"2024-04-24T08:08:51.164310+0000","last_undegraded":"2024-04-24T08:08:51.164310+0000","last_fullsized":"2024-04-24T08:08:51.164310+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:59:35.906720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.163840+0000","last_change":"2024-04-24T08:07:52.064657+0000","last_active":"2024-04-24T08:08:51.163840+0000","last_peered":"2024-04-24T08:08:51.163840+0000","last_clean":"2024-04-24T08:08:51.163840+0000","last_became_active":"2024-04-24T08:07:52.064188+0000","last_became_peered":"2024-04-24T08:07:52.064188+0000","last_unstale":"2024-04-24T08:08:51.163840+0000","last_undegraded":"2024-04-24T08:08:51.163840+0000","last_fullsized":"2024-04-24T08:08:51.163840+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:56:32.139420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.167983+0000","last_change":"2024-04-24T08:07:52.067227+0000","last_active":"2024-04-24T08:08:51.167983+0000","last_peered":"2024-04-24T08:08:51.167983+0000","last_clean":"2024-04-24T08:08:51.167983+0000","last_became_active":"2024-04-24T08:07:52.066854+0000","last_became_peered":"2024-04-24T08:07:52.066854+0000","last_unstale":"2024-04-24T08:08:51.167983+0000","last_undegraded":"2024-04-24T08:08:51.167983+0000","last_fullsized":"2024-04-24T08:08:51.167983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:25:17.580223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.166830+0000","last_change":"2024-04-24T08:07:52.074127+0000","last_active":"2024-04-24T08:08:51.166830+0000","last_peered":"2024-04-24T08:08:51.166830+0000","last_clean":"2024-04-24T08:08:51.166830+0000","last_became_active":"2024-04-24T08:07:52.073806+0000","last_became_peered":"2024-04-24T08:07:52.073806+0000","last_unstale":"2024-04-24T08:08:51.166830+0000","last_undegraded":"2024-04-24T08:08:51.166830+0000","last_fullsized":"2024-04-24T08:08:51.166830+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:25:22.068997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.165003+0000","last_change":"2024-04-24T08:07:52.051239+0000","last_active":"2024-04-24T08:08:51.165003+0000","last_peered":"2024-04-24T08:08:51.165003+0000","last_clean":"2024-04-24T08:08:51.165003+0000","last_became_active":"2024-04-24T08:07:52.049232+0000","last_became_peered":"2024-04-24T08:07:52.049232+0000","last_unstale":"2024-04-24T08:08:51.165003+0000","last_undegraded":"2024-04-24T08:08:51.165003+0000","last_fullsized":"2024-04-24T08:08:51.165003+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:07:39.768537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.169180+0000","last_change":"2024-04-24T08:07:52.062119+0000","last_active":"2024-04-24T08:08:51.169180+0000","last_peered":"2024-04-24T08:08:51.169180+0000","last_clean":"2024-04-24T08:08:51.169180+0000","last_became_active":"2024-04-24T08:07:52.061781+0000","last_became_peered":"2024-04-24T08:07:52.061781+0000","last_unstale":"2024-04-24T08:08:51.169180+0000","last_undegraded":"2024-04-24T08:08:51.169180+0000","last_fullsized":"2024-04-24T08:08:51.169180+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:49:41.595368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.166665+0000","last_change":"2024-04-24T08:07:51.052411+0000","last_active":"2024-04-24T08:08:51.166665+0000","last_peered":"2024-04-24T08:08:51.166665+0000","last_clean":"2024-04-24T08:08:51.166665+0000","last_became_active":"2024-04-24T08:07:51.051704+0000","last_became_peered":"2024-04-24T08:07:51.051704+0000","last_unstale":"2024-04-24T08:08:51.166665+0000","last_undegraded":"2024-04-24T08:08:51.166665+0000","last_fullsized":"2024-04-24T08:08:51.166665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:34:23.001338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.177407+0000","last_change":"2024-04-24T08:07:52.056508+0000","last_active":"2024-04-24T08:08:51.177407+0000","last_peered":"2024-04-24T08:08:51.177407+0000","last_clean":"2024-04-24T08:08:51.177407+0000","last_became_active":"2024-04-24T08:07:52.056158+0000","last_became_peered":"2024-04-24T08:07:52.056158+0000","last_unstale":"2024-04-24T08:08:51.177407+0000","last_undegraded":"2024-04-24T08:08:51.177407+0000","last_fullsized":"2024-04-24T08:08:51.177407+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:52.267984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.178662+0000","last_change":"2024-04-24T08:07:51.080403+0000","last_active":"2024-04-24T08:08:51.178662+0000","last_peered":"2024-04-24T08:08:51.178662+0000","last_clean":"2024-04-24T08:08:51.178662+0000","last_became_active":"2024-04-24T08:07:51.079158+0000","last_became_peered":"2024-04-24T08:07:51.079158+0000","last_unstale":"2024-04-24T08:08:51.178662+0000","last_undegraded":"2024-04-24T08:08:51.178662+0000","last_fullsized":"2024-04-24T08:08:51.178662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:49:59.491850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.165290+0000","last_change":"2024-04-24T08:07:52.066963+0000","last_active":"2024-04-24T08:08:51.165290+0000","last_peered":"2024-04-24T08:08:51.165290+0000","last_clean":"2024-04-24T08:08:51.165290+0000","last_became_active":"2024-04-24T08:07:52.065475+0000","last_became_peered":"2024-04-24T08:07:52.065475+0000","last_unstale":"2024-04-24T08:08:51.165290+0000","last_undegraded":"2024-04-24T08:08:51.165290+0000","last_fullsized":"2024-04-24T08:08:51.165290+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:28:34.220200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.172343+0000","last_change":"2024-04-24T08:07:51.042772+0000","last_active":"2024-04-24T08:08:51.172343+0000","last_peered":"2024-04-24T08:08:51.172343+0000","last_clean":"2024-04-24T08:08:51.172343+0000","last_became_active":"2024-04-24T08:07:51.042128+0000","last_became_peered":"2024-04-24T08:07:51.042128+0000","last_unstale":"2024-04-24T08:08:51.172343+0000","last_undegraded":"2024-04-24T08:08:51.172343+0000","last_fullsized":"2024-04-24T08:08:51.172343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:10:29.207991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.167643+0000","last_change":"2024-04-24T08:07:52.062411+0000","last_active":"2024-04-24T08:08:51.167643+0000","last_peered":"2024-04-24T08:08:51.167643+0000","last_clean":"2024-04-24T08:08:51.167643+0000","last_became_active":"2024-04-24T08:07:52.062037+0000","last_became_peered":"2024-04-24T08:07:52.062037+0000","last_unstale":"2024-04-24T08:08:51.167643+0000","last_undegraded":"2024-04-24T08:08:51.167643+0000","last_fullsized":"2024-04-24T08:08:51.167643+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:16:17.463874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.172264+0000","last_change":"2024-04-24T08:07:51.046046+0000","last_active":"2024-04-24T08:08:51.172264+0000","last_peered":"2024-04-24T08:08:51.172264+0000","last_clean":"2024-04-24T08:08:51.172264+0000","last_became_active":"2024-04-24T08:07:51.045633+0000","last_became_peered":"2024-04-24T08:07:51.045633+0000","last_unstale":"2024-04-24T08:08:51.172264+0000","last_undegraded":"2024-04-24T08:08:51.172264+0000","last_fullsized":"2024-04-24T08:08:51.172264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:18:20.529234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.168988+0000","last_change":"2024-04-24T08:07:52.061167+0000","last_active":"2024-04-24T08:08:51.168988+0000","last_peered":"2024-04-24T08:08:51.168988+0000","last_clean":"2024-04-24T08:08:51.168988+0000","last_became_active":"2024-04-24T08:07:52.060841+0000","last_became_peered":"2024-04-24T08:07:52.060841+0000","last_unstale":"2024-04-24T08:08:51.168988+0000","last_undegraded":"2024-04-24T08:08:51.168988+0000","last_fullsized":"2024-04-24T08:08:51.168988+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:04:27.048760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'6","reported_seq":110,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:17.266716+0000","last_change":"2024-04-24T08:07:51.044274+0000","last_active":"2024-04-24T08:09:17.266716+0000","last_peered":"2024-04-24T08:09:17.266716+0000","last_clean":"2024-04-24T08:09:17.266716+0000","last_became_active":"2024-04-24T08:07:51.043938+0000","last_became_peered":"2024-04-24T08:07:51.043938+0000","last_unstale":"2024-04-24T08:09:17.266716+0000","last_undegraded":"2024-04-24T08:09:17.266716+0000","last_fullsized":"2024-04-24T08:09:17.266716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:05:38.349981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1013,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":85,"num_read_kb":3,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.171941+0000","last_change":"2024-04-24T08:07:52.055777+0000","last_active":"2024-04-24T08:08:51.171941+0000","last_peered":"2024-04-24T08:08:51.171941+0000","last_clean":"2024-04-24T08:08:51.171941+0000","last_became_active":"2024-04-24T08:07:52.055415+0000","last_became_peered":"2024-04-24T08:07:52.055415+0000","last_unstale":"2024-04-24T08:08:51.171941+0000","last_undegraded":"2024-04-24T08:08:51.171941+0000","last_fullsized":"2024-04-24T08:08:51.171941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:18:31.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.164174+0000","last_change":"2024-04-24T08:07:51.072513+0000","last_active":"2024-04-24T08:08:51.164174+0000","last_peered":"2024-04-24T08:08:51.164174+0000","last_clean":"2024-04-24T08:08:51.164174+0000","last_became_active":"2024-04-24T08:07:51.072249+0000","last_became_peered":"2024-04-24T08:07:51.072249+0000","last_unstale":"2024-04-24T08:08:51.164174+0000","last_undegraded":"2024-04-24T08:08:51.164174+0000","last_fullsized":"2024-04-24T08:08:51.164174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:45:07.112617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.587130+0000","last_change":"2024-04-24T08:08:51.587130+0000","last_active":"2024-04-24T08:08:51.587130+0000","last_peered":"2024-04-24T08:08:51.587130+0000","last_clean":"2024-04-24T08:08:51.587130+0000","last_became_active":"2024-04-24T08:08:51.586645+0000","last_became_peered":"2024-04-24T08:08:51.586645+0000","last_unstale":"2024-04-24T08:08:51.587130+0000","last_undegraded":"2024-04-24T08:08:51.587130+0000","last_fullsized":"2024-04-24T08:08:51.587130+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:24:14.677548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.249085+0000","last_change":"2024-04-24T08:07:51.070901+0000","last_active":"2024-04-24T08:08:51.249085+0000","last_peered":"2024-04-24T08:08:51.249085+0000","last_clean":"2024-04-24T08:08:51.249085+0000","last_became_active":"2024-04-24T08:07:51.067178+0000","last_became_peered":"2024-04-24T08:07:51.067178+0000","last_unstale":"2024-04-24T08:08:51.249085+0000","last_undegraded":"2024-04-24T08:08:51.249085+0000","last_fullsized":"2024-04-24T08:08:51.249085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:24:26.807786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.174624+0000","last_change":"2024-04-24T08:07:52.079493+0000","last_active":"2024-04-24T08:08:51.174624+0000","last_peered":"2024-04-24T08:08:51.174624+0000","last_clean":"2024-04-24T08:08:51.174624+0000","last_became_active":"2024-04-24T08:07:52.077266+0000","last_became_peered":"2024-04-24T08:07:52.077266+0000","last_unstale":"2024-04-24T08:08:51.174624+0000","last_undegraded":"2024-04-24T08:08:51.174624+0000","last_fullsized":"2024-04-24T08:08:51.174624+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:40:45.660146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.170025+0000","last_change":"2024-04-24T08:07:51.045963+0000","last_active":"2024-04-24T08:08:51.170025+0000","last_peered":"2024-04-24T08:08:51.170025+0000","last_clean":"2024-04-24T08:08:51.170025+0000","last_became_active":"2024-04-24T08:07:51.045641+0000","last_became_peered":"2024-04-24T08:07:51.045641+0000","last_unstale":"2024-04-24T08:08:51.170025+0000","last_undegraded":"2024-04-24T08:08:51.170025+0000","last_fullsized":"2024-04-24T08:08:51.170025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:07:59.646155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.164364+0000","last_change":"2024-04-24T08:07:52.057280+0000","last_active":"2024-04-24T08:08:51.164364+0000","last_peered":"2024-04-24T08:08:51.164364+0000","last_clean":"2024-04-24T08:08:51.164364+0000","last_became_active":"2024-04-24T08:07:52.054944+0000","last_became_peered":"2024-04-24T08:07:52.054944+0000","last_unstale":"2024-04-24T08:08:51.164364+0000","last_undegraded":"2024-04-24T08:08:51.164364+0000","last_fullsized":"2024-04-24T08:08:51.164364+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:22:45.911472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'3","reported_seq":103,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:16.264463+0000","last_change":"2024-04-24T08:07:51.048898+0000","last_active":"2024-04-24T08:09:16.264463+0000","last_peered":"2024-04-24T08:09:16.264463+0000","last_clean":"2024-04-24T08:09:16.264463+0000","last_became_active":"2024-04-24T08:07:51.048585+0000","last_became_peered":"2024-04-24T08:07:51.048585+0000","last_unstale":"2024-04-24T08:09:16.264463+0000","last_undegraded":"2024-04-24T08:09:16.264463+0000","last_fullsized":"2024-04-24T08:09:16.264463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:40:30.413541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":81,"num_read_kb":81,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.169063+0000","last_change":"2024-04-24T08:07:52.048093+0000","last_active":"2024-04-24T08:08:51.169063+0000","last_peered":"2024-04-24T08:08:51.169063+0000","last_clean":"2024-04-24T08:08:51.169063+0000","last_became_active":"2024-04-24T08:07:52.047727+0000","last_became_peered":"2024-04-24T08:07:52.047727+0000","last_unstale":"2024-04-24T08:08:51.169063+0000","last_undegraded":"2024-04-24T08:08:51.169063+0000","last_fullsized":"2024-04-24T08:08:51.169063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:57:26.248355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.585801+0000","last_change":"2024-04-24T08:07:51.047810+0000","last_active":"2024-04-24T08:08:51.585801+0000","last_peered":"2024-04-24T08:08:51.585801+0000","last_clean":"2024-04-24T08:08:51.585801+0000","last_became_active":"2024-04-24T08:07:51.047501+0000","last_became_peered":"2024-04-24T08:07:51.047501+0000","last_unstale":"2024-04-24T08:08:51.585801+0000","last_undegraded":"2024-04-24T08:08:51.585801+0000","last_fullsized":"2024-04-24T08:08:51.585801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:02:27.585202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.163853+0000","last_change":"2024-04-24T08:07:52.063782+0000","last_active":"2024-04-24T08:08:51.163853+0000","last_peered":"2024-04-24T08:08:51.163853+0000","last_clean":"2024-04-24T08:08:51.163853+0000","last_became_active":"2024-04-24T08:07:52.063056+0000","last_became_peered":"2024-04-24T08:07:52.063056+0000","last_unstale":"2024-04-24T08:08:51.163853+0000","last_undegraded":"2024-04-24T08:08:51.163853+0000","last_fullsized":"2024-04-24T08:08:51.163853+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:41:30.966787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.169138+0000","last_change":"2024-04-24T08:07:51.076139+0000","last_active":"2024-04-24T08:08:51.169138+0000","last_peered":"2024-04-24T08:08:51.169138+0000","last_clean":"2024-04-24T08:08:51.169138+0000","last_became_active":"2024-04-24T08:07:51.075842+0000","last_became_peered":"2024-04-24T08:07:51.075842+0000","last_unstale":"2024-04-24T08:08:51.169138+0000","last_undegraded":"2024-04-24T08:08:51.169138+0000","last_fullsized":"2024-04-24T08:08:51.169138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:46:56.741726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.250881+0000","last_change":"2024-04-24T08:08:51.250881+0000","last_active":"2024-04-24T08:08:51.250881+0000","last_peered":"2024-04-24T08:08:51.250881+0000","last_clean":"2024-04-24T08:08:51.250881+0000","last_became_active":"2024-04-24T08:08:51.250519+0000","last_became_peered":"2024-04-24T08:08:51.250519+0000","last_unstale":"2024-04-24T08:08:51.250881+0000","last_undegraded":"2024-04-24T08:08:51.250881+0000","last_fullsized":"2024-04-24T08:08:51.250881+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:23:04.805687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'28","reported_seq":216,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:17.266157+0000","last_change":"2024-04-24T08:07:51.048079+0000","last_active":"2024-04-24T08:09:17.266157+0000","last_peered":"2024-04-24T08:09:17.266157+0000","last_clean":"2024-04-24T08:09:17.266157+0000","last_became_active":"2024-04-24T08:07:51.047584+0000","last_became_peered":"2024-04-24T08:07:51.047584+0000","last_unstale":"2024-04-24T08:09:17.266157+0000","last_undegraded":"2024-04-24T08:09:17.266157+0000","last_fullsized":"2024-04-24T08:09:17.266157+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":28,"log_dups_size":0,"ondisk_log_size":28,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:09:05.554984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":58880,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":169,"num_read_kb":141,"num_write":29,"num_write_kb":67,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.172106+0000","last_change":"2024-04-24T08:07:52.055894+0000","last_active":"2024-04-24T08:08:51.172106+0000","last_peered":"2024-04-24T08:08:51.172106+0000","last_clean":"2024-04-24T08:08:51.172106+0000","last_became_active":"2024-04-24T08:07:52.055548+0000","last_became_peered":"2024-04-24T08:07:52.055548+0000","last_unstale":"2024-04-24T08:08:51.172106+0000","last_undegraded":"2024-04-24T08:08:51.172106+0000","last_fullsized":"2024-04-24T08:08:51.172106+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:33:22.609865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.188459+0000","last_change":"2024-04-24T08:08:51.188459+0000","last_active":"2024-04-24T08:08:51.188459+0000","last_peered":"2024-04-24T08:08:51.188459+0000","last_clean":"2024-04-24T08:08:51.188459+0000","last_became_active":"2024-04-24T08:08:51.180387+0000","last_became_peered":"2024-04-24T08:08:51.180387+0000","last_unstale":"2024-04-24T08:08:51.188459+0000","last_undegraded":"2024-04-24T08:08:51.188459+0000","last_fullsized":"2024-04-24T08:08:51.188459+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:34:56.592978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.248885+0000","last_change":"2024-04-24T08:07:52.069950+0000","last_active":"2024-04-24T08:08:51.248885+0000","last_peered":"2024-04-24T08:08:51.248885+0000","last_clean":"2024-04-24T08:08:51.248885+0000","last_became_active":"2024-04-24T08:07:52.069515+0000","last_became_peered":"2024-04-24T08:07:52.069515+0000","last_unstale":"2024-04-24T08:08:51.248885+0000","last_undegraded":"2024-04-24T08:08:51.248885+0000","last_fullsized":"2024-04-24T08:08:51.248885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:52:27.530541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.168866+0000","last_change":"2024-04-24T08:07:51.067062+0000","last_active":"2024-04-24T08:08:51.168866+0000","last_peered":"2024-04-24T08:08:51.168866+0000","last_clean":"2024-04-24T08:08:51.168866+0000","last_became_active":"2024-04-24T08:07:51.066712+0000","last_became_peered":"2024-04-24T08:07:51.066712+0000","last_unstale":"2024-04-24T08:08:51.168866+0000","last_undegraded":"2024-04-24T08:08:51.168866+0000","last_fullsized":"2024-04-24T08:08:51.168866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:11:52.974470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.585606+0000","last_change":"2024-04-24T08:07:52.064521+0000","last_active":"2024-04-24T08:08:51.585606+0000","last_peered":"2024-04-24T08:08:51.585606+0000","last_clean":"2024-04-24T08:08:51.585606+0000","last_became_active":"2024-04-24T08:07:52.064184+0000","last_became_peered":"2024-04-24T08:07:52.064184+0000","last_unstale":"2024-04-24T08:08:51.585606+0000","last_undegraded":"2024-04-24T08:08:51.585606+0000","last_fullsized":"2024-04-24T08:08:51.585606+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:00:27.294441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'4","reported_seq":108,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:17.299257+0000","last_change":"2024-04-24T08:07:51.043577+0000","last_active":"2024-04-24T08:09:17.299257+0000","last_peered":"2024-04-24T08:09:17.299257+0000","last_clean":"2024-04-24T08:09:17.299257+0000","last_became_active":"2024-04-24T08:07:51.043215+0000","last_became_peered":"2024-04-24T08:07:51.043215+0000","last_unstale":"2024-04-24T08:09:17.299257+0000","last_undegraded":"2024-04-24T08:09:17.299257+0000","last_fullsized":"2024-04-24T08:09:17.299257+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:25:09.979236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":979,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":4,"num_write":3,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.585284+0000","last_change":"2024-04-24T08:07:52.057633+0000","last_active":"2024-04-24T08:08:51.585284+0000","last_peered":"2024-04-24T08:08:51.585284+0000","last_clean":"2024-04-24T08:08:51.585284+0000","last_became_active":"2024-04-24T08:07:52.057309+0000","last_became_peered":"2024-04-24T08:07:52.057309+0000","last_unstale":"2024-04-24T08:08:51.585284+0000","last_undegraded":"2024-04-24T08:08:51.585284+0000","last_fullsized":"2024-04-24T08:08:51.585284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:49:56.045030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.171746+0000","last_change":"2024-04-24T08:07:51.058957+0000","last_active":"2024-04-24T08:08:51.171746+0000","last_peered":"2024-04-24T08:08:51.171746+0000","last_clean":"2024-04-24T08:08:51.171746+0000","last_became_active":"2024-04-24T08:07:51.058654+0000","last_became_peered":"2024-04-24T08:07:51.058654+0000","last_unstale":"2024-04-24T08:08:51.171746+0000","last_undegraded":"2024-04-24T08:08:51.171746+0000","last_fullsized":"2024-04-24T08:08:51.171746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:13.354590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.168911+0000","last_change":"2024-04-24T08:07:52.051052+0000","last_active":"2024-04-24T08:08:51.168911+0000","last_peered":"2024-04-24T08:08:51.168911+0000","last_clean":"2024-04-24T08:08:51.168911+0000","last_became_active":"2024-04-24T08:07:52.050685+0000","last_became_peered":"2024-04-24T08:07:52.050685+0000","last_unstale":"2024-04-24T08:08:51.168911+0000","last_undegraded":"2024-04-24T08:08:51.168911+0000","last_fullsized":"2024-04-24T08:08:51.168911+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:34.491653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.172455+0000","last_change":"2024-04-24T08:07:51.041088+0000","last_active":"2024-04-24T08:08:51.172455+0000","last_peered":"2024-04-24T08:08:51.172455+0000","last_clean":"2024-04-24T08:08:51.172455+0000","last_became_active":"2024-04-24T08:07:51.040752+0000","last_became_peered":"2024-04-24T08:07:51.040752+0000","last_unstale":"2024-04-24T08:08:51.172455+0000","last_undegraded":"2024-04-24T08:08:51.172455+0000","last_fullsized":"2024-04-24T08:08:51.172455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:48:23.405705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.173331+0000","last_change":"2024-04-24T08:08:51.173331+0000","last_active":"2024-04-24T08:08:51.173331+0000","last_peered":"2024-04-24T08:08:51.173331+0000","last_clean":"2024-04-24T08:08:51.173331+0000","last_became_active":"2024-04-24T08:08:51.172912+0000","last_became_peered":"2024-04-24T08:08:51.172912+0000","last_unstale":"2024-04-24T08:08:51.173331+0000","last_undegraded":"2024-04-24T08:08:51.173331+0000","last_fullsized":"2024-04-24T08:08:51.173331+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:54:35.241315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.164883+0000","last_change":"2024-04-24T08:07:51.075583+0000","last_active":"2024-04-24T08:08:51.164883+0000","last_peered":"2024-04-24T08:08:51.164883+0000","last_clean":"2024-04-24T08:08:51.164883+0000","last_became_active":"2024-04-24T08:07:51.075111+0000","last_became_peered":"2024-04-24T08:07:51.075111+0000","last_unstale":"2024-04-24T08:08:51.164883+0000","last_undegraded":"2024-04-24T08:08:51.164883+0000","last_fullsized":"2024-04-24T08:08:51.164883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:56:00.820057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.164517+0000","last_change":"2024-04-24T08:07:52.063232+0000","last_active":"2024-04-24T08:08:51.164517+0000","last_peered":"2024-04-24T08:08:51.164517+0000","last_clean":"2024-04-24T08:08:51.164517+0000","last_became_active":"2024-04-24T08:07:52.062868+0000","last_became_peered":"2024-04-24T08:07:52.062868+0000","last_unstale":"2024-04-24T08:08:51.164517+0000","last_undegraded":"2024-04-24T08:08:51.164517+0000","last_fullsized":"2024-04-24T08:08:51.164517+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:00:40.333389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.585498+0000","last_change":"2024-04-24T08:07:51.070647+0000","last_active":"2024-04-24T08:08:51.585498+0000","last_peered":"2024-04-24T08:08:51.585498+0000","last_clean":"2024-04-24T08:08:51.585498+0000","last_became_active":"2024-04-24T08:07:51.070298+0000","last_became_peered":"2024-04-24T08:07:51.070298+0000","last_unstale":"2024-04-24T08:08:51.585498+0000","last_undegraded":"2024-04-24T08:08:51.585498+0000","last_fullsized":"2024-04-24T08:08:51.585498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:56:41.706796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.164549+0000","last_change":"2024-04-24T08:07:52.073417+0000","last_active":"2024-04-24T08:08:51.164549+0000","last_peered":"2024-04-24T08:08:51.164549+0000","last_clean":"2024-04-24T08:08:51.164549+0000","last_became_active":"2024-04-24T08:07:52.073051+0000","last_became_peered":"2024-04-24T08:07:52.073051+0000","last_unstale":"2024-04-24T08:08:51.164549+0000","last_undegraded":"2024-04-24T08:08:51.164549+0000","last_fullsized":"2024-04-24T08:08:51.164549+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:54:58.338279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":11,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.174286+0000","last_change":"2024-04-24T08:08:51.172525+0000","last_active":"2024-04-24T08:08:51.174286+0000","last_peered":"2024-04-24T08:08:51.174286+0000","last_clean":"2024-04-24T08:08:51.174286+0000","last_became_active":"2024-04-24T08:08:51.171924+0000","last_became_peered":"2024-04-24T08:08:51.171924+0000","last_unstale":"2024-04-24T08:08:51.174286+0000","last_undegraded":"2024-04-24T08:08:51.174286+0000","last_fullsized":"2024-04-24T08:08:51.174286+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:25:42.067150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":105,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:17.298566+0000","last_change":"2024-04-24T08:07:51.076937+0000","last_active":"2024-04-24T08:09:17.298566+0000","last_peered":"2024-04-24T08:09:17.298566+0000","last_clean":"2024-04-24T08:09:17.298566+0000","last_became_active":"2024-04-24T08:07:51.076621+0000","last_became_peered":"2024-04-24T08:07:51.076621+0000","last_unstale":"2024-04-24T08:09:17.298566+0000","last_undegraded":"2024-04-24T08:09:17.298566+0000","last_fullsized":"2024-04-24T08:09:17.298566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:09.966994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":82,"num_read_kb":82,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.176371+0000","last_change":"2024-04-24T08:07:52.054150+0000","last_active":"2024-04-24T08:08:51.176371+0000","last_peered":"2024-04-24T08:08:51.176371+0000","last_clean":"2024-04-24T08:08:51.176371+0000","last_became_active":"2024-04-24T08:07:52.053817+0000","last_became_peered":"2024-04-24T08:07:52.053817+0000","last_unstale":"2024-04-24T08:08:51.176371+0000","last_undegraded":"2024-04-24T08:08:51.176371+0000","last_fullsized":"2024-04-24T08:08:51.176371+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:46:16.570004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.168147+0000","last_change":"2024-04-24T08:07:51.045431+0000","last_active":"2024-04-24T08:08:51.168147+0000","last_peered":"2024-04-24T08:08:51.168147+0000","last_clean":"2024-04-24T08:08:51.168147+0000","last_became_active":"2024-04-24T08:07:51.045112+0000","last_became_peered":"2024-04-24T08:07:51.045112+0000","last_unstale":"2024-04-24T08:08:51.168147+0000","last_undegraded":"2024-04-24T08:08:51.168147+0000","last_fullsized":"2024-04-24T08:08:51.168147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:52:05.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":352,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.585492+0000","last_change":"2024-04-24T08:06:31.583310+0000","last_active":"2024-04-24T08:08:51.585492+0000","last_peered":"2024-04-24T08:08:51.585492+0000","last_clean":"2024-04-24T08:08:51.585492+0000","last_became_active":"2024-04-24T08:06:31.582735+0000","last_became_peered":"2024-04-24T08:06:31.582735+0000","last_unstale":"2024-04-24T08:08:51.585492+0000","last_undegraded":"2024-04-24T08:08:51.585492+0000","last_fullsized":"2024-04-24T08:08:51.585492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:02:34.411025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:02:34.411025+0000","last_clean_scrub_stamp":"2024-04-24T08:02:34.411025+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:48.816092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.248835+0000","last_change":"2024-04-24T08:07:52.063961+0000","last_active":"2024-04-24T08:08:51.248835+0000","last_peered":"2024-04-24T08:08:51.248835+0000","last_clean":"2024-04-24T08:08:51.248835+0000","last_became_active":"2024-04-24T08:07:52.063608+0000","last_became_peered":"2024-04-24T08:07:52.063608+0000","last_unstale":"2024-04-24T08:08:51.248835+0000","last_undegraded":"2024-04-24T08:08:51.248835+0000","last_fullsized":"2024-04-24T08:08:51.248835+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:40:50.233349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.585566+0000","last_change":"2024-04-24T08:07:51.045626+0000","last_active":"2024-04-24T08:08:51.585566+0000","last_peered":"2024-04-24T08:08:51.585566+0000","last_clean":"2024-04-24T08:08:51.585566+0000","last_became_active":"2024-04-24T08:07:51.045293+0000","last_became_peered":"2024-04-24T08:07:51.045293+0000","last_unstale":"2024-04-24T08:08:51.585566+0000","last_undegraded":"2024-04-24T08:08:51.585566+0000","last_fullsized":"2024-04-24T08:08:51.585566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:13:04.601331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.172222+0000","last_change":"2024-04-24T08:07:51.079228+0000","last_active":"2024-04-24T08:08:51.172222+0000","last_peered":"2024-04-24T08:08:51.172222+0000","last_clean":"2024-04-24T08:08:51.172222+0000","last_became_active":"2024-04-24T08:07:51.078932+0000","last_became_peered":"2024-04-24T08:07:51.078932+0000","last_unstale":"2024-04-24T08:08:51.172222+0000","last_undegraded":"2024-04-24T08:08:51.172222+0000","last_fullsized":"2024-04-24T08:08:51.172222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:58:32.024133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.249758+0000","last_change":"2024-04-24T08:07:52.065943+0000","last_active":"2024-04-24T08:08:51.249758+0000","last_peered":"2024-04-24T08:08:51.249758+0000","last_clean":"2024-04-24T08:08:51.249758+0000","last_became_active":"2024-04-24T08:07:52.062810+0000","last_became_peered":"2024-04-24T08:07:52.062810+0000","last_unstale":"2024-04-24T08:08:51.249758+0000","last_undegraded":"2024-04-24T08:08:51.249758+0000","last_fullsized":"2024-04-24T08:08:51.249758+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:40:08.803652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'9","reported_seq":40,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.193009+0000","last_change":"2024-04-24T08:08:51.193009+0000","last_active":"2024-04-24T08:08:51.193009+0000","last_peered":"2024-04-24T08:08:51.193009+0000","last_clean":"2024-04-24T08:08:51.193009+0000","last_became_active":"2024-04-24T08:08:51.180025+0000","last_became_peered":"2024-04-24T08:08:51.180025+0000","last_unstale":"2024-04-24T08:08:51.193009+0000","last_undegraded":"2024-04-24T08:08:51.193009+0000","last_fullsized":"2024-04-24T08:08:51.193009+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:31:19.397470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":17,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.177766+0000","last_change":"2024-04-24T08:07:52.058764+0000","last_active":"2024-04-24T08:08:51.177766+0000","last_peered":"2024-04-24T08:08:51.177766+0000","last_clean":"2024-04-24T08:08:51.177766+0000","last_became_active":"2024-04-24T08:07:52.058336+0000","last_became_peered":"2024-04-24T08:07:52.058336+0000","last_unstale":"2024-04-24T08:08:51.177766+0000","last_undegraded":"2024-04-24T08:08:51.177766+0000","last_fullsized":"2024-04-24T08:08:51.177766+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:05:37.245521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.168801+0000","last_change":"2024-04-24T08:07:51.067154+0000","last_active":"2024-04-24T08:08:51.168801+0000","last_peered":"2024-04-24T08:08:51.168801+0000","last_clean":"2024-04-24T08:08:51.168801+0000","last_became_active":"2024-04-24T08:07:51.066769+0000","last_became_peered":"2024-04-24T08:07:51.066769+0000","last_unstale":"2024-04-24T08:08:51.168801+0000","last_undegraded":"2024-04-24T08:08:51.168801+0000","last_fullsized":"2024-04-24T08:08:51.168801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:39:25.908189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.163484+0000","last_change":"2024-04-24T08:07:52.072134+0000","last_active":"2024-04-24T08:08:51.163484+0000","last_peered":"2024-04-24T08:08:51.163484+0000","last_clean":"2024-04-24T08:08:51.163484+0000","last_became_active":"2024-04-24T08:07:52.071703+0000","last_became_peered":"2024-04-24T08:07:52.071703+0000","last_unstale":"2024-04-24T08:08:51.163484+0000","last_undegraded":"2024-04-24T08:08:51.163484+0000","last_fullsized":"2024-04-24T08:08:51.163484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:59.829988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.167881+0000","last_change":"2024-04-24T08:07:51.059151+0000","last_active":"2024-04-24T08:08:51.167881+0000","last_peered":"2024-04-24T08:08:51.167881+0000","last_clean":"2024-04-24T08:08:51.167881+0000","last_became_active":"2024-04-24T08:07:51.058872+0000","last_became_peered":"2024-04-24T08:07:51.058872+0000","last_unstale":"2024-04-24T08:08:51.167881+0000","last_undegraded":"2024-04-24T08:08:51.167881+0000","last_fullsized":"2024-04-24T08:08:51.167881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:36.023595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.171761+0000","last_change":"2024-04-24T08:07:52.056919+0000","last_active":"2024-04-24T08:08:51.171761+0000","last_peered":"2024-04-24T08:08:51.171761+0000","last_clean":"2024-04-24T08:08:51.171761+0000","last_became_active":"2024-04-24T08:07:52.056573+0000","last_became_peered":"2024-04-24T08:07:52.056573+0000","last_unstale":"2024-04-24T08:08:51.171761+0000","last_undegraded":"2024-04-24T08:08:51.171761+0000","last_fullsized":"2024-04-24T08:08:51.171761+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:58:17.939465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.175838+0000","last_change":"2024-04-24T08:07:51.056325+0000","last_active":"2024-04-24T08:08:51.175838+0000","last_peered":"2024-04-24T08:08:51.175838+0000","last_clean":"2024-04-24T08:08:51.175838+0000","last_became_active":"2024-04-24T08:07:51.055862+0000","last_became_peered":"2024-04-24T08:07:51.055862+0000","last_unstale":"2024-04-24T08:08:51.175838+0000","last_undegraded":"2024-04-24T08:08:51.175838+0000","last_fullsized":"2024-04-24T08:08:51.175838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:11:44.487530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.165975+0000","last_change":"2024-04-24T08:07:52.065570+0000","last_active":"2024-04-24T08:08:51.165975+0000","last_peered":"2024-04-24T08:08:51.165975+0000","last_clean":"2024-04-24T08:08:51.165975+0000","last_became_active":"2024-04-24T08:07:52.065196+0000","last_became_peered":"2024-04-24T08:07:52.065196+0000","last_unstale":"2024-04-24T08:08:51.165975+0000","last_undegraded":"2024-04-24T08:08:51.165975+0000","last_fullsized":"2024-04-24T08:08:51.165975+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:48:09.474159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.168540+0000","last_change":"2024-04-24T08:07:51.067187+0000","last_active":"2024-04-24T08:08:51.168540+0000","last_peered":"2024-04-24T08:08:51.168540+0000","last_clean":"2024-04-24T08:08:51.168540+0000","last_became_active":"2024-04-24T08:07:51.066808+0000","last_became_peered":"2024-04-24T08:07:51.066808+0000","last_unstale":"2024-04-24T08:08:51.168540+0000","last_undegraded":"2024-04-24T08:08:51.168540+0000","last_fullsized":"2024-04-24T08:08:51.168540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:32:11.852191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.177017+0000","last_change":"2024-04-24T08:07:52.078426+0000","last_active":"2024-04-24T08:08:51.177017+0000","last_peered":"2024-04-24T08:08:51.177017+0000","last_clean":"2024-04-24T08:08:51.177017+0000","last_became_active":"2024-04-24T08:07:52.078076+0000","last_became_peered":"2024-04-24T08:07:52.078076+0000","last_unstale":"2024-04-24T08:08:51.177017+0000","last_undegraded":"2024-04-24T08:08:51.177017+0000","last_fullsized":"2024-04-24T08:08:51.177017+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:55:29.885688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.171843+0000","last_change":"2024-04-24T08:07:51.042601+0000","last_active":"2024-04-24T08:08:51.171843+0000","last_peered":"2024-04-24T08:08:51.171843+0000","last_clean":"2024-04-24T08:08:51.171843+0000","last_became_active":"2024-04-24T08:07:51.042252+0000","last_became_peered":"2024-04-24T08:07:51.042252+0000","last_unstale":"2024-04-24T08:08:51.171843+0000","last_undegraded":"2024-04-24T08:08:51.171843+0000","last_fullsized":"2024-04-24T08:08:51.171843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:11:44.640303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.585859+0000","last_change":"2024-04-24T08:07:52.063142+0000","last_active":"2024-04-24T08:08:51.585859+0000","last_peered":"2024-04-24T08:08:51.585859+0000","last_clean":"2024-04-24T08:08:51.585859+0000","last_became_active":"2024-04-24T08:07:52.062263+0000","last_became_peered":"2024-04-24T08:07:52.062263+0000","last_unstale":"2024-04-24T08:08:51.585859+0000","last_undegraded":"2024-04-24T08:08:51.585859+0000","last_fullsized":"2024-04-24T08:08:51.585859+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:38.879380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.185189+0000","last_change":"2024-04-24T08:08:51.184795+0000","last_active":"2024-04-24T08:08:51.185189+0000","last_peered":"2024-04-24T08:08:51.185189+0000","last_clean":"2024-04-24T08:08:51.185189+0000","last_became_active":"2024-04-24T08:08:51.179560+0000","last_became_peered":"2024-04-24T08:08:51.179560+0000","last_unstale":"2024-04-24T08:08:51.185189+0000","last_undegraded":"2024-04-24T08:08:51.185189+0000","last_fullsized":"2024-04-24T08:08:51.185189+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:31:23.795126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.585801+0000","last_change":"2024-04-24T08:07:52.055958+0000","last_active":"2024-04-24T08:08:51.585801+0000","last_peered":"2024-04-24T08:08:51.585801+0000","last_clean":"2024-04-24T08:08:51.585801+0000","last_became_active":"2024-04-24T08:07:52.055598+0000","last_became_peered":"2024-04-24T08:07:52.055598+0000","last_unstale":"2024-04-24T08:08:51.585801+0000","last_undegraded":"2024-04-24T08:08:51.585801+0000","last_fullsized":"2024-04-24T08:08:51.585801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:12:22.452576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.168668+0000","last_change":"2024-04-24T08:07:51.073234+0000","last_active":"2024-04-24T08:08:51.168668+0000","last_peered":"2024-04-24T08:08:51.168668+0000","last_clean":"2024-04-24T08:08:51.168668+0000","last_became_active":"2024-04-24T08:07:51.072887+0000","last_became_peered":"2024-04-24T08:07:51.072887+0000","last_unstale":"2024-04-24T08:08:51.168668+0000","last_undegraded":"2024-04-24T08:08:51.168668+0000","last_fullsized":"2024-04-24T08:08:51.168668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:28:15.908859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"86'1","reported_seq":18,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.585567+0000","last_change":"2024-04-24T08:07:52.061188+0000","last_active":"2024-04-24T08:08:51.585567+0000","last_peered":"2024-04-24T08:08:51.585567+0000","last_clean":"2024-04-24T08:08:51.585567+0000","last_became_active":"2024-04-24T08:07:52.059616+0000","last_became_peered":"2024-04-24T08:07:52.059616+0000","last_unstale":"2024-04-24T08:08:51.585567+0000","last_undegraded":"2024-04-24T08:08:51.585567+0000","last_fullsized":"2024-04-24T08:08:51.585567+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:43.546091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.249320+0000","last_change":"2024-04-24T08:07:51.078300+0000","last_active":"2024-04-24T08:08:51.249320+0000","last_peered":"2024-04-24T08:08:51.249320+0000","last_clean":"2024-04-24T08:08:51.249320+0000","last_became_active":"2024-04-24T08:07:51.076026+0000","last_became_peered":"2024-04-24T08:07:51.076026+0000","last_unstale":"2024-04-24T08:08:51.249320+0000","last_undegraded":"2024-04-24T08:08:51.249320+0000","last_fullsized":"2024-04-24T08:08:51.249320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:45:31.494213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.585642+0000","last_change":"2024-04-24T08:07:52.056148+0000","last_active":"2024-04-24T08:08:51.585642+0000","last_peered":"2024-04-24T08:08:51.585642+0000","last_clean":"2024-04-24T08:08:51.585642+0000","last_became_active":"2024-04-24T08:07:52.055768+0000","last_became_peered":"2024-04-24T08:07:52.055768+0000","last_unstale":"2024-04-24T08:08:51.585642+0000","last_undegraded":"2024-04-24T08:08:51.585642+0000","last_fullsized":"2024-04-24T08:08:51.585642+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:52:48.503479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.166056+0000","last_change":"2024-04-24T08:07:52.056611+0000","last_active":"2024-04-24T08:08:51.166056+0000","last_peered":"2024-04-24T08:08:51.166056+0000","last_clean":"2024-04-24T08:08:51.166056+0000","last_became_active":"2024-04-24T08:07:52.055788+0000","last_became_peered":"2024-04-24T08:07:52.055788+0000","last_unstale":"2024-04-24T08:08:51.166056+0000","last_undegraded":"2024-04-24T08:08:51.166056+0000","last_fullsized":"2024-04-24T08:08:51.166056+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:21:28.997588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.177574+0000","last_change":"2024-04-24T08:08:51.177574+0000","last_active":"2024-04-24T08:08:51.177574+0000","last_peered":"2024-04-24T08:08:51.177574+0000","last_clean":"2024-04-24T08:08:51.177574+0000","last_became_active":"2024-04-24T08:08:51.177145+0000","last_became_peered":"2024-04-24T08:08:51.177145+0000","last_unstale":"2024-04-24T08:08:51.177574+0000","last_undegraded":"2024-04-24T08:08:51.177574+0000","last_fullsized":"2024-04-24T08:08:51.177574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:49:19.438233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.168773+0000","last_change":"2024-04-24T08:07:52.070899+0000","last_active":"2024-04-24T08:08:51.168773+0000","last_peered":"2024-04-24T08:08:51.168773+0000","last_clean":"2024-04-24T08:08:51.168773+0000","last_became_active":"2024-04-24T08:07:52.070556+0000","last_became_peered":"2024-04-24T08:07:52.070556+0000","last_unstale":"2024-04-24T08:08:51.168773+0000","last_undegraded":"2024-04-24T08:08:51.168773+0000","last_fullsized":"2024-04-24T08:08:51.168773+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:11.381122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.250206+0000","last_change":"2024-04-24T08:08:51.250206+0000","last_active":"2024-04-24T08:08:51.250206+0000","last_peered":"2024-04-24T08:08:51.250206+0000","last_clean":"2024-04-24T08:08:51.250206+0000","last_became_active":"2024-04-24T08:08:51.249677+0000","last_became_peered":"2024-04-24T08:08:51.249677+0000","last_unstale":"2024-04-24T08:08:51.250206+0000","last_undegraded":"2024-04-24T08:08:51.250206+0000","last_fullsized":"2024-04-24T08:08:51.250206+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:06:26.655987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.249806+0000","last_change":"2024-04-24T08:07:52.078604+0000","last_active":"2024-04-24T08:08:51.249806+0000","last_peered":"2024-04-24T08:08:51.249806+0000","last_clean":"2024-04-24T08:08:51.249806+0000","last_became_active":"2024-04-24T08:07:52.078122+0000","last_became_peered":"2024-04-24T08:07:52.078122+0000","last_unstale":"2024-04-24T08:08:51.249806+0000","last_undegraded":"2024-04-24T08:08:51.249806+0000","last_fullsized":"2024-04-24T08:08:51.249806+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:24:19.062821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.166221+0000","last_change":"2024-04-24T08:07:52.055986+0000","last_active":"2024-04-24T08:08:51.166221+0000","last_peered":"2024-04-24T08:08:51.166221+0000","last_clean":"2024-04-24T08:08:51.166221+0000","last_became_active":"2024-04-24T08:07:52.055609+0000","last_became_peered":"2024-04-24T08:07:52.055609+0000","last_unstale":"2024-04-24T08:08:51.166221+0000","last_undegraded":"2024-04-24T08:08:51.166221+0000","last_fullsized":"2024-04-24T08:08:51.166221+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:38:04.731204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.169573+0000","last_change":"2024-04-24T08:07:52.055605+0000","last_active":"2024-04-24T08:08:51.169573+0000","last_peered":"2024-04-24T08:08:51.169573+0000","last_clean":"2024-04-24T08:08:51.169573+0000","last_became_active":"2024-04-24T08:07:52.055268+0000","last_became_peered":"2024-04-24T08:07:52.055268+0000","last_unstale":"2024-04-24T08:08:51.169573+0000","last_undegraded":"2024-04-24T08:08:51.169573+0000","last_fullsized":"2024-04-24T08:08:51.169573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:50:29.197484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.177647+0000","last_change":"2024-04-24T08:07:52.064590+0000","last_active":"2024-04-24T08:08:51.177647+0000","last_peered":"2024-04-24T08:08:51.177647+0000","last_clean":"2024-04-24T08:08:51.177647+0000","last_became_active":"2024-04-24T08:07:52.064113+0000","last_became_peered":"2024-04-24T08:07:52.064113+0000","last_unstale":"2024-04-24T08:08:51.177647+0000","last_undegraded":"2024-04-24T08:08:51.177647+0000","last_fullsized":"2024-04-24T08:08:51.177647+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:36:10.926145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.250599+0000","last_change":"2024-04-24T08:08:51.250599+0000","last_active":"2024-04-24T08:08:51.250599+0000","last_peered":"2024-04-24T08:08:51.250599+0000","last_clean":"2024-04-24T08:08:51.250599+0000","last_became_active":"2024-04-24T08:08:51.250197+0000","last_became_peered":"2024-04-24T08:08:51.250197+0000","last_unstale":"2024-04-24T08:08:51.250599+0000","last_undegraded":"2024-04-24T08:08:51.250599+0000","last_fullsized":"2024-04-24T08:08:51.250599+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:31:30.729911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6],"acting":[5,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.175709+0000","last_change":"2024-04-24T08:07:52.068308+0000","last_active":"2024-04-24T08:08:51.175709+0000","last_peered":"2024-04-24T08:08:51.175709+0000","last_clean":"2024-04-24T08:08:51.175709+0000","last_became_active":"2024-04-24T08:07:52.067786+0000","last_became_peered":"2024-04-24T08:07:52.067786+0000","last_unstale":"2024-04-24T08:08:51.175709+0000","last_undegraded":"2024-04-24T08:08:51.175709+0000","last_fullsized":"2024-04-24T08:08:51.175709+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:26:09.349120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.167874+0000","last_change":"2024-04-24T08:07:52.053089+0000","last_active":"2024-04-24T08:08:51.167874+0000","last_peered":"2024-04-24T08:08:51.167874+0000","last_clean":"2024-04-24T08:08:51.167874+0000","last_became_active":"2024-04-24T08:07:52.052753+0000","last_became_peered":"2024-04-24T08:07:52.052753+0000","last_unstale":"2024-04-24T08:08:51.167874+0000","last_undegraded":"2024-04-24T08:08:51.167874+0000","last_fullsized":"2024-04-24T08:08:51.167874+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:02:14.583922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.249208+0000","last_change":"2024-04-24T08:07:52.076633+0000","last_active":"2024-04-24T08:08:51.249208+0000","last_peered":"2024-04-24T08:08:51.249208+0000","last_clean":"2024-04-24T08:08:51.249208+0000","last_became_active":"2024-04-24T08:07:52.076280+0000","last_became_peered":"2024-04-24T08:07:52.076280+0000","last_unstale":"2024-04-24T08:08:51.249208+0000","last_undegraded":"2024-04-24T08:08:51.249208+0000","last_fullsized":"2024-04-24T08:08:51.249208+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:29:59.823772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.166076+0000","last_change":"2024-04-24T08:07:52.053017+0000","last_active":"2024-04-24T08:08:51.166076+0000","last_peered":"2024-04-24T08:08:51.166076+0000","last_clean":"2024-04-24T08:08:51.166076+0000","last_became_active":"2024-04-24T08:07:52.052325+0000","last_became_peered":"2024-04-24T08:07:52.052325+0000","last_unstale":"2024-04-24T08:08:51.166076+0000","last_undegraded":"2024-04-24T08:08:51.166076+0000","last_fullsized":"2024-04-24T08:08:51.166076+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:05:58.161360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":213304,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63272,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":515,"num_read_kb":323,"num_write":160,"num_write_kb":108,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":616,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":189816,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351008,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36280,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708872,"statfs":{"total":95995035648,"available":95957884928,"internally_reserved":0,"allocated":958464,"data_stored":462211,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612644,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28104,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717048,"statfs":{"total":95995035648,"available":95966257152,"internally_reserved":0,"allocated":1105920,"data_stored":1112929,"data_compressed":41899,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776170,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":970752,"data_stored":462413,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037808,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32132,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713020,"statfs":{"total":95995035648,"available":95962132480,"internally_reserved":0,"allocated":970752,"data_stored":462333,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1019904,"data_stored":1054654,"data_compressed":41899,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495677,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":958464,"data_stored":461802,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574936,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1019904,"data_stored":520475,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":463073,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":521600,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182477,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":974848,"data_stored":462113,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920840,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":966656,"data_stored":463212,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757315,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32192,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712960,"statfs":{"total":95995035648,"available":95962071040,"internally_reserved":0,"allocated":1032192,"data_stored":1053947,"data_compressed":41899,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":58880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":60005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1617,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1867,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2574,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":60849,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:09:22.077 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph pg dump --format=json 2024-04-24T08:09:22.580 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:09:22.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:22 smithi117 ceph-mon[32438]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:09:22.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:22 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:09:22.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:22 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:09:22.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:22 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:09:22.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:22 smithi194 ceph-mon[32836]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:09:22.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:22 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:09:22.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:22 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:09:22.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:22 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:09:22.728 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:22 smithi028 ceph-mon[27786]: from='client.24884 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:09:22.728 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:22 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:09:22.728 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:22 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:09:22.728 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:22 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' 2024-04-24T08:09:23.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:23 smithi028 ceph-mon[27786]: pgmap v364: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-24T08:09:23.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:23 smithi117 ceph-mon[32438]: pgmap v364: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-24T08:09:23.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:23 smithi194 ceph-mon[32836]: pgmap v364: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-04-24T08:09:24.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.804+0000 7f0558c00700 1 -- 172.21.15.28:0/1630177694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055406b1a0 msgr2=0x7f055410f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.804+0000 7f0558c00700 1 --2- 172.21.15.28:0/1630177694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055406b1a0 0x7f055410f9b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f054c00b0a0 tx=0x7f054c00b3b0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.804+0000 7f0558c00700 1 -- 172.21.15.28:0/1630177694 shutdown_connections 2024-04-24T08:09:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.804+0000 7f0558c00700 1 --2- 172.21.15.28:0/1630177694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055406b1a0 0x7f055410f9b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.804+0000 7f0558c00700 1 --2- 172.21.15.28:0/1630177694 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0554108b70 0x7f0554108f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.804+0000 7f0558c00700 1 --2- 172.21.15.28:0/1630177694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f055406a800 0x7f055406ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.804+0000 7f0558c00700 1 -- 172.21.15.28:0/1630177694 >> 172.21.15.28:0/1630177694 conn(0x7f05540789a0 msgr2=0x7f0554078db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:24.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.804+0000 7f0558c00700 1 -- 172.21.15.28:0/1630177694 shutdown_connections 2024-04-24T08:09:24.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.804+0000 7f0558c00700 1 -- 172.21.15.28:0/1630177694 wait complete. 2024-04-24T08:09:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.805+0000 7f0558c00700 1 Processor -- start 2024-04-24T08:09:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.805+0000 7f0558c00700 1 -- start start 2024-04-24T08:09:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.805+0000 7f0558c00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055406a800 0x7f0554199250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.805+0000 7f0558c00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f055406b1a0 0x7f0554199790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.806+0000 7f0558c00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0554108b70 0x7f055419db20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.806+0000 7f0558c00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0554112e20 con 0x7f055406a800 2024-04-24T08:09:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.806+0000 7f0558c00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0554112ca0 con 0x7f055406b1a0 2024-04-24T08:09:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.806+0000 7f0558c00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0554112fa0 con 0x7f0554108b70 2024-04-24T08:09:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.806+0000 7f0551e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055406a800 0x7f0554199250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:24.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.806+0000 7f0551e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055406a800 0x7f0554199250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38312/0 (socket says 172.21.15.28:38312) 2024-04-24T08:09:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.806+0000 7f0551e00700 1 -- 172.21.15.28:0/282505292 learned_addr learned my addr 172.21.15.28:0/282505292 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.806+0000 7f0551400700 1 --2- 172.21.15.28:0/282505292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f055406b1a0 0x7f0554199790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.806+0000 7f0552800700 1 --2- 172.21.15.28:0/282505292 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0554108b70 0x7f055419db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.806+0000 7f0551e00700 1 -- 172.21.15.28:0/282505292 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0554108b70 msgr2=0x7f055419db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.806+0000 7f0551e00700 1 --2- 172.21.15.28:0/282505292 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0554108b70 0x7f055419db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.806+0000 7f0551e00700 1 -- 172.21.15.28:0/282505292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f055406b1a0 msgr2=0x7f0554199790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.806+0000 7f0551e00700 1 --2- 172.21.15.28:0/282505292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f055406b1a0 0x7f0554199790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.806+0000 7f0551e00700 1 -- 172.21.15.28:0/282505292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f055419e260 con 0x7f055406a800 2024-04-24T08:09:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.807+0000 7f0551e00700 1 --2- 172.21.15.28:0/282505292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055406a800 0x7f0554199250 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f054400eb30 tx=0x7f054400ee40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.807+0000 7f054aa00700 1 -- 172.21.15.28:0/282505292 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054400cbd0 con 0x7f055406a800 2024-04-24T08:09:24.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.807+0000 7f054aa00700 1 -- 172.21.15.28:0/282505292 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0544004500 con 0x7f055406a800 2024-04-24T08:09:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.807+0000 7f054aa00700 1 -- 172.21.15.28:0/282505292 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0544010430 con 0x7f055406a800 2024-04-24T08:09:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.807+0000 7f0558c00700 1 -- 172.21.15.28:0/282505292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f055419e550 con 0x7f055406a800 2024-04-24T08:09:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.807+0000 7f0558c00700 1 -- 172.21.15.28:0/282505292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05541a5dd0 con 0x7f055406a800 2024-04-24T08:09:24.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.808+0000 7f0558c00700 1 -- 172.21.15.28:0/282505292 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05540697e0 con 0x7f055406a800 2024-04-24T08:09:24.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.811+0000 7f054aa00700 1 -- 172.21.15.28:0/282505292 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0544003f00 con 0x7f055406a800 2024-04-24T08:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.812+0000 7f054aa00700 1 --2- 172.21.15.28:0/282505292 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f05380a3cc0 0x7f05380a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.812+0000 7f054aa00700 1 -- 172.21.15.28:0/282505292 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f05440d06d0 con 0x7f055406a800 2024-04-24T08:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.812+0000 7f0551400700 1 --2- 172.21.15.28:0/282505292 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f05380a3cc0 0x7f05380a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:24.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.813+0000 7f0551400700 1 --2- 172.21.15.28:0/282505292 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f05380a3cc0 0x7f05380a6180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0554078ea0 tx=0x7f0540009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:24.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:24.817+0000 7f054aa00700 1 -- 172.21.15.28:0/282505292 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f054409d030 con 0x7f055406a800 2024-04-24T08:09:25.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:25.046+0000 7f0558c00700 1 -- 172.21.15.28:0/282505292 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f0554062f30 con 0x7f05380a3cc0 2024-04-24T08:09:25.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:25.063+0000 7f054aa00700 1 -- 172.21.15.28:0/282505292 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+259157 (secure 0 0 0) 0x7f0554062f30 con 0x7f05380a3cc0 2024-04-24T08:09:25.066 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:09:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:25.069+0000 7f0558c00700 1 -- 172.21.15.28:0/282505292 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f05380a3cc0 msgr2=0x7f05380a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:25.069+0000 7f0558c00700 1 --2- 172.21.15.28:0/282505292 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f05380a3cc0 0x7f05380a6180 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f0554078ea0 tx=0x7f0540009260 comp rx=0 tx=0).stop 2024-04-24T08:09:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:25.069+0000 7f0558c00700 1 -- 172.21.15.28:0/282505292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055406a800 msgr2=0x7f0554199250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:25.069+0000 7f0558c00700 1 --2- 172.21.15.28:0/282505292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055406a800 0x7f0554199250 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f054400eb30 tx=0x7f054400ee40 comp rx=0 tx=0).stop 2024-04-24T08:09:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:25.069+0000 7f0558c00700 1 -- 172.21.15.28:0/282505292 shutdown_connections 2024-04-24T08:09:25.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:25.069+0000 7f0558c00700 1 --2- 172.21.15.28:0/282505292 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f05380a3cc0 0x7f05380a6180 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:25.069+0000 7f0558c00700 1 --2- 172.21.15.28:0/282505292 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0554108b70 0x7f055419db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:25.069+0000 7f0558c00700 1 --2- 172.21.15.28:0/282505292 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f055406a800 0x7f0554199250 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:25.069+0000 7f0558c00700 1 --2- 172.21.15.28:0/282505292 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f055406b1a0 0x7f0554199790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:25.069+0000 7f0558c00700 1 -- 172.21.15.28:0/282505292 >> 172.21.15.28:0/282505292 conn(0x7f05540789a0 msgr2=0x7f05540ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:25.069+0000 7f0558c00700 1 -- 172.21.15.28:0/282505292 shutdown_connections 2024-04-24T08:09:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:25.069+0000 7f0558c00700 1 -- 172.21.15.28:0/282505292 wait complete. 2024-04-24T08:09:25.072 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-24T08:09:25.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:25 smithi117 ceph-mon[32438]: pgmap v365: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-24T08:09:25.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:25 smithi194 ceph-mon[32836]: pgmap v365: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-24T08:09:25.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:25 smithi028 ceph-mon[27786]: pgmap v365: 97 pgs: 97 active+clean; 639 KiB data, 352 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-24T08:09:25.999 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-04-24T08:09:24.097677+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654405,"num_objects":64,"num_object_clones":0,"num_object_copies":192,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":684,"num_read_kb":460,"num_write":352,"num_write_kb":3055,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1181352,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":688,"ondisk_log_size":688,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":360880,"kb_used_data":11712,"kb_used_omap":18,"kb_used_meta":349037,"kb_avail":1124580944,"statfs":{"total":1151940427776,"available":1151570886656,"internally_reserved":0,"allocated":11993088,"data_stored":7501716,"data_compressed":177906,"data_compressed_allocated":233472,"data_compressed_original":1966080,"omap_allocated":19140,"internal_metadata":357414204},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":36,"apply_latency_ms":36,"commit_latency_ns":36000000,"apply_latency_ns":36000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":93,"num_read_kb":48,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.005451"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.090356+0000","last_change":"2024-04-24T08:07:52.074319+0000","last_active":"2024-04-24T08:09:20.090356+0000","last_peered":"2024-04-24T08:09:20.090356+0000","last_clean":"2024-04-24T08:09:20.090356+0000","last_became_active":"2024-04-24T08:07:52.073923+0000","last_became_peered":"2024-04-24T08:07:52.073923+0000","last_unstale":"2024-04-24T08:09:20.090356+0000","last_undegraded":"2024-04-24T08:09:20.090356+0000","last_fullsized":"2024-04-24T08:09:20.090356+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:29.325904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.163288+0000","last_change":"2024-04-24T08:07:52.068316+0000","last_active":"2024-04-24T08:08:51.163288+0000","last_peered":"2024-04-24T08:08:51.163288+0000","last_clean":"2024-04-24T08:08:51.163288+0000","last_became_active":"2024-04-24T08:07:52.067961+0000","last_became_peered":"2024-04-24T08:07:52.067961+0000","last_unstale":"2024-04-24T08:08:51.163288+0000","last_undegraded":"2024-04-24T08:08:51.163288+0000","last_fullsized":"2024-04-24T08:08:51.163288+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:35:14.079348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.165619+0000","last_change":"2024-04-24T08:07:52.066870+0000","last_active":"2024-04-24T08:08:51.165619+0000","last_peered":"2024-04-24T08:08:51.165619+0000","last_clean":"2024-04-24T08:08:51.165619+0000","last_became_active":"2024-04-24T08:07:52.066481+0000","last_became_peered":"2024-04-24T08:07:52.066481+0000","last_unstale":"2024-04-24T08:08:51.165619+0000","last_undegraded":"2024-04-24T08:08:51.165619+0000","last_fullsized":"2024-04-24T08:08:51.165619+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:25:45.526667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.069325+0000","last_change":"2024-04-24T08:07:52.061548+0000","last_active":"2024-04-24T08:09:20.069325+0000","last_peered":"2024-04-24T08:09:20.069325+0000","last_clean":"2024-04-24T08:09:20.069325+0000","last_became_active":"2024-04-24T08:07:52.060519+0000","last_became_peered":"2024-04-24T08:07:52.060519+0000","last_unstale":"2024-04-24T08:09:20.069325+0000","last_undegraded":"2024-04-24T08:09:20.069325+0000","last_fullsized":"2024-04-24T08:09:20.069325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:47:20.952035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.083191+0000","last_change":"2024-04-24T08:07:52.063805+0000","last_active":"2024-04-24T08:09:20.083191+0000","last_peered":"2024-04-24T08:09:20.083191+0000","last_clean":"2024-04-24T08:09:20.083191+0000","last_became_active":"2024-04-24T08:07:52.063414+0000","last_became_peered":"2024-04-24T08:07:52.063414+0000","last_unstale":"2024-04-24T08:09:20.083191+0000","last_undegraded":"2024-04-24T08:09:20.083191+0000","last_fullsized":"2024-04-24T08:09:20.083191+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:06:24.465035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.097263+0000","last_change":"2024-04-24T08:07:52.050152+0000","last_active":"2024-04-24T08:09:20.097263+0000","last_peered":"2024-04-24T08:09:20.097263+0000","last_clean":"2024-04-24T08:09:20.097263+0000","last_became_active":"2024-04-24T08:07:52.049805+0000","last_became_peered":"2024-04-24T08:07:52.049805+0000","last_unstale":"2024-04-24T08:09:20.097263+0000","last_undegraded":"2024-04-24T08:09:20.097263+0000","last_fullsized":"2024-04-24T08:09:20.097263+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:47:34.644263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.097549+0000","last_change":"2024-04-24T08:07:52.055726+0000","last_active":"2024-04-24T08:09:20.097549+0000","last_peered":"2024-04-24T08:09:20.097549+0000","last_clean":"2024-04-24T08:09:20.097549+0000","last_became_active":"2024-04-24T08:07:52.054563+0000","last_became_peered":"2024-04-24T08:07:52.054563+0000","last_unstale":"2024-04-24T08:09:20.097549+0000","last_undegraded":"2024-04-24T08:09:20.097549+0000","last_fullsized":"2024-04-24T08:09:20.097549+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:34:57.430232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.101298+0000","last_change":"2024-04-24T08:07:52.053672+0000","last_active":"2024-04-24T08:09:20.101298+0000","last_peered":"2024-04-24T08:09:20.101298+0000","last_clean":"2024-04-24T08:09:20.101298+0000","last_became_active":"2024-04-24T08:07:52.053150+0000","last_became_peered":"2024-04-24T08:07:52.053150+0000","last_unstale":"2024-04-24T08:09:20.101298+0000","last_undegraded":"2024-04-24T08:09:20.101298+0000","last_fullsized":"2024-04-24T08:09:20.101298+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:00:12.272555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.073705+0000","last_change":"2024-04-24T08:07:52.054648+0000","last_active":"2024-04-24T08:09:20.073705+0000","last_peered":"2024-04-24T08:09:20.073705+0000","last_clean":"2024-04-24T08:09:20.073705+0000","last_became_active":"2024-04-24T08:07:52.054301+0000","last_became_peered":"2024-04-24T08:07:52.054301+0000","last_unstale":"2024-04-24T08:09:20.073705+0000","last_undegraded":"2024-04-24T08:09:20.073705+0000","last_fullsized":"2024-04-24T08:09:20.073705+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:52:20.533855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.071111+0000","last_change":"2024-04-24T08:07:52.052994+0000","last_active":"2024-04-24T08:09:20.071111+0000","last_peered":"2024-04-24T08:09:20.071111+0000","last_clean":"2024-04-24T08:09:20.071111+0000","last_became_active":"2024-04-24T08:07:52.052641+0000","last_became_peered":"2024-04-24T08:07:52.052641+0000","last_unstale":"2024-04-24T08:09:20.071111+0000","last_undegraded":"2024-04-24T08:09:20.071111+0000","last_fullsized":"2024-04-24T08:09:20.071111+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:02.665622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.075762+0000","last_change":"2024-04-24T08:07:52.055901+0000","last_active":"2024-04-24T08:09:20.075762+0000","last_peered":"2024-04-24T08:09:20.075762+0000","last_clean":"2024-04-24T08:09:20.075762+0000","last_became_active":"2024-04-24T08:07:52.055277+0000","last_became_peered":"2024-04-24T08:07:52.055277+0000","last_unstale":"2024-04-24T08:09:20.075762+0000","last_undegraded":"2024-04-24T08:09:20.075762+0000","last_fullsized":"2024-04-24T08:09:20.075762+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:46.950578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.069212+0000","last_change":"2024-04-24T08:07:52.060329+0000","last_active":"2024-04-24T08:09:20.069212+0000","last_peered":"2024-04-24T08:09:20.069212+0000","last_clean":"2024-04-24T08:09:20.069212+0000","last_became_active":"2024-04-24T08:07:52.059914+0000","last_became_peered":"2024-04-24T08:07:52.059914+0000","last_unstale":"2024-04-24T08:09:20.069212+0000","last_undegraded":"2024-04-24T08:09:20.069212+0000","last_fullsized":"2024-04-24T08:09:20.069212+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:15:20.263916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.091498+0000","last_change":"2024-04-24T08:07:52.062108+0000","last_active":"2024-04-24T08:09:20.091498+0000","last_peered":"2024-04-24T08:09:20.091498+0000","last_clean":"2024-04-24T08:09:20.091498+0000","last_became_active":"2024-04-24T08:07:52.061665+0000","last_became_peered":"2024-04-24T08:07:52.061665+0000","last_unstale":"2024-04-24T08:09:20.091498+0000","last_undegraded":"2024-04-24T08:09:20.091498+0000","last_fullsized":"2024-04-24T08:09:20.091498+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:55:58.071807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.164310+0000","last_change":"2024-04-24T08:07:52.063033+0000","last_active":"2024-04-24T08:08:51.164310+0000","last_peered":"2024-04-24T08:08:51.164310+0000","last_clean":"2024-04-24T08:08:51.164310+0000","last_became_active":"2024-04-24T08:07:52.062610+0000","last_became_peered":"2024-04-24T08:07:52.062610+0000","last_unstale":"2024-04-24T08:08:51.164310+0000","last_undegraded":"2024-04-24T08:08:51.164310+0000","last_fullsized":"2024-04-24T08:08:51.164310+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:59:35.906720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.163840+0000","last_change":"2024-04-24T08:07:52.064657+0000","last_active":"2024-04-24T08:08:51.163840+0000","last_peered":"2024-04-24T08:08:51.163840+0000","last_clean":"2024-04-24T08:08:51.163840+0000","last_became_active":"2024-04-24T08:07:52.064188+0000","last_became_peered":"2024-04-24T08:07:52.064188+0000","last_unstale":"2024-04-24T08:08:51.163840+0000","last_undegraded":"2024-04-24T08:08:51.163840+0000","last_fullsized":"2024-04-24T08:08:51.163840+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:56:32.139420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.092723+0000","last_change":"2024-04-24T08:07:52.067227+0000","last_active":"2024-04-24T08:09:20.092723+0000","last_peered":"2024-04-24T08:09:20.092723+0000","last_clean":"2024-04-24T08:09:20.092723+0000","last_became_active":"2024-04-24T08:07:52.066854+0000","last_became_peered":"2024-04-24T08:07:52.066854+0000","last_unstale":"2024-04-24T08:09:20.092723+0000","last_undegraded":"2024-04-24T08:09:20.092723+0000","last_fullsized":"2024-04-24T08:09:20.092723+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:25:17.580223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.090460+0000","last_change":"2024-04-24T08:07:52.074127+0000","last_active":"2024-04-24T08:09:20.090460+0000","last_peered":"2024-04-24T08:09:20.090460+0000","last_clean":"2024-04-24T08:09:20.090460+0000","last_became_active":"2024-04-24T08:07:52.073806+0000","last_became_peered":"2024-04-24T08:07:52.073806+0000","last_unstale":"2024-04-24T08:09:20.090460+0000","last_undegraded":"2024-04-24T08:09:20.090460+0000","last_fullsized":"2024-04-24T08:09:20.090460+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:25:22.068997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.103609+0000","last_change":"2024-04-24T08:07:52.051239+0000","last_active":"2024-04-24T08:09:20.103609+0000","last_peered":"2024-04-24T08:09:20.103609+0000","last_clean":"2024-04-24T08:09:20.103609+0000","last_became_active":"2024-04-24T08:07:52.049232+0000","last_became_peered":"2024-04-24T08:07:52.049232+0000","last_unstale":"2024-04-24T08:09:20.103609+0000","last_undegraded":"2024-04-24T08:09:20.103609+0000","last_fullsized":"2024-04-24T08:09:20.103609+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:07:39.768537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.169180+0000","last_change":"2024-04-24T08:07:52.062119+0000","last_active":"2024-04-24T08:08:51.169180+0000","last_peered":"2024-04-24T08:08:51.169180+0000","last_clean":"2024-04-24T08:08:51.169180+0000","last_became_active":"2024-04-24T08:07:52.061781+0000","last_became_peered":"2024-04-24T08:07:52.061781+0000","last_unstale":"2024-04-24T08:08:51.169180+0000","last_undegraded":"2024-04-24T08:08:51.169180+0000","last_fullsized":"2024-04-24T08:08:51.169180+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:49:41.595368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.166665+0000","last_change":"2024-04-24T08:07:51.052411+0000","last_active":"2024-04-24T08:08:51.166665+0000","last_peered":"2024-04-24T08:08:51.166665+0000","last_clean":"2024-04-24T08:08:51.166665+0000","last_became_active":"2024-04-24T08:07:51.051704+0000","last_became_peered":"2024-04-24T08:07:51.051704+0000","last_unstale":"2024-04-24T08:08:51.166665+0000","last_undegraded":"2024-04-24T08:08:51.166665+0000","last_fullsized":"2024-04-24T08:08:51.166665+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:34:23.001338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.083091+0000","last_change":"2024-04-24T08:07:52.056508+0000","last_active":"2024-04-24T08:09:20.083091+0000","last_peered":"2024-04-24T08:09:20.083091+0000","last_clean":"2024-04-24T08:09:20.083091+0000","last_became_active":"2024-04-24T08:07:52.056158+0000","last_became_peered":"2024-04-24T08:07:52.056158+0000","last_unstale":"2024-04-24T08:09:20.083091+0000","last_undegraded":"2024-04-24T08:09:20.083091+0000","last_fullsized":"2024-04-24T08:09:20.083091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:52.267984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.178662+0000","last_change":"2024-04-24T08:07:51.080403+0000","last_active":"2024-04-24T08:08:51.178662+0000","last_peered":"2024-04-24T08:08:51.178662+0000","last_clean":"2024-04-24T08:08:51.178662+0000","last_became_active":"2024-04-24T08:07:51.079158+0000","last_became_peered":"2024-04-24T08:07:51.079158+0000","last_unstale":"2024-04-24T08:08:51.178662+0000","last_undegraded":"2024-04-24T08:08:51.178662+0000","last_fullsized":"2024-04-24T08:08:51.178662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:49:59.491850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.165290+0000","last_change":"2024-04-24T08:07:52.066963+0000","last_active":"2024-04-24T08:08:51.165290+0000","last_peered":"2024-04-24T08:08:51.165290+0000","last_clean":"2024-04-24T08:08:51.165290+0000","last_became_active":"2024-04-24T08:07:52.065475+0000","last_became_peered":"2024-04-24T08:07:52.065475+0000","last_unstale":"2024-04-24T08:08:51.165290+0000","last_undegraded":"2024-04-24T08:08:51.165290+0000","last_fullsized":"2024-04-24T08:08:51.165290+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:28:34.220200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.172343+0000","last_change":"2024-04-24T08:07:51.042772+0000","last_active":"2024-04-24T08:08:51.172343+0000","last_peered":"2024-04-24T08:08:51.172343+0000","last_clean":"2024-04-24T08:08:51.172343+0000","last_became_active":"2024-04-24T08:07:51.042128+0000","last_became_peered":"2024-04-24T08:07:51.042128+0000","last_unstale":"2024-04-24T08:08:51.172343+0000","last_undegraded":"2024-04-24T08:08:51.172343+0000","last_fullsized":"2024-04-24T08:08:51.172343+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:10:29.207991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.096272+0000","last_change":"2024-04-24T08:07:52.062411+0000","last_active":"2024-04-24T08:09:20.096272+0000","last_peered":"2024-04-24T08:09:20.096272+0000","last_clean":"2024-04-24T08:09:20.096272+0000","last_became_active":"2024-04-24T08:07:52.062037+0000","last_became_peered":"2024-04-24T08:07:52.062037+0000","last_unstale":"2024-04-24T08:09:20.096272+0000","last_undegraded":"2024-04-24T08:09:20.096272+0000","last_fullsized":"2024-04-24T08:09:20.096272+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:16:17.463874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.172264+0000","last_change":"2024-04-24T08:07:51.046046+0000","last_active":"2024-04-24T08:08:51.172264+0000","last_peered":"2024-04-24T08:08:51.172264+0000","last_clean":"2024-04-24T08:08:51.172264+0000","last_became_active":"2024-04-24T08:07:51.045633+0000","last_became_peered":"2024-04-24T08:07:51.045633+0000","last_unstale":"2024-04-24T08:08:51.172264+0000","last_undegraded":"2024-04-24T08:08:51.172264+0000","last_fullsized":"2024-04-24T08:08:51.172264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:18:20.529234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.071130+0000","last_change":"2024-04-24T08:07:52.061167+0000","last_active":"2024-04-24T08:09:20.071130+0000","last_peered":"2024-04-24T08:09:20.071130+0000","last_clean":"2024-04-24T08:09:20.071130+0000","last_became_active":"2024-04-24T08:07:52.060841+0000","last_became_peered":"2024-04-24T08:07:52.060841+0000","last_unstale":"2024-04-24T08:09:20.071130+0000","last_undegraded":"2024-04-24T08:09:20.071130+0000","last_fullsized":"2024-04-24T08:09:20.071130+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:04:27.048760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'7","reported_seq":117,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:22.275414+0000","last_change":"2024-04-24T08:07:51.044274+0000","last_active":"2024-04-24T08:09:22.275414+0000","last_peered":"2024-04-24T08:09:22.275414+0000","last_clean":"2024-04-24T08:09:22.275414+0000","last_became_active":"2024-04-24T08:07:51.043938+0000","last_became_peered":"2024-04-24T08:07:51.043938+0000","last_unstale":"2024-04-24T08:09:22.275414+0000","last_undegraded":"2024-04-24T08:09:22.275414+0000","last_fullsized":"2024-04-24T08:09:22.275414+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:05:38.349981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":91,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.095207+0000","last_change":"2024-04-24T08:07:52.055777+0000","last_active":"2024-04-24T08:09:20.095207+0000","last_peered":"2024-04-24T08:09:20.095207+0000","last_clean":"2024-04-24T08:09:20.095207+0000","last_became_active":"2024-04-24T08:07:52.055415+0000","last_became_peered":"2024-04-24T08:07:52.055415+0000","last_unstale":"2024-04-24T08:09:20.095207+0000","last_undegraded":"2024-04-24T08:09:20.095207+0000","last_fullsized":"2024-04-24T08:09:20.095207+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:18:31.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.164174+0000","last_change":"2024-04-24T08:07:51.072513+0000","last_active":"2024-04-24T08:08:51.164174+0000","last_peered":"2024-04-24T08:08:51.164174+0000","last_clean":"2024-04-24T08:08:51.164174+0000","last_became_active":"2024-04-24T08:07:51.072249+0000","last_became_peered":"2024-04-24T08:07:51.072249+0000","last_unstale":"2024-04-24T08:08:51.164174+0000","last_undegraded":"2024-04-24T08:08:51.164174+0000","last_fullsized":"2024-04-24T08:08:51.164174+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:45:07.112617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.091581+0000","last_change":"2024-04-24T08:08:51.587130+0000","last_active":"2024-04-24T08:09:20.091581+0000","last_peered":"2024-04-24T08:09:20.091581+0000","last_clean":"2024-04-24T08:09:20.091581+0000","last_became_active":"2024-04-24T08:08:51.586645+0000","last_became_peered":"2024-04-24T08:08:51.586645+0000","last_unstale":"2024-04-24T08:09:20.091581+0000","last_undegraded":"2024-04-24T08:09:20.091581+0000","last_fullsized":"2024-04-24T08:09:20.091581+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:24:14.677548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.249085+0000","last_change":"2024-04-24T08:07:51.070901+0000","last_active":"2024-04-24T08:08:51.249085+0000","last_peered":"2024-04-24T08:08:51.249085+0000","last_clean":"2024-04-24T08:08:51.249085+0000","last_became_active":"2024-04-24T08:07:51.067178+0000","last_became_peered":"2024-04-24T08:07:51.067178+0000","last_unstale":"2024-04-24T08:08:51.249085+0000","last_undegraded":"2024-04-24T08:08:51.249085+0000","last_fullsized":"2024-04-24T08:08:51.249085+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:24:26.807786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.101123+0000","last_change":"2024-04-24T08:07:52.079493+0000","last_active":"2024-04-24T08:09:20.101123+0000","last_peered":"2024-04-24T08:09:20.101123+0000","last_clean":"2024-04-24T08:09:20.101123+0000","last_became_active":"2024-04-24T08:07:52.077266+0000","last_became_peered":"2024-04-24T08:07:52.077266+0000","last_unstale":"2024-04-24T08:09:20.101123+0000","last_undegraded":"2024-04-24T08:09:20.101123+0000","last_fullsized":"2024-04-24T08:09:20.101123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:40:45.660146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.170025+0000","last_change":"2024-04-24T08:07:51.045963+0000","last_active":"2024-04-24T08:08:51.170025+0000","last_peered":"2024-04-24T08:08:51.170025+0000","last_clean":"2024-04-24T08:08:51.170025+0000","last_became_active":"2024-04-24T08:07:51.045641+0000","last_became_peered":"2024-04-24T08:07:51.045641+0000","last_unstale":"2024-04-24T08:08:51.170025+0000","last_undegraded":"2024-04-24T08:08:51.170025+0000","last_fullsized":"2024-04-24T08:08:51.170025+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:07:59.646155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.107105+0000","last_change":"2024-04-24T08:07:52.057280+0000","last_active":"2024-04-24T08:09:20.107105+0000","last_peered":"2024-04-24T08:09:20.107105+0000","last_clean":"2024-04-24T08:09:20.107105+0000","last_became_active":"2024-04-24T08:07:52.054944+0000","last_became_peered":"2024-04-24T08:07:52.054944+0000","last_unstale":"2024-04-24T08:09:20.107105+0000","last_undegraded":"2024-04-24T08:09:20.107105+0000","last_fullsized":"2024-04-24T08:09:20.107105+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:22:45.911472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'4","reported_seq":109,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:21.273155+0000","last_change":"2024-04-24T08:07:51.048898+0000","last_active":"2024-04-24T08:09:21.273155+0000","last_peered":"2024-04-24T08:09:21.273155+0000","last_clean":"2024-04-24T08:09:21.273155+0000","last_became_active":"2024-04-24T08:07:51.048585+0000","last_became_peered":"2024-04-24T08:07:51.048585+0000","last_unstale":"2024-04-24T08:09:21.273155+0000","last_undegraded":"2024-04-24T08:09:21.273155+0000","last_fullsized":"2024-04-24T08:09:21.273155+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:40:30.413541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":86,"num_read_kb":86,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.102916+0000","last_change":"2024-04-24T08:07:52.048093+0000","last_active":"2024-04-24T08:09:20.102916+0000","last_peered":"2024-04-24T08:09:20.102916+0000","last_clean":"2024-04-24T08:09:20.102916+0000","last_became_active":"2024-04-24T08:07:52.047727+0000","last_became_peered":"2024-04-24T08:07:52.047727+0000","last_unstale":"2024-04-24T08:09:20.102916+0000","last_undegraded":"2024-04-24T08:09:20.102916+0000","last_fullsized":"2024-04-24T08:09:20.102916+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:57:26.248355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.585801+0000","last_change":"2024-04-24T08:07:51.047810+0000","last_active":"2024-04-24T08:08:51.585801+0000","last_peered":"2024-04-24T08:08:51.585801+0000","last_clean":"2024-04-24T08:08:51.585801+0000","last_became_active":"2024-04-24T08:07:51.047501+0000","last_became_peered":"2024-04-24T08:07:51.047501+0000","last_unstale":"2024-04-24T08:08:51.585801+0000","last_undegraded":"2024-04-24T08:08:51.585801+0000","last_fullsized":"2024-04-24T08:08:51.585801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:02:27.585202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.163853+0000","last_change":"2024-04-24T08:07:52.063782+0000","last_active":"2024-04-24T08:08:51.163853+0000","last_peered":"2024-04-24T08:08:51.163853+0000","last_clean":"2024-04-24T08:08:51.163853+0000","last_became_active":"2024-04-24T08:07:52.063056+0000","last_became_peered":"2024-04-24T08:07:52.063056+0000","last_unstale":"2024-04-24T08:08:51.163853+0000","last_undegraded":"2024-04-24T08:08:51.163853+0000","last_fullsized":"2024-04-24T08:08:51.163853+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:41:30.966787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.169138+0000","last_change":"2024-04-24T08:07:51.076139+0000","last_active":"2024-04-24T08:08:51.169138+0000","last_peered":"2024-04-24T08:08:51.169138+0000","last_clean":"2024-04-24T08:08:51.169138+0000","last_became_active":"2024-04-24T08:07:51.075842+0000","last_became_peered":"2024-04-24T08:07:51.075842+0000","last_unstale":"2024-04-24T08:08:51.169138+0000","last_undegraded":"2024-04-24T08:08:51.169138+0000","last_fullsized":"2024-04-24T08:08:51.169138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:46:56.741726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.096318+0000","last_change":"2024-04-24T08:08:51.250881+0000","last_active":"2024-04-24T08:09:20.096318+0000","last_peered":"2024-04-24T08:09:20.096318+0000","last_clean":"2024-04-24T08:09:20.096318+0000","last_became_active":"2024-04-24T08:08:51.250519+0000","last_became_peered":"2024-04-24T08:08:51.250519+0000","last_unstale":"2024-04-24T08:09:20.096318+0000","last_undegraded":"2024-04-24T08:09:20.096318+0000","last_fullsized":"2024-04-24T08:09:20.096318+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:23:04.805687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'31","reported_seq":230,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:22.274799+0000","last_change":"2024-04-24T08:07:51.048079+0000","last_active":"2024-04-24T08:09:22.274799+0000","last_peered":"2024-04-24T08:09:22.274799+0000","last_clean":"2024-04-24T08:09:22.274799+0000","last_became_active":"2024-04-24T08:07:51.047584+0000","last_became_peered":"2024-04-24T08:07:51.047584+0000","last_unstale":"2024-04-24T08:09:22.274799+0000","last_undegraded":"2024-04-24T08:09:22.274799+0000","last_fullsized":"2024-04-24T08:09:22.274799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":31,"log_dups_size":0,"ondisk_log_size":31,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:09:05.554984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":59129,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":180,"num_read_kb":147,"num_write":32,"num_write_kb":70,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.056592+0000","last_change":"2024-04-24T08:07:52.055894+0000","last_active":"2024-04-24T08:09:20.056592+0000","last_peered":"2024-04-24T08:09:20.056592+0000","last_clean":"2024-04-24T08:09:20.056592+0000","last_became_active":"2024-04-24T08:07:52.055548+0000","last_became_peered":"2024-04-24T08:07:52.055548+0000","last_unstale":"2024-04-24T08:09:20.056592+0000","last_undegraded":"2024-04-24T08:09:20.056592+0000","last_fullsized":"2024-04-24T08:09:20.056592+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:33:22.609865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.188459+0000","last_change":"2024-04-24T08:08:51.188459+0000","last_active":"2024-04-24T08:08:51.188459+0000","last_peered":"2024-04-24T08:08:51.188459+0000","last_clean":"2024-04-24T08:08:51.188459+0000","last_became_active":"2024-04-24T08:08:51.180387+0000","last_became_peered":"2024-04-24T08:08:51.180387+0000","last_unstale":"2024-04-24T08:08:51.188459+0000","last_undegraded":"2024-04-24T08:08:51.188459+0000","last_fullsized":"2024-04-24T08:08:51.188459+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:34:56.592978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.107255+0000","last_change":"2024-04-24T08:07:52.069950+0000","last_active":"2024-04-24T08:09:20.107255+0000","last_peered":"2024-04-24T08:09:20.107255+0000","last_clean":"2024-04-24T08:09:20.107255+0000","last_became_active":"2024-04-24T08:07:52.069515+0000","last_became_peered":"2024-04-24T08:07:52.069515+0000","last_unstale":"2024-04-24T08:09:20.107255+0000","last_undegraded":"2024-04-24T08:09:20.107255+0000","last_fullsized":"2024-04-24T08:09:20.107255+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:52:27.530541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.168866+0000","last_change":"2024-04-24T08:07:51.067062+0000","last_active":"2024-04-24T08:08:51.168866+0000","last_peered":"2024-04-24T08:08:51.168866+0000","last_clean":"2024-04-24T08:08:51.168866+0000","last_became_active":"2024-04-24T08:07:51.066712+0000","last_became_peered":"2024-04-24T08:07:51.066712+0000","last_unstale":"2024-04-24T08:08:51.168866+0000","last_undegraded":"2024-04-24T08:08:51.168866+0000","last_fullsized":"2024-04-24T08:08:51.168866+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:11:52.974470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.094274+0000","last_change":"2024-04-24T08:07:52.064521+0000","last_active":"2024-04-24T08:09:20.094274+0000","last_peered":"2024-04-24T08:09:20.094274+0000","last_clean":"2024-04-24T08:09:20.094274+0000","last_became_active":"2024-04-24T08:07:52.064184+0000","last_became_peered":"2024-04-24T08:07:52.064184+0000","last_unstale":"2024-04-24T08:09:20.094274+0000","last_undegraded":"2024-04-24T08:09:20.094274+0000","last_fullsized":"2024-04-24T08:09:20.094274+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:00:27.294441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"88'5","reported_seq":115,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:22.317451+0000","last_change":"2024-04-24T08:07:51.043577+0000","last_active":"2024-04-24T08:09:22.317451+0000","last_peered":"2024-04-24T08:09:22.317451+0000","last_clean":"2024-04-24T08:09:22.317451+0000","last_became_active":"2024-04-24T08:07:51.043215+0000","last_became_peered":"2024-04-24T08:07:51.043215+0000","last_unstale":"2024-04-24T08:09:22.317451+0000","last_undegraded":"2024-04-24T08:09:22.317451+0000","last_fullsized":"2024-04-24T08:09:22.317451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:25:09.979236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1120,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":92,"num_read_kb":5,"num_write":4,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.067411+0000","last_change":"2024-04-24T08:07:52.057633+0000","last_active":"2024-04-24T08:09:20.067411+0000","last_peered":"2024-04-24T08:09:20.067411+0000","last_clean":"2024-04-24T08:09:20.067411+0000","last_became_active":"2024-04-24T08:07:52.057309+0000","last_became_peered":"2024-04-24T08:07:52.057309+0000","last_unstale":"2024-04-24T08:09:20.067411+0000","last_undegraded":"2024-04-24T08:09:20.067411+0000","last_fullsized":"2024-04-24T08:09:20.067411+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:49:56.045030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":35,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.171746+0000","last_change":"2024-04-24T08:07:51.058957+0000","last_active":"2024-04-24T08:08:51.171746+0000","last_peered":"2024-04-24T08:08:51.171746+0000","last_clean":"2024-04-24T08:08:51.171746+0000","last_became_active":"2024-04-24T08:07:51.058654+0000","last_became_peered":"2024-04-24T08:07:51.058654+0000","last_unstale":"2024-04-24T08:08:51.171746+0000","last_undegraded":"2024-04-24T08:08:51.171746+0000","last_fullsized":"2024-04-24T08:08:51.171746+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:13.354590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.069581+0000","last_change":"2024-04-24T08:07:52.051052+0000","last_active":"2024-04-24T08:09:20.069581+0000","last_peered":"2024-04-24T08:09:20.069581+0000","last_clean":"2024-04-24T08:09:20.069581+0000","last_became_active":"2024-04-24T08:07:52.050685+0000","last_became_peered":"2024-04-24T08:07:52.050685+0000","last_unstale":"2024-04-24T08:09:20.069581+0000","last_undegraded":"2024-04-24T08:09:20.069581+0000","last_fullsized":"2024-04-24T08:09:20.069581+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:34.491653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.172455+0000","last_change":"2024-04-24T08:07:51.041088+0000","last_active":"2024-04-24T08:08:51.172455+0000","last_peered":"2024-04-24T08:08:51.172455+0000","last_clean":"2024-04-24T08:08:51.172455+0000","last_became_active":"2024-04-24T08:07:51.040752+0000","last_became_peered":"2024-04-24T08:07:51.040752+0000","last_unstale":"2024-04-24T08:08:51.172455+0000","last_undegraded":"2024-04-24T08:08:51.172455+0000","last_fullsized":"2024-04-24T08:08:51.172455+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:48:23.405705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.071141+0000","last_change":"2024-04-24T08:08:51.173331+0000","last_active":"2024-04-24T08:09:20.071141+0000","last_peered":"2024-04-24T08:09:20.071141+0000","last_clean":"2024-04-24T08:09:20.071141+0000","last_became_active":"2024-04-24T08:08:51.172912+0000","last_became_peered":"2024-04-24T08:08:51.172912+0000","last_unstale":"2024-04-24T08:09:20.071141+0000","last_undegraded":"2024-04-24T08:09:20.071141+0000","last_fullsized":"2024-04-24T08:09:20.071141+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:54:35.241315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.164883+0000","last_change":"2024-04-24T08:07:51.075583+0000","last_active":"2024-04-24T08:08:51.164883+0000","last_peered":"2024-04-24T08:08:51.164883+0000","last_clean":"2024-04-24T08:08:51.164883+0000","last_became_active":"2024-04-24T08:07:51.075111+0000","last_became_peered":"2024-04-24T08:07:51.075111+0000","last_unstale":"2024-04-24T08:08:51.164883+0000","last_undegraded":"2024-04-24T08:08:51.164883+0000","last_fullsized":"2024-04-24T08:08:51.164883+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:56:00.820057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.094725+0000","last_change":"2024-04-24T08:07:52.063232+0000","last_active":"2024-04-24T08:09:20.094725+0000","last_peered":"2024-04-24T08:09:20.094725+0000","last_clean":"2024-04-24T08:09:20.094725+0000","last_became_active":"2024-04-24T08:07:52.062868+0000","last_became_peered":"2024-04-24T08:07:52.062868+0000","last_unstale":"2024-04-24T08:09:20.094725+0000","last_undegraded":"2024-04-24T08:09:20.094725+0000","last_fullsized":"2024-04-24T08:09:20.094725+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:00:40.333389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.585498+0000","last_change":"2024-04-24T08:07:51.070647+0000","last_active":"2024-04-24T08:08:51.585498+0000","last_peered":"2024-04-24T08:08:51.585498+0000","last_clean":"2024-04-24T08:08:51.585498+0000","last_became_active":"2024-04-24T08:07:51.070298+0000","last_became_peered":"2024-04-24T08:07:51.070298+0000","last_unstale":"2024-04-24T08:08:51.585498+0000","last_undegraded":"2024-04-24T08:08:51.585498+0000","last_fullsized":"2024-04-24T08:08:51.585498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:56:41.706796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.089215+0000","last_change":"2024-04-24T08:07:52.073417+0000","last_active":"2024-04-24T08:09:20.089215+0000","last_peered":"2024-04-24T08:09:20.089215+0000","last_clean":"2024-04-24T08:09:20.089215+0000","last_became_active":"2024-04-24T08:07:52.073051+0000","last_became_peered":"2024-04-24T08:07:52.073051+0000","last_unstale":"2024-04-24T08:09:20.089215+0000","last_undegraded":"2024-04-24T08:09:20.089215+0000","last_fullsized":"2024-04-24T08:09:20.089215+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:54:58.338279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.089152+0000","last_change":"2024-04-24T08:08:51.172525+0000","last_active":"2024-04-24T08:09:20.089152+0000","last_peered":"2024-04-24T08:09:20.089152+0000","last_clean":"2024-04-24T08:09:20.089152+0000","last_became_active":"2024-04-24T08:08:51.171924+0000","last_became_peered":"2024-04-24T08:08:51.171924+0000","last_unstale":"2024-04-24T08:09:20.089152+0000","last_undegraded":"2024-04-24T08:09:20.089152+0000","last_fullsized":"2024-04-24T08:09:20.089152+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:25:42.067150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"88'5","reported_seq":111,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:22.316726+0000","last_change":"2024-04-24T08:07:51.076937+0000","last_active":"2024-04-24T08:09:22.316726+0000","last_peered":"2024-04-24T08:09:22.316726+0000","last_clean":"2024-04-24T08:09:22.316726+0000","last_became_active":"2024-04-24T08:07:51.076621+0000","last_became_peered":"2024-04-24T08:07:51.076621+0000","last_unstale":"2024-04-24T08:09:22.316726+0000","last_undegraded":"2024-04-24T08:09:22.316726+0000","last_fullsized":"2024-04-24T08:09:22.316726+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:09.966994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":87,"num_read_kb":87,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.110220+0000","last_change":"2024-04-24T08:07:52.054150+0000","last_active":"2024-04-24T08:09:20.110220+0000","last_peered":"2024-04-24T08:09:20.110220+0000","last_clean":"2024-04-24T08:09:20.110220+0000","last_became_active":"2024-04-24T08:07:52.053817+0000","last_became_peered":"2024-04-24T08:07:52.053817+0000","last_unstale":"2024-04-24T08:09:20.110220+0000","last_undegraded":"2024-04-24T08:09:20.110220+0000","last_fullsized":"2024-04-24T08:09:20.110220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:46:16.570004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.168147+0000","last_change":"2024-04-24T08:07:51.045431+0000","last_active":"2024-04-24T08:08:51.168147+0000","last_peered":"2024-04-24T08:08:51.168147+0000","last_clean":"2024-04-24T08:08:51.168147+0000","last_became_active":"2024-04-24T08:07:51.045112+0000","last_became_peered":"2024-04-24T08:07:51.045112+0000","last_unstale":"2024-04-24T08:08:51.168147+0000","last_undegraded":"2024-04-24T08:08:51.168147+0000","last_fullsized":"2024-04-24T08:08:51.168147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:52:05.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":352,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.585492+0000","last_change":"2024-04-24T08:06:31.583310+0000","last_active":"2024-04-24T08:08:51.585492+0000","last_peered":"2024-04-24T08:08:51.585492+0000","last_clean":"2024-04-24T08:08:51.585492+0000","last_became_active":"2024-04-24T08:06:31.582735+0000","last_became_peered":"2024-04-24T08:06:31.582735+0000","last_unstale":"2024-04-24T08:08:51.585492+0000","last_undegraded":"2024-04-24T08:08:51.585492+0000","last_fullsized":"2024-04-24T08:08:51.585492+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:02:34.411025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:02:34.411025+0000","last_clean_scrub_stamp":"2024-04-24T08:02:34.411025+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:48.816092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.087463+0000","last_change":"2024-04-24T08:07:52.063961+0000","last_active":"2024-04-24T08:09:20.087463+0000","last_peered":"2024-04-24T08:09:20.087463+0000","last_clean":"2024-04-24T08:09:20.087463+0000","last_became_active":"2024-04-24T08:07:52.063608+0000","last_became_peered":"2024-04-24T08:07:52.063608+0000","last_unstale":"2024-04-24T08:09:20.087463+0000","last_undegraded":"2024-04-24T08:09:20.087463+0000","last_fullsized":"2024-04-24T08:09:20.087463+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:40:50.233349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.585566+0000","last_change":"2024-04-24T08:07:51.045626+0000","last_active":"2024-04-24T08:08:51.585566+0000","last_peered":"2024-04-24T08:08:51.585566+0000","last_clean":"2024-04-24T08:08:51.585566+0000","last_became_active":"2024-04-24T08:07:51.045293+0000","last_became_peered":"2024-04-24T08:07:51.045293+0000","last_unstale":"2024-04-24T08:08:51.585566+0000","last_undegraded":"2024-04-24T08:08:51.585566+0000","last_fullsized":"2024-04-24T08:08:51.585566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:13:04.601331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.172222+0000","last_change":"2024-04-24T08:07:51.079228+0000","last_active":"2024-04-24T08:08:51.172222+0000","last_peered":"2024-04-24T08:08:51.172222+0000","last_clean":"2024-04-24T08:08:51.172222+0000","last_became_active":"2024-04-24T08:07:51.078932+0000","last_became_peered":"2024-04-24T08:07:51.078932+0000","last_unstale":"2024-04-24T08:08:51.172222+0000","last_undegraded":"2024-04-24T08:08:51.172222+0000","last_fullsized":"2024-04-24T08:08:51.172222+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:58:32.024133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.098129+0000","last_change":"2024-04-24T08:07:52.065943+0000","last_active":"2024-04-24T08:09:20.098129+0000","last_peered":"2024-04-24T08:09:20.098129+0000","last_clean":"2024-04-24T08:09:20.098129+0000","last_became_active":"2024-04-24T08:07:52.062810+0000","last_became_peered":"2024-04-24T08:07:52.062810+0000","last_unstale":"2024-04-24T08:09:20.098129+0000","last_undegraded":"2024-04-24T08:09:20.098129+0000","last_fullsized":"2024-04-24T08:09:20.098129+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:40:08.803652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'10","reported_seq":41,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.193829+0000","last_change":"2024-04-24T08:08:51.193009+0000","last_active":"2024-04-24T08:08:51.193829+0000","last_peered":"2024-04-24T08:08:51.193829+0000","last_clean":"2024-04-24T08:08:51.193829+0000","last_became_active":"2024-04-24T08:08:51.180025+0000","last_became_peered":"2024-04-24T08:08:51.180025+0000","last_unstale":"2024-04-24T08:08:51.193829+0000","last_undegraded":"2024-04-24T08:08:51.193829+0000","last_fullsized":"2024-04-24T08:08:51.193829+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:31:19.397470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":19,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.088301+0000","last_change":"2024-04-24T08:07:52.058764+0000","last_active":"2024-04-24T08:09:20.088301+0000","last_peered":"2024-04-24T08:09:20.088301+0000","last_clean":"2024-04-24T08:09:20.088301+0000","last_became_active":"2024-04-24T08:07:52.058336+0000","last_became_peered":"2024-04-24T08:07:52.058336+0000","last_unstale":"2024-04-24T08:09:20.088301+0000","last_undegraded":"2024-04-24T08:09:20.088301+0000","last_fullsized":"2024-04-24T08:09:20.088301+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:05:37.245521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.168801+0000","last_change":"2024-04-24T08:07:51.067154+0000","last_active":"2024-04-24T08:08:51.168801+0000","last_peered":"2024-04-24T08:08:51.168801+0000","last_clean":"2024-04-24T08:08:51.168801+0000","last_became_active":"2024-04-24T08:07:51.066769+0000","last_became_peered":"2024-04-24T08:07:51.066769+0000","last_unstale":"2024-04-24T08:08:51.168801+0000","last_undegraded":"2024-04-24T08:08:51.168801+0000","last_fullsized":"2024-04-24T08:08:51.168801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:39:25.908189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.163484+0000","last_change":"2024-04-24T08:07:52.072134+0000","last_active":"2024-04-24T08:08:51.163484+0000","last_peered":"2024-04-24T08:08:51.163484+0000","last_clean":"2024-04-24T08:08:51.163484+0000","last_became_active":"2024-04-24T08:07:52.071703+0000","last_became_peered":"2024-04-24T08:07:52.071703+0000","last_unstale":"2024-04-24T08:08:51.163484+0000","last_undegraded":"2024-04-24T08:08:51.163484+0000","last_fullsized":"2024-04-24T08:08:51.163484+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:59.829988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.167881+0000","last_change":"2024-04-24T08:07:51.059151+0000","last_active":"2024-04-24T08:08:51.167881+0000","last_peered":"2024-04-24T08:08:51.167881+0000","last_clean":"2024-04-24T08:08:51.167881+0000","last_became_active":"2024-04-24T08:07:51.058872+0000","last_became_peered":"2024-04-24T08:07:51.058872+0000","last_unstale":"2024-04-24T08:08:51.167881+0000","last_undegraded":"2024-04-24T08:08:51.167881+0000","last_fullsized":"2024-04-24T08:08:51.167881+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:36.023595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.026883+0000","last_change":"2024-04-24T08:07:52.056919+0000","last_active":"2024-04-24T08:09:20.026883+0000","last_peered":"2024-04-24T08:09:20.026883+0000","last_clean":"2024-04-24T08:09:20.026883+0000","last_became_active":"2024-04-24T08:07:52.056573+0000","last_became_peered":"2024-04-24T08:07:52.056573+0000","last_unstale":"2024-04-24T08:09:20.026883+0000","last_undegraded":"2024-04-24T08:09:20.026883+0000","last_fullsized":"2024-04-24T08:09:20.026883+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:58:17.939465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.175838+0000","last_change":"2024-04-24T08:07:51.056325+0000","last_active":"2024-04-24T08:08:51.175838+0000","last_peered":"2024-04-24T08:08:51.175838+0000","last_clean":"2024-04-24T08:08:51.175838+0000","last_became_active":"2024-04-24T08:07:51.055862+0000","last_became_peered":"2024-04-24T08:07:51.055862+0000","last_unstale":"2024-04-24T08:08:51.175838+0000","last_undegraded":"2024-04-24T08:08:51.175838+0000","last_fullsized":"2024-04-24T08:08:51.175838+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:11:44.487530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.090628+0000","last_change":"2024-04-24T08:07:52.065570+0000","last_active":"2024-04-24T08:09:20.090628+0000","last_peered":"2024-04-24T08:09:20.090628+0000","last_clean":"2024-04-24T08:09:20.090628+0000","last_became_active":"2024-04-24T08:07:52.065196+0000","last_became_peered":"2024-04-24T08:07:52.065196+0000","last_unstale":"2024-04-24T08:09:20.090628+0000","last_undegraded":"2024-04-24T08:09:20.090628+0000","last_fullsized":"2024-04-24T08:09:20.090628+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:48:09.474159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.168540+0000","last_change":"2024-04-24T08:07:51.067187+0000","last_active":"2024-04-24T08:08:51.168540+0000","last_peered":"2024-04-24T08:08:51.168540+0000","last_clean":"2024-04-24T08:08:51.168540+0000","last_became_active":"2024-04-24T08:07:51.066808+0000","last_became_peered":"2024-04-24T08:07:51.066808+0000","last_unstale":"2024-04-24T08:08:51.168540+0000","last_undegraded":"2024-04-24T08:08:51.168540+0000","last_fullsized":"2024-04-24T08:08:51.168540+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:32:11.852191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.106375+0000","last_change":"2024-04-24T08:07:52.078426+0000","last_active":"2024-04-24T08:09:20.106375+0000","last_peered":"2024-04-24T08:09:20.106375+0000","last_clean":"2024-04-24T08:09:20.106375+0000","last_became_active":"2024-04-24T08:07:52.078076+0000","last_became_peered":"2024-04-24T08:07:52.078076+0000","last_unstale":"2024-04-24T08:09:20.106375+0000","last_undegraded":"2024-04-24T08:09:20.106375+0000","last_fullsized":"2024-04-24T08:09:20.106375+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:55:29.885688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.171843+0000","last_change":"2024-04-24T08:07:51.042601+0000","last_active":"2024-04-24T08:08:51.171843+0000","last_peered":"2024-04-24T08:08:51.171843+0000","last_clean":"2024-04-24T08:08:51.171843+0000","last_became_active":"2024-04-24T08:07:51.042252+0000","last_became_peered":"2024-04-24T08:07:51.042252+0000","last_unstale":"2024-04-24T08:08:51.171843+0000","last_undegraded":"2024-04-24T08:08:51.171843+0000","last_fullsized":"2024-04-24T08:08:51.171843+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:11:44.640303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.094565+0000","last_change":"2024-04-24T08:07:52.063142+0000","last_active":"2024-04-24T08:09:20.094565+0000","last_peered":"2024-04-24T08:09:20.094565+0000","last_clean":"2024-04-24T08:09:20.094565+0000","last_became_active":"2024-04-24T08:07:52.062263+0000","last_became_peered":"2024-04-24T08:07:52.062263+0000","last_unstale":"2024-04-24T08:09:20.094565+0000","last_undegraded":"2024-04-24T08:09:20.094565+0000","last_fullsized":"2024-04-24T08:09:20.094565+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:38.879380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.185189+0000","last_change":"2024-04-24T08:08:51.184795+0000","last_active":"2024-04-24T08:08:51.185189+0000","last_peered":"2024-04-24T08:08:51.185189+0000","last_clean":"2024-04-24T08:08:51.185189+0000","last_became_active":"2024-04-24T08:08:51.179560+0000","last_became_peered":"2024-04-24T08:08:51.179560+0000","last_unstale":"2024-04-24T08:08:51.185189+0000","last_undegraded":"2024-04-24T08:08:51.185189+0000","last_fullsized":"2024-04-24T08:08:51.185189+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:31:23.795126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.077181+0000","last_change":"2024-04-24T08:07:52.055958+0000","last_active":"2024-04-24T08:09:20.077181+0000","last_peered":"2024-04-24T08:09:20.077181+0000","last_clean":"2024-04-24T08:09:20.077181+0000","last_became_active":"2024-04-24T08:07:52.055598+0000","last_became_peered":"2024-04-24T08:07:52.055598+0000","last_unstale":"2024-04-24T08:09:20.077181+0000","last_undegraded":"2024-04-24T08:09:20.077181+0000","last_fullsized":"2024-04-24T08:09:20.077181+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:12:22.452576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"85'1","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.168668+0000","last_change":"2024-04-24T08:07:51.073234+0000","last_active":"2024-04-24T08:08:51.168668+0000","last_peered":"2024-04-24T08:08:51.168668+0000","last_clean":"2024-04-24T08:08:51.168668+0000","last_became_active":"2024-04-24T08:07:51.072887+0000","last_became_peered":"2024-04-24T08:07:51.072887+0000","last_unstale":"2024-04-24T08:08:51.168668+0000","last_undegraded":"2024-04-24T08:08:51.168668+0000","last_fullsized":"2024-04-24T08:08:51.168668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:28:15.908859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.067674+0000","last_change":"2024-04-24T08:07:52.061188+0000","last_active":"2024-04-24T08:09:20.067674+0000","last_peered":"2024-04-24T08:09:20.067674+0000","last_clean":"2024-04-24T08:09:20.067674+0000","last_became_active":"2024-04-24T08:07:52.059616+0000","last_became_peered":"2024-04-24T08:07:52.059616+0000","last_unstale":"2024-04-24T08:09:20.067674+0000","last_undegraded":"2024-04-24T08:09:20.067674+0000","last_fullsized":"2024-04-24T08:09:20.067674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:43.546091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.249320+0000","last_change":"2024-04-24T08:07:51.078300+0000","last_active":"2024-04-24T08:08:51.249320+0000","last_peered":"2024-04-24T08:08:51.249320+0000","last_clean":"2024-04-24T08:08:51.249320+0000","last_became_active":"2024-04-24T08:07:51.076026+0000","last_became_peered":"2024-04-24T08:07:51.076026+0000","last_unstale":"2024-04-24T08:08:51.249320+0000","last_undegraded":"2024-04-24T08:08:51.249320+0000","last_fullsized":"2024-04-24T08:08:51.249320+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:45:31.494213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.073565+0000","last_change":"2024-04-24T08:07:52.056148+0000","last_active":"2024-04-24T08:09:20.073565+0000","last_peered":"2024-04-24T08:09:20.073565+0000","last_clean":"2024-04-24T08:09:20.073565+0000","last_became_active":"2024-04-24T08:07:52.055768+0000","last_became_peered":"2024-04-24T08:07:52.055768+0000","last_unstale":"2024-04-24T08:09:20.073565+0000","last_undegraded":"2024-04-24T08:09:20.073565+0000","last_fullsized":"2024-04-24T08:09:20.073565+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:52:48.503479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.071668+0000","last_change":"2024-04-24T08:07:52.056611+0000","last_active":"2024-04-24T08:09:20.071668+0000","last_peered":"2024-04-24T08:09:20.071668+0000","last_clean":"2024-04-24T08:09:20.071668+0000","last_became_active":"2024-04-24T08:07:52.055788+0000","last_became_peered":"2024-04-24T08:07:52.055788+0000","last_unstale":"2024-04-24T08:09:20.071668+0000","last_undegraded":"2024-04-24T08:09:20.071668+0000","last_fullsized":"2024-04-24T08:09:20.071668+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:21:28.997588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.177574+0000","last_change":"2024-04-24T08:08:51.177574+0000","last_active":"2024-04-24T08:08:51.177574+0000","last_peered":"2024-04-24T08:08:51.177574+0000","last_clean":"2024-04-24T08:08:51.177574+0000","last_became_active":"2024-04-24T08:08:51.177145+0000","last_became_peered":"2024-04-24T08:08:51.177145+0000","last_unstale":"2024-04-24T08:08:51.177574+0000","last_undegraded":"2024-04-24T08:08:51.177574+0000","last_fullsized":"2024-04-24T08:08:51.177574+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:49:19.438233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.168773+0000","last_change":"2024-04-24T08:07:52.070899+0000","last_active":"2024-04-24T08:08:51.168773+0000","last_peered":"2024-04-24T08:08:51.168773+0000","last_clean":"2024-04-24T08:08:51.168773+0000","last_became_active":"2024-04-24T08:07:52.070556+0000","last_became_peered":"2024-04-24T08:07:52.070556+0000","last_unstale":"2024-04-24T08:08:51.168773+0000","last_undegraded":"2024-04-24T08:08:51.168773+0000","last_fullsized":"2024-04-24T08:08:51.168773+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:40:11.381122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.250206+0000","last_change":"2024-04-24T08:08:51.250206+0000","last_active":"2024-04-24T08:08:51.250206+0000","last_peered":"2024-04-24T08:08:51.250206+0000","last_clean":"2024-04-24T08:08:51.250206+0000","last_became_active":"2024-04-24T08:08:51.249677+0000","last_became_peered":"2024-04-24T08:08:51.249677+0000","last_unstale":"2024-04-24T08:08:51.250206+0000","last_undegraded":"2024-04-24T08:08:51.250206+0000","last_fullsized":"2024-04-24T08:08:51.250206+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:06:26.655987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.101706+0000","last_change":"2024-04-24T08:07:52.078604+0000","last_active":"2024-04-24T08:09:20.101706+0000","last_peered":"2024-04-24T08:09:20.101706+0000","last_clean":"2024-04-24T08:09:20.101706+0000","last_became_active":"2024-04-24T08:07:52.078122+0000","last_became_peered":"2024-04-24T08:07:52.078122+0000","last_unstale":"2024-04-24T08:09:20.101706+0000","last_undegraded":"2024-04-24T08:09:20.101706+0000","last_fullsized":"2024-04-24T08:09:20.101706+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:24:19.062821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.070864+0000","last_change":"2024-04-24T08:07:52.055986+0000","last_active":"2024-04-24T08:09:20.070864+0000","last_peered":"2024-04-24T08:09:20.070864+0000","last_clean":"2024-04-24T08:09:20.070864+0000","last_became_active":"2024-04-24T08:07:52.055609+0000","last_became_peered":"2024-04-24T08:07:52.055609+0000","last_unstale":"2024-04-24T08:09:20.070864+0000","last_undegraded":"2024-04-24T08:09:20.070864+0000","last_fullsized":"2024-04-24T08:09:20.070864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:38:04.731204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:08:51.169573+0000","last_change":"2024-04-24T08:07:52.055605+0000","last_active":"2024-04-24T08:08:51.169573+0000","last_peered":"2024-04-24T08:08:51.169573+0000","last_clean":"2024-04-24T08:08:51.169573+0000","last_became_active":"2024-04-24T08:07:52.055268+0000","last_became_peered":"2024-04-24T08:07:52.055268+0000","last_unstale":"2024-04-24T08:08:51.169573+0000","last_undegraded":"2024-04-24T08:08:51.169573+0000","last_fullsized":"2024-04-24T08:08:51.169573+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:50:29.197484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.081887+0000","last_change":"2024-04-24T08:07:52.064590+0000","last_active":"2024-04-24T08:09:20.081887+0000","last_peered":"2024-04-24T08:09:20.081887+0000","last_clean":"2024-04-24T08:09:20.081887+0000","last_became_active":"2024-04-24T08:07:52.064113+0000","last_became_peered":"2024-04-24T08:07:52.064113+0000","last_unstale":"2024-04-24T08:09:20.081887+0000","last_undegraded":"2024-04-24T08:09:20.081887+0000","last_fullsized":"2024-04-24T08:09:20.081887+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:36:10.926145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.094851+0000","last_change":"2024-04-24T08:08:51.250599+0000","last_active":"2024-04-24T08:09:20.094851+0000","last_peered":"2024-04-24T08:09:20.094851+0000","last_clean":"2024-04-24T08:09:20.094851+0000","last_became_active":"2024-04-24T08:08:51.250197+0000","last_became_peered":"2024-04-24T08:08:51.250197+0000","last_unstale":"2024-04-24T08:09:20.094851+0000","last_undegraded":"2024-04-24T08:09:20.094851+0000","last_fullsized":"2024-04-24T08:09:20.094851+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:31:30.729911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6],"acting":[5,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.085888+0000","last_change":"2024-04-24T08:07:52.068308+0000","last_active":"2024-04-24T08:09:20.085888+0000","last_peered":"2024-04-24T08:09:20.085888+0000","last_clean":"2024-04-24T08:09:20.085888+0000","last_became_active":"2024-04-24T08:07:52.067786+0000","last_became_peered":"2024-04-24T08:07:52.067786+0000","last_unstale":"2024-04-24T08:09:20.085888+0000","last_undegraded":"2024-04-24T08:09:20.085888+0000","last_fullsized":"2024-04-24T08:09:20.085888+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:26:09.349120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.107687+0000","last_change":"2024-04-24T08:07:52.053089+0000","last_active":"2024-04-24T08:09:20.107687+0000","last_peered":"2024-04-24T08:09:20.107687+0000","last_clean":"2024-04-24T08:09:20.107687+0000","last_became_active":"2024-04-24T08:07:52.052753+0000","last_became_peered":"2024-04-24T08:07:52.052753+0000","last_unstale":"2024-04-24T08:09:20.107687+0000","last_undegraded":"2024-04-24T08:09:20.107687+0000","last_fullsized":"2024-04-24T08:09:20.107687+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:02:14.583922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.104000+0000","last_change":"2024-04-24T08:07:52.076633+0000","last_active":"2024-04-24T08:09:20.104000+0000","last_peered":"2024-04-24T08:09:20.104000+0000","last_clean":"2024-04-24T08:09:20.104000+0000","last_became_active":"2024-04-24T08:07:52.076280+0000","last_became_peered":"2024-04-24T08:07:52.076280+0000","last_unstale":"2024-04-24T08:09:20.104000+0000","last_undegraded":"2024-04-24T08:09:20.104000+0000","last_fullsized":"2024-04-24T08:09:20.104000+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:29:59.823772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-04-24T08:09:20.069876+0000","last_change":"2024-04-24T08:07:52.053017+0000","last_active":"2024-04-24T08:09:20.069876+0000","last_peered":"2024-04-24T08:09:20.069876+0000","last_clean":"2024-04-24T08:09:20.069876+0000","last_became_active":"2024-04-24T08:07:52.052325+0000","last_became_peered":"2024-04-24T08:07:52.052325+0000","last_unstale":"2024-04-24T08:09:20.069876+0000","last_undegraded":"2024-04-24T08:09:20.069876+0000","last_fullsized":"2024-04-24T08:09:20.069876+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:05:58.161360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":406,"ondisk_log_size":406,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":327680,"data_stored":3542208,"data_compressed":213304,"data_compressed_allocated":270336,"data_compressed_original":3506176,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":63803,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":548,"num_read_kb":341,"num_write":169,"num_write_kb":115,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":616,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":191019,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351009,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36272,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708880,"statfs":{"total":95995035648,"available":95957893120,"internally_reserved":0,"allocated":958464,"data_stored":462211,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612645,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28096,"kb_used_data":1080,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717056,"statfs":{"total":95995035648,"available":95966265344,"internally_reserved":0,"allocated":1105920,"data_stored":1113319,"data_compressed":41899,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776171,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":970752,"data_stored":462413,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037809,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32124,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713028,"statfs":{"total":95995035648,"available":95962140672,"internally_reserved":0,"allocated":970752,"data_stored":462333,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332151,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":1054936,"data_compressed":41899,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495678,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":461802,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1019904,"data_stored":520475,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738462,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":463073,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1036288,"data_stored":521600,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182478,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":462113,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920841,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":463353,"data_compressed":5801,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757316,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32184,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712968,"statfs":{"total":95995035648,"available":95962079232,"internally_reserved":0,"allocated":1032192,"data_stored":1054088,"data_compressed":41899,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34456,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":59129,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":60005,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":1758,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2856,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":61239,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":616,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:09:26.003 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-04-24T08:09:26.003 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-04-24T08:09:26.003 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-04-24T08:09:26.003 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph health --format=json 2024-04-24T08:09:26.526 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:09:26.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:26 smithi117 ceph-mon[32438]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:09:26.681 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:26 smithi028 ceph-mon[27786]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:09:26.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:26 smithi194 ceph-mon[32836]: from='client.14790 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:09:27.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:27 smithi117 ceph-mon[32438]: pgmap v366: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-24T08:09:27.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:27 smithi194 ceph-mon[32836]: pgmap v366: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-24T08:09:27.774 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:27 smithi028 ceph-mon[27786]: pgmap v366: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-24T08:09:28.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.788+0000 7efd75000700 1 -- 172.21.15.28:0/2876206169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd70109d30 msgr2=0x7efd7010dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:28.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.788+0000 7efd75000700 1 --2- 172.21.15.28:0/2876206169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd70109d30 0x7efd7010dd80 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7efd580098f0 tx=0x7efd58009c00 comp rx=0 tx=0).stop 2024-04-24T08:09:28.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.788+0000 7efd75000700 1 -- 172.21.15.28:0/2876206169 shutdown_connections 2024-04-24T08:09:28.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.788+0000 7efd75000700 1 --2- 172.21.15.28:0/2876206169 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd7010e460 0x7efd70116030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:28.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.788+0000 7efd75000700 1 --2- 172.21.15.28:0/2876206169 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd70109380 0x7efd70109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:28.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.788+0000 7efd75000700 1 --2- 172.21.15.28:0/2876206169 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd70109d30 0x7efd7010dd80 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:28.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.788+0000 7efd75000700 1 -- 172.21.15.28:0/2876206169 >> 172.21.15.28:0/2876206169 conn(0x7efd700762c0 msgr2=0x7efd70107910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:28.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.788+0000 7efd75000700 1 -- 172.21.15.28:0/2876206169 shutdown_connections 2024-04-24T08:09:28.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.788+0000 7efd75000700 1 -- 172.21.15.28:0/2876206169 wait complete. 2024-04-24T08:09:28.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.790+0000 7efd75000700 1 Processor -- start 2024-04-24T08:09:28.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.790+0000 7efd75000700 1 -- start start 2024-04-24T08:09:28.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.790+0000 7efd75000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd70109380 0x7efd701a5660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:28.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.790+0000 7efd75000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd70109d30 0x7efd701a5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:28.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.790+0000 7efd75000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd7010e460 0x7efd7019f730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:28.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.790+0000 7efd75000700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd70119470 con 0x7efd70109380 2024-04-24T08:09:28.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.790+0000 7efd75000700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efd701192f0 con 0x7efd70109d30 2024-04-24T08:09:28.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.790+0000 7efd75000700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efd701195f0 con 0x7efd7010e460 2024-04-24T08:09:28.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.791+0000 7efd6de00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd70109380 0x7efd701a5660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:28.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.791+0000 7efd67e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd70109d30 0x7efd701a5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:28.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.791+0000 7efd67e00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd70109d30 0x7efd701a5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:48056/0 (socket says 172.21.15.28:48056) 2024-04-24T08:09:28.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.791+0000 7efd67e00700 1 -- 172.21.15.28:0/3109798303 learned_addr learned my addr 172.21.15.28:0/3109798303 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:28.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.791+0000 7efd6e800700 1 --2- 172.21.15.28:0/3109798303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd7010e460 0x7efd7019f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:28.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.791+0000 7efd67e00700 1 -- 172.21.15.28:0/3109798303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd7010e460 msgr2=0x7efd7019f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:28.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.791+0000 7efd67e00700 1 --2- 172.21.15.28:0/3109798303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd7010e460 0x7efd7019f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:28.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.791+0000 7efd67e00700 1 -- 172.21.15.28:0/3109798303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd70109380 msgr2=0x7efd701a5660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:28.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.791+0000 7efd67e00700 1 --2- 172.21.15.28:0/3109798303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd70109380 0x7efd701a5660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:28.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.791+0000 7efd67e00700 1 -- 172.21.15.28:0/3109798303 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd701a0010 con 0x7efd70109d30 2024-04-24T08:09:28.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.791+0000 7efd67e00700 1 --2- 172.21.15.28:0/3109798303 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd70109d30 0x7efd701a5ba0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efd58009fd0 tx=0x7efd580094b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:28.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.792+0000 7efd66a00700 1 -- 172.21.15.28:0/3109798303 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd58025070 con 0x7efd70109d30 2024-04-24T08:09:28.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.792+0000 7efd66a00700 1 -- 172.21.15.28:0/3109798303 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd5800ba40 con 0x7efd70109d30 2024-04-24T08:09:28.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.792+0000 7efd66a00700 1 -- 172.21.15.28:0/3109798303 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd58004b90 con 0x7efd70109d30 2024-04-24T08:09:28.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.792+0000 7efd75000700 1 -- 172.21.15.28:0/3109798303 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd701a02a0 con 0x7efd70109d30 2024-04-24T08:09:28.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.792+0000 7efd75000700 1 -- 172.21.15.28:0/3109798303 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd701ac5e0 con 0x7efd70109d30 2024-04-24T08:09:28.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.793+0000 7efd75000700 1 -- 172.21.15.28:0/3109798303 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd701a0600 con 0x7efd70109d30 2024-04-24T08:09:28.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.795+0000 7efd66a00700 1 -- 172.21.15.28:0/3109798303 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7efd58028070 con 0x7efd70109d30 2024-04-24T08:09:28.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.797+0000 7efd66a00700 1 --2- 172.21.15.28:0/3109798303 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd540a3cc0 0x7efd540a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:28.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.797+0000 7efd66a00700 1 -- 172.21.15.28:0/3109798303 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7efd580d0b10 con 0x7efd70109d30 2024-04-24T08:09:28.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.797+0000 7efd6de00700 1 --2- 172.21.15.28:0/3109798303 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd540a3cc0 0x7efd540a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:28.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.798+0000 7efd6de00700 1 --2- 172.21.15.28:0/3109798303 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd540a3cc0 0x7efd540a6180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efd60005fd0 tx=0x7efd60009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:28.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:28.801+0000 7efd66a00700 1 -- 172.21.15.28:0/3109798303 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efd5805cff0 con 0x7efd70109d30 2024-04-24T08:09:29.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:29.092+0000 7efd75000700 1 -- 172.21.15.28:0/3109798303 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7efd70069810 con 0x7efd70109d30 2024-04-24T08:09:29.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:29.093+0000 7efd66a00700 1 -- 172.21.15.28:0/3109798303 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7efd70069810 con 0x7efd70109d30 2024-04-24T08:09:29.094 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:09:29.094 INFO:teuthology.orchestra.run.smithi028.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-04-24T08:09:29.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:29.096+0000 7efd75000700 1 -- 172.21.15.28:0/3109798303 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd540a3cc0 msgr2=0x7efd540a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:29.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:29.096+0000 7efd75000700 1 --2- 172.21.15.28:0/3109798303 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd540a3cc0 0x7efd540a6180 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7efd60005fd0 tx=0x7efd60009310 comp rx=0 tx=0).stop 2024-04-24T08:09:29.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:29.096+0000 7efd75000700 1 -- 172.21.15.28:0/3109798303 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd70109d30 msgr2=0x7efd701a5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:29.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:29.096+0000 7efd75000700 1 --2- 172.21.15.28:0/3109798303 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd70109d30 0x7efd701a5ba0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7efd58009fd0 tx=0x7efd580094b0 comp rx=0 tx=0).stop 2024-04-24T08:09:29.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:29.096+0000 7efd75000700 1 -- 172.21.15.28:0/3109798303 shutdown_connections 2024-04-24T08:09:29.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:29.096+0000 7efd75000700 1 --2- 172.21.15.28:0/3109798303 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7efd540a3cc0 0x7efd540a6180 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:29.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:29.096+0000 7efd75000700 1 --2- 172.21.15.28:0/3109798303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd7010e460 0x7efd7019f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:29.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:29.096+0000 7efd75000700 1 --2- 172.21.15.28:0/3109798303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd70109380 0x7efd701a5660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:29.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:29.096+0000 7efd75000700 1 --2- 172.21.15.28:0/3109798303 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd70109d30 0x7efd701a5ba0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:29.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:29.096+0000 7efd75000700 1 -- 172.21.15.28:0/3109798303 >> 172.21.15.28:0/3109798303 conn(0x7efd700762c0 msgr2=0x7efd70077ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:29.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:29.096+0000 7efd75000700 1 -- 172.21.15.28:0/3109798303 shutdown_connections 2024-04-24T08:09:29.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:29.096+0000 7efd75000700 1 -- 172.21.15.28:0/3109798303 wait complete. 2024-04-24T08:09:29.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:29 smithi028 ceph-mon[27786]: pgmap v367: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-24T08:09:29.507 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:29 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3109798303' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-24T08:09:29.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:29 smithi117 ceph-mon[32438]: pgmap v367: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-24T08:09:29.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:29 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3109798303' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-24T08:09:29.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:29 smithi194 ceph-mon[32836]: pgmap v367: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-24T08:09:29.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:29 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3109798303' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-04-24T08:09:29.917 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-04-24T08:09:29.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.149+0000 7f095b200640 1 -- 172.21.15.28:0/3749041959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f095c074a60 msgr2=0x7f095c074ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.149+0000 7f095b200640 1 --2- 172.21.15.28:0/3749041959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f095c074a60 0x7f095c074ee0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f0948009900 tx=0x7f094802a400 comp rx=0 tx=0).stop 2024-04-24T08:09:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.149+0000 7f095b200640 1 -- 172.21.15.28:0/3749041959 shutdown_connections 2024-04-24T08:09:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.149+0000 7f095b200640 1 --2- 172.21.15.28:0/3749041959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f095c074a60 0x7f095c074ee0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.149+0000 7f095b200640 1 --2- 172.21.15.28:0/3749041959 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f095c109d30 0x7f095c10c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.149+0000 7f095b200640 1 --2- 172.21.15.28:0/3749041959 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f095c076040 0x7f095c074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.149+0000 7f095b200640 1 -- 172.21.15.28:0/3749041959 >> 172.21.15.28:0/3749041959 conn(0x7f095c0fd3e0 msgr2=0x7f095c0ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.149+0000 7f095b200640 1 -- 172.21.15.28:0/3749041959 shutdown_connections 2024-04-24T08:09:30.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.149+0000 7f095b200640 1 -- 172.21.15.28:0/3749041959 wait complete. 2024-04-24T08:09:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.151+0000 7f095b200640 1 Processor -- start 2024-04-24T08:09:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.151+0000 7f095b200640 1 -- start start 2024-04-24T08:09:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.151+0000 7f095b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f095c074a60 0x7f095c19a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.151+0000 7f095b200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f095c076040 0x7f095c19a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.151+0000 7f095b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f095c109d30 0x7f095c1a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.151+0000 7f095b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f095c10fd40 con 0x7f095c109d30 2024-04-24T08:09:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.151+0000 7f095b200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f095c10fbc0 con 0x7f095c076040 2024-04-24T08:09:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.151+0000 7f095b200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f095c10fec0 con 0x7f095c074a60 2024-04-24T08:09:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.152+0000 7f095a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f095c109d30 0x7f095c1a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.152+0000 7f095a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f095c109d30 0x7f095c1a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59628/0 (socket says 172.21.15.28:59628) 2024-04-24T08:09:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.152+0000 7f095a800640 1 -- 172.21.15.28:0/2857167427 learned_addr learned my addr 172.21.15.28:0/2857167427 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.152+0000 7f0959400640 1 --2- 172.21.15.28:0/2857167427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f095c076040 0x7f095c19a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.152+0000 7f0959e00640 1 --2- 172.21.15.28:0/2857167427 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f095c074a60 0x7f095c19a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.152+0000 7f095a800640 1 -- 172.21.15.28:0/2857167427 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f095c074a60 msgr2=0x7f095c19a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.152+0000 7f095a800640 1 --2- 172.21.15.28:0/2857167427 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f095c074a60 0x7f095c19a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.152+0000 7f095a800640 1 -- 172.21.15.28:0/2857167427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f095c076040 msgr2=0x7f095c19a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.152+0000 7f095a800640 1 --2- 172.21.15.28:0/2857167427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f095c076040 0x7f095c19a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.152+0000 7f095a800640 1 -- 172.21.15.28:0/2857167427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f095c1a1fa0 con 0x7f095c109d30 2024-04-24T08:09:30.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.152+0000 7f095a800640 1 --2- 172.21.15.28:0/2857167427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f095c109d30 0x7f095c1a1930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f095400eb10 tx=0x7f095400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:30.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.153+0000 7f0952a00640 1 -- 172.21.15.28:0/2857167427 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954004b20 con 0x7f095c109d30 2024-04-24T08:09:30.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.153+0000 7f0952a00640 1 -- 172.21.15.28:0/2857167427 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0954004cc0 con 0x7f095c109d30 2024-04-24T08:09:30.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.153+0000 7f095b200640 1 -- 172.21.15.28:0/2857167427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f095c1a2290 con 0x7f095c109d30 2024-04-24T08:09:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.153+0000 7f0952a00640 1 -- 172.21.15.28:0/2857167427 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0954010760 con 0x7f095c109d30 2024-04-24T08:09:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.153+0000 7f095b200640 1 -- 172.21.15.28:0/2857167427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f095c1a2760 con 0x7f095c109d30 2024-04-24T08:09:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.154+0000 7f095b200640 1 -- 172.21.15.28:0/2857167427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f095c108a60 con 0x7f095c109d30 2024-04-24T08:09:30.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.156+0000 7f0952a00640 1 -- 172.21.15.28:0/2857167427 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0954005020 con 0x7f095c109d30 2024-04-24T08:09:30.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.157+0000 7f0952a00640 1 --2- 172.21.15.28:0/2857167427 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f09400a3bb0 0x7f09400a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:30.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.157+0000 7f0952a00640 1 -- 172.21.15.28:0/2857167427 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f09540d09f0 con 0x7f095c109d30 2024-04-24T08:09:30.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.158+0000 7f0959e00640 1 --2- 172.21.15.28:0/2857167427 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f09400a3bb0 0x7f09400a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:30.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.158+0000 7f0959e00640 1 --2- 172.21.15.28:0/2857167427 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f09400a3bb0 0x7f09400a6070 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f094c009bd0 tx=0x7f094c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.162+0000 7f0952a00640 1 -- 172.21.15.28:0/2857167427 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f095409d350 con 0x7f095c109d30 2024-04-24T08:09:30.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.371+0000 7f095b200640 1 -- 172.21.15.28:0/2857167427 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f095c062c00 con 0x7f095c109d30 2024-04-24T08:09:30.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.376+0000 7f0952a00640 1 -- 172.21.15.28:0/2857167427 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22677 (secure 0 0 0) 0x7f095409cc50 con 0x7f095c109d30 2024-04-24T08:09:30.378 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:09:30.378 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:08:51.155712+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:30.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.378+0000 7f095b200640 1 -- 172.21.15.28:0/2857167427 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f09400a3bb0 msgr2=0x7f09400a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:30.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.378+0000 7f095b200640 1 --2- 172.21.15.28:0/2857167427 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f09400a3bb0 0x7f09400a6070 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f094c009bd0 tx=0x7f094c005f90 comp rx=0 tx=0).stop 2024-04-24T08:09:30.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.379+0000 7f095b200640 1 -- 172.21.15.28:0/2857167427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f095c109d30 msgr2=0x7f095c1a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:30.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.379+0000 7f095b200640 1 --2- 172.21.15.28:0/2857167427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f095c109d30 0x7f095c1a1930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f095400eb10 tx=0x7f095400c2d0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.379+0000 7f095b200640 1 -- 172.21.15.28:0/2857167427 shutdown_connections 2024-04-24T08:09:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.379+0000 7f095b200640 1 --2- 172.21.15.28:0/2857167427 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f09400a3bb0 0x7f09400a6070 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.379+0000 7f095b200640 1 --2- 172.21.15.28:0/2857167427 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f095c109d30 0x7f095c1a1930 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.379+0000 7f095b200640 1 --2- 172.21.15.28:0/2857167427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f095c076040 0x7f095c19a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.379+0000 7f095b200640 1 --2- 172.21.15.28:0/2857167427 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f095c074a60 0x7f095c19a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.379+0000 7f095b200640 1 -- 172.21.15.28:0/2857167427 >> 172.21.15.28:0/2857167427 conn(0x7f095c0fd3e0 msgr2=0x7f095c105c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.379+0000 7f095b200640 1 -- 172.21.15.28:0/2857167427 shutdown_connections 2024-04-24T08:09:30.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.379+0000 7f095b200640 1 -- 172.21.15.28:0/2857167427 wait complete. 2024-04-24T08:09:30.401 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:02:34.086371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:07:49.819536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:07:50.493486+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:09:30.401 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:09:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.616+0000 7f4d46400640 1 -- 172.21.15.28:0/2246595818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d400749e0 msgr2=0x7f4d40074e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.616+0000 7f4d46400640 1 --2- 172.21.15.28:0/2246595818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d400749e0 0x7f4d40074e40 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c009900 tx=0x7f4d2c02a480 comp rx=0 tx=0).stop 2024-04-24T08:09:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.616+0000 7f4d46400640 1 -- 172.21.15.28:0/2246595818 shutdown_connections 2024-04-24T08:09:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.616+0000 7f4d46400640 1 --2- 172.21.15.28:0/2246595818 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d40109de0 0x7f4d4010c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.616+0000 7f4d46400640 1 --2- 172.21.15.28:0/2246595818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d400749e0 0x7f4d40074e40 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.616+0000 7f4d46400640 1 --2- 172.21.15.28:0/2246595818 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d40075fc0 0x7f4d40074410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.616+0000 7f4d46400640 1 -- 172.21.15.28:0/2246595818 >> 172.21.15.28:0/2246595818 conn(0x7f4d400fd4a0 msgr2=0x7f4d400ff8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.616+0000 7f4d46400640 1 -- 172.21.15.28:0/2246595818 shutdown_connections 2024-04-24T08:09:30.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.616+0000 7f4d46400640 1 -- 172.21.15.28:0/2246595818 wait complete. 2024-04-24T08:09:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.617+0000 7f4d46400640 1 Processor -- start 2024-04-24T08:09:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.618+0000 7f4d46400640 1 -- start start 2024-04-24T08:09:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.618+0000 7f4d46400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d400749e0 0x7f4d4019a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.618+0000 7f4d46400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d40075fc0 0x7f4d4019a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.618+0000 7f4d46400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d40109de0 0x7f4d401a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.618+0000 7f4d46400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4010fdf0 con 0x7f4d40075fc0 2024-04-24T08:09:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.618+0000 7f4d46400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4010fc70 con 0x7f4d40109de0 2024-04-24T08:09:30.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.618+0000 7f4d46400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4d4010ff70 con 0x7f4d400749e0 2024-04-24T08:09:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.618+0000 7f4d3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d40075fc0 0x7f4d4019a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.618+0000 7f4d3fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d40075fc0 0x7f4d4019a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59642/0 (socket says 172.21.15.28:59642) 2024-04-24T08:09:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.618+0000 7f4d3fe00640 1 -- 172.21.15.28:0/2890193038 learned_addr learned my addr 172.21.15.28:0/2890193038 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.618+0000 7f4d45000640 1 --2- 172.21.15.28:0/2890193038 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d400749e0 0x7f4d4019a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.619+0000 7f4d45a00640 1 --2- 172.21.15.28:0/2890193038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d40109de0 0x7f4d401a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.619+0000 7f4d3fe00640 1 -- 172.21.15.28:0/2890193038 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d400749e0 msgr2=0x7f4d4019a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.619+0000 7f4d3fe00640 1 --2- 172.21.15.28:0/2890193038 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d400749e0 0x7f4d4019a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.619+0000 7f4d3fe00640 1 -- 172.21.15.28:0/2890193038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d40109de0 msgr2=0x7f4d401a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.619+0000 7f4d3fe00640 1 --2- 172.21.15.28:0/2890193038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d40109de0 0x7f4d401a19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.619+0000 7f4d3fe00640 1 -- 172.21.15.28:0/2890193038 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d401a2040 con 0x7f4d40075fc0 2024-04-24T08:09:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.619+0000 7f4d3fe00640 1 --2- 172.21.15.28:0/2890193038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d40075fc0 0x7f4d4019a960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c031440 tx=0x7f4d2c032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.620+0000 7f4d3d600640 1 -- 172.21.15.28:0/2890193038 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c03f030 con 0x7f4d40075fc0 2024-04-24T08:09:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.620+0000 7f4d46400640 1 -- 172.21.15.28:0/2890193038 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d401a22d0 con 0x7f4d40075fc0 2024-04-24T08:09:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.620+0000 7f4d46400640 1 -- 172.21.15.28:0/2890193038 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d401a2810 con 0x7f4d40075fc0 2024-04-24T08:09:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.620+0000 7f4d3d600640 1 -- 172.21.15.28:0/2890193038 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d2c003ce0 con 0x7f4d40075fc0 2024-04-24T08:09:30.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.620+0000 7f4d3d600640 1 -- 172.21.15.28:0/2890193038 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d2c049060 con 0x7f4d40075fc0 2024-04-24T08:09:30.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.621+0000 7f4d46400640 1 -- 172.21.15.28:0/2890193038 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4d200052c0 con 0x7f4d40075fc0 2024-04-24T08:09:30.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.623+0000 7f4d3d600640 1 -- 172.21.15.28:0/2890193038 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4d2c042030 con 0x7f4d40075fc0 2024-04-24T08:09:30.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.626+0000 7f4d3d600640 1 --2- 172.21.15.28:0/2890193038 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4d280a3c30 0x7f4d280a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:30.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.627+0000 7f4d45000640 1 --2- 172.21.15.28:0/2890193038 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4d280a3c30 0x7f4d280a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:30.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.627+0000 7f4d3d600640 1 -- 172.21.15.28:0/2890193038 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4d2c0f04b0 con 0x7f4d40075fc0 2024-04-24T08:09:30.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.627+0000 7f4d45000640 1 --2- 172.21.15.28:0/2890193038 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4d280a3c30 0x7f4d280a60f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4d34009bd0 tx=0x7f4d34005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:30.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.633+0000 7f4d3d600640 1 -- 172.21.15.28:0/2890193038 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4d2c0bc7c0 con 0x7f4d40075fc0 2024-04-24T08:09:30.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.843+0000 7f4d46400640 1 -- 172.21.15.28:0/2890193038 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f4d20005dc0 con 0x7f4d40075fc0 2024-04-24T08:09:30.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.844+0000 7f4d3d600640 1 -- 172.21.15.28:0/2890193038 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7f4d2c0c0ac0 con 0x7f4d40075fc0 2024-04-24T08:09:30.845 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-24T08:09:30.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.847+0000 7f4d46400640 1 -- 172.21.15.28:0/2890193038 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4d280a3c30 msgr2=0x7f4d280a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:30.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.847+0000 7f4d46400640 1 --2- 172.21.15.28:0/2890193038 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4d280a3c30 0x7f4d280a60f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f4d34009bd0 tx=0x7f4d34005f90 comp rx=0 tx=0).stop 2024-04-24T08:09:30.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.847+0000 7f4d46400640 1 -- 172.21.15.28:0/2890193038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d40075fc0 msgr2=0x7f4d4019a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:30.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.847+0000 7f4d46400640 1 --2- 172.21.15.28:0/2890193038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d40075fc0 0x7f4d4019a960 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f4d2c031440 tx=0x7f4d2c032ce0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.847+0000 7f4d46400640 1 -- 172.21.15.28:0/2890193038 shutdown_connections 2024-04-24T08:09:30.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.847+0000 7f4d46400640 1 --2- 172.21.15.28:0/2890193038 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4d280a3c30 0x7f4d280a60f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.847+0000 7f4d46400640 1 --2- 172.21.15.28:0/2890193038 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d40075fc0 0x7f4d4019a960 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.847+0000 7f4d46400640 1 --2- 172.21.15.28:0/2890193038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d40109de0 0x7f4d401a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.847+0000 7f4d46400640 1 --2- 172.21.15.28:0/2890193038 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d400749e0 0x7f4d4019a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:30.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.847+0000 7f4d46400640 1 -- 172.21.15.28:0/2890193038 >> 172.21.15.28:0/2890193038 conn(0x7f4d400fd4a0 msgr2=0x7f4d40105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:30.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.847+0000 7f4d46400640 1 -- 172.21.15.28:0/2890193038 shutdown_connections 2024-04-24T08:09:30.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:30.847+0000 7f4d46400640 1 -- 172.21.15.28:0/2890193038 wait complete. 2024-04-24T08:09:30.870 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:09:31.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.065+0000 7f4b10600640 1 -- 172.21.15.28:0/559634620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b08104260 msgr2=0x7f4b08104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.065+0000 7f4b10600640 1 --2- 172.21.15.28:0/559634620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b08104260 0x7f4b08104680 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4afc009900 tx=0x7f4afc02a400 comp rx=0 tx=0).stop 2024-04-24T08:09:31.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.066+0000 7f4b10600640 1 -- 172.21.15.28:0/559634620 shutdown_connections 2024-04-24T08:09:31.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.066+0000 7f4b10600640 1 --2- 172.21.15.28:0/559634620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b08105450 0x7f4b081058d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.066+0000 7f4b10600640 1 --2- 172.21.15.28:0/559634620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b08105e10 0x7f4b0810c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.066+0000 7f4b10600640 1 --2- 172.21.15.28:0/559634620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b08104260 0x7f4b08104680 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.066+0000 7f4b10600640 1 -- 172.21.15.28:0/559634620 >> 172.21.15.28:0/559634620 conn(0x7f4b080ff7f0 msgr2=0x7f4b08101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:31.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.066+0000 7f4b10600640 1 -- 172.21.15.28:0/559634620 shutdown_connections 2024-04-24T08:09:31.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.066+0000 7f4b10600640 1 -- 172.21.15.28:0/559634620 wait complete. 2024-04-24T08:09:31.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.067+0000 7f4b10600640 1 Processor -- start 2024-04-24T08:09:31.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.067+0000 7f4b10600640 1 -- start start 2024-04-24T08:09:31.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.068+0000 7f4b10600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b08104260 0x7f4b0819a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.068+0000 7f4b10600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b08105450 0x7f4b0819aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.068+0000 7f4b10600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b08105e10 0x7f4b081a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.068+0000 7f4b10600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0810fef0 con 0x7f4b08105450 2024-04-24T08:09:31.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.068+0000 7f4b10600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4b0810fd70 con 0x7f4b08105e10 2024-04-24T08:09:31.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.068+0000 7f4b10600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4b08110070 con 0x7f4b08104260 2024-04-24T08:09:31.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.068+0000 7f4b0dc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b08104260 0x7f4b0819a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.068+0000 7f4b0d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b08105450 0x7f4b0819aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.068+0000 7f4b0dc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b08104260 0x7f4b0819a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:34106/0 (socket says 172.21.15.28:34106) 2024-04-24T08:09:31.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.068+0000 7f4b0dc00640 1 -- 172.21.15.28:0/1146167432 learned_addr learned my addr 172.21.15.28:0/1146167432 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:31.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.068+0000 7f4b0e600640 1 --2- 172.21.15.28:0/1146167432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b08105e10 0x7f4b081a1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.068+0000 7f4b0dc00640 1 -- 172.21.15.28:0/1146167432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b08105e10 msgr2=0x7f4b081a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.068+0000 7f4b0dc00640 1 --2- 172.21.15.28:0/1146167432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b08105e10 0x7f4b081a1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.068+0000 7f4b0dc00640 1 -- 172.21.15.28:0/1146167432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b08105450 msgr2=0x7f4b0819aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.068+0000 7f4b0dc00640 1 --2- 172.21.15.28:0/1146167432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b08105450 0x7f4b0819aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.068+0000 7f4b0dc00640 1 -- 172.21.15.28:0/1146167432 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b081a2190 con 0x7f4b08104260 2024-04-24T08:09:31.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.068+0000 7f4b0d200640 1 --2- 172.21.15.28:0/1146167432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b08105450 0x7f4b0819aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:31.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.069+0000 7f4b0dc00640 1 --2- 172.21.15.28:0/1146167432 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b08104260 0x7f4b0819a570 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4afc031610 tx=0x7f4afc004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:31.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.069+0000 7f4b06000640 1 -- 172.21.15.28:0/1146167432 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc03f030 con 0x7f4b08104260 2024-04-24T08:09:31.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.069+0000 7f4b06000640 1 -- 172.21.15.28:0/1146167432 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4afc003ad0 con 0x7f4b08104260 2024-04-24T08:09:31.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.069+0000 7f4b06000640 1 -- 172.21.15.28:0/1146167432 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4afc035070 con 0x7f4b08104260 2024-04-24T08:09:31.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.069+0000 7f4b10600640 1 -- 172.21.15.28:0/1146167432 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b081a2420 con 0x7f4b08104260 2024-04-24T08:09:31.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.069+0000 7f4b10600640 1 -- 172.21.15.28:0/1146167432 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b081a2960 con 0x7f4b08104260 2024-04-24T08:09:31.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.071+0000 7f4b10600640 1 -- 172.21.15.28:0/1146167432 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4b0810b5d0 con 0x7f4b08104260 2024-04-24T08:09:31.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.073+0000 7f4b06000640 1 -- 172.21.15.28:0/1146167432 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f4afc042030 con 0x7f4b08104260 2024-04-24T08:09:31.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.074+0000 7f4b06000640 1 --2- 172.21.15.28:0/1146167432 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4af00a3c30 0x7f4af00a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.074+0000 7f4b06000640 1 -- 172.21.15.28:0/1146167432 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f4afc0f0a40 con 0x7f4b08104260 2024-04-24T08:09:31.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.074+0000 7f4b0d200640 1 --2- 172.21.15.28:0/1146167432 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4af00a3c30 0x7f4af00a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.075+0000 7f4b0d200640 1 --2- 172.21.15.28:0/1146167432 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4af00a3c30 0x7f4af00a60f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4b0819ba80 tx=0x7f4af4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:31.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.077+0000 7f4b06000640 1 -- 172.21.15.28:0/1146167432 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f4afc0bcd50 con 0x7f4b08104260 2024-04-24T08:09:31.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.282+0000 7f4b10600640 1 -- 172.21.15.28:0/1146167432 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f4b08062c00 con 0x7f4b08104260 2024-04-24T08:09:31.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.282+0000 7f4b06000640 1 -- 172.21.15.28:0/1146167432 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7f4afc0c1050 con 0x7f4b08104260 2024-04-24T08:09:31.284 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-24T08:09:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.285+0000 7f4b10600640 1 -- 172.21.15.28:0/1146167432 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4af00a3c30 msgr2=0x7f4af00a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.285+0000 7f4b10600640 1 --2- 172.21.15.28:0/1146167432 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4af00a3c30 0x7f4af00a60f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f4b0819ba80 tx=0x7f4af4005f90 comp rx=0 tx=0).stop 2024-04-24T08:09:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.285+0000 7f4b10600640 1 -- 172.21.15.28:0/1146167432 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b08104260 msgr2=0x7f4b0819a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.285+0000 7f4b10600640 1 --2- 172.21.15.28:0/1146167432 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b08104260 0x7f4b0819a570 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4afc031610 tx=0x7f4afc004270 comp rx=0 tx=0).stop 2024-04-24T08:09:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.285+0000 7f4b10600640 1 -- 172.21.15.28:0/1146167432 shutdown_connections 2024-04-24T08:09:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.285+0000 7f4b10600640 1 --2- 172.21.15.28:0/1146167432 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f4af00a3c30 0x7f4af00a60f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.285+0000 7f4b10600640 1 --2- 172.21.15.28:0/1146167432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4b08105450 0x7f4b0819aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.285+0000 7f4b10600640 1 --2- 172.21.15.28:0/1146167432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b08105e10 0x7f4b081a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.285+0000 7f4b10600640 1 --2- 172.21.15.28:0/1146167432 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4b08104260 0x7f4b0819a570 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.285+0000 7f4b10600640 1 -- 172.21.15.28:0/1146167432 >> 172.21.15.28:0/1146167432 conn(0x7f4b080ff7f0 msgr2=0x7f4b081087a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.285+0000 7f4b10600640 1 -- 172.21.15.28:0/1146167432 shutdown_connections 2024-04-24T08:09:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.285+0000 7f4b10600640 1 -- 172.21.15.28:0/1146167432 wait complete. 2024-04-24T08:09:31.305 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:09:31.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.521+0000 7fad85000640 1 -- 172.21.15.28:0/3613596398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad80105330 msgr2=0x7fad801057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.521+0000 7fad85000640 1 --2- 172.21.15.28:0/3613596398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad80105330 0x7fad801057b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fad6c009900 tx=0x7fad6c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.521+0000 7fad85000640 1 -- 172.21.15.28:0/3613596398 shutdown_connections 2024-04-24T08:09:31.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.521+0000 7fad85000640 1 --2- 172.21.15.28:0/3613596398 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad80105cf0 0x7fad8010c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.521+0000 7fad85000640 1 --2- 172.21.15.28:0/3613596398 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad80105330 0x7fad801057b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.521+0000 7fad85000640 1 --2- 172.21.15.28:0/3613596398 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad80104140 0x7fad80104560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.521+0000 7fad85000640 1 -- 172.21.15.28:0/3613596398 >> 172.21.15.28:0/3613596398 conn(0x7fad800ff6b0 msgr2=0x7fad80101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:31.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.521+0000 7fad85000640 1 -- 172.21.15.28:0/3613596398 shutdown_connections 2024-04-24T08:09:31.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.521+0000 7fad85000640 1 -- 172.21.15.28:0/3613596398 wait complete. 2024-04-24T08:09:31.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.523+0000 7fad85000640 1 Processor -- start 2024-04-24T08:09:31.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.523+0000 7fad85000640 1 -- start start 2024-04-24T08:09:31.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.523+0000 7fad85000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad80104140 0x7fad8019a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.523+0000 7fad85000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad80105330 0x7fad8019a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.523+0000 7fad85000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad80105cf0 0x7fad801a19a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.523+0000 7fad85000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fad8010fe20 con 0x7fad80105330 2024-04-24T08:09:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.523+0000 7fad85000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fad8010fca0 con 0x7fad80104140 2024-04-24T08:09:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.523+0000 7fad85000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fad8010ffa0 con 0x7fad80105cf0 2024-04-24T08:09:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.523+0000 7fad7f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad80104140 0x7fad8019a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.523+0000 7fad7f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad80104140 0x7fad8019a3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:55392/0 (socket says 172.21.15.28:55392) 2024-04-24T08:09:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.523+0000 7fad7f400640 1 -- 172.21.15.28:0/2801100938 learned_addr learned my addr 172.21.15.28:0/2801100938 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.524+0000 7fad7ea00640 1 --2- 172.21.15.28:0/2801100938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad80105330 0x7fad8019a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.524+0000 7fad7fe00640 1 --2- 172.21.15.28:0/2801100938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad80105cf0 0x7fad801a19a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.524+0000 7fad7f400640 1 -- 172.21.15.28:0/2801100938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad80105cf0 msgr2=0x7fad801a19a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.524+0000 7fad7f400640 1 --2- 172.21.15.28:0/2801100938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad80105cf0 0x7fad801a19a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.524+0000 7fad7f400640 1 -- 172.21.15.28:0/2801100938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad80105330 msgr2=0x7fad8019a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.524+0000 7fad7f400640 1 --2- 172.21.15.28:0/2801100938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad80105330 0x7fad8019a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.524+0000 7fad7f400640 1 -- 172.21.15.28:0/2801100938 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fad801a2010 con 0x7fad80104140 2024-04-24T08:09:31.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.524+0000 7fad7fe00640 1 --2- 172.21.15.28:0/2801100938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad80105cf0 0x7fad801a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:31.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.524+0000 7fad7f400640 1 --2- 172.21.15.28:0/2801100938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad80104140 0x7fad8019a3f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fad7000e9c0 tx=0x7fad7000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:31.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.524+0000 7fad77e00640 1 -- 172.21.15.28:0/2801100938 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad7000cde0 con 0x7fad80104140 2024-04-24T08:09:31.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.524+0000 7fad77e00640 1 -- 172.21.15.28:0/2801100938 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fad70010910 con 0x7fad80104140 2024-04-24T08:09:31.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.525+0000 7fad77e00640 1 -- 172.21.15.28:0/2801100938 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fad70018a00 con 0x7fad80104140 2024-04-24T08:09:31.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.525+0000 7fad85000640 1 -- 172.21.15.28:0/2801100938 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fad801a2300 con 0x7fad80104140 2024-04-24T08:09:31.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.525+0000 7fad85000640 1 -- 172.21.15.28:0/2801100938 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fad801a27d0 con 0x7fad80104140 2024-04-24T08:09:31.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.526+0000 7fad85000640 1 -- 172.21.15.28:0/2801100938 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fad8010b460 con 0x7fad80104140 2024-04-24T08:09:31.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.528+0000 7fad77e00640 1 -- 172.21.15.28:0/2801100938 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fad7001f020 con 0x7fad80104140 2024-04-24T08:09:31.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.529+0000 7fad77e00640 1 --2- 172.21.15.28:0/2801100938 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fad640a3c30 0x7fad640a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.529+0000 7fad77e00640 1 -- 172.21.15.28:0/2801100938 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fad700d08f0 con 0x7fad80104140 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.530+0000 7fad7ea00640 1 --2- 172.21.15.28:0/2801100938 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fad640a3c30 0x7fad640a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.530+0000 7fad7ea00640 1 --2- 172.21.15.28:0/2801100938 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fad640a3c30 0x7fad640a60f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fad6c031440 tx=0x7fad6c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:31.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.532+0000 7fad77e00640 1 -- 172.21.15.28:0/2801100938 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fad7009d250 con 0x7fad80104140 2024-04-24T08:09:31.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:31 smithi117 ceph-mon[32438]: pgmap v368: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-24T08:09:31.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:31 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2857167427' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:09:31.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:31 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2890193038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:09:31.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:31 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1146167432' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:09:31.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:31 smithi194 ceph-mon[32836]: pgmap v368: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-24T08:09:31.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:31 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2857167427' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:09:31.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:31 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2890193038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:09:31.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:31 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1146167432' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:09:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.738+0000 7fad85000640 1 -- 172.21.15.28:0/2801100938 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fad8002d890 con 0x7fad80104140 2024-04-24T08:09:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.739+0000 7fad77e00640 1 -- 172.21.15.28:0/2801100938 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fad7009cb50 con 0x7fad80104140 2024-04-24T08:09:31.741 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-24T08:09:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.742+0000 7fad85000640 1 -- 172.21.15.28:0/2801100938 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fad640a3c30 msgr2=0x7fad640a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.742+0000 7fad85000640 1 --2- 172.21.15.28:0/2801100938 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fad640a3c30 0x7fad640a60f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fad6c031440 tx=0x7fad6c0315b0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.742+0000 7fad85000640 1 -- 172.21.15.28:0/2801100938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad80104140 msgr2=0x7fad8019a3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.742+0000 7fad85000640 1 --2- 172.21.15.28:0/2801100938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad80104140 0x7fad8019a3f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fad7000e9c0 tx=0x7fad7000ee80 comp rx=0 tx=0).stop 2024-04-24T08:09:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.742+0000 7fad85000640 1 -- 172.21.15.28:0/2801100938 shutdown_connections 2024-04-24T08:09:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.742+0000 7fad85000640 1 --2- 172.21.15.28:0/2801100938 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fad640a3c30 0x7fad640a60f0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.743+0000 7fad85000640 1 --2- 172.21.15.28:0/2801100938 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fad80105330 0x7fad8019a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.743+0000 7fad85000640 1 --2- 172.21.15.28:0/2801100938 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fad80105cf0 0x7fad801a19a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.743+0000 7fad85000640 1 --2- 172.21.15.28:0/2801100938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad80104140 0x7fad8019a3f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.743+0000 7fad85000640 1 -- 172.21.15.28:0/2801100938 >> 172.21.15.28:0/2801100938 conn(0x7fad800ff6b0 msgr2=0x7fad80108630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.743+0000 7fad85000640 1 -- 172.21.15.28:0/2801100938 shutdown_connections 2024-04-24T08:09:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.743+0000 7fad85000640 1 -- 172.21.15.28:0/2801100938 wait complete. 2024-04-24T08:09:31.761 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:31 smithi028 ceph-mon[27786]: pgmap v368: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 597 B/s wr, 7 op/s 2024-04-24T08:09:31.761 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:31 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2857167427' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:09:31.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:31 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2890193038' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:09:31.762 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:31 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1146167432' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:09:31.765 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:09:31.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.983+0000 7f72c7e00640 1 -- 172.21.15.28:0/1298138827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72c8105290 msgr2=0x7f72c8105710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.983+0000 7f72c7e00640 1 --2- 172.21.15.28:0/1298138827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72c8105290 0x7f72c8105710 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009900 tx=0x7f72b002a410 comp rx=0 tx=0).stop 2024-04-24T08:09:31.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.983+0000 7f72c7e00640 1 -- 172.21.15.28:0/1298138827 shutdown_connections 2024-04-24T08:09:31.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.983+0000 7f72c7e00640 1 --2- 172.21.15.28:0/1298138827 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72c8105290 0x7f72c8105710 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.983+0000 7f72c7e00640 1 --2- 172.21.15.28:0/1298138827 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72c8105c50 0x7f72c810c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.983+0000 7f72c7e00640 1 --2- 172.21.15.28:0/1298138827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72c81040a0 0x7f72c81044c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.983+0000 7f72c7e00640 1 -- 172.21.15.28:0/1298138827 >> 172.21.15.28:0/1298138827 conn(0x7f72c80ff610 msgr2=0x7f72c8101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:31.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.983+0000 7f72c7e00640 1 -- 172.21.15.28:0/1298138827 shutdown_connections 2024-04-24T08:09:31.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.983+0000 7f72c7e00640 1 -- 172.21.15.28:0/1298138827 wait complete. 2024-04-24T08:09:31.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.985+0000 7f72c7e00640 1 Processor -- start 2024-04-24T08:09:31.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.985+0000 7f72c7e00640 1 -- start start 2024-04-24T08:09:31.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.985+0000 7f72c7e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72c81040a0 0x7f72c819a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.985+0000 7f72c7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72c8105290 0x7f72c819a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.985+0000 7f72c7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72c8105c50 0x7f72c81a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.985+0000 7f72c7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72c810fd80 con 0x7f72c8105c50 2024-04-24T08:09:31.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.985+0000 7f72c7e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f72c810fc00 con 0x7f72c81040a0 2024-04-24T08:09:31.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.985+0000 7f72c7e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f72c810ff00 con 0x7f72c8105290 2024-04-24T08:09:31.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.985+0000 7f72c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72c8105c50 0x7f72c81a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.985+0000 7f72c7400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72c8105c50 0x7f72c81a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59704/0 (socket says 172.21.15.28:59704) 2024-04-24T08:09:31.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.985+0000 7f72c7400640 1 -- 172.21.15.28:0/3797189933 learned_addr learned my addr 172.21.15.28:0/3797189933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:31.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.986+0000 7f72c6a00640 1 --2- 172.21.15.28:0/3797189933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72c81040a0 0x7f72c819a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.986+0000 7f72c6000640 1 --2- 172.21.15.28:0/3797189933 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72c8105290 0x7f72c819a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.986+0000 7f72c7400640 1 -- 172.21.15.28:0/3797189933 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72c8105290 msgr2=0x7f72c819a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.986+0000 7f72c7400640 1 --2- 172.21.15.28:0/3797189933 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72c8105290 0x7f72c819a8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.986+0000 7f72c7400640 1 -- 172.21.15.28:0/3797189933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72c81040a0 msgr2=0x7f72c819a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:31.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.986+0000 7f72c7400640 1 --2- 172.21.15.28:0/3797189933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72c81040a0 0x7f72c819a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:31.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.986+0000 7f72c7400640 1 -- 172.21.15.28:0/3797189933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c81a1fa0 con 0x7f72c8105c50 2024-04-24T08:09:31.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.986+0000 7f72c7400640 1 --2- 172.21.15.28:0/3797189933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72c8105c50 0x7f72c81a1930 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f72b800eb10 tx=0x7f72b800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:31.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.986+0000 7f72bf400640 1 -- 172.21.15.28:0/3797189933 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8004b20 con 0x7f72c8105c50 2024-04-24T08:09:31.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.986+0000 7f72bf400640 1 -- 172.21.15.28:0/3797189933 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72b8004cc0 con 0x7f72c8105c50 2024-04-24T08:09:31.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.987+0000 7f72c7e00640 1 -- 172.21.15.28:0/3797189933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72c81a2290 con 0x7f72c8105c50 2024-04-24T08:09:31.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.987+0000 7f72bf400640 1 -- 172.21.15.28:0/3797189933 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8010760 con 0x7f72c8105c50 2024-04-24T08:09:31.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.987+0000 7f72c7e00640 1 -- 172.21.15.28:0/3797189933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72c81a2800 con 0x7f72c8105c50 2024-04-24T08:09:31.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.988+0000 7f72c7e00640 1 -- 172.21.15.28:0/3797189933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f72c810b3c0 con 0x7f72c8105c50 2024-04-24T08:09:31.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.990+0000 7f72bf400640 1 -- 172.21.15.28:0/3797189933 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f72b8005020 con 0x7f72c8105c50 2024-04-24T08:09:31.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.991+0000 7f72bf400640 1 --2- 172.21.15.28:0/3797189933 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f72ac0a3c30 0x7f72ac0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:31.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.991+0000 7f72bf400640 1 -- 172.21.15.28:0/3797189933 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f72b80d0a90 con 0x7f72c8105c50 2024-04-24T08:09:31.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.991+0000 7f72c6a00640 1 --2- 172.21.15.28:0/3797189933 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f72ac0a3c30 0x7f72ac0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:31.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.992+0000 7f72c6a00640 1 --2- 172.21.15.28:0/3797189933 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f72ac0a3c30 0x7f72ac0a60f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f72c81050f0 tx=0x7f72c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:31.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:31.995+0000 7f72bf400640 1 -- 172.21.15.28:0/3797189933 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f72b809d3f0 con 0x7f72c8105c50 2024-04-24T08:09:32.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.246+0000 7f72c7e00640 1 -- 172.21.15.28:0/3797189933 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f72c802d620 con 0x7f72c8105c50 2024-04-24T08:09:32.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.247+0000 7f72bf400640 1 -- 172.21.15.28:0/3797189933 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11871 (secure 0 0 0) 0x7f72b809ccf0 con 0x7f72c8105c50 2024-04-24T08:09:32.249 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:09:32.249 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:07:57.133790+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:09:32.249 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 23 2024-04-24T08:09:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.249+0000 7f72c7e00640 1 -- 172.21.15.28:0/3797189933 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f72ac0a3c30 msgr2=0x7f72ac0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.249+0000 7f72c7e00640 1 --2- 172.21.15.28:0/3797189933 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f72ac0a3c30 0x7f72ac0a60f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f72c81050f0 tx=0x7f72c0005f90 comp rx=0 tx=0).stop 2024-04-24T08:09:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.249+0000 7f72c7e00640 1 -- 172.21.15.28:0/3797189933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72c8105c50 msgr2=0x7f72c81a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.249+0000 7f72c7e00640 1 --2- 172.21.15.28:0/3797189933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72c8105c50 0x7f72c81a1930 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f72b800eb10 tx=0x7f72b800c2d0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.249+0000 7f72c7e00640 1 -- 172.21.15.28:0/3797189933 shutdown_connections 2024-04-24T08:09:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.249+0000 7f72c7e00640 1 --2- 172.21.15.28:0/3797189933 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f72ac0a3c30 0x7f72ac0a60f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.249+0000 7f72c7e00640 1 --2- 172.21.15.28:0/3797189933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72c8105c50 0x7f72c81a1930 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.249+0000 7f72c7e00640 1 --2- 172.21.15.28:0/3797189933 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72c8105290 0x7f72c819a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.249+0000 7f72c7e00640 1 --2- 172.21.15.28:0/3797189933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72c81040a0 0x7f72c819a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.250+0000 7f72c7e00640 1 -- 172.21.15.28:0/3797189933 >> 172.21.15.28:0/3797189933 conn(0x7f72c80ff610 msgr2=0x7f72c8108590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.250+0000 7f72c7e00640 1 -- 172.21.15.28:0/3797189933 shutdown_connections 2024-04-24T08:09:32.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.250+0000 7f72c7e00640 1 -- 172.21.15.28:0/3797189933 wait complete. 2024-04-24T08:09:32.271 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:32.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.480+0000 7f6e57200640 1 -- 172.21.15.28:0/2302350829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e50105c30 msgr2=0x7f6e5010c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.480+0000 7f6e57200640 1 --2- 172.21.15.28:0/2302350829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e50105c30 0x7f6e5010c4c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009ca0 tx=0x7f6e4c02a8b0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.480+0000 7f6e57200640 1 -- 172.21.15.28:0/2302350829 shutdown_connections 2024-04-24T08:09:32.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.480+0000 7f6e57200640 1 --2- 172.21.15.28:0/2302350829 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e50105c30 0x7f6e5010c4c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.480+0000 7f6e57200640 1 --2- 172.21.15.28:0/2302350829 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e50105270 0x7f6e501056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.480+0000 7f6e57200640 1 --2- 172.21.15.28:0/2302350829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e50104080 0x7f6e501044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.480+0000 7f6e57200640 1 -- 172.21.15.28:0/2302350829 >> 172.21.15.28:0/2302350829 conn(0x7f6e500ff610 msgr2=0x7f6e50101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:32.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.480+0000 7f6e57200640 1 -- 172.21.15.28:0/2302350829 shutdown_connections 2024-04-24T08:09:32.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.481+0000 7f6e57200640 1 -- 172.21.15.28:0/2302350829 wait complete. 2024-04-24T08:09:32.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.482+0000 7f6e57200640 1 Processor -- start 2024-04-24T08:09:32.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.482+0000 7f6e57200640 1 -- start start 2024-04-24T08:09:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.483+0000 7f6e57200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e50104080 0x7f6e5019a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.483+0000 7f6e57200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e50105270 0x7f6e5019a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.483+0000 7f6e57200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e50105c30 0x7f6e501a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.483+0000 7f6e57200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5010fd60 con 0x7f6e50105270 2024-04-24T08:09:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.483+0000 7f6e57200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5010fbe0 con 0x7f6e50104080 2024-04-24T08:09:32.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.483+0000 7f6e57200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6e5010fee0 con 0x7f6e50105c30 2024-04-24T08:09:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.483+0000 7f6e55400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e50105270 0x7f6e5019a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.483+0000 7f6e55400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e50105270 0x7f6e5019a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59708/0 (socket says 172.21.15.28:59708) 2024-04-24T08:09:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.483+0000 7f6e55400640 1 -- 172.21.15.28:0/3047096593 learned_addr learned my addr 172.21.15.28:0/3047096593 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.483+0000 7f6e56800640 1 --2- 172.21.15.28:0/3047096593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e50105c30 0x7f6e501a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:32.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.483+0000 7f6e55e00640 1 --2- 172.21.15.28:0/3047096593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e50104080 0x7f6e5019a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.483+0000 7f6e55400640 1 -- 172.21.15.28:0/3047096593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e50105c30 msgr2=0x7f6e501a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.483+0000 7f6e55400640 1 --2- 172.21.15.28:0/3047096593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e50105c30 0x7f6e501a18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.483+0000 7f6e55400640 1 -- 172.21.15.28:0/3047096593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e50104080 msgr2=0x7f6e5019a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.483+0000 7f6e55400640 1 --2- 172.21.15.28:0/3047096593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e50104080 0x7f6e5019a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.483+0000 7f6e55400640 1 -- 172.21.15.28:0/3047096593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e501a1f10 con 0x7f6e50105270 2024-04-24T08:09:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.484+0000 7f6e55400640 1 --2- 172.21.15.28:0/3047096593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e50105270 0x7f6e5019a830 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000e9f0 tx=0x7f6e4000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.484+0000 7f6e4aa00640 1 -- 172.21.15.28:0/3047096593 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e4000cd80 con 0x7f6e50105270 2024-04-24T08:09:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.484+0000 7f6e4aa00640 1 -- 172.21.15.28:0/3047096593 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e40005500 con 0x7f6e50105270 2024-04-24T08:09:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.484+0000 7f6e57200640 1 -- 172.21.15.28:0/3047096593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e501a2200 con 0x7f6e50105270 2024-04-24T08:09:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.484+0000 7f6e4aa00640 1 -- 172.21.15.28:0/3047096593 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e40010690 con 0x7f6e50105270 2024-04-24T08:09:32.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.484+0000 7f6e57200640 1 -- 172.21.15.28:0/3047096593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e501a2700 con 0x7f6e50105270 2024-04-24T08:09:32.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.485+0000 7f6e57200640 1 -- 172.21.15.28:0/3047096593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e5010b340 con 0x7f6e50105270 2024-04-24T08:09:32.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.487+0000 7f6e4aa00640 1 -- 172.21.15.28:0/3047096593 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6e400046c0 con 0x7f6e50105270 2024-04-24T08:09:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.489+0000 7f6e4aa00640 1 --2- 172.21.15.28:0/3047096593 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6e380a3bb0 0x7f6e380a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.489+0000 7f6e4aa00640 1 -- 172.21.15.28:0/3047096593 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f6e400d0800 con 0x7f6e50105270 2024-04-24T08:09:32.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.489+0000 7f6e55e00640 1 --2- 172.21.15.28:0/3047096593 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6e380a3bb0 0x7f6e380a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:32.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.490+0000 7f6e55e00640 1 --2- 172.21.15.28:0/3047096593 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6e380a3bb0 0x7f6e380a6070 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6e44005fd0 tx=0x7f6e4400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:32.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.495+0000 7f6e4aa00640 1 -- 172.21.15.28:0/3047096593 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6e4009d160 con 0x7f6e50105270 2024-04-24T08:09:32.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:32 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2801100938' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:09:32.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:32 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3797189933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:09:32.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:32 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2801100938' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:09:32.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:32 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3797189933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:09:32.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:32 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2801100938' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:09:32.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:32 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3797189933' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:09:32.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.705+0000 7f6e57200640 1 -- 172.21.15.28:0/3047096593 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6e50062c00 con 0x7f6e50105270 2024-04-24T08:09:32.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.708+0000 7f6e4aa00640 1 -- 172.21.15.28:0/3047096593 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22677 (secure 0 0 0) 0x7f6e4009ca60 con 0x7f6e50105270 2024-04-24T08:09:32.709 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:09:32.710 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:08:51.155712+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:32.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.710+0000 7f6e57200640 1 -- 172.21.15.28:0/3047096593 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6e380a3bb0 msgr2=0x7f6e380a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.711+0000 7f6e57200640 1 --2- 172.21.15.28:0/3047096593 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6e380a3bb0 0x7f6e380a6070 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f6e44005fd0 tx=0x7f6e4400b410 comp rx=0 tx=0).stop 2024-04-24T08:09:32.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.711+0000 7f6e57200640 1 -- 172.21.15.28:0/3047096593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e50105270 msgr2=0x7f6e5019a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.711+0000 7f6e57200640 1 --2- 172.21.15.28:0/3047096593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e50105270 0x7f6e5019a830 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f6e4000e9f0 tx=0x7f6e4000edf0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.711+0000 7f6e57200640 1 -- 172.21.15.28:0/3047096593 shutdown_connections 2024-04-24T08:09:32.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.711+0000 7f6e57200640 1 --2- 172.21.15.28:0/3047096593 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6e380a3bb0 0x7f6e380a6070 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.711+0000 7f6e57200640 1 --2- 172.21.15.28:0/3047096593 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e50105270 0x7f6e5019a830 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.711+0000 7f6e57200640 1 --2- 172.21.15.28:0/3047096593 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e50105c30 0x7f6e501a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.711+0000 7f6e57200640 1 --2- 172.21.15.28:0/3047096593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e50104080 0x7f6e5019a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.711+0000 7f6e57200640 1 -- 172.21.15.28:0/3047096593 >> 172.21.15.28:0/3047096593 conn(0x7f6e500ff610 msgr2=0x7f6e50108510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:32.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.711+0000 7f6e57200640 1 -- 172.21.15.28:0/3047096593 shutdown_connections 2024-04-24T08:09:32.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.711+0000 7f6e57200640 1 -- 172.21.15.28:0/3047096593 wait complete. 2024-04-24T08:09:32.732 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:02:34.086371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:07:49.819536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:07:50.493486+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:09:32.732 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:09:32.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.948+0000 7fbcaee00640 1 -- 172.21.15.28:0/2277597155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca8108660 msgr2=0x7fbca8108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.948+0000 7fbcaee00640 1 --2- 172.21.15.28:0/2277597155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca8108660 0x7fbca8108a80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fbc98009ca0 tx=0x7fbc9802a8c0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.948+0000 7fbcaee00640 1 -- 172.21.15.28:0/2277597155 shutdown_connections 2024-04-24T08:09:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.948+0000 7fbcaee00640 1 --2- 172.21.15.28:0/2277597155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbca810a210 0x7fbca8110aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.948+0000 7fbcaee00640 1 --2- 172.21.15.28:0/2277597155 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbca8109850 0x7fbca8109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.948+0000 7fbcaee00640 1 --2- 172.21.15.28:0/2277597155 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca8108660 0x7fbca8108a80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.948+0000 7fbcaee00640 1 -- 172.21.15.28:0/2277597155 >> 172.21.15.28:0/2277597155 conn(0x7fbca8077600 msgr2=0x7fbca8079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.948+0000 7fbcaee00640 1 -- 172.21.15.28:0/2277597155 shutdown_connections 2024-04-24T08:09:32.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.948+0000 7fbcaee00640 1 -- 172.21.15.28:0/2277597155 wait complete. 2024-04-24T08:09:32.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.949+0000 7fbcaee00640 1 Processor -- start 2024-04-24T08:09:32.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.949+0000 7fbcaee00640 1 -- start start 2024-04-24T08:09:32.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.949+0000 7fbcaee00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbca8108660 0x7fbca819ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.949+0000 7fbcaee00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbca8109850 0x7fbca819efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.950+0000 7fbcaee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca810a210 0x7fbca81a5f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.950+0000 7fbcaee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbca81142e0 con 0x7fbca810a210 2024-04-24T08:09:32.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.950+0000 7fbcaee00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbca8114160 con 0x7fbca8109850 2024-04-24T08:09:32.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.950+0000 7fbcaee00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbca8114460 con 0x7fbca8108660 2024-04-24T08:09:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.950+0000 7fbcace00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca810a210 0x7fbca81a5f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.950+0000 7fbca7400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbca8109850 0x7fbca819efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.950+0000 7fbca7400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbca8109850 0x7fbca819efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:55436/0 (socket says 172.21.15.28:55436) 2024-04-24T08:09:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.950+0000 7fbca7400640 1 -- 172.21.15.28:0/1164428408 learned_addr learned my addr 172.21.15.28:0/1164428408 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.950+0000 7fbca7e00640 1 --2- 172.21.15.28:0/1164428408 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbca8108660 0x7fbca819ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.950+0000 7fbca7400640 1 -- 172.21.15.28:0/1164428408 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbca8108660 msgr2=0x7fbca819ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.950+0000 7fbca7400640 1 --2- 172.21.15.28:0/1164428408 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbca8108660 0x7fbca819ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.950+0000 7fbca7400640 1 -- 172.21.15.28:0/1164428408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca810a210 msgr2=0x7fbca81a5f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.950+0000 7fbca7400640 1 --2- 172.21.15.28:0/1164428408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca810a210 0x7fbca81a5f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.950+0000 7fbca7400640 1 -- 172.21.15.28:0/1164428408 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbca81a65f0 con 0x7fbca8109850 2024-04-24T08:09:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.950+0000 7fbcace00640 1 --2- 172.21.15.28:0/1164428408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca810a210 0x7fbca81a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.951+0000 7fbca7400640 1 --2- 172.21.15.28:0/1164428408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbca8109850 0x7fbca819efb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbc9000e9f0 tx=0x7fbc9000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.951+0000 7fbca4c00640 1 -- 172.21.15.28:0/1164428408 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc9000cd80 con 0x7fbca8109850 2024-04-24T08:09:32.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.951+0000 7fbcaee00640 1 -- 172.21.15.28:0/1164428408 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbca81a68e0 con 0x7fbca8109850 2024-04-24T08:09:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.951+0000 7fbca4c00640 1 -- 172.21.15.28:0/1164428408 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc90005500 con 0x7fbca8109850 2024-04-24T08:09:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.951+0000 7fbca4c00640 1 -- 172.21.15.28:0/1164428408 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc90010690 con 0x7fbca8109850 2024-04-24T08:09:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.951+0000 7fbcaee00640 1 -- 172.21.15.28:0/1164428408 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbca81a6e30 con 0x7fbca8109850 2024-04-24T08:09:32.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.952+0000 7fbcaee00640 1 -- 172.21.15.28:0/1164428408 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbca810f9a0 con 0x7fbca8109850 2024-04-24T08:09:32.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.954+0000 7fbca4c00640 1 -- 172.21.15.28:0/1164428408 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbc900046c0 con 0x7fbca8109850 2024-04-24T08:09:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.956+0000 7fbca4c00640 1 --2- 172.21.15.28:0/1164428408 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbc8c0a3bb0 0x7fbc8c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.956+0000 7fbca4c00640 1 -- 172.21.15.28:0/1164428408 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fbc900d07e0 con 0x7fbca8109850 2024-04-24T08:09:32.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.956+0000 7fbca7e00640 1 --2- 172.21.15.28:0/1164428408 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbc8c0a3bb0 0x7fbc8c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:32.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.957+0000 7fbca7e00640 1 --2- 172.21.15.28:0/1164428408 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbc8c0a3bb0 0x7fbc8c0a6070 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbc98006010 tx=0x7fbc98000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:32.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:32.958+0000 7fbca4c00640 1 -- 172.21.15.28:0/1164428408 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbc9009d140 con 0x7fbca8109850 2024-04-24T08:09:33.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.167+0000 7fbcaee00640 1 -- 172.21.15.28:0/1164428408 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbca802d870 con 0x7fbca8109850 2024-04-24T08:09:33.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.168+0000 7fbca4c00640 1 -- 172.21.15.28:0/1164428408 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) ==== 93+0+10 (secure 0 0 0) 0x7fbc9009ca40 con 0x7fbca8109850 2024-04-24T08:09:33.170 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-24T08:09:33.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.171+0000 7fbcaee00640 1 -- 172.21.15.28:0/1164428408 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbc8c0a3bb0 msgr2=0x7fbc8c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.171+0000 7fbcaee00640 1 --2- 172.21.15.28:0/1164428408 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbc8c0a3bb0 0x7fbc8c0a6070 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbc98006010 tx=0x7fbc98000f40 comp rx=0 tx=0).stop 2024-04-24T08:09:33.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.171+0000 7fbcaee00640 1 -- 172.21.15.28:0/1164428408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbca8109850 msgr2=0x7fbca819efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.171+0000 7fbcaee00640 1 --2- 172.21.15.28:0/1164428408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbca8109850 0x7fbca819efb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbc9000e9f0 tx=0x7fbc9000edf0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.171+0000 7fbcaee00640 1 -- 172.21.15.28:0/1164428408 shutdown_connections 2024-04-24T08:09:33.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.171+0000 7fbcaee00640 1 --2- 172.21.15.28:0/1164428408 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbc8c0a3bb0 0x7fbc8c0a6070 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.171+0000 7fbcaee00640 1 --2- 172.21.15.28:0/1164428408 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbca810a210 0x7fbca81a5f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.171+0000 7fbcaee00640 1 --2- 172.21.15.28:0/1164428408 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbca8109850 0x7fbca819efb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.171+0000 7fbcaee00640 1 --2- 172.21.15.28:0/1164428408 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbca8108660 0x7fbca819ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.171+0000 7fbcaee00640 1 -- 172.21.15.28:0/1164428408 >> 172.21.15.28:0/1164428408 conn(0x7fbca8077600 msgr2=0x7fbca810cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:33.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.171+0000 7fbcaee00640 1 -- 172.21.15.28:0/1164428408 shutdown_connections 2024-04-24T08:09:33.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.171+0000 7fbcaee00640 1 -- 172.21.15.28:0/1164428408 wait complete. 2024-04-24T08:09:33.191 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:09:33.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.408+0000 7ff8e3e00640 1 -- 172.21.15.28:0/1250810237 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8e4105330 msgr2=0x7ff8e41057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.408+0000 7ff8e3e00640 1 --2- 172.21.15.28:0/1250810237 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8e4105330 0x7ff8e41057b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0009900 tx=0x7ff8d002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.408+0000 7ff8e3e00640 1 -- 172.21.15.28:0/1250810237 shutdown_connections 2024-04-24T08:09:33.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.408+0000 7ff8e3e00640 1 --2- 172.21.15.28:0/1250810237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8e4105cf0 0x7ff8e410c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.408+0000 7ff8e3e00640 1 --2- 172.21.15.28:0/1250810237 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8e4105330 0x7ff8e41057b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.408+0000 7ff8e3e00640 1 --2- 172.21.15.28:0/1250810237 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8e4104140 0x7ff8e4104560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.408+0000 7ff8e3e00640 1 -- 172.21.15.28:0/1250810237 >> 172.21.15.28:0/1250810237 conn(0x7ff8e40ff6b0 msgr2=0x7ff8e4101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:33.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.408+0000 7ff8e3e00640 1 -- 172.21.15.28:0/1250810237 shutdown_connections 2024-04-24T08:09:33.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.408+0000 7ff8e3e00640 1 -- 172.21.15.28:0/1250810237 wait complete. 2024-04-24T08:09:33.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.409+0000 7ff8e3e00640 1 Processor -- start 2024-04-24T08:09:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.410+0000 7ff8e3e00640 1 -- start start 2024-04-24T08:09:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.410+0000 7ff8e3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8e4104140 0x7ff8e419a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.410+0000 7ff8e3e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8e4105330 0x7ff8e419a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.410+0000 7ff8e3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8e4105cf0 0x7ff8e41a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.410+0000 7ff8e3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e410fe20 con 0x7ff8e4105cf0 2024-04-24T08:09:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.410+0000 7ff8e3e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e410fca0 con 0x7ff8e4105330 2024-04-24T08:09:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.410+0000 7ff8e3e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff8e410ffa0 con 0x7ff8e4104140 2024-04-24T08:09:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.410+0000 7ff8e3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8e4105cf0 0x7ff8e41a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.410+0000 7ff8e2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8e4104140 0x7ff8e419a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.410+0000 7ff8e2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8e4104140 0x7ff8e419a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:34192/0 (socket says 172.21.15.28:34192) 2024-04-24T08:09:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.410+0000 7ff8e3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8e4105cf0 0x7ff8e41a19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59738/0 (socket says 172.21.15.28:59738) 2024-04-24T08:09:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.410+0000 7ff8e2a00640 1 -- 172.21.15.28:0/666496647 learned_addr learned my addr 172.21.15.28:0/666496647 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:33.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.410+0000 7ff8e2000640 1 --2- 172.21.15.28:0/666496647 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8e4105330 0x7ff8e419a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.411+0000 7ff8e2a00640 1 -- 172.21.15.28:0/666496647 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8e4105330 msgr2=0x7ff8e419a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.411+0000 7ff8e2a00640 1 --2- 172.21.15.28:0/666496647 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8e4105330 0x7ff8e419a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.411+0000 7ff8e2a00640 1 -- 172.21.15.28:0/666496647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8e4105cf0 msgr2=0x7ff8e41a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.411+0000 7ff8e2a00640 1 --2- 172.21.15.28:0/666496647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8e4105cf0 0x7ff8e41a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.411+0000 7ff8e2a00640 1 -- 172.21.15.28:0/666496647 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8e41a2030 con 0x7ff8e4104140 2024-04-24T08:09:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.411+0000 7ff8e3400640 1 --2- 172.21.15.28:0/666496647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8e4105cf0 0x7ff8e41a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.411+0000 7ff8e2a00640 1 --2- 172.21.15.28:0/666496647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8e4104140 0x7ff8e419a410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff8d400e9c0 tx=0x7ff8d400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.411+0000 7ff8db400640 1 -- 172.21.15.28:0/666496647 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d400cd80 con 0x7ff8e4104140 2024-04-24T08:09:33.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.411+0000 7ff8db400640 1 -- 172.21.15.28:0/666496647 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8d40035f0 con 0x7ff8e4104140 2024-04-24T08:09:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.411+0000 7ff8e3e00640 1 -- 172.21.15.28:0/666496647 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8e41a2320 con 0x7ff8e4104140 2024-04-24T08:09:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.411+0000 7ff8db400640 1 -- 172.21.15.28:0/666496647 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8d4010690 con 0x7ff8e4104140 2024-04-24T08:09:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.412+0000 7ff8e3e00640 1 -- 172.21.15.28:0/666496647 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8e41a2890 con 0x7ff8e4104140 2024-04-24T08:09:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.412+0000 7ff8e3e00640 1 -- 172.21.15.28:0/666496647 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff8e410b400 con 0x7ff8e4104140 2024-04-24T08:09:33.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.415+0000 7ff8db400640 1 -- 172.21.15.28:0/666496647 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7ff8d40037a0 con 0x7ff8e4104140 2024-04-24T08:09:33.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.416+0000 7ff8db400640 1 --2- 172.21.15.28:0/666496647 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ff8c80a3c30 0x7ff8c80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.416+0000 7ff8db400640 1 -- 172.21.15.28:0/666496647 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7ff8d40d07e0 con 0x7ff8e4104140 2024-04-24T08:09:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.416+0000 7ff8e2000640 1 --2- 172.21.15.28:0/666496647 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ff8c80a3c30 0x7ff8c80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.417+0000 7ff8e2000640 1 --2- 172.21.15.28:0/666496647 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ff8c80a3c30 0x7ff8c80a60f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0031440 tx=0x7ff8d00315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:33.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.420+0000 7ff8db400640 1 -- 172.21.15.28:0/666496647 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff8d409d140 con 0x7ff8e4104140 2024-04-24T08:09:33.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:33 smithi028 ceph-mon[27786]: pgmap v369: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-24T08:09:33.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:33 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3047096593' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:09:33.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:33 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1164428408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:09:33.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:33 smithi117 ceph-mon[32438]: pgmap v369: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-24T08:09:33.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:33 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3047096593' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:09:33.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:33 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1164428408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:09:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.626+0000 7ff8e3e00640 1 -- 172.21.15.28:0/666496647 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff8e4062c00 con 0x7ff8e4104140 2024-04-24T08:09:33.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.627+0000 7ff8db400640 1 -- 172.21.15.28:0/666496647 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) ==== 104+0+11 (secure 0 0 0) 0x7ff8d409ca40 con 0x7ff8e4104140 2024-04-24T08:09:33.629 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-24T08:09:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.630+0000 7ff8e3e00640 1 -- 172.21.15.28:0/666496647 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ff8c80a3c30 msgr2=0x7ff8c80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.630+0000 7ff8e3e00640 1 --2- 172.21.15.28:0/666496647 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ff8c80a3c30 0x7ff8c80a60f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7ff8d0031440 tx=0x7ff8d00315b0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.630+0000 7ff8e3e00640 1 -- 172.21.15.28:0/666496647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8e4104140 msgr2=0x7ff8e419a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.630+0000 7ff8e3e00640 1 --2- 172.21.15.28:0/666496647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8e4104140 0x7ff8e419a410 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7ff8d400e9c0 tx=0x7ff8d400ee80 comp rx=0 tx=0).stop 2024-04-24T08:09:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.630+0000 7ff8e3e00640 1 -- 172.21.15.28:0/666496647 shutdown_connections 2024-04-24T08:09:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.630+0000 7ff8e3e00640 1 --2- 172.21.15.28:0/666496647 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7ff8c80a3c30 0x7ff8c80a60f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.630+0000 7ff8e3e00640 1 --2- 172.21.15.28:0/666496647 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff8e4105cf0 0x7ff8e41a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.630+0000 7ff8e3e00640 1 --2- 172.21.15.28:0/666496647 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8e4105330 0x7ff8e419a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.630+0000 7ff8e3e00640 1 --2- 172.21.15.28:0/666496647 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff8e4104140 0x7ff8e419a410 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.630+0000 7ff8e3e00640 1 -- 172.21.15.28:0/666496647 >> 172.21.15.28:0/666496647 conn(0x7ff8e40ff6b0 msgr2=0x7ff8e41085d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.630+0000 7ff8e3e00640 1 -- 172.21.15.28:0/666496647 shutdown_connections 2024-04-24T08:09:33.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.630+0000 7ff8e3e00640 1 -- 172.21.15.28:0/666496647 wait complete. 2024-04-24T08:09:33.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:09:33.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:33 smithi194 ceph-mon[32836]: pgmap v369: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 597 B/s wr, 5 op/s 2024-04-24T08:09:33.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:33 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3047096593' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:09:33.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:33 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1164428408' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:09:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.864+0000 7fb682000640 1 -- 172.21.15.28:0/187047134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb67c109dd0 msgr2=0x7fb67c10c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.864+0000 7fb682000640 1 --2- 172.21.15.28:0/187047134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb67c109dd0 0x7fb67c10c260 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fb674009be0 tx=0x7fb67402a6c0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.865+0000 7fb682000640 1 -- 172.21.15.28:0/187047134 shutdown_connections 2024-04-24T08:09:33.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.865+0000 7fb682000640 1 --2- 172.21.15.28:0/187047134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb67c109dd0 0x7fb67c10c260 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.865+0000 7fb682000640 1 --2- 172.21.15.28:0/187047134 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb67c074a60 0x7fb67c074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.865+0000 7fb682000640 1 --2- 172.21.15.28:0/187047134 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb67c076040 0x7fb67c074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.865+0000 7fb682000640 1 -- 172.21.15.28:0/187047134 >> 172.21.15.28:0/187047134 conn(0x7fb67c0fd4a0 msgr2=0x7fb67c0ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:33.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.865+0000 7fb682000640 1 -- 172.21.15.28:0/187047134 shutdown_connections 2024-04-24T08:09:33.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.865+0000 7fb682000640 1 -- 172.21.15.28:0/187047134 wait complete. 2024-04-24T08:09:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.866+0000 7fb682000640 1 Processor -- start 2024-04-24T08:09:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.866+0000 7fb682000640 1 -- start start 2024-04-24T08:09:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.866+0000 7fb682000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb67c074a60 0x7fb67c19a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.866+0000 7fb682000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb67c076040 0x7fb67c19a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.866+0000 7fb682000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb67c109dd0 0x7fb67c1a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.866+0000 7fb682000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c10fd90 con 0x7fb67c109dd0 2024-04-24T08:09:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.866+0000 7fb682000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c10fc10 con 0x7fb67c074a60 2024-04-24T08:09:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.866+0000 7fb682000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb67c10ff10 con 0x7fb67c076040 2024-04-24T08:09:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.867+0000 7fb680c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb67c074a60 0x7fb67c19a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.867+0000 7fb681600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb67c109dd0 0x7fb67c1a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.867+0000 7fb680c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb67c074a60 0x7fb67c19a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:55466/0 (socket says 172.21.15.28:55466) 2024-04-24T08:09:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.867+0000 7fb681600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb67c109dd0 0x7fb67c1a1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59756/0 (socket says 172.21.15.28:59756) 2024-04-24T08:09:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.867+0000 7fb680c00640 1 -- 172.21.15.28:0/2997187819 learned_addr learned my addr 172.21.15.28:0/2997187819 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.867+0000 7fb67be00640 1 --2- 172.21.15.28:0/2997187819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb67c076040 0x7fb67c19a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.867+0000 7fb680c00640 1 -- 172.21.15.28:0/2997187819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb67c076040 msgr2=0x7fb67c19a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.867+0000 7fb680c00640 1 --2- 172.21.15.28:0/2997187819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb67c076040 0x7fb67c19a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.867+0000 7fb680c00640 1 -- 172.21.15.28:0/2997187819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb67c109dd0 msgr2=0x7fb67c1a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.867+0000 7fb680c00640 1 --2- 172.21.15.28:0/2997187819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb67c109dd0 0x7fb67c1a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.867+0000 7fb680c00640 1 -- 172.21.15.28:0/2997187819 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb67c1a1ff0 con 0x7fb67c074a60 2024-04-24T08:09:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.867+0000 7fb681600640 1 --2- 172.21.15.28:0/2997187819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb67c109dd0 0x7fb67c1a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.867+0000 7fb680c00640 1 --2- 172.21.15.28:0/2997187819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb67c074a60 0x7fb67c19a3d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb67000e9c0 tx=0x7fb67000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.868+0000 7fb679600640 1 -- 172.21.15.28:0/2997187819 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb67000cde0 con 0x7fb67c074a60 2024-04-24T08:09:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.868+0000 7fb679600640 1 -- 172.21.15.28:0/2997187819 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6700035f0 con 0x7fb67c074a60 2024-04-24T08:09:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.868+0000 7fb679600640 1 -- 172.21.15.28:0/2997187819 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb670010690 con 0x7fb67c074a60 2024-04-24T08:09:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.868+0000 7fb682000640 1 -- 172.21.15.28:0/2997187819 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb67c1a22e0 con 0x7fb67c074a60 2024-04-24T08:09:33.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.868+0000 7fb682000640 1 -- 172.21.15.28:0/2997187819 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb67c1a27e0 con 0x7fb67c074a60 2024-04-24T08:09:33.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.869+0000 7fb682000640 1 -- 172.21.15.28:0/2997187819 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb67c108b00 con 0x7fb67c074a60 2024-04-24T08:09:33.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.871+0000 7fb679600640 1 -- 172.21.15.28:0/2997187819 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb6700037a0 con 0x7fb67c074a60 2024-04-24T08:09:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.872+0000 7fb679600640 1 --2- 172.21.15.28:0/2997187819 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb6640a3bb0 0x7fb6640a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:33.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.872+0000 7fb679600640 1 -- 172.21.15.28:0/2997187819 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb6700d0a50 con 0x7fb67c074a60 2024-04-24T08:09:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.872+0000 7fb67be00640 1 --2- 172.21.15.28:0/2997187819 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb6640a3bb0 0x7fb6640a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:33.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.873+0000 7fb67be00640 1 --2- 172.21.15.28:0/2997187819 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb6640a3bb0 0x7fb6640a6070 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb67c19b8e0 tx=0x7fb668005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:33.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:33.875+0000 7fb679600640 1 -- 172.21.15.28:0/2997187819 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb67009d3b0 con 0x7fb67c074a60 2024-04-24T08:09:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.077+0000 7fb682000640 1 -- 172.21.15.28:0/2997187819 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fb67c02d8b0 con 0x7fb67c074a60 2024-04-24T08:09:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.078+0000 7fb679600640 1 -- 172.21.15.28:0/2997187819 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) ==== 100+0+11 (secure 0 0 0) 0x7fb67009ccb0 con 0x7fb67c074a60 2024-04-24T08:09:34.080 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-24T08:09:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.081+0000 7fb682000640 1 -- 172.21.15.28:0/2997187819 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb6640a3bb0 msgr2=0x7fb6640a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.081+0000 7fb682000640 1 --2- 172.21.15.28:0/2997187819 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb6640a3bb0 0x7fb6640a6070 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fb67c19b8e0 tx=0x7fb668005f90 comp rx=0 tx=0).stop 2024-04-24T08:09:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.081+0000 7fb682000640 1 -- 172.21.15.28:0/2997187819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb67c074a60 msgr2=0x7fb67c19a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.081+0000 7fb682000640 1 --2- 172.21.15.28:0/2997187819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb67c074a60 0x7fb67c19a3d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fb67000e9c0 tx=0x7fb67000ee80 comp rx=0 tx=0).stop 2024-04-24T08:09:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.081+0000 7fb682000640 1 -- 172.21.15.28:0/2997187819 shutdown_connections 2024-04-24T08:09:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.081+0000 7fb682000640 1 --2- 172.21.15.28:0/2997187819 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb6640a3bb0 0x7fb6640a6070 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.081+0000 7fb682000640 1 --2- 172.21.15.28:0/2997187819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb67c109dd0 0x7fb67c1a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.081+0000 7fb682000640 1 --2- 172.21.15.28:0/2997187819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb67c076040 0x7fb67c19a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.081+0000 7fb682000640 1 --2- 172.21.15.28:0/2997187819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb67c074a60 0x7fb67c19a3d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.081+0000 7fb682000640 1 -- 172.21.15.28:0/2997187819 >> 172.21.15.28:0/2997187819 conn(0x7fb67c0fd4a0 msgr2=0x7fb67c105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.082+0000 7fb682000640 1 -- 172.21.15.28:0/2997187819 shutdown_connections 2024-04-24T08:09:34.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.082+0000 7fb682000640 1 -- 172.21.15.28:0/2997187819 wait complete. 2024-04-24T08:09:34.102 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:09:34.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.315+0000 7f7b86600640 1 -- 172.21.15.28:0/3626086090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b80074a60 msgr2=0x7f7b80074ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.315+0000 7f7b86600640 1 --2- 172.21.15.28:0/3626086090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b80074a60 0x7f7b80074ee0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f7b6c009890 tx=0x7f7b6c02a360 comp rx=0 tx=0).stop 2024-04-24T08:09:34.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.315+0000 7f7b86600640 1 -- 172.21.15.28:0/3626086090 shutdown_connections 2024-04-24T08:09:34.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.315+0000 7f7b86600640 1 --2- 172.21.15.28:0/3626086090 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8010e440 0x7f7b80110830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.315+0000 7f7b86600640 1 --2- 172.21.15.28:0/3626086090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b80074a60 0x7f7b80074ee0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.315+0000 7f7b86600640 1 --2- 172.21.15.28:0/3626086090 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b80076040 0x7f7b80074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.315+0000 7f7b86600640 1 -- 172.21.15.28:0/3626086090 >> 172.21.15.28:0/3626086090 conn(0x7f7b800fd3e0 msgr2=0x7f7b800ff820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:34.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.315+0000 7f7b86600640 1 -- 172.21.15.28:0/3626086090 shutdown_connections 2024-04-24T08:09:34.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.315+0000 7f7b86600640 1 -- 172.21.15.28:0/3626086090 wait complete. 2024-04-24T08:09:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.316+0000 7f7b86600640 1 Processor -- start 2024-04-24T08:09:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.316+0000 7f7b86600640 1 -- start start 2024-04-24T08:09:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.317+0000 7f7b86600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b80074a60 0x7f7b8019e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.317+0000 7f7b86600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b80076040 0x7f7b8019ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.317+0000 7f7b86600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8010e440 0x7f7b801a5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.317+0000 7f7b86600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7b801140c0 con 0x7f7b8010e440 2024-04-24T08:09:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.317+0000 7f7b86600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7b80113f40 con 0x7f7b80074a60 2024-04-24T08:09:34.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.317+0000 7f7b86600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7b80114240 con 0x7f7b80076040 2024-04-24T08:09:34.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.317+0000 7f7b85c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8010e440 0x7f7b801a5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.317+0000 7f7b85c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8010e440 0x7f7b801a5da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59786/0 (socket says 172.21.15.28:59786) 2024-04-24T08:09:34.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.317+0000 7f7b85c00640 1 -- 172.21.15.28:0/160121291 learned_addr learned my addr 172.21.15.28:0/160121291 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:34.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.317+0000 7f7b85200640 1 --2- 172.21.15.28:0/160121291 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b80074a60 0x7f7b8019e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.317+0000 7f7b7fe00640 1 --2- 172.21.15.28:0/160121291 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b80076040 0x7f7b8019ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.317+0000 7f7b85c00640 1 -- 172.21.15.28:0/160121291 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b80076040 msgr2=0x7f7b8019ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.317+0000 7f7b85c00640 1 --2- 172.21.15.28:0/160121291 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b80076040 0x7f7b8019ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.317+0000 7f7b85c00640 1 -- 172.21.15.28:0/160121291 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b80074a60 msgr2=0x7f7b8019e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.317+0000 7f7b85c00640 1 --2- 172.21.15.28:0/160121291 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b80074a60 0x7f7b8019e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.318+0000 7f7b85c00640 1 -- 172.21.15.28:0/160121291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b801a6410 con 0x7f7b8010e440 2024-04-24T08:09:34.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.318+0000 7f7b7fe00640 1 --2- 172.21.15.28:0/160121291 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b80076040 0x7f7b8019ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:34.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.318+0000 7f7b85c00640 1 --2- 172.21.15.28:0/160121291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8010e440 0x7f7b801a5da0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7b7800efc0 tx=0x7f7b7800c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:34.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.318+0000 7f7b7d600640 1 -- 172.21.15.28:0/160121291 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b78003970 con 0x7f7b8010e440 2024-04-24T08:09:34.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.318+0000 7f7b7d600640 1 -- 172.21.15.28:0/160121291 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b78004020 con 0x7f7b8010e440 2024-04-24T08:09:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.318+0000 7f7b7d600640 1 -- 172.21.15.28:0/160121291 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b7800b9f0 con 0x7f7b8010e440 2024-04-24T08:09:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.318+0000 7f7b86600640 1 -- 172.21.15.28:0/160121291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b801a6700 con 0x7f7b8010e440 2024-04-24T08:09:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.318+0000 7f7b86600640 1 -- 172.21.15.28:0/160121291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b801a6bd0 con 0x7f7b8010e440 2024-04-24T08:09:34.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.320+0000 7f7b86600640 1 -- 172.21.15.28:0/160121291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b80104610 con 0x7f7b8010e440 2024-04-24T08:09:34.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.322+0000 7f7b7d600640 1 -- 172.21.15.28:0/160121291 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f7b7801d020 con 0x7f7b8010e440 2024-04-24T08:09:34.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.324+0000 7f7b7d600640 1 --2- 172.21.15.28:0/160121291 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7b680a3bb0 0x7f7b680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.324+0000 7f7b7d600640 1 -- 172.21.15.28:0/160121291 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f7b780d0310 con 0x7f7b8010e440 2024-04-24T08:09:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.324+0000 7f7b85200640 1 --2- 172.21.15.28:0/160121291 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7b680a3bb0 0x7f7b680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.325+0000 7f7b85200640 1 --2- 172.21.15.28:0/160121291 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7b680a3bb0 0x7f7b680a6070 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009bd0 tx=0x7f7b74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.330+0000 7f7b7d600640 1 -- 172.21.15.28:0/160121291 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7b7809cc70 con 0x7f7b8010e440 2024-04-24T08:09:34.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:34 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/666496647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:09:34.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:34 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2997187819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:09:34.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.579+0000 7f7b86600640 1 -- 172.21.15.28:0/160121291 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b80062c00 con 0x7f7b8010e440 2024-04-24T08:09:34.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.581+0000 7f7b7d600640 1 -- 172.21.15.28:0/160121291 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11871 (secure 0 0 0) 0x7f7b7809c570 con 0x7f7b8010e440 2024-04-24T08:09:34.583 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:09:34.583 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:07:57.133790+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:09:34.583 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 23 2024-04-24T08:09:34.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.583+0000 7f7b86600640 1 -- 172.21.15.28:0/160121291 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7b680a3bb0 msgr2=0x7f7b680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.584+0000 7f7b86600640 1 --2- 172.21.15.28:0/160121291 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7b680a3bb0 0x7f7b680a6070 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7b74009bd0 tx=0x7f7b74005f90 comp rx=0 tx=0).stop 2024-04-24T08:09:34.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.584+0000 7f7b86600640 1 -- 172.21.15.28:0/160121291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8010e440 msgr2=0x7f7b801a5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.584+0000 7f7b86600640 1 --2- 172.21.15.28:0/160121291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8010e440 0x7f7b801a5da0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f7b7800efc0 tx=0x7f7b7800c800 comp rx=0 tx=0).stop 2024-04-24T08:09:34.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.584+0000 7f7b86600640 1 -- 172.21.15.28:0/160121291 shutdown_connections 2024-04-24T08:09:34.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.584+0000 7f7b86600640 1 --2- 172.21.15.28:0/160121291 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7b680a3bb0 0x7f7b680a6070 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.584+0000 7f7b86600640 1 --2- 172.21.15.28:0/160121291 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7b8010e440 0x7f7b801a5da0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.584+0000 7f7b86600640 1 --2- 172.21.15.28:0/160121291 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7b80076040 0x7f7b8019ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.584+0000 7f7b86600640 1 --2- 172.21.15.28:0/160121291 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b80074a60 0x7f7b8019e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.584+0000 7f7b86600640 1 -- 172.21.15.28:0/160121291 >> 172.21.15.28:0/160121291 conn(0x7f7b800fd3e0 msgr2=0x7f7b801017e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:34.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.584+0000 7f7b86600640 1 -- 172.21.15.28:0/160121291 shutdown_connections 2024-04-24T08:09:34.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.584+0000 7f7b86600640 1 -- 172.21.15.28:0/160121291 wait complete. 2024-04-24T08:09:34.605 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:09:34.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:34 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/666496647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:09:34.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:34 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2997187819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:09:34.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:34 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/666496647' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:09:34.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:34 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2997187819' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:09:34.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.821+0000 7fdc1da00640 1 -- 172.21.15.28:0/61631966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc18100d80 msgr2=0x7fdc181011a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.821+0000 7fdc1da00640 1 --2- 172.21.15.28:0/61631966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc18100d80 0x7fdc181011a0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdc08009900 tx=0x7fdc0802a410 comp rx=0 tx=0).stop 2024-04-24T08:09:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.821+0000 7fdc1da00640 1 -- 172.21.15.28:0/61631966 shutdown_connections 2024-04-24T08:09:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.821+0000 7fdc1da00640 1 --2- 172.21.15.28:0/61631966 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc18101f70 0x7fdc181023f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.821+0000 7fdc1da00640 1 --2- 172.21.15.28:0/61631966 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc18102930 0x7fdc181091c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.821+0000 7fdc1da00640 1 --2- 172.21.15.28:0/61631966 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc18100d80 0x7fdc181011a0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.821+0000 7fdc1da00640 1 -- 172.21.15.28:0/61631966 >> 172.21.15.28:0/61631966 conn(0x7fdc180fc2d0 msgr2=0x7fdc180fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.821+0000 7fdc1da00640 1 -- 172.21.15.28:0/61631966 shutdown_connections 2024-04-24T08:09:34.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.821+0000 7fdc1da00640 1 -- 172.21.15.28:0/61631966 wait complete. 2024-04-24T08:09:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.822+0000 7fdc1da00640 1 Processor -- start 2024-04-24T08:09:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.822+0000 7fdc1da00640 1 -- start start 2024-04-24T08:09:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.822+0000 7fdc1da00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc18100d80 0x7fdc181101a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.822+0000 7fdc1da00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc18101f70 0x7fdc181106e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.822+0000 7fdc1da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc18102930 0x7fdc1810cd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.822+0000 7fdc1da00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1810b130 con 0x7fdc18102930 2024-04-24T08:09:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.822+0000 7fdc1da00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1810afb0 con 0x7fdc18100d80 2024-04-24T08:09:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.822+0000 7fdc1da00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1810b2b0 con 0x7fdc18101f70 2024-04-24T08:09:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.823+0000 7fdc17400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc18101f70 0x7fdc181106e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.823+0000 7fdc17400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc18101f70 0x7fdc181106e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:34262/0 (socket says 172.21.15.28:34262) 2024-04-24T08:09:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.823+0000 7fdc17400640 1 -- 172.21.15.28:0/2133119994 learned_addr learned my addr 172.21.15.28:0/2133119994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:34.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.823+0000 7fdc17e00640 1 --2- 172.21.15.28:0/2133119994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc18100d80 0x7fdc181101a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.823+0000 7fdc1d000640 1 --2- 172.21.15.28:0/2133119994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc18102930 0x7fdc1810cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.823+0000 7fdc17400640 1 -- 172.21.15.28:0/2133119994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc18100d80 msgr2=0x7fdc181101a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.823+0000 7fdc17400640 1 --2- 172.21.15.28:0/2133119994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc18100d80 0x7fdc181101a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.823+0000 7fdc17400640 1 -- 172.21.15.28:0/2133119994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc18102930 msgr2=0x7fdc1810cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.823+0000 7fdc17400640 1 --2- 172.21.15.28:0/2133119994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc18102930 0x7fdc1810cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.823+0000 7fdc17400640 1 -- 172.21.15.28:0/2133119994 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1810d5d0 con 0x7fdc18101f70 2024-04-24T08:09:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.823+0000 7fdc1d000640 1 --2- 172.21.15.28:0/2133119994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc18102930 0x7fdc1810cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:09:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.823+0000 7fdc17400640 1 --2- 172.21.15.28:0/2133119994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc18101f70 0x7fdc181106e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400e9f0 tx=0x7fdc0400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.824+0000 7fdc14c00640 1 -- 172.21.15.28:0/2133119994 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0400cd80 con 0x7fdc18101f70 2024-04-24T08:09:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.824+0000 7fdc14c00640 1 -- 172.21.15.28:0/2133119994 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc04010910 con 0x7fdc18101f70 2024-04-24T08:09:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.824+0000 7fdc1da00640 1 -- 172.21.15.28:0/2133119994 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc181a6800 con 0x7fdc18101f70 2024-04-24T08:09:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.824+0000 7fdc14c00640 1 -- 172.21.15.28:0/2133119994 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04018990 con 0x7fdc18101f70 2024-04-24T08:09:34.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.824+0000 7fdc1da00640 1 -- 172.21.15.28:0/2133119994 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc181a6c60 con 0x7fdc18101f70 2024-04-24T08:09:34.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.825+0000 7fdc1da00640 1 -- 172.21.15.28:0/2133119994 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc181078c0 con 0x7fdc18101f70 2024-04-24T08:09:34.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.828+0000 7fdc14c00640 1 -- 172.21.15.28:0/2133119994 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fdc04010ac0 con 0x7fdc18101f70 2024-04-24T08:09:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.829+0000 7fdc14c00640 1 --2- 172.21.15.28:0/2133119994 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fdbfc0a3c30 0x7fdbfc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.829+0000 7fdc14c00640 1 -- 172.21.15.28:0/2133119994 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fdc040d0ba0 con 0x7fdc18101f70 2024-04-24T08:09:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.829+0000 7fdc17e00640 1 --2- 172.21.15.28:0/2133119994 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fdbfc0a3c30 0x7fdbfc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:34.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.830+0000 7fdc17e00640 1 --2- 172.21.15.28:0/2133119994 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fdbfc0a3c30 0x7fdbfc0a60f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdc0802aa00 tx=0x7fdc080315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:34.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:34.831+0000 7fdc14c00640 1 -- 172.21.15.28:0/2133119994 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc0409d500 con 0x7fdc18101f70 2024-04-24T08:09:35.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.041+0000 7fdc1da00640 1 -- 172.21.15.28:0/2133119994 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdc18062c00 con 0x7fdc18101f70 2024-04-24T08:09:35.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.043+0000 7fdc14c00640 1 -- 172.21.15.28:0/2133119994 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) ==== 74+0+22677 (secure 0 0 0) 0x7fdc0409ce00 con 0x7fdc18101f70 2024-04-24T08:09:35.045 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:09:35.045 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":88,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:08:51.155712+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:09:35.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.046+0000 7fdc1da00640 1 -- 172.21.15.28:0/2133119994 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fdbfc0a3c30 msgr2=0x7fdbfc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.046+0000 7fdc1da00640 1 --2- 172.21.15.28:0/2133119994 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fdbfc0a3c30 0x7fdbfc0a60f0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fdc0802aa00 tx=0x7fdc080315b0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.046+0000 7fdc1da00640 1 -- 172.21.15.28:0/2133119994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc18101f70 msgr2=0x7fdc181106e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.046+0000 7fdc1da00640 1 --2- 172.21.15.28:0/2133119994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc18101f70 0x7fdc181106e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400e9f0 tx=0x7fdc0400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.046+0000 7fdc1da00640 1 -- 172.21.15.28:0/2133119994 shutdown_connections 2024-04-24T08:09:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.046+0000 7fdc1da00640 1 --2- 172.21.15.28:0/2133119994 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fdbfc0a3c30 0x7fdbfc0a60f0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.046+0000 7fdc1da00640 1 --2- 172.21.15.28:0/2133119994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc18102930 0x7fdc1810cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.046+0000 7fdc1da00640 1 --2- 172.21.15.28:0/2133119994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc18101f70 0x7fdc181106e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.046+0000 7fdc1da00640 1 --2- 172.21.15.28:0/2133119994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc18100d80 0x7fdc181101a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.046+0000 7fdc1da00640 1 -- 172.21.15.28:0/2133119994 >> 172.21.15.28:0/2133119994 conn(0x7fdc180fc2d0 msgr2=0x7fdc180fddd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.046+0000 7fdc1da00640 1 -- 172.21.15.28:0/2133119994 shutdown_connections 2024-04-24T08:09:35.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.046+0000 7fdc1da00640 1 -- 172.21.15.28:0/2133119994 wait complete. 2024-04-24T08:09:35.069 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:09:35.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.284+0000 7f1a0fe00640 1 -- 172.21.15.28:0/2493906855 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a10102ff0 msgr2=0x7f1a10103470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.284+0000 7f1a0fe00640 1 --2- 172.21.15.28:0/2493906855 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a10102ff0 0x7f1a10103470 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f19f8009900 tx=0x7f19f802a410 comp rx=0 tx=0).stop 2024-04-24T08:09:35.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.284+0000 7f1a0fe00640 1 -- 172.21.15.28:0/2493906855 shutdown_connections 2024-04-24T08:09:35.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.284+0000 7f1a0fe00640 1 --2- 172.21.15.28:0/2493906855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a101039b0 0x7f1a1010a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.284+0000 7f1a0fe00640 1 --2- 172.21.15.28:0/2493906855 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a10102ff0 0x7f1a10103470 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.284+0000 7f1a0fe00640 1 --2- 172.21.15.28:0/2493906855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a10101ea0 0x7f1a101022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.284+0000 7f1a0fe00640 1 -- 172.21.15.28:0/2493906855 >> 172.21.15.28:0/2493906855 conn(0x7f1a100fd410 msgr2=0x7f1a100ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:35.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.285+0000 7f1a0fe00640 1 -- 172.21.15.28:0/2493906855 shutdown_connections 2024-04-24T08:09:35.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.285+0000 7f1a0fe00640 1 -- 172.21.15.28:0/2493906855 wait complete. 2024-04-24T08:09:35.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.286+0000 7f1a0fe00640 1 Processor -- start 2024-04-24T08:09:35.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.286+0000 7f1a0fe00640 1 -- start start 2024-04-24T08:09:35.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.287+0000 7f1a0fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a10101ea0 0x7f1a1019a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.287+0000 7f1a0fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a10102ff0 0x7f1a1019a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.287+0000 7f1a0fe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a101039b0 0x7f1a101a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.287+0000 7f1a0fe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1010db80 con 0x7f1a10102ff0 2024-04-24T08:09:35.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.287+0000 7f1a0fe00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1010da00 con 0x7f1a101039b0 2024-04-24T08:09:35.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.287+0000 7f1a0fe00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1a1010dd00 con 0x7f1a10101ea0 2024-04-24T08:09:35.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.287+0000 7f1a06000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a10102ff0 0x7f1a1019a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.287+0000 7f1a06000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a10102ff0 0x7f1a1019a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59824/0 (socket says 172.21.15.28:59824) 2024-04-24T08:09:35.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.287+0000 7f1a06000640 1 -- 172.21.15.28:0/2926155163 learned_addr learned my addr 172.21.15.28:0/2926155163 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:35.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.287+0000 7f1a0ea00640 1 --2- 172.21.15.28:0/2926155163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a10101ea0 0x7f1a1019a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.287+0000 7f1a0f400640 1 --2- 172.21.15.28:0/2926155163 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a101039b0 0x7f1a101a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.287+0000 7f1a0ea00640 1 -- 172.21.15.28:0/2926155163 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a101039b0 msgr2=0x7f1a101a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.288+0000 7f1a0ea00640 1 --2- 172.21.15.28:0/2926155163 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a101039b0 0x7f1a101a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.288+0000 7f1a0ea00640 1 -- 172.21.15.28:0/2926155163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a10102ff0 msgr2=0x7f1a1019a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.288+0000 7f1a0ea00640 1 --2- 172.21.15.28:0/2926155163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a10102ff0 0x7f1a1019a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.288+0000 7f1a0ea00640 1 -- 172.21.15.28:0/2926155163 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1a101a1fd0 con 0x7f1a10101ea0 2024-04-24T08:09:35.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.288+0000 7f1a06000640 1 --2- 172.21.15.28:0/2926155163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a10102ff0 0x7f1a1019a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.288+0000 7f1a0ea00640 1 --2- 172.21.15.28:0/2926155163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a10101ea0 0x7f1a1019a400 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1a0000ec80 tx=0x7f1a0000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.288+0000 7f1a07e00640 1 -- 172.21.15.28:0/2926155163 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a00003b70 con 0x7f1a10101ea0 2024-04-24T08:09:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.288+0000 7f1a07e00640 1 -- 172.21.15.28:0/2926155163 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1a00003d10 con 0x7f1a10101ea0 2024-04-24T08:09:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.288+0000 7f1a0fe00640 1 -- 172.21.15.28:0/2926155163 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1a101a22c0 con 0x7f1a10101ea0 2024-04-24T08:09:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.288+0000 7f1a07e00640 1 -- 172.21.15.28:0/2926155163 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1a00010990 con 0x7f1a10101ea0 2024-04-24T08:09:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.288+0000 7f1a0fe00640 1 -- 172.21.15.28:0/2926155163 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1a101a2830 con 0x7f1a10101ea0 2024-04-24T08:09:35.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.289+0000 7f1a0fe00640 1 -- 172.21.15.28:0/2926155163 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1a101a2af0 con 0x7f1a10101ea0 2024-04-24T08:09:35.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.292+0000 7f1a07e00640 1 -- 172.21.15.28:0/2926155163 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1a00004620 con 0x7f1a10101ea0 2024-04-24T08:09:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.293+0000 7f1a07e00640 1 --2- 172.21.15.28:0/2926155163 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f19f40a3c30 0x7f19f40a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.294+0000 7f1a06000640 1 --2- 172.21.15.28:0/2926155163 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f19f40a3c30 0x7f19f40a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.294+0000 7f1a07e00640 1 -- 172.21.15.28:0/2926155163 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f1a000d0750 con 0x7f1a10101ea0 2024-04-24T08:09:35.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.294+0000 7f1a06000640 1 --2- 172.21.15.28:0/2926155163 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f19f40a3c30 0x7f19f40a60f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f19f8000c00 tx=0x7f19f80315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:35.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.298+0000 7f1a07e00640 1 -- 172.21.15.28:0/2926155163 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1a0009d0b0 con 0x7f1a10101ea0 2024-04-24T08:09:35.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.553+0000 7f1a0fe00640 1 -- 172.21.15.28:0/2926155163 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f1a101a3240 con 0x7f1a10101ea0 2024-04-24T08:09:35.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.555+0000 7f1a07e00640 1 -- 172.21.15.28:0/2926155163 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11871 (secure 0 0 0) 0x7f1a0009c9b0 con 0x7f1a10101ea0 2024-04-24T08:09:35.556 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:09:35.556 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:07:57.133790+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:09:35.556 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 23 2024-04-24T08:09:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.557+0000 7f1a0fe00640 1 -- 172.21.15.28:0/2926155163 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f19f40a3c30 msgr2=0x7f19f40a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.557+0000 7f1a0fe00640 1 --2- 172.21.15.28:0/2926155163 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f19f40a3c30 0x7f19f40a60f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f19f8000c00 tx=0x7f19f80315b0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.557+0000 7f1a0fe00640 1 -- 172.21.15.28:0/2926155163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a10101ea0 msgr2=0x7f1a1019a400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.557+0000 7f1a0fe00640 1 --2- 172.21.15.28:0/2926155163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a10101ea0 0x7f1a1019a400 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f1a0000ec80 tx=0x7f1a0000c6a0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.557+0000 7f1a0fe00640 1 -- 172.21.15.28:0/2926155163 shutdown_connections 2024-04-24T08:09:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.557+0000 7f1a0fe00640 1 --2- 172.21.15.28:0/2926155163 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f19f40a3c30 0x7f19f40a60f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.557+0000 7f1a0fe00640 1 --2- 172.21.15.28:0/2926155163 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1a10102ff0 0x7f1a1019a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.557+0000 7f1a0fe00640 1 --2- 172.21.15.28:0/2926155163 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a101039b0 0x7f1a101a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.557+0000 7f1a0fe00640 1 --2- 172.21.15.28:0/2926155163 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1a10101ea0 0x7f1a1019a400 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.557+0000 7f1a0fe00640 1 -- 172.21.15.28:0/2926155163 >> 172.21.15.28:0/2926155163 conn(0x7f1a100fd410 msgr2=0x7f1a10074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.557+0000 7f1a0fe00640 1 -- 172.21.15.28:0/2926155163 shutdown_connections 2024-04-24T08:09:35.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.557+0000 7f1a0fe00640 1 -- 172.21.15.28:0/2926155163 wait complete. 2024-04-24T08:09:35.579 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-24T08:07:51.585580+0000', 'modified': '2024-04-24T08:07:57.133790+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24574, 'mds_1': 24572, 'mds_2': 24556}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24515': {'gid': 24515, 'name': 'h', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.117:6837/862115546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6836', 'nonce': 862115546}, {'type': 'v1', 'addr': '172.21.15.117:6837', 'nonce': 862115546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.194:6835/356867408', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6834', 'nonce': 356867408}, {'type': 'v1', 'addr': '172.21.15.194:6835', 'nonce': 356867408}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24535': {'gid': 24535, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.194:6837/3046177244', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6836', 'nonce': 3046177244}, {'type': 'v1', 'addr': '172.21.15.194:6837', 'nonce': 3046177244}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24556': {'gid': 24556, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.194:6839/2785495353', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6838', 'nonce': 2785495353}, {'type': 'v1', 'addr': '172.21.15.194:6839', 'nonce': 2785495353}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.117:6839/3606021351', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6838', 'nonce': 3606021351}, {'type': 'v1', 'addr': '172.21.15.117:6839', 'nonce': 3606021351}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24574': {'gid': 24574, 'name': 'g', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.28:6839/1247092633', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 1247092633}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 1247092633}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24574, 'qdb_cluster': [24556, 24572, 24574]} 2024-04-24T08:09:35.579 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-24T08:09:35.579 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-24T08:09:35.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:35 smithi117 ceph-mon[32438]: pgmap v370: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-24T08:09:35.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:35 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/160121291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:09:35.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:35 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2133119994' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:09:35.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:35 smithi028 ceph-mon[27786]: pgmap v370: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-24T08:09:35.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:35 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/160121291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:09:35.654 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:35 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2133119994' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:09:35.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:35 smithi194 ceph-mon[32836]: pgmap v370: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 597 B/s wr, 8 op/s 2024-04-24T08:09:35.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:35 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/160121291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:09:35.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:35 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2133119994' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:09:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.796+0000 7f6889e00640 1 -- 172.21.15.28:0/3278219018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6884106580 msgr2=0x7f6884108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.796+0000 7f6889e00640 1 --2- 172.21.15.28:0/3278219018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6884106580 0x7f6884108970 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f687c009ca0 tx=0x7f687c02a8c0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.796+0000 7f6889e00640 1 -- 172.21.15.28:0/3278219018 shutdown_connections 2024-04-24T08:09:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.796+0000 7f6889e00640 1 --2- 172.21.15.28:0/3278219018 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6884106580 0x7f6884108970 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.796+0000 7f6889e00640 1 --2- 172.21.15.28:0/3278219018 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6884101880 0x7f6884101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.796+0000 7f6889e00640 1 --2- 172.21.15.28:0/3278219018 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68840684b0 0x7f6884101340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.796+0000 7f6889e00640 1 -- 172.21.15.28:0/3278219018 >> 172.21.15.28:0/3278219018 conn(0x7f68840fb6c0 msgr2=0x7f68840fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.796+0000 7f6889e00640 1 -- 172.21.15.28:0/3278219018 shutdown_connections 2024-04-24T08:09:35.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.796+0000 7f6889e00640 1 -- 172.21.15.28:0/3278219018 wait complete. 2024-04-24T08:09:35.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.797+0000 7f6889e00640 1 Processor -- start 2024-04-24T08:09:35.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.797+0000 7f6889e00640 1 -- start start 2024-04-24T08:09:35.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.798+0000 7f6889e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68840684b0 0x7f688410bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.798+0000 7f6889e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6884101880 0x7f688410a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.798+0000 7f6889e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6884106580 0x7f688410a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.798+0000 7f6889e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f688410dbe0 con 0x7f6884106580 2024-04-24T08:09:35.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.798+0000 7f6889e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f688410da60 con 0x7f6884101880 2024-04-24T08:09:35.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.798+0000 7f6889e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f688410dd60 con 0x7f68840684b0 2024-04-24T08:09:35.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.798+0000 7f6889400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6884106580 0x7f688410a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.798+0000 7f6888a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68840684b0 0x7f688410bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.798+0000 7f6889400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6884106580 0x7f688410a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59840/0 (socket says 172.21.15.28:59840) 2024-04-24T08:09:35.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.798+0000 7f6889400640 1 -- 172.21.15.28:0/3888158277 learned_addr learned my addr 172.21.15.28:0/3888158277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:35.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.798+0000 7f6883e00640 1 --2- 172.21.15.28:0/3888158277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6884101880 0x7f688410a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.799+0000 7f6888a00640 1 -- 172.21.15.28:0/3888158277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6884101880 msgr2=0x7f688410a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.799+0000 7f6888a00640 1 --2- 172.21.15.28:0/3888158277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6884101880 0x7f688410a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.799+0000 7f6888a00640 1 -- 172.21.15.28:0/3888158277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6884106580 msgr2=0x7f688410a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.799+0000 7f6888a00640 1 --2- 172.21.15.28:0/3888158277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6884106580 0x7f688410a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.799+0000 7f6888a00640 1 -- 172.21.15.28:0/3888158277 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f688410b0d0 con 0x7f68840684b0 2024-04-24T08:09:35.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.799+0000 7f6889400640 1 --2- 172.21.15.28:0/3888158277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6884106580 0x7f688410a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:35.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.799+0000 7f6888a00640 1 --2- 172.21.15.28:0/3888158277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68840684b0 0x7f688410bcc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f68841011a0 tx=0x7f687800dbb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:35.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.799+0000 7f6881600640 1 -- 172.21.15.28:0/3888158277 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6878017070 con 0x7f68840684b0 2024-04-24T08:09:35.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.799+0000 7f6889e00640 1 -- 172.21.15.28:0/3888158277 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68841a2330 con 0x7f68840684b0 2024-04-24T08:09:35.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.799+0000 7f6881600640 1 -- 172.21.15.28:0/3888158277 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6878010460 con 0x7f68840684b0 2024-04-24T08:09:35.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.799+0000 7f6881600640 1 -- 172.21.15.28:0/3888158277 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687800f620 con 0x7f68840684b0 2024-04-24T08:09:35.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.799+0000 7f6889e00640 1 -- 172.21.15.28:0/3888158277 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68841a2880 con 0x7f68840684b0 2024-04-24T08:09:35.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.804+0000 7f688e381740 1 Processor -- start 2024-04-24T08:09:35.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.804+0000 7f688e381740 1 -- start start 2024-04-24T08:09:35.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.804+0000 7f688e381740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640c260 0x56327640c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.804+0000 7f688e381740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640cc50 0x563276411750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.804+0000 7f688e381740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563276411dc0 0x563276414250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.804+0000 7f688e381740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563276371b10 con 0x56327640cc50 2024-04-24T08:09:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.804+0000 7f688e381740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5632763182b0 con 0x56327640c260 2024-04-24T08:09:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.804+0000 7f688e381740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56327636dad0 con 0x563276411dc0 2024-04-24T08:09:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.804+0000 7f6881600640 1 -- 172.21.15.28:0/3888158277 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f687801a070 con 0x7f68840684b0 2024-04-24T08:09:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.806+0000 7f6889400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640c260 0x56327640c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.806+0000 7f6888a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640cc50 0x563276411750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.806+0000 7f6889400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640c260 0x56327640c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:55538/0 (socket says 172.21.15.28:55538) 2024-04-24T08:09:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.806+0000 7f6889400640 1 -- 172.21.15.28:0/2328800263 learned_addr learned my addr 172.21.15.28:0/2328800263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.806+0000 7f6883e00640 1 --2- 172.21.15.28:0/2328800263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563276411dc0 0x563276414250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.806+0000 7f6889400640 1 -- 172.21.15.28:0/2328800263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563276411dc0 msgr2=0x563276414250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.806+0000 7f6889400640 1 --2- 172.21.15.28:0/2328800263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563276411dc0 0x563276414250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.806+0000 7f6889400640 1 -- 172.21.15.28:0/2328800263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640cc50 msgr2=0x563276411750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:35.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.806+0000 7f6889400640 1 --2- 172.21.15.28:0/2328800263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640cc50 0x563276411750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.806+0000 7f6889400640 1 -- 172.21.15.28:0/2328800263 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632760842d0 con 0x56327640c260 2024-04-24T08:09:35.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.806+0000 7f6883e00640 1 --2- 172.21.15.28:0/2328800263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563276411dc0 0x563276414250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:35.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.806+0000 7f6888a00640 1 --2- 172.21.15.28:0/2328800263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640cc50 0x563276411750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:35.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.807+0000 7f6889400640 1 --2- 172.21.15.28:0/2328800263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640c260 0x56327640c680 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f687c009430 tx=0x7f687c005ef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26d5862a7d6132eb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:35.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.807+0000 7f6875c00640 1 -- 172.21.15.28:0/2328800263 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687c049070 con 0x56327640c260 2024-04-24T08:09:35.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.807+0000 7f6875c00640 1 -- 172.21.15.28:0/2328800263 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f687c036040 con 0x56327640c260 2024-04-24T08:09:35.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.807+0000 7f6875c00640 1 -- 172.21.15.28:0/2328800263 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687c0305c0 con 0x56327640c260 2024-04-24T08:09:35.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f688e381740 1 -- 172.21.15.28:0/2328800263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640c260 msgr2=0x56327640c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f688e381740 1 --2- 172.21.15.28:0/2328800263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640c260 0x56327640c680 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f687c009430 tx=0x7f687c005ef0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f6881600640 1 --2- 172.21.15.28:0/3888158277 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f686c0a3bb0 0x7f686c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f688e381740 1 -- 172.21.15.28:0/2328800263 shutdown_connections 2024-04-24T08:09:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f688e381740 1 --2- 172.21.15.28:0/2328800263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640cc50 0x563276411750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f688e381740 1 --2- 172.21.15.28:0/2328800263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563276411dc0 0x563276414250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f688e381740 1 --2- 172.21.15.28:0/2328800263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640c260 0x56327640c680 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f6881600640 1 -- 172.21.15.28:0/3888158277 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f68780cfd80 con 0x7f68840684b0 2024-04-24T08:09:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f688e381740 1 -- 172.21.15.28:0/2328800263 >> 172.21.15.28:0/2328800263 conn(0x5632764078d0 msgr2=0x563276407cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f6888a00640 1 --2- 172.21.15.28:0/3888158277 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f686c0a3bb0 0x7f686c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f688e381740 1 -- 172.21.15.28:0/2328800263 shutdown_connections 2024-04-24T08:09:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f688e381740 1 -- 172.21.15.28:0/2328800263 wait complete. 2024-04-24T08:09:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f6888a00640 1 --2- 172.21.15.28:0/3888158277 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f686c0a3bb0 0x7f686c0a6070 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f68780cd800 tx=0x7f68780d37c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f688e381740 1 -- start start 2024-04-24T08:09:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f688e381740 1 Processor -- start 2024-04-24T08:09:35.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f688e381740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640c260 0x563276416510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f688e381740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56327640cc50 0x563276416a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f688e381740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563276411dc0 0x56327641da20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f688e381740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563276414b90 con 0x56327640c260 2024-04-24T08:09:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f688e381740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56327632b590 con 0x563276411dc0 2024-04-24T08:09:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.808+0000 7f688e381740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563276414d10 con 0x56327640cc50 2024-04-24T08:09:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.809+0000 7f688e381740 10 client.0 _ll_register_callbacks cb 0x56327630e3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:09:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.809+0000 7f6889400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640c260 0x563276416510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.809+0000 7f6889400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640c260 0x563276416510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59868/0 (socket says 172.21.15.28:59868) 2024-04-24T08:09:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.809+0000 7f6889400640 1 -- 172.21.15.28:0/3737340043 learned_addr learned my addr 172.21.15.28:0/3737340043 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.809+0000 7f6883e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56327640cc50 0x563276416a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.809+0000 7f6889400640 1 --2- 172.21.15.28:0/3737340043 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563276411dc0 0x56327641da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.809+0000 7f6883e00640 1 -- 172.21.15.28:0/3737340043 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563276411dc0 msgr2=0x56327641da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.809+0000 7f6883e00640 1 --2- 172.21.15.28:0/3737340043 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563276411dc0 0x56327641da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.809+0000 7f6883e00640 1 -- 172.21.15.28:0/3737340043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640c260 msgr2=0x563276416510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.809+0000 7f6883e00640 1 --2- 172.21.15.28:0/3737340043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640c260 0x563276416510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.809+0000 7f6883e00640 1 -- 172.21.15.28:0/3737340043 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56327614e6b0 con 0x56327640cc50 2024-04-24T08:09:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.809+0000 7f6889400640 1 --2- 172.21.15.28:0/3737340043 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563276411dc0 0x56327641da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:09:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.810+0000 7f6883e00640 1 --2- 172.21.15.28:0/3737340043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56327640cc50 0x563276416a50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f688410ba30 tx=0x7f6870012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6e0261d304f37de server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.810+0000 7f686a000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:09:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.810+0000 7f686a000640 1 -- 172.21.15.28:0/3737340043 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f687000f890 con 0x56327640cc50 2024-04-24T08:09:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.810+0000 7f688e381740 1 -- 172.21.15.28:0/3737340043 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f687c036040 con 0x56327640cc50 2024-04-24T08:09:35.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.810+0000 7f686a000640 1 -- 172.21.15.28:0/3737340043 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f687000fa30 con 0x56327640cc50 2024-04-24T08:09:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.810+0000 7f686a000640 1 -- 172.21.15.28:0/3737340043 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f687000fc30 con 0x56327640cc50 2024-04-24T08:09:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.810+0000 7f688e381740 10 client.25003 fetch_fsmap learned FSMap version 23 2024-04-24T08:09:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.810+0000 7f688e381740 1 -- 172.21.15.28:0/3737340043 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x563276420eb0 con 0x56327640cc50 2024-04-24T08:09:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.811+0000 7f686a000640 1 -- 172.21.15.28:0/3737340043 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f687c036040 con 0x56327640cc50 2024-04-24T08:09:35.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.811+0000 7f686a000640 1 -- 172.21.15.28:0/3737340043 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f68700104c0 con 0x56327640cc50 2024-04-24T08:09:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.811+0000 7f688e381740 10 client.25003 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:09:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.811+0000 7f688e381740 10 client.25003 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-24T08:09:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.811+0000 7f688e381740 20 client.25003 populate_metadata read hostname 'smithi028' 2024-04-24T08:09:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.811+0000 7f688e381740 1 --2- 172.21.15.28:0/3737340043 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5632764234c0 0x563276424930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.811+0000 7f688e381740 4 client.25003 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:09:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.811+0000 7f688e381740 1 -- 172.21.15.28:0/3737340043 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56327637f530 con 0x5632764234c0 2024-04-24T08:09:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.812+0000 7f6889400640 1 --2- 172.21.15.28:0/3737340043 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5632764234c0 0x563276424930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.814+0000 7f6889400640 1 --2- 172.21.15.28:0/3737340043 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5632764234c0 0x563276424930 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=da97a6b14a267485 server_cookie=e1d7de69bf3a00ca in_seq=0 out_seq=0 2024-04-24T08:09:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.814+0000 7f686a000640 10 client.25003 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:09:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.814+0000 7f686a000640 1 -- 172.21.15.28:0/3737340043 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56327637f530 con 0x5632764234c0 2024-04-24T08:09:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.815+0000 7f686a000640 10 client.25003 handle_command_reply: tid=0 2024-04-24T08:09:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.815+0000 7f688e381740 1 client.25003 shutdown 2024-04-24T08:09:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.815+0000 7f688e381740 1 -- 172.21.15.28:0/3737340043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56327640cc50 msgr2=0x563276416a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.815+0000 7f688e381740 1 --2- 172.21.15.28:0/3737340043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56327640cc50 0x563276416a50 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f688410ba30 tx=0x7f6870012b60 comp rx=0 tx=0).stop 2024-04-24T08:09:35.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.815+0000 7f688e381740 1 -- 172.21.15.28:0/3737340043 shutdown_connections 2024-04-24T08:09:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.815+0000 7f688e381740 1 --2- 172.21.15.28:0/3737340043 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5632764234c0 0x563276424930 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.815+0000 7f688e381740 1 --2- 172.21.15.28:0/3737340043 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563276411dc0 0x56327641da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.815+0000 7f688e381740 1 --2- 172.21.15.28:0/3737340043 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56327640cc50 0x563276416a50 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.815+0000 7f688e381740 1 --2- 172.21.15.28:0/3737340043 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640c260 0x563276416510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.815+0000 7f686a000640 0 client.25003 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:09:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.815+0000 7f688e381740 1 -- 172.21.15.28:0/3737340043 >> 172.21.15.28:0/3737340043 conn(0x5632764078d0 msgr2=0x563276406970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.815+0000 7f688e381740 1 -- 172.21.15.28:0/3737340043 shutdown_connections 2024-04-24T08:09:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.815+0000 7f688e381740 1 -- 172.21.15.28:0/3737340043 wait complete. 2024-04-24T08:09:35.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.815+0000 7f688e381740 20 client.25003 trim_cache size 0 max 16384 2024-04-24T08:09:35.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.834+0000 7f688e381740 1 Processor -- start 2024-04-24T08:09:35.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.835+0000 7f688e381740 1 -- start start 2024-04-24T08:09:35.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.835+0000 7f688e381740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640c260 0x563276416120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.835+0000 7f688e381740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640cc50 0x56327641d9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.835+0000 7f688e381740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563276411dc0 0x563276416660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.835+0000 7f688e381740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563276422c70 con 0x56327640c260 2024-04-24T08:09:35.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.835+0000 7f688e381740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563276423340 con 0x56327640cc50 2024-04-24T08:09:35.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.835+0000 7f688e381740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563276423a10 con 0x563276411dc0 2024-04-24T08:09:35.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.835+0000 7f6889400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640c260 0x563276416120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.835+0000 7f6889400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640c260 0x563276416120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59872/0 (socket says 172.21.15.28:59872) 2024-04-24T08:09:35.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.835+0000 7f6889400640 1 -- 172.21.15.28:0/2788661619 learned_addr learned my addr 172.21.15.28:0/2788661619 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:35.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.835+0000 7f6883e00640 1 --2- 172.21.15.28:0/2788661619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640cc50 0x56327641d9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.835+0000 7f6889400640 1 --2- 172.21.15.28:0/2788661619 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563276411dc0 0x563276416660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.836+0000 7f6883e00640 1 -- 172.21.15.28:0/2788661619 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563276411dc0 msgr2=0x563276416660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.836+0000 7f6883e00640 1 --2- 172.21.15.28:0/2788661619 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563276411dc0 0x563276416660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.836+0000 7f6883e00640 1 -- 172.21.15.28:0/2788661619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640c260 msgr2=0x563276416120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.836+0000 7f6883e00640 1 --2- 172.21.15.28:0/2788661619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640c260 0x563276416120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.836+0000 7f6883e00640 1 -- 172.21.15.28:0/2788661619 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563276407700 con 0x56327640cc50 2024-04-24T08:09:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.836+0000 7f6883e00640 1 --2- 172.21.15.28:0/2788661619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640cc50 0x56327641d9e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6870000c00 tx=0x7f687000fed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47af398be31a9699 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.836+0000 7f6875200640 1 -- 172.21.15.28:0/2788661619 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6870010ba0 con 0x56327640cc50 2024-04-24T08:09:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.836+0000 7f6875200640 1 -- 172.21.15.28:0/2788661619 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6870010d40 con 0x56327640cc50 2024-04-24T08:09:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.836+0000 7f6875200640 1 -- 172.21.15.28:0/2788661619 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6870010450 con 0x56327640cc50 2024-04-24T08:09:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.838+0000 7f688e381740 1 -- 172.21.15.28:0/2788661619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640cc50 msgr2=0x56327641d9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.838+0000 7f688e381740 1 --2- 172.21.15.28:0/2788661619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640cc50 0x56327641d9e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6870000c00 tx=0x7f687000fed0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.838+0000 7f688e381740 1 -- 172.21.15.28:0/2788661619 shutdown_connections 2024-04-24T08:09:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.838+0000 7f688e381740 1 --2- 172.21.15.28:0/2788661619 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563276411dc0 0x563276416660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.838+0000 7f688e381740 1 --2- 172.21.15.28:0/2788661619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640cc50 0x56327641d9e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.838+0000 7f688e381740 1 --2- 172.21.15.28:0/2788661619 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56327640c260 0x563276416120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.838+0000 7f688e381740 1 -- 172.21.15.28:0/2788661619 >> 172.21.15.28:0/2788661619 conn(0x5632764078d0 msgr2=0x56327640e150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:35.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.838+0000 7f688e381740 1 -- 172.21.15.28:0/2788661619 shutdown_connections 2024-04-24T08:09:35.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.838+0000 7f688e381740 1 -- 172.21.15.28:0/2788661619 wait complete. 2024-04-24T08:09:35.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.838+0000 7f688e381740 1 -- start start 2024-04-24T08:09:35.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.839+0000 7f688e381740 1 Processor -- start 2024-04-24T08:09:35.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.839+0000 7f688e381740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640c260 0x56327641d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.839+0000 7f688e381740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56327640cc50 0x56327641a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.839+0000 7f688e381740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563276411dc0 0x56327641a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.839+0000 7f688e381740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5632764247b0 con 0x563276411dc0 2024-04-24T08:09:35.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.839+0000 7f688e381740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5632764240e0 con 0x56327640c260 2024-04-24T08:09:35.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.839+0000 7f688e381740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563276424e80 con 0x56327640cc50 2024-04-24T08:09:35.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.839+0000 7f688e381740 10 client.0 _ll_register_callbacks cb 0x7f6870009e60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:09:35.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.839+0000 7f6889400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640c260 0x56327641d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.839+0000 7f6883e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56327640cc50 0x56327641a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.839+0000 7f6889400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563276411dc0 0x56327641a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.839+0000 7f6883e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56327640cc50 0x56327641a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:34348/0 (socket says 172.21.15.28:34348) 2024-04-24T08:09:35.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.839+0000 7f6883e00640 1 -- 172.21.15.28:0/2351487522 learned_addr learned my addr 172.21.15.28:0/2351487522 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:35.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.840+0000 7f6883e00640 1 -- 172.21.15.28:0/2351487522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640c260 msgr2=0x56327641d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:35.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.840+0000 7f6883e00640 1 --2- 172.21.15.28:0/2351487522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640c260 0x56327641d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.840+0000 7f6883e00640 1 -- 172.21.15.28:0/2351487522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563276411dc0 msgr2=0x56327641a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:35.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.840+0000 7f6883e00640 1 --2- 172.21.15.28:0/2351487522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563276411dc0 0x56327641a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.840+0000 7f6883e00640 1 -- 172.21.15.28:0/2351487522 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56327640c090 con 0x56327640cc50 2024-04-24T08:09:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.840+0000 7f6883e00640 1 --2- 172.21.15.28:0/2351487522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56327640cc50 0x56327641a130 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6870010aa0 tx=0x7f687000f9c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=40af515ae9501c9b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.840+0000 7f686a000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:09:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.840+0000 7f686a000640 1 -- 172.21.15.28:0/2351487522 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6870010ba0 con 0x56327640cc50 2024-04-24T08:09:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.840+0000 7f686a000640 1 -- 172.21.15.28:0/2351487522 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f68700050d0 con 0x56327640cc50 2024-04-24T08:09:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.840+0000 7f688e381740 1 -- 172.21.15.28:0/2351487522 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6870010d40 con 0x56327640cc50 2024-04-24T08:09:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.841+0000 7f686a000640 1 -- 172.21.15.28:0/2351487522 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f68700052d0 con 0x56327640cc50 2024-04-24T08:09:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.841+0000 7f688e381740 10 client.25009 fetch_fsmap learned FSMap version 23 2024-04-24T08:09:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.841+0000 7f688e381740 1 -- 172.21.15.28:0/2351487522 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x5632764929e0 con 0x56327640cc50 2024-04-24T08:09:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.841+0000 7f686a000640 1 -- 172.21.15.28:0/2351487522 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6870010d40 con 0x56327640cc50 2024-04-24T08:09:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.841+0000 7f686a000640 1 -- 172.21.15.28:0/2351487522 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f687001c500 con 0x56327640cc50 2024-04-24T08:09:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.841+0000 7f688e381740 10 client.25009 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:09:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.841+0000 7f688e381740 10 client.25009 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-24T08:09:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.841+0000 7f688e381740 20 client.25009 populate_metadata read hostname 'smithi028' 2024-04-24T08:09:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.841+0000 7f688e381740 1 --2- 172.21.15.28:0/2351487522 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563276497100 0x563276493c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:35.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.841+0000 7f688e381740 4 client.25009 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:09:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.841+0000 7f688e381740 1 -- 172.21.15.28:0/2351487522 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "status"}) -- 0x5632763793a0 con 0x563276497100 2024-04-24T08:09:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.842+0000 7f6889400640 1 --2- 172.21.15.28:0/2351487522 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563276497100 0x563276493c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.842+0000 7f6889400640 1 --2- 172.21.15.28:0/2351487522 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563276497100 0x563276493c10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2c0bf14757ec05a6 server_cookie=56d1a84661b7d5f in_seq=0 out_seq=0 2024-04-24T08:09:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.842+0000 7f686a000640 10 client.25009 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:09:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.842+0000 7f686a000640 1 -- 172.21.15.28:0/2351487522 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5632763793a0 con 0x563276497100 2024-04-24T08:09:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.842+0000 7f686a000640 10 client.25009 handle_command_reply: tid=0 2024-04-24T08:09:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.842+0000 7f688e381740 1 client.25009 shutdown 2024-04-24T08:09:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.843+0000 7f688e381740 1 -- 172.21.15.28:0/2351487522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56327640cc50 msgr2=0x56327641a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.843+0000 7f688e381740 1 --2- 172.21.15.28:0/2351487522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56327640cc50 0x56327641a130 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6870010aa0 tx=0x7f687000f9c0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.843+0000 7f688e381740 1 -- 172.21.15.28:0/2351487522 shutdown_connections 2024-04-24T08:09:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.843+0000 7f688e381740 1 --2- 172.21.15.28:0/2351487522 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563276497100 0x563276493c10 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.843+0000 7f688e381740 1 --2- 172.21.15.28:0/2351487522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563276411dc0 0x56327641a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.843+0000 7f688e381740 1 --2- 172.21.15.28:0/2351487522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56327640cc50 0x56327641a130 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.843+0000 7f688e381740 1 --2- 172.21.15.28:0/2351487522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56327640c260 0x56327641d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.843+0000 7f686a000640 0 client.25009 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:09:35.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.843+0000 7f688e381740 1 -- 172.21.15.28:0/2351487522 >> 172.21.15.28:0/2351487522 conn(0x5632764078d0 msgr2=0x563276416120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:35.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.843+0000 7f688e381740 1 -- 172.21.15.28:0/2351487522 shutdown_connections 2024-04-24T08:09:35.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.843+0000 7f688e381740 1 -- 172.21.15.28:0/2351487522 wait complete. 2024-04-24T08:09:35.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.843+0000 7f688e381740 20 client.25009 trim_cache size 0 max 16384 2024-04-24T08:09:35.847 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:09:35.847 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "473aaf58-0210-11ef-bc93-c7b262605968", 2024-04-24T08:09:35.847 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-04-24T08:09:35.847 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24556, 2024-04-24T08:09:35.847 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-24T08:09:35.848 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-24T08:09:35.848 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-24T08:09:35.848 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 99.717260211999999, 2024-04-24T08:09:35.848 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 23, 2024-04-24T08:09:35.848 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 88, 2024-04-24T08:09:35.848 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 86, 2024-04-24T08:09:35.848 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 121.318453097 2024-04-24T08:09:35.848 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.844+0000 7f6889e00640 1 -- 172.21.15.28:0/3888158277 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f686c0a3bb0 msgr2=0x7f686c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.844+0000 7f6889e00640 1 --2- 172.21.15.28:0/3888158277 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f686c0a3bb0 0x7f686c0a6070 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f68780cd800 tx=0x7f68780d37c0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.844+0000 7f6889e00640 1 -- 172.21.15.28:0/3888158277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68840684b0 msgr2=0x7f688410bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.844+0000 7f6889e00640 1 --2- 172.21.15.28:0/3888158277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68840684b0 0x7f688410bcc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f68841011a0 tx=0x7f687800dbb0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.844+0000 7f6889e00640 1 -- 172.21.15.28:0/3888158277 shutdown_connections 2024-04-24T08:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.844+0000 7f6889e00640 1 --2- 172.21.15.28:0/3888158277 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f686c0a3bb0 0x7f686c0a6070 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.844+0000 7f6889e00640 1 --2- 172.21.15.28:0/3888158277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6884106580 0x7f688410a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.844+0000 7f6889e00640 1 --2- 172.21.15.28:0/3888158277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6884101880 0x7f688410a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.844+0000 7f6889e00640 1 --2- 172.21.15.28:0/3888158277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f68840684b0 0x7f688410bcc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.844+0000 7f6889e00640 1 -- 172.21.15.28:0/3888158277 >> 172.21.15.28:0/3888158277 conn(0x7f68840fb6c0 msgr2=0x7f6884102560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.844+0000 7f6889e00640 1 -- 172.21.15.28:0/3888158277 shutdown_connections 2024-04-24T08:09:35.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:35.844+0000 7f6889e00640 1 -- 172.21.15.28:0/3888158277 wait complete. 2024-04-24T08:09:35.862 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-24T08:09:36.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.067+0000 7fe1fb200640 1 -- 172.21.15.28:0/1277820430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1fc101f10 msgr2=0x7fe1fc102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.067+0000 7fe1fb200640 1 --2- 172.21.15.28:0/1277820430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1fc101f10 0x7fe1fc102330 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009900 tx=0x7fe1ec02a410 comp rx=0 tx=0).stop 2024-04-24T08:09:36.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.067+0000 7fe1fb200640 1 -- 172.21.15.28:0/1277820430 shutdown_connections 2024-04-24T08:09:36.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.067+0000 7fe1fb200640 1 --2- 172.21.15.28:0/1277820430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1fc103a20 0x7fe1fc10a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.067+0000 7fe1fb200640 1 --2- 172.21.15.28:0/1277820430 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1fc103060 0x7fe1fc1034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.067+0000 7fe1fb200640 1 --2- 172.21.15.28:0/1277820430 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1fc101f10 0x7fe1fc102330 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.067+0000 7fe1fb200640 1 -- 172.21.15.28:0/1277820430 >> 172.21.15.28:0/1277820430 conn(0x7fe1fc0fd4a0 msgr2=0x7fe1fc0ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:36.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.067+0000 7fe1fb200640 1 -- 172.21.15.28:0/1277820430 shutdown_connections 2024-04-24T08:09:36.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.067+0000 7fe1fb200640 1 -- 172.21.15.28:0/1277820430 wait complete. 2024-04-24T08:09:36.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.068+0000 7fe1fb200640 1 Processor -- start 2024-04-24T08:09:36.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.068+0000 7fe1fb200640 1 -- start start 2024-04-24T08:09:36.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1fb200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1fc101f10 0x7fe1fc19a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1fb200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1fc103060 0x7fe1fc19a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1fb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1fc103a20 0x7fe1fc1a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1fb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe1fc10dbf0 con 0x7fe1fc103a20 2024-04-24T08:09:36.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1fb200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe1fc10da70 con 0x7fe1fc101f10 2024-04-24T08:09:36.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1fb200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe1fc10dd70 con 0x7fe1fc103060 2024-04-24T08:09:36.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1fa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1fc103a20 0x7fe1fc1a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1f9e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1fc101f10 0x7fe1fc19a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1f9400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1fc103060 0x7fe1fc19a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1fa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1fc103a20 0x7fe1fc1a1a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59894/0 (socket says 172.21.15.28:59894) 2024-04-24T08:09:36.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1fa800640 1 -- 172.21.15.28:0/2446137189 learned_addr learned my addr 172.21.15.28:0/2446137189 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:36.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1f9e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1fc101f10 0x7fe1fc19a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:55582/0 (socket says 172.21.15.28:55582) 2024-04-24T08:09:36.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1fa800640 1 -- 172.21.15.28:0/2446137189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1fc103060 msgr2=0x7fe1fc19a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1fa800640 1 --2- 172.21.15.28:0/2446137189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1fc103060 0x7fe1fc19a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1fa800640 1 -- 172.21.15.28:0/2446137189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1fc101f10 msgr2=0x7fe1fc19a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1fa800640 1 --2- 172.21.15.28:0/2446137189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1fc101f10 0x7fe1fc19a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1fa800640 1 -- 172.21.15.28:0/2446137189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1fc1a2070 con 0x7fe1fc103a20 2024-04-24T08:09:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.069+0000 7fe1f9e00640 1 --2- 172.21.15.28:0/2446137189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1fc101f10 0x7fe1fc19a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.070+0000 7fe1fa800640 1 --2- 172.21.15.28:0/2446137189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1fc103a20 0x7fe1fc1a1a00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400efd0 tx=0x7fe1f400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.070+0000 7fe1f2a00640 1 -- 172.21.15.28:0/2446137189 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f4003b70 con 0x7fe1fc103a20 2024-04-24T08:09:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.070+0000 7fe1f2a00640 1 -- 172.21.15.28:0/2446137189 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1f4003d10 con 0x7fe1fc103a20 2024-04-24T08:09:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.070+0000 7fe1fb200640 1 -- 172.21.15.28:0/2446137189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1fc1a2360 con 0x7fe1fc103a20 2024-04-24T08:09:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.070+0000 7fe1f2a00640 1 -- 172.21.15.28:0/2446137189 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f40109f0 con 0x7fe1fc103a20 2024-04-24T08:09:36.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.070+0000 7fe1fb200640 1 -- 172.21.15.28:0/2446137189 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1fc1a2860 con 0x7fe1fc103a20 2024-04-24T08:09:36.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.073+0000 7fe203bfc740 1 Processor -- start 2024-04-24T08:09:36.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.073+0000 7fe1f2a00640 1 -- 172.21.15.28:0/2446137189 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fe1f4004020 con 0x7fe1fc103a20 2024-04-24T08:09:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.074+0000 7fe203bfc740 1 -- start start 2024-04-24T08:09:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.074+0000 7fe203bfc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d99260 0x55ca58d99680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.074+0000 7fe203bfc740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99c50 0x55ca58d9e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.074+0000 7fe203bfc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d9edc0 0x55ca58da1250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.074+0000 7fe203bfc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ca58cfeb10 con 0x55ca58d99260 2024-04-24T08:09:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.074+0000 7fe203bfc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ca58ca52b0 con 0x55ca58d9edc0 2024-04-24T08:09:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.074+0000 7fe203bfc740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ca58cfaad0 con 0x55ca58d99c50 2024-04-24T08:09:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.074+0000 7fe1f9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99c50 0x55ca58d9e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.074+0000 7fe1f9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d99260 0x55ca58d99680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.074+0000 7fe1f9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99c50 0x55ca58d9e750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:34366/0 (socket says 172.21.15.28:34366) 2024-04-24T08:09:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.074+0000 7fe1f9e00640 1 -- 172.21.15.28:0/3371765367 learned_addr learned my addr 172.21.15.28:0/3371765367 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:36.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.075+0000 7fe1f9e00640 1 -- 172.21.15.28:0/3371765367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d9edc0 msgr2=0x55ca58da1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:36.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.075+0000 7fe1f9e00640 1 --2- 172.21.15.28:0/3371765367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d9edc0 0x55ca58da1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.075+0000 7fe1f9e00640 1 -- 172.21.15.28:0/3371765367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d99260 msgr2=0x55ca58d99680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:36.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.075+0000 7fe1f9e00640 1 --2- 172.21.15.28:0/3371765367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d99260 0x55ca58d99680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.075+0000 7fe1f9e00640 1 -- 172.21.15.28:0/3371765367 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca58a112d0 con 0x55ca58d99c50 2024-04-24T08:09:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.075+0000 7fe1f9e00640 1 --2- 172.21.15.28:0/3371765367 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99c50 0x55ca58d9e750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009900 tx=0x7fe1ec0043f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf37cdcb547370a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.075+0000 7fe1f2a00640 1 --2- 172.21.15.28:0/2446137189 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe1e00a3bb0 0x7fe1e00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.075+0000 7fe1e7400640 1 -- 172.21.15.28:0/3371765367 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec035040 con 0x55ca58d99c50 2024-04-24T08:09:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.075+0000 7fe1e7400640 1 -- 172.21.15.28:0/3371765367 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1ec004770 con 0x55ca58d99c50 2024-04-24T08:09:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.075+0000 7fe1e7400640 1 -- 172.21.15.28:0/3371765367 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec031a30 con 0x55ca58d99c50 2024-04-24T08:09:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.076+0000 7fe1f2a00640 1 -- 172.21.15.28:0/2446137189 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe1f40d08d0 con 0x7fe1fc103a20 2024-04-24T08:09:36.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.076+0000 7fe1f9400640 1 --2- 172.21.15.28:0/2446137189 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe1e00a3bb0 0x7fe1e00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.076+0000 7fe203bfc740 1 -- 172.21.15.28:0/3371765367 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99c50 msgr2=0x55ca58d9e750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.076+0000 7fe203bfc740 1 --2- 172.21.15.28:0/3371765367 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99c50 0x55ca58d9e750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009900 tx=0x7fe1ec0043f0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.076+0000 7fe1f9400640 1 --2- 172.21.15.28:0/2446137189 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe1e00a3bb0 0x7fe1e00a6070 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x55ca58d9a4b0 tx=0x7fe1e8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.076+0000 7fe203bfc740 1 -- 172.21.15.28:0/3371765367 shutdown_connections 2024-04-24T08:09:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.076+0000 7fe203bfc740 1 --2- 172.21.15.28:0/3371765367 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d9edc0 0x55ca58da1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.076+0000 7fe203bfc740 1 --2- 172.21.15.28:0/3371765367 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99c50 0x55ca58d9e750 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.076+0000 7fe203bfc740 1 --2- 172.21.15.28:0/3371765367 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d99260 0x55ca58d99680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.076+0000 7fe203bfc740 1 -- 172.21.15.28:0/3371765367 >> 172.21.15.28:0/3371765367 conn(0x55ca58d948d0 msgr2=0x55ca58d94cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.076+0000 7fe203bfc740 1 -- 172.21.15.28:0/3371765367 shutdown_connections 2024-04-24T08:09:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.076+0000 7fe203bfc740 1 -- 172.21.15.28:0/3371765367 wait complete. 2024-04-24T08:09:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.077+0000 7fe203bfc740 1 -- start start 2024-04-24T08:09:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.077+0000 7fe203bfc740 1 Processor -- start 2024-04-24T08:09:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.077+0000 7fe203bfc740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 0x55ca58da3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.078+0000 7fe203bfc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d99c50 0x55ca58da3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.078+0000 7fe203bfc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 0x55ca58daabd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.078+0000 7fe203bfc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ca58da1d10 con 0x55ca58d9edc0 2024-04-24T08:09:36.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.078+0000 7fe203bfc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ca58cb8590 con 0x55ca58d99c50 2024-04-24T08:09:36.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.078+0000 7fe203bfc740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ca58da1e90 con 0x55ca58d99260 2024-04-24T08:09:36.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.078+0000 7fe203bfc740 10 client.0 _ll_register_callbacks cb 0x55ca58c9b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:09:36.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.078+0000 7fe1f9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 0x55ca58da3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.078+0000 7fe1fa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 0x55ca58daabd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.078+0000 7fe1f9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 0x55ca58da3650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:34376/0 (socket says 172.21.15.28:34376) 2024-04-24T08:09:36.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.078+0000 7fe1f9e00640 1 -- 172.21.15.28:0/2119306121 learned_addr learned my addr 172.21.15.28:0/2119306121 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:36.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.078+0000 7fe1f9400640 1 --2- 172.21.15.28:0/2119306121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d99c50 0x55ca58da3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.078+0000 7fe1fa800640 1 -- 172.21.15.28:0/2119306121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 msgr2=0x55ca58da3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:36.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.078+0000 7fe1fa800640 1 --2- 172.21.15.28:0/2119306121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 0x55ca58da3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.078+0000 7fe1fa800640 1 -- 172.21.15.28:0/2119306121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d99c50 msgr2=0x55ca58da3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:36.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.078+0000 7fe1fa800640 1 --2- 172.21.15.28:0/2119306121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d99c50 0x55ca58da3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.078+0000 7fe1fa800640 1 -- 172.21.15.28:0/2119306121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca58adb6b0 con 0x55ca58d9edc0 2024-04-24T08:09:36.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.079+0000 7fe1fa800640 1 --2- 172.21.15.28:0/2119306121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 0x55ca58daabd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400cba0 tx=0x7fe1f409c550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7207832dce4e3282 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:36.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.079+0000 7fe1dbe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:09:36.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.079+0000 7fe1dbe00640 1 -- 172.21.15.28:0/2119306121 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f40b6060 con 0x55ca58d9edc0 2024-04-24T08:09:36.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.079+0000 7fe203bfc740 1 -- 172.21.15.28:0/2119306121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1ec004770 con 0x55ca58d9edc0 2024-04-24T08:09:36.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.079+0000 7fe1dbe00640 1 -- 172.21.15.28:0/2119306121 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe1f4004620 con 0x55ca58d9edc0 2024-04-24T08:09:36.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.079+0000 7fe1dbe00640 1 -- 172.21.15.28:0/2119306121 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe1f409de60 con 0x55ca58d9edc0 2024-04-24T08:09:36.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.079+0000 7fe203bfc740 10 client.14898 fetch_fsmap learned FSMap version 23 2024-04-24T08:09:36.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.079+0000 7fe203bfc740 1 -- 172.21.15.28:0/2119306121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ca58dae060 con 0x55ca58d9edc0 2024-04-24T08:09:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.079+0000 7fe1dbe00640 1 -- 172.21.15.28:0/2119306121 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1ec004770 con 0x55ca58d9edc0 2024-04-24T08:09:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.080+0000 7fe1dbe00640 1 -- 172.21.15.28:0/2119306121 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe1f401f020 con 0x55ca58d9edc0 2024-04-24T08:09:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.080+0000 7fe203bfc740 10 client.14898 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:09:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.080+0000 7fe203bfc740 10 client.14898 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T08:09:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.080+0000 7fe203bfc740 20 client.14898 populate_metadata read hostname 'smithi028' 2024-04-24T08:09:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.080+0000 7fe203bfc740 1 --2- 172.21.15.28:0/2119306121 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55ca58db06a0 0x55ca58db1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.080+0000 7fe203bfc740 4 client.14898 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:09:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.080+0000 7fe203bfc740 1 -- 172.21.15.28:0/2119306121 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ca58d0c530 con 0x55ca58db06a0 2024-04-24T08:09:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.080+0000 7fe1f9e00640 1 --2- 172.21.15.28:0/2119306121 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55ca58db06a0 0x55ca58db1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.081+0000 7fe1f9e00640 1 --2- 172.21.15.28:0/2119306121 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55ca58db06a0 0x55ca58db1b10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d796ef8e8e3a9852 server_cookie=5374fbee7cbd3aa5 in_seq=0 out_seq=0 2024-04-24T08:09:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.081+0000 7fe1dbe00640 10 client.14898 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:09:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.082+0000 7fe1dbe00640 1 -- 172.21.15.28:0/2119306121 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ca58d0c530 con 0x55ca58db06a0 2024-04-24T08:09:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.082+0000 7fe1dbe00640 10 client.14898 handle_command_reply: tid=0 2024-04-24T08:09:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.082+0000 7fe203bfc740 1 client.14898 shutdown 2024-04-24T08:09:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.082+0000 7fe203bfc740 1 -- 172.21.15.28:0/2119306121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 msgr2=0x55ca58daabd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.082+0000 7fe203bfc740 1 --2- 172.21.15.28:0/2119306121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 0x55ca58daabd0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400cba0 tx=0x7fe1f409c550 comp rx=0 tx=0).stop 2024-04-24T08:09:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.082+0000 7fe203bfc740 1 -- 172.21.15.28:0/2119306121 shutdown_connections 2024-04-24T08:09:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.082+0000 7fe203bfc740 1 --2- 172.21.15.28:0/2119306121 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55ca58db06a0 0x55ca58db1b10 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.082+0000 7fe203bfc740 1 --2- 172.21.15.28:0/2119306121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 0x55ca58daabd0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.082+0000 7fe203bfc740 1 --2- 172.21.15.28:0/2119306121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d99c50 0x55ca58da3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.082+0000 7fe203bfc740 1 --2- 172.21.15.28:0/2119306121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 0x55ca58da3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.082+0000 7fe1dbe00640 0 client.14898 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:09:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.082+0000 7fe203bfc740 1 -- 172.21.15.28:0/2119306121 >> 172.21.15.28:0/2119306121 conn(0x55ca58d948d0 msgr2=0x55ca58d93970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.082+0000 7fe203bfc740 1 -- 172.21.15.28:0/2119306121 shutdown_connections 2024-04-24T08:09:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.082+0000 7fe203bfc740 1 -- 172.21.15.28:0/2119306121 wait complete. 2024-04-24T08:09:36.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.083+0000 7fe203bfc740 20 client.14898 trim_cache size 0 max 16384 2024-04-24T08:09:36.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.103+0000 7fe203bfc740 1 Processor -- start 2024-04-24T08:09:36.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.104+0000 7fe203bfc740 1 -- start start 2024-04-24T08:09:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.104+0000 7fe203bfc740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 0x55ca58d9e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.104+0000 7fe203bfc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d99c50 0x55ca58daab90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.104+0000 7fe203bfc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 0x55ca58da2290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.104+0000 7fe203bfc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ca58dafe50 con 0x55ca58d9edc0 2024-04-24T08:09:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.104+0000 7fe203bfc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ca58db0520 con 0x55ca58d99c50 2024-04-24T08:09:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.104+0000 7fe203bfc740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ca58db0bf0 con 0x55ca58d99260 2024-04-24T08:09:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.104+0000 7fe1f9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 0x55ca58d9e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.104+0000 7fe1f9400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d99c50 0x55ca58daab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.104+0000 7fe1f9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 0x55ca58d9e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:34380/0 (socket says 172.21.15.28:34380) 2024-04-24T08:09:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.104+0000 7fe1f9e00640 1 -- 172.21.15.28:0/3886164934 learned_addr learned my addr 172.21.15.28:0/3886164934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.105+0000 7fe1fa800640 1 --2- 172.21.15.28:0/3886164934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 0x55ca58da2290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.105+0000 7fe1f9e00640 1 -- 172.21.15.28:0/3886164934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d99c50 msgr2=0x55ca58daab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.105+0000 7fe1f9e00640 1 --2- 172.21.15.28:0/3886164934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d99c50 0x55ca58daab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.105+0000 7fe1f9e00640 1 -- 172.21.15.28:0/3886164934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 msgr2=0x55ca58da2290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.105+0000 7fe1f9e00640 1 --2- 172.21.15.28:0/3886164934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 0x55ca58da2290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.105+0000 7fe1f9e00640 1 -- 172.21.15.28:0/3886164934 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ca58d9a030 con 0x55ca58d99260 2024-04-24T08:09:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.105+0000 7fe1f9e00640 1 --2- 172.21.15.28:0/3886164934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 0x55ca58d9e680 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec004650 tx=0x7fe1ec0045e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a49856308826718b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.105+0000 7fe1e6a00640 1 -- 172.21.15.28:0/3886164934 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec035040 con 0x55ca58d99260 2024-04-24T08:09:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.106+0000 7fe1e6a00640 1 -- 172.21.15.28:0/3886164934 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1ec031d90 con 0x55ca58d99260 2024-04-24T08:09:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.106+0000 7fe1e6a00640 1 -- 172.21.15.28:0/3886164934 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1ec0035f0 con 0x55ca58d99260 2024-04-24T08:09:36.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.106+0000 7fe203bfc740 1 -- 172.21.15.28:0/3886164934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 msgr2=0x55ca58d9e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.106+0000 7fe203bfc740 1 --2- 172.21.15.28:0/3886164934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 0x55ca58d9e680 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec004650 tx=0x7fe1ec0045e0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.106+0000 7fe203bfc740 1 -- 172.21.15.28:0/3886164934 shutdown_connections 2024-04-24T08:09:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.106+0000 7fe203bfc740 1 --2- 172.21.15.28:0/3886164934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 0x55ca58da2290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.106+0000 7fe203bfc740 1 --2- 172.21.15.28:0/3886164934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d99c50 0x55ca58daab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.106+0000 7fe203bfc740 1 --2- 172.21.15.28:0/3886164934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 0x55ca58d9e680 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.106+0000 7fe203bfc740 1 -- 172.21.15.28:0/3886164934 >> 172.21.15.28:0/3886164934 conn(0x55ca58d948d0 msgr2=0x55ca58da2c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.107+0000 7fe203bfc740 1 -- 172.21.15.28:0/3886164934 shutdown_connections 2024-04-24T08:09:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.107+0000 7fe203bfc740 1 -- 172.21.15.28:0/3886164934 wait complete. 2024-04-24T08:09:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.107+0000 7fe203bfc740 1 -- start start 2024-04-24T08:09:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.107+0000 7fe203bfc740 1 Processor -- start 2024-04-24T08:09:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.107+0000 7fe203bfc740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 0x55ca58da5970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.108+0000 7fe203bfc740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d99c50 0x55ca58da5eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.108+0000 7fe203bfc740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 0x55ca58e14640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.108+0000 7fe203bfc740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ca58db1990 con 0x55ca58d9edc0 2024-04-24T08:09:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.108+0000 7fe203bfc740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ca58db12c0 con 0x55ca58d99c50 2024-04-24T08:09:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.108+0000 7fe203bfc740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ca58db2060 con 0x55ca58d99260 2024-04-24T08:09:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.108+0000 7fe203bfc740 10 client.0 _ll_register_callbacks cb 0x7fe1f409c670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:09:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.108+0000 7fe1fa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 0x55ca58e14640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.108+0000 7fe1fa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 0x55ca58e14640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59946/0 (socket says 172.21.15.28:59946) 2024-04-24T08:09:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.108+0000 7fe1fa800640 1 -- 172.21.15.28:0/3798625373 learned_addr learned my addr 172.21.15.28:0/3798625373 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.108+0000 7fe1f9e00640 1 --2- 172.21.15.28:0/3798625373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 0x55ca58da5970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.108+0000 7fe1f9400640 1 --2- 172.21.15.28:0/3798625373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d99c50 0x55ca58da5eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.108+0000 7fe1fa800640 1 -- 172.21.15.28:0/3798625373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 msgr2=0x55ca58da5970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.108+0000 7fe1fa800640 1 --2- 172.21.15.28:0/3798625373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 0x55ca58da5970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.108+0000 7fe1fa800640 1 -- 172.21.15.28:0/3798625373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d99c50 msgr2=0x55ca58da5eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:36.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.108+0000 7fe1fa800640 1 --2- 172.21.15.28:0/3798625373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d99c50 0x55ca58da5eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.108+0000 7fe1fa800640 1 -- 172.21.15.28:0/3798625373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ca58e14b80 con 0x55ca58d9edc0 2024-04-24T08:09:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.109+0000 7fe1fa800640 1 --2- 172.21.15.28:0/3798625373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 0x55ca58e14640 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe1f409c830 tx=0x7fe1f409c5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=988c6daf74baeb40 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.109+0000 7fe1dbe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:09:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.109+0000 7fe1dbe00640 1 -- 172.21.15.28:0/3798625373 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1f409c180 con 0x55ca58d9edc0 2024-04-24T08:09:36.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.109+0000 7fe1dbe00640 1 -- 172.21.15.28:0/3798625373 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fe1f4098880 con 0x55ca58d9edc0 2024-04-24T08:09:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.109+0000 7fe203bfc740 1 -- 172.21.15.28:0/3798625373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe1ec031d90 con 0x55ca58d9edc0 2024-04-24T08:09:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.109+0000 7fe1dbe00640 1 -- 172.21.15.28:0/3798625373 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fe1f4010b90 con 0x55ca58d9edc0 2024-04-24T08:09:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.109+0000 7fe203bfc740 10 client.14904 fetch_fsmap learned FSMap version 23 2024-04-24T08:09:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.109+0000 7fe203bfc740 1 -- 172.21.15.28:0/3798625373 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ca58e15db0 con 0x55ca58d9edc0 2024-04-24T08:09:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.109+0000 7fe1dbe00640 1 -- 172.21.15.28:0/3798625373 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1ec031d90 con 0x55ca58d9edc0 2024-04-24T08:09:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.109+0000 7fe1dbe00640 1 -- 172.21.15.28:0/3798625373 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fe1f401f070 con 0x55ca58d9edc0 2024-04-24T08:09:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.109+0000 7fe203bfc740 10 client.14904 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:09:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.109+0000 7fe203bfc740 10 client.14904 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T08:09:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.109+0000 7fe203bfc740 20 client.14904 populate_metadata read hostname 'smithi028' 2024-04-24T08:09:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.110+0000 7fe203bfc740 1 --2- 172.21.15.28:0/3798625373 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55ca58e1a790 0x55ca58e194a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.110+0000 7fe203bfc740 4 client.14904 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:09:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.110+0000 7fe203bfc740 1 -- 172.21.15.28:0/3798625373 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "status"}) -- 0x55ca58d063a0 con 0x55ca58e1a790 2024-04-24T08:09:36.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.110+0000 7fe1f9e00640 1 --2- 172.21.15.28:0/3798625373 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55ca58e1a790 0x55ca58e194a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.110+0000 7fe1f9e00640 1 --2- 172.21.15.28:0/3798625373 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55ca58e1a790 0x55ca58e194a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=37b13156c5788493 server_cookie=1f6a23c364fbc2e3 in_seq=0 out_seq=0 2024-04-24T08:09:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.110+0000 7fe1dbe00640 10 client.14904 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:09:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.111+0000 7fe1dbe00640 1 -- 172.21.15.28:0/3798625373 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55ca58d063a0 con 0x55ca58e1a790 2024-04-24T08:09:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.111+0000 7fe1dbe00640 10 client.14904 handle_command_reply: tid=0 2024-04-24T08:09:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.111+0000 7fe203bfc740 1 client.14904 shutdown 2024-04-24T08:09:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.111+0000 7fe203bfc740 1 -- 172.21.15.28:0/3798625373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 msgr2=0x55ca58e14640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.111+0000 7fe203bfc740 1 --2- 172.21.15.28:0/3798625373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 0x55ca58e14640 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe1f409c830 tx=0x7fe1f409c5c0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.111+0000 7fe203bfc740 1 -- 172.21.15.28:0/3798625373 shutdown_connections 2024-04-24T08:09:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.111+0000 7fe203bfc740 1 --2- 172.21.15.28:0/3798625373 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55ca58e1a790 0x55ca58e194a0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.111+0000 7fe203bfc740 1 --2- 172.21.15.28:0/3798625373 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ca58d9edc0 0x55ca58e14640 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.111+0000 7fe203bfc740 1 --2- 172.21.15.28:0/3798625373 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ca58d99c50 0x55ca58da5eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.111+0000 7fe203bfc740 1 --2- 172.21.15.28:0/3798625373 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ca58d99260 0x55ca58da5970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.111+0000 7fe1dbe00640 0 client.14904 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:09:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.111+0000 7fe203bfc740 1 -- 172.21.15.28:0/3798625373 >> 172.21.15.28:0/3798625373 conn(0x55ca58d948d0 msgr2=0x55ca58da2290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.111+0000 7fe203bfc740 1 -- 172.21.15.28:0/3798625373 shutdown_connections 2024-04-24T08:09:36.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.111+0000 7fe203bfc740 1 -- 172.21.15.28:0/3798625373 wait complete. 2024-04-24T08:09:36.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.111+0000 7fe203bfc740 20 client.14904 trim_cache size 0 max 16384 2024-04-24T08:09:36.116 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:09:36.116 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "473aaf58-0210-11ef-bc93-c7b262605968", 2024-04-24T08:09:36.116 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-04-24T08:09:36.116 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24572, 2024-04-24T08:09:36.116 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-24T08:09:36.116 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-24T08:09:36.116 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-24T08:09:36.117 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 101.005297852, 2024-04-24T08:09:36.117 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 23, 2024-04-24T08:09:36.117 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 88, 2024-04-24T08:09:36.117 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 86, 2024-04-24T08:09:36.117 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 110.498709006 2024-04-24T08:09:36.117 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:09:36.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.112+0000 7fe1fb200640 1 -- 172.21.15.28:0/2446137189 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe1e00a3bb0 msgr2=0x7fe1e00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.112+0000 7fe1fb200640 1 --2- 172.21.15.28:0/2446137189 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe1e00a3bb0 0x7fe1e00a6070 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x55ca58d9a4b0 tx=0x7fe1e8005850 comp rx=0 tx=0).stop 2024-04-24T08:09:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.112+0000 7fe1fb200640 1 -- 172.21.15.28:0/2446137189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1fc103a20 msgr2=0x7fe1fc1a1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.112+0000 7fe1fb200640 1 --2- 172.21.15.28:0/2446137189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1fc103a20 0x7fe1fc1a1a00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fe1f400efd0 tx=0x7fe1f400c6a0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.112+0000 7fe1fb200640 1 -- 172.21.15.28:0/2446137189 shutdown_connections 2024-04-24T08:09:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.112+0000 7fe1fb200640 1 --2- 172.21.15.28:0/2446137189 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fe1e00a3bb0 0x7fe1e00a6070 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.112+0000 7fe1fb200640 1 --2- 172.21.15.28:0/2446137189 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe1fc103a20 0x7fe1fc1a1a00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.112+0000 7fe1fb200640 1 --2- 172.21.15.28:0/2446137189 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe1fc103060 0x7fe1fc19a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.112+0000 7fe1fb200640 1 --2- 172.21.15.28:0/2446137189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1fc101f10 0x7fe1fc19a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.112+0000 7fe1fb200640 1 -- 172.21.15.28:0/2446137189 >> 172.21.15.28:0/2446137189 conn(0x7fe1fc0fd4a0 msgr2=0x7fe1fc074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.112+0000 7fe1fb200640 1 -- 172.21.15.28:0/2446137189 shutdown_connections 2024-04-24T08:09:36.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.112+0000 7fe1fb200640 1 -- 172.21.15.28:0/2446137189 wait complete. 2024-04-24T08:09:36.132 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-24T08:09:36.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.346+0000 7fb0e5e00640 1 -- 172.21.15.28:0/2172997383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0e0109dd0 msgr2=0x7fb0e010c260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.346+0000 7fb0e5e00640 1 --2- 172.21.15.28:0/2172997383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0e0109dd0 0x7fb0e010c260 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009ca0 tx=0x7fb0d802a8c0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.346+0000 7fb0e5e00640 1 -- 172.21.15.28:0/2172997383 shutdown_connections 2024-04-24T08:09:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.346+0000 7fb0e5e00640 1 --2- 172.21.15.28:0/2172997383 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0e0109dd0 0x7fb0e010c260 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.346+0000 7fb0e5e00640 1 --2- 172.21.15.28:0/2172997383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e0074a60 0x7fb0e0074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.346+0000 7fb0e5e00640 1 --2- 172.21.15.28:0/2172997383 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e0076040 0x7fb0e0074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.346+0000 7fb0e5e00640 1 -- 172.21.15.28:0/2172997383 >> 172.21.15.28:0/2172997383 conn(0x7fb0e00fd4a0 msgr2=0x7fb0e00ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.346+0000 7fb0e5e00640 1 -- 172.21.15.28:0/2172997383 shutdown_connections 2024-04-24T08:09:36.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.346+0000 7fb0e5e00640 1 -- 172.21.15.28:0/2172997383 wait complete. 2024-04-24T08:09:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.347+0000 7fb0e5e00640 1 Processor -- start 2024-04-24T08:09:36.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.348+0000 7fb0e5e00640 1 -- start start 2024-04-24T08:09:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.348+0000 7fb0e5e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0e0074a60 0x7fb0e019a360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.348+0000 7fb0e5e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e0076040 0x7fb0e019a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.348+0000 7fb0e5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e0109dd0 0x7fb0e01a1910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.348+0000 7fb0e5e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e010fde0 con 0x7fb0e0109dd0 2024-04-24T08:09:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.348+0000 7fb0e5e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e010fc60 con 0x7fb0e0076040 2024-04-24T08:09:36.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.348+0000 7fb0e5e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb0e010ff60 con 0x7fb0e0074a60 2024-04-24T08:09:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.348+0000 7fb0e4a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0e0074a60 0x7fb0e019a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.348+0000 7fb0dfe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e0076040 0x7fb0e019a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.348+0000 7fb0dfe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e0076040 0x7fb0e019a8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:55642/0 (socket says 172.21.15.28:55642) 2024-04-24T08:09:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.348+0000 7fb0e4a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0e0074a60 0x7fb0e019a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:34400/0 (socket says 172.21.15.28:34400) 2024-04-24T08:09:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.348+0000 7fb0dfe00640 1 -- 172.21.15.28:0/2575838733 learned_addr learned my addr 172.21.15.28:0/2575838733 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.348+0000 7fb0e5400640 1 --2- 172.21.15.28:0/2575838733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e0109dd0 0x7fb0e01a1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.349+0000 7fb0e4a00640 1 -- 172.21.15.28:0/2575838733 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e0076040 msgr2=0x7fb0e019a8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.349+0000 7fb0e4a00640 1 --2- 172.21.15.28:0/2575838733 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e0076040 0x7fb0e019a8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.349+0000 7fb0e4a00640 1 -- 172.21.15.28:0/2575838733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e0109dd0 msgr2=0x7fb0e01a1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.349+0000 7fb0e4a00640 1 --2- 172.21.15.28:0/2575838733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e0109dd0 0x7fb0e01a1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.349+0000 7fb0e4a00640 1 -- 172.21.15.28:0/2575838733 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0e01a1f80 con 0x7fb0e0074a60 2024-04-24T08:09:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.349+0000 7fb0e5400640 1 --2- 172.21.15.28:0/2575838733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e0109dd0 0x7fb0e01a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:09:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.349+0000 7fb0dfe00640 1 --2- 172.21.15.28:0/2575838733 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e0076040 0x7fb0e019a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:36.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.349+0000 7fb0e4a00640 1 --2- 172.21.15.28:0/2575838733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0e0074a60 0x7fb0e019a360 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb0d400e9f0 tx=0x7fb0d400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:36.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.352+0000 7fb0dd600640 1 -- 172.21.15.28:0/2575838733 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d400ccc0 con 0x7fb0e0074a60 2024-04-24T08:09:36.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.352+0000 7fb0dd600640 1 -- 172.21.15.28:0/2575838733 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d4010910 con 0x7fb0e0074a60 2024-04-24T08:09:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.352+0000 7fb0dd600640 1 -- 172.21.15.28:0/2575838733 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d4018960 con 0x7fb0e0074a60 2024-04-24T08:09:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.352+0000 7fb0e5e00640 1 -- 172.21.15.28:0/2575838733 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0e01a2270 con 0x7fb0e0074a60 2024-04-24T08:09:36.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.352+0000 7fb0e5e00640 1 -- 172.21.15.28:0/2575838733 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0e01a2770 con 0x7fb0e0074a60 2024-04-24T08:09:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.356+0000 7fb0ea383740 1 Processor -- start 2024-04-24T08:09:36.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.356+0000 7fb0ea383740 1 -- start start 2024-04-24T08:09:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.356+0000 7fb0ea383740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbc260 0x55ddcddbc680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.356+0000 7fb0ea383740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbcc50 0x55ddcddc1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.357+0000 7fb0ea383740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddc1dc0 0x55ddcddc4250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.357+0000 7fb0ea383740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ddcdd21b10 con 0x55ddcddbc260 2024-04-24T08:09:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.357+0000 7fb0ea383740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ddcdcc82b0 con 0x55ddcddc1dc0 2024-04-24T08:09:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.357+0000 7fb0ea383740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ddcdd1dad0 con 0x55ddcddbcc50 2024-04-24T08:09:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.357+0000 7fb0e5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbc260 0x55ddcddbc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.357+0000 7fb0e5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbc260 0x55ddcddbc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59976/0 (socket says 172.21.15.28:59976) 2024-04-24T08:09:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.357+0000 7fb0e5400640 1 -- 172.21.15.28:0/2836619549 learned_addr learned my addr 172.21.15.28:0/2836619549 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:36.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.357+0000 7fb0dfe00640 1 --2- 172.21.15.28:0/2836619549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddc1dc0 0x55ddcddc4250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.357+0000 7fb0e5400640 1 -- 172.21.15.28:0/2836619549 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbcc50 msgr2=0x55ddcddc1750 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:09:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.357+0000 7fb0e5400640 1 --2- 172.21.15.28:0/2836619549 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbcc50 0x55ddcddc1750 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.357+0000 7fb0e5400640 1 -- 172.21.15.28:0/2836619549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddc1dc0 msgr2=0x55ddcddc4250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.357+0000 7fb0e5400640 1 --2- 172.21.15.28:0/2836619549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddc1dc0 0x55ddcddc4250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.357+0000 7fb0e5400640 1 -- 172.21.15.28:0/2836619549 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddcda342d0 con 0x55ddcddbc260 2024-04-24T08:09:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.358+0000 7fb0e5400640 1 --2- 172.21.15.28:0/2836619549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbc260 0x55ddcddbc680 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8005e30 tx=0x7fb0d8005f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ccb88b781cba1b2b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:36.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.358+0000 7fb0dd600640 1 -- 172.21.15.28:0/2575838733 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb0d4010ac0 con 0x7fb0e0074a60 2024-04-24T08:09:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.359+0000 7fb0d1c00640 1 -- 172.21.15.28:0/2836619549 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8031650 con 0x55ddcddbc260 2024-04-24T08:09:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.360+0000 7fb0d1c00640 1 -- 172.21.15.28:0/2836619549 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d8036040 con 0x55ddcddbc260 2024-04-24T08:09:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.360+0000 7fb0d1c00640 1 -- 172.21.15.28:0/2836619549 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d8004770 con 0x55ddcddbc260 2024-04-24T08:09:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.360+0000 7fb0dd600640 1 --2- 172.21.15.28:0/2575838733 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb0c80a3c30 0x7fb0c80a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.360+0000 7fb0dd600640 1 -- 172.21.15.28:0/2575838733 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb0d40d07d0 con 0x7fb0e0074a60 2024-04-24T08:09:36.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.360+0000 7fb0e4a00640 1 --2- 172.21.15.28:0/2575838733 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb0c80a3c30 0x7fb0c80a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.361+0000 7fb0ea383740 1 -- 172.21.15.28:0/2836619549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbc260 msgr2=0x55ddcddbc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.361+0000 7fb0ea383740 1 --2- 172.21.15.28:0/2836619549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbc260 0x55ddcddbc680 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8005e30 tx=0x7fb0d8005f70 comp rx=0 tx=0).stop 2024-04-24T08:09:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.361+0000 7fb0ea383740 1 -- 172.21.15.28:0/2836619549 shutdown_connections 2024-04-24T08:09:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.361+0000 7fb0ea383740 1 --2- 172.21.15.28:0/2836619549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddc1dc0 0x55ddcddc4250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.361+0000 7fb0ea383740 1 --2- 172.21.15.28:0/2836619549 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbcc50 0x55ddcddc1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.361+0000 7fb0ea383740 1 --2- 172.21.15.28:0/2836619549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbc260 0x55ddcddbc680 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.361+0000 7fb0ea383740 1 -- 172.21.15.28:0/2836619549 >> 172.21.15.28:0/2836619549 conn(0x55ddcddb78d0 msgr2=0x55ddcddb7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.361+0000 7fb0e4a00640 1 --2- 172.21.15.28:0/2575838733 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb0c80a3c30 0x7fb0c80a60f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb0d409c790 tx=0x7fb0d400cf60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.361+0000 7fb0ea383740 1 -- 172.21.15.28:0/2836619549 shutdown_connections 2024-04-24T08:09:36.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.361+0000 7fb0ea383740 1 -- 172.21.15.28:0/2836619549 wait complete. 2024-04-24T08:09:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.361+0000 7fb0ea383740 1 -- start start 2024-04-24T08:09:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.362+0000 7fb0ea383740 1 Processor -- start 2024-04-24T08:09:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.362+0000 7fb0ea383740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddbc260 0x55ddcddc6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.362+0000 7fb0ea383740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbcc50 0x55ddcddc6bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.362+0000 7fb0ea383740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddc1dc0 0x55ddcddcdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.362+0000 7fb0ea383740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ddcddc4d10 con 0x55ddcddc1dc0 2024-04-24T08:09:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.362+0000 7fb0ea383740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ddcdcdb590 con 0x55ddcddbc260 2024-04-24T08:09:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.362+0000 7fb0ea383740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ddcddc4e90 con 0x55ddcddbcc50 2024-04-24T08:09:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.362+0000 7fb0ea383740 10 client.0 _ll_register_callbacks cb 0x55ddcdcbe3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:09:36.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.363+0000 7fb0e5400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddbc260 0x55ddcddc6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.363+0000 7fb0e5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddc1dc0 0x55ddcddcdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.363+0000 7fb0dfe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbcc50 0x55ddcddc6bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.363+0000 7fb0e5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddc1dc0 0x55ddcddcdc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59982/0 (socket says 172.21.15.28:59982) 2024-04-24T08:09:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.363+0000 7fb0dfe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbcc50 0x55ddcddc6bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:34410/0 (socket says 172.21.15.28:34410) 2024-04-24T08:09:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.363+0000 7fb0e5400640 1 -- 172.21.15.28:0/3531641887 learned_addr learned my addr 172.21.15.28:0/3531641887 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.363+0000 7fb0dfe00640 1 -- 172.21.15.28:0/3531641887 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddbc260 msgr2=0x55ddcddc6680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.363+0000 7fb0dfe00640 1 --2- 172.21.15.28:0/3531641887 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddbc260 0x55ddcddc6680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.363+0000 7fb0dfe00640 1 -- 172.21.15.28:0/3531641887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddc1dc0 msgr2=0x55ddcddcdc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.363+0000 7fb0dfe00640 1 --2- 172.21.15.28:0/3531641887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddc1dc0 0x55ddcddcdc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.363+0000 7fb0dfe00640 1 -- 172.21.15.28:0/3531641887 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddcdafe6b0 con 0x55ddcddbcc50 2024-04-24T08:09:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.363+0000 7fb0e5400640 1 --2- 172.21.15.28:0/3531641887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddc1dc0 0x55ddcddcdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.363+0000 7fb0dfe00640 1 --2- 172.21.15.28:0/3531641887 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbcc50 0x55ddcddc6bc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009640 tx=0x7fb0cc00f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20bd1741b98fe250 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.363+0000 7fb0c6000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:09:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.364+0000 7fb0c6000640 1 -- 172.21.15.28:0/3531641887 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0cc0120d0 con 0x55ddcddbcc50 2024-04-24T08:09:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.364+0000 7fb0c6000640 1 -- 172.21.15.28:0/3531641887 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb0cc005020 con 0x55ddcddbcc50 2024-04-24T08:09:36.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.364+0000 7fb0ea383740 1 -- 172.21.15.28:0/3531641887 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0d8036040 con 0x55ddcddbcc50 2024-04-24T08:09:36.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.364+0000 7fb0c6000640 1 -- 172.21.15.28:0/3531641887 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb0cc012270 con 0x55ddcddbcc50 2024-04-24T08:09:36.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.364+0000 7fb0ea383740 10 client.25036 fetch_fsmap learned FSMap version 23 2024-04-24T08:09:36.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.364+0000 7fb0ea383740 1 -- 172.21.15.28:0/3531641887 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ddcddd1040 con 0x55ddcddbcc50 2024-04-24T08:09:36.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.364+0000 7fb0c6000640 1 -- 172.21.15.28:0/3531641887 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d8036040 con 0x55ddcddbcc50 2024-04-24T08:09:36.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.364+0000 7fb0c6000640 1 -- 172.21.15.28:0/3531641887 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb0cc01c470 con 0x55ddcddbcc50 2024-04-24T08:09:36.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.365+0000 7fb0ea383740 10 client.25036 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:09:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.365+0000 7fb0ea383740 10 client.25036 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-24T08:09:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.365+0000 7fb0ea383740 20 client.25036 populate_metadata read hostname 'smithi028' 2024-04-24T08:09:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.365+0000 7fb0ea383740 1 --2- 172.21.15.28:0/3531641887 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ddcddd36c0 0x55ddcddd4b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.365+0000 7fb0ea383740 4 client.25036 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:09:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.365+0000 7fb0ea383740 1 -- 172.21.15.28:0/3531641887 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ddcdd2f530 con 0x55ddcddd36c0 2024-04-24T08:09:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.365+0000 7fb0e5400640 1 --2- 172.21.15.28:0/3531641887 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ddcddd36c0 0x55ddcddd4b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.365+0000 7fb0e5400640 1 --2- 172.21.15.28:0/3531641887 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ddcddd36c0 0x55ddcddd4b30 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8a5d35dfb1ba1e3 server_cookie=af1001becb87fef9 in_seq=0 out_seq=0 2024-04-24T08:09:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.365+0000 7fb0c6000640 10 client.25036 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:09:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.366+0000 7fb0c6000640 1 -- 172.21.15.28:0/3531641887 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ddcdd2f530 con 0x55ddcddd36c0 2024-04-24T08:09:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.366+0000 7fb0c6000640 10 client.25036 handle_command_reply: tid=0 2024-04-24T08:09:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.366+0000 7fb0ea383740 1 client.25036 shutdown 2024-04-24T08:09:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.367+0000 7fb0ea383740 1 -- 172.21.15.28:0/3531641887 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbcc50 msgr2=0x55ddcddc6bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.367+0000 7fb0ea383740 1 --2- 172.21.15.28:0/3531641887 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbcc50 0x55ddcddc6bc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fb0cc009640 tx=0x7fb0cc00f790 comp rx=0 tx=0).stop 2024-04-24T08:09:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.367+0000 7fb0ea383740 1 -- 172.21.15.28:0/3531641887 shutdown_connections 2024-04-24T08:09:36.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.367+0000 7fb0ea383740 1 --2- 172.21.15.28:0/3531641887 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ddcddd36c0 0x55ddcddd4b30 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.367+0000 7fb0ea383740 1 --2- 172.21.15.28:0/3531641887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddc1dc0 0x55ddcddcdc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.367+0000 7fb0ea383740 1 --2- 172.21.15.28:0/3531641887 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbcc50 0x55ddcddc6bc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.367+0000 7fb0ea383740 1 --2- 172.21.15.28:0/3531641887 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddbc260 0x55ddcddc6680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.367+0000 7fb0ea383740 1 -- 172.21.15.28:0/3531641887 >> 172.21.15.28:0/3531641887 conn(0x55ddcddb78d0 msgr2=0x55ddcddb6970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.367+0000 7fb0c6000640 0 client.25036 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:09:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.367+0000 7fb0ea383740 1 -- 172.21.15.28:0/3531641887 shutdown_connections 2024-04-24T08:09:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.367+0000 7fb0ea383740 1 -- 172.21.15.28:0/3531641887 wait complete. 2024-04-24T08:09:36.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.367+0000 7fb0ea383740 20 client.25036 trim_cache size 0 max 16384 2024-04-24T08:09:36.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.387+0000 7fb0ea383740 1 Processor -- start 2024-04-24T08:09:36.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.387+0000 7fb0ea383740 1 -- start start 2024-04-24T08:09:36.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.387+0000 7fb0ea383740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbc260 0x55ddcddc5230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.387+0000 7fb0ea383740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbcc50 0x55ddcddcdbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.387+0000 7fb0ea383740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddc1dc0 0x55ddcddbe0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.387+0000 7fb0ea383740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ddcddd2e70 con 0x55ddcddbc260 2024-04-24T08:09:36.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.387+0000 7fb0ea383740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ddcddd3540 con 0x55ddcddc1dc0 2024-04-24T08:09:36.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.387+0000 7fb0ea383740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ddcddd3c10 con 0x55ddcddbcc50 2024-04-24T08:09:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.388+0000 7fb0e5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbc260 0x55ddcddc5230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.388+0000 7fb0e5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbc260 0x55ddcddc5230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59994/0 (socket says 172.21.15.28:59994) 2024-04-24T08:09:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.388+0000 7fb0e5400640 1 -- 172.21.15.28:0/3516062453 learned_addr learned my addr 172.21.15.28:0/3516062453 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.388+0000 7fb0dfe00640 1 --2- 172.21.15.28:0/3516062453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbcc50 0x55ddcddcdbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.388+0000 7fb0e5400640 1 --2- 172.21.15.28:0/3516062453 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddc1dc0 0x55ddcddbe0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.388+0000 7fb0e5400640 1 -- 172.21.15.28:0/3516062453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbcc50 msgr2=0x55ddcddcdbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.388+0000 7fb0e5400640 1 --2- 172.21.15.28:0/3516062453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbcc50 0x55ddcddcdbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.388+0000 7fb0e5400640 1 -- 172.21.15.28:0/3516062453 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddc1dc0 msgr2=0x55ddcddbe0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.388+0000 7fb0e5400640 1 --2- 172.21.15.28:0/3516062453 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddc1dc0 0x55ddcddbe0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.388+0000 7fb0e5400640 1 -- 172.21.15.28:0/3516062453 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ddcddc7440 con 0x55ddcddbc260 2024-04-24T08:09:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.388+0000 7fb0dfe00640 1 --2- 172.21.15.28:0/3516062453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbcc50 0x55ddcddcdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.389+0000 7fb0e5400640 1 --2- 172.21.15.28:0/3516062453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbc260 0x55ddcddc5230 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009ca0 tx=0x7fb0d8005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f5153a74b5198c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.389+0000 7fb0d1200640 1 -- 172.21.15.28:0/3516062453 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d804a040 con 0x55ddcddbc260 2024-04-24T08:09:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.389+0000 7fb0d1200640 1 -- 172.21.15.28:0/3516062453 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d8036040 con 0x55ddcddbc260 2024-04-24T08:09:36.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.389+0000 7fb0d1200640 1 -- 172.21.15.28:0/3516062453 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d802f870 con 0x55ddcddbc260 2024-04-24T08:09:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.390+0000 7fb0ea383740 1 -- 172.21.15.28:0/3516062453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbc260 msgr2=0x55ddcddc5230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.390+0000 7fb0ea383740 1 --2- 172.21.15.28:0/3516062453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbc260 0x55ddcddc5230 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8009ca0 tx=0x7fb0d8005da0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.390+0000 7fb0ea383740 1 -- 172.21.15.28:0/3516062453 shutdown_connections 2024-04-24T08:09:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.390+0000 7fb0ea383740 1 --2- 172.21.15.28:0/3516062453 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddc1dc0 0x55ddcddbe0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.390+0000 7fb0ea383740 1 --2- 172.21.15.28:0/3516062453 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbcc50 0x55ddcddcdbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.390+0000 7fb0ea383740 1 --2- 172.21.15.28:0/3516062453 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbc260 0x55ddcddc5230 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.390+0000 7fb0ea383740 1 -- 172.21.15.28:0/3516062453 >> 172.21.15.28:0/3516062453 conn(0x55ddcddb78d0 msgr2=0x55ddcddb7cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.390+0000 7fb0ea383740 1 -- 172.21.15.28:0/3516062453 shutdown_connections 2024-04-24T08:09:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.390+0000 7fb0ea383740 1 -- 172.21.15.28:0/3516062453 wait complete. 2024-04-24T08:09:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.390+0000 7fb0ea383740 1 -- start start 2024-04-24T08:09:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.390+0000 7fb0ea383740 1 Processor -- start 2024-04-24T08:09:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.391+0000 7fb0ea383740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbc260 0x55ddcddc86d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.391+0000 7fb0ea383740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbcc50 0x55ddcddcdc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.391+0000 7fb0ea383740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddc1dc0 0x55ddcddc8ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.391+0000 7fb0ea383740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ddcddd49b0 con 0x55ddcddbcc50 2024-04-24T08:09:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.391+0000 7fb0ea383740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ddcddd42e0 con 0x55ddcddc1dc0 2024-04-24T08:09:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.391+0000 7fb0ea383740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ddcddd5080 con 0x55ddcddbc260 2024-04-24T08:09:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.391+0000 7fb0ea383740 10 client.0 _ll_register_callbacks cb 0x7fb0cc0127f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:09:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.391+0000 7fb0e5400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbc260 0x55ddcddc86d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.391+0000 7fb0dfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbcc50 0x55ddcddcdc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.391+0000 7fb0e5400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbc260 0x55ddcddc86d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:34428/0 (socket says 172.21.15.28:34428) 2024-04-24T08:09:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.391+0000 7fb0e5400640 1 -- 172.21.15.28:0/2840785956 learned_addr learned my addr 172.21.15.28:0/2840785956 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.391+0000 7fb0e5400640 1 --2- 172.21.15.28:0/2840785956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddc1dc0 0x55ddcddc8ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.392+0000 7fb0e5400640 1 -- 172.21.15.28:0/2840785956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddc1dc0 msgr2=0x55ddcddc8ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.392+0000 7fb0e5400640 1 --2- 172.21.15.28:0/2840785956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddc1dc0 0x55ddcddc8ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.392+0000 7fb0e5400640 1 -- 172.21.15.28:0/2840785956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbcc50 msgr2=0x55ddcddcdc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:09:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.392+0000 7fb0e5400640 1 --2- 172.21.15.28:0/2840785956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbcc50 0x55ddcddcdc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.392+0000 7fb0e5400640 1 -- 172.21.15.28:0/2840785956 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ddcde376e0 con 0x55ddcddbc260 2024-04-24T08:09:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.392+0000 7fb0dfe00640 1 --2- 172.21.15.28:0/2840785956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbcc50 0x55ddcddcdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.392+0000 7fb0e5400640 1 --2- 172.21.15.28:0/2840785956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbc260 0x55ddcddc86d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8031c10 tx=0x7fb0d8005cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94ce21277e8f3019 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.392+0000 7fb0c6000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:09:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.392+0000 7fb0c6000640 1 -- 172.21.15.28:0/2840785956 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0d804a040 con 0x55ddcddbc260 2024-04-24T08:09:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.392+0000 7fb0c6000640 1 -- 172.21.15.28:0/2840785956 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7fb0d8004240 con 0x55ddcddbc260 2024-04-24T08:09:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.392+0000 7fb0ea383740 1 -- 172.21.15.28:0/2840785956 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0d8036040 con 0x55ddcddbc260 2024-04-24T08:09:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.392+0000 7fb0c6000640 1 -- 172.21.15.28:0/2840785956 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fb0d8004b40 con 0x55ddcddbc260 2024-04-24T08:09:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.393+0000 7fb0ea383740 10 client.25042 fetch_fsmap learned FSMap version 23 2024-04-24T08:09:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.393+0000 7fb0ea383740 1 -- 172.21.15.28:0/2840785956 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ddcde389e0 con 0x55ddcddbc260 2024-04-24T08:09:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.393+0000 7fb0c6000640 1 -- 172.21.15.28:0/2840785956 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0d8036040 con 0x55ddcddbc260 2024-04-24T08:09:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.393+0000 7fb0c6000640 1 -- 172.21.15.28:0/2840785956 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fb0d802fad0 con 0x55ddcddbc260 2024-04-24T08:09:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.393+0000 7fb0ea383740 10 client.25042 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:09:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.393+0000 7fb0ea383740 10 client.25042 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-24T08:09:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.393+0000 7fb0ea383740 20 client.25042 populate_metadata read hostname 'smithi028' 2024-04-24T08:09:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.393+0000 7fb0ea383740 1 --2- 172.21.15.28:0/2840785956 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ddcde3d040 0x55ddcde3bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.393+0000 7fb0ea383740 4 client.25042 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:09:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.393+0000 7fb0ea383740 1 -- 172.21.15.28:0/2840785956 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "status"}) -- 0x55ddcdd293a0 con 0x55ddcde3d040 2024-04-24T08:09:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.393+0000 7fb0dfe00640 1 --2- 172.21.15.28:0/2840785956 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ddcde3d040 0x55ddcde3bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.394+0000 7fb0dfe00640 1 --2- 172.21.15.28:0/2840785956 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ddcde3d040 0x55ddcde3bd50 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca684abbe0672e74 server_cookie=cc6cb8e0ba0bb492 in_seq=0 out_seq=0 2024-04-24T08:09:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.394+0000 7fb0c6000640 10 client.25042 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:09:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.394+0000 7fb0c6000640 1 -- 172.21.15.28:0/2840785956 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x55ddcdd293a0 con 0x55ddcde3d040 2024-04-24T08:09:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.394+0000 7fb0c6000640 10 client.25042 handle_command_reply: tid=0 2024-04-24T08:09:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.394+0000 7fb0ea383740 1 client.25042 shutdown 2024-04-24T08:09:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.395+0000 7fb0ea383740 1 -- 172.21.15.28:0/2840785956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbc260 msgr2=0x55ddcddc86d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.395+0000 7fb0ea383740 1 --2- 172.21.15.28:0/2840785956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbc260 0x55ddcddc86d0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fb0d8031c10 tx=0x7fb0d8005cf0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.395+0000 7fb0ea383740 1 -- 172.21.15.28:0/2840785956 shutdown_connections 2024-04-24T08:09:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.395+0000 7fb0ea383740 1 --2- 172.21.15.28:0/2840785956 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ddcde3d040 0x55ddcde3bd50 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.395+0000 7fb0ea383740 1 --2- 172.21.15.28:0/2840785956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ddcddbcc50 0x55ddcddcdc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.395+0000 7fb0ea383740 1 --2- 172.21.15.28:0/2840785956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ddcddc1dc0 0x55ddcddc8ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.395+0000 7fb0c6000640 0 client.25042 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:09:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.395+0000 7fb0ea383740 1 --2- 172.21.15.28:0/2840785956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ddcddbc260 0x55ddcddc86d0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.395+0000 7fb0ea383740 1 -- 172.21.15.28:0/2840785956 >> 172.21.15.28:0/2840785956 conn(0x55ddcddb78d0 msgr2=0x55ddcddc5230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.395+0000 7fb0ea383740 1 -- 172.21.15.28:0/2840785956 shutdown_connections 2024-04-24T08:09:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.395+0000 7fb0ea383740 1 -- 172.21.15.28:0/2840785956 wait complete. 2024-04-24T08:09:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.395+0000 7fb0ea383740 20 client.25042 trim_cache size 0 max 16384 2024-04-24T08:09:36.399 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:09:36.399 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "473aaf58-0210-11ef-bc93-c7b262605968", 2024-04-24T08:09:36.399 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-24T08:09:36.399 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24574, 2024-04-24T08:09:36.399 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-24T08:09:36.399 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-24T08:09:36.399 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-24T08:09:36.399 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 104.318390433, 2024-04-24T08:09:36.399 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 23, 2024-04-24T08:09:36.399 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 88, 2024-04-24T08:09:36.399 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 86, 2024-04-24T08:09:36.399 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 117.784852126 2024-04-24T08:09:36.400 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:09:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.396+0000 7fb0e5e00640 1 -- 172.21.15.28:0/2575838733 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb0c80a3c30 msgr2=0x7fb0c80a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.396+0000 7fb0e5e00640 1 --2- 172.21.15.28:0/2575838733 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb0c80a3c30 0x7fb0c80a60f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb0d409c790 tx=0x7fb0d400cf60 comp rx=0 tx=0).stop 2024-04-24T08:09:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.396+0000 7fb0e5e00640 1 -- 172.21.15.28:0/2575838733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0e0074a60 msgr2=0x7fb0e019a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.396+0000 7fb0e5e00640 1 --2- 172.21.15.28:0/2575838733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0e0074a60 0x7fb0e019a360 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fb0d400e9f0 tx=0x7fb0d400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.396+0000 7fb0e5e00640 1 -- 172.21.15.28:0/2575838733 shutdown_connections 2024-04-24T08:09:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.396+0000 7fb0e5e00640 1 --2- 172.21.15.28:0/2575838733 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb0c80a3c30 0x7fb0c80a60f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.396+0000 7fb0e5e00640 1 --2- 172.21.15.28:0/2575838733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb0e0109dd0 0x7fb0e01a1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.396+0000 7fb0e5e00640 1 --2- 172.21.15.28:0/2575838733 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e0076040 0x7fb0e019a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.396+0000 7fb0e5e00640 1 --2- 172.21.15.28:0/2575838733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb0e0074a60 0x7fb0e019a360 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.396+0000 7fb0e5e00640 1 -- 172.21.15.28:0/2575838733 >> 172.21.15.28:0/2575838733 conn(0x7fb0e00fd4a0 msgr2=0x7fb0e0105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.396+0000 7fb0e5e00640 1 -- 172.21.15.28:0/2575838733 shutdown_connections 2024-04-24T08:09:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:36.396+0000 7fb0e5e00640 1 -- 172.21.15.28:0/2575838733 wait complete. 2024-04-24T08:09:36.413 INFO:tasks.cephadm:Setup complete, yielding 2024-04-24T08:09:36.414 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-24T08:09:36.423 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-24T08:09:36.423 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- bash -c 'ceph orch status' 2024-04-24T08:09:36.495 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:36 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2926155163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:09:36.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:36 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2926155163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:09:36.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:36 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2926155163' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:09:36.877 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:09:37.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:37 smithi117 ceph-mon[32438]: pgmap v371: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:09:37.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:37 smithi194 ceph-mon[32836]: pgmap v371: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:09:37.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:37 smithi028 ceph-mon[27786]: pgmap v371: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:09:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.262+0000 7f541fe00700 1 -- 172.21.15.28:0/425135489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5420105e90 msgr2=0x7f54201062f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.262+0000 7f541fe00700 1 --2- 172.21.15.28:0/425135489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5420105e90 0x7f54201062f0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f540c0099c0 tx=0x7f540c009cd0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.262+0000 7f541fe00700 1 -- 172.21.15.28:0/425135489 shutdown_connections 2024-04-24T08:09:39.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.262+0000 7f541fe00700 1 --2- 172.21.15.28:0/425135489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5420106830 0x7f542010d0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.262+0000 7f541fe00700 1 --2- 172.21.15.28:0/425135489 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5420104c90 0x7f54201050b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.262+0000 7f541fe00700 1 --2- 172.21.15.28:0/425135489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5420105e90 0x7f54201062f0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.262+0000 7f541fe00700 1 -- 172.21.15.28:0/425135489 >> 172.21.15.28:0/425135489 conn(0x7f5420100210 msgr2=0x7f5420102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.262+0000 7f541fe00700 1 -- 172.21.15.28:0/425135489 shutdown_connections 2024-04-24T08:09:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.262+0000 7f541fe00700 1 -- 172.21.15.28:0/425135489 wait complete. 2024-04-24T08:09:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.263+0000 7f541fe00700 1 Processor -- start 2024-04-24T08:09:39.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.263+0000 7f541fe00700 1 -- start start 2024-04-24T08:09:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.264+0000 7f541fe00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5420104c90 0x7f542019ae30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.264+0000 7f541fe00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5420105e90 0x7f542019b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.264+0000 7f541fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5420106830 0x7f54201a23f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.264+0000 7f541fe00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5420110850 con 0x7f5420106830 2024-04-24T08:09:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.264+0000 7f541fe00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f54201106d0 con 0x7f5420104c90 2024-04-24T08:09:39.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.264+0000 7f541fe00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f54201109d0 con 0x7f5420105e90 2024-04-24T08:09:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.264+0000 7f541e000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5420105e90 0x7f542019b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.264+0000 7f541ea00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5420104c90 0x7f542019ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.264+0000 7f541ea00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5420104c90 0x7f542019ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:55684/0 (socket says 172.21.15.28:55684) 2024-04-24T08:09:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.264+0000 7f541e000700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5420105e90 0x7f542019b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:34454/0 (socket says 172.21.15.28:34454) 2024-04-24T08:09:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.264+0000 7f541ea00700 1 -- 172.21.15.28:0/3873627221 learned_addr learned my addr 172.21.15.28:0/3873627221 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.264+0000 7f541f400700 1 --2- 172.21.15.28:0/3873627221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5420106830 0x7f54201a23f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.264+0000 7f541e000700 1 -- 172.21.15.28:0/3873627221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5420104c90 msgr2=0x7f542019ae30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.265+0000 7f541e000700 1 --2- 172.21.15.28:0/3873627221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5420104c90 0x7f542019ae30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.265+0000 7f541e000700 1 -- 172.21.15.28:0/3873627221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5420106830 msgr2=0x7f54201a23f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.265+0000 7f541e000700 1 --2- 172.21.15.28:0/3873627221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5420106830 0x7f54201a23f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.265+0000 7f541e000700 1 -- 172.21.15.28:0/3873627221 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54201a2a90 con 0x7f5420105e90 2024-04-24T08:09:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.265+0000 7f541ea00700 1 --2- 172.21.15.28:0/3873627221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5420104c90 0x7f542019ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.265+0000 7f541f400700 1 --2- 172.21.15.28:0/3873627221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5420106830 0x7f54201a23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:39.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.265+0000 7f541e000700 1 --2- 172.21.15.28:0/3873627221 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5420105e90 0x7f542019b370 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f540c0099c0 tx=0x7f540c00bf60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.265+0000 7f5417400700 1 -- 172.21.15.28:0/3873627221 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c009e60 con 0x7f5420105e90 2024-04-24T08:09:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.265+0000 7f5417400700 1 -- 172.21.15.28:0/3873627221 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f540c00f460 con 0x7f5420105e90 2024-04-24T08:09:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.265+0000 7f541fe00700 1 -- 172.21.15.28:0/3873627221 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54201a2d20 con 0x7f5420105e90 2024-04-24T08:09:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.265+0000 7f5417400700 1 -- 172.21.15.28:0/3873627221 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f540c017630 con 0x7f5420105e90 2024-04-24T08:09:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.266+0000 7f541fe00700 1 -- 172.21.15.28:0/3873627221 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54201a31b0 con 0x7f5420105e90 2024-04-24T08:09:39.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.267+0000 7f541fe00700 1 -- 172.21.15.28:0/3873627221 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54200697e0 con 0x7f5420105e90 2024-04-24T08:09:39.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.269+0000 7f5417400700 1 -- 172.21.15.28:0/3873627221 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f540c00f610 con 0x7f5420105e90 2024-04-24T08:09:39.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.270+0000 7f5417400700 1 --2- 172.21.15.28:0/3873627221 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f54040a3c40 0x7f54040a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:39.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.270+0000 7f5417400700 1 -- 172.21.15.28:0/3873627221 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f540c0d1250 con 0x7f5420105e90 2024-04-24T08:09:39.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.270+0000 7f541ea00700 1 --2- 172.21.15.28:0/3873627221 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f54040a3c40 0x7f54040a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:39.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.271+0000 7f541ea00700 1 --2- 172.21.15.28:0/3873627221 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f54040a3c40 0x7f54040a6100 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f54100105c0 tx=0x7f541000b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:39.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.272+0000 7f5417400700 1 -- 172.21.15.28:0/3873627221 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f540c013070 con 0x7f5420105e90 2024-04-24T08:09:39.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.529+0000 7f541fe00700 1 -- 172.21.15.28:0/3873627221 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f5420062f30 con 0x7f54040a3c40 2024-04-24T08:09:39.529 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:39 smithi028 ceph-mon[27786]: pgmap v372: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:09:39.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.532+0000 7f5417400700 1 -- 172.21.15.28:0/3873627221 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f5420062f30 con 0x7f54040a3c40 2024-04-24T08:09:39.533 INFO:teuthology.orchestra.run.smithi028.stdout:Backend: cephadm 2024-04-24T08:09:39.533 INFO:teuthology.orchestra.run.smithi028.stdout:Available: Yes 2024-04-24T08:09:39.533 INFO:teuthology.orchestra.run.smithi028.stdout:Paused: No 2024-04-24T08:09:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.535+0000 7f541fe00700 1 -- 172.21.15.28:0/3873627221 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f54040a3c40 msgr2=0x7f54040a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.535+0000 7f541fe00700 1 --2- 172.21.15.28:0/3873627221 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f54040a3c40 0x7f54040a6100 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f54100105c0 tx=0x7f541000b4a0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.535+0000 7f541fe00700 1 -- 172.21.15.28:0/3873627221 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5420105e90 msgr2=0x7f542019b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:39.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.535+0000 7f541fe00700 1 --2- 172.21.15.28:0/3873627221 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5420105e90 0x7f542019b370 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f540c0099c0 tx=0x7f540c00bf60 comp rx=0 tx=0).stop 2024-04-24T08:09:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.535+0000 7f541fe00700 1 -- 172.21.15.28:0/3873627221 shutdown_connections 2024-04-24T08:09:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.535+0000 7f541fe00700 1 --2- 172.21.15.28:0/3873627221 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f54040a3c40 0x7f54040a6100 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.535+0000 7f541fe00700 1 --2- 172.21.15.28:0/3873627221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5420106830 0x7f54201a23f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.535+0000 7f541fe00700 1 --2- 172.21.15.28:0/3873627221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5420104c90 0x7f542019ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.535+0000 7f541fe00700 1 --2- 172.21.15.28:0/3873627221 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5420105e90 0x7f542019b370 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:39.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.535+0000 7f541fe00700 1 -- 172.21.15.28:0/3873627221 >> 172.21.15.28:0/3873627221 conn(0x7f5420100210 msgr2=0x7f5420109190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:39.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.536+0000 7f541fe00700 1 -- 172.21.15.28:0/3873627221 shutdown_connections 2024-04-24T08:09:39.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:39.536+0000 7f541fe00700 1 -- 172.21.15.28:0/3873627221 wait complete. 2024-04-24T08:09:39.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:39 smithi117 ceph-mon[32438]: pgmap v372: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:09:39.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:39 smithi194 ceph-mon[32836]: pgmap v372: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:09:40.368 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ps' 2024-04-24T08:09:40.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:40 smithi028 ceph-mon[27786]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:09:40.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:40 smithi117 ceph-mon[32438]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:09:40.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:40 smithi194 ceph-mon[32836]: from='client.25048 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:09:40.827 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:09:41.500 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:41 smithi028 ceph-mon[27786]: pgmap v373: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:09:41.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:41 smithi117 ceph-mon[32438]: pgmap v373: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:09:41.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:41 smithi194 ceph-mon[32836]: pgmap v373: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:09:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.118+0000 7f6b51400700 1 -- 172.21.15.28:0/4186784022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4c108840 msgr2=0x7f6b4c10acb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.118+0000 7f6b51400700 1 --2- 172.21.15.28:0/4186784022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4c108840 0x7f6b4c10acb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6b4400b0a0 tx=0x7f6b4400b3b0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.118+0000 7f6b51400700 1 -- 172.21.15.28:0/4186784022 shutdown_connections 2024-04-24T08:09:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.118+0000 7f6b51400700 1 --2- 172.21.15.28:0/4186784022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4c108840 0x7f6b4c10acb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.118+0000 7f6b51400700 1 --2- 172.21.15.28:0/4186784022 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b4c1035e0 0x7f6b4c1059d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.118+0000 7f6b51400700 1 --2- 172.21.15.28:0/4186784022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b4c105f10 0x7f6b4c108300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.118+0000 7f6b51400700 1 -- 172.21.15.28:0/4186784022 >> 172.21.15.28:0/4186784022 conn(0x7f6b4c0fcff0 msgr2=0x7f6b4c0ff430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:43.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.118+0000 7f6b51400700 1 -- 172.21.15.28:0/4186784022 shutdown_connections 2024-04-24T08:09:43.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.118+0000 7f6b51400700 1 -- 172.21.15.28:0/4186784022 wait complete. 2024-04-24T08:09:43.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.120+0000 7f6b51400700 1 Processor -- start 2024-04-24T08:09:43.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.120+0000 7f6b51400700 1 -- start start 2024-04-24T08:09:43.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b51400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b4c1035e0 0x7f6b4c19af20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:43.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b51400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b4c105f10 0x7f6b4c19b460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:43.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b51400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4c108840 0x7f6b4c1a24e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:43.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b51400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c10e700 con 0x7f6b4c108840 2024-04-24T08:09:43.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b51400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c10e580 con 0x7f6b4c105f10 2024-04-24T08:09:43.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b51400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6b4c10e880 con 0x7f6b4c1035e0 2024-04-24T08:09:43.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b4b200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4c108840 0x7f6b4c1a24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:43.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b4a800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b4c1035e0 0x7f6b4c19af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:43.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b4b200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4c108840 0x7f6b4c1a24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46612/0 (socket says 172.21.15.28:46612) 2024-04-24T08:09:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b4a800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b4c1035e0 0x7f6b4c19af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:38936/0 (socket says 172.21.15.28:38936) 2024-04-24T08:09:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b4b200700 1 -- 172.21.15.28:0/844211131 learned_addr learned my addr 172.21.15.28:0/844211131 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b41e00700 1 --2- 172.21.15.28:0/844211131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b4c105f10 0x7f6b4c19b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b4b200700 1 -- 172.21.15.28:0/844211131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b4c1035e0 msgr2=0x7f6b4c19af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b4b200700 1 --2- 172.21.15.28:0/844211131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b4c1035e0 0x7f6b4c19af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b4b200700 1 -- 172.21.15.28:0/844211131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b4c105f10 msgr2=0x7f6b4c19b460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b4b200700 1 --2- 172.21.15.28:0/844211131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b4c105f10 0x7f6b4c19b460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b4b200700 1 -- 172.21.15.28:0/844211131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b4c1a2b80 con 0x7f6b4c108840 2024-04-24T08:09:43.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.121+0000 7f6b41e00700 1 --2- 172.21.15.28:0/844211131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b4c105f10 0x7f6b4c19b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.122+0000 7f6b4b200700 1 --2- 172.21.15.28:0/844211131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4c108840 0x7f6b4c1a24e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6b44004d00 tx=0x7f6b44004d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.122+0000 7f6b43e00700 1 -- 172.21.15.28:0/844211131 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b44021070 con 0x7f6b4c108840 2024-04-24T08:09:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.122+0000 7f6b43e00700 1 -- 172.21.15.28:0/844211131 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b44007b80 con 0x7f6b4c108840 2024-04-24T08:09:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.122+0000 7f6b43e00700 1 -- 172.21.15.28:0/844211131 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b440052a0 con 0x7f6b4c108840 2024-04-24T08:09:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.122+0000 7f6b51400700 1 -- 172.21.15.28:0/844211131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b4c1a2e10 con 0x7f6b4c108840 2024-04-24T08:09:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.122+0000 7f6b51400700 1 -- 172.21.15.28:0/844211131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b4c1a3210 con 0x7f6b4c108840 2024-04-24T08:09:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.124+0000 7f6b51400700 1 -- 172.21.15.28:0/844211131 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b4c069820 con 0x7f6b4c108840 2024-04-24T08:09:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.126+0000 7f6b43e00700 1 -- 172.21.15.28:0/844211131 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6b44024070 con 0x7f6b4c108840 2024-04-24T08:09:43.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.128+0000 7f6b43e00700 1 --2- 172.21.15.28:0/844211131 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6b2c0a3cc0 0x7f6b2c0a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:43.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.128+0000 7f6b43e00700 1 -- 172.21.15.28:0/844211131 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f6b4400f020 con 0x7f6b4c108840 2024-04-24T08:09:43.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.128+0000 7f6b4a800700 1 --2- 172.21.15.28:0/844211131 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6b2c0a3cc0 0x7f6b2c0a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.129+0000 7f6b4a800700 1 --2- 172.21.15.28:0/844211131 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6b2c0a3cc0 0x7f6b2c0a6180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c009c60 tx=0x7f6b3c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:43.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.133+0000 7f6b43e00700 1 -- 172.21.15.28:0/844211131 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6b4402b020 con 0x7f6b4c108840 2024-04-24T08:09:43.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.375+0000 7f6b51400700 1 -- 172.21.15.28:0/844211131 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7f6b4c062f30 con 0x7f6b2c0a3cc0 2024-04-24T08:09:43.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.414+0000 7f6b43e00700 1 -- 172.21.15.28:0/844211131 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7f6b4c062f30 con 0x7f6b2c0a3cc0 2024-04-24T08:09:43.415 INFO:teuthology.orchestra.run.smithi028.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-04-24T08:09:43.415 INFO:teuthology.orchestra.run.smithi028.stdout:mds.a smithi028 running (2m) 22s ago 2m 12.4M - 19.0.0-3116-g1f81fb39 b98aea5aa06f e9d1c0734f93 2024-04-24T08:09:43.415 INFO:teuthology.orchestra.run.smithi028.stdout:mds.b smithi117 running (118s) 49s ago 117s 16.4M - 19.0.0-3116-g1f81fb39 b98aea5aa06f cc15683a7e2d 2024-04-24T08:09:43.415 INFO:teuthology.orchestra.run.smithi028.stdout:mds.c smithi194 running (2m) 49s ago 2m 18.1M - 19.0.0-3116-g1f81fb39 b98aea5aa06f d1d18f3a853e 2024-04-24T08:09:43.415 INFO:teuthology.orchestra.run.smithi028.stdout:mds.d smithi028 running (2m) 22s ago 2m 12.5M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 9ef75adc945f 2024-04-24T08:09:43.415 INFO:teuthology.orchestra.run.smithi028.stdout:mds.e smithi117 running (2m) 49s ago 2m 16.4M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 25e09601568a 2024-04-24T08:09:43.415 INFO:teuthology.orchestra.run.smithi028.stdout:mds.f smithi194 running (2m) 49s ago 2m 19.4M - 19.0.0-3116-g1f81fb39 b98aea5aa06f fc34c6ee65bf 2024-04-24T08:09:43.416 INFO:teuthology.orchestra.run.smithi028.stdout:mds.g smithi028 running (2m) 22s ago 2m 24.6M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 5dce281f50a2 2024-04-24T08:09:43.416 INFO:teuthology.orchestra.run.smithi028.stdout:mds.h smithi117 running (2m) 49s ago 2m 20.3M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 070fa8e2068b 2024-04-24T08:09:43.416 INFO:teuthology.orchestra.run.smithi028.stdout:mds.i smithi194 running (2m) 49s ago 2m 18.9M - 19.0.0-3116-g1f81fb39 b98aea5aa06f a5cb61bed1c1 2024-04-24T08:09:43.416 INFO:teuthology.orchestra.run.smithi028.stdout:mds.j smithi028 running (2m) 22s ago 2m 12.5M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 98ab3eba529b 2024-04-24T08:09:43.416 INFO:teuthology.orchestra.run.smithi028.stdout:mds.k smithi117 running (2m) 49s ago 2m 15.4M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 8a8f543f281c 2024-04-24T08:09:43.416 INFO:teuthology.orchestra.run.smithi028.stdout:mds.l smithi194 running (2m) 49s ago 2m 14.2M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 95974d00367b 2024-04-24T08:09:43.416 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.x smithi028 *:9283,8765 running (10m) 22s ago 10m 515M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 8e12d69a1c64 2024-04-24T08:09:43.416 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.y smithi117 *:8443,8765 running (8m) 49s ago 8m 452M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 8cbd65784aa5 2024-04-24T08:09:43.416 INFO:teuthology.orchestra.run.smithi028.stdout:mgr.z smithi194 *:8443,8765 running (8m) 49s ago 8m 455M - 19.0.0-3116-g1f81fb39 b98aea5aa06f 91b29e229a6f 2024-04-24T08:09:43.416 INFO:teuthology.orchestra.run.smithi028.stdout:mon.a smithi028 running (10m) 22s ago 11m 51.6M 2048M 19.0.0-3116-g1f81fb39 b98aea5aa06f 29ad43385eba 2024-04-24T08:09:43.416 INFO:teuthology.orchestra.run.smithi028.stdout:mon.b smithi117 running (8m) 49s ago 8m 39.8M 2048M 19.0.0-3116-g1f81fb39 b98aea5aa06f abc0e1bdf477 2024-04-24T08:09:43.416 INFO:teuthology.orchestra.run.smithi028.stdout:mon.c smithi194 running (8m) 49s ago 8m 39.1M 2048M 19.0.0-3116-g1f81fb39 b98aea5aa06f 082e362d480d 2024-04-24T08:09:43.416 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 smithi028 running (8m) 22s ago 8m 43.6M 4273M 19.0.0-3116-g1f81fb39 b98aea5aa06f 53d01449da7e 2024-04-24T08:09:43.416 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 smithi117 running (7m) 49s ago 7m 55.6M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f b2878dac5932 2024-04-24T08:09:43.417 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 smithi194 running (7m) 49s ago 7m 45.7M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f 0ee6ffc09e0e 2024-04-24T08:09:43.417 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 smithi028 running (6m) 22s ago 6m 41.4M 4273M 19.0.0-3116-g1f81fb39 b98aea5aa06f ff511362c1af 2024-04-24T08:09:43.417 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 smithi117 running (6m) 49s ago 6m 42.0M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f 198a698228fb 2024-04-24T08:09:43.417 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 smithi194 running (5m) 49s ago 5m 57.0M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f 117b04849d0d 2024-04-24T08:09:43.417 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 smithi028 running (5m) 22s ago 5m 41.0M 4273M 19.0.0-3116-g1f81fb39 b98aea5aa06f c5d12798c355 2024-04-24T08:09:43.417 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 smithi117 running (4m) 49s ago 4m 58.7M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f e42eeee52349 2024-04-24T08:09:43.417 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 smithi194 running (4m) 49s ago 4m 54.1M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f 7d0aa61c5f07 2024-04-24T08:09:43.417 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 smithi028 running (3m) 22s ago 3m 39.6M 4273M 19.0.0-3116-g1f81fb39 b98aea5aa06f 481df7831341 2024-04-24T08:09:43.417 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 smithi117 running (3m) 49s ago 3m 41.8M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f 189f0ab883c5 2024-04-24T08:09:43.417 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 smithi194 running (2m) 49s ago 2m 53.4M 4272M 19.0.0-3116-g1f81fb39 b98aea5aa06f 0807c0b5e45b 2024-04-24T08:09:43.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.417+0000 7f6b51400700 1 -- 172.21.15.28:0/844211131 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6b2c0a3cc0 msgr2=0x7f6b2c0a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:43.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.417+0000 7f6b51400700 1 --2- 172.21.15.28:0/844211131 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6b2c0a3cc0 0x7f6b2c0a6180 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f6b3c009c60 tx=0x7f6b3c009260 comp rx=0 tx=0).stop 2024-04-24T08:09:43.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.417+0000 7f6b51400700 1 -- 172.21.15.28:0/844211131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4c108840 msgr2=0x7f6b4c1a24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.417+0000 7f6b51400700 1 --2- 172.21.15.28:0/844211131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4c108840 0x7f6b4c1a24e0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f6b44004d00 tx=0x7f6b44004d30 comp rx=0 tx=0).stop 2024-04-24T08:09:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.417+0000 7f6b51400700 1 -- 172.21.15.28:0/844211131 shutdown_connections 2024-04-24T08:09:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.417+0000 7f6b51400700 1 --2- 172.21.15.28:0/844211131 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6b2c0a3cc0 0x7f6b2c0a6180 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.417+0000 7f6b51400700 1 --2- 172.21.15.28:0/844211131 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6b4c108840 0x7f6b4c1a24e0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.417+0000 7f6b51400700 1 --2- 172.21.15.28:0/844211131 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6b4c1035e0 0x7f6b4c19af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.417+0000 7f6b51400700 1 --2- 172.21.15.28:0/844211131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b4c105f10 0x7f6b4c19b460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.417+0000 7f6b51400700 1 -- 172.21.15.28:0/844211131 >> 172.21.15.28:0/844211131 conn(0x7f6b4c0fcff0 msgr2=0x7f6b4c102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.418+0000 7f6b51400700 1 -- 172.21.15.28:0/844211131 shutdown_connections 2024-04-24T08:09:43.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:43.418+0000 7f6b51400700 1 -- 172.21.15.28:0/844211131 wait complete. 2024-04-24T08:09:43.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:43 smithi117 ceph-mon[32438]: pgmap v374: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:09:43.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:43 smithi194 ceph-mon[32836]: pgmap v374: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:09:43.784 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:43 smithi028 ceph-mon[27786]: pgmap v374: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:09:44.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- bash -c 'ceph orch ls' 2024-04-24T08:09:44.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:44 smithi028 ceph-mon[27786]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:09:44.674 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:09:44.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:44 smithi194 ceph-mon[32836]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:09:44.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:44 smithi117 ceph-mon[32438]: from='client.14934 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:09:45.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:45 smithi028 ceph-mon[27786]: pgmap v375: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:09:45.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:45 smithi194 ceph-mon[32836]: pgmap v375: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:09:45.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:45 smithi117 ceph-mon[32438]: pgmap v375: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:09:46.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.920+0000 7f58fd200700 1 -- 172.21.15.28:0/3152583495 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58f80790b0 msgr2=0x7f58f8079930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:46.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.920+0000 7f58fd200700 1 --2- 172.21.15.28:0/3152583495 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58f80790b0 0x7f58f8079930 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f58f0009f90 tx=0x7f58f000b330 comp rx=0 tx=0).stop 2024-04-24T08:09:46.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.920+0000 7f58fd200700 1 -- 172.21.15.28:0/3152583495 shutdown_connections 2024-04-24T08:09:46.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.920+0000 7f58fd200700 1 --2- 172.21.15.28:0/3152583495 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58f80790b0 0x7f58f8079930 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:46.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.920+0000 7f58fd200700 1 --2- 172.21.15.28:0/3152583495 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58f80774a0 0x7f58f80778c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:46.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.920+0000 7f58fd200700 1 --2- 172.21.15.28:0/3152583495 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58f80786f0 0x7f58f8078b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:46.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.920+0000 7f58fd200700 1 -- 172.21.15.28:0/3152583495 >> 172.21.15.28:0/3152583495 conn(0x7f58f8100340 msgr2=0x7f58f8102780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:46.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.920+0000 7f58fd200700 1 -- 172.21.15.28:0/3152583495 shutdown_connections 2024-04-24T08:09:46.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.920+0000 7f58fd200700 1 -- 172.21.15.28:0/3152583495 wait complete. 2024-04-24T08:09:46.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.921+0000 7f58fd200700 1 Processor -- start 2024-04-24T08:09:46.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.921+0000 7f58fd200700 1 -- start start 2024-04-24T08:09:46.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.922+0000 7f58fd200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58f80774a0 0x7f58f819f320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:46.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.922+0000 7f58fd200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58f80786f0 0x7f58f819f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:46.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.922+0000 7f58fd200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58f80790b0 0x7f58f81a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:46.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.922+0000 7f58fd200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f58f8114de0 con 0x7f58f80774a0 2024-04-24T08:09:46.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.922+0000 7f58fd200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f58f8114c60 con 0x7f58f80786f0 2024-04-24T08:09:46.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.922+0000 7f58fd200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f58f8114f60 con 0x7f58f80790b0 2024-04-24T08:09:46.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.922+0000 7f58f5e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58f80774a0 0x7f58f819f320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:46.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.922+0000 7f58f6800700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58f80790b0 0x7f58f81a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:46.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.922+0000 7f58f5e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58f80774a0 0x7f58f819f320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46630/0 (socket says 172.21.15.28:46630) 2024-04-24T08:09:46.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.922+0000 7f58f5e00700 1 -- 172.21.15.28:0/326716214 learned_addr learned my addr 172.21.15.28:0/326716214 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:46.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.922+0000 7f58efe00700 1 --2- 172.21.15.28:0/326716214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58f80786f0 0x7f58f819f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:46.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.923+0000 7f58f6800700 1 -- 172.21.15.28:0/326716214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58f80786f0 msgr2=0x7f58f819f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:46.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.923+0000 7f58f6800700 1 --2- 172.21.15.28:0/326716214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58f80786f0 0x7f58f819f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:46.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.923+0000 7f58f6800700 1 -- 172.21.15.28:0/326716214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58f80774a0 msgr2=0x7f58f819f320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:46.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.923+0000 7f58f6800700 1 --2- 172.21.15.28:0/326716214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58f80774a0 0x7f58f819f320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:46.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.923+0000 7f58f6800700 1 -- 172.21.15.28:0/326716214 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58f81a6f80 con 0x7f58f80790b0 2024-04-24T08:09:46.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.923+0000 7f58f6800700 1 --2- 172.21.15.28:0/326716214 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58f80790b0 0x7f58f81a68e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f58f0015000 tx=0x7f58f0007c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:46.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.923+0000 7f58eea00700 1 -- 172.21.15.28:0/326716214 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f001f070 con 0x7f58f80790b0 2024-04-24T08:09:46.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.923+0000 7f58eea00700 1 -- 172.21.15.28:0/326716214 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f0005530 con 0x7f58f80790b0 2024-04-24T08:09:46.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.923+0000 7f58eea00700 1 -- 172.21.15.28:0/326716214 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f001b680 con 0x7f58f80790b0 2024-04-24T08:09:46.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.923+0000 7f58fd200700 1 -- 172.21.15.28:0/326716214 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58f81a7210 con 0x7f58f80790b0 2024-04-24T08:09:46.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.923+0000 7f58fd200700 1 -- 172.21.15.28:0/326716214 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58f81a7670 con 0x7f58f80790b0 2024-04-24T08:09:46.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.925+0000 7f58fd200700 1 -- 172.21.15.28:0/326716214 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58f80697e0 con 0x7f58f80790b0 2024-04-24T08:09:46.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.927+0000 7f58eea00700 1 -- 172.21.15.28:0/326716214 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f58f0022070 con 0x7f58f80790b0 2024-04-24T08:09:46.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.928+0000 7f58eea00700 1 --2- 172.21.15.28:0/326716214 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f58dc0a3c40 0x7f58dc0a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:46.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.928+0000 7f58eea00700 1 -- 172.21.15.28:0/326716214 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8269+0+0 (secure 0 0 0) 0x7f58f00d1380 con 0x7f58f80790b0 2024-04-24T08:09:46.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.928+0000 7f58f5e00700 1 --2- 172.21.15.28:0/326716214 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f58dc0a3c40 0x7f58dc0a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:46.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.929+0000 7f58f5e00700 1 --2- 172.21.15.28:0/326716214 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f58dc0a3c40 0x7f58dc0a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f58e8005950 tx=0x7f58e8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:46.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:46.930+0000 7f58eea00700 1 -- 172.21.15.28:0/326716214 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f58f009dd70 con 0x7f58f80790b0 2024-04-24T08:09:47.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:47.172+0000 7f58fd200700 1 -- 172.21.15.28:0/326716214 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7f58f8062f30 con 0x7f58dc0a3c40 2024-04-24T08:09:47.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:47.190+0000 7f58eea00700 1 -- 172.21.15.28:0/326716214 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7f58f8062f30 con 0x7f58dc0a3c40 2024-04-24T08:09:47.191 INFO:teuthology.orchestra.run.smithi028.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-04-24T08:09:47.191 INFO:teuthology.orchestra.run.smithi028.stdout:mds.all 12/12 53s ago 2m smithi028=a;smithi028=d;smithi028=g;smithi028=j;smithi117=b;smithi117=e;smithi117=h;smithi117=k;smithi194=c;smithi194=f;smithi194=i;smithi194=l;count:12 2024-04-24T08:09:47.191 INFO:teuthology.orchestra.run.smithi028.stdout:mgr 3/3 53s ago 8m smithi028=x;smithi117=y;smithi194=z;count:3 2024-04-24T08:09:47.191 INFO:teuthology.orchestra.run.smithi028.stdout:mon 3/3 53s ago 9m smithi028:172.21.15.28=a;smithi117:172.21.15.117=b;smithi194:172.21.15.194=c;count:3 2024-04-24T08:09:47.191 INFO:teuthology.orchestra.run.smithi028.stdout:osd 12 53s ago - 2024-04-24T08:09:47.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:47.193+0000 7f58fd200700 1 -- 172.21.15.28:0/326716214 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f58dc0a3c40 msgr2=0x7f58dc0a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:47.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:47.193+0000 7f58fd200700 1 --2- 172.21.15.28:0/326716214 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f58dc0a3c40 0x7f58dc0a6100 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f58e8005950 tx=0x7f58e8009310 comp rx=0 tx=0).stop 2024-04-24T08:09:47.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:47.193+0000 7f58fd200700 1 -- 172.21.15.28:0/326716214 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58f80790b0 msgr2=0x7f58f81a68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:47.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:47.193+0000 7f58fd200700 1 --2- 172.21.15.28:0/326716214 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58f80790b0 0x7f58f81a68e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f58f0015000 tx=0x7f58f0007c50 comp rx=0 tx=0).stop 2024-04-24T08:09:47.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:47.193+0000 7f58fd200700 1 -- 172.21.15.28:0/326716214 shutdown_connections 2024-04-24T08:09:47.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:47.193+0000 7f58fd200700 1 --2- 172.21.15.28:0/326716214 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f58dc0a3c40 0x7f58dc0a6100 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:47.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:47.193+0000 7f58fd200700 1 --2- 172.21.15.28:0/326716214 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f58f80790b0 0x7f58f81a68e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:47.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:47.193+0000 7f58fd200700 1 --2- 172.21.15.28:0/326716214 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f58f80774a0 0x7f58f819f320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:47.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:47.193+0000 7f58fd200700 1 --2- 172.21.15.28:0/326716214 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58f80786f0 0x7f58f819f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:47.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:47.193+0000 7f58fd200700 1 -- 172.21.15.28:0/326716214 >> 172.21.15.28:0/326716214 conn(0x7f58f8100340 msgr2=0x7f58f8101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:47.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:47.193+0000 7f58fd200700 1 -- 172.21.15.28:0/326716214 shutdown_connections 2024-04-24T08:09:47.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:47.193+0000 7f58fd200700 1 -- 172.21.15.28:0/326716214 wait complete. 2024-04-24T08:09:47.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:47 smithi028 ceph-mon[27786]: pgmap v376: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:09:47.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:47 smithi194 ceph-mon[32836]: pgmap v376: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:09:47.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:47 smithi117 ceph-mon[32438]: pgmap v376: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:09:47.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- bash -c 'ceph orch host ls' 2024-04-24T08:09:48.456 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:09:48.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:48 smithi194 ceph-mon[32836]: from='client.25060 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:09:48.759 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:48 smithi028 ceph-mon[27786]: from='client.25060 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:09:48.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:48 smithi117 ceph-mon[32438]: from='client.25060 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:09:49.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:49 smithi194 ceph-mon[32836]: pgmap v377: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:09:49.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:49 smithi028 ceph-mon[27786]: pgmap v377: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:09:49.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:49 smithi117 ceph-mon[32438]: pgmap v377: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:09:50.435 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:50 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-04-24T08:09:50.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:50 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-04-24T08:09:50.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.731+0000 7fb45fa00700 1 -- 172.21.15.28:0/703124060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb458076ac0 msgr2=0x7fb458074f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.731+0000 7fb45fa00700 1 --2- 172.21.15.28:0/703124060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb458076ac0 0x7fb458074f20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fb44c009990 tx=0x7fb44c009ca0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.731+0000 7fb45fa00700 1 -- 172.21.15.28:0/703124060 shutdown_connections 2024-04-24T08:09:50.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.731+0000 7fb45fa00700 1 --2- 172.21.15.28:0/703124060 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb45810aa30 0x7fb45810cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.731+0000 7fb45fa00700 1 --2- 172.21.15.28:0/703124060 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb458076ac0 0x7fb458074f20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.731+0000 7fb45fa00700 1 --2- 172.21.15.28:0/703124060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4580754f0 0x7fb458075970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.731+0000 7fb45fa00700 1 -- 172.21.15.28:0/703124060 >> 172.21.15.28:0/703124060 conn(0x7fb4580fe0f0 msgr2=0x7fb458100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:50.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.732+0000 7fb45fa00700 1 -- 172.21.15.28:0/703124060 shutdown_connections 2024-04-24T08:09:50.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.732+0000 7fb45fa00700 1 -- 172.21.15.28:0/703124060 wait complete. 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.733+0000 7fb45fa00700 1 Processor -- start 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.733+0000 7fb45fa00700 1 -- start start 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.733+0000 7fb45fa00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4580754f0 0x7fb45819af40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.733+0000 7fb45fa00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb458076ac0 0x7fb45819b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.733+0000 7fb45fa00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb45810aa30 0x7fb4581a2500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.733+0000 7fb45fa00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb458110940 con 0x7fb458076ac0 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.733+0000 7fb45fa00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb4581107c0 con 0x7fb4580754f0 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.733+0000 7fb45fa00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb458110ac0 con 0x7fb45810aa30 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.734+0000 7fb457e00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb458076ac0 0x7fb45819b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.734+0000 7fb45d000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4580754f0 0x7fb45819af40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.734+0000 7fb45d000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4580754f0 0x7fb45819af40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:38300/0 (socket says 172.21.15.28:38300) 2024-04-24T08:09:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.734+0000 7fb45d000700 1 -- 172.21.15.28:0/2836622680 learned_addr learned my addr 172.21.15.28:0/2836622680 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.734+0000 7fb45da00700 1 --2- 172.21.15.28:0/2836622680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb45810aa30 0x7fb4581a2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.734+0000 7fb457e00700 1 -- 172.21.15.28:0/2836622680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb45810aa30 msgr2=0x7fb4581a2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.734+0000 7fb457e00700 1 --2- 172.21.15.28:0/2836622680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb45810aa30 0x7fb4581a2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.734+0000 7fb457e00700 1 -- 172.21.15.28:0/2836622680 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4580754f0 msgr2=0x7fb45819af40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.734+0000 7fb457e00700 1 --2- 172.21.15.28:0/2836622680 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4580754f0 0x7fb45819af40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.734+0000 7fb457e00700 1 -- 172.21.15.28:0/2836622680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4581a2ba0 con 0x7fb458076ac0 2024-04-24T08:09:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.734+0000 7fb45d000700 1 --2- 172.21.15.28:0/2836622680 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4580754f0 0x7fb45819af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.734+0000 7fb45da00700 1 --2- 172.21.15.28:0/2836622680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb45810aa30 0x7fb4581a2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:50.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.735+0000 7fb457e00700 1 --2- 172.21.15.28:0/2836622680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb458076ac0 0x7fb45819b480 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb44400ea20 tx=0x7fb44400ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.735+0000 7fb455600700 1 -- 172.21.15.28:0/2836622680 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb44400cbd0 con 0x7fb458076ac0 2024-04-24T08:09:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.735+0000 7fb455600700 1 -- 172.21.15.28:0/2836622680 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb44400ee60 con 0x7fb458076ac0 2024-04-24T08:09:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.735+0000 7fb45fa00700 1 -- 172.21.15.28:0/2836622680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4581a2e90 con 0x7fb458076ac0 2024-04-24T08:09:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.735+0000 7fb455600700 1 -- 172.21.15.28:0/2836622680 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb444018680 con 0x7fb458076ac0 2024-04-24T08:09:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.735+0000 7fb45fa00700 1 -- 172.21.15.28:0/2836622680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4581a3340 con 0x7fb458076ac0 2024-04-24T08:09:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.736+0000 7fb45fa00700 1 -- 172.21.15.28:0/2836622680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb4580697e0 con 0x7fb458076ac0 2024-04-24T08:09:50.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.739+0000 7fb455600700 1 -- 172.21.15.28:0/2836622680 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb444018820 con 0x7fb458076ac0 2024-04-24T08:09:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.741+0000 7fb455600700 1 --2- 172.21.15.28:0/2836622680 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb4400a3c40 0x7fb4400a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.741+0000 7fb455600700 1 -- 172.21.15.28:0/2836622680 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8298+0+0 (secure 0 0 0) 0x7fb4440d09d0 con 0x7fb458076ac0 2024-04-24T08:09:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.741+0000 7fb45d000700 1 --2- 172.21.15.28:0/2836622680 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb4400a3c40 0x7fb4400a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:50.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.742+0000 7fb45d000700 1 --2- 172.21.15.28:0/2836622680 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb4400a3c40 0x7fb4400a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00b2e0 tx=0x7fb44c00b880 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:50.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.743+0000 7fb455600700 1 -- 172.21.15.28:0/2836622680 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb44409d310 con 0x7fb458076ac0 2024-04-24T08:09:50.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:50 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]: dispatch 2024-04-24T08:09:50.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.981+0000 7fb45fa00700 1 -- 172.21.15.28:0/2836622680 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7fb458062f30 con 0x7fb4400a3c40 2024-04-24T08:09:50.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.986+0000 7fb455600700 1 -- 172.21.15.28:0/2836622680 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7fb458062f30 con 0x7fb4400a3c40 2024-04-24T08:09:50.987 INFO:teuthology.orchestra.run.smithi028.stdout:HOST ADDR LABELS STATUS 2024-04-24T08:09:50.987 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 172.21.15.28 2024-04-24T08:09:50.987 INFO:teuthology.orchestra.run.smithi028.stdout:smithi117 172.21.15.117 2024-04-24T08:09:50.987 INFO:teuthology.orchestra.run.smithi028.stdout:smithi194 172.21.15.194 2024-04-24T08:09:50.987 INFO:teuthology.orchestra.run.smithi028.stdout:3 hosts in cluster 2024-04-24T08:09:50.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.989+0000 7fb45fa00700 1 -- 172.21.15.28:0/2836622680 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb4400a3c40 msgr2=0x7fb4400a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.989+0000 7fb45fa00700 1 --2- 172.21.15.28:0/2836622680 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb4400a3c40 0x7fb4400a6100 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fb44c00b2e0 tx=0x7fb44c00b880 comp rx=0 tx=0).stop 2024-04-24T08:09:50.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.989+0000 7fb45fa00700 1 -- 172.21.15.28:0/2836622680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb458076ac0 msgr2=0x7fb45819b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:50.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.989+0000 7fb45fa00700 1 --2- 172.21.15.28:0/2836622680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb458076ac0 0x7fb45819b480 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb44400ea20 tx=0x7fb44400ed30 comp rx=0 tx=0).stop 2024-04-24T08:09:50.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.989+0000 7fb45fa00700 1 -- 172.21.15.28:0/2836622680 shutdown_connections 2024-04-24T08:09:50.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.989+0000 7fb45fa00700 1 --2- 172.21.15.28:0/2836622680 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb4400a3c40 0x7fb4400a6100 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.989+0000 7fb45fa00700 1 --2- 172.21.15.28:0/2836622680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb45810aa30 0x7fb4581a2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.989+0000 7fb45fa00700 1 --2- 172.21.15.28:0/2836622680 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4580754f0 0x7fb45819af40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.989+0000 7fb45fa00700 1 --2- 172.21.15.28:0/2836622680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb458076ac0 0x7fb45819b480 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:50.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.989+0000 7fb45fa00700 1 -- 172.21.15.28:0/2836622680 >> 172.21.15.28:0/2836622680 conn(0x7fb4580fe0f0 msgr2=0x7fb458106980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:50.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.990+0000 7fb45fa00700 1 -- 172.21.15.28:0/2836622680 shutdown_connections 2024-04-24T08:09:50.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:50.990+0000 7fb45fa00700 1 -- 172.21.15.28:0/2836622680 wait complete. 2024-04-24T08:09:51.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:51 smithi194 ceph-mon[32836]: pgmap v378: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:09:51.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:51 smithi194 ceph-mon[32836]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-04-24T08:09:51.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:51 smithi194 ceph-mon[32836]: osdmap e89: 12 total, 12 up, 12 in 2024-04-24T08:09:51.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:51 smithi028 ceph-mon[27786]: pgmap v378: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:09:51.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:51 smithi028 ceph-mon[27786]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-04-24T08:09:51.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:51 smithi028 ceph-mon[27786]: osdmap e89: 12 total, 12 up, 12 in 2024-04-24T08:09:51.852 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- bash -c 'ceph orch device ls' 2024-04-24T08:09:51.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:51 smithi117 ceph-mon[32438]: pgmap v378: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:09:51.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:51 smithi117 ceph-mon[32438]: from='mgr.14152 172.21.15.28:0/718841999' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.22", "id": [3, 9]}]': finished 2024-04-24T08:09:51.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:51 smithi117 ceph-mon[32438]: osdmap e89: 12 total, 12 up, 12 in 2024-04-24T08:09:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:09:52.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:52 smithi194 ceph-mon[32836]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:09:52.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:52 smithi194 ceph-mon[32836]: osdmap e90: 12 total, 12 up, 12 in 2024-04-24T08:09:52.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:52 smithi028 ceph-mon[27786]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:09:52.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:52 smithi028 ceph-mon[27786]: osdmap e90: 12 total, 12 up, 12 in 2024-04-24T08:09:52.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:52 smithi117 ceph-mon[32438]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:09:52.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:52 smithi117 ceph-mon[32438]: osdmap e90: 12 total, 12 up, 12 in 2024-04-24T08:09:53.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:53 smithi028 ceph-mon[27786]: pgmap v381: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-24T08:09:53.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:53 smithi117 ceph-mon[32438]: pgmap v381: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-24T08:09:53.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:53 smithi194 ceph-mon[32836]: pgmap v381: 97 pgs: 97 active+clean; 639 KiB data, 356 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 KiB/s rd, 3 op/s 2024-04-24T08:09:54.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.598+0000 7f8087400700 1 -- 172.21.15.28:0/3981948535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8080103e80 msgr2=0x7f80801042e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:54.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.598+0000 7f8087400700 1 --2- 172.21.15.28:0/3981948535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8080103e80 0x7f80801042e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f806c0098f0 tx=0x7f806c009c00 comp rx=0 tx=0).stop 2024-04-24T08:09:54.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.598+0000 7f8087400700 1 -- 172.21.15.28:0/3981948535 shutdown_connections 2024-04-24T08:09:54.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.598+0000 7f8087400700 1 --2- 172.21.15.28:0/3981948535 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8080104820 0x7f808010b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.598+0000 7f8087400700 1 --2- 172.21.15.28:0/3981948535 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080100e00 0x7f8080101220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.598+0000 7f8087400700 1 --2- 172.21.15.28:0/3981948535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8080103e80 0x7f80801042e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.598+0000 7f8087400700 1 -- 172.21.15.28:0/3981948535 >> 172.21.15.28:0/3981948535 conn(0x7f80800fc780 msgr2=0x7f80800feba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:54.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.598+0000 7f8087400700 1 -- 172.21.15.28:0/3981948535 shutdown_connections 2024-04-24T08:09:54.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.598+0000 7f8087400700 1 -- 172.21.15.28:0/3981948535 wait complete. 2024-04-24T08:09:54.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.600+0000 7f8087400700 1 Processor -- start 2024-04-24T08:09:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.600+0000 7f8087400700 1 -- start start 2024-04-24T08:09:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.600+0000 7f8087400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8080100e00 0x7f808019aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.600+0000 7f8087400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080103e80 0x7f808019b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.600+0000 7f8087400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8080104820 0x7f80801a25b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.600+0000 7f8087400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f808010e740 con 0x7f8080103e80 2024-04-24T08:09:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.600+0000 7f8087400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f808010e5c0 con 0x7f8080100e00 2024-04-24T08:09:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.600+0000 7f8087400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f808010e8c0 con 0x7f8080104820 2024-04-24T08:09:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.601+0000 7f8084a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8080100e00 0x7f808019aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.601+0000 7f807fe00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080103e80 0x7f808019b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.601+0000 7f8084a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8080100e00 0x7f808019aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:38314/0 (socket says 172.21.15.28:38314) 2024-04-24T08:09:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.601+0000 7f8084a00700 1 -- 172.21.15.28:0/3142795693 learned_addr learned my addr 172.21.15.28:0/3142795693 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:54.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.601+0000 7f8085400700 1 --2- 172.21.15.28:0/3142795693 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8080104820 0x7f80801a25b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:54.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.601+0000 7f8084a00700 1 -- 172.21.15.28:0/3142795693 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8080104820 msgr2=0x7f80801a25b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:54.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.601+0000 7f8084a00700 1 --2- 172.21.15.28:0/3142795693 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8080104820 0x7f80801a25b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.601+0000 7f8084a00700 1 -- 172.21.15.28:0/3142795693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080103e80 msgr2=0x7f808019b530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:54.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.601+0000 7f8084a00700 1 --2- 172.21.15.28:0/3142795693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080103e80 0x7f808019b530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.601+0000 7f8084a00700 1 -- 172.21.15.28:0/3142795693 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80801a2c50 con 0x7f8080100e00 2024-04-24T08:09:54.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.601+0000 7f8084a00700 1 --2- 172.21.15.28:0/3142795693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8080100e00 0x7f808019aff0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f807400e970 tx=0x7f807400ec80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:54.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.602+0000 7f807d600700 1 -- 172.21.15.28:0/3142795693 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f807400cbd0 con 0x7f8080100e00 2024-04-24T08:09:54.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.602+0000 7f807d600700 1 -- 172.21.15.28:0/3142795693 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8074004500 con 0x7f8080100e00 2024-04-24T08:09:54.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.602+0000 7f807d600700 1 -- 172.21.15.28:0/3142795693 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8074003e30 con 0x7f8080100e00 2024-04-24T08:09:54.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.602+0000 7f8087400700 1 -- 172.21.15.28:0/3142795693 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80801a2f40 con 0x7f8080100e00 2024-04-24T08:09:54.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.602+0000 7f8087400700 1 -- 172.21.15.28:0/3142795693 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80801a3340 con 0x7f8080100e00 2024-04-24T08:09:54.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.603+0000 7f8087400700 1 -- 172.21.15.28:0/3142795693 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80801a35d0 con 0x7f8080100e00 2024-04-24T08:09:54.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.605+0000 7f807d600700 1 -- 172.21.15.28:0/3142795693 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f8074004000 con 0x7f8080100e00 2024-04-24T08:09:54.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.607+0000 7f807d600700 1 --2- 172.21.15.28:0/3142795693 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f80680a3cc0 0x7f80680a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:54.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.607+0000 7f807d600700 1 -- 172.21.15.28:0/3142795693 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f80740d0740 con 0x7f8080100e00 2024-04-24T08:09:54.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.607+0000 7f807fe00700 1 --2- 172.21.15.28:0/3142795693 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f80680a3cc0 0x7f80680a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:54.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.608+0000 7f807fe00700 1 --2- 172.21.15.28:0/3142795693 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f80680a3cc0 0x7f80680a6180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f806c00fb70 tx=0x7f806c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:54.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.611+0000 7f807d600700 1 -- 172.21.15.28:0/3142795693 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f807409d080 con 0x7f8080100e00 2024-04-24T08:09:54.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.856+0000 7f8087400700 1 -- 172.21.15.28:0/3142795693 --> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f8080103390 con 0x7f80680a3cc0 2024-04-24T08:09:54.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.860+0000 7f807d600700 1 -- 172.21.15.28:0/3142795693 <== mgr.14152 v2:172.21.15.28:6800/361125225 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f8080103390 con 0x7f80680a3cc0 2024-04-24T08:09:54.862 INFO:teuthology.orchestra.run.smithi028.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-04-24T08:09:54.862 INFO:teuthology.orchestra.run.smithi028.stdout:smithi028 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747401F4480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-04-24T08:09:54.862 INFO:teuthology.orchestra.run.smithi028.stdout:smithi117 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400V6400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-04-24T08:09:54.862 INFO:teuthology.orchestra.run.smithi028.stdout:smithi194 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6240002A400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-04-24T08:09:54.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.864+0000 7f8087400700 1 -- 172.21.15.28:0/3142795693 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f80680a3cc0 msgr2=0x7f80680a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:54.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.864+0000 7f8087400700 1 --2- 172.21.15.28:0/3142795693 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f80680a3cc0 0x7f80680a6180 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f806c00fb70 tx=0x7f806c019040 comp rx=0 tx=0).stop 2024-04-24T08:09:54.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.864+0000 7f8087400700 1 -- 172.21.15.28:0/3142795693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8080100e00 msgr2=0x7f808019aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:54.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.864+0000 7f8087400700 1 --2- 172.21.15.28:0/3142795693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8080100e00 0x7f808019aff0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f807400e970 tx=0x7f807400ec80 comp rx=0 tx=0).stop 2024-04-24T08:09:54.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.864+0000 7f8087400700 1 -- 172.21.15.28:0/3142795693 shutdown_connections 2024-04-24T08:09:54.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.864+0000 7f8087400700 1 --2- 172.21.15.28:0/3142795693 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f80680a3cc0 0x7f80680a6180 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.864+0000 7f8087400700 1 --2- 172.21.15.28:0/3142795693 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8080104820 0x7f80801a25b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.864+0000 7f8087400700 1 --2- 172.21.15.28:0/3142795693 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8080100e00 0x7f808019aff0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.864+0000 7f8087400700 1 --2- 172.21.15.28:0/3142795693 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8080103e80 0x7f808019b530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:54.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.864+0000 7f8087400700 1 -- 172.21.15.28:0/3142795693 >> 172.21.15.28:0/3142795693 conn(0x7f80800fc780 msgr2=0x7f8080101c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:54.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.864+0000 7f8087400700 1 -- 172.21.15.28:0/3142795693 shutdown_connections 2024-04-24T08:09:54.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:54.864+0000 7f8087400700 1 -- 172.21.15.28:0/3142795693 wait complete. 2024-04-24T08:09:55.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:55 smithi028 ceph-mon[27786]: pgmap v382: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:09:55.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:55 smithi028 ceph-mon[27786]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:09:55.712 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-04-24T08:09:55.721 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-24T08:09:55.721 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- bash -c 'ceph fs dump' 2024-04-24T08:09:55.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:55 smithi117 ceph-mon[32438]: pgmap v382: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:09:55.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:55 smithi117 ceph-mon[32438]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:09:55.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:55 smithi194 ceph-mon[32836]: pgmap v382: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:09:55.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:55 smithi194 ceph-mon[32836]: from='client.25046 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:09:56.189 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:09:57.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:57 smithi028 ceph-mon[27786]: pgmap v383: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:09:57.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:57 smithi117 ceph-mon[32438]: pgmap v383: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:09:57.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:57 smithi194 ceph-mon[32836]: pgmap v383: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:09:58.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.516+0000 7fc10a200700 1 -- 172.21.15.28:0/1359050947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc10410f130 msgr2=0x7fc104111520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.516+0000 7fc10a200700 1 --2- 172.21.15.28:0/1359050947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc10410f130 0x7fc104111520 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fc0fc00b0a0 tx=0x7fc0fc00b3b0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.516+0000 7fc10a200700 1 -- 172.21.15.28:0/1359050947 shutdown_connections 2024-04-24T08:09:58.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.516+0000 7fc10a200700 1 --2- 172.21.15.28:0/1359050947 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc10410f130 0x7fc104111520 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.516+0000 7fc10a200700 1 --2- 172.21.15.28:0/1359050947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc104076b20 0x7fc104074f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.516+0000 7fc10a200700 1 --2- 172.21.15.28:0/1359050947 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc104075550 0x7fc1040759d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.516+0000 7fc10a200700 1 -- 172.21.15.28:0/1359050947 >> 172.21.15.28:0/1359050947 conn(0x7fc1040fe0f0 msgr2=0x7fc104100530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:58.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.516+0000 7fc10a200700 1 -- 172.21.15.28:0/1359050947 shutdown_connections 2024-04-24T08:09:58.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.516+0000 7fc10a200700 1 -- 172.21.15.28:0/1359050947 wait complete. 2024-04-24T08:09:58.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.518+0000 7fc10a200700 1 Processor -- start 2024-04-24T08:09:58.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.518+0000 7fc10a200700 1 -- start start 2024-04-24T08:09:58.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.518+0000 7fc10a200700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc104075550 0x7fc10419f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.519+0000 7fc10a200700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc104076b20 0x7fc10419f890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.519+0000 7fc10a200700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc10410f130 0x7fc1041a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.519+0000 7fc10a200700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc104114d90 con 0x7fc104075550 2024-04-24T08:09:58.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.519+0000 7fc10a200700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc104114c10 con 0x7fc104076b20 2024-04-24T08:09:58.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.519+0000 7fc10a200700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc104114f10 con 0x7fc10410f130 2024-04-24T08:09:58.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.519+0000 7fc103400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc104075550 0x7fc10419f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:58.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.519+0000 7fc102a00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc104076b20 0x7fc10419f890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:58.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.519+0000 7fc103400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc104075550 0x7fc10419f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59688/0 (socket says 172.21.15.28:59688) 2024-04-24T08:09:58.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.519+0000 7fc103400700 1 -- 172.21.15.28:0/330214475 learned_addr learned my addr 172.21.15.28:0/330214475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:09:58.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.519+0000 7fc103e00700 1 --2- 172.21.15.28:0/330214475 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc10410f130 0x7fc1041a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:58.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.520+0000 7fc102a00700 1 -- 172.21.15.28:0/330214475 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc10410f130 msgr2=0x7fc1041a6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.520+0000 7fc102a00700 1 --2- 172.21.15.28:0/330214475 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc10410f130 0x7fc1041a6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.520+0000 7fc102a00700 1 -- 172.21.15.28:0/330214475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc104075550 msgr2=0x7fc10419f350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.520+0000 7fc102a00700 1 --2- 172.21.15.28:0/330214475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc104075550 0x7fc10419f350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.520+0000 7fc102a00700 1 -- 172.21.15.28:0/330214475 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc1041a6fb0 con 0x7fc104076b20 2024-04-24T08:09:58.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.520+0000 7fc103400700 1 --2- 172.21.15.28:0/330214475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc104075550 0x7fc10419f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:09:58.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.520+0000 7fc102a00700 1 --2- 172.21.15.28:0/330214475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc104076b20 0x7fc10419f890 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc0f000e9c0 tx=0x7fc0f000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:58.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.520+0000 7fc0fbe00700 1 -- 172.21.15.28:0/330214475 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f0004220 con 0x7fc104076b20 2024-04-24T08:09:58.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.520+0000 7fc10a200700 1 -- 172.21.15.28:0/330214475 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc1041a72a0 con 0x7fc104076b20 2024-04-24T08:09:58.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.520+0000 7fc0fbe00700 1 -- 172.21.15.28:0/330214475 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0f00043c0 con 0x7fc104076b20 2024-04-24T08:09:58.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.520+0000 7fc0fbe00700 1 -- 172.21.15.28:0/330214475 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f0003d80 con 0x7fc104076b20 2024-04-24T08:09:58.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.521+0000 7fc10a200700 1 -- 172.21.15.28:0/330214475 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc1041a7830 con 0x7fc104076b20 2024-04-24T08:09:58.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.522+0000 7fc10a200700 1 -- 172.21.15.28:0/330214475 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc1040697e0 con 0x7fc104076b20 2024-04-24T08:09:58.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.524+0000 7fc0fbe00700 1 -- 172.21.15.28:0/330214475 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc0f000c7c0 con 0x7fc104076b20 2024-04-24T08:09:58.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.525+0000 7fc0fbe00700 1 --2- 172.21.15.28:0/330214475 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc0e80a3cc0 0x7fc0e80a6180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:09:58.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.525+0000 7fc0fbe00700 1 -- 172.21.15.28:0/330214475 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7fc0f00d0620 con 0x7fc104076b20 2024-04-24T08:09:58.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.525+0000 7fc103400700 1 --2- 172.21.15.28:0/330214475 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc0e80a3cc0 0x7fc0e80a6180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:09:58.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.526+0000 7fc103400700 1 --2- 172.21.15.28:0/330214475 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc0e80a3cc0 0x7fc0e80a6180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4005c50 tx=0x7fc0f4007480 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:09:58.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.529+0000 7fc0fbe00700 1 -- 172.21.15.28:0/330214475 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc0f009cf60 con 0x7fc104076b20 2024-04-24T08:09:58.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.810+0000 7fc10a200700 1 -- 172.21.15.28:0/330214475 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fc104063700 con 0x7fc104076b20 2024-04-24T08:09:58.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.811+0000 7fc0fbe00700 1 -- 172.21.15.28:0/330214475 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 23 v23) ==== 76+0+3050 (secure 0 0 0) 0x7fc0f009c860 con 0x7fc104076b20 2024-04-24T08:09:58.812 INFO:teuthology.orchestra.run.smithi028.stdout:e23 2024-04-24T08:09:58.813 INFO:teuthology.orchestra.run.smithi028.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-04-24T08:09:58.813 INFO:teuthology.orchestra.run.smithi028.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-24T08:09:58.813 INFO:teuthology.orchestra.run.smithi028.stdout:legacy client fscid: 1 2024-04-24T08:09:58.813 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:09:58.813 INFO:teuthology.orchestra.run.smithi028.stdout:Filesystem 'cephfs' (1) 2024-04-24T08:09:58.813 INFO:teuthology.orchestra.run.smithi028.stdout:fs_name cephfs 2024-04-24T08:09:58.813 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 23 2024-04-24T08:09:58.813 INFO:teuthology.orchestra.run.smithi028.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-04-24T08:09:58.813 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-04-24T08:07:51.585580+0000 2024-04-24T08:09:58.813 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-04-24T08:07:57.133790+0000 2024-04-24T08:09:58.813 INFO:teuthology.orchestra.run.smithi028.stdout:tableserver 0 2024-04-24T08:09:58.814 INFO:teuthology.orchestra.run.smithi028.stdout:root 0 2024-04-24T08:09:58.814 INFO:teuthology.orchestra.run.smithi028.stdout:session_timeout 300 2024-04-24T08:09:58.814 INFO:teuthology.orchestra.run.smithi028.stdout:session_autoclose 300 2024-04-24T08:09:58.814 INFO:teuthology.orchestra.run.smithi028.stdout:max_file_size 1099511627776 2024-04-24T08:09:58.814 INFO:teuthology.orchestra.run.smithi028.stdout:max_xattr_size 65536 2024-04-24T08:09:58.814 INFO:teuthology.orchestra.run.smithi028.stdout:required_client_features {} 2024-04-24T08:09:58.814 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure 0 2024-04-24T08:09:58.814 INFO:teuthology.orchestra.run.smithi028.stdout:last_failure_osd_epoch 0 2024-04-24T08:09:58.814 INFO:teuthology.orchestra.run.smithi028.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-04-24T08:09:58.814 INFO:teuthology.orchestra.run.smithi028.stdout:max_mds 3 2024-04-24T08:09:58.814 INFO:teuthology.orchestra.run.smithi028.stdout:in 0,1,2 2024-04-24T08:09:58.814 INFO:teuthology.orchestra.run.smithi028.stdout:up {0=24574,1=24572,2=24556} 2024-04-24T08:09:58.814 INFO:teuthology.orchestra.run.smithi028.stdout:failed 2024-04-24T08:09:58.815 INFO:teuthology.orchestra.run.smithi028.stdout:damaged 2024-04-24T08:09:58.815 INFO:teuthology.orchestra.run.smithi028.stdout:stopped 2024-04-24T08:09:58.815 INFO:teuthology.orchestra.run.smithi028.stdout:data_pools [3] 2024-04-24T08:09:58.815 INFO:teuthology.orchestra.run.smithi028.stdout:metadata_pool 2 2024-04-24T08:09:58.815 INFO:teuthology.orchestra.run.smithi028.stdout:inline_data disabled 2024-04-24T08:09:58.815 INFO:teuthology.orchestra.run.smithi028.stdout:balancer 2024-04-24T08:09:58.815 INFO:teuthology.orchestra.run.smithi028.stdout:bal_rank_mask -1 2024-04-24T08:09:58.815 INFO:teuthology.orchestra.run.smithi028.stdout:standby_count_wanted 0 2024-04-24T08:09:58.815 INFO:teuthology.orchestra.run.smithi028.stdout:qdb_cluster leader: 24574 members: 24556,24572,24574 2024-04-24T08:09:58.815 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.g{0:24574} state up:active seq 5 addr [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:09:58.815 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.c{0:24535} state up:standby-replay seq 1 addr [v2:172.21.15.194:6836/3046177244,v1:172.21.15.194:6837/3046177244] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:09:58.815 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.b{1:24572} state up:active seq 4 addr [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:09:58.816 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.f{1:24523} state up:standby-replay seq 1 addr [v2:172.21.15.194:6834/356867408,v1:172.21.15.194:6835/356867408] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:09:58.816 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.i{2:24556} state up:active seq 7 addr [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:09:58.816 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.h{2:24515} state up:standby-replay seq 1 addr [v2:172.21.15.117:6836/862115546,v1:172.21.15.117:6837/862115546] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:09:58.816 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:09:58.816 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:09:58.816 INFO:teuthology.orchestra.run.smithi028.stdout:Standby daemons: 2024-04-24T08:09:58.816 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:09:58.816 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.j{-1:14385} state up:standby seq 1 addr [v2:172.21.15.28:6834/1751430859,v1:172.21.15.28:6835/1751430859] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:09:58.816 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.a{-1:14466} state up:standby seq 1 addr [v2:172.21.15.28:6840/666043045,v1:172.21.15.28:6841/666043045] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:09:58.816 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.k{-1:24434} state up:standby seq 1 addr [v2:172.21.15.117:6832/4248331494,v1:172.21.15.117:6833/4248331494] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:09:58.816 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.l{-1:24466} state up:standby seq 1 addr [v2:172.21.15.194:6832/2281745957,v1:172.21.15.194:6833/2281745957] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:09:58.816 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.d{-1:24470} state up:standby seq 1 addr [v2:172.21.15.28:6836/711950790,v1:172.21.15.28:6837/711950790] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:09:58.816 INFO:teuthology.orchestra.run.smithi028.stdout:[mds.e{-1:24482} state up:standby seq 1 addr [v2:172.21.15.117:6834/2458524548,v1:172.21.15.117:6835/2458524548] compat {c=[1],r=[1],i=[1fff]}] 2024-04-24T08:09:58.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.814+0000 7fc10a200700 1 -- 172.21.15.28:0/330214475 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc0e80a3cc0 msgr2=0x7fc0e80a6180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.814+0000 7fc10a200700 1 --2- 172.21.15.28:0/330214475 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc0e80a3cc0 0x7fc0e80a6180 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc0f4005c50 tx=0x7fc0f4007480 comp rx=0 tx=0).stop 2024-04-24T08:09:58.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.814+0000 7fc10a200700 1 -- 172.21.15.28:0/330214475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc104076b20 msgr2=0x7fc10419f890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:09:58.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.814+0000 7fc10a200700 1 --2- 172.21.15.28:0/330214475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc104076b20 0x7fc10419f890 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fc0f000e9c0 tx=0x7fc0f000ecd0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.814+0000 7fc10a200700 1 -- 172.21.15.28:0/330214475 shutdown_connections 2024-04-24T08:09:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.814+0000 7fc10a200700 1 --2- 172.21.15.28:0/330214475 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc0e80a3cc0 0x7fc0e80a6180 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.814+0000 7fc10a200700 1 --2- 172.21.15.28:0/330214475 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc10410f130 0x7fc1041a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.814+0000 7fc10a200700 1 --2- 172.21.15.28:0/330214475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc104075550 0x7fc10419f350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.814+0000 7fc10a200700 1 --2- 172.21.15.28:0/330214475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc104076b20 0x7fc10419f890 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:09:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.814+0000 7fc10a200700 1 -- 172.21.15.28:0/330214475 >> 172.21.15.28:0/330214475 conn(0x7fc1040fe0f0 msgr2=0x7fc1041024d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:09:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.814+0000 7fc10a200700 1 -- 172.21.15.28:0/330214475 shutdown_connections 2024-04-24T08:09:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:09:58.814+0000 7fc10a200700 1 -- 172.21.15.28:0/330214475 wait complete. 2024-04-24T08:09:58.818 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 23 2024-04-24T08:09:59.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- bash -c 'ceph osd dump' 2024-04-24T08:09:59.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:59 smithi028 ceph-mon[27786]: pgmap v384: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-24T08:09:59.713 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:09:59 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/330214475' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-24T08:09:59.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:59 smithi117 ceph-mon[32438]: pgmap v384: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-24T08:09:59.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:09:59 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/330214475' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-24T08:09:59.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:59 smithi194 ceph-mon[32836]: pgmap v384: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-24T08:09:59.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:09:59 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/330214475' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-04-24T08:10:00.108 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:10:00.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:00 smithi028 ceph-mon[27786]: overall HEALTH_OK 2024-04-24T08:10:00.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:00 smithi117 ceph-mon[32438]: overall HEALTH_OK 2024-04-24T08:10:00.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:00 smithi194 ceph-mon[32836]: overall HEALTH_OK 2024-04-24T08:10:01.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:01 smithi028 ceph-mon[27786]: pgmap v385: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-24T08:10:01.842 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:01 smithi194 ceph-mon[32836]: pgmap v385: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-24T08:10:01.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:01 smithi117 ceph-mon[32438]: pgmap v385: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-24T08:10:02.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.398+0000 7fb284400700 1 -- 172.21.15.28:0/1815892288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb27c105f90 msgr2=0x7fb27c1063f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:02.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.399+0000 7fb284400700 1 --2- 172.21.15.28:0/1815892288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb27c105f90 0x7fb27c1063f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb2680099c0 tx=0x7fb268009cd0 comp rx=0 tx=0).stop 2024-04-24T08:10:02.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.399+0000 7fb284400700 1 -- 172.21.15.28:0/1815892288 shutdown_connections 2024-04-24T08:10:02.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.399+0000 7fb284400700 1 --2- 172.21.15.28:0/1815892288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb27c106930 0x7fb27c10d1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:02.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.399+0000 7fb284400700 1 --2- 172.21.15.28:0/1815892288 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb27c104d90 0x7fb27c1051b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:02.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.399+0000 7fb284400700 1 --2- 172.21.15.28:0/1815892288 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb27c105f90 0x7fb27c1063f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:02.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.399+0000 7fb284400700 1 -- 172.21.15.28:0/1815892288 >> 172.21.15.28:0/1815892288 conn(0x7fb27c100350 msgr2=0x7fb27c102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:02.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.399+0000 7fb284400700 1 -- 172.21.15.28:0/1815892288 shutdown_connections 2024-04-24T08:10:02.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.399+0000 7fb284400700 1 -- 172.21.15.28:0/1815892288 wait complete. 2024-04-24T08:10:02.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.400+0000 7fb284400700 1 Processor -- start 2024-04-24T08:10:02.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.401+0000 7fb284400700 1 -- start start 2024-04-24T08:10:02.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.401+0000 7fb284400700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb27c104d90 0x7fb27c19af50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:02.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.401+0000 7fb284400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb27c105f90 0x7fb27c19b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:02.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.401+0000 7fb284400700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb27c106930 0x7fb27c1a2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:02.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.401+0000 7fb284400700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c1109a0 con 0x7fb27c106930 2024-04-24T08:10:02.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.401+0000 7fb284400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c110820 con 0x7fb27c105f90 2024-04-24T08:10:02.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.401+0000 7fb284400700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb27c110b20 con 0x7fb27c104d90 2024-04-24T08:10:02.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.401+0000 7fb281000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb27c105f90 0x7fb27c19b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:02.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.401+0000 7fb281000700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb27c105f90 0x7fb27c19b490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:52342/0 (socket says 172.21.15.28:52342) 2024-04-24T08:10:02.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.401+0000 7fb281000700 1 -- 172.21.15.28:0/3939570403 learned_addr learned my addr 172.21.15.28:0/3939570403 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:02.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.401+0000 7fb281a00700 1 --2- 172.21.15.28:0/3939570403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb27c104d90 0x7fb27c19af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:02.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.401+0000 7fb282400700 1 --2- 172.21.15.28:0/3939570403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb27c106930 0x7fb27c1a2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:02.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.402+0000 7fb281a00700 1 -- 172.21.15.28:0/3939570403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb27c105f90 msgr2=0x7fb27c19b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:02.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.402+0000 7fb281a00700 1 --2- 172.21.15.28:0/3939570403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb27c105f90 0x7fb27c19b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:02.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.402+0000 7fb281a00700 1 -- 172.21.15.28:0/3939570403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb27c106930 msgr2=0x7fb27c1a2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:02.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.402+0000 7fb281a00700 1 --2- 172.21.15.28:0/3939570403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb27c106930 0x7fb27c1a2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:02.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.402+0000 7fb281a00700 1 -- 172.21.15.28:0/3939570403 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb27c1a2bb0 con 0x7fb27c104d90 2024-04-24T08:10:02.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.402+0000 7fb282400700 1 --2- 172.21.15.28:0/3939570403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb27c106930 0x7fb27c1a2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:10:02.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.402+0000 7fb281a00700 1 --2- 172.21.15.28:0/3939570403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb27c104d90 0x7fb27c19af50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb27000d770 tx=0x7fb27000da80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:02.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.403+0000 7fb27a000700 1 -- 172.21.15.28:0/3939570403 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb270017070 con 0x7fb27c104d90 2024-04-24T08:10:02.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.403+0000 7fb284400700 1 -- 172.21.15.28:0/3939570403 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb27c1a2ea0 con 0x7fb27c104d90 2024-04-24T08:10:02.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.403+0000 7fb27a000700 1 -- 172.21.15.28:0/3939570403 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb27000fb70 con 0x7fb27c104d90 2024-04-24T08:10:02.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.403+0000 7fb27a000700 1 -- 172.21.15.28:0/3939570403 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb270003b90 con 0x7fb27c104d90 2024-04-24T08:10:02.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.403+0000 7fb284400700 1 -- 172.21.15.28:0/3939570403 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb27c1a3300 con 0x7fb27c104d90 2024-04-24T08:10:02.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.404+0000 7fb284400700 1 -- 172.21.15.28:0/3939570403 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb27c0697e0 con 0x7fb27c104d90 2024-04-24T08:10:02.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.406+0000 7fb27a000700 1 -- 172.21.15.28:0/3939570403 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fb270003d30 con 0x7fb27c104d90 2024-04-24T08:10:02.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.408+0000 7fb27a000700 1 --2- 172.21.15.28:0/3939570403 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2640a3c40 0x7fb2640a6100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:02.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.408+0000 7fb27a000700 1 -- 172.21.15.28:0/3939570403 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7fb2700cfda0 con 0x7fb27c104d90 2024-04-24T08:10:02.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.408+0000 7fb281000700 1 --2- 172.21.15.28:0/3939570403 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2640a3c40 0x7fb2640a6100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:02.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.409+0000 7fb281000700 1 --2- 172.21.15.28:0/3939570403 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2640a3c40 0x7fb2640a6100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb268000c00 tx=0x7fb268005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:02.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.410+0000 7fb27a000700 1 -- 172.21.15.28:0/3939570403 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fb27009c6e0 con 0x7fb27c104d90 2024-04-24T08:10:02.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.646+0000 7fb284400700 1 -- 172.21.15.28:0/3939570403 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7fb27c02e0f0 con 0x7fb27c104d90 2024-04-24T08:10:02.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.647+0000 7fb27a000700 1 -- 172.21.15.28:0/3939570403 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) ==== 56+0+5356 (secure 0 0 0) 0x7fb27009bfe0 con 0x7fb27c104d90 2024-04-24T08:10:02.650 INFO:teuthology.orchestra.run.smithi028.stdout:epoch 90 2024-04-24T08:10:02.650 INFO:teuthology.orchestra.run.smithi028.stdout:fsid 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:10:02.650 INFO:teuthology.orchestra.run.smithi028.stdout:created 2024-04-24T07:58:42.318512+0000 2024-04-24T08:10:02.650 INFO:teuthology.orchestra.run.smithi028.stdout:modified 2024-04-24T08:09:51.414366+0000 2024-04-24T08:10:02.650 INFO:teuthology.orchestra.run.smithi028.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-04-24T08:10:02.650 INFO:teuthology.orchestra.run.smithi028.stdout:crush_version 26 2024-04-24T08:10:02.651 INFO:teuthology.orchestra.run.smithi028.stdout:full_ratio 0.95 2024-04-24T08:10:02.651 INFO:teuthology.orchestra.run.smithi028.stdout:backfillfull_ratio 0.9 2024-04-24T08:10:02.651 INFO:teuthology.orchestra.run.smithi028.stdout:nearfull_ratio 0.85 2024-04-24T08:10:02.651 INFO:teuthology.orchestra.run.smithi028.stdout:require_min_compat_client luminous 2024-04-24T08:10:02.651 INFO:teuthology.orchestra.run.smithi028.stdout:min_compat_client luminous 2024-04-24T08:10:02.651 INFO:teuthology.orchestra.run.smithi028.stdout:require_osd_release squid 2024-04-24T08:10:02.651 INFO:teuthology.orchestra.run.smithi028.stdout:stretch_mode_enabled false 2024-04-24T08:10:02.651 INFO:teuthology.orchestra.run.smithi028.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-04-24T08:10:02.651 INFO:teuthology.orchestra.run.smithi028.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 2.25 2024-04-24T08:10:02.652 INFO:teuthology.orchestra.run.smithi028.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-04-24T08:10:02.652 INFO:teuthology.orchestra.run.smithi028.stdout:max_osd 12 2024-04-24T08:10:02.652 INFO:teuthology.orchestra.run.smithi028.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6802/1414246664,v1:172.21.15.28:6803/1414246664] [v2:172.21.15.28:6804/1414246664,v1:172.21.15.28:6805/1414246664] exists,up 9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a 2024-04-24T08:10:02.652 INFO:teuthology.orchestra.run.smithi028.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6800/3269221814,v1:172.21.15.117:6801/3269221814] [v2:172.21.15.117:6802/3269221814,v1:172.21.15.117:6803/3269221814] exists,up a1b6b23d-4deb-4210-af96-64b7217b06aa 2024-04-24T08:10:02.652 INFO:teuthology.orchestra.run.smithi028.stdout:osd.2 up in weight 1 up_from 18 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6800/65787773,v1:172.21.15.194:6801/65787773] [v2:172.21.15.194:6802/65787773,v1:172.21.15.194:6803/65787773] exists,up 223ab66d-7bb6-4225-b292-009846c746d4 2024-04-24T08:10:02.652 INFO:teuthology.orchestra.run.smithi028.stdout:osd.3 up in weight 1 up_from 25 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6810/3962695585,v1:172.21.15.28:6811/3962695585] [v2:172.21.15.28:6812/3962695585,v1:172.21.15.28:6813/3962695585] exists,up 32e5f179-3598-4a9b-93c4-48728d942883 2024-04-24T08:10:02.652 INFO:teuthology.orchestra.run.smithi028.stdout:osd.4 up in weight 1 up_from 33 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6808/1649446681,v1:172.21.15.117:6809/1649446681] [v2:172.21.15.117:6810/1649446681,v1:172.21.15.117:6811/1649446681] exists,up c07ffaec-979d-4a1d-b2fd-0d92b3fd303a 2024-04-24T08:10:02.652 INFO:teuthology.orchestra.run.smithi028.stdout:osd.5 up in weight 1 up_from 38 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6808/1399608260,v1:172.21.15.194:6809/1399608260] [v2:172.21.15.194:6810/1399608260,v1:172.21.15.194:6811/1399608260] exists,up fcbda2ee-300e-498a-bb5c-c1d112d03f05 2024-04-24T08:10:02.652 INFO:teuthology.orchestra.run.smithi028.stdout:osd.6 up in weight 1 up_from 46 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6818/1356466279,v1:172.21.15.28:6819/1356466279] [v2:172.21.15.28:6820/1356466279,v1:172.21.15.28:6821/1356466279] exists,up 6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4 2024-04-24T08:10:02.652 INFO:teuthology.orchestra.run.smithi028.stdout:osd.7 up in weight 1 up_from 51 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6816/1472212690,v1:172.21.15.117:6817/1472212690] [v2:172.21.15.117:6818/1472212690,v1:172.21.15.117:6819/1472212690] exists,up 657dc8f6-2eae-4b01-b57c-9bd7fbc5c541 2024-04-24T08:10:02.652 INFO:teuthology.orchestra.run.smithi028.stdout:osd.8 up in weight 1 up_from 60 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6816/301057688,v1:172.21.15.194:6817/301057688] [v2:172.21.15.194:6818/301057688,v1:172.21.15.194:6819/301057688] exists,up 7bd54baf-23e4-4f9d-8c03-79bc64308158 2024-04-24T08:10:02.653 INFO:teuthology.orchestra.run.smithi028.stdout:osd.9 up in weight 1 up_from 68 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.28:6826/2537306688,v1:172.21.15.28:6827/2537306688] [v2:172.21.15.28:6828/2537306688,v1:172.21.15.28:6829/2537306688] exists,up c9268da7-6a4a-4f31-b4ff-99e934adc71e 2024-04-24T08:10:02.653 INFO:teuthology.orchestra.run.smithi028.stdout:osd.10 up in weight 1 up_from 73 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.117:6824/2345072654,v1:172.21.15.117:6825/2345072654] [v2:172.21.15.117:6826/2345072654,v1:172.21.15.117:6827/2345072654] exists,up 107dfd4b-d52f-4fdd-925e-d1fcac3cffe7 2024-04-24T08:10:02.653 INFO:teuthology.orchestra.run.smithi028.stdout:osd.11 up in weight 1 up_from 81 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.194:6824/275366773,v1:172.21.15.194:6825/275366773] [v2:172.21.15.194:6826/275366773,v1:172.21.15.194:6827/275366773] exists,up 22674865-b8f4-445d-8152-ba0a903586b4 2024-04-24T08:10:02.653 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.6 [8,11] 2024-04-24T08:10:02.653 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.c [2,4] 2024-04-24T08:10:02.653 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 2.13 [2,11] 2024-04-24T08:10:02.653 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.0 [5,9] 2024-04-24T08:10:02.654 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.e [4,8] 2024-04-24T08:10:02.654 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.14 [4,6] 2024-04-24T08:10:02.654 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.1b [4,7] 2024-04-24T08:10:02.654 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.21 [5,9] 2024-04-24T08:10:02.654 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.22 [3,9] 2024-04-24T08:10:02.654 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.23 [4,8] 2024-04-24T08:10:02.654 INFO:teuthology.orchestra.run.smithi028.stdout:pg_upmap_items 3.28 [3,6] 2024-04-24T08:10:02.654 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/3308597107 expires 2024-04-25T07:59:49.915756+0000 2024-04-24T08:10:02.654 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/409391969 expires 2024-04-25T07:59:49.915756+0000 2024-04-24T08:10:02.654 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/3308597107 expires 2024-04-25T07:59:49.915756+0000 2024-04-24T08:10:02.654 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/3378984747 expires 2024-04-25T07:59:49.915756+0000 2024-04-24T08:10:02.655 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/4131458386 expires 2024-04-25T07:59:15.381957+0000 2024-04-24T08:10:02.655 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/2417007332 expires 2024-04-25T07:59:15.381957+0000 2024-04-24T08:10:02.655 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/234090859 expires 2024-04-25T07:59:15.381957+0000 2024-04-24T08:10:02.655 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6800/2371619813 expires 2024-04-25T07:59:15.381957+0000 2024-04-24T08:10:02.655 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:0/4081129566 expires 2024-04-25T07:59:49.915756+0000 2024-04-24T08:10:02.655 INFO:teuthology.orchestra.run.smithi028.stdout:blocklist 172.21.15.28:6801/2371619813 expires 2024-04-25T07:59:15.381957+0000 2024-04-24T08:10:02.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.650+0000 7fb284400700 1 -- 172.21.15.28:0/3939570403 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2640a3c40 msgr2=0x7fb2640a6100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.650+0000 7fb284400700 1 --2- 172.21.15.28:0/3939570403 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2640a3c40 0x7fb2640a6100 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fb268000c00 tx=0x7fb268005fb0 comp rx=0 tx=0).stop 2024-04-24T08:10:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.650+0000 7fb284400700 1 -- 172.21.15.28:0/3939570403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb27c104d90 msgr2=0x7fb27c19af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.650+0000 7fb284400700 1 --2- 172.21.15.28:0/3939570403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb27c104d90 0x7fb27c19af50 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb27000d770 tx=0x7fb27000da80 comp rx=0 tx=0).stop 2024-04-24T08:10:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.650+0000 7fb284400700 1 -- 172.21.15.28:0/3939570403 shutdown_connections 2024-04-24T08:10:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.650+0000 7fb284400700 1 --2- 172.21.15.28:0/3939570403 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fb2640a3c40 0x7fb2640a6100 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.650+0000 7fb284400700 1 --2- 172.21.15.28:0/3939570403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb27c106930 0x7fb27c1a2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.650+0000 7fb284400700 1 --2- 172.21.15.28:0/3939570403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb27c104d90 0x7fb27c19af50 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.650+0000 7fb284400700 1 --2- 172.21.15.28:0/3939570403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb27c105f90 0x7fb27c19b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.650+0000 7fb284400700 1 -- 172.21.15.28:0/3939570403 >> 172.21.15.28:0/3939570403 conn(0x7fb27c100350 msgr2=0x7fb27c109290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.651+0000 7fb284400700 1 -- 172.21.15.28:0/3939570403 shutdown_connections 2024-04-24T08:10:02.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:02.651+0000 7fb284400700 1 -- 172.21.15.28:0/3939570403 wait complete. 2024-04-24T08:10:03.460 INFO:teuthology.run_tasks:Running task fs.ready... 2024-04-24T08:10:03.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:10:03.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.700+0000 7f1bb0600640 1 -- 172.21.15.28:0/1572746846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba8105450 msgr2=0x7f1ba81058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.700+0000 7f1bb0600640 1 --2- 172.21.15.28:0/1572746846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba8105450 0x7f1ba81058d0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f1b94009860 tx=0x7f1b9402a340 comp rx=0 tx=0).stop 2024-04-24T08:10:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.700+0000 7f1bb0600640 1 -- 172.21.15.28:0/1572746846 shutdown_connections 2024-04-24T08:10:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.700+0000 7f1bb0600640 1 --2- 172.21.15.28:0/1572746846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ba8105e10 0x7f1ba810c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.700+0000 7f1bb0600640 1 --2- 172.21.15.28:0/1572746846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba8105450 0x7f1ba81058d0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.700+0000 7f1bb0600640 1 --2- 172.21.15.28:0/1572746846 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ba8104260 0x7f1ba8104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.700+0000 7f1bb0600640 1 -- 172.21.15.28:0/1572746846 >> 172.21.15.28:0/1572746846 conn(0x7f1ba80ff7f0 msgr2=0x7f1ba8101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.700+0000 7f1bb0600640 1 -- 172.21.15.28:0/1572746846 shutdown_connections 2024-04-24T08:10:03.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.700+0000 7f1bb0600640 1 -- 172.21.15.28:0/1572746846 wait complete. 2024-04-24T08:10:03.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.701+0000 7f1bb0600640 1 Processor -- start 2024-04-24T08:10:03.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.702+0000 7f1bb0600640 1 -- start start 2024-04-24T08:10:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.702+0000 7f1bb0600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba8104260 0x7f1ba819a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.702+0000 7f1bb0600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ba8105450 0x7f1ba819aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.702+0000 7f1bb0600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ba8105e10 0x7f1ba81a1b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.702+0000 7f1bb0600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba810fef0 con 0x7f1ba8105450 2024-04-24T08:10:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.702+0000 7f1bb0600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba810fd70 con 0x7f1ba8105e10 2024-04-24T08:10:03.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.702+0000 7f1bb0600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1ba8110070 con 0x7f1ba8104260 2024-04-24T08:10:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.702+0000 7f1badc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba8104260 0x7f1ba819a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.702+0000 7f1badc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba8104260 0x7f1ba819a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:57878/0 (socket says 172.21.15.28:57878) 2024-04-24T08:10:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.702+0000 7f1badc00640 1 -- 172.21.15.28:0/3799801320 learned_addr learned my addr 172.21.15.28:0/3799801320 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.702+0000 7f1bad200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ba8105450 0x7f1ba819aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.703+0000 7f1bae600640 1 --2- 172.21.15.28:0/3799801320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ba8105e10 0x7f1ba81a1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.703+0000 7f1badc00640 1 -- 172.21.15.28:0/3799801320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ba8105e10 msgr2=0x7f1ba81a1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.703+0000 7f1badc00640 1 --2- 172.21.15.28:0/3799801320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ba8105e10 0x7f1ba81a1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.703+0000 7f1badc00640 1 -- 172.21.15.28:0/3799801320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ba8105450 msgr2=0x7f1ba819aab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.703+0000 7f1badc00640 1 --2- 172.21.15.28:0/3799801320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ba8105450 0x7f1ba819aab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.703+0000 7f1badc00640 1 -- 172.21.15.28:0/3799801320 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ba81a2190 con 0x7f1ba8104260 2024-04-24T08:10:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.703+0000 7f1bad200640 1 --2- 172.21.15.28:0/3799801320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ba8105450 0x7f1ba819aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:03.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.703+0000 7f1badc00640 1 --2- 172.21.15.28:0/3799801320 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba8104260 0x7f1ba819a570 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800e9c0 tx=0x7f1b9800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.703+0000 7f1ba6000640 1 -- 172.21.15.28:0/3799801320 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b9800cd80 con 0x7f1ba8104260 2024-04-24T08:10:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.703+0000 7f1ba6000640 1 -- 172.21.15.28:0/3799801320 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b980035f0 con 0x7f1ba8104260 2024-04-24T08:10:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.703+0000 7f1ba6000640 1 -- 172.21.15.28:0/3799801320 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b98010690 con 0x7f1ba8104260 2024-04-24T08:10:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.703+0000 7f1bb0600640 1 -- 172.21.15.28:0/3799801320 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ba81a2480 con 0x7f1ba8104260 2024-04-24T08:10:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.704+0000 7f1bb0600640 1 -- 172.21.15.28:0/3799801320 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ba81a29a0 con 0x7f1ba8104260 2024-04-24T08:10:03.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.705+0000 7f1bb0600640 1 -- 172.21.15.28:0/3799801320 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1ba810b580 con 0x7f1ba8104260 2024-04-24T08:10:03.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.707+0000 7f1ba6000640 1 -- 172.21.15.28:0/3799801320 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f1b980037a0 con 0x7f1ba8104260 2024-04-24T08:10:03.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.708+0000 7f1ba6000640 1 --2- 172.21.15.28:0/3799801320 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1b8c0a3c30 0x7f1b8c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:03.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.708+0000 7f1ba6000640 1 -- 172.21.15.28:0/3799801320 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f1b980d0780 con 0x7f1ba8104260 2024-04-24T08:10:03.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.708+0000 7f1bad200640 1 --2- 172.21.15.28:0/3799801320 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1b8c0a3c30 0x7f1b8c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:03.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.709+0000 7f1bad200640 1 --2- 172.21.15.28:0/3799801320 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1b8c0a3c30 0x7f1b8c0a60f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1b94009830 tx=0x7f1b94030550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:03.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.712+0000 7f1ba6000640 1 -- 172.21.15.28:0/3799801320 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1b9809d0c0 con 0x7f1ba8104260 2024-04-24T08:10:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:03 smithi028 ceph-mon[27786]: pgmap v386: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-24T08:10:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:03 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3939570403' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-24T08:10:03.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:03 smithi117 ceph-mon[32438]: pgmap v386: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-24T08:10:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:03 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3939570403' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-24T08:10:03.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.920+0000 7f1bb0600640 1 -- 172.21.15.28:0/3799801320 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f1ba802d620 con 0x7f1ba8104260 2024-04-24T08:10:03.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.922+0000 7f1ba6000640 1 -- 172.21.15.28:0/3799801320 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22724 (secure 0 0 0) 0x7f1b9809c9c0 con 0x7f1ba8104260 2024-04-24T08:10:03.924 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:10:03.924 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:09:51.414366+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:10:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.925+0000 7f1bb0600640 1 -- 172.21.15.28:0/3799801320 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1b8c0a3c30 msgr2=0x7f1b8c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.925+0000 7f1bb0600640 1 --2- 172.21.15.28:0/3799801320 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1b8c0a3c30 0x7f1b8c0a60f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f1b94009830 tx=0x7f1b94030550 comp rx=0 tx=0).stop 2024-04-24T08:10:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.925+0000 7f1bb0600640 1 -- 172.21.15.28:0/3799801320 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba8104260 msgr2=0x7f1ba819a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.925+0000 7f1bb0600640 1 --2- 172.21.15.28:0/3799801320 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba8104260 0x7f1ba819a570 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f1b9800e9c0 tx=0x7f1b9800ee80 comp rx=0 tx=0).stop 2024-04-24T08:10:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.925+0000 7f1bb0600640 1 -- 172.21.15.28:0/3799801320 shutdown_connections 2024-04-24T08:10:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.925+0000 7f1bb0600640 1 --2- 172.21.15.28:0/3799801320 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1b8c0a3c30 0x7f1b8c0a60f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.925+0000 7f1bb0600640 1 --2- 172.21.15.28:0/3799801320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ba8105450 0x7f1ba819aab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.925+0000 7f1bb0600640 1 --2- 172.21.15.28:0/3799801320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ba8105e10 0x7f1ba81a1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.925+0000 7f1bb0600640 1 --2- 172.21.15.28:0/3799801320 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ba8104260 0x7f1ba819a570 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:03.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.925+0000 7f1bb0600640 1 -- 172.21.15.28:0/3799801320 >> 172.21.15.28:0/3799801320 conn(0x7f1ba80ff7f0 msgr2=0x7f1ba8108750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.926+0000 7f1bb0600640 1 -- 172.21.15.28:0/3799801320 shutdown_connections 2024-04-24T08:10:03.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:03.926+0000 7f1bb0600640 1 -- 172.21.15.28:0/3799801320 wait complete. 2024-04-24T08:10:03.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:03 smithi194 ceph-mon[32836]: pgmap v386: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-24T08:10:03.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:03 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3939570403' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-04-24T08:10:03.948 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:02:34.086371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:07:49.819536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:07:50.493486+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:10:03.949 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:10:04.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.171+0000 7f3d39600640 1 -- 172.21.15.28:0/2358221181 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d34102990 msgr2=0x7f3d34104d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:04.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.171+0000 7f3d39600640 1 --2- 172.21.15.28:0/2358221181 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d34102990 0x7f3d34104d80 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f3d24009860 tx=0x7f3d2402a340 comp rx=0 tx=0).stop 2024-04-24T08:10:04.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.172+0000 7f3d39600640 1 -- 172.21.15.28:0/2358221181 shutdown_connections 2024-04-24T08:10:04.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.172+0000 7f3d39600640 1 --2- 172.21.15.28:0/2358221181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d341052c0 0x7f3d341076b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.172+0000 7f3d39600640 1 --2- 172.21.15.28:0/2358221181 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d34107bf0 0x7f3d3410a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.172+0000 7f3d39600640 1 --2- 172.21.15.28:0/2358221181 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d34102990 0x7f3d34104d80 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.172+0000 7f3d39600640 1 -- 172.21.15.28:0/2358221181 >> 172.21.15.28:0/2358221181 conn(0x7f3d340fc3a0 msgr2=0x7f3d340fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:04.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.172+0000 7f3d39600640 1 -- 172.21.15.28:0/2358221181 shutdown_connections 2024-04-24T08:10:04.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.172+0000 7f3d39600640 1 -- 172.21.15.28:0/2358221181 wait complete. 2024-04-24T08:10:04.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.173+0000 7f3d39600640 1 Processor -- start 2024-04-24T08:10:04.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.173+0000 7f3d39600640 1 -- start start 2024-04-24T08:10:04.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.174+0000 7f3d39600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d34102990 0x7f3d3419a410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:04.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.174+0000 7f3d39600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d341052c0 0x7f3d3419a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:04.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.174+0000 7f3d39600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d34107bf0 0x7f3d341a19c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:04.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.174+0000 7f3d39600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3410db80 con 0x7f3d341052c0 2024-04-24T08:10:04.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.174+0000 7f3d39600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3410da00 con 0x7f3d34107bf0 2024-04-24T08:10:04.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.174+0000 7f3d39600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3d3410dd00 con 0x7f3d34102990 2024-04-24T08:10:04.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.174+0000 7f3d33400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d341052c0 0x7f3d3419a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:04.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.174+0000 7f3d33e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d34102990 0x7f3d3419a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.174+0000 7f3d33400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d341052c0 0x7f3d3419a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50418/0 (socket says 172.21.15.28:50418) 2024-04-24T08:10:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.174+0000 7f3d33400640 1 -- 172.21.15.28:0/171164999 learned_addr learned my addr 172.21.15.28:0/171164999 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.174+0000 7f3d38c00640 1 --2- 172.21.15.28:0/171164999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d34107bf0 0x7f3d341a19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.174+0000 7f3d33e00640 1 -- 172.21.15.28:0/171164999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d34107bf0 msgr2=0x7f3d341a19c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.174+0000 7f3d33e00640 1 --2- 172.21.15.28:0/171164999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d34107bf0 0x7f3d341a19c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.174+0000 7f3d33e00640 1 -- 172.21.15.28:0/171164999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d341052c0 msgr2=0x7f3d3419a950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.174+0000 7f3d33e00640 1 --2- 172.21.15.28:0/171164999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d341052c0 0x7f3d3419a950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.174+0000 7f3d33e00640 1 -- 172.21.15.28:0/171164999 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d341a2030 con 0x7f3d34102990 2024-04-24T08:10:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.174+0000 7f3d33400640 1 --2- 172.21.15.28:0/171164999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d341052c0 0x7f3d3419a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:04.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.175+0000 7f3d33e00640 1 --2- 172.21.15.28:0/171164999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d34102990 0x7f3d3419a410 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3d24035040 tx=0x7f3d24033d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.175+0000 7f3d30c00640 1 -- 172.21.15.28:0/171164999 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24037040 con 0x7f3d34102990 2024-04-24T08:10:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.175+0000 7f3d39600640 1 -- 172.21.15.28:0/171164999 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d341a22c0 con 0x7f3d34102990 2024-04-24T08:10:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.175+0000 7f3d30c00640 1 -- 172.21.15.28:0/171164999 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d24030850 con 0x7f3d34102990 2024-04-24T08:10:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.175+0000 7f3d30c00640 1 -- 172.21.15.28:0/171164999 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d24004920 con 0x7f3d34102990 2024-04-24T08:10:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.175+0000 7f3d39600640 1 -- 172.21.15.28:0/171164999 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d341a2780 con 0x7f3d34102990 2024-04-24T08:10:04.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.176+0000 7f3d39600640 1 -- 172.21.15.28:0/171164999 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d34109740 con 0x7f3d34102990 2024-04-24T08:10:04.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.179+0000 7f3d30c00640 1 -- 172.21.15.28:0/171164999 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f3d24004050 con 0x7f3d34102990 2024-04-24T08:10:04.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.180+0000 7f3d30c00640 1 --2- 172.21.15.28:0/171164999 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3d140a3bb0 0x7f3d140a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:04.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.180+0000 7f3d30c00640 1 -- 172.21.15.28:0/171164999 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f3d240f0270 con 0x7f3d34102990 2024-04-24T08:10:04.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.180+0000 7f3d33400640 1 --2- 172.21.15.28:0/171164999 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3d140a3bb0 0x7f3d140a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:04.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.181+0000 7f3d33400640 1 --2- 172.21.15.28:0/171164999 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3d140a3bb0 0x7f3d140a6070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3d3419b920 tx=0x7f3d1c0095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:04.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.183+0000 7f3d30c00640 1 -- 172.21.15.28:0/171164999 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3d240bcbb0 con 0x7f3d34102990 2024-04-24T08:10:04.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.392+0000 7f3d39600640 1 -- 172.21.15.28:0/171164999 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3d3402d7b0 con 0x7f3d34102990 2024-04-24T08:10:04.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.393+0000 7f3d30c00640 1 -- 172.21.15.28:0/171164999 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f3d240bc4b0 con 0x7f3d34102990 2024-04-24T08:10:04.394 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-24T08:10:04.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.395+0000 7f3d39600640 1 -- 172.21.15.28:0/171164999 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3d140a3bb0 msgr2=0x7f3d140a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:04.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.395+0000 7f3d39600640 1 --2- 172.21.15.28:0/171164999 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3d140a3bb0 0x7f3d140a6070 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f3d3419b920 tx=0x7f3d1c0095c0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.395+0000 7f3d39600640 1 -- 172.21.15.28:0/171164999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d34102990 msgr2=0x7f3d3419a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:04.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.395+0000 7f3d39600640 1 --2- 172.21.15.28:0/171164999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d34102990 0x7f3d3419a410 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3d24035040 tx=0x7f3d24033d90 comp rx=0 tx=0).stop 2024-04-24T08:10:04.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.396+0000 7f3d39600640 1 -- 172.21.15.28:0/171164999 shutdown_connections 2024-04-24T08:10:04.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.396+0000 7f3d39600640 1 --2- 172.21.15.28:0/171164999 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f3d140a3bb0 0x7f3d140a6070 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.396+0000 7f3d39600640 1 --2- 172.21.15.28:0/171164999 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3d341052c0 0x7f3d3419a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.396+0000 7f3d39600640 1 --2- 172.21.15.28:0/171164999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d34107bf0 0x7f3d341a19c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.396+0000 7f3d39600640 1 --2- 172.21.15.28:0/171164999 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3d34102990 0x7f3d3419a410 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.396+0000 7f3d39600640 1 -- 172.21.15.28:0/171164999 >> 172.21.15.28:0/171164999 conn(0x7f3d340fc3a0 msgr2=0x7f3d341020c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:04.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.396+0000 7f3d39600640 1 -- 172.21.15.28:0/171164999 shutdown_connections 2024-04-24T08:10:04.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.396+0000 7f3d39600640 1 -- 172.21.15.28:0/171164999 wait complete. 2024-04-24T08:10:04.417 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:10:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.637+0000 7f0d89600640 1 -- 172.21.15.28:0/4128965113 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d84103c00 msgr2=0x7f0d8410a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.637+0000 7f0d89600640 1 --2- 172.21.15.28:0/4128965113 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d84103c00 0x7f0d8410a490 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f0d7c009900 tx=0x7f0d7c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.637+0000 7f0d89600640 1 -- 172.21.15.28:0/4128965113 shutdown_connections 2024-04-24T08:10:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.637+0000 7f0d89600640 1 --2- 172.21.15.28:0/4128965113 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d84103c00 0x7f0d8410a490 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.637+0000 7f0d89600640 1 --2- 172.21.15.28:0/4128965113 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d84103240 0x7f0d841036c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.637+0000 7f0d89600640 1 --2- 172.21.15.28:0/4128965113 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d84102050 0x7f0d84102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.637+0000 7f0d89600640 1 -- 172.21.15.28:0/4128965113 >> 172.21.15.28:0/4128965113 conn(0x7f0d840fd5e0 msgr2=0x7f0d840ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:04.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.637+0000 7f0d89600640 1 -- 172.21.15.28:0/4128965113 shutdown_connections 2024-04-24T08:10:04.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.638+0000 7f0d89600640 1 -- 172.21.15.28:0/4128965113 wait complete. 2024-04-24T08:10:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.638+0000 7f0d89600640 1 Processor -- start 2024-04-24T08:10:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.639+0000 7f0d89600640 1 -- start start 2024-04-24T08:10:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.639+0000 7f0d89600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d84102050 0x7f0d8419a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.639+0000 7f0d89600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d84103240 0x7f0d8419aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.639+0000 7f0d89600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d84103c00 0x7f0d841a1af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.639+0000 7f0d89600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8410dd30 con 0x7f0d84102050 2024-04-24T08:10:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.639+0000 7f0d89600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8410dbb0 con 0x7f0d84103c00 2024-04-24T08:10:04.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.639+0000 7f0d89600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0d8410deb0 con 0x7f0d84103240 2024-04-24T08:10:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.639+0000 7f0d82a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d84102050 0x7f0d8419a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.639+0000 7f0d82000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d84103240 0x7f0d8419aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.639+0000 7f0d82000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d84103240 0x7f0d8419aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:57922/0 (socket says 172.21.15.28:57922) 2024-04-24T08:10:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.639+0000 7f0d82000640 1 -- 172.21.15.28:0/55691564 learned_addr learned my addr 172.21.15.28:0/55691564 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.639+0000 7f0d83400640 1 --2- 172.21.15.28:0/55691564 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d84103c00 0x7f0d841a1af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.640+0000 7f0d82000640 1 -- 172.21.15.28:0/55691564 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d84103c00 msgr2=0x7f0d841a1af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.640+0000 7f0d82000640 1 --2- 172.21.15.28:0/55691564 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d84103c00 0x7f0d841a1af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.640+0000 7f0d82000640 1 -- 172.21.15.28:0/55691564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d84102050 msgr2=0x7f0d8419a540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.640+0000 7f0d82000640 1 --2- 172.21.15.28:0/55691564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d84102050 0x7f0d8419a540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.640+0000 7f0d82000640 1 -- 172.21.15.28:0/55691564 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0d841a2160 con 0x7f0d84103240 2024-04-24T08:10:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.640+0000 7f0d82000640 1 --2- 172.21.15.28:0/55691564 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d84103240 0x7f0d8419aa80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0d7000eb90 tx=0x7f0d7000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:04.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.640+0000 7f0d7b400640 1 -- 172.21.15.28:0/55691564 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d70004d20 con 0x7f0d84103240 2024-04-24T08:10:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.640+0000 7f0d7b400640 1 -- 172.21.15.28:0/55691564 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0d7000ee90 con 0x7f0d84103240 2024-04-24T08:10:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.640+0000 7f0d89600640 1 -- 172.21.15.28:0/55691564 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0d841a2450 con 0x7f0d84103240 2024-04-24T08:10:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.640+0000 7f0d7b400640 1 -- 172.21.15.28:0/55691564 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0d70010850 con 0x7f0d84103240 2024-04-24T08:10:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.640+0000 7f0d89600640 1 -- 172.21.15.28:0/55691564 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0d841a2950 con 0x7f0d84103240 2024-04-24T08:10:04.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.641+0000 7f0d89600640 1 -- 172.21.15.28:0/55691564 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0d841a2c10 con 0x7f0d84103240 2024-04-24T08:10:04.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.644+0000 7f0d7b400640 1 -- 172.21.15.28:0/55691564 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0d70005020 con 0x7f0d84103240 2024-04-24T08:10:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.646+0000 7f0d7b400640 1 --2- 172.21.15.28:0/55691564 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0d680a3bb0 0x7f0d680a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.646+0000 7f0d7b400640 1 -- 172.21.15.28:0/55691564 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f0d700d0b80 con 0x7f0d84103240 2024-04-24T08:10:04.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.646+0000 7f0d82a00640 1 --2- 172.21.15.28:0/55691564 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0d680a3bb0 0x7f0d680a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:04.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.647+0000 7f0d82a00640 1 --2- 172.21.15.28:0/55691564 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0d680a3bb0 0x7f0d680a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0d841030a0 tx=0x7f0d7400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:04.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.650+0000 7f0d7b400640 1 -- 172.21.15.28:0/55691564 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f0d7009d4c0 con 0x7f0d84103240 2024-04-24T08:10:04.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:04 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3799801320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:04.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:04 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/171164999' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:10:04.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.857+0000 7f0d89600640 1 -- 172.21.15.28:0/55691564 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f0d841a2de0 con 0x7f0d84103240 2024-04-24T08:10:04.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.857+0000 7f0d7b400640 1 -- 172.21.15.28:0/55691564 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f0d7009cdc0 con 0x7f0d84103240 2024-04-24T08:10:04.858 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-24T08:10:04.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.860+0000 7f0d89600640 1 -- 172.21.15.28:0/55691564 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0d680a3bb0 msgr2=0x7f0d680a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:04.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.860+0000 7f0d89600640 1 --2- 172.21.15.28:0/55691564 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0d680a3bb0 0x7f0d680a6070 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f0d841030a0 tx=0x7f0d7400b390 comp rx=0 tx=0).stop 2024-04-24T08:10:04.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.860+0000 7f0d89600640 1 -- 172.21.15.28:0/55691564 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d84103240 msgr2=0x7f0d8419aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:04.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.860+0000 7f0d89600640 1 --2- 172.21.15.28:0/55691564 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d84103240 0x7f0d8419aa80 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f0d7000eb90 tx=0x7f0d7000c6a0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.860+0000 7f0d89600640 1 -- 172.21.15.28:0/55691564 shutdown_connections 2024-04-24T08:10:04.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.860+0000 7f0d89600640 1 --2- 172.21.15.28:0/55691564 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f0d680a3bb0 0x7f0d680a6070 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.860+0000 7f0d89600640 1 --2- 172.21.15.28:0/55691564 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d84103c00 0x7f0d841a1af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.860+0000 7f0d89600640 1 --2- 172.21.15.28:0/55691564 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0d84103240 0x7f0d8419aa80 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.860+0000 7f0d89600640 1 --2- 172.21.15.28:0/55691564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0d84102050 0x7f0d8419a540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:04.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.860+0000 7f0d89600640 1 -- 172.21.15.28:0/55691564 >> 172.21.15.28:0/55691564 conn(0x7f0d840fd5e0 msgr2=0x7f0d84074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:04.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.860+0000 7f0d89600640 1 -- 172.21.15.28:0/55691564 shutdown_connections 2024-04-24T08:10:04.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:04.861+0000 7f0d89600640 1 -- 172.21.15.28:0/55691564 wait complete. 2024-04-24T08:10:04.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:04 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3799801320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:04.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:04 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/171164999' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:10:04.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:10:04.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:04 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3799801320' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:04.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:04 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/171164999' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:10:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.094+0000 7fcb0b200640 1 -- 172.21.15.28:0/881084160 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb0c103a20 msgr2=0x7fcb0c10a340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.094+0000 7fcb0b200640 1 --2- 172.21.15.28:0/881084160 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb0c103a20 0x7fcb0c10a340 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcb04009be0 tx=0x7fcb0402a6d0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.094+0000 7fcb0b200640 1 -- 172.21.15.28:0/881084160 shutdown_connections 2024-04-24T08:10:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.094+0000 7fcb0b200640 1 --2- 172.21.15.28:0/881084160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb0c103060 0x7fcb0c1034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.094+0000 7fcb0b200640 1 --2- 172.21.15.28:0/881084160 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb0c103a20 0x7fcb0c10a340 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.094+0000 7fcb0b200640 1 --2- 172.21.15.28:0/881084160 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcb0c101f10 0x7fcb0c102330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.094+0000 7fcb0b200640 1 -- 172.21.15.28:0/881084160 >> 172.21.15.28:0/881084160 conn(0x7fcb0c0fd4a0 msgr2=0x7fcb0c0ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.094+0000 7fcb0b200640 1 -- 172.21.15.28:0/881084160 shutdown_connections 2024-04-24T08:10:05.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.094+0000 7fcb0b200640 1 -- 172.21.15.28:0/881084160 wait complete. 2024-04-24T08:10:05.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.095+0000 7fcb0b200640 1 Processor -- start 2024-04-24T08:10:05.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.095+0000 7fcb0b200640 1 -- start start 2024-04-24T08:10:05.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.096+0000 7fcb0b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb0c101f10 0x7fcb0c075db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:05.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.096+0000 7fcb0b200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb0c103060 0x7fcb0c0743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:05.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.096+0000 7fcb0b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcb0c103a20 0x7fcb0c074930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:05.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.096+0000 7fcb0b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c10dba0 con 0x7fcb0c101f10 2024-04-24T08:10:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.096+0000 7fcb0b200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c10da20 con 0x7fcb0c103060 2024-04-24T08:10:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.096+0000 7fcb0b200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcb0c10dd20 con 0x7fcb0c103a20 2024-04-24T08:10:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.096+0000 7fcb03e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb0c103060 0x7fcb0c0743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.096+0000 7fcb09e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb0c101f10 0x7fcb0c075db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.096+0000 7fcb03e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb0c103060 0x7fcb0c0743f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:52386/0 (socket says 172.21.15.28:52386) 2024-04-24T08:10:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.096+0000 7fcb03e00640 1 -- 172.21.15.28:0/683016752 learned_addr learned my addr 172.21.15.28:0/683016752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.096+0000 7fcb0a800640 1 --2- 172.21.15.28:0/683016752 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcb0c103a20 0x7fcb0c074930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.097+0000 7fcb03e00640 1 -- 172.21.15.28:0/683016752 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcb0c103a20 msgr2=0x7fcb0c074930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.097+0000 7fcb03e00640 1 --2- 172.21.15.28:0/683016752 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcb0c103a20 0x7fcb0c074930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.097+0000 7fcb03e00640 1 -- 172.21.15.28:0/683016752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb0c101f10 msgr2=0x7fcb0c075db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.097+0000 7fcb03e00640 1 --2- 172.21.15.28:0/683016752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb0c101f10 0x7fcb0c075db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.097+0000 7fcb03e00640 1 -- 172.21.15.28:0/683016752 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcb0c0751c0 con 0x7fcb0c103060 2024-04-24T08:10:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.097+0000 7fcb03e00640 1 --2- 172.21.15.28:0/683016752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb0c103060 0x7fcb0c0743f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcaf400e9c0 tx=0x7fcaf400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.097+0000 7fcb02a00640 1 -- 172.21.15.28:0/683016752 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf400cd80 con 0x7fcb0c103060 2024-04-24T08:10:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.097+0000 7fcb0b200640 1 -- 172.21.15.28:0/683016752 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcb0c1a22a0 con 0x7fcb0c103060 2024-04-24T08:10:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.097+0000 7fcb02a00640 1 -- 172.21.15.28:0/683016752 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcaf40035f0 con 0x7fcb0c103060 2024-04-24T08:10:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.097+0000 7fcb02a00640 1 -- 172.21.15.28:0/683016752 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4010690 con 0x7fcb0c103060 2024-04-24T08:10:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.097+0000 7fcb0b200640 1 -- 172.21.15.28:0/683016752 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcb0c1a27f0 con 0x7fcb0c103060 2024-04-24T08:10:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.099+0000 7fcb0b200640 1 -- 172.21.15.28:0/683016752 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcb0c1091c0 con 0x7fcb0c103060 2024-04-24T08:10:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.101+0000 7fcb02a00640 1 -- 172.21.15.28:0/683016752 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fcaf40037a0 con 0x7fcb0c103060 2024-04-24T08:10:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.102+0000 7fcb02a00640 1 --2- 172.21.15.28:0/683016752 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fcaf00a3bb0 0x7fcaf00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.102+0000 7fcb02a00640 1 -- 172.21.15.28:0/683016752 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7fcaf40d08c0 con 0x7fcb0c103060 2024-04-24T08:10:05.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.102+0000 7fcb09e00640 1 --2- 172.21.15.28:0/683016752 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fcaf00a3bb0 0x7fcaf00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:05.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.103+0000 7fcb09e00640 1 --2- 172.21.15.28:0/683016752 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fcaf00a3bb0 0x7fcaf00a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009530 tx=0x7fcafc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:05.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.107+0000 7fcb02a00640 1 -- 172.21.15.28:0/683016752 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fcaf409d200 con 0x7fcb0c103060 2024-04-24T08:10:05.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.313+0000 7fcb0b200640 1 -- 172.21.15.28:0/683016752 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fcb0c062c00 con 0x7fcb0c103060 2024-04-24T08:10:05.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.314+0000 7fcb02a00640 1 -- 172.21.15.28:0/683016752 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fcaf409cb00 con 0x7fcb0c103060 2024-04-24T08:10:05.318 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-24T08:10:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.317+0000 7fcb0b200640 1 -- 172.21.15.28:0/683016752 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fcaf00a3bb0 msgr2=0x7fcaf00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.317+0000 7fcb0b200640 1 --2- 172.21.15.28:0/683016752 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fcaf00a3bb0 0x7fcaf00a6070 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fcafc009530 tx=0x7fcafc009260 comp rx=0 tx=0).stop 2024-04-24T08:10:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.317+0000 7fcb0b200640 1 -- 172.21.15.28:0/683016752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb0c103060 msgr2=0x7fcb0c0743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.317+0000 7fcb0b200640 1 --2- 172.21.15.28:0/683016752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb0c103060 0x7fcb0c0743f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fcaf400e9c0 tx=0x7fcaf400ee80 comp rx=0 tx=0).stop 2024-04-24T08:10:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.317+0000 7fcb0b200640 1 -- 172.21.15.28:0/683016752 shutdown_connections 2024-04-24T08:10:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.317+0000 7fcb0b200640 1 --2- 172.21.15.28:0/683016752 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fcaf00a3bb0 0x7fcaf00a6070 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.317+0000 7fcb0b200640 1 --2- 172.21.15.28:0/683016752 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcb0c103a20 0x7fcb0c074930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.317+0000 7fcb0b200640 1 --2- 172.21.15.28:0/683016752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb0c103060 0x7fcb0c0743f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.317+0000 7fcb0b200640 1 --2- 172.21.15.28:0/683016752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcb0c101f10 0x7fcb0c075db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.317+0000 7fcb0b200640 1 -- 172.21.15.28:0/683016752 >> 172.21.15.28:0/683016752 conn(0x7fcb0c0fd4a0 msgr2=0x7fcb0c106390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:05.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.317+0000 7fcb0b200640 1 -- 172.21.15.28:0/683016752 shutdown_connections 2024-04-24T08:10:05.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.317+0000 7fcb0b200640 1 -- 172.21.15.28:0/683016752 wait complete. 2024-04-24T08:10:05.338 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:10:05.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.547+0000 7f2c87a00640 1 -- 172.21.15.28:0/3746425279 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c80103240 msgr2=0x7f2c801036c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:05.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.547+0000 7f2c87a00640 1 --2- 172.21.15.28:0/3746425279 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c80103240 0x7f2c801036c0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2c68009900 tx=0x7f2c6802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.548+0000 7f2c87a00640 1 -- 172.21.15.28:0/3746425279 shutdown_connections 2024-04-24T08:10:05.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.548+0000 7f2c87a00640 1 --2- 172.21.15.28:0/3746425279 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80103c00 0x7f2c8010a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.548+0000 7f2c87a00640 1 --2- 172.21.15.28:0/3746425279 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c80103240 0x7f2c801036c0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.548+0000 7f2c87a00640 1 --2- 172.21.15.28:0/3746425279 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c80102050 0x7f2c80102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.548+0000 7f2c87a00640 1 -- 172.21.15.28:0/3746425279 >> 172.21.15.28:0/3746425279 conn(0x7f2c800fd600 msgr2=0x7f2c800ffa20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:05.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.548+0000 7f2c87a00640 1 -- 172.21.15.28:0/3746425279 shutdown_connections 2024-04-24T08:10:05.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.548+0000 7f2c87a00640 1 -- 172.21.15.28:0/3746425279 wait complete. 2024-04-24T08:10:05.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.549+0000 7f2c87a00640 1 Processor -- start 2024-04-24T08:10:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.549+0000 7f2c87a00640 1 -- start start 2024-04-24T08:10:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.549+0000 7f2c87a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c80102050 0x7f2c8019a560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.549+0000 7f2c87a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80103240 0x7f2c8019aaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.549+0000 7f2c87a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c80103c00 0x7f2c801a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.549+0000 7f2c87a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8010dd30 con 0x7f2c80103240 2024-04-24T08:10:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.549+0000 7f2c87a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8010dbb0 con 0x7f2c80102050 2024-04-24T08:10:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.549+0000 7f2c87a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2c8010deb0 con 0x7f2c80103c00 2024-04-24T08:10:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.549+0000 7f2c7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80103240 0x7f2c8019aaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.549+0000 7f2c85000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c80102050 0x7f2c8019a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.549+0000 7f2c7fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80103240 0x7f2c8019aaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50484/0 (socket says 172.21.15.28:50484) 2024-04-24T08:10:05.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.549+0000 7f2c7fe00640 1 -- 172.21.15.28:0/1517742041 learned_addr learned my addr 172.21.15.28:0/1517742041 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.550+0000 7f2c85a00640 1 --2- 172.21.15.28:0/1517742041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c80103c00 0x7f2c801a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.550+0000 7f2c85000640 1 -- 172.21.15.28:0/1517742041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c80103c00 msgr2=0x7f2c801a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.550+0000 7f2c85000640 1 --2- 172.21.15.28:0/1517742041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c80103c00 0x7f2c801a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.550+0000 7f2c85000640 1 -- 172.21.15.28:0/1517742041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80103240 msgr2=0x7f2c8019aaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.550+0000 7f2c85000640 1 --2- 172.21.15.28:0/1517742041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80103240 0x7f2c8019aaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.550+0000 7f2c85000640 1 -- 172.21.15.28:0/1517742041 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2c801a2180 con 0x7f2c80102050 2024-04-24T08:10:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.550+0000 7f2c7fe00640 1 --2- 172.21.15.28:0/1517742041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80103240 0x7f2c8019aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.550+0000 7f2c85000640 1 --2- 172.21.15.28:0/1517742041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c80102050 0x7f2c8019a560 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2c70005fd0 tx=0x7f2c7000ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.550+0000 7f2c7d600640 1 -- 172.21.15.28:0/1517742041 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c70003970 con 0x7f2c80102050 2024-04-24T08:10:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.551+0000 7f2c87a00640 1 -- 172.21.15.28:0/1517742041 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2c801a2470 con 0x7f2c80102050 2024-04-24T08:10:05.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.551+0000 7f2c7d600640 1 -- 172.21.15.28:0/1517742041 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c70004020 con 0x7f2c80102050 2024-04-24T08:10:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.551+0000 7f2c7d600640 1 -- 172.21.15.28:0/1517742041 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c700107b0 con 0x7f2c80102050 2024-04-24T08:10:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.551+0000 7f2c87a00640 1 -- 172.21.15.28:0/1517742041 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2c801a29c0 con 0x7f2c80102050 2024-04-24T08:10:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.551+0000 7f2c87a00640 1 -- 172.21.15.28:0/1517742041 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2c801a2c80 con 0x7f2c80102050 2024-04-24T08:10:05.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.554+0000 7f2c7d600640 1 -- 172.21.15.28:0/1517742041 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f2c700041d0 con 0x7f2c80102050 2024-04-24T08:10:05.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.555+0000 7f2c7d600640 1 --2- 172.21.15.28:0/1517742041 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2c640a3bb0 0x7f2c640a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:05.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.555+0000 7f2c7d600640 1 -- 172.21.15.28:0/1517742041 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f2c700d0c00 con 0x7f2c80102050 2024-04-24T08:10:05.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.555+0000 7f2c7fe00640 1 --2- 172.21.15.28:0/1517742041 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2c640a3bb0 0x7f2c640a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:05.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.556+0000 7f2c7fe00640 1 --2- 172.21.15.28:0/1517742041 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2c640a3bb0 0x7f2c640a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2c8019ba70 tx=0x7f2c680315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:05.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.557+0000 7f2c7d600640 1 -- 172.21.15.28:0/1517742041 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c7009d5d0 con 0x7f2c80102050 2024-04-24T08:10:05.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:05 smithi028 ceph-mon[27786]: pgmap v387: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:10:05.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:05 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/55691564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:10:05.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:05 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/683016752' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:10:05.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.807+0000 7f2c87a00640 1 -- 172.21.15.28:0/1517742041 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2c801a2e50 con 0x7f2c80102050 2024-04-24T08:10:05.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.808+0000 7f2c7d600640 1 -- 172.21.15.28:0/1517742041 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11871 (secure 0 0 0) 0x7f2c7009ced0 con 0x7f2c80102050 2024-04-24T08:10:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:10:05.810 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:07:57.133790+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:10:05.810 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 23 2024-04-24T08:10:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.811+0000 7f2c87a00640 1 -- 172.21.15.28:0/1517742041 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2c640a3bb0 msgr2=0x7f2c640a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.811+0000 7f2c87a00640 1 --2- 172.21.15.28:0/1517742041 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2c640a3bb0 0x7f2c640a6070 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f2c8019ba70 tx=0x7f2c680315b0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.811+0000 7f2c87a00640 1 -- 172.21.15.28:0/1517742041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c80102050 msgr2=0x7f2c8019a560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.811+0000 7f2c87a00640 1 --2- 172.21.15.28:0/1517742041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c80102050 0x7f2c8019a560 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f2c70005fd0 tx=0x7f2c7000ef90 comp rx=0 tx=0).stop 2024-04-24T08:10:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.811+0000 7f2c87a00640 1 -- 172.21.15.28:0/1517742041 shutdown_connections 2024-04-24T08:10:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.811+0000 7f2c87a00640 1 --2- 172.21.15.28:0/1517742041 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f2c640a3bb0 0x7f2c640a6070 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.811+0000 7f2c87a00640 1 --2- 172.21.15.28:0/1517742041 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2c80103240 0x7f2c8019aaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.811+0000 7f2c87a00640 1 --2- 172.21.15.28:0/1517742041 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2c80103c00 0x7f2c801a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.811+0000 7f2c87a00640 1 --2- 172.21.15.28:0/1517742041 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c80102050 0x7f2c8019a560 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:05.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.811+0000 7f2c87a00640 1 -- 172.21.15.28:0/1517742041 >> 172.21.15.28:0/1517742041 conn(0x7f2c800fd600 msgr2=0x7f2c80074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:05.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.812+0000 7f2c87a00640 1 -- 172.21.15.28:0/1517742041 shutdown_connections 2024-04-24T08:10:05.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:05.812+0000 7f2c87a00640 1 -- 172.21.15.28:0/1517742041 wait complete. 2024-04-24T08:10:05.833 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:10:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:05 smithi117 ceph-mon[32438]: pgmap v387: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:10:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:05 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/55691564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:10:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:05 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/683016752' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:10:05.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:05 smithi194 ceph-mon[32836]: pgmap v387: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:10:05.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:05 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/55691564' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:10:05.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:05 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/683016752' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:10:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.046+0000 7fbc8de00640 1 -- 172.21.15.28:0/218889969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc88109850 msgr2=0x7fbc88109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.046+0000 7fbc8de00640 1 --2- 172.21.15.28:0/218889969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc88109850 0x7fbc88109cd0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbc70009900 tx=0x7fbc7002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.046+0000 7fbc8de00640 1 -- 172.21.15.28:0/218889969 shutdown_connections 2024-04-24T08:10:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.046+0000 7fbc8de00640 1 --2- 172.21.15.28:0/218889969 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc88109850 0x7fbc88109cd0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.046+0000 7fbc8de00640 1 --2- 172.21.15.28:0/218889969 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc8810a210 0x7fbc88110aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.046+0000 7fbc8de00640 1 --2- 172.21.15.28:0/218889969 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc88108660 0x7fbc88108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.046+0000 7fbc8de00640 1 -- 172.21.15.28:0/218889969 >> 172.21.15.28:0/218889969 conn(0x7fbc88077620 msgr2=0x7fbc88079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.046+0000 7fbc8de00640 1 -- 172.21.15.28:0/218889969 shutdown_connections 2024-04-24T08:10:06.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.046+0000 7fbc8de00640 1 -- 172.21.15.28:0/218889969 wait complete. 2024-04-24T08:10:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.047+0000 7fbc8de00640 1 Processor -- start 2024-04-24T08:10:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.047+0000 7fbc8de00640 1 -- start start 2024-04-24T08:10:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.048+0000 7fbc8de00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc88108660 0x7fbc8819e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.048+0000 7fbc8de00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc88109850 0x7fbc8819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.048+0000 7fbc8de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc8810a210 0x7fbc881a5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.048+0000 7fbc8de00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbc88114310 con 0x7fbc8810a210 2024-04-24T08:10:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.048+0000 7fbc8de00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbc88114190 con 0x7fbc88108660 2024-04-24T08:10:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.048+0000 7fbc8de00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbc88114490 con 0x7fbc88109850 2024-04-24T08:10:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.048+0000 7fbc7e000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc88109850 0x7fbc8819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.048+0000 7fbc87400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc8810a210 0x7fbc881a5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:06.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.048+0000 7fbc7e000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc88109850 0x7fbc8819eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:57984/0 (socket says 172.21.15.28:57984) 2024-04-24T08:10:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.048+0000 7fbc7e000640 1 -- 172.21.15.28:0/2137718210 learned_addr learned my addr 172.21.15.28:0/2137718210 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.048+0000 7fbc86a00640 1 --2- 172.21.15.28:0/2137718210 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc88108660 0x7fbc8819e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.048+0000 7fbc7e000640 1 -- 172.21.15.28:0/2137718210 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc88108660 msgr2=0x7fbc8819e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.048+0000 7fbc7e000640 1 --2- 172.21.15.28:0/2137718210 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc88108660 0x7fbc8819e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.048+0000 7fbc7e000640 1 -- 172.21.15.28:0/2137718210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc8810a210 msgr2=0x7fbc881a5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.048+0000 7fbc7e000640 1 --2- 172.21.15.28:0/2137718210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc8810a210 0x7fbc881a5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.048+0000 7fbc7e000640 1 -- 172.21.15.28:0/2137718210 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbc881a65d0 con 0x7fbc88109850 2024-04-24T08:10:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.049+0000 7fbc7e000640 1 --2- 172.21.15.28:0/2137718210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc88109850 0x7fbc8819eef0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbc70031440 tx=0x7fbc70032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.049+0000 7fbc8ca00640 1 -- 172.21.15.28:0/2137718210 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc7003f030 con 0x7fbc88109850 2024-04-24T08:10:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.049+0000 7fbc8ca00640 1 -- 172.21.15.28:0/2137718210 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbc70003ce0 con 0x7fbc88109850 2024-04-24T08:10:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.049+0000 7fbc8de00640 1 -- 172.21.15.28:0/2137718210 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbc881a6860 con 0x7fbc88109850 2024-04-24T08:10:06.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.049+0000 7fbc8ca00640 1 -- 172.21.15.28:0/2137718210 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbc70035070 con 0x7fbc88109850 2024-04-24T08:10:06.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.049+0000 7fbc8de00640 1 -- 172.21.15.28:0/2137718210 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbc881a6d50 con 0x7fbc88109850 2024-04-24T08:10:06.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.050+0000 7fbc8de00640 1 -- 172.21.15.28:0/2137718210 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbc8810f980 con 0x7fbc88109850 2024-04-24T08:10:06.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.053+0000 7fbc8ca00640 1 -- 172.21.15.28:0/2137718210 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fbc70042030 con 0x7fbc88109850 2024-04-24T08:10:06.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.054+0000 7fbc8ca00640 1 --2- 172.21.15.28:0/2137718210 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbc6c0a3bb0 0x7fbc6c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.054+0000 7fbc8ca00640 1 -- 172.21.15.28:0/2137718210 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7fbc700f0ee0 con 0x7fbc88109850 2024-04-24T08:10:06.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.054+0000 7fbc86a00640 1 --2- 172.21.15.28:0/2137718210 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbc6c0a3bb0 0x7fbc6c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:06.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.055+0000 7fbc86a00640 1 --2- 172.21.15.28:0/2137718210 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbc6c0a3bb0 0x7fbc6c0a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbc881096b0 tx=0x7fbc78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:06.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.056+0000 7fbc8ca00640 1 -- 172.21.15.28:0/2137718210 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbc700bd820 con 0x7fbc88109850 2024-04-24T08:10:06.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.264+0000 7fbc8de00640 1 -- 172.21.15.28:0/2137718210 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbc8802d7f0 con 0x7fbc88109850 2024-04-24T08:10:06.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.267+0000 7fbc8ca00640 1 -- 172.21.15.28:0/2137718210 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22724 (secure 0 0 0) 0x7fbc700bd120 con 0x7fbc88109850 2024-04-24T08:10:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:10:06.268 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:09:51.414366+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:10:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.269+0000 7fbc8de00640 1 -- 172.21.15.28:0/2137718210 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbc6c0a3bb0 msgr2=0x7fbc6c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.269+0000 7fbc8de00640 1 --2- 172.21.15.28:0/2137718210 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbc6c0a3bb0 0x7fbc6c0a6070 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbc881096b0 tx=0x7fbc78005f90 comp rx=0 tx=0).stop 2024-04-24T08:10:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.269+0000 7fbc8de00640 1 -- 172.21.15.28:0/2137718210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc88109850 msgr2=0x7fbc8819eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.269+0000 7fbc8de00640 1 --2- 172.21.15.28:0/2137718210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc88109850 0x7fbc8819eef0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fbc70031440 tx=0x7fbc70032ce0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.270+0000 7fbc8de00640 1 -- 172.21.15.28:0/2137718210 shutdown_connections 2024-04-24T08:10:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.270+0000 7fbc8de00640 1 --2- 172.21.15.28:0/2137718210 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fbc6c0a3bb0 0x7fbc6c0a6070 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.270+0000 7fbc8de00640 1 --2- 172.21.15.28:0/2137718210 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbc8810a210 0x7fbc881a5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.270+0000 7fbc8de00640 1 --2- 172.21.15.28:0/2137718210 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbc88109850 0x7fbc8819eef0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.270+0000 7fbc8de00640 1 --2- 172.21.15.28:0/2137718210 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc88108660 0x7fbc8819e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.270+0000 7fbc8de00640 1 -- 172.21.15.28:0/2137718210 >> 172.21.15.28:0/2137718210 conn(0x7fbc88077620 msgr2=0x7fbc8810cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:06.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.270+0000 7fbc8de00640 1 -- 172.21.15.28:0/2137718210 shutdown_connections 2024-04-24T08:10:06.272 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.270+0000 7fbc8de00640 1 -- 172.21.15.28:0/2137718210 wait complete. 2024-04-24T08:10:06.292 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:02:34.086371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:07:49.819536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:07:50.493486+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:10:06.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:10:06.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.521+0000 7f0273800640 1 -- 172.21.15.28:0/765148235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f026c109850 msgr2=0x7f026c109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.521+0000 7f0273800640 1 --2- 172.21.15.28:0/765148235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f026c109850 0x7f026c109cd0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f0258009900 tx=0x7f025802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.521+0000 7f0273800640 1 -- 172.21.15.28:0/765148235 shutdown_connections 2024-04-24T08:10:06.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.521+0000 7f0273800640 1 --2- 172.21.15.28:0/765148235 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f026c109850 0x7f026c109cd0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.521+0000 7f0273800640 1 --2- 172.21.15.28:0/765148235 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f026c10a210 0x7f026c110aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.521+0000 7f0273800640 1 --2- 172.21.15.28:0/765148235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f026c108660 0x7f026c108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.521+0000 7f0273800640 1 -- 172.21.15.28:0/765148235 >> 172.21.15.28:0/765148235 conn(0x7f026c077600 msgr2=0x7f026c079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:06.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.521+0000 7f0273800640 1 -- 172.21.15.28:0/765148235 shutdown_connections 2024-04-24T08:10:06.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.521+0000 7f0273800640 1 -- 172.21.15.28:0/765148235 wait complete. 2024-04-24T08:10:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.522+0000 7f0273800640 1 Processor -- start 2024-04-24T08:10:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.522+0000 7f0273800640 1 -- start start 2024-04-24T08:10:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.522+0000 7f0273800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f026c108660 0x7f026c19e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.522+0000 7f0273800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f026c109850 0x7f026c19eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.522+0000 7f0273800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f026c10a210 0x7f026c1a5f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.522+0000 7f0273800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f026c114330 con 0x7f026c109850 2024-04-24T08:10:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.522+0000 7f0273800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f026c1141b0 con 0x7f026c108660 2024-04-24T08:10:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.522+0000 7f0273800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f026c1144b0 con 0x7f026c10a210 2024-04-24T08:10:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.523+0000 7f026be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f026c109850 0x7f026c19eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.523+0000 7f026be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f026c109850 0x7f026c19eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50510/0 (socket says 172.21.15.28:50510) 2024-04-24T08:10:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.523+0000 7f026be00640 1 -- 172.21.15.28:0/2936297426 learned_addr learned my addr 172.21.15.28:0/2936297426 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.523+0000 7f0270e00640 1 --2- 172.21.15.28:0/2936297426 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f026c108660 0x7f026c19e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.523+0000 7f0271800640 1 --2- 172.21.15.28:0/2936297426 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f026c10a210 0x7f026c1a5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.523+0000 7f026be00640 1 -- 172.21.15.28:0/2936297426 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f026c10a210 msgr2=0x7f026c1a5f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.523+0000 7f026be00640 1 --2- 172.21.15.28:0/2936297426 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f026c10a210 0x7f026c1a5f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.523+0000 7f026be00640 1 -- 172.21.15.28:0/2936297426 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f026c108660 msgr2=0x7f026c19e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.523+0000 7f026be00640 1 --2- 172.21.15.28:0/2936297426 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f026c108660 0x7f026c19e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.523+0000 7f026be00640 1 -- 172.21.15.28:0/2936297426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f026c1a6590 con 0x7f026c109850 2024-04-24T08:10:06.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.523+0000 7f026be00640 1 --2- 172.21.15.28:0/2936297426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f026c109850 0x7f026c19eeb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0258031440 tx=0x7f0258032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.524+0000 7f0269600640 1 -- 172.21.15.28:0/2936297426 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f025803f030 con 0x7f026c109850 2024-04-24T08:10:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.524+0000 7f0269600640 1 -- 172.21.15.28:0/2936297426 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0258003ce0 con 0x7f026c109850 2024-04-24T08:10:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.524+0000 7f0269600640 1 -- 172.21.15.28:0/2936297426 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0258035070 con 0x7f026c109850 2024-04-24T08:10:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.524+0000 7f0273800640 1 -- 172.21.15.28:0/2936297426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f026c1a6820 con 0x7f026c109850 2024-04-24T08:10:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.524+0000 7f0273800640 1 -- 172.21.15.28:0/2936297426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f026c1a6d60 con 0x7f026c109850 2024-04-24T08:10:06.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.525+0000 7f0273800640 1 -- 172.21.15.28:0/2936297426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f026c10f980 con 0x7f026c109850 2024-04-24T08:10:06.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.527+0000 7f0269600640 1 -- 172.21.15.28:0/2936297426 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f0258042030 con 0x7f026c109850 2024-04-24T08:10:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.529+0000 7f0269600640 1 --2- 172.21.15.28:0/2936297426 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f02540a3c30 0x7f02540a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.529+0000 7f0269600640 1 -- 172.21.15.28:0/2936297426 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f02580f08d0 con 0x7f026c109850 2024-04-24T08:10:06.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.529+0000 7f0270e00640 1 --2- 172.21.15.28:0/2936297426 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f02540a3c30 0x7f02540a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:06.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.530+0000 7f0270e00640 1 --2- 172.21.15.28:0/2936297426 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f02540a3c30 0x7f02540a60f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f026c1096b0 tx=0x7f0260005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:06.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.534+0000 7f0269600640 1 -- 172.21.15.28:0/2936297426 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f02580bcbc0 con 0x7f026c109850 2024-04-24T08:10:06.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:06 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1517742041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:10:06.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:06 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2137718210' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.739+0000 7f0273800640 1 -- 172.21.15.28:0/2936297426 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f026c062c00 con 0x7f026c109850 2024-04-24T08:10:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.740+0000 7f0269600640 1 -- 172.21.15.28:0/2936297426 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) ==== 93+0+10 (secure 0 0 0) 0x7f02580c0ec0 con 0x7f026c109850 2024-04-24T08:10:06.741 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-24T08:10:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.742+0000 7f0273800640 1 -- 172.21.15.28:0/2936297426 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f02540a3c30 msgr2=0x7f02540a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.742+0000 7f0273800640 1 --2- 172.21.15.28:0/2936297426 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f02540a3c30 0x7f02540a60f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f026c1096b0 tx=0x7f0260005f90 comp rx=0 tx=0).stop 2024-04-24T08:10:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.742+0000 7f0273800640 1 -- 172.21.15.28:0/2936297426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f026c109850 msgr2=0x7f026c19eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.742+0000 7f0273800640 1 --2- 172.21.15.28:0/2936297426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f026c109850 0x7f026c19eeb0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f0258031440 tx=0x7f0258032ce0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.742+0000 7f0273800640 1 -- 172.21.15.28:0/2936297426 shutdown_connections 2024-04-24T08:10:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.742+0000 7f0273800640 1 --2- 172.21.15.28:0/2936297426 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f02540a3c30 0x7f02540a60f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.742+0000 7f0273800640 1 --2- 172.21.15.28:0/2936297426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f026c109850 0x7f026c19eeb0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.742+0000 7f0273800640 1 --2- 172.21.15.28:0/2936297426 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f026c10a210 0x7f026c1a5f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.742+0000 7f0273800640 1 --2- 172.21.15.28:0/2936297426 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f026c108660 0x7f026c19e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.742+0000 7f0273800640 1 -- 172.21.15.28:0/2936297426 >> 172.21.15.28:0/2936297426 conn(0x7f026c077600 msgr2=0x7f026c10cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.743+0000 7f0273800640 1 -- 172.21.15.28:0/2936297426 shutdown_connections 2024-04-24T08:10:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.743+0000 7f0273800640 1 -- 172.21.15.28:0/2936297426 wait complete. 2024-04-24T08:10:06.763 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:10:06.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:06 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1517742041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:10:06.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:06 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2137718210' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:06.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:06 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1517742041' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:10:06.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:06 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2137718210' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:06.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.983+0000 7f5ccfe00640 1 -- 172.21.15.28:0/2543201055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cc8077c00 msgr2=0x7f5cc8078060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.983+0000 7f5ccfe00640 1 --2- 172.21.15.28:0/2543201055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cc8077c00 0x7f5cc8078060 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4009900 tx=0x7f5cb402a410 comp rx=0 tx=0).stop 2024-04-24T08:10:06.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.983+0000 7f5ccfe00640 1 -- 172.21.15.28:0/2543201055 shutdown_connections 2024-04-24T08:10:06.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.983+0000 7f5ccfe00640 1 --2- 172.21.15.28:0/2543201055 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cc8077c00 0x7f5cc8078060 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.983+0000 7f5ccfe00640 1 --2- 172.21.15.28:0/2543201055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cc80785a0 0x7f5cc8078e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.983+0000 7f5ccfe00640 1 --2- 172.21.15.28:0/2543201055 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5cc80769c0 0x7f5cc8076de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.983+0000 7f5ccfe00640 1 -- 172.21.15.28:0/2543201055 >> 172.21.15.28:0/2543201055 conn(0x7f5cc80ff7f0 msgr2=0x7f5cc8101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:06.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.983+0000 7f5ccfe00640 1 -- 172.21.15.28:0/2543201055 shutdown_connections 2024-04-24T08:10:06.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.983+0000 7f5ccfe00640 1 -- 172.21.15.28:0/2543201055 wait complete. 2024-04-24T08:10:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.984+0000 7f5ccfe00640 1 Processor -- start 2024-04-24T08:10:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.984+0000 7f5ccfe00640 1 -- start start 2024-04-24T08:10:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.985+0000 7f5ccfe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cc80769c0 0x7f5cc819e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.985+0000 7f5ccfe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5cc8077c00 0x7f5cc819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.985+0000 7f5ccfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cc80785a0 0x7f5cc81a5f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.985+0000 7f5ccfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc8114350 con 0x7f5cc80785a0 2024-04-24T08:10:06.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.985+0000 7f5ccfe00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc81141d0 con 0x7f5cc80769c0 2024-04-24T08:10:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.985+0000 7f5ccfe00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5cc81144d0 con 0x7f5cc8077c00 2024-04-24T08:10:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.985+0000 7f5ccde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cc80785a0 0x7f5cc81a5f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.985+0000 7f5ccd400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cc80769c0 0x7f5cc819e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.985+0000 7f5ccde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cc80785a0 0x7f5cc81a5f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50524/0 (socket says 172.21.15.28:50524) 2024-04-24T08:10:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.985+0000 7f5ccde00640 1 -- 172.21.15.28:0/651133047 learned_addr learned my addr 172.21.15.28:0/651133047 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.985+0000 7f5ccca00640 1 --2- 172.21.15.28:0/651133047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5cc8077c00 0x7f5cc819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.986+0000 7f5ccd400640 1 -- 172.21.15.28:0/651133047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5cc8077c00 msgr2=0x7f5cc819eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.986+0000 7f5ccd400640 1 --2- 172.21.15.28:0/651133047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5cc8077c00 0x7f5cc819eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.986+0000 7f5ccd400640 1 -- 172.21.15.28:0/651133047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cc80785a0 msgr2=0x7f5cc81a5f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.986+0000 7f5ccd400640 1 --2- 172.21.15.28:0/651133047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cc80785a0 0x7f5cc81a5f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.986+0000 7f5ccd400640 1 -- 172.21.15.28:0/651133047 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cc81a65d0 con 0x7f5cc80769c0 2024-04-24T08:10:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.986+0000 7f5ccde00640 1 --2- 172.21.15.28:0/651133047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cc80785a0 0x7f5cc81a5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:06.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.986+0000 7f5ccd400640 1 --2- 172.21.15.28:0/651133047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cc80769c0 0x7f5cc819e9b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0009fd0 tx=0x7f5cc000ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.986+0000 7f5cc6000640 1 -- 172.21.15.28:0/651133047 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc0004b20 con 0x7f5cc80769c0 2024-04-24T08:10:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.986+0000 7f5cc6000640 1 -- 172.21.15.28:0/651133047 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cc0004cc0 con 0x7f5cc80769c0 2024-04-24T08:10:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.986+0000 7f5ccfe00640 1 -- 172.21.15.28:0/651133047 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cc81a68c0 con 0x7f5cc80769c0 2024-04-24T08:10:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.986+0000 7f5cc6000640 1 -- 172.21.15.28:0/651133047 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cc00107b0 con 0x7f5cc80769c0 2024-04-24T08:10:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.987+0000 7f5ccfe00640 1 -- 172.21.15.28:0/651133047 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cc81a6de0 con 0x7f5cc80769c0 2024-04-24T08:10:06.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.988+0000 7f5ccfe00640 1 -- 172.21.15.28:0/651133047 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cc8110bf0 con 0x7f5cc80769c0 2024-04-24T08:10:06.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.990+0000 7f5cc6000640 1 -- 172.21.15.28:0/651133047 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f5cc0005020 con 0x7f5cc80769c0 2024-04-24T08:10:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.991+0000 7f5cc6000640 1 --2- 172.21.15.28:0/651133047 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5cac0a3c30 0x7f5cac0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.991+0000 7f5cc6000640 1 -- 172.21.15.28:0/651133047 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f5cc00d0fb0 con 0x7f5cc80769c0 2024-04-24T08:10:06.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.991+0000 7f5ccca00640 1 --2- 172.21.15.28:0/651133047 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5cac0a3c30 0x7f5cac0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:06.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.992+0000 7f5ccca00640 1 --2- 172.21.15.28:0/651133047 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5cac0a3c30 0x7f5cac0a60f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4031440 tx=0x7f5cb40315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:06.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:06.993+0000 7f5cc6000640 1 -- 172.21.15.28:0/651133047 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5cc009d8f0 con 0x7f5cc80769c0 2024-04-24T08:10:07.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.203+0000 7f5ccfe00640 1 -- 172.21.15.28:0/651133047 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f5cc802d8b0 con 0x7f5cc80769c0 2024-04-24T08:10:07.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.204+0000 7f5cc6000640 1 -- 172.21.15.28:0/651133047 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) ==== 104+0+11 (secure 0 0 0) 0x7f5cc009d1f0 con 0x7f5cc80769c0 2024-04-24T08:10:07.206 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-24T08:10:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.207+0000 7f5ccfe00640 1 -- 172.21.15.28:0/651133047 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5cac0a3c30 msgr2=0x7f5cac0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.207+0000 7f5ccfe00640 1 --2- 172.21.15.28:0/651133047 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5cac0a3c30 0x7f5cac0a60f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f5cb4031440 tx=0x7f5cb40315b0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.207+0000 7f5ccfe00640 1 -- 172.21.15.28:0/651133047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cc80769c0 msgr2=0x7f5cc819e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.207+0000 7f5ccfe00640 1 --2- 172.21.15.28:0/651133047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cc80769c0 0x7f5cc819e9b0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f5cc0009fd0 tx=0x7f5cc000ef90 comp rx=0 tx=0).stop 2024-04-24T08:10:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.207+0000 7f5ccfe00640 1 -- 172.21.15.28:0/651133047 shutdown_connections 2024-04-24T08:10:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.207+0000 7f5ccfe00640 1 --2- 172.21.15.28:0/651133047 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f5cac0a3c30 0x7f5cac0a60f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.207+0000 7f5ccfe00640 1 --2- 172.21.15.28:0/651133047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cc80785a0 0x7f5cc81a5f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.207+0000 7f5ccfe00640 1 --2- 172.21.15.28:0/651133047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5cc8077c00 0x7f5cc819eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.207+0000 7f5ccfe00640 1 --2- 172.21.15.28:0/651133047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cc80769c0 0x7f5cc819e9b0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.208+0000 7f5ccfe00640 1 -- 172.21.15.28:0/651133047 >> 172.21.15.28:0/651133047 conn(0x7f5cc80ff7f0 msgr2=0x7f5cc81012f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.208+0000 7f5ccfe00640 1 -- 172.21.15.28:0/651133047 shutdown_connections 2024-04-24T08:10:07.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.208+0000 7f5ccfe00640 1 -- 172.21.15.28:0/651133047 wait complete. 2024-04-24T08:10:07.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:10:07.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.441+0000 7fd8a1000640 1 -- 172.21.15.28:0/2353864694 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd89c074a60 msgr2=0x7fd89c074ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:07.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.441+0000 7fd8a1000640 1 --2- 172.21.15.28:0/2353864694 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd89c074a60 0x7fd89c074ee0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fd884009860 tx=0x7fd88402a340 comp rx=0 tx=0).stop 2024-04-24T08:10:07.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.441+0000 7fd8a1000640 1 -- 172.21.15.28:0/2353864694 shutdown_connections 2024-04-24T08:10:07.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.441+0000 7fd8a1000640 1 --2- 172.21.15.28:0/2353864694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd89c10e4e0 0x7fd89c1108d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.442+0000 7fd8a1000640 1 --2- 172.21.15.28:0/2353864694 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd89c074a60 0x7fd89c074ee0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.442+0000 7fd8a1000640 1 --2- 172.21.15.28:0/2353864694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd89c076040 0x7fd89c074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.442+0000 7fd8a1000640 1 -- 172.21.15.28:0/2353864694 >> 172.21.15.28:0/2353864694 conn(0x7fd89c0fd4a0 msgr2=0x7fd89c0ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.442+0000 7fd8a1000640 1 -- 172.21.15.28:0/2353864694 shutdown_connections 2024-04-24T08:10:07.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.442+0000 7fd8a1000640 1 -- 172.21.15.28:0/2353864694 wait complete. 2024-04-24T08:10:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.443+0000 7fd8a1000640 1 Processor -- start 2024-04-24T08:10:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.443+0000 7fd8a1000640 1 -- start start 2024-04-24T08:10:07.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.443+0000 7fd8a1000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd89c074a60 0x7fd89c19e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.443+0000 7fd8a1000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd89c076040 0x7fd89c19ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.444+0000 7fd8a1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd89c10e4e0 0x7fd89c1a5da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.444+0000 7fd8a1000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd89c114160 con 0x7fd89c10e4e0 2024-04-24T08:10:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.444+0000 7fd8a1000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd89c113fe0 con 0x7fd89c076040 2024-04-24T08:10:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.444+0000 7fd8a1000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd89c1142e0 con 0x7fd89c074a60 2024-04-24T08:10:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.444+0000 7fd89b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd89c074a60 0x7fd89c19e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.444+0000 7fd89b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd89c074a60 0x7fd89c19e7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:58040/0 (socket says 172.21.15.28:58040) 2024-04-24T08:10:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.444+0000 7fd89b400640 1 -- 172.21.15.28:0/2152149390 learned_addr learned my addr 172.21.15.28:0/2152149390 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.444+0000 7fd89be00640 1 --2- 172.21.15.28:0/2152149390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd89c10e4e0 0x7fd89c1a5da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.444+0000 7fd893e00640 1 --2- 172.21.15.28:0/2152149390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd89c076040 0x7fd89c19ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.444+0000 7fd89be00640 1 -- 172.21.15.28:0/2152149390 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd89c074a60 msgr2=0x7fd89c19e7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:07.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.444+0000 7fd89be00640 1 --2- 172.21.15.28:0/2152149390 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd89c074a60 0x7fd89c19e7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.444+0000 7fd89be00640 1 -- 172.21.15.28:0/2152149390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd89c076040 msgr2=0x7fd89c19ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.444+0000 7fd89be00640 1 --2- 172.21.15.28:0/2152149390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd89c076040 0x7fd89c19ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.444+0000 7fd89be00640 1 -- 172.21.15.28:0/2152149390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd89c1a6410 con 0x7fd89c10e4e0 2024-04-24T08:10:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.444+0000 7fd89b400640 1 --2- 172.21.15.28:0/2152149390 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd89c074a60 0x7fd89c19e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.445+0000 7fd89be00640 1 --2- 172.21.15.28:0/2152149390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd89c10e4e0 0x7fd89c1a5da0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd894009fd0 tx=0x7fd89400ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.445+0000 7fd898c00640 1 -- 172.21.15.28:0/2152149390 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd89400ce20 con 0x7fd89c10e4e0 2024-04-24T08:10:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.445+0000 7fd8a1000640 1 -- 172.21.15.28:0/2152149390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd89c1a6700 con 0x7fd89c10e4e0 2024-04-24T08:10:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.445+0000 7fd898c00640 1 -- 172.21.15.28:0/2152149390 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd894005020 con 0x7fd89c10e4e0 2024-04-24T08:10:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.445+0000 7fd898c00640 1 -- 172.21.15.28:0/2152149390 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd894010720 con 0x7fd89c10e4e0 2024-04-24T08:10:07.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.445+0000 7fd8a1000640 1 -- 172.21.15.28:0/2152149390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd89c1a6bd0 con 0x7fd89c10e4e0 2024-04-24T08:10:07.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.447+0000 7fd8a1000640 1 -- 172.21.15.28:0/2152149390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd89c1046b0 con 0x7fd89c10e4e0 2024-04-24T08:10:07.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.449+0000 7fd898c00640 1 -- 172.21.15.28:0/2152149390 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fd894005540 con 0x7fd89c10e4e0 2024-04-24T08:10:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.450+0000 7fd898c00640 1 --2- 172.21.15.28:0/2152149390 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd8800a3bb0 0x7fd8800a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.450+0000 7fd898c00640 1 -- 172.21.15.28:0/2152149390 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7fd8940d0b50 con 0x7fd89c10e4e0 2024-04-24T08:10:07.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.450+0000 7fd89b400640 1 --2- 172.21.15.28:0/2152149390 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd8800a3bb0 0x7fd8800a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:07.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.451+0000 7fd89b400640 1 --2- 172.21.15.28:0/2152149390 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd8800a3bb0 0x7fd8800a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd88c0096c0 tx=0x7fd88c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:07.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.454+0000 7fd898c00640 1 -- 172.21.15.28:0/2152149390 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd89409d490 con 0x7fd89c10e4e0 2024-04-24T08:10:07.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.663+0000 7fd8a1000640 1 -- 172.21.15.28:0/2152149390 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fd89c062c00 con 0x7fd89c10e4e0 2024-04-24T08:10:07.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.665+0000 7fd898c00640 1 -- 172.21.15.28:0/2152149390 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) ==== 100+0+11 (secure 0 0 0) 0x7fd89409cd90 con 0x7fd89c10e4e0 2024-04-24T08:10:07.666 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-24T08:10:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.667+0000 7fd8a1000640 1 -- 172.21.15.28:0/2152149390 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd8800a3bb0 msgr2=0x7fd8800a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.667+0000 7fd8a1000640 1 --2- 172.21.15.28:0/2152149390 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd8800a3bb0 0x7fd8800a6070 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fd88c0096c0 tx=0x7fd88c009260 comp rx=0 tx=0).stop 2024-04-24T08:10:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.667+0000 7fd8a1000640 1 -- 172.21.15.28:0/2152149390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd89c10e4e0 msgr2=0x7fd89c1a5da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.667+0000 7fd8a1000640 1 --2- 172.21.15.28:0/2152149390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd89c10e4e0 0x7fd89c1a5da0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd894009fd0 tx=0x7fd89400ef10 comp rx=0 tx=0).stop 2024-04-24T08:10:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.667+0000 7fd8a1000640 1 -- 172.21.15.28:0/2152149390 shutdown_connections 2024-04-24T08:10:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.667+0000 7fd8a1000640 1 --2- 172.21.15.28:0/2152149390 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fd8800a3bb0 0x7fd8800a6070 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.667+0000 7fd8a1000640 1 --2- 172.21.15.28:0/2152149390 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd89c10e4e0 0x7fd89c1a5da0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.667+0000 7fd8a1000640 1 --2- 172.21.15.28:0/2152149390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd89c076040 0x7fd89c19ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.667+0000 7fd8a1000640 1 --2- 172.21.15.28:0/2152149390 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd89c074a60 0x7fd89c19e7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.667+0000 7fd8a1000640 1 -- 172.21.15.28:0/2152149390 >> 172.21.15.28:0/2152149390 conn(0x7fd89c0fd4a0 msgr2=0x7fd89c101880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.668+0000 7fd8a1000640 1 -- 172.21.15.28:0/2152149390 shutdown_connections 2024-04-24T08:10:07.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.668+0000 7fd8a1000640 1 -- 172.21.15.28:0/2152149390 wait complete. 2024-04-24T08:10:07.689 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:10:07.751 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:07 smithi028 ceph-mon[27786]: pgmap v388: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:10:07.751 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:07 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2936297426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:10:07.751 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:07 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/651133047' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:10:07.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:07 smithi117 ceph-mon[32438]: pgmap v388: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:10:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:07 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2936297426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:10:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:07 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/651133047' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:10:07.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.889+0000 7f677dc00640 1 -- 172.21.15.28:0/700321231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6778104080 msgr2=0x7f67781044a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:07.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.889+0000 7f677dc00640 1 --2- 172.21.15.28:0/700321231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6778104080 0x7f67781044a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6768009ca0 tx=0x7f676802a8c0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.890+0000 7f677dc00640 1 -- 172.21.15.28:0/700321231 shutdown_connections 2024-04-24T08:10:07.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.890+0000 7f677dc00640 1 --2- 172.21.15.28:0/700321231 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6778105c30 0x7f677810c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.890+0000 7f677dc00640 1 --2- 172.21.15.28:0/700321231 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6778105270 0x7f67781056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.890+0000 7f677dc00640 1 --2- 172.21.15.28:0/700321231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6778104080 0x7f67781044a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.890+0000 7f677dc00640 1 -- 172.21.15.28:0/700321231 >> 172.21.15.28:0/700321231 conn(0x7f67780ff610 msgr2=0x7f6778101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:07.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.890+0000 7f677dc00640 1 -- 172.21.15.28:0/700321231 shutdown_connections 2024-04-24T08:10:07.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.890+0000 7f677dc00640 1 -- 172.21.15.28:0/700321231 wait complete. 2024-04-24T08:10:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.891+0000 7f677dc00640 1 Processor -- start 2024-04-24T08:10:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.891+0000 7f677dc00640 1 -- start start 2024-04-24T08:10:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.891+0000 7f677dc00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6778104080 0x7f677819a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.891+0000 7f677dc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6778105270 0x7f677819a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.891+0000 7f677dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6778105c30 0x7f67781a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.891+0000 7f677dc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f677810fd10 con 0x7f6778105c30 2024-04-24T08:10:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.891+0000 7f677dc00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f677810fb90 con 0x7f6778104080 2024-04-24T08:10:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.891+0000 7f677dc00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f677810fe90 con 0x7f6778105270 2024-04-24T08:10:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.891+0000 7f6777e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6778104080 0x7f677819a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.891+0000 7f6777e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6778104080 0x7f677819a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:52504/0 (socket says 172.21.15.28:52504) 2024-04-24T08:10:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.892+0000 7f6777e00640 1 -- 172.21.15.28:0/458030845 learned_addr learned my addr 172.21.15.28:0/458030845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.892+0000 7f677d200640 1 --2- 172.21.15.28:0/458030845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6778105c30 0x7f67781a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.892+0000 7f676f400640 1 --2- 172.21.15.28:0/458030845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6778105270 0x7f677819a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.892+0000 7f6777e00640 1 -- 172.21.15.28:0/458030845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6778105270 msgr2=0x7f677819a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.892+0000 7f6777e00640 1 --2- 172.21.15.28:0/458030845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6778105270 0x7f677819a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.892+0000 7f6777e00640 1 -- 172.21.15.28:0/458030845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6778105c30 msgr2=0x7f67781a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.892+0000 7f6777e00640 1 --2- 172.21.15.28:0/458030845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6778105c30 0x7f67781a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.892+0000 7f6777e00640 1 -- 172.21.15.28:0/458030845 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67781a1fc0 con 0x7f6778104080 2024-04-24T08:10:07.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.892+0000 7f6777e00640 1 --2- 172.21.15.28:0/458030845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6778104080 0x7f677819a3a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6768009bf0 tx=0x7f6768031ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.893+0000 7f6775600640 1 -- 172.21.15.28:0/458030845 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f676802fa20 con 0x7f6778104080 2024-04-24T08:10:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.893+0000 7f6775600640 1 -- 172.21.15.28:0/458030845 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f676802fbc0 con 0x7f6778104080 2024-04-24T08:10:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.893+0000 7f6775600640 1 -- 172.21.15.28:0/458030845 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6768046750 con 0x7f6778104080 2024-04-24T08:10:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.893+0000 7f677dc00640 1 -- 172.21.15.28:0/458030845 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67781a2250 con 0x7f6778104080 2024-04-24T08:10:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.893+0000 7f677dc00640 1 -- 172.21.15.28:0/458030845 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67781a27f0 con 0x7f6778104080 2024-04-24T08:10:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.894+0000 7f677dc00640 1 -- 172.21.15.28:0/458030845 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f677810b3a0 con 0x7f6778104080 2024-04-24T08:10:07.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.896+0000 7f6775600640 1 -- 172.21.15.28:0/458030845 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f676802fd70 con 0x7f6778104080 2024-04-24T08:10:07.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.897+0000 7f6775600640 1 --2- 172.21.15.28:0/458030845 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f675c0a3c30 0x7f675c0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:07.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.898+0000 7f6775600640 1 -- 172.21.15.28:0/458030845 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f67680f1880 con 0x7f6778104080 2024-04-24T08:10:07.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.898+0000 7f676f400640 1 --2- 172.21.15.28:0/458030845 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f675c0a3c30 0x7f675c0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:07.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.898+0000 7f676f400640 1 --2- 172.21.15.28:0/458030845 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f675c0a3c30 0x7f675c0a60f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f67600095f0 tx=0x7f6760005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:07.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:07.900+0000 7f6775600640 1 -- 172.21.15.28:0/458030845 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f67680be1c0 con 0x7f6778104080 2024-04-24T08:10:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:07 smithi194 ceph-mon[32836]: pgmap v388: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:10:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:07 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2936297426' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:10:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:07 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/651133047' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:10:08.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.148+0000 7f677dc00640 1 -- 172.21.15.28:0/458030845 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f677802d620 con 0x7f6778104080 2024-04-24T08:10:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.150+0000 7f6775600640 1 -- 172.21.15.28:0/458030845 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) ==== 94+0+11871 (secure 0 0 0) 0x7f67680bdac0 con 0x7f6778104080 2024-04-24T08:10:08.151 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:10:08.151 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":23,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":23,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:07:57.133790+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:10:08.151 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 23 2024-04-24T08:10:08.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.152+0000 7f677dc00640 1 -- 172.21.15.28:0/458030845 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f675c0a3c30 msgr2=0x7f675c0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:08.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.152+0000 7f677dc00640 1 --2- 172.21.15.28:0/458030845 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f675c0a3c30 0x7f675c0a60f0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f67600095f0 tx=0x7f6760005f90 comp rx=0 tx=0).stop 2024-04-24T08:10:08.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.152+0000 7f677dc00640 1 -- 172.21.15.28:0/458030845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6778104080 msgr2=0x7f677819a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:08.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.152+0000 7f677dc00640 1 --2- 172.21.15.28:0/458030845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6778104080 0x7f677819a3a0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f6768009bf0 tx=0x7f6768031ee0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.152+0000 7f677dc00640 1 -- 172.21.15.28:0/458030845 shutdown_connections 2024-04-24T08:10:08.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.152+0000 7f677dc00640 1 --2- 172.21.15.28:0/458030845 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f675c0a3c30 0x7f675c0a60f0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.152+0000 7f677dc00640 1 --2- 172.21.15.28:0/458030845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6778105c30 0x7f67781a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.152+0000 7f677dc00640 1 --2- 172.21.15.28:0/458030845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6778105270 0x7f677819a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.152+0000 7f677dc00640 1 --2- 172.21.15.28:0/458030845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6778104080 0x7f677819a3a0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.152+0000 7f677dc00640 1 -- 172.21.15.28:0/458030845 >> 172.21.15.28:0/458030845 conn(0x7f67780ff610 msgr2=0x7f6778108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:08.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.153+0000 7f677dc00640 1 -- 172.21.15.28:0/458030845 shutdown_connections 2024-04-24T08:10:08.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.153+0000 7f677dc00640 1 -- 172.21.15.28:0/458030845 wait complete. 2024-04-24T08:10:08.174 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:10:08.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.391+0000 7f6e6be00640 1 -- 172.21.15.28:0/4280945459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e6c105270 msgr2=0x7f6e6c1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:08.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.391+0000 7f6e6be00640 1 --2- 172.21.15.28:0/4280945459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e6c105270 0x7f6e6c1056f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f6e54009900 tx=0x7f6e5402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.391+0000 7f6e6be00640 1 -- 172.21.15.28:0/4280945459 shutdown_connections 2024-04-24T08:10:08.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.391+0000 7f6e6be00640 1 --2- 172.21.15.28:0/4280945459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e6c105270 0x7f6e6c1056f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.391+0000 7f6e6be00640 1 --2- 172.21.15.28:0/4280945459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e6c105c30 0x7f6e6c10c4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.391+0000 7f6e6be00640 1 --2- 172.21.15.28:0/4280945459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e6c104080 0x7f6e6c1044a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.391+0000 7f6e6be00640 1 -- 172.21.15.28:0/4280945459 >> 172.21.15.28:0/4280945459 conn(0x7f6e6c0ff610 msgr2=0x7f6e6c101a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:08.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.391+0000 7f6e6be00640 1 -- 172.21.15.28:0/4280945459 shutdown_connections 2024-04-24T08:10:08.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.392+0000 7f6e6be00640 1 -- 172.21.15.28:0/4280945459 wait complete. 2024-04-24T08:10:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.392+0000 7f6e6be00640 1 Processor -- start 2024-04-24T08:10:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.393+0000 7f6e6be00640 1 -- start start 2024-04-24T08:10:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.393+0000 7f6e6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e6c104080 0x7f6e6c19a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.393+0000 7f6e6be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e6c105270 0x7f6e6c19a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.393+0000 7f6e6be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e6c105c30 0x7f6e6c1a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.393+0000 7f6e6be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6c10fd10 con 0x7f6e6c104080 2024-04-24T08:10:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.393+0000 7f6e6be00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6c10fb90 con 0x7f6e6c105270 2024-04-24T08:10:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.393+0000 7f6e6be00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6c10fe90 con 0x7f6e6c105c30 2024-04-24T08:10:08.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.393+0000 7f6e6aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e6c104080 0x7f6e6c19a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.393+0000 7f6e6aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e6c104080 0x7f6e6c19a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50600/0 (socket says 172.21.15.28:50600) 2024-04-24T08:10:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.393+0000 7f6e6aa00640 1 -- 172.21.15.28:0/3297248246 learned_addr learned my addr 172.21.15.28:0/3297248246 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.393+0000 7f6e6b400640 1 --2- 172.21.15.28:0/3297248246 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e6c105c30 0x7f6e6c1a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.393+0000 7f6e62000640 1 --2- 172.21.15.28:0/3297248246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e6c105270 0x7f6e6c19a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.394+0000 7f6e6aa00640 1 -- 172.21.15.28:0/3297248246 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e6c105c30 msgr2=0x7f6e6c1a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.394+0000 7f6e6aa00640 1 --2- 172.21.15.28:0/3297248246 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e6c105c30 0x7f6e6c1a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.394+0000 7f6e6aa00640 1 -- 172.21.15.28:0/3297248246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e6c105270 msgr2=0x7f6e6c19a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.394+0000 7f6e6aa00640 1 --2- 172.21.15.28:0/3297248246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e6c105270 0x7f6e6c19a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.394+0000 7f6e6aa00640 1 -- 172.21.15.28:0/3297248246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e6c1a1f90 con 0x7f6e6c104080 2024-04-24T08:10:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.394+0000 7f6e6aa00640 1 --2- 172.21.15.28:0/3297248246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e6c104080 0x7f6e6c19a370 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c00e9c0 tx=0x7f6e5c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.394+0000 7f6e63e00640 1 -- 172.21.15.28:0/3297248246 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c00cd80 con 0x7f6e6c104080 2024-04-24T08:10:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.394+0000 7f6e63e00640 1 -- 172.21.15.28:0/3297248246 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e5c010910 con 0x7f6e6c104080 2024-04-24T08:10:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.394+0000 7f6e63e00640 1 -- 172.21.15.28:0/3297248246 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c018950 con 0x7f6e6c104080 2024-04-24T08:10:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.394+0000 7f6e6be00640 1 -- 172.21.15.28:0/3297248246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e6c1a2280 con 0x7f6e6c104080 2024-04-24T08:10:08.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.394+0000 7f6e6be00640 1 -- 172.21.15.28:0/3297248246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e6c1a27f0 con 0x7f6e6c104080 2024-04-24T08:10:08.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.396+0000 7f6e6be00640 1 -- 172.21.15.28:0/3297248246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e6c10b3a0 con 0x7f6e6c104080 2024-04-24T08:10:08.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.398+0000 7f6e63e00640 1 -- 172.21.15.28:0/3297248246 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f6e5c01f020 con 0x7f6e6c104080 2024-04-24T08:10:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.399+0000 7f6e63e00640 1 --2- 172.21.15.28:0/3297248246 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6e500a3c30 0x7f6e500a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.400+0000 7f6e63e00640 1 -- 172.21.15.28:0/3297248246 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f6e5c0d04b0 con 0x7f6e6c104080 2024-04-24T08:10:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.400+0000 7f6e62000640 1 --2- 172.21.15.28:0/3297248246 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6e500a3c30 0x7f6e500a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.401+0000 7f6e62000640 1 --2- 172.21.15.28:0/3297248246 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6e500a3c30 0x7f6e500a60f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6e540314e0 tx=0x7f6e54000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:08.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.404+0000 7f6e63e00640 1 -- 172.21.15.28:0/3297248246 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6e5c09cdf0 con 0x7f6e6c104080 2024-04-24T08:10:08.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.606+0000 7f6e6be00640 1 -- 172.21.15.28:0/3297248246 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6e6c062c00 con 0x7f6e6c104080 2024-04-24T08:10:08.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.611+0000 7f6e63e00640 1 -- 172.21.15.28:0/3297248246 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) ==== 74+0+22724 (secure 0 0 0) 0x7f6e5c09c6f0 con 0x7f6e6c104080 2024-04-24T08:10:08.613 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:10:08.613 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":90,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:09:51.414366+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:10:08.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.613+0000 7f6e6be00640 1 -- 172.21.15.28:0/3297248246 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6e500a3c30 msgr2=0x7f6e500a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:08.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.613+0000 7f6e6be00640 1 --2- 172.21.15.28:0/3297248246 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6e500a3c30 0x7f6e500a60f0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f6e540314e0 tx=0x7f6e54000bc0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.613+0000 7f6e6be00640 1 -- 172.21.15.28:0/3297248246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e6c104080 msgr2=0x7f6e6c19a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:08.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.613+0000 7f6e6be00640 1 --2- 172.21.15.28:0/3297248246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e6c104080 0x7f6e6c19a370 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c00e9c0 tx=0x7f6e5c00edc0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.614+0000 7f6e6be00640 1 -- 172.21.15.28:0/3297248246 shutdown_connections 2024-04-24T08:10:08.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.614+0000 7f6e6be00640 1 --2- 172.21.15.28:0/3297248246 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f6e500a3c30 0x7f6e500a60f0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.614+0000 7f6e6be00640 1 --2- 172.21.15.28:0/3297248246 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6e6c105c30 0x7f6e6c1a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.614+0000 7f6e6be00640 1 --2- 172.21.15.28:0/3297248246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e6c105270 0x7f6e6c19a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.614+0000 7f6e6be00640 1 --2- 172.21.15.28:0/3297248246 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6e6c104080 0x7f6e6c19a370 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.614+0000 7f6e6be00640 1 -- 172.21.15.28:0/3297248246 >> 172.21.15.28:0/3297248246 conn(0x7f6e6c0ff610 msgr2=0x7f6e6c108570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:08.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.614+0000 7f6e6be00640 1 -- 172.21.15.28:0/3297248246 shutdown_connections 2024-04-24T08:10:08.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.614+0000 7f6e6be00640 1 -- 172.21.15.28:0/3297248246 wait complete. 2024-04-24T08:10:08.635 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 23, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-04-24T08:07:51.585580+0000', 'modified': '2024-04-24T08:07:57.133790+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24574, 'mds_1': 24572, 'mds_2': 24556}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24515': {'gid': 24515, 'name': 'h', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.117:6837/862115546', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6836', 'nonce': 862115546}, {'type': 'v1', 'addr': '172.21.15.117:6837', 'nonce': 862115546}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24523': {'gid': 24523, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.194:6835/356867408', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6834', 'nonce': 356867408}, {'type': 'v1', 'addr': '172.21.15.194:6835', 'nonce': 356867408}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24535': {'gid': 24535, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.194:6837/3046177244', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6836', 'nonce': 3046177244}, {'type': 'v1', 'addr': '172.21.15.194:6837', 'nonce': 3046177244}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24556': {'gid': 24556, 'name': 'i', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 7, 'addr': '172.21.15.194:6839/2785495353', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.194:6838', 'nonce': 2785495353}, {'type': 'v1', 'addr': '172.21.15.194:6839', 'nonce': 2785495353}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24572': {'gid': 24572, 'name': 'b', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.117:6839/3606021351', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6838', 'nonce': 3606021351}, {'type': 'v1', 'addr': '172.21.15.117:6839', 'nonce': 3606021351}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24574': {'gid': 24574, 'name': 'g', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 5, 'addr': '172.21.15.28:6839/1247092633', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.28:6838', 'nonce': 1247092633}, {'type': 'v1', 'addr': '172.21.15.28:6839', 'nonce': 1247092633}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24574, 'qdb_cluster': [24556, 24572, 24574]} 2024-04-24T08:10:08.635 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-04-24T08:10:08.636 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-04-24T08:10:08.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:08 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2152149390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:10:08.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:08 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/458030845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:10:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.847+0000 7fc3a5a00640 1 -- 172.21.15.28:0/3216120207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a0105450 msgr2=0x7fc3a01058d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.847+0000 7fc3a5a00640 1 --2- 172.21.15.28:0/3216120207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a0105450 0x7fc3a01058d0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fc388009900 tx=0x7fc38802a410 comp rx=0 tx=0).stop 2024-04-24T08:10:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.847+0000 7fc3a5a00640 1 -- 172.21.15.28:0/3216120207 shutdown_connections 2024-04-24T08:10:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.847+0000 7fc3a5a00640 1 --2- 172.21.15.28:0/3216120207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a0105450 0x7fc3a01058d0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.847+0000 7fc3a5a00640 1 --2- 172.21.15.28:0/3216120207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3a0105e10 0x7fc3a010c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.847+0000 7fc3a5a00640 1 --2- 172.21.15.28:0/3216120207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3a0104260 0x7fc3a0104680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.847+0000 7fc3a5a00640 1 -- 172.21.15.28:0/3216120207 >> 172.21.15.28:0/3216120207 conn(0x7fc3a00ff7f0 msgr2=0x7fc3a0101c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.847+0000 7fc3a5a00640 1 -- 172.21.15.28:0/3216120207 shutdown_connections 2024-04-24T08:10:08.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.848+0000 7fc3a5a00640 1 -- 172.21.15.28:0/3216120207 wait complete. 2024-04-24T08:10:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.848+0000 7fc3a5a00640 1 Processor -- start 2024-04-24T08:10:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.849+0000 7fc3a5a00640 1 -- start start 2024-04-24T08:10:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.849+0000 7fc3a5a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3a0104260 0x7fc3a019a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.849+0000 7fc3a5a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3a0105450 0x7fc3a019aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.849+0000 7fc3a5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a0105e10 0x7fc3a01a1ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.849+0000 7fc3a5a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a010ff40 con 0x7fc3a0105e10 2024-04-24T08:10:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.849+0000 7fc3a5a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a010fdc0 con 0x7fc3a0104260 2024-04-24T08:10:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.849+0000 7fc3a5a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc3a01100c0 con 0x7fc3a0105450 2024-04-24T08:10:08.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.849+0000 7fc39f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a0105e10 0x7fc3a01a1ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.849+0000 7fc39ea00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3a0104260 0x7fc3a019a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.849+0000 7fc39f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a0105e10 0x7fc3a01a1ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50618/0 (socket says 172.21.15.28:50618) 2024-04-24T08:10:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.849+0000 7fc39f400640 1 -- 172.21.15.28:0/2444924749 learned_addr learned my addr 172.21.15.28:0/2444924749 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.849+0000 7fc396000640 1 --2- 172.21.15.28:0/2444924749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3a0105450 0x7fc3a019aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.850+0000 7fc39f400640 1 -- 172.21.15.28:0/2444924749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3a0105450 msgr2=0x7fc3a019aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.850+0000 7fc39f400640 1 --2- 172.21.15.28:0/2444924749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3a0105450 0x7fc3a019aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.850+0000 7fc39f400640 1 -- 172.21.15.28:0/2444924749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3a0104260 msgr2=0x7fc3a019a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.850+0000 7fc39f400640 1 --2- 172.21.15.28:0/2444924749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3a0104260 0x7fc3a019a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.850+0000 7fc39f400640 1 -- 172.21.15.28:0/2444924749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3a01a2140 con 0x7fc3a0105e10 2024-04-24T08:10:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.850+0000 7fc39ea00640 1 --2- 172.21.15.28:0/2444924749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3a0104260 0x7fc3a019a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.850+0000 7fc39f400640 1 --2- 172.21.15.28:0/2444924749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a0105e10 0x7fc3a01a1ad0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc39800eaa0 tx=0x7fc39800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.850+0000 7fc397e00640 1 -- 172.21.15.28:0/2444924749 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39800cdb0 con 0x7fc3a0105e10 2024-04-24T08:10:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.850+0000 7fc397e00640 1 -- 172.21.15.28:0/2444924749 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc398005020 con 0x7fc3a0105e10 2024-04-24T08:10:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.850+0000 7fc3a5a00640 1 -- 172.21.15.28:0/2444924749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3a01a2430 con 0x7fc3a0105e10 2024-04-24T08:10:08.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.850+0000 7fc397e00640 1 -- 172.21.15.28:0/2444924749 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3980116b0 con 0x7fc3a0105e10 2024-04-24T08:10:08.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.850+0000 7fc3a5a00640 1 -- 172.21.15.28:0/2444924749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3a01a2900 con 0x7fc3a0105e10 2024-04-24T08:10:08.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.854+0000 7fc3a87d2740 1 Processor -- start 2024-04-24T08:10:08.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.854+0000 7fc397e00640 1 -- 172.21.15.28:0/2444924749 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7fc398005560 con 0x7fc3a0105e10 2024-04-24T08:10:08.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.854+0000 7fc3a87d2740 1 -- start start 2024-04-24T08:10:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.854+0000 7fc3a87d2740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f36090 0x55ccb5f364b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.854+0000 7fc3a87d2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36a80 0x55ccb5f3b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.854+0000 7fc3a87d2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5f3e080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.854+0000 7fc3a87d2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ccb5e9bb10 con 0x55ccb5f3bbf0 2024-04-24T08:10:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.854+0000 7fc3a87d2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ccb5e422b0 con 0x55ccb5f36a80 2024-04-24T08:10:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.854+0000 7fc3a87d2740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ccb5e97ad0 con 0x55ccb5f36090 2024-04-24T08:10:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.855+0000 7fc396000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5f3e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.855+0000 7fc39ea00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36a80 0x55ccb5f3b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.855+0000 7fc39ea00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36a80 0x55ccb5f3b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:52566/0 (socket says 172.21.15.28:52566) 2024-04-24T08:10:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.855+0000 7fc39ea00640 1 -- 172.21.15.28:0/4021312345 learned_addr learned my addr 172.21.15.28:0/4021312345 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:08.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.855+0000 7fc396000640 1 --2- 172.21.15.28:0/4021312345 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f36090 0x55ccb5f364b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.855+0000 7fc39ea00640 1 -- 172.21.15.28:0/4021312345 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f36090 msgr2=0x55ccb5f364b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.855+0000 7fc39ea00640 1 --2- 172.21.15.28:0/4021312345 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f36090 0x55ccb5f364b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.855+0000 7fc39ea00640 1 -- 172.21.15.28:0/4021312345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 msgr2=0x55ccb5f3e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.855+0000 7fc39ea00640 1 --2- 172.21.15.28:0/4021312345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5f3e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.855+0000 7fc39ea00640 1 -- 172.21.15.28:0/4021312345 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccb5bae2d0 con 0x55ccb5f36a80 2024-04-24T08:10:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.855+0000 7fc39ea00640 1 --2- 172.21.15.28:0/4021312345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36a80 0x55ccb5f3b580 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc3900114c0 tx=0x7fc390011980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ed674bb27ba11fc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.855+0000 7fc38fe00640 1 -- 172.21.15.28:0/4021312345 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc390004020 con 0x55ccb5f36a80 2024-04-24T08:10:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.856+0000 7fc38fe00640 1 -- 172.21.15.28:0/4021312345 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc390005500 con 0x55ccb5f36a80 2024-04-24T08:10:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.856+0000 7fc38fe00640 1 -- 172.21.15.28:0/4021312345 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc390004df0 con 0x55ccb5f36a80 2024-04-24T08:10:08.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.856+0000 7fc397e00640 1 --2- 172.21.15.28:0/2444924749 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc3840a3bb0 0x7fc3840a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.856+0000 7fc397e00640 1 -- 172.21.15.28:0/2444924749 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7fc3980d0630 con 0x7fc3a0105e10 2024-04-24T08:10:08.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.856+0000 7fc396000640 1 --2- 172.21.15.28:0/2444924749 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc3840a3bb0 0x7fc3840a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.856+0000 7fc3a87d2740 1 -- 172.21.15.28:0/4021312345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36a80 msgr2=0x55ccb5f3b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:08.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.856+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/4021312345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36a80 0x55ccb5f3b580 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fc3900114c0 tx=0x7fc390011980 comp rx=0 tx=0).stop 2024-04-24T08:10:08.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.856+0000 7fc3a87d2740 1 -- 172.21.15.28:0/4021312345 shutdown_connections 2024-04-24T08:10:08.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.856+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/4021312345 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5f3e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.856+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/4021312345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36a80 0x55ccb5f3b580 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.856+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/4021312345 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f36090 0x55ccb5f364b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.856+0000 7fc3a87d2740 1 -- 172.21.15.28:0/4021312345 >> 172.21.15.28:0/4021312345 conn(0x55ccb5f31700 msgr2=0x55ccb5f31b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:08.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.857+0000 7fc3a87d2740 1 -- 172.21.15.28:0/4021312345 shutdown_connections 2024-04-24T08:10:08.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.857+0000 7fc3a87d2740 1 -- 172.21.15.28:0/4021312345 wait complete. 2024-04-24T08:10:08.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.857+0000 7fc396000640 1 --2- 172.21.15.28:0/2444924749 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc3840a3bb0 0x7fc3840a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc388031440 tx=0x7fc388000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:08.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.857+0000 7fc3a87d2740 1 -- start start 2024-04-24T08:10:08.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.857+0000 7fc3a87d2740 1 Processor -- start 2024-04-24T08:10:08.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.858+0000 7fc3a87d2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36090 0x55ccb5f404a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.858+0000 7fc3a87d2740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f36a80 0x55ccb5f409e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.858+0000 7fc3a87d2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5f47a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.858+0000 7fc3a87d2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ccb5f3eba0 con 0x55ccb5f3bbf0 2024-04-24T08:10:08.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.858+0000 7fc3a87d2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ccb5e55590 con 0x55ccb5f36090 2024-04-24T08:10:08.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.858+0000 7fc3a87d2740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ccb5f3ed20 con 0x55ccb5f36a80 2024-04-24T08:10:08.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.858+0000 7fc3a87d2740 10 client.0 _ll_register_callbacks cb 0x55ccb5e383a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:10:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.858+0000 7fc39f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5f47a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.858+0000 7fc39f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5f47a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50636/0 (socket says 172.21.15.28:50636) 2024-04-24T08:10:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.858+0000 7fc39f400640 1 -- 172.21.15.28:0/1624592394 learned_addr learned my addr 172.21.15.28:0/1624592394 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.858+0000 7fc39ea00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36090 0x55ccb5f404a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.858+0000 7fc396000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f36a80 0x55ccb5f409e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.858+0000 7fc39f400640 1 -- 172.21.15.28:0/1624592394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f36a80 msgr2=0x55ccb5f409e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.858+0000 7fc39f400640 1 --2- 172.21.15.28:0/1624592394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f36a80 0x55ccb5f409e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.858+0000 7fc39f400640 1 -- 172.21.15.28:0/1624592394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36090 msgr2=0x55ccb5f404a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.858+0000 7fc39f400640 1 --2- 172.21.15.28:0/1624592394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36090 0x55ccb5f404a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.858+0000 7fc39f400640 1 -- 172.21.15.28:0/1624592394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccb5c786b0 con 0x55ccb5f3bbf0 2024-04-24T08:10:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.859+0000 7fc39f400640 1 --2- 172.21.15.28:0/1624592394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5f47a20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc39809c8b0 tx=0x7fc3980a54e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b701b4f2c63ac68 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.859+0000 7fc38cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:10:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.859+0000 7fc38cc00640 1 -- 172.21.15.28:0/1624592394 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39801f040 con 0x55ccb5f3bbf0 2024-04-24T08:10:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.859+0000 7fc38cc00640 1 -- 172.21.15.28:0/1624592394 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7fc3980872c0 con 0x55ccb5f3bbf0 2024-04-24T08:10:08.864 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.859+0000 7fc3a87d2740 1 -- 172.21.15.28:0/1624592394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc390005500 con 0x55ccb5f3bbf0 2024-04-24T08:10:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.859+0000 7fc38cc00640 1 -- 172.21.15.28:0/1624592394 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc39809dba0 con 0x55ccb5f3bbf0 2024-04-24T08:10:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.859+0000 7fc3a87d2740 10 client.15042 fetch_fsmap learned FSMap version 23 2024-04-24T08:10:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.859+0000 7fc3a87d2740 1 -- 172.21.15.28:0/1624592394 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ccb5f4ae00 con 0x55ccb5f3bbf0 2024-04-24T08:10:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.859+0000 7fc38cc00640 1 -- 172.21.15.28:0/1624592394 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc390005500 con 0x55ccb5f3bbf0 2024-04-24T08:10:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.859+0000 7fc38cc00640 1 -- 172.21.15.28:0/1624592394 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc39800f040 con 0x55ccb5f3bbf0 2024-04-24T08:10:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.860+0000 7fc3a87d2740 10 client.15042 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:10:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.860+0000 7fc3a87d2740 10 client.15042 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-24T08:10:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.860+0000 7fc3a87d2740 20 client.15042 populate_metadata read hostname 'smithi028' 2024-04-24T08:10:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.860+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/1624592394 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55ccb5f4d510 0x55ccb5f4e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.860+0000 7fc3a87d2740 4 client.15042 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:10:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.860+0000 7fc3a87d2740 1 -- 172.21.15.28:0/1624592394 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ccb5ea9530 con 0x55ccb5f4d510 2024-04-24T08:10:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.860+0000 7fc39ea00640 1 --2- 172.21.15.28:0/1624592394 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55ccb5f4d510 0x55ccb5f4e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.861+0000 7fc39ea00640 1 --2- 172.21.15.28:0/1624592394 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55ccb5f4d510 0x55ccb5f4e980 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cbfdb30549f4b02d server_cookie=101d9d82d029e30d in_seq=0 out_seq=0 2024-04-24T08:10:08.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.861+0000 7fc38cc00640 10 client.15042 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:10:08.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.862+0000 7fc38cc00640 1 -- 172.21.15.28:0/1624592394 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ccb5ea9530 con 0x55ccb5f4d510 2024-04-24T08:10:08.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.862+0000 7fc38cc00640 10 client.15042 handle_command_reply: tid=0 2024-04-24T08:10:08.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.862+0000 7fc3a87d2740 1 client.15042 shutdown 2024-04-24T08:10:08.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.862+0000 7fc3a87d2740 1 -- 172.21.15.28:0/1624592394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 msgr2=0x55ccb5f47a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:08.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.862+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/1624592394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5f47a20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fc39809c8b0 tx=0x7fc3980a54e0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.862+0000 7fc3a87d2740 1 -- 172.21.15.28:0/1624592394 shutdown_connections 2024-04-24T08:10:08.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.862+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/1624592394 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55ccb5f4d510 0x55ccb5f4e980 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.862+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/1624592394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5f47a20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.862+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/1624592394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f36a80 0x55ccb5f409e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.862+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/1624592394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36090 0x55ccb5f404a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.862+0000 7fc38cc00640 0 client.15042 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:10:08.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.862+0000 7fc3a87d2740 1 -- 172.21.15.28:0/1624592394 >> 172.21.15.28:0/1624592394 conn(0x55ccb5f31700 msgr2=0x55ccb5f308c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:08.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.862+0000 7fc3a87d2740 1 -- 172.21.15.28:0/1624592394 shutdown_connections 2024-04-24T08:10:08.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.862+0000 7fc3a87d2740 1 -- 172.21.15.28:0/1624592394 wait complete. 2024-04-24T08:10:08.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.862+0000 7fc3a87d2740 20 client.15042 trim_cache size 0 max 16384 2024-04-24T08:10:08.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:08 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2152149390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:10:08.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:08 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/458030845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:10:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.881+0000 7fc3a87d2740 1 Processor -- start 2024-04-24T08:10:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.881+0000 7fc3a87d2740 1 -- start start 2024-04-24T08:10:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.882+0000 7fc3a87d2740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f36090 0x55ccb5f3b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.882+0000 7fc3a87d2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36a80 0x55ccb5f479e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.882+0000 7fc3a87d2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5f2c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.882+0000 7fc3a87d2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ccb5f4ccc0 con 0x55ccb5f3bbf0 2024-04-24T08:10:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.882+0000 7fc3a87d2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ccb5f4d390 con 0x55ccb5f36a80 2024-04-24T08:10:08.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.882+0000 7fc3a87d2740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ccb5f4da60 con 0x55ccb5f36090 2024-04-24T08:10:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.882+0000 7fc39f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5f2c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.882+0000 7fc39f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5f2c750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50640/0 (socket says 172.21.15.28:50640) 2024-04-24T08:10:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.882+0000 7fc39f400640 1 -- 172.21.15.28:0/739872522 learned_addr learned my addr 172.21.15.28:0/739872522 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.882+0000 7fc39ea00640 1 --2- 172.21.15.28:0/739872522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f36090 0x55ccb5f3b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.882+0000 7fc396000640 1 --2- 172.21.15.28:0/739872522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36a80 0x55ccb5f479e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.882+0000 7fc39f400640 1 -- 172.21.15.28:0/739872522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f36090 msgr2=0x55ccb5f3b4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.882+0000 7fc39f400640 1 --2- 172.21.15.28:0/739872522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f36090 0x55ccb5f3b4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.882+0000 7fc39f400640 1 -- 172.21.15.28:0/739872522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36a80 msgr2=0x55ccb5f479e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.882+0000 7fc39f400640 1 --2- 172.21.15.28:0/739872522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36a80 0x55ccb5f479e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.882+0000 7fc39f400640 1 -- 172.21.15.28:0/739872522 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ccb5f35620 con 0x55ccb5f3bbf0 2024-04-24T08:10:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.883+0000 7fc39f400640 1 --2- 172.21.15.28:0/739872522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5f2c750 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc398098d80 tx=0x7fc3980054d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50a65627ee2cb8e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.883+0000 7fc38f400640 1 -- 172.21.15.28:0/739872522 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39801f070 con 0x55ccb5f3bbf0 2024-04-24T08:10:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.883+0000 7fc38f400640 1 -- 172.21.15.28:0/739872522 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3980bb020 con 0x55ccb5f3bbf0 2024-04-24T08:10:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.883+0000 7fc38f400640 1 -- 172.21.15.28:0/739872522 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc39809b7e0 con 0x55ccb5f3bbf0 2024-04-24T08:10:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.883+0000 7fc3a87d2740 1 -- 172.21.15.28:0/739872522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 msgr2=0x55ccb5f2c750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.883+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/739872522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5f2c750 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc398098d80 tx=0x7fc3980054d0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.884+0000 7fc3a87d2740 1 -- 172.21.15.28:0/739872522 shutdown_connections 2024-04-24T08:10:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.884+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/739872522 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5f2c750 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.884+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/739872522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36a80 0x55ccb5f479e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.884+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/739872522 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f36090 0x55ccb5f3b4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.884+0000 7fc3a87d2740 1 -- 172.21.15.28:0/739872522 >> 172.21.15.28:0/739872522 conn(0x55ccb5f31700 msgr2=0x55ccb5f31ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.884+0000 7fc3a87d2740 1 -- 172.21.15.28:0/739872522 shutdown_connections 2024-04-24T08:10:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.884+0000 7fc3a87d2740 1 -- 172.21.15.28:0/739872522 wait complete. 2024-04-24T08:10:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.884+0000 7fc3a87d2740 1 -- start start 2024-04-24T08:10:08.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.884+0000 7fc3a87d2740 1 Processor -- start 2024-04-24T08:10:08.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc3a87d2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36090 0x55ccb5f459d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc3a87d2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f36a80 0x55ccb5fb7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc3a87d2740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5fba200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc3a87d2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ccb5f4e800 con 0x55ccb5f36a80 2024-04-24T08:10:08.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc3a87d2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ccb5f4e130 con 0x55ccb5f36090 2024-04-24T08:10:08.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc3a87d2740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ccb5f4eed0 con 0x55ccb5f3bbf0 2024-04-24T08:10:08.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc3a87d2740 10 client.0 _ll_register_callbacks cb 0x7fc39809c9d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:10:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc396000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f36a80 0x55ccb5fb7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc396000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f36a80 0x55ccb5fb7cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50650/0 (socket says 172.21.15.28:50650) 2024-04-24T08:10:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc396000640 1 -- 172.21.15.28:0/2414572786 learned_addr learned my addr 172.21.15.28:0/2414572786 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc39ea00640 1 --2- 172.21.15.28:0/2414572786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36090 0x55ccb5f459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc39f400640 1 --2- 172.21.15.28:0/2414572786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5fba200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc396000640 1 -- 172.21.15.28:0/2414572786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f3bbf0 msgr2=0x55ccb5fba200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc396000640 1 --2- 172.21.15.28:0/2414572786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5fba200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc396000640 1 -- 172.21.15.28:0/2414572786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36090 msgr2=0x55ccb5f459d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc396000640 1 --2- 172.21.15.28:0/2414572786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36090 0x55ccb5f459d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc396000640 1 -- 172.21.15.28:0/2414572786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ccb5fba940 con 0x55ccb5f36a80 2024-04-24T08:10:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc396000640 1 --2- 172.21.15.28:0/2414572786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f36a80 0x55ccb5fb7cb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc388009900 tx=0x7fc388005780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71fc7df1a770b979 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.885+0000 7fc38cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:10:08.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.886+0000 7fc38cc00640 1 -- 172.21.15.28:0/2414572786 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc388057060 con 0x55ccb5f36a80 2024-04-24T08:10:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.886+0000 7fc38cc00640 1 -- 172.21.15.28:0/2414572786 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7fc3880037f0 con 0x55ccb5f36a80 2024-04-24T08:10:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.886+0000 7fc3a87d2740 1 -- 172.21.15.28:0/2414572786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc3980bb020 con 0x55ccb5f36a80 2024-04-24T08:10:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.886+0000 7fc38cc00640 1 -- 172.21.15.28:0/2414572786 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7fc388035070 con 0x55ccb5f36a80 2024-04-24T08:10:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.886+0000 7fc3a87d2740 10 client.15048 fetch_fsmap learned FSMap version 23 2024-04-24T08:10:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.886+0000 7fc3a87d2740 1 -- 172.21.15.28:0/2414572786 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x55ccb5fbbcb0 con 0x55ccb5f36a80 2024-04-24T08:10:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.886+0000 7fc38cc00640 1 -- 172.21.15.28:0/2414572786 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3980bb020 con 0x55ccb5f36a80 2024-04-24T08:10:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.886+0000 7fc38cc00640 1 -- 172.21.15.28:0/2414572786 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7fc388051070 con 0x55ccb5f36a80 2024-04-24T08:10:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.886+0000 7fc3a87d2740 10 client.15048 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:10:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.886+0000 7fc3a87d2740 10 client.15048 resolve_mds: resolved name 'i' to daemon mds.i 2024-04-24T08:10:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.887+0000 7fc3a87d2740 20 client.15048 populate_metadata read hostname 'smithi028' 2024-04-24T08:10:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.887+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/2414572786 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55ccb5fc0860 0x55ccb5fbf570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.887+0000 7fc3a87d2740 4 client.15048 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:10:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.887+0000 7fc3a87d2740 1 -- 172.21.15.28:0/2414572786 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "status"}) -- 0x55ccb5ea33a0 con 0x55ccb5fc0860 2024-04-24T08:10:08.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.887+0000 7fc39ea00640 1 --2- 172.21.15.28:0/2414572786 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55ccb5fc0860 0x55ccb5fbf570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:08.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.888+0000 7fc39ea00640 1 --2- 172.21.15.28:0/2414572786 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55ccb5fc0860 0x55ccb5fbf570 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9054b4d91109b151 server_cookie=41cb4b460c49de6 in_seq=0 out_seq=0 2024-04-24T08:10:08.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.888+0000 7fc38cc00640 10 client.15048 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:10:08.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.888+0000 7fc38cc00640 1 -- 172.21.15.28:0/2414572786 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55ccb5ea33a0 con 0x55ccb5fc0860 2024-04-24T08:10:08.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.888+0000 7fc38cc00640 10 client.15048 handle_command_reply: tid=0 2024-04-24T08:10:08.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.888+0000 7fc3a87d2740 1 client.15048 shutdown 2024-04-24T08:10:08.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.888+0000 7fc3a87d2740 1 -- 172.21.15.28:0/2414572786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f36a80 msgr2=0x55ccb5fb7cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:08.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.888+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/2414572786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f36a80 0x55ccb5fb7cb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc388009900 tx=0x7fc388005780 comp rx=0 tx=0).stop 2024-04-24T08:10:08.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.888+0000 7fc3a87d2740 1 -- 172.21.15.28:0/2414572786 shutdown_connections 2024-04-24T08:10:08.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.888+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/2414572786 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55ccb5fc0860 0x55ccb5fbf570 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.888+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/2414572786 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ccb5f36a80 0x55ccb5fb7cb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.888+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/2414572786 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ccb5f3bbf0 0x55ccb5fba200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.888+0000 7fc38cc00640 0 client.15048 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:10:08.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.888+0000 7fc3a87d2740 1 --2- 172.21.15.28:0/2414572786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ccb5f36090 0x55ccb5f459d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.888+0000 7fc3a87d2740 1 -- 172.21.15.28:0/2414572786 >> 172.21.15.28:0/2414572786 conn(0x55ccb5f31700 msgr2=0x55ccb5f2c750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:08.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.888+0000 7fc3a87d2740 1 -- 172.21.15.28:0/2414572786 shutdown_connections 2024-04-24T08:10:08.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.888+0000 7fc3a87d2740 1 -- 172.21.15.28:0/2414572786 wait complete. 2024-04-24T08:10:08.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.888+0000 7fc3a87d2740 20 client.15048 trim_cache size 0 max 16384 2024-04-24T08:10:08.894 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:10:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "473aaf58-0210-11ef-bc93-c7b262605968", 2024-04-24T08:10:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 2, 2024-04-24T08:10:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24556, 2024-04-24T08:10:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-24T08:10:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-24T08:10:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-24T08:10:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 132.76371571799999, 2024-04-24T08:10:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 23, 2024-04-24T08:10:08.894 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 90, 2024-04-24T08:10:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 86, 2024-04-24T08:10:08.895 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 154.36490791400001 2024-04-24T08:10:08.895 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:10:08.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.889+0000 7fc3a5a00640 1 -- 172.21.15.28:0/2444924749 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc3840a3bb0 msgr2=0x7fc3840a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:08.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.889+0000 7fc3a5a00640 1 --2- 172.21.15.28:0/2444924749 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc3840a3bb0 0x7fc3840a6070 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fc388031440 tx=0x7fc388000bc0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.889+0000 7fc3a5a00640 1 -- 172.21.15.28:0/2444924749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a0105e10 msgr2=0x7fc3a01a1ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:08.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.889+0000 7fc3a5a00640 1 --2- 172.21.15.28:0/2444924749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a0105e10 0x7fc3a01a1ad0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fc39800eaa0 tx=0x7fc39800ef60 comp rx=0 tx=0).stop 2024-04-24T08:10:08.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.889+0000 7fc3a5a00640 1 -- 172.21.15.28:0/2444924749 shutdown_connections 2024-04-24T08:10:08.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.889+0000 7fc3a5a00640 1 --2- 172.21.15.28:0/2444924749 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fc3840a3bb0 0x7fc3840a6070 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.889+0000 7fc3a5a00640 1 --2- 172.21.15.28:0/2444924749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3a0105e10 0x7fc3a01a1ad0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.889+0000 7fc3a5a00640 1 --2- 172.21.15.28:0/2444924749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3a0105450 0x7fc3a019aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.889+0000 7fc3a5a00640 1 --2- 172.21.15.28:0/2444924749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3a0104260 0x7fc3a019a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:08.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.889+0000 7fc3a5a00640 1 -- 172.21.15.28:0/2444924749 >> 172.21.15.28:0/2444924749 conn(0x7fc3a00ff7f0 msgr2=0x7fc3a01086f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:08.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.889+0000 7fc3a5a00640 1 -- 172.21.15.28:0/2444924749 shutdown_connections 2024-04-24T08:10:08.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:08.889+0000 7fc3a5a00640 1 -- 172.21.15.28:0/2444924749 wait complete. 2024-04-24T08:10:08.908 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-04-24T08:10:08.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:08 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2152149390' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:10:08.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:08 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/458030845' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:10:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.110+0000 7f56db200640 1 -- 172.21.15.28:0/3331528290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56dc0785a0 msgr2=0x7f56dc078e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.110+0000 7f56db200640 1 --2- 172.21.15.28:0/3331528290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56dc0785a0 0x7f56dc078e40 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f56d4009be0 tx=0x7f56d402a6d0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.110+0000 7f56db200640 1 -- 172.21.15.28:0/3331528290 shutdown_connections 2024-04-24T08:10:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.110+0000 7f56db200640 1 --2- 172.21.15.28:0/3331528290 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56dc0785a0 0x7f56dc078e40 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.110+0000 7f56db200640 1 --2- 172.21.15.28:0/3331528290 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56dc077c00 0x7f56dc078060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.110+0000 7f56db200640 1 --2- 172.21.15.28:0/3331528290 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56dc0769c0 0x7f56dc076de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.110+0000 7f56db200640 1 -- 172.21.15.28:0/3331528290 >> 172.21.15.28:0/3331528290 conn(0x7f56dc0ff6b0 msgr2=0x7f56dc101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.110+0000 7f56db200640 1 -- 172.21.15.28:0/3331528290 shutdown_connections 2024-04-24T08:10:09.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.110+0000 7f56db200640 1 -- 172.21.15.28:0/3331528290 wait complete. 2024-04-24T08:10:09.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.111+0000 7f56db200640 1 Processor -- start 2024-04-24T08:10:09.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.112+0000 7f56db200640 1 -- start start 2024-04-24T08:10:09.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.112+0000 7f56db200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56dc0769c0 0x7f56dc19e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.112+0000 7f56db200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56dc077c00 0x7f56dc19ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.112+0000 7f56db200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56dc0785a0 0x7f56dc1a5db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.112+0000 7f56db200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f56dc1141d0 con 0x7f56dc0769c0 2024-04-24T08:10:09.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.112+0000 7f56db200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f56dc114050 con 0x7f56dc0785a0 2024-04-24T08:10:09.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.112+0000 7f56db200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f56dc114350 con 0x7f56dc077c00 2024-04-24T08:10:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.112+0000 7f56d9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56dc0769c0 0x7f56dc19e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.112+0000 7f56d9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56dc0769c0 0x7f56dc19e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50662/0 (socket says 172.21.15.28:50662) 2024-04-24T08:10:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.112+0000 7f56d9e00640 1 -- 172.21.15.28:0/3030668241 learned_addr learned my addr 172.21.15.28:0/3030668241 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.112+0000 7f56d9400640 1 --2- 172.21.15.28:0/3030668241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56dc077c00 0x7f56dc19ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.113+0000 7f56da800640 1 --2- 172.21.15.28:0/3030668241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56dc0785a0 0x7f56dc1a5db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.113+0000 7f56d9e00640 1 -- 172.21.15.28:0/3030668241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56dc077c00 msgr2=0x7f56dc19ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.113+0000 7f56d9e00640 1 --2- 172.21.15.28:0/3030668241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56dc077c00 0x7f56dc19ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.113+0000 7f56d9e00640 1 -- 172.21.15.28:0/3030668241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56dc0785a0 msgr2=0x7f56dc1a5db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.113+0000 7f56d9e00640 1 --2- 172.21.15.28:0/3030668241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56dc0785a0 0x7f56dc1a5db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.113+0000 7f56d9e00640 1 -- 172.21.15.28:0/3030668241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f56dc1a6420 con 0x7f56dc0769c0 2024-04-24T08:10:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.113+0000 7f56d9e00640 1 --2- 172.21.15.28:0/3030668241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56dc0769c0 0x7f56dc19e800 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00e9c0 tx=0x7f56cc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.114+0000 7f56d2a00640 1 -- 172.21.15.28:0/3030668241 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc00ccc0 con 0x7f56dc0769c0 2024-04-24T08:10:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.114+0000 7f56db200640 1 -- 172.21.15.28:0/3030668241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f56dc1a6710 con 0x7f56dc0769c0 2024-04-24T08:10:09.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.114+0000 7f56d2a00640 1 -- 172.21.15.28:0/3030668241 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56cc005500 con 0x7f56dc0769c0 2024-04-24T08:10:09.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.114+0000 7f56d2a00640 1 -- 172.21.15.28:0/3030668241 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc010430 con 0x7f56dc0769c0 2024-04-24T08:10:09.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.114+0000 7f56db200640 1 -- 172.21.15.28:0/3030668241 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f56dc1a6c60 con 0x7f56dc0769c0 2024-04-24T08:10:09.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.117+0000 7f56e3c03740 1 Processor -- start 2024-04-24T08:10:09.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.117+0000 7f56d2a00640 1 -- 172.21.15.28:0/3030668241 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f56cc0046c0 con 0x7f56dc0769c0 2024-04-24T08:10:09.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.117+0000 7f56e3c03740 1 -- start start 2024-04-24T08:10:09.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.117+0000 7f56e3c03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90260 0x562e1ae90680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.117+0000 7f56e3c03740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90c50 0x562e1ae95750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.117+0000 7f56e3c03740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae95dc0 0x562e1ae98250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.117+0000 7f56e3c03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562e1adf5b10 con 0x562e1ae90260 2024-04-24T08:10:09.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.117+0000 7f56e3c03740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562e1ad9c2b0 con 0x562e1ae90c50 2024-04-24T08:10:09.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.117+0000 7f56e3c03740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562e1adf1ad0 con 0x562e1ae95dc0 2024-04-24T08:10:09.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.118+0000 7f56da800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90260 0x562e1ae90680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.118+0000 7f56d9e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90c50 0x562e1ae95750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.118+0000 7f56d9400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae95dc0 0x562e1ae98250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.118+0000 7f56d9e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90c50 0x562e1ae95750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:52620/0 (socket says 172.21.15.28:52620) 2024-04-24T08:10:09.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.118+0000 7f56d9e00640 1 -- 172.21.15.28:0/1130912317 learned_addr learned my addr 172.21.15.28:0/1130912317 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:09.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.118+0000 7f56d9e00640 1 -- 172.21.15.28:0/1130912317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae95dc0 msgr2=0x562e1ae98250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.118+0000 7f56d9e00640 1 --2- 172.21.15.28:0/1130912317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae95dc0 0x562e1ae98250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.118+0000 7f56d9e00640 1 -- 172.21.15.28:0/1130912317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90260 msgr2=0x562e1ae90680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.118+0000 7f56d9e00640 1 --2- 172.21.15.28:0/1130912317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90260 0x562e1ae90680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.118+0000 7f56d9e00640 1 -- 172.21.15.28:0/1130912317 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e1ab082d0 con 0x562e1ae90c50 2024-04-24T08:10:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.119+0000 7f56d9e00640 1 --2- 172.21.15.28:0/1130912317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90c50 0x562e1ae95750 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f56cc0d6a90 tx=0x7f56cc0d6b80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d7bb0eb6149cd062 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.119+0000 7f56d2a00640 1 --2- 172.21.15.28:0/3030668241 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f56c00a3bb0 0x7f56c00a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.119+0000 7f56c7400640 1 -- 172.21.15.28:0/1130912317 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc0df070 con 0x562e1ae90c50 2024-04-24T08:10:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.119+0000 7f56c7400640 1 -- 172.21.15.28:0/1130912317 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56cc09d130 con 0x562e1ae90c50 2024-04-24T08:10:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.119+0000 7f56c7400640 1 -- 172.21.15.28:0/1130912317 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc0b61b0 con 0x562e1ae90c50 2024-04-24T08:10:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.119+0000 7f56da800640 1 --2- 172.21.15.28:0/3030668241 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f56c00a3bb0 0x7f56c00a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.119+0000 7f56d2a00640 1 -- 172.21.15.28:0/3030668241 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f56cc0d07f0 con 0x7f56dc0769c0 2024-04-24T08:10:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.120+0000 7f56e3c03740 1 -- 172.21.15.28:0/1130912317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90c50 msgr2=0x562e1ae95750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.120+0000 7f56e3c03740 1 --2- 172.21.15.28:0/1130912317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90c50 0x562e1ae95750 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f56cc0d6a90 tx=0x7f56cc0d6b80 comp rx=0 tx=0).stop 2024-04-24T08:10:09.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.120+0000 7f56da800640 1 --2- 172.21.15.28:0/3030668241 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f56c00a3bb0 0x7f56c00a6070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f56d402abf0 tx=0x7f56d4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.120+0000 7f56e3c03740 1 -- 172.21.15.28:0/1130912317 shutdown_connections 2024-04-24T08:10:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.120+0000 7f56e3c03740 1 --2- 172.21.15.28:0/1130912317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae95dc0 0x562e1ae98250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.120+0000 7f56e3c03740 1 --2- 172.21.15.28:0/1130912317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90c50 0x562e1ae95750 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.120+0000 7f56e3c03740 1 --2- 172.21.15.28:0/1130912317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90260 0x562e1ae90680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.120+0000 7f56e3c03740 1 -- 172.21.15.28:0/1130912317 >> 172.21.15.28:0/1130912317 conn(0x562e1ae8b8d0 msgr2=0x562e1ae8bcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.120+0000 7f56e3c03740 1 -- 172.21.15.28:0/1130912317 shutdown_connections 2024-04-24T08:10:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.120+0000 7f56e3c03740 1 -- 172.21.15.28:0/1130912317 wait complete. 2024-04-24T08:10:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.120+0000 7f56e3c03740 1 -- start start 2024-04-24T08:10:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.121+0000 7f56e3c03740 1 Processor -- start 2024-04-24T08:10:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.121+0000 7f56e3c03740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae90260 0x562e1ae9a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.121+0000 7f56e3c03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90c50 0x562e1ae9ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.121+0000 7f56e3c03740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae95dc0 0x562e1aea1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.121+0000 7f56e3c03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562e1ae98d70 con 0x562e1ae90c50 2024-04-24T08:10:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.121+0000 7f56e3c03740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562e1adaf590 con 0x562e1ae95dc0 2024-04-24T08:10:09.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.121+0000 7f56e3c03740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562e1ae98ef0 con 0x562e1ae90260 2024-04-24T08:10:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.121+0000 7f56e3c03740 10 client.0 _ll_register_callbacks cb 0x562e1ad923a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:10:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.121+0000 7f56d9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90c50 0x562e1ae9ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.121+0000 7f56da800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae95dc0 0x562e1aea1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.121+0000 7f56d9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90c50 0x562e1ae9ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50676/0 (socket says 172.21.15.28:50676) 2024-04-24T08:10:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.121+0000 7f56d9400640 1 -- 172.21.15.28:0/273771137 learned_addr learned my addr 172.21.15.28:0/273771137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.121+0000 7f56d9e00640 1 --2- 172.21.15.28:0/273771137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae90260 0x562e1ae9a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.122+0000 7f56da800640 1 -- 172.21.15.28:0/273771137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae90260 msgr2=0x562e1ae9a640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.122+0000 7f56da800640 1 --2- 172.21.15.28:0/273771137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae90260 0x562e1ae9a640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.122+0000 7f56da800640 1 -- 172.21.15.28:0/273771137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90c50 msgr2=0x562e1ae9ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.122+0000 7f56da800640 1 --2- 172.21.15.28:0/273771137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90c50 0x562e1ae9ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.122+0000 7f56da800640 1 -- 172.21.15.28:0/273771137 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e1abd26b0 con 0x562e1ae95dc0 2024-04-24T08:10:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.122+0000 7f56d9e00640 1 --2- 172.21.15.28:0/273771137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae90260 0x562e1ae9a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:09.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.122+0000 7f56d9400640 1 --2- 172.21.15.28:0/273771137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90c50 0x562e1ae9ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.122+0000 7f56da800640 1 --2- 172.21.15.28:0/273771137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae95dc0 0x562e1aea1b50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f56d4034040 tx=0x7f56d402a6d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d51815bcec22a61 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.122+0000 7f56bbe00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:10:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.122+0000 7f56bbe00640 1 -- 172.21.15.28:0/273771137 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d404f020 con 0x562e1ae95dc0 2024-04-24T08:10:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.122+0000 7f56bbe00640 1 -- 172.21.15.28:0/273771137 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f56d4003b70 con 0x562e1ae95dc0 2024-04-24T08:10:09.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.122+0000 7f56e3c03740 1 -- 172.21.15.28:0/273771137 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56cc09d130 con 0x562e1ae95dc0 2024-04-24T08:10:09.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.123+0000 7f56bbe00640 1 -- 172.21.15.28:0/273771137 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f56d405a050 con 0x562e1ae95dc0 2024-04-24T08:10:09.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.123+0000 7f56e3c03740 10 client.25154 fetch_fsmap learned FSMap version 23 2024-04-24T08:10:09.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.123+0000 7f56e3c03740 1 -- 172.21.15.28:0/273771137 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562e1aea4f30 con 0x562e1ae95dc0 2024-04-24T08:10:09.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.123+0000 7f56bbe00640 1 -- 172.21.15.28:0/273771137 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56cc09d130 con 0x562e1ae95dc0 2024-04-24T08:10:09.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.123+0000 7f56bbe00640 1 -- 172.21.15.28:0/273771137 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f56d404b640 con 0x562e1ae95dc0 2024-04-24T08:10:09.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.123+0000 7f56e3c03740 10 client.25154 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:10:09.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.123+0000 7f56e3c03740 10 client.25154 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T08:10:09.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.123+0000 7f56e3c03740 20 client.25154 populate_metadata read hostname 'smithi028' 2024-04-24T08:10:09.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.123+0000 7f56e3c03740 1 --2- 172.21.15.28:0/273771137 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x562e1aea75a0 0x562e1aea8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.124+0000 7f56e3c03740 4 client.25154 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:10:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.124+0000 7f56e3c03740 1 -- 172.21.15.28:0/273771137 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562e1ae03530 con 0x562e1aea75a0 2024-04-24T08:10:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.124+0000 7f56d9e00640 1 --2- 172.21.15.28:0/273771137 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x562e1aea75a0 0x562e1aea8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.124+0000 7f56d9e00640 1 --2- 172.21.15.28:0/273771137 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x562e1aea75a0 0x562e1aea8a10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f5eeb0d53b802c9e server_cookie=19d8b0215ac46a59 in_seq=0 out_seq=0 2024-04-24T08:10:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.124+0000 7f56bbe00640 10 client.25154 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:10:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.125+0000 7f56bbe00640 1 -- 172.21.15.28:0/273771137 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562e1ae03530 con 0x562e1aea75a0 2024-04-24T08:10:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.125+0000 7f56bbe00640 10 client.25154 handle_command_reply: tid=0 2024-04-24T08:10:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.125+0000 7f56e3c03740 1 client.25154 shutdown 2024-04-24T08:10:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.125+0000 7f56e3c03740 1 -- 172.21.15.28:0/273771137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae95dc0 msgr2=0x562e1aea1b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.125+0000 7f56e3c03740 1 --2- 172.21.15.28:0/273771137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae95dc0 0x562e1aea1b50 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f56d4034040 tx=0x7f56d402a6d0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.126+0000 7f56e3c03740 1 -- 172.21.15.28:0/273771137 shutdown_connections 2024-04-24T08:10:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.126+0000 7f56e3c03740 1 --2- 172.21.15.28:0/273771137 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x562e1aea75a0 0x562e1aea8a10 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.126+0000 7f56e3c03740 1 --2- 172.21.15.28:0/273771137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90c50 0x562e1ae9ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.126+0000 7f56e3c03740 1 --2- 172.21.15.28:0/273771137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae95dc0 0x562e1aea1b50 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.126+0000 7f56e3c03740 1 --2- 172.21.15.28:0/273771137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae90260 0x562e1ae9a640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.126+0000 7f56bbe00640 0 client.25154 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:10:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.126+0000 7f56e3c03740 1 -- 172.21.15.28:0/273771137 >> 172.21.15.28:0/273771137 conn(0x562e1ae8b8d0 msgr2=0x562e1ae8a970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.126+0000 7f56e3c03740 1 -- 172.21.15.28:0/273771137 shutdown_connections 2024-04-24T08:10:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.126+0000 7f56e3c03740 1 -- 172.21.15.28:0/273771137 wait complete. 2024-04-24T08:10:09.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.126+0000 7f56e3c03740 20 client.25154 trim_cache size 0 max 16384 2024-04-24T08:10:09.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.146+0000 7f56e3c03740 1 Processor -- start 2024-04-24T08:10:09.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.146+0000 7f56e3c03740 1 -- start start 2024-04-24T08:10:09.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.146+0000 7f56e3c03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90260 0x562e1ae95680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.146+0000 7f56e3c03740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90c50 0x562e1aea1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.146+0000 7f56e3c03740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae95dc0 0x562e1ae981b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.146+0000 7f56e3c03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562e1aea6d50 con 0x562e1ae90260 2024-04-24T08:10:09.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.146+0000 7f56e3c03740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562e1aea7420 con 0x562e1ae90c50 2024-04-24T08:10:09.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.146+0000 7f56e3c03740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562e1aea7af0 con 0x562e1ae95dc0 2024-04-24T08:10:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.147+0000 7f56d9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90260 0x562e1ae95680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.147+0000 7f56d9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90260 0x562e1ae95680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50690/0 (socket says 172.21.15.28:50690) 2024-04-24T08:10:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.147+0000 7f56d9e00640 1 -- 172.21.15.28:0/912638172 learned_addr learned my addr 172.21.15.28:0/912638172 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.147+0000 7f56da800640 1 --2- 172.21.15.28:0/912638172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae95dc0 0x562e1ae981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.147+0000 7f56d9400640 1 --2- 172.21.15.28:0/912638172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90c50 0x562e1aea1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.147+0000 7f56d9e00640 1 -- 172.21.15.28:0/912638172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae95dc0 msgr2=0x562e1ae981b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.147+0000 7f56d9e00640 1 --2- 172.21.15.28:0/912638172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae95dc0 0x562e1ae981b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.147+0000 7f56d9e00640 1 -- 172.21.15.28:0/912638172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90c50 msgr2=0x562e1aea1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.147+0000 7f56d9e00640 1 --2- 172.21.15.28:0/912638172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90c50 0x562e1aea1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.147+0000 7f56d9e00640 1 -- 172.21.15.28:0/912638172 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562e1aea7200 con 0x562e1ae90260 2024-04-24T08:10:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.147+0000 7f56d9400640 1 --2- 172.21.15.28:0/912638172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90c50 0x562e1aea1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:10:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.147+0000 7f56d9e00640 1 --2- 172.21.15.28:0/912638172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90260 0x562e1ae95680 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f56cc099400 tx=0x7f56cc005880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34100b51c26ef468 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.148+0000 7f56c6a00640 1 -- 172.21.15.28:0/912638172 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc0b9850 con 0x562e1ae90260 2024-04-24T08:10:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.148+0000 7f56c6a00640 1 -- 172.21.15.28:0/912638172 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56cc0bbe50 con 0x562e1ae90260 2024-04-24T08:10:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.148+0000 7f56c6a00640 1 -- 172.21.15.28:0/912638172 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56cc0b9bd0 con 0x562e1ae90260 2024-04-24T08:10:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.148+0000 7f56e3c03740 1 -- 172.21.15.28:0/912638172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90260 msgr2=0x562e1ae95680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.148+0000 7f56e3c03740 1 --2- 172.21.15.28:0/912638172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90260 0x562e1ae95680 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f56cc099400 tx=0x7f56cc005880 comp rx=0 tx=0).stop 2024-04-24T08:10:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.148+0000 7f56e3c03740 1 -- 172.21.15.28:0/912638172 shutdown_connections 2024-04-24T08:10:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.148+0000 7f56e3c03740 1 --2- 172.21.15.28:0/912638172 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae95dc0 0x562e1ae981b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.148+0000 7f56e3c03740 1 --2- 172.21.15.28:0/912638172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90c50 0x562e1aea1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.148+0000 7f56e3c03740 1 --2- 172.21.15.28:0/912638172 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae90260 0x562e1ae95680 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.148+0000 7f56e3c03740 1 -- 172.21.15.28:0/912638172 >> 172.21.15.28:0/912638172 conn(0x562e1ae8b8d0 msgr2=0x562e1ae8bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:09.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.148+0000 7f56e3c03740 1 -- 172.21.15.28:0/912638172 shutdown_connections 2024-04-24T08:10:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.149+0000 7f56e3c03740 1 -- 172.21.15.28:0/912638172 wait complete. 2024-04-24T08:10:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.149+0000 7f56e3c03740 1 -- start start 2024-04-24T08:10:09.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.149+0000 7f56e3c03740 1 Processor -- start 2024-04-24T08:10:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.149+0000 7f56e3c03740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90260 0x562e1ae94590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.149+0000 7f56e3c03740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae90c50 0x562e1ae94ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.149+0000 7f56e3c03740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae95dc0 0x562e1af0b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.149+0000 7f56e3c03740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562e1aea8890 con 0x562e1ae95dc0 2024-04-24T08:10:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.149+0000 7f56e3c03740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562e1aea81c0 con 0x562e1ae90260 2024-04-24T08:10:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.149+0000 7f56e3c03740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562e1aea8f60 con 0x562e1ae90c50 2024-04-24T08:10:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.149+0000 7f56e3c03740 10 client.0 _ll_register_callbacks cb 0x7f56c8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:10:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.150+0000 7f56da800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae95dc0 0x562e1af0b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.150+0000 7f56da800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae95dc0 0x562e1af0b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50692/0 (socket says 172.21.15.28:50692) 2024-04-24T08:10:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.150+0000 7f56da800640 1 -- 172.21.15.28:0/2531189507 learned_addr learned my addr 172.21.15.28:0/2531189507 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.150+0000 7f56d9e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90260 0x562e1ae94590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.150+0000 7f56d9400640 1 --2- 172.21.15.28:0/2531189507 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae90c50 0x562e1ae94ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.150+0000 7f56da800640 1 -- 172.21.15.28:0/2531189507 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae90c50 msgr2=0x562e1ae94ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.150+0000 7f56da800640 1 --2- 172.21.15.28:0/2531189507 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae90c50 0x562e1ae94ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.150+0000 7f56da800640 1 -- 172.21.15.28:0/2531189507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90260 msgr2=0x562e1ae94590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.150+0000 7f56da800640 1 --2- 172.21.15.28:0/2531189507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90260 0x562e1ae94590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.150+0000 7f56da800640 1 -- 172.21.15.28:0/2531189507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562e1af0ba80 con 0x562e1ae95dc0 2024-04-24T08:10:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.150+0000 7f56d9e00640 1 --2- 172.21.15.28:0/2531189507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90260 0x562e1ae94590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.150+0000 7f56da800640 1 --2- 172.21.15.28:0/2531189507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae95dc0 0x562e1af0b540 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f56d404bc70 tx=0x7f56d404b6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb24cab16c5f3f8e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.150+0000 7f56bbe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:10:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.151+0000 7f56bbe00640 1 -- 172.21.15.28:0/2531189507 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f56d4062020 con 0x562e1ae95dc0 2024-04-24T08:10:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.151+0000 7f56bbe00640 1 -- 172.21.15.28:0/2531189507 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f56d4003d80 con 0x562e1ae95dc0 2024-04-24T08:10:09.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.151+0000 7f56e3c03740 1 -- 172.21.15.28:0/2531189507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f56cc0bbe50 con 0x562e1ae95dc0 2024-04-24T08:10:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.151+0000 7f56bbe00640 1 -- 172.21.15.28:0/2531189507 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f56d404f040 con 0x562e1ae95dc0 2024-04-24T08:10:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.151+0000 7f56e3c03740 10 client.15066 fetch_fsmap learned FSMap version 23 2024-04-24T08:10:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.151+0000 7f56e3c03740 1 -- 172.21.15.28:0/2531189507 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x562e1af0cc30 con 0x562e1ae95dc0 2024-04-24T08:10:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.151+0000 7f56bbe00640 1 -- 172.21.15.28:0/2531189507 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f56cc0bbe50 con 0x562e1ae95dc0 2024-04-24T08:10:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.151+0000 7f56bbe00640 1 -- 172.21.15.28:0/2531189507 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f56d4031490 con 0x562e1ae95dc0 2024-04-24T08:10:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.151+0000 7f56e3c03740 10 client.15066 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:10:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.152+0000 7f56e3c03740 10 client.15066 resolve_mds: resolved name 'b' to daemon mds.b 2024-04-24T08:10:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.152+0000 7f56e3c03740 20 client.15066 populate_metadata read hostname 'smithi028' 2024-04-24T08:10:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.152+0000 7f56e3c03740 1 --2- 172.21.15.28:0/2531189507 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x562e1af11850 0x562e1af10560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.152+0000 7f56e3c03740 4 client.15066 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:10:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.152+0000 7f56e3c03740 1 -- 172.21.15.28:0/2531189507 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "status"}) -- 0x562e1adfd3a0 con 0x562e1af11850 2024-04-24T08:10:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.152+0000 7f56d9e00640 1 --2- 172.21.15.28:0/2531189507 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x562e1af11850 0x562e1af10560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.152+0000 7f56d9e00640 1 --2- 172.21.15.28:0/2531189507 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x562e1af11850 0x562e1af10560 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=1ee817b9b33b6b1f server_cookie=3ad9bf9cefff7a0c in_seq=0 out_seq=0 2024-04-24T08:10:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.152+0000 7f56bbe00640 10 client.15066 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:10:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.153+0000 7f56bbe00640 1 -- 172.21.15.28:0/2531189507 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x562e1adfd3a0 con 0x562e1af11850 2024-04-24T08:10:09.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.153+0000 7f56bbe00640 10 client.15066 handle_command_reply: tid=0 2024-04-24T08:10:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.153+0000 7f56e3c03740 1 client.15066 shutdown 2024-04-24T08:10:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.153+0000 7f56e3c03740 1 -- 172.21.15.28:0/2531189507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae95dc0 msgr2=0x562e1af0b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.153+0000 7f56e3c03740 1 --2- 172.21.15.28:0/2531189507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae95dc0 0x562e1af0b540 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f56d404bc70 tx=0x7f56d404b6b0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.153+0000 7f56e3c03740 1 -- 172.21.15.28:0/2531189507 shutdown_connections 2024-04-24T08:10:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.153+0000 7f56e3c03740 1 --2- 172.21.15.28:0/2531189507 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x562e1af11850 0x562e1af10560 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.153+0000 7f56e3c03740 1 --2- 172.21.15.28:0/2531189507 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562e1ae95dc0 0x562e1af0b540 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.153+0000 7f56e3c03740 1 --2- 172.21.15.28:0/2531189507 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562e1ae90c50 0x562e1ae94ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.153+0000 7f56e3c03740 1 --2- 172.21.15.28:0/2531189507 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562e1ae90260 0x562e1ae94590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.153+0000 7f56bbe00640 0 client.15066 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:10:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.153+0000 7f56e3c03740 1 -- 172.21.15.28:0/2531189507 >> 172.21.15.28:0/2531189507 conn(0x562e1ae8b8d0 msgr2=0x562e1ae8bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.153+0000 7f56e3c03740 1 -- 172.21.15.28:0/2531189507 shutdown_connections 2024-04-24T08:10:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.153+0000 7f56e3c03740 1 -- 172.21.15.28:0/2531189507 wait complete. 2024-04-24T08:10:09.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.153+0000 7f56e3c03740 20 client.15066 trim_cache size 0 max 16384 2024-04-24T08:10:09.159 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:10:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "473aaf58-0210-11ef-bc93-c7b262605968", 2024-04-24T08:10:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 1, 2024-04-24T08:10:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24572, 2024-04-24T08:10:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-24T08:10:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-24T08:10:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-24T08:10:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 134.048279718, 2024-04-24T08:10:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 23, 2024-04-24T08:10:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 90, 2024-04-24T08:10:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 86, 2024-04-24T08:10:09.159 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 143.54169133299999 2024-04-24T08:10:09.159 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:10:09.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.154+0000 7f56db200640 1 -- 172.21.15.28:0/3030668241 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f56c00a3bb0 msgr2=0x7f56c00a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.154+0000 7f56db200640 1 --2- 172.21.15.28:0/3030668241 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f56c00a3bb0 0x7f56c00a6070 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f56d402abf0 tx=0x7f56d4005fd0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.154+0000 7f56db200640 1 -- 172.21.15.28:0/3030668241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56dc0769c0 msgr2=0x7f56dc19e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.154+0000 7f56db200640 1 --2- 172.21.15.28:0/3030668241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56dc0769c0 0x7f56dc19e800 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f56cc00e9c0 tx=0x7f56cc00ee80 comp rx=0 tx=0).stop 2024-04-24T08:10:09.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.154+0000 7f56db200640 1 -- 172.21.15.28:0/3030668241 shutdown_connections 2024-04-24T08:10:09.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.154+0000 7f56db200640 1 --2- 172.21.15.28:0/3030668241 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f56c00a3bb0 0x7f56c00a6070 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.154+0000 7f56db200640 1 --2- 172.21.15.28:0/3030668241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56dc0785a0 0x7f56dc1a5db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.154+0000 7f56db200640 1 --2- 172.21.15.28:0/3030668241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f56dc077c00 0x7f56dc19ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.154+0000 7f56db200640 1 --2- 172.21.15.28:0/3030668241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f56dc0769c0 0x7f56dc19e800 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.154+0000 7f56db200640 1 -- 172.21.15.28:0/3030668241 >> 172.21.15.28:0/3030668241 conn(0x7f56dc0ff6b0 msgr2=0x7f56dc1011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:09.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.154+0000 7f56db200640 1 -- 172.21.15.28:0/3030668241 shutdown_connections 2024-04-24T08:10:09.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.154+0000 7f56db200640 1 -- 172.21.15.28:0/3030668241 wait complete. 2024-04-24T08:10:09.173 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-04-24T08:10:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.380+0000 7f966e400640 1 -- 172.21.15.28:0/2213059840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9668109850 msgr2=0x7f9668109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.380+0000 7f966e400640 1 --2- 172.21.15.28:0/2213059840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9668109850 0x7f9668109cd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f9654009ca0 tx=0x7f965402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.381+0000 7f966e400640 1 -- 172.21.15.28:0/2213059840 shutdown_connections 2024-04-24T08:10:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.381+0000 7f966e400640 1 --2- 172.21.15.28:0/2213059840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9668109850 0x7f9668109cd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.381+0000 7f966e400640 1 --2- 172.21.15.28:0/2213059840 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f966810a210 0x7f9668110aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.381+0000 7f966e400640 1 --2- 172.21.15.28:0/2213059840 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9668108660 0x7f9668108a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.381+0000 7f966e400640 1 -- 172.21.15.28:0/2213059840 >> 172.21.15.28:0/2213059840 conn(0x7f9668077600 msgr2=0x7f9668079a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.381+0000 7f966e400640 1 -- 172.21.15.28:0/2213059840 shutdown_connections 2024-04-24T08:10:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.381+0000 7f966e400640 1 -- 172.21.15.28:0/2213059840 wait complete. 2024-04-24T08:10:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.382+0000 7f966e400640 1 Processor -- start 2024-04-24T08:10:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.382+0000 7f966e400640 1 -- start start 2024-04-24T08:10:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.382+0000 7f966e400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9668108660 0x7f966819e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.382+0000 7f966e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9668109850 0x7f966819eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.382+0000 7f966e400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f966810a210 0x7f96681a5f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.382+0000 7f966e400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96681142d0 con 0x7f9668109850 2024-04-24T08:10:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.382+0000 7f966e400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9668114150 con 0x7f9668108660 2024-04-24T08:10:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.382+0000 7f966e400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9668114450 con 0x7f966810a210 2024-04-24T08:10:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.382+0000 7f9666a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9668109850 0x7f966819eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.383+0000 7f9666a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9668109850 0x7f966819eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50698/0 (socket says 172.21.15.28:50698) 2024-04-24T08:10:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.383+0000 7f9666a00640 1 -- 172.21.15.28:0/4142234059 learned_addr learned my addr 172.21.15.28:0/4142234059 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.383+0000 7f9667400640 1 --2- 172.21.15.28:0/4142234059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9668108660 0x7f966819e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.383+0000 7f9667e00640 1 --2- 172.21.15.28:0/4142234059 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f966810a210 0x7f96681a5f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.383+0000 7f9666a00640 1 -- 172.21.15.28:0/4142234059 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f966810a210 msgr2=0x7f96681a5f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.383+0000 7f9666a00640 1 --2- 172.21.15.28:0/4142234059 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f966810a210 0x7f96681a5f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.383+0000 7f9666a00640 1 -- 172.21.15.28:0/4142234059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9668108660 msgr2=0x7f966819e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.383+0000 7f9666a00640 1 --2- 172.21.15.28:0/4142234059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9668108660 0x7f966819e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.383+0000 7f9666a00640 1 -- 172.21.15.28:0/4142234059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96681a65b0 con 0x7f9668109850 2024-04-24T08:10:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.383+0000 7f9666a00640 1 --2- 172.21.15.28:0/4142234059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9668109850 0x7f966819eed0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9654031a40 tx=0x7f9654031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.384+0000 7f965fe00640 1 -- 172.21.15.28:0/4142234059 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965402fa20 con 0x7f9668109850 2024-04-24T08:10:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.384+0000 7f966e400640 1 -- 172.21.15.28:0/4142234059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96681a6840 con 0x7f9668109850 2024-04-24T08:10:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.384+0000 7f965fe00640 1 -- 172.21.15.28:0/4142234059 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f965402fbc0 con 0x7f9668109850 2024-04-24T08:10:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.384+0000 7f965fe00640 1 -- 172.21.15.28:0/4142234059 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965403a780 con 0x7f9668109850 2024-04-24T08:10:09.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.384+0000 7f966e400640 1 -- 172.21.15.28:0/4142234059 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96681a6d90 con 0x7f9668109850 2024-04-24T08:10:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.387+0000 7f967117f740 1 Processor -- start 2024-04-24T08:10:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.387+0000 7f967117f740 1 -- start start 2024-04-24T08:10:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.387+0000 7f965fe00640 1 -- 172.21.15.28:0/4142234059 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f965403a920 con 0x7f9668109850 2024-04-24T08:10:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.387+0000 7f967117f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48090 0x557d04b484b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.387+0000 7f967117f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48a80 0x557d04b4d580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.387+0000 7f967117f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b4dbf0 0x557d04b50080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.387+0000 7f967117f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d04aadb10 con 0x557d04b48a80 2024-04-24T08:10:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.387+0000 7f967117f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557d04a542b0 con 0x557d04b48090 2024-04-24T08:10:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.387+0000 7f967117f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557d04aa9ad0 con 0x557d04b4dbf0 2024-04-24T08:10:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.388+0000 7f9667400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48a80 0x557d04b4d580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.388+0000 7f9667e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48090 0x557d04b484b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.388+0000 7f9667400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48a80 0x557d04b4d580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50700/0 (socket says 172.21.15.28:50700) 2024-04-24T08:10:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.388+0000 7f9667400640 1 -- 172.21.15.28:0/45631874 learned_addr learned my addr 172.21.15.28:0/45631874 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.388+0000 7f9666a00640 1 --2- 172.21.15.28:0/45631874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b4dbf0 0x557d04b50080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.388+0000 7f9667400640 1 -- 172.21.15.28:0/45631874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b4dbf0 msgr2=0x557d04b50080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.388+0000 7f9667400640 1 --2- 172.21.15.28:0/45631874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b4dbf0 0x557d04b50080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.388+0000 7f9667400640 1 -- 172.21.15.28:0/45631874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48090 msgr2=0x557d04b484b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.388+0000 7f9667400640 1 --2- 172.21.15.28:0/45631874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48090 0x557d04b484b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.388+0000 7f9667400640 1 -- 172.21.15.28:0/45631874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d047c02d0 con 0x557d04b48a80 2024-04-24T08:10:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.388+0000 7f9666a00640 1 --2- 172.21.15.28:0/45631874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b4dbf0 0x557d04b50080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:10:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.388+0000 7f9667e00640 1 --2- 172.21.15.28:0/45631874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48090 0x557d04b484b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.388+0000 7f9667400640 1 --2- 172.21.15.28:0/45631874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48a80 0x557d04b4d580 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f965800b7d0 tx=0x7f9658004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a608caaf7e451cbb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:09.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.389+0000 7f965cc00640 1 -- 172.21.15.28:0/45631874 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9658003dd0 con 0x557d04b48a80 2024-04-24T08:10:09.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.389+0000 7f965cc00640 1 -- 172.21.15.28:0/45631874 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9658015070 con 0x557d04b48a80 2024-04-24T08:10:09.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.389+0000 7f965cc00640 1 -- 172.21.15.28:0/45631874 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96580117c0 con 0x557d04b48a80 2024-04-24T08:10:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.389+0000 7f965fe00640 1 --2- 172.21.15.28:0/4142234059 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f964c0a3bb0 0x7f964c0a6070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.389+0000 7f965fe00640 1 -- 172.21.15.28:0/4142234059 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f96540f19b0 con 0x7f9668109850 2024-04-24T08:10:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.389+0000 7f9667e00640 1 --2- 172.21.15.28:0/4142234059 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f964c0a3bb0 0x7f964c0a6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.390+0000 7f967117f740 1 -- 172.21.15.28:0/45631874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48a80 msgr2=0x557d04b4d580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.390+0000 7f967117f740 1 --2- 172.21.15.28:0/45631874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48a80 0x557d04b4d580 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f965800b7d0 tx=0x7f9658004610 comp rx=0 tx=0).stop 2024-04-24T08:10:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.390+0000 7f967117f740 1 -- 172.21.15.28:0/45631874 shutdown_connections 2024-04-24T08:10:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.390+0000 7f967117f740 1 --2- 172.21.15.28:0/45631874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48a80 0x557d04b4d580 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.390+0000 7f967117f740 1 --2- 172.21.15.28:0/45631874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b4dbf0 0x557d04b50080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.390+0000 7f967117f740 1 --2- 172.21.15.28:0/45631874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48090 0x557d04b484b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.390+0000 7f967117f740 1 -- 172.21.15.28:0/45631874 >> 172.21.15.28:0/45631874 conn(0x557d04b43700 msgr2=0x557d04b43b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.390+0000 7f967117f740 1 -- 172.21.15.28:0/45631874 shutdown_connections 2024-04-24T08:10:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.390+0000 7f9667e00640 1 --2- 172.21.15.28:0/4142234059 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f964c0a3bb0 0x7f964c0a6070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x557d04b492e0 tx=0x7f9660005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.390+0000 7f967117f740 1 -- 172.21.15.28:0/45631874 wait complete. 2024-04-24T08:10:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.390+0000 7f967117f740 1 -- start start 2024-04-24T08:10:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.391+0000 7f967117f740 1 Processor -- start 2024-04-24T08:10:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.391+0000 7f967117f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48090 0x557d04b52540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.391+0000 7f967117f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48a80 0x557d04b52a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.391+0000 7f967117f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b4dbf0 0x557d04b59ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.391+0000 7f967117f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d04b50ba0 con 0x557d04b48090 2024-04-24T08:10:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.391+0000 7f967117f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557d04a67590 con 0x557d04b48a80 2024-04-24T08:10:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.391+0000 7f967117f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557d04b50d20 con 0x557d04b4dbf0 2024-04-24T08:10:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.391+0000 7f967117f740 10 client.0 _ll_register_callbacks cb 0x557d04a4a3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:10:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.391+0000 7f9667400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48090 0x557d04b52540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.391+0000 7f9667400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48090 0x557d04b52540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50714/0 (socket says 172.21.15.28:50714) 2024-04-24T08:10:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.391+0000 7f9667400640 1 -- 172.21.15.28:0/2791900420 learned_addr learned my addr 172.21.15.28:0/2791900420 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.391+0000 7f9666a00640 1 --2- 172.21.15.28:0/2791900420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48a80 0x557d04b52a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.392+0000 7f9667e00640 1 --2- 172.21.15.28:0/2791900420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b4dbf0 0x557d04b59ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.392+0000 7f9666a00640 1 -- 172.21.15.28:0/2791900420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b4dbf0 msgr2=0x557d04b59ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.392+0000 7f9666a00640 1 --2- 172.21.15.28:0/2791900420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b4dbf0 0x557d04b59ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.392+0000 7f9666a00640 1 -- 172.21.15.28:0/2791900420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48090 msgr2=0x557d04b52540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.392+0000 7f9666a00640 1 --2- 172.21.15.28:0/2791900420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48090 0x557d04b52540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.392+0000 7f9666a00640 1 -- 172.21.15.28:0/2791900420 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d0488a6b0 con 0x557d04b48a80 2024-04-24T08:10:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.392+0000 7f9667400640 1 --2- 172.21.15.28:0/2791900420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48090 0x557d04b52540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.392+0000 7f9666a00640 1 --2- 172.21.15.28:0/2791900420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48a80 0x557d04b52a80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f96540bd8f0 tx=0x7f965402a8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31f8fd48db995c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.392+0000 7f9651600640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:10:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.392+0000 7f9651600640 1 -- 172.21.15.28:0/2791900420 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96540bdbf0 con 0x557d04b48a80 2024-04-24T08:10:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.392+0000 7f9651600640 1 -- 172.21.15.28:0/2791900420 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f96540de0b0 con 0x557d04b48a80 2024-04-24T08:10:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.392+0000 7f967117f740 1 -- 172.21.15.28:0/2791900420 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9658015070 con 0x557d04b48a80 2024-04-24T08:10:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.393+0000 7f9651600640 1 -- 172.21.15.28:0/2791900420 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f96540c1fa0 con 0x557d04b48a80 2024-04-24T08:10:09.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.393+0000 7f967117f740 10 client.25172 fetch_fsmap learned FSMap version 23 2024-04-24T08:10:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.393+0000 7f967117f740 1 -- 172.21.15.28:0/2791900420 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557d04b50ba0 con 0x557d04b48a80 2024-04-24T08:10:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.393+0000 7f9651600640 1 -- 172.21.15.28:0/2791900420 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9658015070 con 0x557d04b48a80 2024-04-24T08:10:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.393+0000 7f9651600640 1 -- 172.21.15.28:0/2791900420 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f96540c63d0 con 0x557d04b48a80 2024-04-24T08:10:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.393+0000 7f967117f740 10 client.25172 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:10:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.393+0000 7f967117f740 10 client.25172 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-24T08:10:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.393+0000 7f967117f740 20 client.25172 populate_metadata read hostname 'smithi028' 2024-04-24T08:10:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.393+0000 7f967117f740 1 --2- 172.21.15.28:0/2791900420 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557d04b5f470 0x557d04b608e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.393+0000 7f967117f740 4 client.25172 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:10:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.393+0000 7f967117f740 1 -- 172.21.15.28:0/2791900420 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d04abb530 con 0x557d04b5f470 2024-04-24T08:10:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.394+0000 7f9667400640 1 --2- 172.21.15.28:0/2791900420 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557d04b5f470 0x557d04b608e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.394+0000 7f9667400640 1 --2- 172.21.15.28:0/2791900420 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557d04b5f470 0x557d04b608e0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa0b07bfec2b501e server_cookie=6ae84eb0c76cc66c in_seq=0 out_seq=0 2024-04-24T08:10:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.394+0000 7f9651600640 10 client.25172 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:10:09.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.395+0000 7f9651600640 1 -- 172.21.15.28:0/2791900420 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557d04abb530 con 0x557d04b5f470 2024-04-24T08:10:09.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.395+0000 7f9651600640 10 client.25172 handle_command_reply: tid=0 2024-04-24T08:10:09.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.395+0000 7f967117f740 1 client.25172 shutdown 2024-04-24T08:10:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.395+0000 7f967117f740 1 -- 172.21.15.28:0/2791900420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48a80 msgr2=0x557d04b52a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.395+0000 7f967117f740 1 --2- 172.21.15.28:0/2791900420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48a80 0x557d04b52a80 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f96540bd8f0 tx=0x7f965402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.395+0000 7f967117f740 1 -- 172.21.15.28:0/2791900420 shutdown_connections 2024-04-24T08:10:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.395+0000 7f967117f740 1 --2- 172.21.15.28:0/2791900420 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557d04b5f470 0x557d04b608e0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.395+0000 7f967117f740 1 --2- 172.21.15.28:0/2791900420 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b4dbf0 0x557d04b59ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.395+0000 7f967117f740 1 --2- 172.21.15.28:0/2791900420 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48a80 0x557d04b52a80 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.395+0000 7f967117f740 1 --2- 172.21.15.28:0/2791900420 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48090 0x557d04b52540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.395+0000 7f9651600640 0 client.25172 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:10:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.395+0000 7f967117f740 1 -- 172.21.15.28:0/2791900420 >> 172.21.15.28:0/2791900420 conn(0x557d04b43700 msgr2=0x557d04b428c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.396+0000 7f967117f740 1 -- 172.21.15.28:0/2791900420 shutdown_connections 2024-04-24T08:10:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.396+0000 7f967117f740 1 -- 172.21.15.28:0/2791900420 wait complete. 2024-04-24T08:10:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.396+0000 7f967117f740 20 client.25172 trim_cache size 0 max 16384 2024-04-24T08:10:09.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.416+0000 7f967117f740 1 Processor -- start 2024-04-24T08:10:09.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.416+0000 7f967117f740 1 -- start start 2024-04-24T08:10:09.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.416+0000 7f967117f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b48090 0x557d04b5dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.416+0000 7f967117f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48a80 0x557d04b59a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.416+0000 7f967117f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b4dbf0 0x557d04b5e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.416+0000 7f967117f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d04b5ec20 con 0x557d04b48a80 2024-04-24T08:10:09.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.416+0000 7f967117f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557d04b5f2f0 con 0x557d04b4dbf0 2024-04-24T08:10:09.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.416+0000 7f967117f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557d04b5f9c0 con 0x557d04b48090 2024-04-24T08:10:09.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.416+0000 7f9667e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b4dbf0 0x557d04b5e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.416+0000 7f9667e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b4dbf0 0x557d04b5e4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:52718/0 (socket says 172.21.15.28:52718) 2024-04-24T08:10:09.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.416+0000 7f9667e00640 1 -- 172.21.15.28:0/1743915805 learned_addr learned my addr 172.21.15.28:0/1743915805 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:09.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.417+0000 7f9666a00640 1 --2- 172.21.15.28:0/1743915805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48a80 0x557d04b59a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.417+0000 7f9667400640 1 --2- 172.21.15.28:0/1743915805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b48090 0x557d04b5dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.417+0000 7f9667e00640 1 -- 172.21.15.28:0/1743915805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b48090 msgr2=0x557d04b5dfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:09.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.417+0000 7f9667e00640 1 --2- 172.21.15.28:0/1743915805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b48090 0x557d04b5dfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.417+0000 7f9667e00640 1 -- 172.21.15.28:0/1743915805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48a80 msgr2=0x557d04b59a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:09.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.417+0000 7f9667e00640 1 --2- 172.21.15.28:0/1743915805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48a80 0x557d04b59a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.417+0000 7f9667e00640 1 -- 172.21.15.28:0/1743915805 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d04b48e60 con 0x557d04b4dbf0 2024-04-24T08:10:09.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.417+0000 7f9666a00640 1 --2- 172.21.15.28:0/1743915805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48a80 0x557d04b59a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:09.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.417+0000 7f9667e00640 1 --2- 172.21.15.28:0/1743915805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b4dbf0 0x557d04b5e4f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9660004380 tx=0x7f966001b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81b465804e182796 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:09.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.418+0000 7f9653e00640 1 -- 172.21.15.28:0/1743915805 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966002e070 con 0x557d04b4dbf0 2024-04-24T08:10:09.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.418+0000 7f9653e00640 1 -- 172.21.15.28:0/1743915805 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f966001e950 con 0x557d04b4dbf0 2024-04-24T08:10:09.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.418+0000 7f9653e00640 1 -- 172.21.15.28:0/1743915805 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9660029880 con 0x557d04b4dbf0 2024-04-24T08:10:09.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.418+0000 7f967117f740 1 -- 172.21.15.28:0/1743915805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b4dbf0 msgr2=0x557d04b5e4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.418+0000 7f967117f740 1 --2- 172.21.15.28:0/1743915805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b4dbf0 0x557d04b5e4f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f9660004380 tx=0x7f966001b000 comp rx=0 tx=0).stop 2024-04-24T08:10:09.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.418+0000 7f967117f740 1 -- 172.21.15.28:0/1743915805 shutdown_connections 2024-04-24T08:10:09.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.418+0000 7f967117f740 1 --2- 172.21.15.28:0/1743915805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b48a80 0x557d04b59a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.418+0000 7f967117f740 1 --2- 172.21.15.28:0/1743915805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b4dbf0 0x557d04b5e4f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.418+0000 7f967117f740 1 --2- 172.21.15.28:0/1743915805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b48090 0x557d04b5dfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.418+0000 7f967117f740 1 -- 172.21.15.28:0/1743915805 >> 172.21.15.28:0/1743915805 conn(0x557d04b43700 msgr2=0x557d04b43ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:09.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.419+0000 7f967117f740 1 -- 172.21.15.28:0/1743915805 shutdown_connections 2024-04-24T08:10:09.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.419+0000 7f967117f740 1 -- 172.21.15.28:0/1743915805 wait complete. 2024-04-24T08:10:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.419+0000 7f967117f740 1 -- start start 2024-04-24T08:10:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.419+0000 7f967117f740 1 Processor -- start 2024-04-24T08:10:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.419+0000 7f967117f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b48090 0x557d04b55370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.419+0000 7f967117f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48a80 0x557d04b558b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f967117f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b4dbf0 0x557d04b58880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f967117f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557d04b60760 con 0x557d04b4dbf0 2024-04-24T08:10:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f967117f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557d04b60090 con 0x557d04b48a80 2024-04-24T08:10:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f967117f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557d04b60e30 con 0x557d04b48090 2024-04-24T08:10:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f967117f740 10 client.0 _ll_register_callbacks cb 0x7f96540bda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:10:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f9666a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48a80 0x557d04b558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f9667e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b4dbf0 0x557d04b58880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f9666a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48a80 0x557d04b558b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:52720/0 (socket says 172.21.15.28:52720) 2024-04-24T08:10:09.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f9666a00640 1 -- 172.21.15.28:0/1382409816 learned_addr learned my addr 172.21.15.28:0/1382409816 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f9667400640 1 --2- 172.21.15.28:0/1382409816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b48090 0x557d04b55370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f9666a00640 1 -- 172.21.15.28:0/1382409816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b48090 msgr2=0x557d04b55370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f9666a00640 1 --2- 172.21.15.28:0/1382409816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b48090 0x557d04b55370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f9666a00640 1 -- 172.21.15.28:0/1382409816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b4dbf0 msgr2=0x557d04b58880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f9666a00640 1 --2- 172.21.15.28:0/1382409816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b4dbf0 0x557d04b58880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f9666a00640 1 -- 172.21.15.28:0/1382409816 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d04b58e70 con 0x557d04b48a80 2024-04-24T08:10:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f9667400640 1 --2- 172.21.15.28:0/1382409816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b48090 0x557d04b55370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:10:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f9667e00640 1 --2- 172.21.15.28:0/1382409816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b4dbf0 0x557d04b58880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f9666a00640 1 --2- 172.21.15.28:0/1382409816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48a80 0x557d04b558b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f96540ba5c0 tx=0x7f96540bd470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f3f8e36ce25650e6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.420+0000 7f9651600640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:10:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.421+0000 7f9651600640 1 -- 172.21.15.28:0/1382409816 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96540be0d0 con 0x557d04b48a80 2024-04-24T08:10:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.421+0000 7f9651600640 1 -- 172.21.15.28:0/1382409816 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f96540de0b0 con 0x557d04b48a80 2024-04-24T08:10:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.421+0000 7f967117f740 1 -- 172.21.15.28:0/1382409816 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f966001e950 con 0x557d04b48a80 2024-04-24T08:10:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.421+0000 7f9651600640 1 -- 172.21.15.28:0/1382409816 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=23) ==== 24+0+0 (secure 0 0 0) 0x7f96540a3050 con 0x557d04b48a80 2024-04-24T08:10:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.421+0000 7f967117f740 10 client.25178 fetch_fsmap learned FSMap version 23 2024-04-24T08:10:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.421+0000 7f967117f740 1 -- 172.21.15.28:0/1382409816 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=23,monmap=4+}) -- 0x557d04bc5640 con 0x557d04b48a80 2024-04-24T08:10:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.421+0000 7f9651600640 1 -- 172.21.15.28:0/1382409816 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f966001e950 con 0x557d04b48a80 2024-04-24T08:10:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.421+0000 7f9651600640 1 -- 172.21.15.28:0/1382409816 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 23) ==== 8338+0+0 (secure 0 0 0) 0x7f96540bccb0 con 0x557d04b48a80 2024-04-24T08:10:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.421+0000 7f967117f740 10 client.25178 fetch_fsmap finished waiting for FSMap version 23 2024-04-24T08:10:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.422+0000 7f967117f740 10 client.25178 resolve_mds: resolved name 'g' to daemon mds.g 2024-04-24T08:10:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.422+0000 7f967117f740 20 client.25178 populate_metadata read hostname 'smithi028' 2024-04-24T08:10:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.422+0000 7f967117f740 1 --2- 172.21.15.28:0/1382409816 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557d04bc9fb0 0x557d04bc6ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.422+0000 7f967117f740 4 client.25178 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "status"}] 2024-04-24T08:10:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.422+0000 7f967117f740 1 -- 172.21.15.28:0/1382409816 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "status"}) -- 0x557d04ab53a0 con 0x557d04bc9fb0 2024-04-24T08:10:09.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.422+0000 7f9667400640 1 --2- 172.21.15.28:0/1382409816 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557d04bc9fb0 0x557d04bc6ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.422+0000 7f9667400640 1 --2- 172.21.15.28:0/1382409816 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557d04bc9fb0 0x557d04bc6ac0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c9d0ee1e405bb65 server_cookie=66879d3c88ab1a67 in_seq=0 out_seq=0 2024-04-24T08:10:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.422+0000 7f9651600640 10 client.25178 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:10:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.423+0000 7f9651600640 1 -- 172.21.15.28:0/1382409816 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x557d04ab53a0 con 0x557d04bc9fb0 2024-04-24T08:10:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.423+0000 7f9651600640 10 client.25178 handle_command_reply: tid=0 2024-04-24T08:10:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.423+0000 7f967117f740 1 client.25178 shutdown 2024-04-24T08:10:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.423+0000 7f967117f740 1 -- 172.21.15.28:0/1382409816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48a80 msgr2=0x557d04b558b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.423+0000 7f967117f740 1 --2- 172.21.15.28:0/1382409816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48a80 0x557d04b558b0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f96540ba5c0 tx=0x7f96540bd470 comp rx=0 tx=0).stop 2024-04-24T08:10:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.423+0000 7f967117f740 1 -- 172.21.15.28:0/1382409816 shutdown_connections 2024-04-24T08:10:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.423+0000 7f967117f740 1 --2- 172.21.15.28:0/1382409816 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557d04bc9fb0 0x557d04bc6ac0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.423+0000 7f967117f740 1 --2- 172.21.15.28:0/1382409816 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557d04b4dbf0 0x557d04b58880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.423+0000 7f967117f740 1 --2- 172.21.15.28:0/1382409816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557d04b48a80 0x557d04b558b0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.423+0000 7f9651600640 0 client.25178 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:10:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.423+0000 7f967117f740 1 --2- 172.21.15.28:0/1382409816 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557d04b48090 0x557d04b55370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.423+0000 7f967117f740 1 -- 172.21.15.28:0/1382409816 >> 172.21.15.28:0/1382409816 conn(0x557d04b43700 msgr2=0x557d04b5dfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:09.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.423+0000 7f967117f740 1 -- 172.21.15.28:0/1382409816 shutdown_connections 2024-04-24T08:10:09.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.423+0000 7f967117f740 1 -- 172.21.15.28:0/1382409816 wait complete. 2024-04-24T08:10:09.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.423+0000 7f967117f740 20 client.25178 trim_cache size 0 max 16384 2024-04-24T08:10:09.429 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:10:09.429 INFO:teuthology.orchestra.run.smithi028.stdout: "cluster_fsid": "473aaf58-0210-11ef-bc93-c7b262605968", 2024-04-24T08:10:09.429 INFO:teuthology.orchestra.run.smithi028.stdout: "whoami": 0, 2024-04-24T08:10:09.429 INFO:teuthology.orchestra.run.smithi028.stdout: "id": 24574, 2024-04-24T08:10:09.429 INFO:teuthology.orchestra.run.smithi028.stdout: "want_state": "up:active", 2024-04-24T08:10:09.429 INFO:teuthology.orchestra.run.smithi028.stdout: "state": "up:active", 2024-04-24T08:10:09.429 INFO:teuthology.orchestra.run.smithi028.stdout: "fs_name": "cephfs", 2024-04-24T08:10:09.430 INFO:teuthology.orchestra.run.smithi028.stdout: "rank_uptime": 137.34796536499999, 2024-04-24T08:10:09.430 INFO:teuthology.orchestra.run.smithi028.stdout: "mdsmap_epoch": 23, 2024-04-24T08:10:09.430 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch": 90, 2024-04-24T08:10:09.430 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_epoch_barrier": 86, 2024-04-24T08:10:09.430 INFO:teuthology.orchestra.run.smithi028.stdout: "uptime": 150.814427483 2024-04-24T08:10:09.430 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:10:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.424+0000 7f966e400640 1 -- 172.21.15.28:0/4142234059 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f964c0a3bb0 msgr2=0x7f964c0a6070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.424+0000 7f966e400640 1 --2- 172.21.15.28:0/4142234059 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f964c0a3bb0 0x7f964c0a6070 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x557d04b492e0 tx=0x7f9660005f90 comp rx=0 tx=0).stop 2024-04-24T08:10:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.424+0000 7f966e400640 1 -- 172.21.15.28:0/4142234059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9668109850 msgr2=0x7f966819eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.424+0000 7f966e400640 1 --2- 172.21.15.28:0/4142234059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9668109850 0x7f966819eed0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9654031a40 tx=0x7f9654031ee0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.424+0000 7f966e400640 1 -- 172.21.15.28:0/4142234059 shutdown_connections 2024-04-24T08:10:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.424+0000 7f966e400640 1 --2- 172.21.15.28:0/4142234059 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f964c0a3bb0 0x7f964c0a6070 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.424+0000 7f966e400640 1 --2- 172.21.15.28:0/4142234059 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9668109850 0x7f966819eed0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.424+0000 7f966e400640 1 --2- 172.21.15.28:0/4142234059 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f966810a210 0x7f96681a5f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.424+0000 7f966e400640 1 --2- 172.21.15.28:0/4142234059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9668108660 0x7f966819e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.424+0000 7f966e400640 1 -- 172.21.15.28:0/4142234059 >> 172.21.15.28:0/4142234059 conn(0x7f9668077600 msgr2=0x7f966810cc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.424+0000 7f966e400640 1 -- 172.21.15.28:0/4142234059 shutdown_connections 2024-04-24T08:10:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.424+0000 7f966e400640 1 -- 172.21.15.28:0/4142234059 wait complete. 2024-04-24T08:10:09.444 INFO:teuthology.run_tasks:Running task sequential... 2024-04-24T08:10:09.454 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-04-24T08:10:09.454 INFO:teuthology.task.sequential:In sequential, running task print... 2024-04-24T08:10:09.454 INFO:teuthology.task.print:Enabling mgr modules 2024-04-24T08:10:09.455 INFO:teuthology.task.sequential:In sequential, running task exec... 2024-04-24T08:10:09.455 INFO:teuthology.task.exec:Executing custom commands... 2024-04-24T08:10:09.455 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-24T08:10:09.456 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-04-24T08:10:09.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.662+0000 7f03b9800640 1 -- 172.21.15.28:0/2872135012 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03b41039b0 msgr2=0x7f03b410a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.662+0000 7f03b9800640 1 --2- 172.21.15.28:0/2872135012 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03b41039b0 0x7f03b410a2d0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f03ac009ca0 tx=0x7f03ac02a8b0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.662+0000 7f03b9800640 1 -- 172.21.15.28:0/2872135012 shutdown_connections 2024-04-24T08:10:09.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.662+0000 7f03b9800640 1 --2- 172.21.15.28:0/2872135012 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03b4102ff0 0x7f03b4103470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.662+0000 7f03b9800640 1 --2- 172.21.15.28:0/2872135012 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03b41039b0 0x7f03b410a2d0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.662+0000 7f03b9800640 1 --2- 172.21.15.28:0/2872135012 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03b4101ea0 0x7f03b41022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.662+0000 7f03b9800640 1 -- 172.21.15.28:0/2872135012 >> 172.21.15.28:0/2872135012 conn(0x7f03b40fd410 msgr2=0x7f03b40ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:09.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.662+0000 7f03b9800640 1 -- 172.21.15.28:0/2872135012 shutdown_connections 2024-04-24T08:10:09.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.662+0000 7f03b9800640 1 -- 172.21.15.28:0/2872135012 wait complete. 2024-04-24T08:10:09.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.664+0000 7f03b9800640 1 Processor -- start 2024-04-24T08:10:09.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.664+0000 7f03b9800640 1 -- start start 2024-04-24T08:10:09.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.664+0000 7f03b9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03b4101ea0 0x7f03b419a370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.664+0000 7f03b9800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03b4102ff0 0x7f03b419a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.664+0000 7f03b9800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03b41039b0 0x7f03b41a1920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.664+0000 7f03b9800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f03b410dba0 con 0x7f03b4101ea0 2024-04-24T08:10:09.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.665+0000 7f03b9800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f03b410da20 con 0x7f03b41039b0 2024-04-24T08:10:09.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.665+0000 7f03b9800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f03b410dd20 con 0x7f03b4102ff0 2024-04-24T08:10:09.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.665+0000 7f03b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03b4101ea0 0x7f03b419a370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.665+0000 7f03b3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03b4101ea0 0x7f03b419a370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41426/0 (socket says 172.21.15.28:41426) 2024-04-24T08:10:09.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.665+0000 7f03b3e00640 1 -- 172.21.15.28:0/3341204514 learned_addr learned my addr 172.21.15.28:0/3341204514 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:09.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.665+0000 7f03b3400640 1 --2- 172.21.15.28:0/3341204514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03b4102ff0 0x7f03b419a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.665+0000 7f03b8e00640 1 --2- 172.21.15.28:0/3341204514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03b41039b0 0x7f03b41a1920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.665+0000 7f03b3e00640 1 -- 172.21.15.28:0/3341204514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03b4102ff0 msgr2=0x7f03b419a8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.665+0000 7f03b3e00640 1 --2- 172.21.15.28:0/3341204514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03b4102ff0 0x7f03b419a8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.665+0000 7f03b3e00640 1 -- 172.21.15.28:0/3341204514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03b41039b0 msgr2=0x7f03b41a1920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:09.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.665+0000 7f03b3e00640 1 --2- 172.21.15.28:0/3341204514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03b41039b0 0x7f03b41a1920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:09.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.665+0000 7f03b3e00640 1 -- 172.21.15.28:0/3341204514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03b41a1f90 con 0x7f03b4101ea0 2024-04-24T08:10:09.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.666+0000 7f03b3e00640 1 --2- 172.21.15.28:0/3341204514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03b4101ea0 0x7f03b419a370 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f03a400d770 tx=0x7f03a400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:09.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.666+0000 7f03b0c00640 1 -- 172.21.15.28:0/3341204514 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a4017070 con 0x7f03b4101ea0 2024-04-24T08:10:09.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.666+0000 7f03b0c00640 1 -- 172.21.15.28:0/3341204514 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03a4005500 con 0x7f03b4101ea0 2024-04-24T08:10:09.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.666+0000 7f03b0c00640 1 -- 172.21.15.28:0/3341204514 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03a4004e00 con 0x7f03b4101ea0 2024-04-24T08:10:09.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.666+0000 7f03b9800640 1 -- 172.21.15.28:0/3341204514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03b41a2280 con 0x7f03b4101ea0 2024-04-24T08:10:09.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.666+0000 7f03b9800640 1 -- 172.21.15.28:0/3341204514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03b41a2780 con 0x7f03b4101ea0 2024-04-24T08:10:09.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.667+0000 7f03b9800640 1 -- 172.21.15.28:0/3341204514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f03b41a29e0 con 0x7f03b4101ea0 2024-04-24T08:10:09.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.669+0000 7f03b0c00640 1 -- 172.21.15.28:0/3341204514 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 18) ==== 138646+0+0 (secure 0 0 0) 0x7f03a4004fa0 con 0x7f03b4101ea0 2024-04-24T08:10:09.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.671+0000 7f03b0c00640 1 --2- 172.21.15.28:0/3341204514 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f03980a3ba0 0x7f03980a6060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:09.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.671+0000 7f03b0c00640 1 -- 172.21.15.28:0/3341204514 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f03a40cfe60 con 0x7f03b4101ea0 2024-04-24T08:10:09.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.671+0000 7f03b3400640 1 --2- 172.21.15.28:0/3341204514 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f03980a3ba0 0x7f03980a6060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:09.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.672+0000 7f03b3400640 1 --2- 172.21.15.28:0/3341204514 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f03980a3ba0 0x7f03980a6060 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f03a00095f0 tx=0x7f03a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:09.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.676+0000 7f03b0c00640 1 -- 172.21.15.28:0/3341204514 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f03a409c150 con 0x7f03b4101ea0 2024-04-24T08:10:09.779 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:09 smithi117 ceph-mon[32438]: pgmap v389: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:10:09.779 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:09 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3297248246' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:09.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:09 smithi028 ceph-mon[27786]: pgmap v389: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:10:09.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:09 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3297248246' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:09.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:09 smithi194 ceph-mon[32836]: pgmap v389: 97 pgs: 97 active+clean; 639 KiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:10:09.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:09 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3297248246' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:09.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:09.941+0000 7f03b9800640 1 -- 172.21.15.28:0/3341204514 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) -- 0x7f03b41a2cd0 con 0x7f03b4101ea0 2024-04-24T08:10:10.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.520+0000 7f03b0c00640 1 -- 172.21.15.28:0/3341204514 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) ==== 92+0+0 (secure 0 0 0) 0x7f03a40a0450 con 0x7f03b4101ea0 2024-04-24T08:10:10.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.524+0000 7f03b9800640 1 -- 172.21.15.28:0/3341204514 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f03980a3ba0 msgr2=0x7f03980a6060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:10.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.524+0000 7f03b9800640 1 --2- 172.21.15.28:0/3341204514 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f03980a3ba0 0x7f03980a6060 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f03a00095f0 tx=0x7f03a0005f90 comp rx=0 tx=0).stop 2024-04-24T08:10:10.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.524+0000 7f03b9800640 1 -- 172.21.15.28:0/3341204514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03b4101ea0 msgr2=0x7f03b419a370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:10.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.524+0000 7f03b9800640 1 --2- 172.21.15.28:0/3341204514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03b4101ea0 0x7f03b419a370 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f03a400d770 tx=0x7f03a400dc30 comp rx=0 tx=0).stop 2024-04-24T08:10:10.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.524+0000 7f03b9800640 1 -- 172.21.15.28:0/3341204514 shutdown_connections 2024-04-24T08:10:10.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.524+0000 7f03b9800640 1 --2- 172.21.15.28:0/3341204514 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f03980a3ba0 0x7f03980a6060 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.524+0000 7f03b9800640 1 --2- 172.21.15.28:0/3341204514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03b41039b0 0x7f03b41a1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.524+0000 7f03b9800640 1 --2- 172.21.15.28:0/3341204514 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f03b4102ff0 0x7f03b419a8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.524+0000 7f03b9800640 1 --2- 172.21.15.28:0/3341204514 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f03b4101ea0 0x7f03b419a370 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.524+0000 7f03b9800640 1 -- 172.21.15.28:0/3341204514 >> 172.21.15.28:0/3341204514 conn(0x7f03b40fd410 msgr2=0x7f03b4074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:10.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.526+0000 7f03b9800640 1 -- 172.21.15.28:0/3341204514 shutdown_connections 2024-04-24T08:10:10.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.526+0000 7f03b9800640 1 -- 172.21.15.28:0/3341204514 wait complete. 2024-04-24T08:10:10.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-04-24T08:10:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.742+0000 7f7edee00640 1 -- 172.21.15.28:0/1709906004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ed8101e70 msgr2=0x7f7ed8102290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.742+0000 7f7edee00640 1 --2- 172.21.15.28:0/1709906004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ed8101e70 0x7f7ed8102290 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f7ecc009900 tx=0x7f7ecc02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.742+0000 7f7edee00640 1 -- 172.21.15.28:0/1709906004 shutdown_connections 2024-04-24T08:10:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.742+0000 7f7edee00640 1 --2- 172.21.15.28:0/1709906004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed8103a10 0x7f7ed810a330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.742+0000 7f7edee00640 1 --2- 172.21.15.28:0/1709906004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ed8103050 0x7f7ed81034d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.742+0000 7f7edee00640 1 --2- 172.21.15.28:0/1709906004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ed8101e70 0x7f7ed8102290 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.742+0000 7f7edee00640 1 -- 172.21.15.28:0/1709906004 >> 172.21.15.28:0/1709906004 conn(0x7f7ed80fd430 msgr2=0x7f7ed80ff870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.743+0000 7f7edee00640 1 -- 172.21.15.28:0/1709906004 shutdown_connections 2024-04-24T08:10:10.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.743+0000 7f7edee00640 1 -- 172.21.15.28:0/1709906004 wait complete. 2024-04-24T08:10:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.744+0000 7f7edee00640 1 Processor -- start 2024-04-24T08:10:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.744+0000 7f7edee00640 1 -- start start 2024-04-24T08:10:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.744+0000 7f7edee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed8101e70 0x7f7ed819a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.744+0000 7f7edee00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ed8103050 0x7f7ed819a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.744+0000 7f7edee00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ed8103a10 0x7f7ed81a1960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.744+0000 7f7edee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed810db80 con 0x7f7ed8101e70 2024-04-24T08:10:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.744+0000 7f7edee00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed810da00 con 0x7f7ed8103a10 2024-04-24T08:10:10.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.744+0000 7f7edee00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7ed810dd00 con 0x7f7ed8103050 2024-04-24T08:10:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.744+0000 7f7edda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed8101e70 0x7f7ed819a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.745+0000 7f7edd000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ed8103050 0x7f7ed819a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.745+0000 7f7edda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed8101e70 0x7f7ed819a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41446/0 (socket says 172.21.15.28:41446) 2024-04-24T08:10:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.745+0000 7f7edd000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ed8103050 0x7f7ed819a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:46564/0 (socket says 172.21.15.28:46564) 2024-04-24T08:10:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.745+0000 7f7edda00640 1 -- 172.21.15.28:0/687273049 learned_addr learned my addr 172.21.15.28:0/687273049 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.745+0000 7f7ede400640 1 --2- 172.21.15.28:0/687273049 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ed8103a10 0x7f7ed81a1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.745+0000 7f7edd000640 1 -- 172.21.15.28:0/687273049 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ed8103a10 msgr2=0x7f7ed81a1960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.745+0000 7f7edd000640 1 --2- 172.21.15.28:0/687273049 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ed8103a10 0x7f7ed81a1960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.745+0000 7f7edd000640 1 -- 172.21.15.28:0/687273049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed8101e70 msgr2=0x7f7ed819a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.745+0000 7f7edd000640 1 --2- 172.21.15.28:0/687273049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed8101e70 0x7f7ed819a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.745+0000 7f7edd000640 1 -- 172.21.15.28:0/687273049 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ed81a1fd0 con 0x7f7ed8103050 2024-04-24T08:10:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.745+0000 7f7edda00640 1 --2- 172.21.15.28:0/687273049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed8101e70 0x7f7ed819a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.745+0000 7f7edd000640 1 --2- 172.21.15.28:0/687273049 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ed8103050 0x7f7ed819a8f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400e9f0 tx=0x7f7ec400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:10.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.746+0000 7f7ed6000640 1 -- 172.21.15.28:0/687273049 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec400ccc0 con 0x7f7ed8103050 2024-04-24T08:10:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.746+0000 7f7edee00640 1 -- 172.21.15.28:0/687273049 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ed81a22c0 con 0x7f7ed8103050 2024-04-24T08:10:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.746+0000 7f7ed6000640 1 -- 172.21.15.28:0/687273049 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ec4010910 con 0x7f7ed8103050 2024-04-24T08:10:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.746+0000 7f7ed6000640 1 -- 172.21.15.28:0/687273049 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ec4018960 con 0x7f7ed8103050 2024-04-24T08:10:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.746+0000 7f7edee00640 1 -- 172.21.15.28:0/687273049 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ed81a2860 con 0x7f7ed8103050 2024-04-24T08:10:10.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.747+0000 7f7edee00640 1 -- 172.21.15.28:0/687273049 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ed81a2ac0 con 0x7f7ed8103050 2024-04-24T08:10:10.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.750+0000 7f7ed6000640 1 -- 172.21.15.28:0/687273049 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f7ec4010ac0 con 0x7f7ed8103050 2024-04-24T08:10:10.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.751+0000 7f7ed6000640 1 --2- 172.21.15.28:0/687273049 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7ec00a3c70 0x7f7ec00a6130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:10.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.751+0000 7f7ed6000640 1 -- 172.21.15.28:0/687273049 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f7ec40d0800 con 0x7f7ed8103050 2024-04-24T08:10:10.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.751+0000 7f7edda00640 1 -- 172.21.15.28:0/687273049 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7ec00a3c70 msgr2=0x7f7ec00a6130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/361125225 2024-04-24T08:10:10.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.751+0000 7f7edda00640 1 --2- 172.21.15.28:0/687273049 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7ec00a3c70 0x7f7ec00a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:10:10.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.755+0000 7f7ed6000640 1 -- 172.21.15.28:0/687273049 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7ec409caf0 con 0x7f7ed8103050 2024-04-24T08:10:10.788 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:10 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: ignoring --setuser ceph since I am not root 2024-04-24T08:10:10.788 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:10 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: ignoring --setgroup ceph since I am not root 2024-04-24T08:10:10.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:10 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3341204514' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-24T08:10:10.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:10 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3341204514' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-24T08:10:10.878 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:10 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: ignoring --setuser ceph since I am not root 2024-04-24T08:10:10.878 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:10 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: ignoring --setgroup ceph since I am not root 2024-04-24T08:10:10.878 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:10 smithi117 ceph-mgr[33856]: -- 172.21.15.117:0/2591448886 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ba7dd425a0 con 0x55ba7dd16800 2024-04-24T08:10:10.878 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:10 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:10.703+0000 7fedcbd4f200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T08:10:10.890 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:10 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3341204514' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-04-24T08:10:10.890 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:10 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: ignoring --setuser ceph since I am not root 2024-04-24T08:10:10.890 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:10 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: ignoring --setgroup ceph since I am not root 2024-04-24T08:10:10.890 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:10 smithi194 ceph-mgr[34728]: -- 172.21.15.194:0/3977870331 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a8fdb605a0 con 0x55a8fdb34c00 2024-04-24T08:10:10.890 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:10 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:10.703+0000 7fc9944fb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T08:10:10.890 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:10 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:10.888+0000 7fc9944fb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:10:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.952+0000 7f7edda00640 1 -- 172.21.15.28:0/687273049 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7ec00a3c70 msgr2=0x7f7ec00a6130 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/361125225 2024-04-24T08:10:10.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.952+0000 7f7edda00640 1 --2- 172.21.15.28:0/687273049 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7ec00a3c70 0x7f7ec00a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T08:10:10.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.963+0000 7f7edee00640 1 -- 172.21.15.28:0/687273049 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) -- 0x7f7ed81a31e0 con 0x7f7ed8103050 2024-04-24T08:10:10.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.970+0000 7f7ed6000640 1 -- 172.21.15.28:0/687273049 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v28)=0 v28) ==== 138+0+0 (secure 0 0 0) 0x7f7ec40a0df0 con 0x7f7ed8103050 2024-04-24T08:10:10.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.972+0000 7f7edee00640 1 -- 172.21.15.28:0/687273049 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7ec00a3c70 msgr2=0x7f7ec00a6130 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:10:10.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.972+0000 7f7edee00640 1 --2- 172.21.15.28:0/687273049 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7ec00a3c70 0x7f7ec00a6130 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.972+0000 7f7edee00640 1 -- 172.21.15.28:0/687273049 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ed8103050 msgr2=0x7f7ed819a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:10.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.972+0000 7f7edee00640 1 --2- 172.21.15.28:0/687273049 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ed8103050 0x7f7ed819a8f0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f7ec400e9f0 tx=0x7f7ec400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.973+0000 7f7edee00640 1 -- 172.21.15.28:0/687273049 shutdown_connections 2024-04-24T08:10:10.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.973+0000 7f7edee00640 1 --2- 172.21.15.28:0/687273049 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f7ec00a3c70 0x7f7ec00a6130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.973+0000 7f7edee00640 1 --2- 172.21.15.28:0/687273049 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ed8103a10 0x7f7ed81a1960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.973+0000 7f7edee00640 1 --2- 172.21.15.28:0/687273049 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ed8103050 0x7f7ed819a8f0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.973+0000 7f7edee00640 1 --2- 172.21.15.28:0/687273049 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ed8101e70 0x7f7ed819a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:10.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.973+0000 7f7edee00640 1 -- 172.21.15.28:0/687273049 >> 172.21.15.28:0/687273049 conn(0x7f7ed80fd430 msgr2=0x7f7ed8074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:10.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.973+0000 7f7edee00640 1 -- 172.21.15.28:0/687273049 shutdown_connections 2024-04-24T08:10:10.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:10.973+0000 7f7edee00640 1 -- 172.21.15.28:0/687273049 wait complete. 2024-04-24T08:10:10.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-04-24T08:10:11.101 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:10 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:10.847+0000 7f8c3b8e2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-04-24T08:10:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.181+0000 7fa83b200640 1 -- 172.21.15.28:0/3673198280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa83c074a60 msgr2=0x7fa83c074ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.181+0000 7fa83b200640 1 --2- 172.21.15.28:0/3673198280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa83c074a60 0x7fa83c074ee0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa828009900 tx=0x7fa82802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.181+0000 7fa83b200640 1 -- 172.21.15.28:0/3673198280 shutdown_connections 2024-04-24T08:10:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.181+0000 7fa83b200640 1 --2- 172.21.15.28:0/3673198280 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa83c109de0 0x7fa83c10c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.181+0000 7fa83b200640 1 --2- 172.21.15.28:0/3673198280 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa83c074a60 0x7fa83c074ee0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.181+0000 7fa83b200640 1 --2- 172.21.15.28:0/3673198280 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa83c076040 0x7fa83c074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.181+0000 7fa83b200640 1 -- 172.21.15.28:0/3673198280 >> 172.21.15.28:0/3673198280 conn(0x7fa83c0fd490 msgr2=0x7fa83c0ff8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.181+0000 7fa83b200640 1 -- 172.21.15.28:0/3673198280 shutdown_connections 2024-04-24T08:10:11.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.181+0000 7fa83b200640 1 -- 172.21.15.28:0/3673198280 wait complete. 2024-04-24T08:10:11.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.183+0000 7fa83b200640 1 Processor -- start 2024-04-24T08:10:11.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.183+0000 7fa83b200640 1 -- start start 2024-04-24T08:10:11.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.183+0000 7fa83b200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa83c074a60 0x7fa83c19a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:11.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.183+0000 7fa83b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa83c076040 0x7fa83c19a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:11.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.183+0000 7fa83b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa83c109de0 0x7fa83c1a1950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:11.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.183+0000 7fa83b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa83c10fd80 con 0x7fa83c109de0 2024-04-24T08:10:11.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.183+0000 7fa83b200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa83c10fc00 con 0x7fa83c074a60 2024-04-24T08:10:11.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.183+0000 7fa83b200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa83c10ff00 con 0x7fa83c076040 2024-04-24T08:10:11.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.184+0000 7fa839e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa83c074a60 0x7fa83c19a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:11.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.184+0000 7fa839e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa83c074a60 0x7fa83c19a3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:59986/0 (socket says 172.21.15.28:59986) 2024-04-24T08:10:11.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.184+0000 7fa839e00640 1 -- 172.21.15.28:0/3819037703 learned_addr learned my addr 172.21.15.28:0/3819037703 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:11.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.184+0000 7fa83a800640 1 --2- 172.21.15.28:0/3819037703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa83c109de0 0x7fa83c1a1950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:11.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.184+0000 7fa839400640 1 --2- 172.21.15.28:0/3819037703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa83c076040 0x7fa83c19a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:11.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.184+0000 7fa839e00640 1 -- 172.21.15.28:0/3819037703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa83c076040 msgr2=0x7fa83c19a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:11.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.184+0000 7fa839e00640 1 --2- 172.21.15.28:0/3819037703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa83c076040 0x7fa83c19a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:11.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.184+0000 7fa839e00640 1 -- 172.21.15.28:0/3819037703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa83c109de0 msgr2=0x7fa83c1a1950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:11.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.184+0000 7fa839e00640 1 --2- 172.21.15.28:0/3819037703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa83c109de0 0x7fa83c1a1950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:11.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.184+0000 7fa839e00640 1 -- 172.21.15.28:0/3819037703 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83c1a1fc0 con 0x7fa83c074a60 2024-04-24T08:10:11.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.184+0000 7fa839400640 1 --2- 172.21.15.28:0/3819037703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa83c076040 0x7fa83c19a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:11.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.185+0000 7fa839e00640 1 --2- 172.21.15.28:0/3819037703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa83c074a60 0x7fa83c19a3a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa82c00e9f0 tx=0x7fa82c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:11.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.185+0000 7fa832a00640 1 -- 172.21.15.28:0/3819037703 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c00cd80 con 0x7fa83c074a60 2024-04-24T08:10:11.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.185+0000 7fa832a00640 1 -- 172.21.15.28:0/3819037703 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa82c010910 con 0x7fa83c074a60 2024-04-24T08:10:11.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.185+0000 7fa832a00640 1 -- 172.21.15.28:0/3819037703 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa82c018990 con 0x7fa83c074a60 2024-04-24T08:10:11.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.185+0000 7fa83b200640 1 -- 172.21.15.28:0/3819037703 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa83c1a22b0 con 0x7fa83c074a60 2024-04-24T08:10:11.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.185+0000 7fa83b200640 1 -- 172.21.15.28:0/3819037703 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa83c1a2780 con 0x7fa83c074a60 2024-04-24T08:10:11.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.186+0000 7fa83b200640 1 -- 172.21.15.28:0/3819037703 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa83c108b10 con 0x7fa83c074a60 2024-04-24T08:10:11.190 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:10 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:10.992+0000 7fc9944fb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T08:10:11.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.190+0000 7fa832a00640 1 -- 172.21.15.28:0/3819037703 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7fa82c010ac0 con 0x7fa83c074a60 2024-04-24T08:10:11.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.191+0000 7fa832a00640 1 --2- 172.21.15.28:0/3819037703 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa8200a3bf0 0x7fa8200a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:11.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.191+0000 7fa839400640 1 -- 172.21.15.28:0/3819037703 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa8200a3bf0 msgr2=0x7fa8200a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/361125225 2024-04-24T08:10:11.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.191+0000 7fa832a00640 1 -- 172.21.15.28:0/3819037703 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7fa82c0d07c0 con 0x7fa83c074a60 2024-04-24T08:10:11.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.191+0000 7fa839400640 1 --2- 172.21.15.28:0/3819037703 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa8200a3bf0 0x7fa8200a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:10:11.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.195+0000 7fa832a00640 1 -- 172.21.15.28:0/3819037703 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa82c09cb10 con 0x7fa83c074a60 2024-04-24T08:10:11.298 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:10 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:10.886+0000 7fedcbd4f200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:10:11.298 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:10 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:10.989+0000 7fedcbd4f200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T08:10:11.298 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:11 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:11.297+0000 7fedcbd4f200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T08:10:11.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.392+0000 7fa839400640 1 -- 172.21.15.28:0/3819037703 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa8200a3bf0 msgr2=0x7fa8200a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/361125225 2024-04-24T08:10:11.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.392+0000 7fa839400640 1 --2- 172.21.15.28:0/3819037703 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa8200a3bf0 0x7fa8200a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T08:10:11.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.437+0000 7fa83b200640 1 -- 172.21.15.28:0/3819037703 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) -- 0x7fa83c02d620 con 0x7fa83c074a60 2024-04-24T08:10:11.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.443+0000 7fa832a00640 1 -- 172.21.15.28:0/3819037703 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v29)=0 v29) ==== 133+0+0 (secure 0 0 0) 0x7fa82c0a0e10 con 0x7fa83c074a60 2024-04-24T08:10:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.446+0000 7fa83b200640 1 -- 172.21.15.28:0/3819037703 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa8200a3bf0 msgr2=0x7fa8200a60b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:10:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.446+0000 7fa83b200640 1 --2- 172.21.15.28:0/3819037703 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa8200a3bf0 0x7fa8200a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:11.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.446+0000 7fa83b200640 1 -- 172.21.15.28:0/3819037703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa83c074a60 msgr2=0x7fa83c19a3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:11.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.446+0000 7fa83b200640 1 --2- 172.21.15.28:0/3819037703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa83c074a60 0x7fa83c19a3a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa82c00e9f0 tx=0x7fa82c00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:10:11.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.446+0000 7fa83b200640 1 -- 172.21.15.28:0/3819037703 shutdown_connections 2024-04-24T08:10:11.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.446+0000 7fa83b200640 1 --2- 172.21.15.28:0/3819037703 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7fa8200a3bf0 0x7fa8200a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:11.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.446+0000 7fa83b200640 1 --2- 172.21.15.28:0/3819037703 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa83c109de0 0x7fa83c1a1950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:11.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.446+0000 7fa83b200640 1 --2- 172.21.15.28:0/3819037703 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa83c076040 0x7fa83c19a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:11.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.446+0000 7fa83b200640 1 --2- 172.21.15.28:0/3819037703 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa83c074a60 0x7fa83c19a3a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:11.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.446+0000 7fa83b200640 1 -- 172.21.15.28:0/3819037703 >> 172.21.15.28:0/3819037703 conn(0x7fa83c0fd490 msgr2=0x7fa83c105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:11.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.446+0000 7fa83b200640 1 -- 172.21.15.28:0/3819037703 shutdown_connections 2024-04-24T08:10:11.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:11.446+0000 7fa83b200640 1 -- 172.21.15.28:0/3819037703 wait complete. 2024-04-24T08:10:11.463 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:11 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:11.098+0000 7f8c3b8e2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:10:11.463 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:11 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:11.276+0000 7f8c3b8e2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-04-24T08:10:11.466 INFO:teuthology.run_tasks:Running task kclient... 2024-04-24T08:10:11.492 INFO:tasks.kclient:Mounting kernel clients... 2024-04-24T08:10:11.492 INFO:tasks.kclient:config is {'client.0': {}} 2024-04-24T08:10:11.492 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:11.492 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-24T08:10:11.548 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:11.548 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-04-24T08:10:11.621 INFO:teuthology.orchestra.run.smithi028.stderr:Cannot find device "ceph-brx" 2024-04-24T08:10:11.624 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:10:11.624 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=secure', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-04-24T08:10:11.624 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-04-24T08:10:11.625 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:10:11.625 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph fs ls 2024-04-24T08:10:11.627 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:11 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:11.383+0000 7fedcbd4f200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T08:10:11.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:11 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3341204514' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-24T08:10:11.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:11 smithi117 ceph-mon[32438]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-24T08:10:11.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:11 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' 2024-04-24T08:10:11.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:11 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' 2024-04-24T08:10:11.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:11 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3341204514' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-24T08:10:11.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:11 smithi194 ceph-mon[32836]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-24T08:10:11.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:11 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' 2024-04-24T08:10:11.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:11 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' 2024-04-24T08:10:11.691 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:11 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:11.302+0000 7fc9944fb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T08:10:11.691 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:11 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:11.388+0000 7fc9944fb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T08:10:11.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:11 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3341204514' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-04-24T08:10:11.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:11 smithi028 ceph-mon[27786]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-04-24T08:10:11.723 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:11 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' 2024-04-24T08:10:11.724 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:11 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' 2024-04-24T08:10:12.039 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:11 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:11.722+0000 7f8c3b8e2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-04-24T08:10:12.039 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:11 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:11.841+0000 7f8c3b8e2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-04-24T08:10:12.062 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:10:12.877 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:12 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:12.555+0000 7fedcbd4f200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T08:10:12.878 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:12 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:12.723+0000 7fedcbd4f200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T08:10:12.912 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:12 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:12.571+0000 7fc9944fb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T08:10:12.912 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:12 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:12.739+0000 7fc9944fb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T08:10:13.179 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:12 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:12.892+0000 7fedcbd4f200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T08:10:13.190 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:12 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:12.910+0000 7fc9944fb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T08:10:13.475 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:13 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:13.178+0000 7fedcbd4f200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T08:10:13.475 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:13 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:13.271+0000 7fedcbd4f200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T08:10:13.475 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:13 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:13.474+0000 7fedcbd4f200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T08:10:13.502 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:13 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:13.202+0000 7fc9944fb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T08:10:13.502 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:13 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:13.295+0000 7fc9944fb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T08:10:13.710 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:13 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:13.531+0000 7f8c3b8e2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-04-24T08:10:13.877 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:13 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:13.666+0000 7fedcbd4f200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:10:13.940 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:13 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:13.500+0000 7fc9944fb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T08:10:13.940 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:13 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:13.694+0000 7fc9944fb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:10:13.998 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:13 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:13.772+0000 7f8c3b8e2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-04-24T08:10:14.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.195+0000 7f887ba00700 1 -- 172.21.15.28:0/2843781126 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8874104f90 msgr2=0x7f8874105370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:14.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.195+0000 7f887ba00700 1 --2- 172.21.15.28:0/2843781126 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8874104f90 0x7f8874105370 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f88680099c0 tx=0x7f8868009cd0 comp rx=0 tx=0).stop 2024-04-24T08:10:14.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.195+0000 7f887ba00700 1 -- 172.21.15.28:0/2843781126 shutdown_connections 2024-04-24T08:10:14.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.195+0000 7f887ba00700 1 --2- 172.21.15.28:0/2843781126 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f887410a070 0x7f8874111c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:14.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.195+0000 7f887ba00700 1 --2- 172.21.15.28:0/2843781126 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8874104f90 0x7f8874105370 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:14.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.195+0000 7f887ba00700 1 --2- 172.21.15.28:0/2843781126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8874105940 0x7f8874109990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:14.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.195+0000 7f887ba00700 1 -- 172.21.15.28:0/2843781126 >> 172.21.15.28:0/2843781126 conn(0x7f8874100820 msgr2=0x7f8874102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:14.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.195+0000 7f887ba00700 1 -- 172.21.15.28:0/2843781126 shutdown_connections 2024-04-24T08:10:14.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.195+0000 7f887ba00700 1 -- 172.21.15.28:0/2843781126 wait complete. 2024-04-24T08:10:14.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.197+0000 7f887ba00700 1 Processor -- start 2024-04-24T08:10:14.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.197+0000 7f887ba00700 1 -- start start 2024-04-24T08:10:14.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.197+0000 7f887ba00700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8874104f90 0x7f887419b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:14.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.197+0000 7f887ba00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8874105940 0x7f887419baa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:14.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.197+0000 7f887ba00700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f887410a070 0x7f887419fe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:14.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.197+0000 7f887ba00700 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8874115050 con 0x7f8874104f90 2024-04-24T08:10:14.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.197+0000 7f887ba00700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8874114ed0 con 0x7f887410a070 2024-04-24T08:10:14.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.197+0000 7f887ba00700 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f88741151d0 con 0x7f8874105940 2024-04-24T08:10:14.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.198+0000 7f8879000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8874104f90 0x7f887419b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:14.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.198+0000 7f8873e00700 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8874105940 0x7f887419baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:14.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.198+0000 7f8879000700 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8874104f90 0x7f887419b560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41486/0 (socket says 172.21.15.28:41486) 2024-04-24T08:10:14.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.198+0000 7f8879000700 1 -- 172.21.15.28:0/826587665 learned_addr learned my addr 172.21.15.28:0/826587665 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:14.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.198+0000 7f8879a00700 1 --2- 172.21.15.28:0/826587665 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f887410a070 0x7f887419fe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:14.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.198+0000 7f8873e00700 1 -- 172.21.15.28:0/826587665 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f887410a070 msgr2=0x7f887419fe30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:14.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.198+0000 7f8873e00700 1 --2- 172.21.15.28:0/826587665 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f887410a070 0x7f887419fe30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:14.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.198+0000 7f8873e00700 1 -- 172.21.15.28:0/826587665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8874104f90 msgr2=0x7f887419b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:14.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.198+0000 7f8873e00700 1 --2- 172.21.15.28:0/826587665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8874104f90 0x7f887419b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:14.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.198+0000 7f8873e00700 1 -- 172.21.15.28:0/826587665 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88741a0570 con 0x7f8874105940 2024-04-24T08:10:14.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.198+0000 7f8873e00700 1 --2- 172.21.15.28:0/826587665 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8874105940 0x7f887419baa0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f886000e9c0 tx=0x7f886000ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:14.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.199+0000 7f8871600700 1 -- 172.21.15.28:0/826587665 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f886000cbd0 con 0x7f8874105940 2024-04-24T08:10:14.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.199+0000 7f8871600700 1 -- 172.21.15.28:0/826587665 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f886000cd70 con 0x7f8874105940 2024-04-24T08:10:14.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.199+0000 7f887ba00700 1 -- 172.21.15.28:0/826587665 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88741a0860 con 0x7f8874105940 2024-04-24T08:10:14.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.199+0000 7f8871600700 1 -- 172.21.15.28:0/826587665 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8860018680 con 0x7f8874105940 2024-04-24T08:10:14.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.199+0000 7f887ba00700 1 -- 172.21.15.28:0/826587665 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88741a80e0 con 0x7f8874105940 2024-04-24T08:10:14.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.200+0000 7f887ba00700 1 -- 172.21.15.28:0/826587665 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88740697e0 con 0x7f8874105940 2024-04-24T08:10:14.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.202+0000 7f8871600700 1 -- 172.21.15.28:0/826587665 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 19) ==== 138663+0+0 (secure 0 0 0) 0x7f8860010430 con 0x7f8874105940 2024-04-24T08:10:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.203+0000 7f8871600700 1 --2- 172.21.15.28:0/826587665 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f885c0a3d10 0x7f885c0a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.203+0000 7f8879000700 1 -- 172.21.15.28:0/826587665 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f885c0a3d10 msgr2=0x7f885c0a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/361125225 2024-04-24T08:10:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.203+0000 7f8871600700 1 -- 172.21.15.28:0/826587665 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f88600d0650 con 0x7f8874105940 2024-04-24T08:10:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.203+0000 7f8879000700 1 --2- 172.21.15.28:0/826587665 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f885c0a3d10 0x7f885c0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:10:14.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.207+0000 7f8871600700 1 -- 172.21.15.28:0/826587665 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f886009cf90 con 0x7f8874105940 2024-04-24T08:10:14.289 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:13 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:13.996+0000 7f8c3b8e2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-04-24T08:10:14.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.404+0000 7f8879000700 1 -- 172.21.15.28:0/826587665 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f885c0a3d10 msgr2=0x7f885c0a61d0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/361125225 2024-04-24T08:10:14.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.404+0000 7f8879000700 1 --2- 172.21.15.28:0/826587665 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f885c0a3d10 0x7f885c0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T08:10:14.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.483+0000 7f887ba00700 1 -- 172.21.15.28:0/826587665 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f887402dfe0 con 0x7f8874105940 2024-04-24T08:10:14.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.484+0000 7f8871600700 1 -- 172.21.15.28:0/826587665 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v23) ==== 53+0+73 (secure 0 0 0) 0x7f886009c890 con 0x7f8874105940 2024-04-24T08:10:14.484 INFO:teuthology.orchestra.run.smithi028.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-04-24T08:10:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.486+0000 7f887ba00700 1 -- 172.21.15.28:0/826587665 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f885c0a3d10 msgr2=0x7f885c0a61d0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:10:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.486+0000 7f887ba00700 1 --2- 172.21.15.28:0/826587665 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f885c0a3d10 0x7f885c0a61d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.486+0000 7f887ba00700 1 -- 172.21.15.28:0/826587665 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8874105940 msgr2=0x7f887419baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.486+0000 7f887ba00700 1 --2- 172.21.15.28:0/826587665 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8874105940 0x7f887419baa0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f886000e9c0 tx=0x7f886000ecd0 comp rx=0 tx=0).stop 2024-04-24T08:10:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.487+0000 7f887ba00700 1 -- 172.21.15.28:0/826587665 shutdown_connections 2024-04-24T08:10:14.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.487+0000 7f887ba00700 1 --2- 172.21.15.28:0/826587665 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f885c0a3d10 0x7f885c0a61d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.487+0000 7f887ba00700 1 --2- 172.21.15.28:0/826587665 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f887410a070 0x7f887419fe30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.487+0000 7f887ba00700 1 --2- 172.21.15.28:0/826587665 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8874104f90 0x7f887419b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.487+0000 7f887ba00700 1 --2- 172.21.15.28:0/826587665 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8874105940 0x7f887419baa0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.487+0000 7f887ba00700 1 -- 172.21.15.28:0/826587665 >> 172.21.15.28:0/826587665 conn(0x7f8874100820 msgr2=0x7f8874102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.487+0000 7f887ba00700 1 -- 172.21.15.28:0/826587665 shutdown_connections 2024-04-24T08:10:14.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:14.487+0000 7f887ba00700 1 -- 172.21.15.28:0/826587665 wait complete. 2024-04-24T08:10:14.537 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:14 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:14.258+0000 7fedcbd4f200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T08:10:14.690 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:14 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:14.293+0000 7fc9944fb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T08:10:14.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:14 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/826587665' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-24T08:10:14.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:14 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/826587665' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-24T08:10:14.789 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:14 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:14.403+0000 7f8c3b8e2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-04-24T08:10:14.789 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:14 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:14.533+0000 7f8c3b8e2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-04-24T08:10:14.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:14 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/826587665' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-04-24T08:10:14.878 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:14 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:14.678+0000 7fedcbd4f200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T08:10:14.878 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:14 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:14.774+0000 7fedcbd4f200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T08:10:15.009 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:14 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:14.714+0000 7fc9944fb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T08:10:15.009 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:14 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:14.810+0000 7fc9944fb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T08:10:15.075 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:14 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:14.803+0000 7f8c3b8e2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-04-24T08:10:15.318 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-04-24T08:10:15.319 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-04-24T08:10:15.319 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi028.front.sepia.ceph.com 2024-04-24T08:10:15.319 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-04-24T08:10:15.319 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-04-24T08:10:15.319 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-04-24T08:10:15.319 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-04-24T08:10:15.319 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-04-24T08:10:15.319 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:10:15.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:15.319 DEBUG:teuthology.orchestra.run.smithi028:> ip addr 2024-04-24T08:10:15.374 INFO:teuthology.orchestra.run.smithi028.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-04-24T08:10:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-04-24T08:10:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: inet 127.0.0.1/8 scope host lo 2024-04-24T08:10:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:10:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 ::1/128 scope host 2024-04-24T08:10:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:10:15.374 INFO:teuthology.orchestra.run.smithi028.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:10:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:16 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:10:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f0 2024-04-24T08:10:15.374 INFO:teuthology.orchestra.run.smithi028.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:10:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether 0c:c4:7a:6b:ee:17 brd ff:ff:ff:ff:ff:ff 2024-04-24T08:10:15.374 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp7s0f1 2024-04-24T08:10:15.375 INFO:teuthology.orchestra.run.smithi028.stdout:4: ens1f0: mtu 1500 qdisc mq state UP group default qlen 1000 2024-04-24T08:10:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3c brd ff:ff:ff:ff:ff:ff 2024-04-24T08:10:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f0 2024-04-24T08:10:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: inet 172.21.15.28/20 brd 172.21.15.255 scope global dynamic ens1f0 2024-04-24T08:10:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft 41952sec preferred_lft 41952sec 2024-04-24T08:10:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: inet6 fe80::ae1f:6bff:fea5:ca3c/64 scope link noprefixroute 2024-04-24T08:10:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: valid_lft forever preferred_lft forever 2024-04-24T08:10:15.375 INFO:teuthology.orchestra.run.smithi028.stdout:5: ens1f1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-04-24T08:10:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: link/ether ac:1f:6b:a5:ca:3d brd ff:ff:ff:ff:ff:ff 2024-04-24T08:10:15.375 INFO:teuthology.orchestra.run.smithi028.stdout: altname enp1s0f1 2024-04-24T08:10:15.375 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-04-24T08:10:15.376 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:10:15.376 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-24T08:10:15.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add name ceph-brx type bridge 2024-04-24T08:10:15.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr flush dev ceph-brx 2024-04-24T08:10:15.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set ceph-brx up 2024-04-24T08:10:15.376 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-04-24T08:10:15.376 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-24T08:10:15.377 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:14 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:14.974+0000 7fedcbd4f200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T08:10:15.440 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:15 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:15.008+0000 7fc9944fb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T08:10:15.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:15 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:10:15.503 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:15 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:15.074+0000 7f8c3b8e2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-04-24T08:10:15.828 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:15 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:15.493+0000 7fedcbd4f200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T08:10:15.828 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:15 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:15.585+0000 7fedcbd4f200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T08:10:15.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:15 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:10:15.844 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:15.845 DEBUG:teuthology.orchestra.run.smithi028:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-04-24T08:10:15.871 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:15 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:15.536+0000 7fc9944fb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T08:10:15.871 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:15 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:15.627+0000 7fc9944fb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T08:10:15.911 INFO:teuthology.orchestra.run.smithi028.stdout:1 2024-04-24T08:10:15.915 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:15.915 DEBUG:teuthology.orchestra.run.smithi028:> route 2024-04-24T08:10:15.963 INFO:teuthology.orchestra.run.smithi028.stdout:Kernel IP routing table 2024-04-24T08:10:15.964 INFO:teuthology.orchestra.run.smithi028.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-04-24T08:10:15.964 INFO:teuthology.orchestra.run.smithi028.stdout:default _gateway 0.0.0.0 UG 100 0 0 ens1f0 2024-04-24T08:10:15.964 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 ens1f0 2024-04-24T08:10:15.964 INFO:teuthology.orchestra.run.smithi028.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-04-24T08:10:15.965 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:10:15.965 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-24T08:10:15.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -o ens1f0 -i ceph-brx -j ACCEPT 2024-04-24T08:10:15.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -A FORWARD -i ens1f0 -o ceph-brx -j ACCEPT 2024-04-24T08:10:15.965 DEBUG:teuthology.orchestra.run.smithi028:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o ens1f0 -j MASQUERADE 2024-04-24T08:10:15.965 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-24T08:10:16.038 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:15 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:15.911+0000 7f8c3b8e2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-04-24T08:10:16.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:10:16.127 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:15 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:15.828+0000 7fedcbd4f200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T08:10:16.128 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:15 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:15.958+0000 7fedcbd4f200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T08:10:16.128 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:16 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:16.052+0000 7fedcbd4f200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T08:10:16.190 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:15 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:15.870+0000 7fc9944fb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T08:10:16.190 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:16 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:16.002+0000 7fc9944fb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T08:10:16.190 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:16 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:16.098+0000 7fc9944fb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T08:10:16.468 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:16 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:16.170+0000 7fedcbd4f200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T08:10:16.524 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:16 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:16.216+0000 7fc9944fb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T08:10:16.642 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:16 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:16.506+0000 7f8c3b8e2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-04-24T08:10:16.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:16 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:10:16.729 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:16.730 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-24T08:10:16.775 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:16.775 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list-id 2024-04-24T08:10:16.863 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:10:16.864 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-24T08:10:16.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:10:16.864 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-04-24T08:10:16.864 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-24T08:10:16.877 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:16 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:16.468+0000 7fedcbd4f200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T08:10:16.925 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:16 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:16.641+0000 7f8c3b8e2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-04-24T08:10:16.940 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:16 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:16.522+0000 7fc9944fb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T08:10:16.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:16 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:10:17.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:10:17.028 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-04-24T08:10:17.028 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:10:17.028 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-24T08:10:17.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-04-24T08:10:17.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-04-24T08:10:17.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-04-24T08:10:17.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-04-24T08:10:17.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-04-24T08:10:17.029 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-24T08:10:17.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:10:17.182 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:16 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:16.925+0000 7f8c3b8e2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-04-24T08:10:17.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:10:17.326 DEBUG:teuthology.orchestra.run.smithi028:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-04-24T08:10:17.326 DEBUG:teuthology.orchestra.run.smithi028:> set -e 2024-04-24T08:10:17.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set brx.0 up 2024-04-24T08:10:17.327 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link set dev brx.0 master ceph-brx 2024-04-24T08:10:17.327 DEBUG:teuthology.orchestra.run.smithi028:> ') 2024-04-24T08:10:17.347 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:17 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:17.063+0000 7fedcbd4f200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T08:10:17.348 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:10:17 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y[33852]: 2024-04-24T08:10:17.277+0000 7fedcbd4f200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T08:10:17.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:17 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:10:17.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:17 smithi194 ceph-mon[32836]: Standby manager daemon y restarted 2024-04-24T08:10:17.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:17 smithi194 ceph-mon[32836]: Standby manager daemon y started 2024-04-24T08:10:17.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:17 smithi194 ceph-mon[32836]: from='mgr.? 172.21.15.117:0/2509877270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T08:10:17.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:17 smithi194 ceph-mon[32836]: from='mgr.? 172.21.15.117:0/2509877270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:10:17.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:17 smithi194 ceph-mon[32836]: from='mgr.? 172.21.15.117:0/2509877270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T08:10:17.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:17 smithi194 ceph-mon[32836]: from='mgr.? 172.21.15.117:0/2509877270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:10:17.441 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:17 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:17.124+0000 7fc9944fb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T08:10:17.441 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:10:17 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z[34724]: 2024-04-24T08:10:17.339+0000 7fc9944fb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T08:10:17.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:17 smithi028 ceph-mon[27786]: Standby manager daemon y restarted 2024-04-24T08:10:17.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:17 smithi028 ceph-mon[27786]: Standby manager daemon y started 2024-04-24T08:10:17.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:17 smithi028 ceph-mon[27786]: from='mgr.? 172.21.15.117:0/2509877270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T08:10:17.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:17 smithi028 ceph-mon[27786]: from='mgr.? 172.21.15.117:0/2509877270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:10:17.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:17 smithi028 ceph-mon[27786]: from='mgr.? 172.21.15.117:0/2509877270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T08:10:17.445 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:17 smithi028 ceph-mon[27786]: from='mgr.? 172.21.15.117:0/2509877270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:10:17.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:17 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:10:17.525 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:10:17.525 DEBUG:teuthology.orchestra.run.smithi028:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-04-24T08:10:17.566 INFO:teuthology.orchestra.run.smithi028.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-04-24T08:10:17.567 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-04-24T08:10:17.567 DEBUG:teuthology.orchestra.run.smithi028:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:10:17.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:17 smithi117 ceph-mon[32438]: Standby manager daemon y restarted 2024-04-24T08:10:17.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:17 smithi117 ceph-mon[32438]: Standby manager daemon y started 2024-04-24T08:10:17.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:17 smithi117 ceph-mon[32438]: from='mgr.? 172.21.15.117:0/2509877270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-04-24T08:10:17.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:17 smithi117 ceph-mon[32438]: from='mgr.? 172.21.15.117:0/2509877270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:10:17.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:17 smithi117 ceph-mon[32438]: from='mgr.? 172.21.15.117:0/2509877270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-04-24T08:10:17.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:17 smithi117 ceph-mon[32438]: from='mgr.? 172.21.15.117:0/2509877270' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:10:17.651 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-04-24T08:10:17.651 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-04-24T08:10:17.651 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/30aae337-e674-40f1-ba6b-0c04150e17c4 2024-04-24T08:10:17.651 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:17.651 DEBUG:teuthology.orchestra.run.smithi028:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/30aae337-e674-40f1-ba6b-0c04150e17c4 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-04-24T08:10:17.747 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:17 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:17.633+0000 7f8c3b8e2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-04-24T08:10:17.809 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-04-24T08:10:17.809 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync". 2024-04-24T08:10:17.809 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-04-24T08:10:18.039 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:17 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:17.765+0000 7f8c3b8e2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-04-24T08:10:18.354 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:18 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:18.105+0000 7f8c3b8e2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-04-24T08:10:18.355 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:18 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:18.298+0000 7f8c3b8e2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-04-24T08:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync 2024-04-24T08:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync". 2024-04-24T08:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout:invalid new device string format 2024-04-24T08:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: resolved to: "172.21.15.28:3300,172.21.15.117:3300,172.21.15.194:3300" 2024-04-24T08:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.28:3300,172.21.15.117:3300,172.21.15.194:3300:/volumes/qa/sv_1/30aae337-e674-40f1-ba6b-0c04150e17c4 2024-04-24T08:10:18.592 INFO:teuthology.orchestra.run.smithi028.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=secure,nowsync,key=0,fsid=473aaf58-0210-11ef-bc93-c7b262605968" will pass to kernel 2024-04-24T08:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-04-24T08:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout: You just mounted a file system that supports labels which does not 2024-04-24T08:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-04-24T08:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout: applications will generate AVC messages and not be allowed access to 2024-04-24T08:10:18.596 INFO:teuthology.orchestra.run.smithi028.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-04-24T08:10:18.599 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-04-24T08:10:18.600 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:18.600 DEBUG:teuthology.orchestra.run.smithi028:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-04-24T08:10:18.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:18 smithi117 ceph-mon[32438]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-24T08:10:18.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:18 smithi117 ceph-mon[32438]: from='mgr.? 172.21.15.194:0/1165973498' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-24T08:10:18.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:18 smithi117 ceph-mon[32438]: Standby manager daemon z restarted 2024-04-24T08:10:18.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:18 smithi117 ceph-mon[32438]: Standby manager daemon z started 2024-04-24T08:10:18.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:18 smithi117 ceph-mon[32438]: from='mgr.? 172.21.15.194:0/1165973498' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:10:18.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:18 smithi117 ceph-mon[32438]: from='mgr.? 172.21.15.194:0/1165973498' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-24T08:10:18.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:18 smithi117 ceph-mon[32438]: from='mgr.? 172.21.15.194:0/1165973498' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:10:18.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:18 smithi028 ceph-mon[27786]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-24T08:10:18.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:18 smithi028 ceph-mon[27786]: from='mgr.? 172.21.15.194:0/1165973498' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-24T08:10:18.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:18 smithi028 ceph-mon[27786]: Standby manager daemon z restarted 2024-04-24T08:10:18.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:18 smithi028 ceph-mon[27786]: Standby manager daemon z started 2024-04-24T08:10:18.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:18 smithi028 ceph-mon[27786]: from='mgr.? 172.21.15.194:0/1165973498' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:10:18.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:18 smithi028 ceph-mon[27786]: from='mgr.? 172.21.15.194:0/1165973498' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-24T08:10:18.655 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:18 smithi028 ceph-mon[27786]: from='mgr.? 172.21.15.194:0/1165973498' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:10:18.656 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:18 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:18.435+0000 7f8c3b8e2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-04-24T08:10:18.656 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:18 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:18.597+0000 7f8c3b8e2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-04-24T08:10:18.673 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-24T08:10:18.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:18 smithi194 ceph-mon[32836]: mgrmap e20: x(active, since 10m), standbys: z, y 2024-04-24T08:10:18.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:18 smithi194 ceph-mon[32836]: from='mgr.? 172.21.15.194:0/1165973498' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-04-24T08:10:18.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:18 smithi194 ceph-mon[32836]: Standby manager daemon z restarted 2024-04-24T08:10:18.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:18 smithi194 ceph-mon[32836]: Standby manager daemon z started 2024-04-24T08:10:18.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:18 smithi194 ceph-mon[32836]: from='mgr.? 172.21.15.194:0/1165973498' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-04-24T08:10:18.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:18 smithi194 ceph-mon[32836]: from='mgr.? 172.21.15.194:0/1165973498' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-04-24T08:10:18.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:18 smithi194 ceph-mon[32836]: from='mgr.? 172.21.15.194:0/1165973498' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-04-24T08:10:18.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-24T08:10:18.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:10:18.785 INFO:teuthology.orchestra.run.smithi028.stdout:client25246 2024-04-24T08:10:18.786 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T08:10:18.786 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:10:18.835 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-24T08:10:18.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:18 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-24T08:10:18.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:18 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:10:19.002 INFO:teuthology.orchestra.run.smithi028.stdout:client25246 2024-04-24T08:10:19.002 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-04-24T08:10:19.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-04-24T08:10:19.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:10:19.119 INFO:teuthology.orchestra.run.smithi028.stdout:client25246 2024-04-24T08:10:19.120 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/473aaf58-0210-11ef-bc93-c7b262605968.client25246/status') 2024-04-24T08:10:19.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-04-24T08:10:19.243 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:18 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:18.981+0000 7f8c3b8e2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-04-24T08:10:19.251 INFO:teuthology.orchestra.run.smithi028.stdout:instance: client.25246 (3)172.21.15.28:0/173581649 2024-04-24T08:10:19.251 INFO:teuthology.orchestra.run.smithi028.stdout:blocklisted: false 2024-04-24T08:10:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records in 2024-04-24T08:10:19.251 INFO:teuthology.orchestra.run.smithi028.stderr:0+1 records out 2024-04-24T08:10:19.252 INFO:teuthology.orchestra.run.smithi028.stderr:70 bytes copied, 0.000102499 s, 683 kB/s 2024-04-24T08:10:19.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-04-24T08:10:19.264 INFO:teuthology.run_tasks:Running task exec... 2024-04-24T08:10:19.274 INFO:teuthology.task.exec:Executing custom commands... 2024-04-24T08:10:19.274 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-24T08:10:19.274 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-04-24T08:10:19.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.456+0000 7f1ca1400640 1 -- 172.21.15.28:0/3265085305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c1039b0 msgr2=0x7f1c9c10a2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.456+0000 7f1ca1400640 1 --2- 172.21.15.28:0/3265085305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c1039b0 0x7f1c9c10a2d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f1c94009ca0 tx=0x7f1c9402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.456+0000 7f1ca1400640 1 -- 172.21.15.28:0/3265085305 shutdown_connections 2024-04-24T08:10:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.456+0000 7f1ca1400640 1 --2- 172.21.15.28:0/3265085305 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c1039b0 0x7f1c9c10a2d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.456+0000 7f1ca1400640 1 --2- 172.21.15.28:0/3265085305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c9c102ff0 0x7f1c9c103470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.456+0000 7f1ca1400640 1 --2- 172.21.15.28:0/3265085305 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c9c101ea0 0x7f1c9c1022c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.456+0000 7f1ca1400640 1 -- 172.21.15.28:0/3265085305 >> 172.21.15.28:0/3265085305 conn(0x7f1c9c0fd410 msgr2=0x7f1c9c0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.456+0000 7f1ca1400640 1 -- 172.21.15.28:0/3265085305 shutdown_connections 2024-04-24T08:10:19.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.456+0000 7f1ca1400640 1 -- 172.21.15.28:0/3265085305 wait complete. 2024-04-24T08:10:19.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.457+0000 7f1ca1400640 1 Processor -- start 2024-04-24T08:10:19.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.457+0000 7f1ca1400640 1 -- start start 2024-04-24T08:10:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.457+0000 7f1ca1400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c9c101ea0 0x7f1c9c075db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.457+0000 7f1ca1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c102ff0 0x7f1c9c0743f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.457+0000 7f1ca1400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c9c1039b0 0x7f1c9c074930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.457+0000 7f1ca1400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c10dba0 con 0x7f1c9c102ff0 2024-04-24T08:10:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.457+0000 7f1ca1400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c10da20 con 0x7f1c9c1039b0 2024-04-24T08:10:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.457+0000 7f1ca1400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1c9c10dd20 con 0x7f1c9c101ea0 2024-04-24T08:10:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.458+0000 7f1c9b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c102ff0 0x7f1c9c0743f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.458+0000 7f1c9b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c102ff0 0x7f1c9c0743f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:41504/0 (socket says 172.21.15.28:41504) 2024-04-24T08:10:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.458+0000 7f1c9b400640 1 -- 172.21.15.28:0/1839044973 learned_addr learned my addr 172.21.15.28:0/1839044973 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.458+0000 7f1c9be00640 1 --2- 172.21.15.28:0/1839044973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c9c101ea0 0x7f1c9c075db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.458+0000 7f1ca0a00640 1 --2- 172.21.15.28:0/1839044973 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c9c1039b0 0x7f1c9c074930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.458+0000 7f1c9b400640 1 -- 172.21.15.28:0/1839044973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c9c101ea0 msgr2=0x7f1c9c075db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.458+0000 7f1c9b400640 1 --2- 172.21.15.28:0/1839044973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c9c101ea0 0x7f1c9c075db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.458+0000 7f1c9b400640 1 -- 172.21.15.28:0/1839044973 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c9c1039b0 msgr2=0x7f1c9c074930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.458+0000 7f1c9b400640 1 --2- 172.21.15.28:0/1839044973 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c9c1039b0 0x7f1c9c074930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.458+0000 7f1c9b400640 1 -- 172.21.15.28:0/1839044973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1c9c0751c0 con 0x7f1c9c102ff0 2024-04-24T08:10:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.459+0000 7f1c9b400640 1 --2- 172.21.15.28:0/1839044973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c102ff0 0x7f1c9c0743f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1c8800e9c0 tx=0x7f1c8800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.459+0000 7f1c98c00640 1 -- 172.21.15.28:0/1839044973 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c8800cde0 con 0x7f1c9c102ff0 2024-04-24T08:10:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.459+0000 7f1c98c00640 1 -- 172.21.15.28:0/1839044973 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1c88010910 con 0x7f1c9c102ff0 2024-04-24T08:10:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.459+0000 7f1c98c00640 1 -- 172.21.15.28:0/1839044973 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1c880189c0 con 0x7f1c9c102ff0 2024-04-24T08:10:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.459+0000 7f1ca1400640 1 -- 172.21.15.28:0/1839044973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1c9c1a2240 con 0x7f1c9c102ff0 2024-04-24T08:10:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.459+0000 7f1ca1400640 1 -- 172.21.15.28:0/1839044973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1c9c1a2740 con 0x7f1c9c102ff0 2024-04-24T08:10:19.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.460+0000 7f1ca1400640 1 -- 172.21.15.28:0/1839044973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1c9c1090e0 con 0x7f1c9c102ff0 2024-04-24T08:10:19.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.463+0000 7f1c98c00640 1 -- 172.21.15.28:0/1839044973 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 21) ==== 138663+0+0 (secure 0 0 0) 0x7f1c88010ac0 con 0x7f1c9c102ff0 2024-04-24T08:10:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.464+0000 7f1c98c00640 1 --2- 172.21.15.28:0/1839044973 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1c800a3bf0 0x7f1c800a60b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.465+0000 7f1c9be00640 1 -- 172.21.15.28:0/1839044973 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1c800a3bf0 msgr2=0x7f1c800a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/361125225 2024-04-24T08:10:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.465+0000 7f1c98c00640 1 -- 172.21.15.28:0/1839044973 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(90..90 src has 1..90) ==== 8298+0+0 (secure 0 0 0) 0x7f1c880d0a20 con 0x7f1c9c102ff0 2024-04-24T08:10:19.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.465+0000 7f1c9be00640 1 --2- 172.21.15.28:0/1839044973 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1c800a3bf0 0x7f1c800a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-04-24T08:10:19.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.474+0000 7f1c98c00640 1 -- 172.21.15.28:0/1839044973 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1c88053ff0 con 0x7f1c9c102ff0 2024-04-24T08:10:19.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:19 smithi028 ceph-mon[27786]: mgrmap e21: x(active, since 10m), standbys: y, z 2024-04-24T08:10:19.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:19 smithi117 ceph-mon[32438]: mgrmap e21: x(active, since 10m), standbys: y, z 2024-04-24T08:10:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.666+0000 7f1c9be00640 1 -- 172.21.15.28:0/1839044973 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1c800a3bf0 msgr2=0x7f1c800a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/361125225 2024-04-24T08:10:19.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.666+0000 7f1c9be00640 1 --2- 172.21.15.28:0/1839044973 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1c800a3bf0 0x7f1c800a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-04-24T08:10:19.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:19 smithi194 ceph-mon[32836]: mgrmap e21: x(active, since 10m), standbys: y, z 2024-04-24T08:10:19.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:19.713+0000 7f1ca1400640 1 -- 172.21.15.28:0/1839044973 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) -- 0x7f1c9c062c00 con 0x7f1c9c102ff0 2024-04-24T08:10:20.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.067+0000 7f1c9be00640 1 -- 172.21.15.28:0/1839044973 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1c800a3bf0 msgr2=0x7f1c800a60b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.28:6800/361125225 2024-04-24T08:10:20.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.067+0000 7f1c9be00640 1 --2- 172.21.15.28:0/1839044973 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1c800a3bf0 0x7f1c800a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-04-24T08:10:20.111 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:19 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:19.810+0000 7f8c3b8e2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-04-24T08:10:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.135+0000 7f1c98c00640 1 -- 172.21.15.28:0/1839044973 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 22) ==== 138331+0+0 (secure 0 0 0) 0x7f1c880992d0 con 0x7f1c9c102ff0 2024-04-24T08:10:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.135+0000 7f1c98c00640 1 -- 172.21.15.28:0/1839044973 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1c800a3bf0 msgr2=0x7f1c800a60b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:10:20.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.135+0000 7f1c98c00640 1 --2- 172.21.15.28:0/1839044973 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1c800a3bf0 0x7f1c800a60b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.362 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1839044973' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-24T08:10:20.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: Active manager daemon x restarted 2024-04-24T08:10:20.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: Activating manager daemon x 2024-04-24T08:10:20.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: osdmap e91: 12 total, 12 up, 12 in 2024-04-24T08:10:20.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: mgrmap e22: x(active, starting, since 0.0165374s), standbys: y, z 2024-04-24T08:10:20.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:10:20.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:10:20.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:10:20.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-24T08:10:20.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T08:10:20.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T08:10:20.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T08:10:20.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-24T08:10:20.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T08:10:20.363 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:10:20.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T08:10:20.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:10:20.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:10:20.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:10:20.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:10:20.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T08:10:20.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T08:10:20.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-24T08:10:20.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:10:20.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:10:20.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:10:20.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:10:20.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:10:20.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:10:20.364 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:10:20.365 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:10:20.365 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:10:20.365 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:20.110+0000 7f8c3b8e2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-04-24T08:10:20.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.366+0000 7f1c98c00640 1 -- 172.21.15.28:0/1839044973 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v24) ==== 118+0+0 (secure 0 0 0) 0x7f1c880bbe20 con 0x7f1c9c102ff0 2024-04-24T08:10:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.370+0000 7f1ca1400640 1 -- 172.21.15.28:0/1839044973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c102ff0 msgr2=0x7f1c9c0743f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.370+0000 7f1ca1400640 1 --2- 172.21.15.28:0/1839044973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c102ff0 0x7f1c9c0743f0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f1c8800e9c0 tx=0x7f1c8800ee80 comp rx=0 tx=0).stop 2024-04-24T08:10:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.370+0000 7f1ca1400640 1 -- 172.21.15.28:0/1839044973 shutdown_connections 2024-04-24T08:10:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.370+0000 7f1ca1400640 1 --2- 172.21.15.28:0/1839044973 >> [v2:172.21.15.28:6800/361125225,v1:172.21.15.28:6801/361125225] conn(0x7f1c800a3bf0 0x7f1c800a60b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.370+0000 7f1ca1400640 1 --2- 172.21.15.28:0/1839044973 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1c9c102ff0 0x7f1c9c0743f0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.370+0000 7f1ca1400640 1 --2- 172.21.15.28:0/1839044973 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c9c1039b0 0x7f1c9c074930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.370+0000 7f1ca1400640 1 --2- 172.21.15.28:0/1839044973 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1c9c101ea0 0x7f1c9c075db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.370+0000 7f1ca1400640 1 -- 172.21.15.28:0/1839044973 >> 172.21.15.28:0/1839044973 conn(0x7f1c9c0fd410 msgr2=0x7f1c9c1062b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.371+0000 7f1ca1400640 1 -- 172.21.15.28:0/1839044973 shutdown_connections 2024-04-24T08:10:20.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.371+0000 7f1ca1400640 1 -- 172.21.15.28:0/1839044973 wait complete. 2024-04-24T08:10:20.395 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-04-24T08:10:20.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.607+0000 7f3993000640 1 -- 172.21.15.28:0/1573378941 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f398c076040 msgr2=0x7f398c074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.607+0000 7f3993000640 1 --2- 172.21.15.28:0/1573378941 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f398c076040 0x7f398c074490 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3980009900 tx=0x7f398002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.607+0000 7f3993000640 1 -- 172.21.15.28:0/1573378941 shutdown_connections 2024-04-24T08:10:20.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.607+0000 7f3993000640 1 --2- 172.21.15.28:0/1573378941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f398c109d50 0x7f398c10c1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.607+0000 7f3993000640 1 --2- 172.21.15.28:0/1573378941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f398c074a60 0x7f398c074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.607+0000 7f3993000640 1 --2- 172.21.15.28:0/1573378941 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f398c076040 0x7f398c074490 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.607+0000 7f3993000640 1 -- 172.21.15.28:0/1573378941 >> 172.21.15.28:0/1573378941 conn(0x7f398c0fd400 msgr2=0x7f398c0ff840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:20.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.607+0000 7f3993000640 1 -- 172.21.15.28:0/1573378941 shutdown_connections 2024-04-24T08:10:20.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.607+0000 7f3993000640 1 -- 172.21.15.28:0/1573378941 wait complete. 2024-04-24T08:10:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.609+0000 7f3993000640 1 Processor -- start 2024-04-24T08:10:20.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.609+0000 7f3993000640 1 -- start start 2024-04-24T08:10:20.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.609+0000 7f3993000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f398c074a60 0x7f398c19a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:20.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.609+0000 7f3993000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f398c076040 0x7f398c19a890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:20.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.609+0000 7f3993000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f398c109d50 0x7f398c1a1900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:20.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.609+0000 7f3993000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f398c10fcd0 con 0x7f398c074a60 2024-04-24T08:10:20.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.609+0000 7f3993000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f398c10fb50 con 0x7f398c109d50 2024-04-24T08:10:20.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.609+0000 7f3993000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f398c10fe50 con 0x7f398c076040 2024-04-24T08:10:20.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.610+0000 7f3991c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f398c074a60 0x7f398c19a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:20.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.610+0000 7f3992600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f398c109d50 0x7f398c1a1900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:20.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.610+0000 7f3991200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f398c076040 0x7f398c19a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:20.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.610+0000 7f3991c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f398c074a60 0x7f398c19a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33670/0 (socket says 172.21.15.28:33670) 2024-04-24T08:10:20.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.610+0000 7f3991200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f398c076040 0x7f398c19a890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:51564/0 (socket says 172.21.15.28:51564) 2024-04-24T08:10:20.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.610+0000 7f3991c00640 1 -- 172.21.15.28:0/2299333652 learned_addr learned my addr 172.21.15.28:0/2299333652 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:20.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.610+0000 7f3992600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f398c109d50 0x7f398c1a1900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:37856/0 (socket says 172.21.15.28:37856) 2024-04-24T08:10:20.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.610+0000 7f3992600640 1 -- 172.21.15.28:0/2299333652 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f398c076040 msgr2=0x7f398c19a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.610+0000 7f3992600640 1 --2- 172.21.15.28:0/2299333652 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f398c076040 0x7f398c19a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.610+0000 7f3992600640 1 -- 172.21.15.28:0/2299333652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f398c074a60 msgr2=0x7f398c19a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:20.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.610+0000 7f3992600640 1 --2- 172.21.15.28:0/2299333652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f398c074a60 0x7f398c19a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:20.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.610+0000 7f3992600640 1 -- 172.21.15.28:0/2299333652 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f398c1a1f70 con 0x7f398c109d50 2024-04-24T08:10:20.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.610+0000 7f3992600640 1 --2- 172.21.15.28:0/2299333652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f398c109d50 0x7f398c1a1900 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3984009fd0 tx=0x7f398400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:20.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.611+0000 7f398a000640 1 -- 172.21.15.28:0/2299333652 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f398400ce20 con 0x7f398c109d50 2024-04-24T08:10:20.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.611+0000 7f398a000640 1 -- 172.21.15.28:0/2299333652 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3984005020 con 0x7f398c109d50 2024-04-24T08:10:20.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.611+0000 7f3993000640 1 -- 172.21.15.28:0/2299333652 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f398c1a2260 con 0x7f398c109d50 2024-04-24T08:10:20.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.611+0000 7f398a000640 1 -- 172.21.15.28:0/2299333652 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3984010720 con 0x7f398c109d50 2024-04-24T08:10:20.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.611+0000 7f3993000640 1 -- 172.21.15.28:0/2299333652 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f398c1a2760 con 0x7f398c109d50 2024-04-24T08:10:20.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.612+0000 7f3993000640 1 -- 172.21.15.28:0/2299333652 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f398c108a80 con 0x7f398c109d50 2024-04-24T08:10:20.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.614+0000 7f398a000640 1 -- 172.21.15.28:0/2299333652 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 22) ==== 138331+0+0 (secure 0 0 0) 0x7f3984005540 con 0x7f398c109d50 2024-04-24T08:10:20.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.615+0000 7f398a000640 1 -- 172.21.15.28:0/2299333652 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f39840d0710 con 0x7f398c109d50 2024-04-24T08:10:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:20.619+0000 7f398a000640 1 -- 172.21.15.28:0/2299333652 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f398409d350 con 0x7f398c109d50 2024-04-24T08:10:20.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1839044973' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-24T08:10:20.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: Active manager daemon x restarted 2024-04-24T08:10:20.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: Activating manager daemon x 2024-04-24T08:10:20.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: osdmap e91: 12 total, 12 up, 12 in 2024-04-24T08:10:20.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: mgrmap e22: x(active, starting, since 0.0165374s), standbys: y, z 2024-04-24T08:10:20.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:10:20.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:10:20.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:10:20.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-24T08:10:20.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T08:10:20.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T08:10:20.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T08:10:20.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-24T08:10:20.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T08:10:20.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:10:20.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T08:10:20.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:10:20.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:10:20.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:10:20.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:10:20.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T08:10:20.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T08:10:20.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-24T08:10:20.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:10:20.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:10:20.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:10:20.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:10:20.630 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:10:20.631 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:10:20.631 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:10:20.631 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:10:20.631 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:10:20.631 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:10:20.631 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:10:20.631 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:10:20.631 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T08:10:20.631 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T08:10:20.631 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T08:10:20.631 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: Manager daemon x is now available 2024-04-24T08:10:20.631 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:10:20.631 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T08:10:20.631 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T08:10:20.631 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T08:10:20.631 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:20 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1839044973' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-04-24T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: Active manager daemon x restarted 2024-04-24T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: Activating manager daemon x 2024-04-24T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: osdmap e91: 12 total, 12 up, 12 in 2024-04-24T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: mgrmap e22: x(active, starting, since 0.0165374s), standbys: y, z 2024-04-24T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-04-24T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-04-24T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-04-24T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-04-24T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-04-24T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-04-24T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-04-24T08:10:20.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-04-24T08:10:20.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-04-24T08:10:20.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:10:20.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T08:10:20.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:10:20.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:10:20.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:10:20.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:10:20.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-04-24T08:10:20.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-04-24T08:10:20.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-04-24T08:10:20.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-04-24T08:10:20.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-04-24T08:10:20.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-04-24T08:10:20.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-04-24T08:10:20.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-04-24T08:10:20.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-04-24T08:10:20.693 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-04-24T08:10:20.693 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-04-24T08:10:20.693 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-04-24T08:10:20.693 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:10:20.693 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:10:20.693 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:10:20.693 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T08:10:20.693 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T08:10:20.693 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T08:10:20.693 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: Manager daemon x is now available 2024-04-24T08:10:20.693 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:10:20.693 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T08:10:20.693 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T08:10:20.693 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T08:10:20.694 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:20 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T08:10:20.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-04-24T08:10:20.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-04-24T08:10:20.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-04-24T08:10:20.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-04-24T08:10:20.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-04-24T08:10:20.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-04-24T08:10:20.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: Manager daemon x is now available 2024-04-24T08:10:20.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:10:20.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T08:10:20.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-04-24T08:10:20.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T08:10:20.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-04-24T08:10:20.790 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:20.489+0000 7f8b8cc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:10:20.790 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:20.489+0000 7f8b8cc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:10:20.790 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:20.489+0000 7f8b8cc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:10:20.790 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:20.489+0000 7f8b8cc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:10:20.791 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:20.489+0000 7f8b8cc00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:10:20.791 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:20.489+0000 7f8b8fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:10:20.791 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:20.489+0000 7f8b8fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:10:20.791 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:20.489+0000 7f8b8fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:10:20.791 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:20.490+0000 7f8b8fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:10:20.791 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:20 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:20.490+0000 7f8b8fe00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:10:21.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.146+0000 7f398a000640 1 -- 172.21.15.28:0/2299333652 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7f39840989a0 con 0x7f398c109d50 2024-04-24T08:10:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.149+0000 7f398a000640 1 --2- 172.21.15.28:0/2299333652 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f39740aaa20 0x7f39740ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.149+0000 7f398a000640 1 -- 172.21.15.28:0/2299333652 --> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) -- 0x7f3984017ec0 con 0x7f39740aaa20 2024-04-24T08:10:21.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.152+0000 7f3991c00640 1 --2- 172.21.15.28:0/2299333652 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f39740aaa20 0x7f39740ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.164+0000 7f3991c00640 1 --2- 172.21.15.28:0/2299333652 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f39740aaa20 0x7f39740ace10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3980031440 tx=0x7f39800315b0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:21.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.194+0000 7f398a000640 1 -- 172.21.15.28:0/2299333652 <== mgr.25225 v2:172.21.15.28:6800/353036813 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+3 (secure 0 0 0) 0x7f3984017ec0 con 0x7f39740aaa20 2024-04-24T08:10:21.195 INFO:teuthology.orchestra.run.smithi028.stdout:{} 2024-04-24T08:10:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.197+0000 7f3993000640 1 -- 172.21.15.28:0/2299333652 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f39740aaa20 msgr2=0x7f39740ace10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.197+0000 7f3993000640 1 --2- 172.21.15.28:0/2299333652 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f39740aaa20 0x7f39740ace10 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f3980031440 tx=0x7f39800315b0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.197+0000 7f3993000640 1 -- 172.21.15.28:0/2299333652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f398c109d50 msgr2=0x7f398c1a1900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.197+0000 7f3993000640 1 --2- 172.21.15.28:0/2299333652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f398c109d50 0x7f398c1a1900 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3984009fd0 tx=0x7f398400ef10 comp rx=0 tx=0).stop 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.197+0000 7f3993000640 1 -- 172.21.15.28:0/2299333652 shutdown_connections 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.197+0000 7f3993000640 1 --2- 172.21.15.28:0/2299333652 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f39740aaa20 0x7f39740ace10 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.197+0000 7f3993000640 1 --2- 172.21.15.28:0/2299333652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f398c109d50 0x7f398c1a1900 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.197+0000 7f3993000640 1 --2- 172.21.15.28:0/2299333652 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f398c076040 0x7f398c19a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.197+0000 7f3993000640 1 --2- 172.21.15.28:0/2299333652 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f398c074a60 0x7f398c19a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.197+0000 7f3993000640 1 -- 172.21.15.28:0/2299333652 >> 172.21.15.28:0/2299333652 conn(0x7f398c0fd400 msgr2=0x7f398c105c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.197+0000 7f3993000640 1 -- 172.21.15.28:0/2299333652 shutdown_connections 2024-04-24T08:10:21.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.197+0000 7f3993000640 1 -- 172.21.15.28:0/2299333652 wait complete. 2024-04-24T08:10:21.213 INFO:teuthology.run_tasks:Running task check-counter... 2024-04-24T08:10:21.224 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-04-24T08:10:21.277 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:10:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.470+0000 7f9d50a00640 1 -- 172.21.15.28:0/1813347160 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d4c101ea0 msgr2=0x7f9d4c1022c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.470+0000 7f9d50a00640 1 --2- 172.21.15.28:0/1813347160 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d4c101ea0 0x7f9d4c1022c0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c009900 tx=0x7f9d3c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.470+0000 7f9d50a00640 1 -- 172.21.15.28:0/1813347160 shutdown_connections 2024-04-24T08:10:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.470+0000 7f9d50a00640 1 --2- 172.21.15.28:0/1813347160 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4c1039b0 0x7f9d4c10a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.470+0000 7f9d50a00640 1 --2- 172.21.15.28:0/1813347160 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d4c102ff0 0x7f9d4c103470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.470+0000 7f9d50a00640 1 --2- 172.21.15.28:0/1813347160 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d4c101ea0 0x7f9d4c1022c0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.470+0000 7f9d50a00640 1 -- 172.21.15.28:0/1813347160 >> 172.21.15.28:0/1813347160 conn(0x7f9d4c0fd410 msgr2=0x7f9d4c0ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.470+0000 7f9d50a00640 1 -- 172.21.15.28:0/1813347160 shutdown_connections 2024-04-24T08:10:21.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.470+0000 7f9d50a00640 1 -- 172.21.15.28:0/1813347160 wait complete. 2024-04-24T08:10:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.472+0000 7f9d50a00640 1 Processor -- start 2024-04-24T08:10:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.472+0000 7f9d50a00640 1 -- start start 2024-04-24T08:10:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.472+0000 7f9d50a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4c101ea0 0x7f9d4c19a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.472+0000 7f9d50a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d4c102ff0 0x7f9d4c19a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.472+0000 7f9d50a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d4c1039b0 0x7f9d4c1a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.472+0000 7f9d50a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c10db40 con 0x7f9d4c101ea0 2024-04-24T08:10:21.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.472+0000 7f9d50a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c10d9c0 con 0x7f9d4c1039b0 2024-04-24T08:10:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.472+0000 7f9d50a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9d4c10dcc0 con 0x7f9d4c102ff0 2024-04-24T08:10:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.472+0000 7f9d49e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d4c102ff0 0x7f9d4c19a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.472+0000 7f9d4a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4c101ea0 0x7f9d4c19a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.473+0000 7f9d49e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d4c102ff0 0x7f9d4c19a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:51610/0 (socket says 172.21.15.28:51610) 2024-04-24T08:10:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.473+0000 7f9d49e00640 1 -- 172.21.15.28:0/1608827400 learned_addr learned my addr 172.21.15.28:0/1608827400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.473+0000 7f9d4b200640 1 --2- 172.21.15.28:0/1608827400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d4c1039b0 0x7f9d4c1a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.473+0000 7f9d49e00640 1 -- 172.21.15.28:0/1608827400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d4c1039b0 msgr2=0x7f9d4c1a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.473+0000 7f9d49e00640 1 --2- 172.21.15.28:0/1608827400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d4c1039b0 0x7f9d4c1a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.473+0000 7f9d49e00640 1 -- 172.21.15.28:0/1608827400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4c101ea0 msgr2=0x7f9d4c19a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.473+0000 7f9d49e00640 1 --2- 172.21.15.28:0/1608827400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4c101ea0 0x7f9d4c19a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.473+0000 7f9d49e00640 1 -- 172.21.15.28:0/1608827400 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d4c1a1fe0 con 0x7f9d4c102ff0 2024-04-24T08:10:21.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.473+0000 7f9d49e00640 1 --2- 172.21.15.28:0/1608827400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d4c102ff0 0x7f9d4c19a9a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800e9c0 tx=0x7f9d3800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.473+0000 7f9d43400640 1 -- 172.21.15.28:0/1608827400 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d3800cd80 con 0x7f9d4c102ff0 2024-04-24T08:10:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.473+0000 7f9d43400640 1 -- 172.21.15.28:0/1608827400 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d380035f0 con 0x7f9d4c102ff0 2024-04-24T08:10:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.474+0000 7f9d43400640 1 -- 172.21.15.28:0/1608827400 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d38010690 con 0x7f9d4c102ff0 2024-04-24T08:10:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.474+0000 7f9d50a00640 1 -- 172.21.15.28:0/1608827400 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d4c1a22d0 con 0x7f9d4c102ff0 2024-04-24T08:10:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.474+0000 7f9d50a00640 1 -- 172.21.15.28:0/1608827400 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d4c1a27d0 con 0x7f9d4c102ff0 2024-04-24T08:10:21.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.475+0000 7f9d50a00640 1 -- 172.21.15.28:0/1608827400 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d4c1a2a90 con 0x7f9d4c102ff0 2024-04-24T08:10:21.477 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.477+0000 7f9d43400640 1 -- 172.21.15.28:0/1608827400 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7f9d380037a0 con 0x7f9d4c102ff0 2024-04-24T08:10:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.478+0000 7f9d43400640 1 --2- 172.21.15.28:0/1608827400 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d300a3a30 0x7f9d300a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.479+0000 7f9d43400640 1 -- 172.21.15.28:0/1608827400 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f9d380d0460 con 0x7f9d4c102ff0 2024-04-24T08:10:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.479+0000 7f9d4a800640 1 --2- 172.21.15.28:0/1608827400 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d300a3a30 0x7f9d300a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.480+0000 7f9d4a800640 1 --2- 172.21.15.28:0/1608827400 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d300a3a30 0x7f9d300a5ef0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c0314e0 tx=0x7f9d3c000bc0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:21.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.482+0000 7f9d43400640 1 -- 172.21.15.28:0/1608827400 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9d3809cfb0 con 0x7f9d4c102ff0 2024-04-24T08:10:21.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:21 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1839044973' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-24T08:10:21.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:21 smithi028 ceph-mon[27786]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:10:21.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:21 smithi028 ceph-mon[27786]: mgrmap e23: x(active, since 1.03553s), standbys: y, z 2024-04-24T08:10:21.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:21 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:21.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:21 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:21.539 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:21.178+0000 7f8b8a400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:10:21.539 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:21.178+0000 7f8b8a400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:10:21.539 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:21.178+0000 7f8b8a400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:10:21.540 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:21.178+0000 7f8b8a400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:10:21.540 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:10:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:10:21.178+0000 7f8b8a400700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:10:21.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:21 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1839044973' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-24T08:10:21.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:21 smithi117 ceph-mon[32438]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:10:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:21 smithi117 ceph-mon[32438]: mgrmap e23: x(active, since 1.03553s), standbys: y, z 2024-04-24T08:10:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:21 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:21.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:21 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:21.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.690+0000 7f9d50a00640 1 -- 172.21.15.28:0/1608827400 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9d4c1a2c60 con 0x7f9d4c102ff0 2024-04-24T08:10:21.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:21 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1839044973' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-04-24T08:10:21.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:21 smithi194 ceph-mon[32836]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:10:21.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:21 smithi194 ceph-mon[32836]: mgrmap e23: x(active, since 1.03553s), standbys: y, z 2024-04-24T08:10:21.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:21 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:21.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:21 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:21.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.692+0000 7f9d43400640 1 -- 172.21.15.28:0/1608827400 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23099 (secure 0 0 0) 0x7f9d3809c8b0 con 0x7f9d4c102ff0 2024-04-24T08:10:21.692 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:10:21.693 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:10:20.116137+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1819419617":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6801/361125225":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/3758227512":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/874707228":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/2660639233":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/361125225":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:10:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.694+0000 7f9d50a00640 1 -- 172.21.15.28:0/1608827400 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d300a3a30 msgr2=0x7f9d300a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.694+0000 7f9d50a00640 1 --2- 172.21.15.28:0/1608827400 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d300a3a30 0x7f9d300a5ef0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f9d3c0314e0 tx=0x7f9d3c000bc0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.694+0000 7f9d50a00640 1 -- 172.21.15.28:0/1608827400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d4c102ff0 msgr2=0x7f9d4c19a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.694+0000 7f9d50a00640 1 --2- 172.21.15.28:0/1608827400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d4c102ff0 0x7f9d4c19a9a0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9d3800e9c0 tx=0x7f9d3800ee80 comp rx=0 tx=0).stop 2024-04-24T08:10:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.694+0000 7f9d50a00640 1 -- 172.21.15.28:0/1608827400 shutdown_connections 2024-04-24T08:10:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.694+0000 7f9d50a00640 1 --2- 172.21.15.28:0/1608827400 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d300a3a30 0x7f9d300a5ef0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.694+0000 7f9d50a00640 1 --2- 172.21.15.28:0/1608827400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d4c1039b0 0x7f9d4c1a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.694+0000 7f9d50a00640 1 --2- 172.21.15.28:0/1608827400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d4c102ff0 0x7f9d4c19a9a0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.694+0000 7f9d50a00640 1 --2- 172.21.15.28:0/1608827400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d4c101ea0 0x7f9d4c19a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.694+0000 7f9d50a00640 1 -- 172.21.15.28:0/1608827400 >> 172.21.15.28:0/1608827400 conn(0x7f9d4c0fd410 msgr2=0x7f9d4c074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.694+0000 7f9d50a00640 1 -- 172.21.15.28:0/1608827400 shutdown_connections 2024-04-24T08:10:21.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.694+0000 7f9d50a00640 1 -- 172.21.15.28:0/1608827400 wait complete. 2024-04-24T08:10:21.713 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:02:34.086371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:07:49.819536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:07:50.493486+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:10:21.713 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:10:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.910+0000 7f114d800640 1 -- 172.21.15.28:0/1579641759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1148102a30 msgr2=0x7f1148104e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.910+0000 7f114d800640 1 --2- 172.21.15.28:0/1579641759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1148102a30 0x7f1148104e20 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1138009900 tx=0x7f113802a410 comp rx=0 tx=0).stop 2024-04-24T08:10:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.910+0000 7f114d800640 1 -- 172.21.15.28:0/1579641759 shutdown_connections 2024-04-24T08:10:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.910+0000 7f114d800640 1 --2- 172.21.15.28:0/1579641759 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1148107c90 0x7f114810a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.910+0000 7f114d800640 1 --2- 172.21.15.28:0/1579641759 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1148105360 0x7f1148107750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.910+0000 7f114d800640 1 --2- 172.21.15.28:0/1579641759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1148102a30 0x7f1148104e20 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.910+0000 7f114d800640 1 -- 172.21.15.28:0/1579641759 >> 172.21.15.28:0/1579641759 conn(0x7f11480fc3a0 msgr2=0x7f11480fe7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.910+0000 7f114d800640 1 -- 172.21.15.28:0/1579641759 shutdown_connections 2024-04-24T08:10:21.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.910+0000 7f114d800640 1 -- 172.21.15.28:0/1579641759 wait complete. 2024-04-24T08:10:21.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.911+0000 7f114d800640 1 Processor -- start 2024-04-24T08:10:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.912+0000 7f114d800640 1 -- start start 2024-04-24T08:10:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.912+0000 7f114d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1148102a30 0x7f11481023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.912+0000 7f114d800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1148105360 0x7f11481009e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.912+0000 7f114d800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1148107c90 0x7f1148100f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.912+0000 7f114d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f114810dbf0 con 0x7f1148102a30 2024-04-24T08:10:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.912+0000 7f114d800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f114810da70 con 0x7f1148105360 2024-04-24T08:10:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.912+0000 7f114d800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f114810dd70 con 0x7f1148107c90 2024-04-24T08:10:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.912+0000 7f1147e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1148102a30 0x7f11481023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.912+0000 7f1147400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1148105360 0x7f11481009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.912+0000 7f1147400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1148105360 0x7f11481009e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:37918/0 (socket says 172.21.15.28:37918) 2024-04-24T08:10:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.912+0000 7f114ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1148107c90 0x7f1148100f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.912+0000 7f1147400640 1 -- 172.21.15.28:0/1039161795 learned_addr learned my addr 172.21.15.28:0/1039161795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:21.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.912+0000 7f1147e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1148102a30 0x7f11481023a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33742/0 (socket says 172.21.15.28:33742) 2024-04-24T08:10:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.913+0000 7f1147400640 1 -- 172.21.15.28:0/1039161795 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1148107c90 msgr2=0x7f1148100f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.913+0000 7f1147400640 1 --2- 172.21.15.28:0/1039161795 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1148107c90 0x7f1148100f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.913+0000 7f1147400640 1 -- 172.21.15.28:0/1039161795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1148102a30 msgr2=0x7f11481023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.913+0000 7f1147400640 1 --2- 172.21.15.28:0/1039161795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1148102a30 0x7f11481023a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.913+0000 7f1147400640 1 -- 172.21.15.28:0/1039161795 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11481017b0 con 0x7f1148105360 2024-04-24T08:10:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.913+0000 7f1147400640 1 --2- 172.21.15.28:0/1039161795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1148105360 0x7f11481009e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f113400e9f0 tx=0x7f113400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.913+0000 7f1144c00640 1 -- 172.21.15.28:0/1039161795 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f113400ccc0 con 0x7f1148105360 2024-04-24T08:10:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.913+0000 7f114d800640 1 -- 172.21.15.28:0/1039161795 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11481a2380 con 0x7f1148105360 2024-04-24T08:10:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.913+0000 7f1144c00640 1 -- 172.21.15.28:0/1039161795 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1134005500 con 0x7f1148105360 2024-04-24T08:10:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.913+0000 7f1144c00640 1 -- 172.21.15.28:0/1039161795 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1134010690 con 0x7f1148105360 2024-04-24T08:10:21.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.913+0000 7f114d800640 1 -- 172.21.15.28:0/1039161795 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11481a28d0 con 0x7f1148105360 2024-04-24T08:10:21.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.915+0000 7f114d800640 1 -- 172.21.15.28:0/1039161795 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f11480fde90 con 0x7f1148105360 2024-04-24T08:10:21.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.917+0000 7f1144c00640 1 -- 172.21.15.28:0/1039161795 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7f11340046c0 con 0x7f1148105360 2024-04-24T08:10:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.918+0000 7f1144c00640 1 --2- 172.21.15.28:0/1039161795 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f112c0a3a30 0x7f112c0a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.919+0000 7f1144c00640 1 -- 172.21.15.28:0/1039161795 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f11340d04c0 con 0x7f1148105360 2024-04-24T08:10:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.919+0000 7f1147e00640 1 --2- 172.21.15.28:0/1039161795 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f112c0a3a30 0x7f112c0a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:21.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.920+0000 7f1147e00640 1 --2- 172.21.15.28:0/1039161795 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f112c0a3a30 0x7f112c0a5ef0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1138031440 tx=0x7f11380315b0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:21.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:21.923+0000 7f1144c00640 1 -- 172.21.15.28:0/1039161795 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f113409d010 con 0x7f1148105360 2024-04-24T08:10:22.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.133+0000 7f114d800640 1 -- 172.21.15.28:0/1039161795 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f1148062c00 con 0x7f1148105360 2024-04-24T08:10:22.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.133+0000 7f1144c00640 1 -- 172.21.15.28:0/1039161795 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f113409c910 con 0x7f1148105360 2024-04-24T08:10:22.134 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-24T08:10:22.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.137+0000 7f114d800640 1 -- 172.21.15.28:0/1039161795 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f112c0a3a30 msgr2=0x7f112c0a5ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.137+0000 7f114d800640 1 --2- 172.21.15.28:0/1039161795 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f112c0a3a30 0x7f112c0a5ef0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f1138031440 tx=0x7f11380315b0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.137+0000 7f114d800640 1 -- 172.21.15.28:0/1039161795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1148105360 msgr2=0x7f11481009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.137+0000 7f114d800640 1 --2- 172.21.15.28:0/1039161795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1148105360 0x7f11481009e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f113400e9f0 tx=0x7f113400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.137+0000 7f114d800640 1 -- 172.21.15.28:0/1039161795 shutdown_connections 2024-04-24T08:10:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.137+0000 7f114d800640 1 --2- 172.21.15.28:0/1039161795 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f112c0a3a30 0x7f112c0a5ef0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.137+0000 7f114d800640 1 --2- 172.21.15.28:0/1039161795 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1148107c90 0x7f1148100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.137+0000 7f114d800640 1 --2- 172.21.15.28:0/1039161795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1148105360 0x7f11481009e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.137+0000 7f114d800640 1 --2- 172.21.15.28:0/1039161795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1148102a30 0x7f11481023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.137+0000 7f114d800640 1 -- 172.21.15.28:0/1039161795 >> 172.21.15.28:0/1039161795 conn(0x7f11480fc3a0 msgr2=0x7f1148103430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.137+0000 7f114d800640 1 -- 172.21.15.28:0/1039161795 shutdown_connections 2024-04-24T08:10:22.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.137+0000 7f114d800640 1 -- 172.21.15.28:0/1039161795 wait complete. 2024-04-24T08:10:22.156 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:10:22.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.342+0000 7f783ac00640 1 -- 172.21.15.28:0/939859416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7834101220 msgr2=0x7f7834101680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.342+0000 7f783ac00640 1 --2- 172.21.15.28:0/939859416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7834101220 0x7f7834101680 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f7820009900 tx=0x7f782002a410 comp rx=0 tx=0).stop 2024-04-24T08:10:22.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.342+0000 7f783ac00640 1 -- 172.21.15.28:0/939859416 shutdown_connections 2024-04-24T08:10:22.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.342+0000 7f783ac00640 1 --2- 172.21.15.28:0/939859416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7834101220 0x7f7834101680 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.342+0000 7f783ac00640 1 --2- 172.21.15.28:0/939859416 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7834101bc0 0x7f7834102040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.342+0000 7f783ac00640 1 --2- 172.21.15.28:0/939859416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f78341008c0 0x7f7834100ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.342+0000 7f783ac00640 1 -- 172.21.15.28:0/939859416 >> 172.21.15.28:0/939859416 conn(0x7f78340fc4c0 msgr2=0x7f78340fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.342+0000 7f783ac00640 1 -- 172.21.15.28:0/939859416 shutdown_connections 2024-04-24T08:10:22.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.342+0000 7f783ac00640 1 -- 172.21.15.28:0/939859416 wait complete. 2024-04-24T08:10:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.343+0000 7f783ac00640 1 Processor -- start 2024-04-24T08:10:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.343+0000 7f783ac00640 1 -- start start 2024-04-24T08:10:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.343+0000 7f783ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78341008c0 0x7f783419a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.343+0000 7f783ac00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7834101220 0x7f783419aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.343+0000 7f783ac00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7834101bc0 0x7f78341a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.343+0000 7f783ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f783410fef0 con 0x7f78341008c0 2024-04-24T08:10:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.343+0000 7f783ac00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f783410fd70 con 0x7f7834101bc0 2024-04-24T08:10:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.343+0000 7f783ac00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7834110070 con 0x7f7834101220 2024-04-24T08:10:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.343+0000 7f7833e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78341008c0 0x7f783419a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.343+0000 7f7833e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78341008c0 0x7f783419a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33778/0 (socket says 172.21.15.28:33778) 2024-04-24T08:10:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.343+0000 7f7833e00640 1 -- 172.21.15.28:0/2767381748 learned_addr learned my addr 172.21.15.28:0/2767381748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.344+0000 7f7833400640 1 --2- 172.21.15.28:0/2767381748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7834101220 0x7f783419aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.344+0000 7f7838c00640 1 --2- 172.21.15.28:0/2767381748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7834101bc0 0x7f78341a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.344+0000 7f7833e00640 1 -- 172.21.15.28:0/2767381748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7834101220 msgr2=0x7f783419aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.344+0000 7f7833e00640 1 --2- 172.21.15.28:0/2767381748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7834101220 0x7f783419aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.344+0000 7f7833e00640 1 -- 172.21.15.28:0/2767381748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7834101bc0 msgr2=0x7f78341a1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.344+0000 7f7833e00640 1 --2- 172.21.15.28:0/2767381748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7834101bc0 0x7f78341a1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.344+0000 7f7833e00640 1 -- 172.21.15.28:0/2767381748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f78341a21b0 con 0x7f78341008c0 2024-04-24T08:10:22.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.344+0000 7f7833e00640 1 --2- 172.21.15.28:0/2767381748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78341008c0 0x7f783419a590 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f782400e9f0 tx=0x7f782400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:22.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.344+0000 7f7830c00640 1 -- 172.21.15.28:0/2767381748 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f782400ccc0 con 0x7f78341008c0 2024-04-24T08:10:22.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.345+0000 7f783ac00640 1 -- 172.21.15.28:0/2767381748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f78341a24a0 con 0x7f78341008c0 2024-04-24T08:10:22.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.345+0000 7f783ac00640 1 -- 172.21.15.28:0/2767381748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f78341a2a10 con 0x7f78341008c0 2024-04-24T08:10:22.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.345+0000 7f7830c00640 1 -- 172.21.15.28:0/2767381748 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7824010910 con 0x7f78341008c0 2024-04-24T08:10:22.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.345+0000 7f7830c00640 1 -- 172.21.15.28:0/2767381748 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f78240188a0 con 0x7f78341008c0 2024-04-24T08:10:22.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.346+0000 7f783ac00640 1 -- 172.21.15.28:0/2767381748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f78140052c0 con 0x7f78341008c0 2024-04-24T08:10:22.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.348+0000 7f7830c00640 1 -- 172.21.15.28:0/2767381748 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 23) ==== 138458+0+0 (secure 0 0 0) 0x7f7824010ac0 con 0x7f78341008c0 2024-04-24T08:10:22.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.349+0000 7f7830c00640 1 --2- 172.21.15.28:0/2767381748 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f78180a39b0 0x7f78180a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.350+0000 7f7830c00640 1 -- 172.21.15.28:0/2767381748 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f78240d09a0 con 0x7f78341008c0 2024-04-24T08:10:22.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.350+0000 7f7833400640 1 --2- 172.21.15.28:0/2767381748 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f78180a39b0 0x7f78180a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.351+0000 7f7833400640 1 --2- 172.21.15.28:0/2767381748 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f78180a39b0 0x7f78180a5e70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f78200314e0 tx=0x7f7820000bc0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:22.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.354+0000 7f7830c00640 1 -- 172.21.15.28:0/2767381748 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f782409d4f0 con 0x7f78341008c0 2024-04-24T08:10:22.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.538+0000 7f7830c00640 1 -- 172.21.15.28:0/2767381748 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mgrmap(e 24) ==== 138817+0+0 (secure 0 0 0) 0x7f7824099460 con 0x7f78341008c0 2024-04-24T08:10:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.586+0000 7f783ac00640 1 -- 172.21.15.28:0/2767381748 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7814005880 con 0x7f78341008c0 2024-04-24T08:10:22.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.587+0000 7f7830c00640 1 -- 172.21.15.28:0/2767381748 <== mon.0 v2:172.21.15.28:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f78240287c0 con 0x7f78341008c0 2024-04-24T08:10:22.587 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-24T08:10:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.589+0000 7f783ac00640 1 -- 172.21.15.28:0/2767381748 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f78180a39b0 msgr2=0x7f78180a5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.589+0000 7f783ac00640 1 --2- 172.21.15.28:0/2767381748 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f78180a39b0 0x7f78180a5e70 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f78200314e0 tx=0x7f7820000bc0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.589+0000 7f783ac00640 1 -- 172.21.15.28:0/2767381748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78341008c0 msgr2=0x7f783419a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.589+0000 7f783ac00640 1 --2- 172.21.15.28:0/2767381748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78341008c0 0x7f783419a590 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f782400e9f0 tx=0x7f782400edf0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.589+0000 7f783ac00640 1 -- 172.21.15.28:0/2767381748 shutdown_connections 2024-04-24T08:10:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.589+0000 7f783ac00640 1 --2- 172.21.15.28:0/2767381748 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f78180a39b0 0x7f78180a5e70 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.589+0000 7f783ac00640 1 --2- 172.21.15.28:0/2767381748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7834101bc0 0x7f78341a1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.589+0000 7f783ac00640 1 --2- 172.21.15.28:0/2767381748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7834101220 0x7f783419aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.589+0000 7f783ac00640 1 --2- 172.21.15.28:0/2767381748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f78341008c0 0x7f783419a590 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.590+0000 7f783ac00640 1 -- 172.21.15.28:0/2767381748 >> 172.21.15.28:0/2767381748 conn(0x7f78340fc4c0 msgr2=0x7f78340fdc70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.590+0000 7f783ac00640 1 -- 172.21.15.28:0/2767381748 shutdown_connections 2024-04-24T08:10:22.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.590+0000 7f783ac00640 1 -- 172.21.15.28:0/2767381748 wait complete. 2024-04-24T08:10:22.607 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:10:22.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:22 smithi028 ceph-mon[27786]: [24/Apr/2024:08:10:20] ENGINE Bus STARTING 2024-04-24T08:10:22.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:22 smithi028 ceph-mon[27786]: [24/Apr/2024:08:10:20] ENGINE Serving on https://172.21.15.28:7150 2024-04-24T08:10:22.780 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:22 smithi028 ceph-mon[27786]: [24/Apr/2024:08:10:21] ENGINE Serving on http://172.21.15.28:8765 2024-04-24T08:10:22.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:22 smithi028 ceph-mon[27786]: [24/Apr/2024:08:10:21] ENGINE Bus STARTED 2024-04-24T08:10:22.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:22 smithi028 ceph-mon[27786]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:22.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:22 smithi028 ceph-mon[27786]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:10:22.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:22 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:22.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:22 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:22.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:22 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1608827400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:22.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:22 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1039161795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:10:22.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:22 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:22.781 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:22 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:22.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.808+0000 7efcacc00640 1 -- 172.21.15.28:0/396944272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efca8103180 msgr2=0x7efca8103600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.808+0000 7efcacc00640 1 --2- 172.21.15.28:0/396944272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efca8103180 0x7efca8103600 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7efc94009900 tx=0x7efc9402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.808+0000 7efcacc00640 1 -- 172.21.15.28:0/396944272 shutdown_connections 2024-04-24T08:10:22.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.808+0000 7efcacc00640 1 --2- 172.21.15.28:0/396944272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efca8103b40 0x7efca810a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.808+0000 7efcacc00640 1 --2- 172.21.15.28:0/396944272 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efca8103180 0x7efca8103600 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.808+0000 7efcacc00640 1 --2- 172.21.15.28:0/396944272 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efca8101f90 0x7efca81023b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.808+0000 7efcacc00640 1 -- 172.21.15.28:0/396944272 >> 172.21.15.28:0/396944272 conn(0x7efca80fd500 msgr2=0x7efca80ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:22.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.808+0000 7efcacc00640 1 -- 172.21.15.28:0/396944272 shutdown_connections 2024-04-24T08:10:22.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.808+0000 7efcacc00640 1 -- 172.21.15.28:0/396944272 wait complete. 2024-04-24T08:10:22.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.809+0000 7efcacc00640 1 Processor -- start 2024-04-24T08:10:22.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.809+0000 7efcacc00640 1 -- start start 2024-04-24T08:10:22.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.810+0000 7efcacc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efca8101f90 0x7efca819a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.810+0000 7efcacc00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efca8103180 0x7efca819a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.810+0000 7efcacc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efca8103b40 0x7efca81a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.810+0000 7efcacc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efca810dc00 con 0x7efca8103b40 2024-04-24T08:10:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.810+0000 7efcacc00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efca810da80 con 0x7efca8103180 2024-04-24T08:10:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.810+0000 7efcacc00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efca810dd80 con 0x7efca8101f90 2024-04-24T08:10:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.810+0000 7efca7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efca8101f90 0x7efca819a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.810+0000 7efca6a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efca8103180 0x7efca819a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.810+0000 7efca7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efca8101f90 0x7efca819a420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:51668/0 (socket says 172.21.15.28:51668) 2024-04-24T08:10:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.810+0000 7efca7400640 1 -- 172.21.15.28:0/2108785316 learned_addr learned my addr 172.21.15.28:0/2108785316 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.810+0000 7efca7e00640 1 --2- 172.21.15.28:0/2108785316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efca8103b40 0x7efca81a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.811+0000 7efca7400640 1 -- 172.21.15.28:0/2108785316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efca8103180 msgr2=0x7efca819a960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.811+0000 7efca7400640 1 --2- 172.21.15.28:0/2108785316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efca8103180 0x7efca819a960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.811+0000 7efca7400640 1 -- 172.21.15.28:0/2108785316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efca8103b40 msgr2=0x7efca81a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.811+0000 7efca7400640 1 --2- 172.21.15.28:0/2108785316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efca8103b40 0x7efca81a19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:22.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.811+0000 7efca7400640 1 -- 172.21.15.28:0/2108785316 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efca81a2040 con 0x7efca8101f90 2024-04-24T08:10:22.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.811+0000 7efca6a00640 1 --2- 172.21.15.28:0/2108785316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efca8103180 0x7efca819a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:22.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.811+0000 7efca7400640 1 --2- 172.21.15.28:0/2108785316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efca8101f90 0x7efca819a420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efc9800c970 tx=0x7efc9800ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:22.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.811+0000 7efc9fe00640 1 -- 172.21.15.28:0/2108785316 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc98004220 con 0x7efca8101f90 2024-04-24T08:10:22.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.811+0000 7efcacc00640 1 -- 172.21.15.28:0/2108785316 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efca81a2330 con 0x7efca8101f90 2024-04-24T08:10:22.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.811+0000 7efc9fe00640 1 -- 172.21.15.28:0/2108785316 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc980043c0 con 0x7efca8101f90 2024-04-24T08:10:22.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.811+0000 7efc9fe00640 1 -- 172.21.15.28:0/2108785316 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc9800f450 con 0x7efca8101f90 2024-04-24T08:10:22.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.811+0000 7efcacc00640 1 -- 172.21.15.28:0/2108785316 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efca81a2850 con 0x7efca8101f90 2024-04-24T08:10:22.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.813+0000 7efcacc00640 1 -- 172.21.15.28:0/2108785316 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efca81a2b10 con 0x7efca8101f90 2024-04-24T08:10:22.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.815+0000 7efc9fe00640 1 -- 172.21.15.28:0/2108785316 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 24) ==== 138817+0+0 (secure 0 0 0) 0x7efc980079b0 con 0x7efca8101f90 2024-04-24T08:10:22.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.816+0000 7efc9fe00640 1 --2- 172.21.15.28:0/2108785316 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efc8c0a3db0 0x7efc8c0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:22.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.816+0000 7efc9fe00640 1 -- 172.21.15.28:0/2108785316 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7efc980d0ba0 con 0x7efca8101f90 2024-04-24T08:10:22.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.816+0000 7efca6a00640 1 --2- 172.21.15.28:0/2108785316 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efc8c0a3db0 0x7efc8c0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:22.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.817+0000 7efca6a00640 1 --2- 172.21.15.28:0/2108785316 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efc8c0a3db0 0x7efc8c0a6270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efc94031440 tx=0x7efc940315b0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:22.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:22.822+0000 7efc9fe00640 1 -- 172.21.15.28:0/2108785316 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7efc980d0e20 con 0x7efca8101f90 2024-04-24T08:10:22.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:22 smithi117 ceph-mon[32438]: [24/Apr/2024:08:10:20] ENGINE Bus STARTING 2024-04-24T08:10:22.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:22 smithi117 ceph-mon[32438]: [24/Apr/2024:08:10:20] ENGINE Serving on https://172.21.15.28:7150 2024-04-24T08:10:22.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:22 smithi117 ceph-mon[32438]: [24/Apr/2024:08:10:21] ENGINE Serving on http://172.21.15.28:8765 2024-04-24T08:10:22.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:22 smithi117 ceph-mon[32438]: [24/Apr/2024:08:10:21] ENGINE Bus STARTED 2024-04-24T08:10:22.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:22 smithi117 ceph-mon[32438]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:22.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:22 smithi117 ceph-mon[32438]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:10:22.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:22 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:22.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:22 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:22.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:22 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1608827400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:22.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:22 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1039161795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:10:22.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:22 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:22.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:22 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:22.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:22 smithi194 ceph-mon[32836]: [24/Apr/2024:08:10:20] ENGINE Bus STARTING 2024-04-24T08:10:22.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:22 smithi194 ceph-mon[32836]: [24/Apr/2024:08:10:20] ENGINE Serving on https://172.21.15.28:7150 2024-04-24T08:10:22.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:22 smithi194 ceph-mon[32836]: [24/Apr/2024:08:10:21] ENGINE Serving on http://172.21.15.28:8765 2024-04-24T08:10:22.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:22 smithi194 ceph-mon[32836]: [24/Apr/2024:08:10:21] ENGINE Bus STARTED 2024-04-24T08:10:22.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:22 smithi194 ceph-mon[32836]: from='client.25262 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:22.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:22 smithi194 ceph-mon[32836]: pgmap v3: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:10:22.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:22 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:22.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:22 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:22.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:22 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1608827400' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:22.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:22 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1039161795' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:10:22.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:22 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:22.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:22 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:23.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.039+0000 7efcacc00640 1 -- 172.21.15.28:0/2108785316 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7efca81a2ce0 con 0x7efca8101f90 2024-04-24T08:10:23.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.039+0000 7efc9fe00640 1 -- 172.21.15.28:0/2108785316 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7efc98014030 con 0x7efca8101f90 2024-04-24T08:10:23.040 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-24T08:10:23.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.043+0000 7efcacc00640 1 -- 172.21.15.28:0/2108785316 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efc8c0a3db0 msgr2=0x7efc8c0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.043+0000 7efcacc00640 1 --2- 172.21.15.28:0/2108785316 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efc8c0a3db0 0x7efc8c0a6270 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7efc94031440 tx=0x7efc940315b0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.043 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.043+0000 7efcacc00640 1 -- 172.21.15.28:0/2108785316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efca8101f90 msgr2=0x7efca819a420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.043+0000 7efcacc00640 1 --2- 172.21.15.28:0/2108785316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efca8101f90 0x7efca819a420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7efc9800c970 tx=0x7efc9800ce30 comp rx=0 tx=0).stop 2024-04-24T08:10:23.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.043+0000 7efcacc00640 1 -- 172.21.15.28:0/2108785316 shutdown_connections 2024-04-24T08:10:23.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.043+0000 7efcacc00640 1 --2- 172.21.15.28:0/2108785316 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efc8c0a3db0 0x7efc8c0a6270 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.043+0000 7efcacc00640 1 --2- 172.21.15.28:0/2108785316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efca8103b40 0x7efca81a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.043+0000 7efcacc00640 1 --2- 172.21.15.28:0/2108785316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efca8103180 0x7efca819a960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.043+0000 7efcacc00640 1 --2- 172.21.15.28:0/2108785316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efca8101f90 0x7efca819a420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.043+0000 7efcacc00640 1 -- 172.21.15.28:0/2108785316 >> 172.21.15.28:0/2108785316 conn(0x7efca80fd500 msgr2=0x7efca8074230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:23.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.044+0000 7efcacc00640 1 -- 172.21.15.28:0/2108785316 shutdown_connections 2024-04-24T08:10:23.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.044+0000 7efcacc00640 1 -- 172.21.15.28:0/2108785316 wait complete. 2024-04-24T08:10:23.064 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:10:23.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.273+0000 7f7cd9c00640 1 -- 172.21.15.28:0/911796873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cd4107d30 msgr2=0x7f7cd410a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.273+0000 7f7cd9c00640 1 --2- 172.21.15.28:0/911796873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cd4107d30 0x7f7cd410a180 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f7ccc009ca0 tx=0x7f7ccc02a8b0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.273+0000 7f7cd9c00640 1 -- 172.21.15.28:0/911796873 shutdown_connections 2024-04-24T08:10:23.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.273+0000 7f7cd9c00640 1 --2- 172.21.15.28:0/911796873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd4105400 0x7f7cd41077f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.274 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.273+0000 7f7cd9c00640 1 --2- 172.21.15.28:0/911796873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cd4107d30 0x7f7cd410a180 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.273+0000 7f7cd9c00640 1 --2- 172.21.15.28:0/911796873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cd4102ad0 0x7f7cd4104ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.273+0000 7f7cd9c00640 1 -- 172.21.15.28:0/911796873 >> 172.21.15.28:0/911796873 conn(0x7f7cd40fc4e0 msgr2=0x7f7cd40fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:23.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.273+0000 7f7cd9c00640 1 -- 172.21.15.28:0/911796873 shutdown_connections 2024-04-24T08:10:23.275 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.274+0000 7f7cd9c00640 1 -- 172.21.15.28:0/911796873 wait complete. 2024-04-24T08:10:23.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.275+0000 7f7cd9c00640 1 Processor -- start 2024-04-24T08:10:23.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.275+0000 7f7cd9c00640 1 -- start start 2024-04-24T08:10:23.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.275+0000 7f7cd9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd4102ad0 0x7f7cd4102440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.276+0000 7f7cd9c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cd4105400 0x7f7cd4100a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.276+0000 7f7cd9c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cd4107d30 0x7f7cd4100fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.276+0000 7f7cd9c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd410dd30 con 0x7f7cd4102ad0 2024-04-24T08:10:23.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.276+0000 7f7cd9c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd410dbb0 con 0x7f7cd4107d30 2024-04-24T08:10:23.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.276+0000 7f7cd9c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7cd410deb0 con 0x7f7cd4105400 2024-04-24T08:10:23.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.276+0000 7f7cd2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd4102ad0 0x7f7cd4102440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.276+0000 7f7cd3400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cd4107d30 0x7f7cd4100fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.276+0000 7f7cd2000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cd4105400 0x7f7cd4100a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.276+0000 7f7cd3400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cd4107d30 0x7f7cd4100fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:37990/0 (socket says 172.21.15.28:37990) 2024-04-24T08:10:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.276+0000 7f7cd2000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cd4105400 0x7f7cd4100a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:51694/0 (socket says 172.21.15.28:51694) 2024-04-24T08:10:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.276+0000 7f7cd3400640 1 -- 172.21.15.28:0/3624802610 learned_addr learned my addr 172.21.15.28:0/3624802610 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.276+0000 7f7cd2000640 1 -- 172.21.15.28:0/3624802610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cd4107d30 msgr2=0x7f7cd4100fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.276+0000 7f7cd2000640 1 --2- 172.21.15.28:0/3624802610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cd4107d30 0x7f7cd4100fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.276+0000 7f7cd2000640 1 -- 172.21.15.28:0/3624802610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd4102ad0 msgr2=0x7f7cd4102440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.276+0000 7f7cd2000640 1 --2- 172.21.15.28:0/3624802610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd4102ad0 0x7f7cd4102440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.276+0000 7f7cd2000640 1 -- 172.21.15.28:0/3624802610 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cd4101850 con 0x7f7cd4105400 2024-04-24T08:10:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.276+0000 7f7cd3400640 1 --2- 172.21.15.28:0/3624802610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cd4107d30 0x7f7cd4100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.277+0000 7f7cd2000640 1 --2- 172.21.15.28:0/3624802610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cd4105400 0x7f7cd4100a80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000e9c0 tx=0x7f7cc000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:23.278 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.277+0000 7f7ccb400640 1 -- 172.21.15.28:0/3624802610 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc000ccc0 con 0x7f7cd4105400 2024-04-24T08:10:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.277+0000 7f7ccb400640 1 -- 172.21.15.28:0/3624802610 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cc0010910 con 0x7f7cd4105400 2024-04-24T08:10:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.277+0000 7f7cd9c00640 1 -- 172.21.15.28:0/3624802610 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cd41a2400 con 0x7f7cd4105400 2024-04-24T08:10:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.277+0000 7f7ccb400640 1 -- 172.21.15.28:0/3624802610 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cc00188c0 con 0x7f7cd4105400 2024-04-24T08:10:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.277+0000 7f7cd9c00640 1 -- 172.21.15.28:0/3624802610 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cd41a2950 con 0x7f7cd4105400 2024-04-24T08:10:23.279 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.278+0000 7f7cd9c00640 1 -- 172.21.15.28:0/3624802610 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7cd40fdfd0 con 0x7f7cd4105400 2024-04-24T08:10:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.281+0000 7f7ccb400640 1 -- 172.21.15.28:0/3624802610 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 24) ==== 138817+0+0 (secure 0 0 0) 0x7f7cc0010ac0 con 0x7f7cd4105400 2024-04-24T08:10:23.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.282+0000 7f7ccb400640 1 --2- 172.21.15.28:0/3624802610 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7cb80a3db0 0x7f7cb80a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.283+0000 7f7ccb400640 1 -- 172.21.15.28:0/3624802610 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f7cc00d0d10 con 0x7f7cd4105400 2024-04-24T08:10:23.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.283+0000 7f7cd2a00640 1 --2- 172.21.15.28:0/3624802610 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7cb80a3db0 0x7f7cb80a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:23.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.284+0000 7f7cd2a00640 1 --2- 172.21.15.28:0/3624802610 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7cb80a3db0 0x7f7cb80a6270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009bd0 tx=0x7f7cc4005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:23.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.286+0000 7f7ccb400640 1 -- 172.21.15.28:0/3624802610 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7cc009cbd0 con 0x7f7cd4105400 2024-04-24T08:10:23.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.517+0000 7f7cd9c00640 1 -- 172.21.15.28:0/3624802610 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7cd402d870 con 0x7f7cd4105400 2024-04-24T08:10:23.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.519+0000 7f7ccb400640 1 -- 172.21.15.28:0/3624802610 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23099 (secure 0 0 0) 0x7f7cc00a0ed0 con 0x7f7cd4105400 2024-04-24T08:10:23.520 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:10:23.520 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:10:20.116137+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1819419617":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6801/361125225":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/3758227512":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/874707228":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/2660639233":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/361125225":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:10:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.522+0000 7f7cd9c00640 1 -- 172.21.15.28:0/3624802610 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7cb80a3db0 msgr2=0x7f7cb80a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.522+0000 7f7cd9c00640 1 --2- 172.21.15.28:0/3624802610 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7cb80a3db0 0x7f7cb80a6270 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f7cc4009bd0 tx=0x7f7cc4005f90 comp rx=0 tx=0).stop 2024-04-24T08:10:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.522+0000 7f7cd9c00640 1 -- 172.21.15.28:0/3624802610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cd4105400 msgr2=0x7f7cd4100a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.522+0000 7f7cd9c00640 1 --2- 172.21.15.28:0/3624802610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cd4105400 0x7f7cd4100a80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f7cc000e9c0 tx=0x7f7cc000ee80 comp rx=0 tx=0).stop 2024-04-24T08:10:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.522+0000 7f7cd9c00640 1 -- 172.21.15.28:0/3624802610 shutdown_connections 2024-04-24T08:10:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.522+0000 7f7cd9c00640 1 --2- 172.21.15.28:0/3624802610 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7cb80a3db0 0x7f7cb80a6270 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.522+0000 7f7cd9c00640 1 --2- 172.21.15.28:0/3624802610 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cd4107d30 0x7f7cd4100fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.522+0000 7f7cd9c00640 1 --2- 172.21.15.28:0/3624802610 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cd4105400 0x7f7cd4100a80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.522+0000 7f7cd9c00640 1 --2- 172.21.15.28:0/3624802610 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cd4102ad0 0x7f7cd4102440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.522+0000 7f7cd9c00640 1 -- 172.21.15.28:0/3624802610 >> 172.21.15.28:0/3624802610 conn(0x7f7cd40fc4e0 msgr2=0x7f7cd4103410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:23.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.522+0000 7f7cd9c00640 1 -- 172.21.15.28:0/3624802610 shutdown_connections 2024-04-24T08:10:23.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.522+0000 7f7cd9c00640 1 -- 172.21.15.28:0/3624802610 wait complete. 2024-04-24T08:10:23.542 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:02:34.086371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:07:49.819536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:07:50.493486+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:10:23.542 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:10:23.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.746+0000 7f6aee800640 1 -- 172.21.15.28:0/2772566255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ae8105330 msgr2=0x7f6ae81057b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.746+0000 7f6aee800640 1 --2- 172.21.15.28:0/2772566255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ae8105330 0x7f6ae81057b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4009900 tx=0x7f6ad402a410 comp rx=0 tx=0).stop 2024-04-24T08:10:23.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.746+0000 7f6aee800640 1 -- 172.21.15.28:0/2772566255 shutdown_connections 2024-04-24T08:10:23.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.746+0000 7f6aee800640 1 --2- 172.21.15.28:0/2772566255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ae8105cf0 0x7f6ae810c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.746+0000 7f6aee800640 1 --2- 172.21.15.28:0/2772566255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ae8105330 0x7f6ae81057b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.746+0000 7f6aee800640 1 --2- 172.21.15.28:0/2772566255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ae8104140 0x7f6ae8104560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.746+0000 7f6aee800640 1 -- 172.21.15.28:0/2772566255 >> 172.21.15.28:0/2772566255 conn(0x7f6ae80ff6b0 msgr2=0x7f6ae8101af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:23.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.746+0000 7f6aee800640 1 -- 172.21.15.28:0/2772566255 shutdown_connections 2024-04-24T08:10:23.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.746+0000 7f6aee800640 1 -- 172.21.15.28:0/2772566255 wait complete. 2024-04-24T08:10:23.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.747+0000 7f6aee800640 1 Processor -- start 2024-04-24T08:10:23.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.748+0000 7f6aee800640 1 -- start start 2024-04-24T08:10:23.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.748+0000 7f6aee800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ae8104140 0x7f6ae819a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.748+0000 7f6aee800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ae8105330 0x7f6ae819a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.748+0000 7f6aee800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ae8105cf0 0x7f6ae81a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.748+0000 7f6aee800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae810fe20 con 0x7f6ae8105cf0 2024-04-24T08:10:23.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.748+0000 7f6aee800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae810fca0 con 0x7f6ae8104140 2024-04-24T08:10:23.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.748+0000 7f6aee800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6ae810ffa0 con 0x7f6ae8105330 2024-04-24T08:10:23.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.748+0000 7f6aed400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ae8104140 0x7f6ae819a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:23.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.748+0000 7f6aed400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ae8104140 0x7f6ae819a3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:38008/0 (socket says 172.21.15.28:38008) 2024-04-24T08:10:23.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.748+0000 7f6aed400640 1 -- 172.21.15.28:0/272976103 learned_addr learned my addr 172.21.15.28:0/272976103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:23.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.748+0000 7f6aede00640 1 --2- 172.21.15.28:0/272976103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ae8105cf0 0x7f6ae81a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:23.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.748+0000 7f6aeca00640 1 --2- 172.21.15.28:0/272976103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ae8105330 0x7f6ae819a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:23.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.749+0000 7f6aed400640 1 -- 172.21.15.28:0/272976103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ae8105330 msgr2=0x7f6ae819a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.749+0000 7f6aed400640 1 --2- 172.21.15.28:0/272976103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ae8105330 0x7f6ae819a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.749+0000 7f6aed400640 1 -- 172.21.15.28:0/272976103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ae8105cf0 msgr2=0x7f6ae81a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.749+0000 7f6aed400640 1 --2- 172.21.15.28:0/272976103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ae8105cf0 0x7f6ae81a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.749+0000 7f6aed400640 1 -- 172.21.15.28:0/272976103 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ae81a1ff0 con 0x7f6ae8104140 2024-04-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.749+0000 7f6aeca00640 1 --2- 172.21.15.28:0/272976103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ae8105330 0x7f6ae819a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.749+0000 7f6aed400640 1 --2- 172.21.15.28:0/272976103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ae8104140 0x7f6ae819a3d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00e9c0 tx=0x7f6adc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.749+0000 7f6ae6000640 1 -- 172.21.15.28:0/272976103 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc00cd80 con 0x7f6ae8104140 2024-04-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.749+0000 7f6ae6000640 1 -- 172.21.15.28:0/272976103 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6adc0035f0 con 0x7f6ae8104140 2024-04-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.749+0000 7f6ae6000640 1 -- 172.21.15.28:0/272976103 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6adc010690 con 0x7f6ae8104140 2024-04-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.749+0000 7f6aee800640 1 -- 172.21.15.28:0/272976103 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ae81a22e0 con 0x7f6ae8104140 2024-04-24T08:10:23.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.750+0000 7f6aee800640 1 -- 172.21.15.28:0/272976103 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ae81a27b0 con 0x7f6ae8104140 2024-04-24T08:10:23.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.751+0000 7f6aee800640 1 -- 172.21.15.28:0/272976103 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6ae810b460 con 0x7f6ae8104140 2024-04-24T08:10:23.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.753+0000 7f6ae6000640 1 -- 172.21.15.28:0/272976103 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 24) ==== 138817+0+0 (secure 0 0 0) 0x7f6adc0037a0 con 0x7f6ae8104140 2024-04-24T08:10:23.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.754+0000 7f6ae6000640 1 --2- 172.21.15.28:0/272976103 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6ad00a3e30 0x7f6ad00a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:23.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.755+0000 7f6ae6000640 1 -- 172.21.15.28:0/272976103 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f6adc0d0e40 con 0x7f6ae8104140 2024-04-24T08:10:23.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.755+0000 7f6aeca00640 1 --2- 172.21.15.28:0/272976103 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6ad00a3e30 0x7f6ad00a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:23.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.755+0000 7f6aeca00640 1 --2- 172.21.15.28:0/272976103 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6ad00a3e30 0x7f6ad00a62f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4006010 tx=0x7f6ad4000bc0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:23.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.758+0000 7f6ae6000640 1 -- 172.21.15.28:0/272976103 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6adc09d2f0 con 0x7f6ae8104140 2024-04-24T08:10:23.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:23 smithi028 ceph-mon[27786]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:10:23.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:23 smithi028 ceph-mon[27786]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-04-24T08:10:23.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:23 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2767381748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:10:23.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:23 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:23.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:23 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:23.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:23 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2108785316' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:10:23.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:23 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3624802610' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:23.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:23 smithi117 ceph-mon[32438]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:10:23.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:23 smithi117 ceph-mon[32438]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-04-24T08:10:23.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:23 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2767381748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:10:23.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:23 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:23.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:23 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:23.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:23 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2108785316' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:10:23.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:23 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3624802610' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:23.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:23 smithi194 ceph-mon[32836]: pgmap v4: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:10:23.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:23 smithi194 ceph-mon[32836]: mgrmap e24: x(active, since 2s), standbys: y, z 2024-04-24T08:10:23.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:23 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2767381748' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:10:23.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:23 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:23.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:23 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:23.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:23 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2108785316' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:10:23.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:23 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3624802610' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:23.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.978+0000 7f6aee800640 1 -- 172.21.15.28:0/272976103 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6ae8062c00 con 0x7f6ae8104140 2024-04-24T08:10:23.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.978+0000 7f6ae6000640 1 -- 172.21.15.28:0/272976103 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f6adc09cbf0 con 0x7f6ae8104140 2024-04-24T08:10:23.979 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-24T08:10:23.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.982+0000 7f6aee800640 1 -- 172.21.15.28:0/272976103 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6ad00a3e30 msgr2=0x7f6ad00a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.982+0000 7f6aee800640 1 --2- 172.21.15.28:0/272976103 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6ad00a3e30 0x7f6ad00a62f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f6ad4006010 tx=0x7f6ad4000bc0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.982+0000 7f6aee800640 1 -- 172.21.15.28:0/272976103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ae8104140 msgr2=0x7f6ae819a3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.982+0000 7f6aee800640 1 --2- 172.21.15.28:0/272976103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ae8104140 0x7f6ae819a3d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f6adc00e9c0 tx=0x7f6adc00ee80 comp rx=0 tx=0).stop 2024-04-24T08:10:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.982+0000 7f6aee800640 1 -- 172.21.15.28:0/272976103 shutdown_connections 2024-04-24T08:10:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.982+0000 7f6aee800640 1 --2- 172.21.15.28:0/272976103 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6ad00a3e30 0x7f6ad00a62f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.982+0000 7f6aee800640 1 --2- 172.21.15.28:0/272976103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ae8105cf0 0x7f6ae81a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.982+0000 7f6aee800640 1 --2- 172.21.15.28:0/272976103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ae8105330 0x7f6ae819a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.982+0000 7f6aee800640 1 --2- 172.21.15.28:0/272976103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ae8104140 0x7f6ae819a3d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.982+0000 7f6aee800640 1 -- 172.21.15.28:0/272976103 >> 172.21.15.28:0/272976103 conn(0x7f6ae80ff6b0 msgr2=0x7f6ae8108630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.982+0000 7f6aee800640 1 -- 172.21.15.28:0/272976103 shutdown_connections 2024-04-24T08:10:23.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:23.982+0000 7f6aee800640 1 -- 172.21.15.28:0/272976103 wait complete. 2024-04-24T08:10:24.004 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:10:24.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.206+0000 7f7697e00640 1 -- 172.21.15.28:0/2701156221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7698102a80 msgr2=0x7f7698104e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.206+0000 7f7697e00640 1 --2- 172.21.15.28:0/2701156221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7698102a80 0x7f7698104e70 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f7688009900 tx=0x7f768802a410 comp rx=0 tx=0).stop 2024-04-24T08:10:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.206+0000 7f7697e00640 1 -- 172.21.15.28:0/2701156221 shutdown_connections 2024-04-24T08:10:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.206+0000 7f7697e00640 1 --2- 172.21.15.28:0/2701156221 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7698107ce0 0x7f769810a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.206+0000 7f7697e00640 1 --2- 172.21.15.28:0/2701156221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76981053b0 0x7f76981077a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.206+0000 7f7697e00640 1 --2- 172.21.15.28:0/2701156221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7698102a80 0x7f7698104e70 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.206+0000 7f7697e00640 1 -- 172.21.15.28:0/2701156221 >> 172.21.15.28:0/2701156221 conn(0x7f76980fc3b0 msgr2=0x7f76980fe7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.206+0000 7f7697e00640 1 -- 172.21.15.28:0/2701156221 shutdown_connections 2024-04-24T08:10:24.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.206+0000 7f7697e00640 1 -- 172.21.15.28:0/2701156221 wait complete. 2024-04-24T08:10:24.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.207+0000 7f7697e00640 1 Processor -- start 2024-04-24T08:10:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.207+0000 7f7697e00640 1 -- start start 2024-04-24T08:10:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.207+0000 7f7697e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7698102a80 0x7f769819a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.207+0000 7f7697e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76981053b0 0x7f769819a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.207+0000 7f7697e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7698107ce0 0x7f76981a1a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.207+0000 7f7697e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f769810dc40 con 0x7f7698102a80 2024-04-24T08:10:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.207+0000 7f7697e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f769810dac0 con 0x7f7698107ce0 2024-04-24T08:10:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.207+0000 7f7697e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f769810ddc0 con 0x7f76981053b0 2024-04-24T08:10:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.208+0000 7f7696a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7698102a80 0x7f769819a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.208+0000 7f7696000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76981053b0 0x7f769819a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.208+0000 7f7697400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7698107ce0 0x7f76981a1a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.208+0000 7f7696000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76981053b0 0x7f769819a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:51732/0 (socket says 172.21.15.28:51732) 2024-04-24T08:10:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.208+0000 7f7696000640 1 -- 172.21.15.28:0/1321411887 learned_addr learned my addr 172.21.15.28:0/1321411887 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:24.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.208+0000 7f7696a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7698102a80 0x7f769819a450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33846/0 (socket says 172.21.15.28:33846) 2024-04-24T08:10:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.208+0000 7f7697400640 1 -- 172.21.15.28:0/1321411887 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76981053b0 msgr2=0x7f769819a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.208+0000 7f7697400640 1 --2- 172.21.15.28:0/1321411887 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76981053b0 0x7f769819a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.208+0000 7f7697400640 1 -- 172.21.15.28:0/1321411887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7698102a80 msgr2=0x7f769819a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.208+0000 7f7697400640 1 --2- 172.21.15.28:0/1321411887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7698102a80 0x7f769819a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.208+0000 7f7697400640 1 -- 172.21.15.28:0/1321411887 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76981a2070 con 0x7f7698107ce0 2024-04-24T08:10:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.208+0000 7f7696000640 1 --2- 172.21.15.28:0/1321411887 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76981053b0 0x7f769819a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.209+0000 7f7697400640 1 --2- 172.21.15.28:0/1321411887 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7698107ce0 0x7f76981a1a00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7690009fd0 tx=0x7f769000ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.209+0000 7f768f400640 1 -- 172.21.15.28:0/1321411887 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f769000ce20 con 0x7f7698107ce0 2024-04-24T08:10:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.209+0000 7f768f400640 1 -- 172.21.15.28:0/1321411887 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7690005020 con 0x7f7698107ce0 2024-04-24T08:10:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.209+0000 7f768f400640 1 -- 172.21.15.28:0/1321411887 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7690010720 con 0x7f7698107ce0 2024-04-24T08:10:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.209+0000 7f7697e00640 1 -- 172.21.15.28:0/1321411887 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76981a2360 con 0x7f7698107ce0 2024-04-24T08:10:24.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.209+0000 7f7697e00640 1 -- 172.21.15.28:0/1321411887 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76981a2900 con 0x7f7698107ce0 2024-04-24T08:10:24.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.210+0000 7f7697e00640 1 -- 172.21.15.28:0/1321411887 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f76981097d0 con 0x7f7698107ce0 2024-04-24T08:10:24.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.212+0000 7f768f400640 1 -- 172.21.15.28:0/1321411887 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 24) ==== 138817+0+0 (secure 0 0 0) 0x7f7690005540 con 0x7f7698107ce0 2024-04-24T08:10:24.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.213+0000 7f768f400640 1 --2- 172.21.15.28:0/1321411887 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f767c0a3e30 0x7f767c0a62f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:24.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.214+0000 7f768f400640 1 -- 172.21.15.28:0/1321411887 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f76900d0f90 con 0x7f7698107ce0 2024-04-24T08:10:24.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.214+0000 7f7696a00640 1 --2- 172.21.15.28:0/1321411887 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f767c0a3e30 0x7f767c0a62f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:24.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.214+0000 7f7696a00640 1 --2- 172.21.15.28:0/1321411887 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f767c0a3e30 0x7f767c0a62f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7688031440 tx=0x7f76880315b0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:24.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.217+0000 7f768f400640 1 -- 172.21.15.28:0/1321411887 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f769009d440 con 0x7f7698107ce0 2024-04-24T08:10:24.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.425+0000 7f7697e00640 1 -- 172.21.15.28:0/1321411887 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7698062c00 con 0x7f7698107ce0 2024-04-24T08:10:24.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.425+0000 7f768f400640 1 -- 172.21.15.28:0/1321411887 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f769009cd40 con 0x7f7698107ce0 2024-04-24T08:10:24.426 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-24T08:10:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.428+0000 7f7697e00640 1 -- 172.21.15.28:0/1321411887 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f767c0a3e30 msgr2=0x7f767c0a62f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.428+0000 7f7697e00640 1 --2- 172.21.15.28:0/1321411887 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f767c0a3e30 0x7f767c0a62f0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f7688031440 tx=0x7f76880315b0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.428+0000 7f7697e00640 1 -- 172.21.15.28:0/1321411887 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7698107ce0 msgr2=0x7f76981a1a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:24.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.428+0000 7f7697e00640 1 --2- 172.21.15.28:0/1321411887 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7698107ce0 0x7f76981a1a00 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f7690009fd0 tx=0x7f769000ef10 comp rx=0 tx=0).stop 2024-04-24T08:10:24.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.429+0000 7f7697e00640 1 -- 172.21.15.28:0/1321411887 shutdown_connections 2024-04-24T08:10:24.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.429+0000 7f7697e00640 1 --2- 172.21.15.28:0/1321411887 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f767c0a3e30 0x7f767c0a62f0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.429+0000 7f7697e00640 1 --2- 172.21.15.28:0/1321411887 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7698107ce0 0x7f76981a1a00 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.429+0000 7f7697e00640 1 --2- 172.21.15.28:0/1321411887 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f76981053b0 0x7f769819a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.429+0000 7f7697e00640 1 --2- 172.21.15.28:0/1321411887 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7698102a80 0x7f769819a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.429+0000 7f7697e00640 1 -- 172.21.15.28:0/1321411887 >> 172.21.15.28:0/1321411887 conn(0x7f76980fc3b0 msgr2=0x7f7698102180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:24.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.429+0000 7f7697e00640 1 -- 172.21.15.28:0/1321411887 shutdown_connections 2024-04-24T08:10:24.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.429+0000 7f7697e00640 1 -- 172.21.15.28:0/1321411887 wait complete. 2024-04-24T08:10:24.447 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:10:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.635+0000 7f5c17e00640 1 -- 172.21.15.28:0/4276313034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c18076040 msgr2=0x7f5c18074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:24.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.635+0000 7f5c17e00640 1 --2- 172.21.15.28:0/4276313034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c18076040 0x7f5c18074490 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f5c08009900 tx=0x7f5c0802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.635+0000 7f5c17e00640 1 -- 172.21.15.28:0/4276313034 shutdown_connections 2024-04-24T08:10:24.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.635+0000 7f5c17e00640 1 --2- 172.21.15.28:0/4276313034 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c18074a60 0x7f5c18074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.635+0000 7f5c17e00640 1 --2- 172.21.15.28:0/4276313034 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c18109e50 0x7f5c1810c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.635+0000 7f5c17e00640 1 --2- 172.21.15.28:0/4276313034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c18076040 0x7f5c18074490 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.635+0000 7f5c17e00640 1 -- 172.21.15.28:0/4276313034 >> 172.21.15.28:0/4276313034 conn(0x7f5c180fd500 msgr2=0x7f5c180ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:24.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.635+0000 7f5c17e00640 1 -- 172.21.15.28:0/4276313034 shutdown_connections 2024-04-24T08:10:24.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.635+0000 7f5c17e00640 1 -- 172.21.15.28:0/4276313034 wait complete. 2024-04-24T08:10:24.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.636+0000 7f5c17e00640 1 Processor -- start 2024-04-24T08:10:24.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.636+0000 7f5c17e00640 1 -- start start 2024-04-24T08:10:24.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.637+0000 7f5c17e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c18074a60 0x7f5c1819a420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:24.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.637+0000 7f5c17e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c18076040 0x7f5c1819a960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.637+0000 7f5c17e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c18109e50 0x7f5c181a19d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.637+0000 7f5c17e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1810fdc0 con 0x7f5c18109e50 2024-04-24T08:10:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.637+0000 7f5c17e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1810fc40 con 0x7f5c18074a60 2024-04-24T08:10:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.637+0000 7f5c17e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5c1810ff40 con 0x7f5c18076040 2024-04-24T08:10:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.637+0000 7f5c16000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c18076040 0x7f5c1819a960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.637+0000 7f5c17400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c18109e50 0x7f5c181a19d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.637+0000 7f5c16000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c18076040 0x7f5c1819a960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:51750/0 (socket says 172.21.15.28:51750) 2024-04-24T08:10:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.637+0000 7f5c16a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c18074a60 0x7f5c1819a420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.637+0000 7f5c16000640 1 -- 172.21.15.28:0/2234724785 learned_addr learned my addr 172.21.15.28:0/2234724785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:24.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.637+0000 7f5c16000640 1 -- 172.21.15.28:0/2234724785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c18074a60 msgr2=0x7f5c1819a420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.637+0000 7f5c16000640 1 --2- 172.21.15.28:0/2234724785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c18074a60 0x7f5c1819a420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.637+0000 7f5c16000640 1 -- 172.21.15.28:0/2234724785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c18109e50 msgr2=0x7f5c181a19d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.637+0000 7f5c16000640 1 --2- 172.21.15.28:0/2234724785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c18109e50 0x7f5c181a19d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.637+0000 7f5c16000640 1 -- 172.21.15.28:0/2234724785 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c181a2040 con 0x7f5c18076040 2024-04-24T08:10:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.637+0000 7f5c17400640 1 --2- 172.21.15.28:0/2234724785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c18109e50 0x7f5c181a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.638+0000 7f5c16000640 1 --2- 172.21.15.28:0/2234724785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c18076040 0x7f5c1819a960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400e9f0 tx=0x7f5c0400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.638+0000 7f5c0f400640 1 -- 172.21.15.28:0/2234724785 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c0400cd80 con 0x7f5c18076040 2024-04-24T08:10:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.638+0000 7f5c0f400640 1 -- 172.21.15.28:0/2234724785 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c04010910 con 0x7f5c18076040 2024-04-24T08:10:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.638+0000 7f5c0f400640 1 -- 172.21.15.28:0/2234724785 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c04018990 con 0x7f5c18076040 2024-04-24T08:10:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.638+0000 7f5c17e00640 1 -- 172.21.15.28:0/2234724785 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c181a2330 con 0x7f5c18076040 2024-04-24T08:10:24.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.638+0000 7f5c17e00640 1 -- 172.21.15.28:0/2234724785 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c181a2880 con 0x7f5c18076040 2024-04-24T08:10:24.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.639+0000 7f5c17e00640 1 -- 172.21.15.28:0/2234724785 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c18108b80 con 0x7f5c18076040 2024-04-24T08:10:24.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.642+0000 7f5c0f400640 1 -- 172.21.15.28:0/2234724785 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 24) ==== 138817+0+0 (secure 0 0 0) 0x7f5c04010ac0 con 0x7f5c18076040 2024-04-24T08:10:24.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.643+0000 7f5c0f400640 1 --2- 172.21.15.28:0/2234724785 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5bfc0a3db0 0x7f5bfc0a6270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:24.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.643+0000 7f5c0f400640 1 -- 172.21.15.28:0/2234724785 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f5c040d0d50 con 0x7f5c18076040 2024-04-24T08:10:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.643+0000 7f5c16a00640 1 --2- 172.21.15.28:0/2234724785 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5bfc0a3db0 0x7f5bfc0a6270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:24.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.644+0000 7f5c16a00640 1 --2- 172.21.15.28:0/2234724785 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5bfc0a3db0 0x7f5bfc0a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5c08031440 tx=0x7f5c080315b0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:24.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.648+0000 7f5c0f400640 1 -- 172.21.15.28:0/2234724785 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f5c0409cbb0 con 0x7f5c18076040 2024-04-24T08:10:24.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.673+0000 7f5c0f400640 1 -- 172.21.15.28:0/2234724785 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7f5c04099220 con 0x7f5c18076040 2024-04-24T08:10:24.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:24.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:24.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-24T08:10:24.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/272976103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:10:24.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:24.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:24.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.792 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.792 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:24.792 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:24.792 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:24 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1321411887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:10:24.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.875+0000 7f5c17e00640 1 -- 172.21.15.28:0/2234724785 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f5c1802d840 con 0x7f5c18076040 2024-04-24T08:10:24.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.876+0000 7f5c0f400640 1 -- 172.21.15.28:0/2234724785 <== mon.1 v2:172.21.15.194:3300/0 8 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f5c0409d2b0 con 0x7f5c18076040 2024-04-24T08:10:24.876 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-24T08:10:24.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.878+0000 7f5c17e00640 1 -- 172.21.15.28:0/2234724785 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5bfc0a3db0 msgr2=0x7f5bfc0a6270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:24.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.878+0000 7f5c17e00640 1 --2- 172.21.15.28:0/2234724785 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5bfc0a3db0 0x7f5bfc0a6270 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f5c08031440 tx=0x7f5c080315b0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.878+0000 7f5c17e00640 1 -- 172.21.15.28:0/2234724785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c18076040 msgr2=0x7f5c1819a960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:24.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.878+0000 7f5c17e00640 1 --2- 172.21.15.28:0/2234724785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c18076040 0x7f5c1819a960 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5c0400e9f0 tx=0x7f5c0400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.878+0000 7f5c17e00640 1 -- 172.21.15.28:0/2234724785 shutdown_connections 2024-04-24T08:10:24.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.878+0000 7f5c17e00640 1 --2- 172.21.15.28:0/2234724785 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5bfc0a3db0 0x7f5bfc0a6270 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.878+0000 7f5c17e00640 1 --2- 172.21.15.28:0/2234724785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c18109e50 0x7f5c181a19d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.878+0000 7f5c17e00640 1 --2- 172.21.15.28:0/2234724785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c18076040 0x7f5c1819a960 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.878+0000 7f5c17e00640 1 --2- 172.21.15.28:0/2234724785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c18074a60 0x7f5c1819a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:24.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.878+0000 7f5c17e00640 1 -- 172.21.15.28:0/2234724785 >> 172.21.15.28:0/2234724785 conn(0x7f5c180fd500 msgr2=0x7f5c18105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:24.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.878+0000 7f5c17e00640 1 -- 172.21.15.28:0/2234724785 shutdown_connections 2024-04-24T08:10:24.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:24.878+0000 7f5c17e00640 1 -- 172.21.15.28:0/2234724785 wait complete. 2024-04-24T08:10:24.897 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-04-24T08:10:24.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:10:24.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:24.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:24.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-24T08:10:24.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/272976103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:10:24.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:24.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:24.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.943 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.943 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.943 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:24.943 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:24.943 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:24.943 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:24 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1321411887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:10:25.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.082+0000 7f57c5a00640 1 -- 172.21.15.28:0/2775103727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c0109d60 msgr2=0x7f57c010c1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.082+0000 7f57c5a00640 1 --2- 172.21.15.28:0/2775103727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c0109d60 0x7f57c010c1f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f57b8009ca0 tx=0x7f57b802a8c0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.082+0000 7f57c5a00640 1 -- 172.21.15.28:0/2775103727 shutdown_connections 2024-04-24T08:10:25.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.082+0000 7f57c5a00640 1 --2- 172.21.15.28:0/2775103727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c0109d60 0x7f57c010c1f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.082+0000 7f57c5a00640 1 --2- 172.21.15.28:0/2775103727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57c0074a60 0x7f57c0074ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.082+0000 7f57c5a00640 1 --2- 172.21.15.28:0/2775103727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57c0076040 0x7f57c0074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.082+0000 7f57c5a00640 1 -- 172.21.15.28:0/2775103727 >> 172.21.15.28:0/2775103727 conn(0x7f57c00fd410 msgr2=0x7f57c00ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:25.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.083+0000 7f57c5a00640 1 -- 172.21.15.28:0/2775103727 shutdown_connections 2024-04-24T08:10:25.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.083+0000 7f57c5a00640 1 -- 172.21.15.28:0/2775103727 wait complete. 2024-04-24T08:10:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.084+0000 7f57c5a00640 1 Processor -- start 2024-04-24T08:10:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.084+0000 7f57c5a00640 1 -- start start 2024-04-24T08:10:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.084+0000 7f57c5a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c0074a60 0x7f57c019a340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.084+0000 7f57c5a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57c0076040 0x7f57c019a880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.084+0000 7f57c5a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57c0109d60 0x7f57c01a18f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.084+0000 7f57c5a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f57c010fd20 con 0x7f57c0109d60 2024-04-24T08:10:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.084+0000 7f57c5a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f57c010fba0 con 0x7f57c0074a60 2024-04-24T08:10:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.084+0000 7f57c5a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f57c010fea0 con 0x7f57c0076040 2024-04-24T08:10:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.085+0000 7f57bfe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c0074a60 0x7f57c019a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.085+0000 7f57c5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57c0109d60 0x7f57c01a18f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.085+0000 7f57bf400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57c0076040 0x7f57c019a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:25.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.085+0000 7f57c5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57c0109d60 0x7f57c01a18f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33886/0 (socket says 172.21.15.28:33886) 2024-04-24T08:10:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.085+0000 7f57bfe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c0074a60 0x7f57c019a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:38078/0 (socket says 172.21.15.28:38078) 2024-04-24T08:10:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.085+0000 7f57c5000640 1 -- 172.21.15.28:0/447577061 learned_addr learned my addr 172.21.15.28:0/447577061 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.085+0000 7f57bf400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57c0076040 0x7f57c019a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:51770/0 (socket says 172.21.15.28:51770) 2024-04-24T08:10:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.085+0000 7f57bf400640 1 -- 172.21.15.28:0/447577061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c0074a60 msgr2=0x7f57c019a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.085+0000 7f57bf400640 1 --2- 172.21.15.28:0/447577061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c0074a60 0x7f57c019a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.085+0000 7f57bf400640 1 -- 172.21.15.28:0/447577061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57c0109d60 msgr2=0x7f57c01a18f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.085+0000 7f57bf400640 1 --2- 172.21.15.28:0/447577061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57c0109d60 0x7f57c01a18f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.085+0000 7f57bf400640 1 -- 172.21.15.28:0/447577061 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57c01a1f60 con 0x7f57c0076040 2024-04-24T08:10:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.085+0000 7f57c5000640 1 --2- 172.21.15.28:0/447577061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57c0109d60 0x7f57c01a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:25.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.086+0000 7f57bf400640 1 --2- 172.21.15.28:0/447577061 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57c0076040 0x7f57c019a880 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00e9f0 tx=0x7f57ac00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.086+0000 7f57bcc00640 1 -- 172.21.15.28:0/447577061 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac00ccc0 con 0x7f57c0076040 2024-04-24T08:10:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.086+0000 7f57c5a00640 1 -- 172.21.15.28:0/447577061 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57c01a2250 con 0x7f57c0076040 2024-04-24T08:10:25.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.086+0000 7f57bcc00640 1 -- 172.21.15.28:0/447577061 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57ac005020 con 0x7f57c0076040 2024-04-24T08:10:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.086+0000 7f57bcc00640 1 -- 172.21.15.28:0/447577061 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57ac0049e0 con 0x7f57c0076040 2024-04-24T08:10:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.086+0000 7f57c5a00640 1 -- 172.21.15.28:0/447577061 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57c01a27a0 con 0x7f57c0076040 2024-04-24T08:10:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.087+0000 7f57c5a00640 1 -- 172.21.15.28:0/447577061 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f57c0108a90 con 0x7f57c0076040 2024-04-24T08:10:25.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.089+0000 7f57bcc00640 1 -- 172.21.15.28:0/447577061 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7f57ac005560 con 0x7f57c0076040 2024-04-24T08:10:25.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.091+0000 7f57bcc00640 1 --2- 172.21.15.28:0/447577061 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f57a40a3e40 0x7f57a40a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:25.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.092+0000 7f57bcc00640 1 -- 172.21.15.28:0/447577061 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f57ac0d0a60 con 0x7f57c0076040 2024-04-24T08:10:25.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.092+0000 7f57bfe00640 1 --2- 172.21.15.28:0/447577061 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f57a40a3e40 0x7f57a40a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:25.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.092+0000 7f57bfe00640 1 --2- 172.21.15.28:0/447577061 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f57a40a3e40 0x7f57a40a6300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f57b0005d00 tx=0x7f57b000b410 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:25.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.093+0000 7f57bcc00640 1 -- 172.21.15.28:0/447577061 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f57ac0051d0 con 0x7f57c0076040 2024-04-24T08:10:25.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:25.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:25.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:25.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:25.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:25.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:25.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:25.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:25.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:25.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:25.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-24T08:10:25.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/272976103' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:10:25.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:25.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:25.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:25.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:25.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:25.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:25.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:25.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:25.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:25.130 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:25.130 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:25.130 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:25.130 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:24 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1321411887' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:10:25.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.339+0000 7f57c5a00640 1 -- 172.21.15.28:0/447577061 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f57c002d620 con 0x7f57c0076040 2024-04-24T08:10:25.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.340+0000 7f57bcc00640 1 -- 172.21.15.28:0/447577061 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11871 (secure 0 0 0) 0x7f57ac09cf40 con 0x7f57c0076040 2024-04-24T08:10:25.341 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:10:25.341 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:10:19.714831+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24574,24556,24572]},"id":1}]} 2024-04-24T08:10:25.341 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-24T08:10:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.343+0000 7f57c5a00640 1 -- 172.21.15.28:0/447577061 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f57a40a3e40 msgr2=0x7f57a40a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.343+0000 7f57c5a00640 1 --2- 172.21.15.28:0/447577061 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f57a40a3e40 0x7f57a40a6300 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f57b0005d00 tx=0x7f57b000b410 comp rx=0 tx=0).stop 2024-04-24T08:10:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.343+0000 7f57c5a00640 1 -- 172.21.15.28:0/447577061 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57c0076040 msgr2=0x7f57c019a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.343+0000 7f57c5a00640 1 --2- 172.21.15.28:0/447577061 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57c0076040 0x7f57c019a880 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f57ac00e9f0 tx=0x7f57ac00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.343+0000 7f57c5a00640 1 -- 172.21.15.28:0/447577061 shutdown_connections 2024-04-24T08:10:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.343+0000 7f57c5a00640 1 --2- 172.21.15.28:0/447577061 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f57a40a3e40 0x7f57a40a6300 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.343+0000 7f57c5a00640 1 --2- 172.21.15.28:0/447577061 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f57c0109d60 0x7f57c01a18f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.343+0000 7f57c5a00640 1 --2- 172.21.15.28:0/447577061 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f57c0076040 0x7f57c019a880 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.343+0000 7f57c5a00640 1 --2- 172.21.15.28:0/447577061 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57c0074a60 0x7f57c019a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.343+0000 7f57c5a00640 1 -- 172.21.15.28:0/447577061 >> 172.21.15.28:0/447577061 conn(0x7f57c00fd410 msgr2=0x7f57c0105c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.343+0000 7f57c5a00640 1 -- 172.21.15.28:0/447577061 shutdown_connections 2024-04-24T08:10:25.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.343+0000 7f57c5a00640 1 -- 172.21.15.28:0/447577061 wait complete. 2024-04-24T08:10:25.365 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-04-24T08:10:25.365 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-04-24T08:10:25.365 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:10:25.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.560+0000 7ff7f5800640 1 -- 172.21.15.28:0/2618265566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f0077c00 msgr2=0x7ff7f0078060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.560+0000 7ff7f5800640 1 --2- 172.21.15.28:0/2618265566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f0077c00 0x7ff7f0078060 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff7d8009900 tx=0x7ff7d802a410 comp rx=0 tx=0).stop 2024-04-24T08:10:25.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.560+0000 7ff7f5800640 1 -- 172.21.15.28:0/2618265566 shutdown_connections 2024-04-24T08:10:25.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.560+0000 7ff7f5800640 1 --2- 172.21.15.28:0/2618265566 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f0077c00 0x7ff7f0078060 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.560+0000 7ff7f5800640 1 --2- 172.21.15.28:0/2618265566 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7f00785a0 0x7ff7f0078e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.560+0000 7ff7f5800640 1 --2- 172.21.15.28:0/2618265566 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7f00769c0 0x7ff7f0076de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.560+0000 7ff7f5800640 1 -- 172.21.15.28:0/2618265566 >> 172.21.15.28:0/2618265566 conn(0x7ff7f00ff7f0 msgr2=0x7ff7f0101c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:25.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.560+0000 7ff7f5800640 1 -- 172.21.15.28:0/2618265566 shutdown_connections 2024-04-24T08:10:25.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.560+0000 7ff7f5800640 1 -- 172.21.15.28:0/2618265566 wait complete. 2024-04-24T08:10:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.561+0000 7ff7f5800640 1 Processor -- start 2024-04-24T08:10:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.562+0000 7ff7f5800640 1 -- start start 2024-04-24T08:10:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.562+0000 7ff7f5800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7f00769c0 0x7ff7f019e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.562+0000 7ff7f5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f0077c00 0x7ff7f019eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.562+0000 7ff7f5800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7f00785a0 0x7ff7f01a5f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.562+0000 7ff7f5800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f0114310 con 0x7ff7f0077c00 2024-04-24T08:10:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.562+0000 7ff7f5800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f0114190 con 0x7ff7f00769c0 2024-04-24T08:10:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.562+0000 7ff7f5800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff7f0114490 con 0x7ff7f00785a0 2024-04-24T08:10:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.562+0000 7ff7e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f0077c00 0x7ff7f019eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.562+0000 7ff7e7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f0077c00 0x7ff7f019eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33906/0 (socket says 172.21.15.28:33906) 2024-04-24T08:10:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.562+0000 7ff7e7e00640 1 -- 172.21.15.28:0/2744339140 learned_addr learned my addr 172.21.15.28:0/2744339140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.562+0000 7ff7ef400640 1 --2- 172.21.15.28:0/2744339140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7f00785a0 0x7ff7f01a5f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:25.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.563+0000 7ff7eea00640 1 --2- 172.21.15.28:0/2744339140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7f00769c0 0x7ff7f019e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:25.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.563+0000 7ff7ef400640 1 -- 172.21.15.28:0/2744339140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7f00769c0 msgr2=0x7ff7f019e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.563+0000 7ff7ef400640 1 --2- 172.21.15.28:0/2744339140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7f00769c0 0x7ff7f019e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.563+0000 7ff7ef400640 1 -- 172.21.15.28:0/2744339140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f0077c00 msgr2=0x7ff7f019eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.563+0000 7ff7ef400640 1 --2- 172.21.15.28:0/2744339140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f0077c00 0x7ff7f019eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.563+0000 7ff7ef400640 1 -- 172.21.15.28:0/2744339140 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff7f01a65a0 con 0x7ff7f00785a0 2024-04-24T08:10:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.563+0000 7ff7e7e00640 1 --2- 172.21.15.28:0/2744339140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f0077c00 0x7ff7f019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.563+0000 7ff7ef400640 1 --2- 172.21.15.28:0/2744339140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7f00785a0 0x7ff7f01a5f30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800da10 tx=0x7ff7e800ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.564+0000 7ff7e7400640 1 -- 172.21.15.28:0/2744339140 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e800bd70 con 0x7ff7f00785a0 2024-04-24T08:10:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.564+0000 7ff7e7400640 1 -- 172.21.15.28:0/2744339140 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff7e8003e50 con 0x7ff7f00785a0 2024-04-24T08:10:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.564+0000 7ff7e7400640 1 -- 172.21.15.28:0/2744339140 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff7e80106a0 con 0x7ff7f00785a0 2024-04-24T08:10:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.564+0000 7ff7f5800640 1 -- 172.21.15.28:0/2744339140 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff7f01a6890 con 0x7ff7f00785a0 2024-04-24T08:10:25.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.564+0000 7ff7f5800640 1 -- 172.21.15.28:0/2744339140 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff7f01a6db0 con 0x7ff7f00785a0 2024-04-24T08:10:25.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.565+0000 7ff7f5800640 1 -- 172.21.15.28:0/2744339140 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff7f0110dd0 con 0x7ff7f00785a0 2024-04-24T08:10:25.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.567+0000 7ff7e7400640 1 -- 172.21.15.28:0/2744339140 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7ff7e8004b40 con 0x7ff7f00785a0 2024-04-24T08:10:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.568+0000 7ff7e7400640 1 --2- 172.21.15.28:0/2744339140 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff7d40a3ec0 0x7ff7d40a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.569+0000 7ff7e7400640 1 -- 172.21.15.28:0/2744339140 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7ff7e80d1020 con 0x7ff7f00785a0 2024-04-24T08:10:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.569+0000 7ff7eea00640 1 --2- 172.21.15.28:0/2744339140 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff7d40a3ec0 0x7ff7d40a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:25.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.570+0000 7ff7eea00640 1 --2- 172.21.15.28:0/2744339140 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff7d40a3ec0 0x7ff7d40a6380 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000bbe0 tx=0x7ff7e000b390 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:25.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.571+0000 7ff7e7400640 1 -- 172.21.15.28:0/2744339140 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff7e809d4d0 con 0x7ff7f00785a0 2024-04-24T08:10:25.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.780+0000 7ff7f5800640 1 -- 172.21.15.28:0/2744339140 --> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7ff7f0062440 con 0x7ff7d40a3ec0 2024-04-24T08:10:25.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.800+0000 7ff7e7400640 1 -- 172.21.15.28:0/2744339140 <== mgr.25225 v2:172.21.15.28:6800/353036813 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258061 (secure 0 0 0) 0x7ff7f0062440 con 0x7ff7d40a3ec0 2024-04-24T08:10:25.804 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:10:25.806 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-24T08:10:25.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.805+0000 7ff7f5800640 1 -- 172.21.15.28:0/2744339140 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff7d40a3ec0 msgr2=0x7ff7d40a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.805+0000 7ff7f5800640 1 --2- 172.21.15.28:0/2744339140 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff7d40a3ec0 0x7ff7d40a6380 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7ff7e000bbe0 tx=0x7ff7e000b390 comp rx=0 tx=0).stop 2024-04-24T08:10:25.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.805+0000 7ff7f5800640 1 -- 172.21.15.28:0/2744339140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7f00785a0 msgr2=0x7ff7f01a5f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:25.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.805+0000 7ff7f5800640 1 --2- 172.21.15.28:0/2744339140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7f00785a0 0x7ff7f01a5f30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7ff7e800da10 tx=0x7ff7e800ded0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.805+0000 7ff7f5800640 1 -- 172.21.15.28:0/2744339140 shutdown_connections 2024-04-24T08:10:25.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.805+0000 7ff7f5800640 1 --2- 172.21.15.28:0/2744339140 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff7d40a3ec0 0x7ff7d40a6380 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.805+0000 7ff7f5800640 1 --2- 172.21.15.28:0/2744339140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff7f0077c00 0x7ff7f019eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.805+0000 7ff7f5800640 1 --2- 172.21.15.28:0/2744339140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff7f00785a0 0x7ff7f01a5f30 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.805+0000 7ff7f5800640 1 --2- 172.21.15.28:0/2744339140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7f00769c0 0x7ff7f019e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:25.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.805+0000 7ff7f5800640 1 -- 172.21.15.28:0/2744339140 >> 172.21.15.28:0/2744339140 conn(0x7ff7f00ff7f0 msgr2=0x7ff7f0100f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:25.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.806+0000 7ff7f5800640 1 -- 172.21.15.28:0/2744339140 shutdown_connections 2024-04-24T08:10:25.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:25.806+0000 7ff7f5800640 1 -- 172.21.15.28:0/2744339140 wait complete. 2024-04-24T08:10:25.827 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":5,"stamp":"2024-04-24T08:10:24.156306+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1185328,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1123,"num_read_kb":714,"num_write":476,"num_write_kb":4377,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1181352,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":868,"ondisk_log_size":868,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365592,"kb_used_data":12232,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576232,"statfs":{"total":1151940427776,"available":1151566061568,"internally_reserved":0,"allocated":12525568,"data_stored":8868101,"data_compressed":330175,"data_compressed_allocated":442368,"data_compressed_original":3284992,"omap_allocated":19140,"internal_metadata":361674044},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"2.978942"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.289087+0000","last_change":"2024-04-24T08:07:52.074319+0000","last_active":"2024-04-24T08:10:20.289087+0000","last_peered":"2024-04-24T08:10:20.289087+0000","last_clean":"2024-04-24T08:10:20.289087+0000","last_became_active":"2024-04-24T08:07:52.073923+0000","last_became_peered":"2024-04-24T08:07:52.073923+0000","last_unstale":"2024-04-24T08:10:20.289087+0000","last_undegraded":"2024-04-24T08:10:20.289087+0000","last_fullsized":"2024-04-24T08:10:20.289087+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:29.325904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.504231+0000","last_change":"2024-04-24T08:07:52.068316+0000","last_active":"2024-04-24T08:10:20.504231+0000","last_peered":"2024-04-24T08:10:20.504231+0000","last_clean":"2024-04-24T08:10:20.504231+0000","last_became_active":"2024-04-24T08:07:52.067961+0000","last_became_peered":"2024-04-24T08:07:52.067961+0000","last_unstale":"2024-04-24T08:10:20.504231+0000","last_undegraded":"2024-04-24T08:10:20.504231+0000","last_fullsized":"2024-04-24T08:10:20.504231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:35:14.079348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.503658+0000","last_change":"2024-04-24T08:07:52.066870+0000","last_active":"2024-04-24T08:10:20.503658+0000","last_peered":"2024-04-24T08:10:20.503658+0000","last_clean":"2024-04-24T08:10:20.503658+0000","last_became_active":"2024-04-24T08:07:52.066481+0000","last_became_peered":"2024-04-24T08:07:52.066481+0000","last_unstale":"2024-04-24T08:10:20.503658+0000","last_undegraded":"2024-04-24T08:10:20.503658+0000","last_fullsized":"2024-04-24T08:10:20.503658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:25:45.526667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.129237+0000","last_change":"2024-04-24T08:07:52.061548+0000","last_active":"2024-04-24T08:10:20.129237+0000","last_peered":"2024-04-24T08:10:20.129237+0000","last_clean":"2024-04-24T08:10:20.129237+0000","last_became_active":"2024-04-24T08:07:52.060519+0000","last_became_peered":"2024-04-24T08:07:52.060519+0000","last_unstale":"2024-04-24T08:10:20.129237+0000","last_undegraded":"2024-04-24T08:10:20.129237+0000","last_fullsized":"2024-04-24T08:10:20.129237+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:47:20.952035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128104+0000","last_change":"2024-04-24T08:07:52.063805+0000","last_active":"2024-04-24T08:10:20.128104+0000","last_peered":"2024-04-24T08:10:20.128104+0000","last_clean":"2024-04-24T08:10:20.128104+0000","last_became_active":"2024-04-24T08:07:52.063414+0000","last_became_peered":"2024-04-24T08:07:52.063414+0000","last_unstale":"2024-04-24T08:10:20.128104+0000","last_undegraded":"2024-04-24T08:10:20.128104+0000","last_fullsized":"2024-04-24T08:10:20.128104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:06:24.465035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.301125+0000","last_change":"2024-04-24T08:07:52.050152+0000","last_active":"2024-04-24T08:10:20.301125+0000","last_peered":"2024-04-24T08:10:20.301125+0000","last_clean":"2024-04-24T08:10:20.301125+0000","last_became_active":"2024-04-24T08:07:52.049805+0000","last_became_peered":"2024-04-24T08:07:52.049805+0000","last_unstale":"2024-04-24T08:10:20.301125+0000","last_undegraded":"2024-04-24T08:10:20.301125+0000","last_fullsized":"2024-04-24T08:10:20.301125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:47:34.644263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.258824+0000","last_change":"2024-04-24T08:07:52.055726+0000","last_active":"2024-04-24T08:10:20.258824+0000","last_peered":"2024-04-24T08:10:20.258824+0000","last_clean":"2024-04-24T08:10:20.258824+0000","last_became_active":"2024-04-24T08:07:52.054563+0000","last_became_peered":"2024-04-24T08:07:52.054563+0000","last_unstale":"2024-04-24T08:10:20.258824+0000","last_undegraded":"2024-04-24T08:10:20.258824+0000","last_fullsized":"2024-04-24T08:10:20.258824+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:34:57.430232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288218+0000","last_change":"2024-04-24T08:07:52.053672+0000","last_active":"2024-04-24T08:10:20.288218+0000","last_peered":"2024-04-24T08:10:20.288218+0000","last_clean":"2024-04-24T08:10:20.288218+0000","last_became_active":"2024-04-24T08:07:52.053150+0000","last_became_peered":"2024-04-24T08:07:52.053150+0000","last_unstale":"2024-04-24T08:10:20.288218+0000","last_undegraded":"2024-04-24T08:10:20.288218+0000","last_fullsized":"2024-04-24T08:10:20.288218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:00:12.272555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498877+0000","last_change":"2024-04-24T08:07:52.054648+0000","last_active":"2024-04-24T08:10:20.498877+0000","last_peered":"2024-04-24T08:10:20.498877+0000","last_clean":"2024-04-24T08:10:20.498877+0000","last_became_active":"2024-04-24T08:07:52.054301+0000","last_became_peered":"2024-04-24T08:07:52.054301+0000","last_unstale":"2024-04-24T08:10:20.498877+0000","last_undegraded":"2024-04-24T08:10:20.498877+0000","last_fullsized":"2024-04-24T08:10:20.498877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:52:20.533855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.301467+0000","last_change":"2024-04-24T08:07:52.052994+0000","last_active":"2024-04-24T08:10:20.301467+0000","last_peered":"2024-04-24T08:10:20.301467+0000","last_clean":"2024-04-24T08:10:20.301467+0000","last_became_active":"2024-04-24T08:07:52.052641+0000","last_became_peered":"2024-04-24T08:07:52.052641+0000","last_unstale":"2024-04-24T08:10:20.301467+0000","last_undegraded":"2024-04-24T08:10:20.301467+0000","last_fullsized":"2024-04-24T08:10:20.301467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:02.665622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259119+0000","last_change":"2024-04-24T08:07:52.055901+0000","last_active":"2024-04-24T08:10:20.259119+0000","last_peered":"2024-04-24T08:10:20.259119+0000","last_clean":"2024-04-24T08:10:20.259119+0000","last_became_active":"2024-04-24T08:07:52.055277+0000","last_became_peered":"2024-04-24T08:07:52.055277+0000","last_unstale":"2024-04-24T08:10:20.259119+0000","last_undegraded":"2024-04-24T08:10:20.259119+0000","last_fullsized":"2024-04-24T08:10:20.259119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:46.950578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128973+0000","last_change":"2024-04-24T08:07:52.060329+0000","last_active":"2024-04-24T08:10:20.128973+0000","last_peered":"2024-04-24T08:10:20.128973+0000","last_clean":"2024-04-24T08:10:20.128973+0000","last_became_active":"2024-04-24T08:07:52.059914+0000","last_became_peered":"2024-04-24T08:07:52.059914+0000","last_unstale":"2024-04-24T08:10:20.128973+0000","last_undegraded":"2024-04-24T08:10:20.128973+0000","last_fullsized":"2024-04-24T08:10:20.128973+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:15:20.263916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.443115+0000","last_change":"2024-04-24T08:07:52.062108+0000","last_active":"2024-04-24T08:10:20.443115+0000","last_peered":"2024-04-24T08:10:20.443115+0000","last_clean":"2024-04-24T08:10:20.443115+0000","last_became_active":"2024-04-24T08:07:52.061665+0000","last_became_peered":"2024-04-24T08:07:52.061665+0000","last_unstale":"2024-04-24T08:10:20.443115+0000","last_undegraded":"2024-04-24T08:10:20.443115+0000","last_fullsized":"2024-04-24T08:10:20.443115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:55:58.071807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.503324+0000","last_change":"2024-04-24T08:07:52.063033+0000","last_active":"2024-04-24T08:10:20.503324+0000","last_peered":"2024-04-24T08:10:20.503324+0000","last_clean":"2024-04-24T08:10:20.503324+0000","last_became_active":"2024-04-24T08:07:52.062610+0000","last_became_peered":"2024-04-24T08:07:52.062610+0000","last_unstale":"2024-04-24T08:10:20.503324+0000","last_undegraded":"2024-04-24T08:10:20.503324+0000","last_fullsized":"2024-04-24T08:10:20.503324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:59:35.906720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.503122+0000","last_change":"2024-04-24T08:07:52.064657+0000","last_active":"2024-04-24T08:10:20.503122+0000","last_peered":"2024-04-24T08:10:20.503122+0000","last_clean":"2024-04-24T08:10:20.503122+0000","last_became_active":"2024-04-24T08:07:52.064188+0000","last_became_peered":"2024-04-24T08:07:52.064188+0000","last_unstale":"2024-04-24T08:10:20.503122+0000","last_undegraded":"2024-04-24T08:10:20.503122+0000","last_fullsized":"2024-04-24T08:10:20.503122+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:56:32.139420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498229+0000","last_change":"2024-04-24T08:07:52.067227+0000","last_active":"2024-04-24T08:10:20.498229+0000","last_peered":"2024-04-24T08:10:20.498229+0000","last_clean":"2024-04-24T08:10:20.498229+0000","last_became_active":"2024-04-24T08:07:52.066854+0000","last_became_peered":"2024-04-24T08:07:52.066854+0000","last_unstale":"2024-04-24T08:10:20.498229+0000","last_undegraded":"2024-04-24T08:10:20.498229+0000","last_fullsized":"2024-04-24T08:10:20.498229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:25:17.580223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.289276+0000","last_change":"2024-04-24T08:07:52.074127+0000","last_active":"2024-04-24T08:10:20.289276+0000","last_peered":"2024-04-24T08:10:20.289276+0000","last_clean":"2024-04-24T08:10:20.289276+0000","last_became_active":"2024-04-24T08:07:52.073806+0000","last_became_peered":"2024-04-24T08:07:52.073806+0000","last_unstale":"2024-04-24T08:10:20.289276+0000","last_undegraded":"2024-04-24T08:10:20.289276+0000","last_fullsized":"2024-04-24T08:10:20.289276+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:25:22.068997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.131220+0000","last_change":"2024-04-24T08:07:52.051239+0000","last_active":"2024-04-24T08:10:20.131220+0000","last_peered":"2024-04-24T08:10:20.131220+0000","last_clean":"2024-04-24T08:10:20.131220+0000","last_became_active":"2024-04-24T08:07:52.049232+0000","last_became_peered":"2024-04-24T08:07:52.049232+0000","last_unstale":"2024-04-24T08:10:20.131220+0000","last_undegraded":"2024-04-24T08:10:20.131220+0000","last_fullsized":"2024-04-24T08:10:20.131220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:07:39.768537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.497883+0000","last_change":"2024-04-24T08:07:52.062119+0000","last_active":"2024-04-24T08:10:20.497883+0000","last_peered":"2024-04-24T08:10:20.497883+0000","last_clean":"2024-04-24T08:10:20.497883+0000","last_became_active":"2024-04-24T08:07:52.061781+0000","last_became_peered":"2024-04-24T08:07:52.061781+0000","last_unstale":"2024-04-24T08:10:20.497883+0000","last_undegraded":"2024-04-24T08:10:20.497883+0000","last_fullsized":"2024-04-24T08:10:20.497883+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:49:41.595368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.129438+0000","last_change":"2024-04-24T08:07:51.052411+0000","last_active":"2024-04-24T08:10:20.129438+0000","last_peered":"2024-04-24T08:10:20.129438+0000","last_clean":"2024-04-24T08:10:20.129438+0000","last_became_active":"2024-04-24T08:07:51.051704+0000","last_became_peered":"2024-04-24T08:07:51.051704+0000","last_unstale":"2024-04-24T08:10:20.129438+0000","last_undegraded":"2024-04-24T08:10:20.129438+0000","last_fullsized":"2024-04-24T08:10:20.129438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:34:23.001338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288658+0000","last_change":"2024-04-24T08:07:52.056508+0000","last_active":"2024-04-24T08:10:20.288658+0000","last_peered":"2024-04-24T08:10:20.288658+0000","last_clean":"2024-04-24T08:10:20.288658+0000","last_became_active":"2024-04-24T08:07:52.056158+0000","last_became_peered":"2024-04-24T08:07:52.056158+0000","last_unstale":"2024-04-24T08:10:20.288658+0000","last_undegraded":"2024-04-24T08:10:20.288658+0000","last_fullsized":"2024-04-24T08:10:20.288658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:52.267984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288783+0000","last_change":"2024-04-24T08:07:51.080403+0000","last_active":"2024-04-24T08:10:20.288783+0000","last_peered":"2024-04-24T08:10:20.288783+0000","last_clean":"2024-04-24T08:10:20.288783+0000","last_became_active":"2024-04-24T08:07:51.079158+0000","last_became_peered":"2024-04-24T08:07:51.079158+0000","last_unstale":"2024-04-24T08:10:20.288783+0000","last_undegraded":"2024-04-24T08:10:20.288783+0000","last_fullsized":"2024-04-24T08:10:20.288783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:49:59.491850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.504255+0000","last_change":"2024-04-24T08:07:52.066963+0000","last_active":"2024-04-24T08:10:20.504255+0000","last_peered":"2024-04-24T08:10:20.504255+0000","last_clean":"2024-04-24T08:10:20.504255+0000","last_became_active":"2024-04-24T08:07:52.065475+0000","last_became_peered":"2024-04-24T08:07:52.065475+0000","last_unstale":"2024-04-24T08:10:20.504255+0000","last_undegraded":"2024-04-24T08:10:20.504255+0000","last_fullsized":"2024-04-24T08:10:20.504255+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:28:34.220200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128324+0000","last_change":"2024-04-24T08:07:51.042772+0000","last_active":"2024-04-24T08:10:20.128324+0000","last_peered":"2024-04-24T08:10:20.128324+0000","last_clean":"2024-04-24T08:10:20.128324+0000","last_became_active":"2024-04-24T08:07:51.042128+0000","last_became_peered":"2024-04-24T08:07:51.042128+0000","last_unstale":"2024-04-24T08:10:20.128324+0000","last_undegraded":"2024-04-24T08:10:20.128324+0000","last_fullsized":"2024-04-24T08:10:20.128324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:10:29.207991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498210+0000","last_change":"2024-04-24T08:07:52.062411+0000","last_active":"2024-04-24T08:10:20.498210+0000","last_peered":"2024-04-24T08:10:20.498210+0000","last_clean":"2024-04-24T08:10:20.498210+0000","last_became_active":"2024-04-24T08:07:52.062037+0000","last_became_peered":"2024-04-24T08:07:52.062037+0000","last_unstale":"2024-04-24T08:10:20.498210+0000","last_undegraded":"2024-04-24T08:10:20.498210+0000","last_fullsized":"2024-04-24T08:10:20.498210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:16:17.463874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128356+0000","last_change":"2024-04-24T08:07:51.046046+0000","last_active":"2024-04-24T08:10:20.128356+0000","last_peered":"2024-04-24T08:10:20.128356+0000","last_clean":"2024-04-24T08:10:20.128356+0000","last_became_active":"2024-04-24T08:07:51.045633+0000","last_became_peered":"2024-04-24T08:07:51.045633+0000","last_unstale":"2024-04-24T08:10:20.128356+0000","last_undegraded":"2024-04-24T08:10:20.128356+0000","last_fullsized":"2024-04-24T08:10:20.128356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:18:20.529234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297863+0000","last_change":"2024-04-24T08:07:52.061167+0000","last_active":"2024-04-24T08:10:20.297863+0000","last_peered":"2024-04-24T08:10:20.297863+0000","last_clean":"2024-04-24T08:10:20.297863+0000","last_became_active":"2024-04-24T08:07:52.060841+0000","last_became_peered":"2024-04-24T08:07:52.060841+0000","last_unstale":"2024-04-24T08:10:20.297863+0000","last_undegraded":"2024-04-24T08:10:20.297863+0000","last_fullsized":"2024-04-24T08:10:20.297863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:04:27.048760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"91'9","reported_seq":184,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.502848+0000","last_change":"2024-04-24T08:07:51.044274+0000","last_active":"2024-04-24T08:10:20.502848+0000","last_peered":"2024-04-24T08:10:20.502848+0000","last_clean":"2024-04-24T08:10:20.502848+0000","last_became_active":"2024-04-24T08:07:51.043938+0000","last_became_peered":"2024-04-24T08:07:51.043938+0000","last_unstale":"2024-04-24T08:10:20.502848+0000","last_undegraded":"2024-04-24T08:10:20.502848+0000","last_fullsized":"2024-04-24T08:10:20.502848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:05:38.349981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1855,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128148+0000","last_change":"2024-04-24T08:07:52.055777+0000","last_active":"2024-04-24T08:10:20.128148+0000","last_peered":"2024-04-24T08:10:20.128148+0000","last_clean":"2024-04-24T08:10:20.128148+0000","last_became_active":"2024-04-24T08:07:52.055415+0000","last_became_peered":"2024-04-24T08:07:52.055415+0000","last_unstale":"2024-04-24T08:10:20.128148+0000","last_undegraded":"2024-04-24T08:10:20.128148+0000","last_fullsized":"2024-04-24T08:10:20.128148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:18:31.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.301075+0000","last_change":"2024-04-24T08:07:51.072513+0000","last_active":"2024-04-24T08:10:20.301075+0000","last_peered":"2024-04-24T08:10:20.301075+0000","last_clean":"2024-04-24T08:10:20.301075+0000","last_became_active":"2024-04-24T08:07:51.072249+0000","last_became_peered":"2024-04-24T08:07:51.072249+0000","last_unstale":"2024-04-24T08:10:20.301075+0000","last_undegraded":"2024-04-24T08:10:20.301075+0000","last_fullsized":"2024-04-24T08:10:20.301075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:45:07.112617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297787+0000","last_change":"2024-04-24T08:08:51.587130+0000","last_active":"2024-04-24T08:10:20.297787+0000","last_peered":"2024-04-24T08:10:20.297787+0000","last_clean":"2024-04-24T08:10:20.297787+0000","last_became_active":"2024-04-24T08:08:51.586645+0000","last_became_peered":"2024-04-24T08:08:51.586645+0000","last_unstale":"2024-04-24T08:10:20.297787+0000","last_undegraded":"2024-04-24T08:10:20.297787+0000","last_fullsized":"2024-04-24T08:10:20.297787+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:24:14.677548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.443164+0000","last_change":"2024-04-24T08:07:51.070901+0000","last_active":"2024-04-24T08:10:20.443164+0000","last_peered":"2024-04-24T08:10:20.443164+0000","last_clean":"2024-04-24T08:10:20.443164+0000","last_became_active":"2024-04-24T08:07:51.067178+0000","last_became_peered":"2024-04-24T08:07:51.067178+0000","last_unstale":"2024-04-24T08:10:20.443164+0000","last_undegraded":"2024-04-24T08:10:20.443164+0000","last_fullsized":"2024-04-24T08:10:20.443164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:24:26.807786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288202+0000","last_change":"2024-04-24T08:07:52.079493+0000","last_active":"2024-04-24T08:10:20.288202+0000","last_peered":"2024-04-24T08:10:20.288202+0000","last_clean":"2024-04-24T08:10:20.288202+0000","last_became_active":"2024-04-24T08:07:52.077266+0000","last_became_peered":"2024-04-24T08:07:52.077266+0000","last_unstale":"2024-04-24T08:10:20.288202+0000","last_undegraded":"2024-04-24T08:10:20.288202+0000","last_fullsized":"2024-04-24T08:10:20.288202+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:40:45.660146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297546+0000","last_change":"2024-04-24T08:07:51.045963+0000","last_active":"2024-04-24T08:10:20.297546+0000","last_peered":"2024-04-24T08:10:20.297546+0000","last_clean":"2024-04-24T08:10:20.297546+0000","last_became_active":"2024-04-24T08:07:51.045641+0000","last_became_peered":"2024-04-24T08:07:51.045641+0000","last_unstale":"2024-04-24T08:10:20.297546+0000","last_undegraded":"2024-04-24T08:10:20.297546+0000","last_fullsized":"2024-04-24T08:10:20.297546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:07:59.646155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128896+0000","last_change":"2024-04-24T08:07:52.057280+0000","last_active":"2024-04-24T08:10:20.128896+0000","last_peered":"2024-04-24T08:10:20.128896+0000","last_clean":"2024-04-24T08:10:20.128896+0000","last_became_active":"2024-04-24T08:07:52.054944+0000","last_became_peered":"2024-04-24T08:07:52.054944+0000","last_unstale":"2024-04-24T08:10:20.128896+0000","last_undegraded":"2024-04-24T08:10:20.128896+0000","last_fullsized":"2024-04-24T08:10:20.128896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:22:45.911472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":175,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.372717+0000","last_change":"2024-04-24T08:07:51.048898+0000","last_active":"2024-04-24T08:10:20.372717+0000","last_peered":"2024-04-24T08:10:20.372717+0000","last_clean":"2024-04-24T08:10:20.372717+0000","last_became_active":"2024-04-24T08:07:51.048585+0000","last_became_peered":"2024-04-24T08:07:51.048585+0000","last_unstale":"2024-04-24T08:10:20.372717+0000","last_undegraded":"2024-04-24T08:10:20.372717+0000","last_fullsized":"2024-04-24T08:10:20.372717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:40:30.413541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498865+0000","last_change":"2024-04-24T08:07:52.048093+0000","last_active":"2024-04-24T08:10:20.498865+0000","last_peered":"2024-04-24T08:10:20.498865+0000","last_clean":"2024-04-24T08:10:20.498865+0000","last_became_active":"2024-04-24T08:07:52.047727+0000","last_became_peered":"2024-04-24T08:07:52.047727+0000","last_unstale":"2024-04-24T08:10:20.498865+0000","last_undegraded":"2024-04-24T08:10:20.498865+0000","last_fullsized":"2024-04-24T08:10:20.498865+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:57:26.248355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259256+0000","last_change":"2024-04-24T08:07:51.047810+0000","last_active":"2024-04-24T08:10:20.259256+0000","last_peered":"2024-04-24T08:10:20.259256+0000","last_clean":"2024-04-24T08:10:20.259256+0000","last_became_active":"2024-04-24T08:07:51.047501+0000","last_became_peered":"2024-04-24T08:07:51.047501+0000","last_unstale":"2024-04-24T08:10:20.259256+0000","last_undegraded":"2024-04-24T08:10:20.259256+0000","last_fullsized":"2024-04-24T08:10:20.259256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:02:27.585202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.504166+0000","last_change":"2024-04-24T08:07:52.063782+0000","last_active":"2024-04-24T08:10:20.504166+0000","last_peered":"2024-04-24T08:10:20.504166+0000","last_clean":"2024-04-24T08:10:20.504166+0000","last_became_active":"2024-04-24T08:07:52.063056+0000","last_became_peered":"2024-04-24T08:07:52.063056+0000","last_unstale":"2024-04-24T08:10:20.504166+0000","last_undegraded":"2024-04-24T08:10:20.504166+0000","last_fullsized":"2024-04-24T08:10:20.504166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:41:30.966787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.298089+0000","last_change":"2024-04-24T08:07:51.076139+0000","last_active":"2024-04-24T08:10:20.298089+0000","last_peered":"2024-04-24T08:10:20.298089+0000","last_clean":"2024-04-24T08:10:20.298089+0000","last_became_active":"2024-04-24T08:07:51.075842+0000","last_became_peered":"2024-04-24T08:07:51.075842+0000","last_unstale":"2024-04-24T08:10:20.298089+0000","last_undegraded":"2024-04-24T08:10:20.298089+0000","last_fullsized":"2024-04-24T08:10:20.298089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:46:56.741726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498434+0000","last_change":"2024-04-24T08:08:51.250881+0000","last_active":"2024-04-24T08:10:20.498434+0000","last_peered":"2024-04-24T08:10:20.498434+0000","last_clean":"2024-04-24T08:10:20.498434+0000","last_became_active":"2024-04-24T08:08:51.250519+0000","last_became_peered":"2024-04-24T08:08:51.250519+0000","last_unstale":"2024-04-24T08:10:20.498434+0000","last_undegraded":"2024-04-24T08:10:20.498434+0000","last_fullsized":"2024-04-24T08:10:20.498434+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:23:04.805687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'34","reported_seq":357,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.497897+0000","last_change":"2024-04-24T08:07:51.048079+0000","last_active":"2024-04-24T08:10:20.497897+0000","last_peered":"2024-04-24T08:10:20.497897+0000","last_clean":"2024-04-24T08:10:20.497897+0000","last_became_active":"2024-04-24T08:07:51.047584+0000","last_became_peered":"2024-04-24T08:07:51.047584+0000","last_unstale":"2024-04-24T08:10:20.497897+0000","last_undegraded":"2024-04-24T08:10:20.497897+0000","last_fullsized":"2024-04-24T08:10:20.497897+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":34,"log_dups_size":0,"ondisk_log_size":34,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:09:05.554984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":64144,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":297,"num_read_kb":206,"num_write":35,"num_write_kb":77,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128091+0000","last_change":"2024-04-24T08:07:52.055894+0000","last_active":"2024-04-24T08:10:20.128091+0000","last_peered":"2024-04-24T08:10:20.128091+0000","last_clean":"2024-04-24T08:10:20.128091+0000","last_became_active":"2024-04-24T08:07:52.055548+0000","last_became_peered":"2024-04-24T08:07:52.055548+0000","last_unstale":"2024-04-24T08:10:20.128091+0000","last_undegraded":"2024-04-24T08:10:20.128091+0000","last_fullsized":"2024-04-24T08:10:20.128091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:33:22.609865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288830+0000","last_change":"2024-04-24T08:08:51.188459+0000","last_active":"2024-04-24T08:10:20.288830+0000","last_peered":"2024-04-24T08:10:20.288830+0000","last_clean":"2024-04-24T08:10:20.288830+0000","last_became_active":"2024-04-24T08:08:51.180387+0000","last_became_peered":"2024-04-24T08:08:51.180387+0000","last_unstale":"2024-04-24T08:10:20.288830+0000","last_undegraded":"2024-04-24T08:10:20.288830+0000","last_fullsized":"2024-04-24T08:10:20.288830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:34:56.592978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.443550+0000","last_change":"2024-04-24T08:07:52.069950+0000","last_active":"2024-04-24T08:10:20.443550+0000","last_peered":"2024-04-24T08:10:20.443550+0000","last_clean":"2024-04-24T08:10:20.443550+0000","last_became_active":"2024-04-24T08:07:52.069515+0000","last_became_peered":"2024-04-24T08:07:52.069515+0000","last_unstale":"2024-04-24T08:10:20.443550+0000","last_undegraded":"2024-04-24T08:10:20.443550+0000","last_fullsized":"2024-04-24T08:10:20.443550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:52:27.530541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297642+0000","last_change":"2024-04-24T08:07:51.067062+0000","last_active":"2024-04-24T08:10:20.297642+0000","last_peered":"2024-04-24T08:10:20.297642+0000","last_clean":"2024-04-24T08:10:20.297642+0000","last_became_active":"2024-04-24T08:07:51.066712+0000","last_became_peered":"2024-04-24T08:07:51.066712+0000","last_unstale":"2024-04-24T08:10:20.297642+0000","last_undegraded":"2024-04-24T08:10:20.297642+0000","last_fullsized":"2024-04-24T08:10:20.297642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:11:52.974470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259346+0000","last_change":"2024-04-24T08:07:52.064521+0000","last_active":"2024-04-24T08:10:20.259346+0000","last_peered":"2024-04-24T08:10:20.259346+0000","last_clean":"2024-04-24T08:10:20.259346+0000","last_became_active":"2024-04-24T08:07:52.064184+0000","last_became_peered":"2024-04-24T08:07:52.064184+0000","last_unstale":"2024-04-24T08:10:20.259346+0000","last_undegraded":"2024-04-24T08:10:20.259346+0000","last_fullsized":"2024-04-24T08:10:20.259346+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:00:27.294441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'6","reported_seq":180,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498217+0000","last_change":"2024-04-24T08:07:51.043577+0000","last_active":"2024-04-24T08:10:20.498217+0000","last_peered":"2024-04-24T08:10:20.498217+0000","last_clean":"2024-04-24T08:10:20.498217+0000","last_became_active":"2024-04-24T08:07:51.043215+0000","last_became_peered":"2024-04-24T08:07:51.043215+0000","last_unstale":"2024-04-24T08:10:20.498217+0000","last_undegraded":"2024-04-24T08:10:20.498217+0000","last_fullsized":"2024-04-24T08:10:20.498217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:25:09.979236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1511,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":5,"num_write":5,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.258879+0000","last_change":"2024-04-24T08:07:52.057633+0000","last_active":"2024-04-24T08:10:20.258879+0000","last_peered":"2024-04-24T08:10:20.258879+0000","last_clean":"2024-04-24T08:10:20.258879+0000","last_became_active":"2024-04-24T08:07:52.057309+0000","last_became_peered":"2024-04-24T08:07:52.057309+0000","last_unstale":"2024-04-24T08:10:20.258879+0000","last_undegraded":"2024-04-24T08:10:20.258879+0000","last_fullsized":"2024-04-24T08:10:20.258879+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:49:56.045030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.127851+0000","last_change":"2024-04-24T08:07:51.058957+0000","last_active":"2024-04-24T08:10:20.127851+0000","last_peered":"2024-04-24T08:10:20.127851+0000","last_clean":"2024-04-24T08:10:20.127851+0000","last_became_active":"2024-04-24T08:07:51.058654+0000","last_became_peered":"2024-04-24T08:07:51.058654+0000","last_unstale":"2024-04-24T08:10:20.127851+0000","last_undegraded":"2024-04-24T08:10:20.127851+0000","last_fullsized":"2024-04-24T08:10:20.127851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:13.354590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297541+0000","last_change":"2024-04-24T08:07:52.051052+0000","last_active":"2024-04-24T08:10:20.297541+0000","last_peered":"2024-04-24T08:10:20.297541+0000","last_clean":"2024-04-24T08:10:20.297541+0000","last_became_active":"2024-04-24T08:07:52.050685+0000","last_became_peered":"2024-04-24T08:07:52.050685+0000","last_unstale":"2024-04-24T08:10:20.297541+0000","last_undegraded":"2024-04-24T08:10:20.297541+0000","last_fullsized":"2024-04-24T08:10:20.297541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:34.491653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128105+0000","last_change":"2024-04-24T08:07:51.041088+0000","last_active":"2024-04-24T08:10:20.128105+0000","last_peered":"2024-04-24T08:10:20.128105+0000","last_clean":"2024-04-24T08:10:20.128105+0000","last_became_active":"2024-04-24T08:07:51.040752+0000","last_became_peered":"2024-04-24T08:07:51.040752+0000","last_unstale":"2024-04-24T08:10:20.128105+0000","last_undegraded":"2024-04-24T08:10:20.128105+0000","last_fullsized":"2024-04-24T08:10:20.128105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:48:23.405705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.289018+0000","last_change":"2024-04-24T08:08:51.173331+0000","last_active":"2024-04-24T08:10:20.289018+0000","last_peered":"2024-04-24T08:10:20.289018+0000","last_clean":"2024-04-24T08:10:20.289018+0000","last_became_active":"2024-04-24T08:08:51.172912+0000","last_became_peered":"2024-04-24T08:08:51.172912+0000","last_unstale":"2024-04-24T08:10:20.289018+0000","last_undegraded":"2024-04-24T08:10:20.289018+0000","last_fullsized":"2024-04-24T08:10:20.289018+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:54:35.241315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.134622+0000","last_change":"2024-04-24T08:07:51.075583+0000","last_active":"2024-04-24T08:10:20.134622+0000","last_peered":"2024-04-24T08:10:20.134622+0000","last_clean":"2024-04-24T08:10:20.134622+0000","last_became_active":"2024-04-24T08:07:51.075111+0000","last_became_peered":"2024-04-24T08:07:51.075111+0000","last_unstale":"2024-04-24T08:10:20.134622+0000","last_undegraded":"2024-04-24T08:10:20.134622+0000","last_fullsized":"2024-04-24T08:10:20.134622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:56:00.820057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.301341+0000","last_change":"2024-04-24T08:07:52.063232+0000","last_active":"2024-04-24T08:10:20.301341+0000","last_peered":"2024-04-24T08:10:20.301341+0000","last_clean":"2024-04-24T08:10:20.301341+0000","last_became_active":"2024-04-24T08:07:52.062868+0000","last_became_peered":"2024-04-24T08:07:52.062868+0000","last_unstale":"2024-04-24T08:10:20.301341+0000","last_undegraded":"2024-04-24T08:10:20.301341+0000","last_fullsized":"2024-04-24T08:10:20.301341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:00:40.333389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.258895+0000","last_change":"2024-04-24T08:07:51.070647+0000","last_active":"2024-04-24T08:10:20.258895+0000","last_peered":"2024-04-24T08:10:20.258895+0000","last_clean":"2024-04-24T08:10:20.258895+0000","last_became_active":"2024-04-24T08:07:51.070298+0000","last_became_peered":"2024-04-24T08:07:51.070298+0000","last_unstale":"2024-04-24T08:10:20.258895+0000","last_undegraded":"2024-04-24T08:10:20.258895+0000","last_fullsized":"2024-04-24T08:10:20.258895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:56:41.706796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.301283+0000","last_change":"2024-04-24T08:07:52.073417+0000","last_active":"2024-04-24T08:10:20.301283+0000","last_peered":"2024-04-24T08:10:20.301283+0000","last_clean":"2024-04-24T08:10:20.301283+0000","last_became_active":"2024-04-24T08:07:52.073051+0000","last_became_peered":"2024-04-24T08:07:52.073051+0000","last_unstale":"2024-04-24T08:10:20.301283+0000","last_undegraded":"2024-04-24T08:10:20.301283+0000","last_fullsized":"2024-04-24T08:10:20.301283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:54:58.338279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128064+0000","last_change":"2024-04-24T08:08:51.172525+0000","last_active":"2024-04-24T08:10:20.128064+0000","last_peered":"2024-04-24T08:10:20.128064+0000","last_clean":"2024-04-24T08:10:20.128064+0000","last_became_active":"2024-04-24T08:08:51.171924+0000","last_became_peered":"2024-04-24T08:08:51.171924+0000","last_unstale":"2024-04-24T08:10:20.128064+0000","last_undegraded":"2024-04-24T08:10:20.128064+0000","last_fullsized":"2024-04-24T08:10:20.128064+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:25:42.067150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":176,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.443285+0000","last_change":"2024-04-24T08:07:51.076937+0000","last_active":"2024-04-24T08:10:20.443285+0000","last_peered":"2024-04-24T08:10:20.443285+0000","last_clean":"2024-04-24T08:10:20.443285+0000","last_became_active":"2024-04-24T08:07:51.076621+0000","last_became_peered":"2024-04-24T08:07:51.076621+0000","last_unstale":"2024-04-24T08:10:20.443285+0000","last_undegraded":"2024-04-24T08:10:20.443285+0000","last_fullsized":"2024-04-24T08:10:20.443285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:09.966994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288284+0000","last_change":"2024-04-24T08:07:52.054150+0000","last_active":"2024-04-24T08:10:20.288284+0000","last_peered":"2024-04-24T08:10:20.288284+0000","last_clean":"2024-04-24T08:10:20.288284+0000","last_became_active":"2024-04-24T08:07:52.053817+0000","last_became_peered":"2024-04-24T08:07:52.053817+0000","last_unstale":"2024-04-24T08:10:20.288284+0000","last_undegraded":"2024-04-24T08:10:20.288284+0000","last_fullsized":"2024-04-24T08:10:20.288284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:46:16.570004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128899+0000","last_change":"2024-04-24T08:07:51.045431+0000","last_active":"2024-04-24T08:10:20.128899+0000","last_peered":"2024-04-24T08:10:20.128899+0000","last_clean":"2024-04-24T08:10:20.128899+0000","last_became_active":"2024-04-24T08:07:51.045112+0000","last_became_peered":"2024-04-24T08:07:51.045112+0000","last_unstale":"2024-04-24T08:10:20.128899+0000","last_undegraded":"2024-04-24T08:10:20.128899+0000","last_fullsized":"2024-04-24T08:10:20.128899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:52:05.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":358,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.258842+0000","last_change":"2024-04-24T08:06:31.583310+0000","last_active":"2024-04-24T08:10:20.258842+0000","last_peered":"2024-04-24T08:10:20.258842+0000","last_clean":"2024-04-24T08:10:20.258842+0000","last_became_active":"2024-04-24T08:06:31.582735+0000","last_became_peered":"2024-04-24T08:06:31.582735+0000","last_unstale":"2024-04-24T08:10:20.258842+0000","last_undegraded":"2024-04-24T08:10:20.258842+0000","last_fullsized":"2024-04-24T08:10:20.258842+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:02:34.411025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:02:34.411025+0000","last_clean_scrub_stamp":"2024-04-24T08:02:34.411025+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:48.816092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.442781+0000","last_change":"2024-04-24T08:07:52.063961+0000","last_active":"2024-04-24T08:10:20.442781+0000","last_peered":"2024-04-24T08:10:20.442781+0000","last_clean":"2024-04-24T08:10:20.442781+0000","last_became_active":"2024-04-24T08:07:52.063608+0000","last_became_peered":"2024-04-24T08:07:52.063608+0000","last_unstale":"2024-04-24T08:10:20.442781+0000","last_undegraded":"2024-04-24T08:10:20.442781+0000","last_fullsized":"2024-04-24T08:10:20.442781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:40:50.233349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259215+0000","last_change":"2024-04-24T08:07:51.045626+0000","last_active":"2024-04-24T08:10:20.259215+0000","last_peered":"2024-04-24T08:10:20.259215+0000","last_clean":"2024-04-24T08:10:20.259215+0000","last_became_active":"2024-04-24T08:07:51.045293+0000","last_became_peered":"2024-04-24T08:07:51.045293+0000","last_unstale":"2024-04-24T08:10:20.259215+0000","last_undegraded":"2024-04-24T08:10:20.259215+0000","last_fullsized":"2024-04-24T08:10:20.259215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:13:04.601331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128055+0000","last_change":"2024-04-24T08:07:51.079228+0000","last_active":"2024-04-24T08:10:20.128055+0000","last_peered":"2024-04-24T08:10:20.128055+0000","last_clean":"2024-04-24T08:10:20.128055+0000","last_became_active":"2024-04-24T08:07:51.078932+0000","last_became_peered":"2024-04-24T08:07:51.078932+0000","last_unstale":"2024-04-24T08:10:20.128055+0000","last_undegraded":"2024-04-24T08:10:20.128055+0000","last_fullsized":"2024-04-24T08:10:20.128055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:58:32.024133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.442826+0000","last_change":"2024-04-24T08:07:52.065943+0000","last_active":"2024-04-24T08:10:20.442826+0000","last_peered":"2024-04-24T08:10:20.442826+0000","last_clean":"2024-04-24T08:10:20.442826+0000","last_became_active":"2024-04-24T08:07:52.062810+0000","last_became_peered":"2024-04-24T08:07:52.062810+0000","last_unstale":"2024-04-24T08:10:20.442826+0000","last_undegraded":"2024-04-24T08:10:20.442826+0000","last_fullsized":"2024-04-24T08:10:20.442826+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:40:08.803652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'45","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.444289+0000","last_change":"2024-04-24T08:08:51.193009+0000","last_active":"2024-04-24T08:10:20.444289+0000","last_peered":"2024-04-24T08:10:20.444289+0000","last_clean":"2024-04-24T08:10:20.444289+0000","last_became_active":"2024-04-24T08:08:51.180025+0000","last_became_peered":"2024-04-24T08:08:51.180025+0000","last_unstale":"2024-04-24T08:10:20.444289+0000","last_undegraded":"2024-04-24T08:10:20.444289+0000","last_fullsized":"2024-04-24T08:10:20.444289+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:31:19.397470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":81,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288903+0000","last_change":"2024-04-24T08:07:52.058764+0000","last_active":"2024-04-24T08:10:20.288903+0000","last_peered":"2024-04-24T08:10:20.288903+0000","last_clean":"2024-04-24T08:10:20.288903+0000","last_became_active":"2024-04-24T08:07:52.058336+0000","last_became_peered":"2024-04-24T08:07:52.058336+0000","last_unstale":"2024-04-24T08:10:20.288903+0000","last_undegraded":"2024-04-24T08:10:20.288903+0000","last_fullsized":"2024-04-24T08:10:20.288903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:05:37.245521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.129060+0000","last_change":"2024-04-24T08:07:51.067154+0000","last_active":"2024-04-24T08:10:20.129060+0000","last_peered":"2024-04-24T08:10:20.129060+0000","last_clean":"2024-04-24T08:10:20.129060+0000","last_became_active":"2024-04-24T08:07:51.066769+0000","last_became_peered":"2024-04-24T08:07:51.066769+0000","last_unstale":"2024-04-24T08:10:20.129060+0000","last_undegraded":"2024-04-24T08:10:20.129060+0000","last_fullsized":"2024-04-24T08:10:20.129060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:39:25.908189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.504066+0000","last_change":"2024-04-24T08:07:52.072134+0000","last_active":"2024-04-24T08:10:20.504066+0000","last_peered":"2024-04-24T08:10:20.504066+0000","last_clean":"2024-04-24T08:10:20.504066+0000","last_became_active":"2024-04-24T08:07:52.071703+0000","last_became_peered":"2024-04-24T08:07:52.071703+0000","last_unstale":"2024-04-24T08:10:20.504066+0000","last_undegraded":"2024-04-24T08:10:20.504066+0000","last_fullsized":"2024-04-24T08:10:20.504066+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:59.829988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498172+0000","last_change":"2024-04-24T08:07:51.059151+0000","last_active":"2024-04-24T08:10:20.498172+0000","last_peered":"2024-04-24T08:10:20.498172+0000","last_clean":"2024-04-24T08:10:20.498172+0000","last_became_active":"2024-04-24T08:07:51.058872+0000","last_became_peered":"2024-04-24T08:07:51.058872+0000","last_unstale":"2024-04-24T08:10:20.498172+0000","last_undegraded":"2024-04-24T08:10:20.498172+0000","last_fullsized":"2024-04-24T08:10:20.498172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:36.023595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.127663+0000","last_change":"2024-04-24T08:07:52.056919+0000","last_active":"2024-04-24T08:10:20.127663+0000","last_peered":"2024-04-24T08:10:20.127663+0000","last_clean":"2024-04-24T08:10:20.127663+0000","last_became_active":"2024-04-24T08:07:52.056573+0000","last_became_peered":"2024-04-24T08:07:52.056573+0000","last_unstale":"2024-04-24T08:10:20.127663+0000","last_undegraded":"2024-04-24T08:10:20.127663+0000","last_fullsized":"2024-04-24T08:10:20.127663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:58:17.939465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288463+0000","last_change":"2024-04-24T08:07:51.056325+0000","last_active":"2024-04-24T08:10:20.288463+0000","last_peered":"2024-04-24T08:10:20.288463+0000","last_clean":"2024-04-24T08:10:20.288463+0000","last_became_active":"2024-04-24T08:07:51.055862+0000","last_became_peered":"2024-04-24T08:07:51.055862+0000","last_unstale":"2024-04-24T08:10:20.288463+0000","last_undegraded":"2024-04-24T08:10:20.288463+0000","last_fullsized":"2024-04-24T08:10:20.288463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:11:44.487530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288781+0000","last_change":"2024-04-24T08:07:52.065570+0000","last_active":"2024-04-24T08:10:20.288781+0000","last_peered":"2024-04-24T08:10:20.288781+0000","last_clean":"2024-04-24T08:10:20.288781+0000","last_became_active":"2024-04-24T08:07:52.065196+0000","last_became_peered":"2024-04-24T08:07:52.065196+0000","last_unstale":"2024-04-24T08:10:20.288781+0000","last_undegraded":"2024-04-24T08:10:20.288781+0000","last_fullsized":"2024-04-24T08:10:20.288781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:48:09.474159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.497754+0000","last_change":"2024-04-24T08:07:51.067187+0000","last_active":"2024-04-24T08:10:20.497754+0000","last_peered":"2024-04-24T08:10:20.497754+0000","last_clean":"2024-04-24T08:10:20.497754+0000","last_became_active":"2024-04-24T08:07:51.066808+0000","last_became_peered":"2024-04-24T08:07:51.066808+0000","last_unstale":"2024-04-24T08:10:20.497754+0000","last_undegraded":"2024-04-24T08:10:20.497754+0000","last_fullsized":"2024-04-24T08:10:20.497754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:32:11.852191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288593+0000","last_change":"2024-04-24T08:07:52.078426+0000","last_active":"2024-04-24T08:10:20.288593+0000","last_peered":"2024-04-24T08:10:20.288593+0000","last_clean":"2024-04-24T08:10:20.288593+0000","last_became_active":"2024-04-24T08:07:52.078076+0000","last_became_peered":"2024-04-24T08:07:52.078076+0000","last_unstale":"2024-04-24T08:10:20.288593+0000","last_undegraded":"2024-04-24T08:10:20.288593+0000","last_fullsized":"2024-04-24T08:10:20.288593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:55:29.885688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.127956+0000","last_change":"2024-04-24T08:07:51.042601+0000","last_active":"2024-04-24T08:10:20.127956+0000","last_peered":"2024-04-24T08:10:20.127956+0000","last_clean":"2024-04-24T08:10:20.127956+0000","last_became_active":"2024-04-24T08:07:51.042252+0000","last_became_peered":"2024-04-24T08:07:51.042252+0000","last_unstale":"2024-04-24T08:10:20.127956+0000","last_undegraded":"2024-04-24T08:10:20.127956+0000","last_fullsized":"2024-04-24T08:10:20.127956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:11:44.640303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259102+0000","last_change":"2024-04-24T08:07:52.063142+0000","last_active":"2024-04-24T08:10:20.259102+0000","last_peered":"2024-04-24T08:10:20.259102+0000","last_clean":"2024-04-24T08:10:20.259102+0000","last_became_active":"2024-04-24T08:07:52.062263+0000","last_became_peered":"2024-04-24T08:07:52.062263+0000","last_unstale":"2024-04-24T08:10:20.259102+0000","last_undegraded":"2024-04-24T08:10:20.259102+0000","last_fullsized":"2024-04-24T08:10:20.259102+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:38.879380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.129102+0000","last_change":"2024-04-24T08:08:51.184795+0000","last_active":"2024-04-24T08:10:20.129102+0000","last_peered":"2024-04-24T08:10:20.129102+0000","last_clean":"2024-04-24T08:10:20.129102+0000","last_became_active":"2024-04-24T08:08:51.179560+0000","last_became_peered":"2024-04-24T08:08:51.179560+0000","last_unstale":"2024-04-24T08:10:20.129102+0000","last_undegraded":"2024-04-24T08:10:20.129102+0000","last_fullsized":"2024-04-24T08:10:20.129102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:31:23.795126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259332+0000","last_change":"2024-04-24T08:07:52.055958+0000","last_active":"2024-04-24T08:10:20.259332+0000","last_peered":"2024-04-24T08:10:20.259332+0000","last_clean":"2024-04-24T08:10:20.259332+0000","last_became_active":"2024-04-24T08:07:52.055598+0000","last_became_peered":"2024-04-24T08:07:52.055598+0000","last_unstale":"2024-04-24T08:10:20.259332+0000","last_undegraded":"2024-04-24T08:10:20.259332+0000","last_fullsized":"2024-04-24T08:10:20.259332+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:12:22.452576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'39","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.439516+0000","last_change":"2024-04-24T08:07:51.073234+0000","last_active":"2024-04-24T08:10:20.439516+0000","last_peered":"2024-04-24T08:10:20.439516+0000","last_clean":"2024-04-24T08:10:20.439516+0000","last_became_active":"2024-04-24T08:07:51.072887+0000","last_became_peered":"2024-04-24T08:07:51.072887+0000","last_unstale":"2024-04-24T08:10:20.439516+0000","last_undegraded":"2024-04-24T08:10:20.439516+0000","last_fullsized":"2024-04-24T08:10:20.439516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:28:15.908859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259151+0000","last_change":"2024-04-24T08:07:52.061188+0000","last_active":"2024-04-24T08:10:20.259151+0000","last_peered":"2024-04-24T08:10:20.259151+0000","last_clean":"2024-04-24T08:10:20.259151+0000","last_became_active":"2024-04-24T08:07:52.059616+0000","last_became_peered":"2024-04-24T08:07:52.059616+0000","last_unstale":"2024-04-24T08:10:20.259151+0000","last_undegraded":"2024-04-24T08:10:20.259151+0000","last_fullsized":"2024-04-24T08:10:20.259151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:43.546091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.442965+0000","last_change":"2024-04-24T08:07:51.078300+0000","last_active":"2024-04-24T08:10:20.442965+0000","last_peered":"2024-04-24T08:10:20.442965+0000","last_clean":"2024-04-24T08:10:20.442965+0000","last_became_active":"2024-04-24T08:07:51.076026+0000","last_became_peered":"2024-04-24T08:07:51.076026+0000","last_unstale":"2024-04-24T08:10:20.442965+0000","last_undegraded":"2024-04-24T08:10:20.442965+0000","last_fullsized":"2024-04-24T08:10:20.442965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:45:31.494213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259123+0000","last_change":"2024-04-24T08:07:52.056148+0000","last_active":"2024-04-24T08:10:20.259123+0000","last_peered":"2024-04-24T08:10:20.259123+0000","last_clean":"2024-04-24T08:10:20.259123+0000","last_became_active":"2024-04-24T08:07:52.055768+0000","last_became_peered":"2024-04-24T08:07:52.055768+0000","last_unstale":"2024-04-24T08:10:20.259123+0000","last_undegraded":"2024-04-24T08:10:20.259123+0000","last_fullsized":"2024-04-24T08:10:20.259123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:52:48.503479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288880+0000","last_change":"2024-04-24T08:07:52.056611+0000","last_active":"2024-04-24T08:10:20.288880+0000","last_peered":"2024-04-24T08:10:20.288880+0000","last_clean":"2024-04-24T08:10:20.288880+0000","last_became_active":"2024-04-24T08:07:52.055788+0000","last_became_peered":"2024-04-24T08:07:52.055788+0000","last_unstale":"2024-04-24T08:10:20.288880+0000","last_undegraded":"2024-04-24T08:10:20.288880+0000","last_fullsized":"2024-04-24T08:10:20.288880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:21:28.997588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.497715+0000","last_change":"2024-04-24T08:08:51.177574+0000","last_active":"2024-04-24T08:10:20.497715+0000","last_peered":"2024-04-24T08:10:20.497715+0000","last_clean":"2024-04-24T08:10:20.497715+0000","last_became_active":"2024-04-24T08:08:51.177145+0000","last_became_peered":"2024-04-24T08:08:51.177145+0000","last_unstale":"2024-04-24T08:10:20.497715+0000","last_undegraded":"2024-04-24T08:10:20.497715+0000","last_fullsized":"2024-04-24T08:10:20.497715+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:49:19.438233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.497647+0000","last_change":"2024-04-24T08:09:51.441560+0000","last_active":"2024-04-24T08:10:20.497647+0000","last_peered":"2024-04-24T08:10:20.497647+0000","last_clean":"2024-04-24T08:10:20.497647+0000","last_became_active":"2024-04-24T08:09:51.441192+0000","last_became_peered":"2024-04-24T08:09:51.441192+0000","last_unstale":"2024-04-24T08:10:20.497647+0000","last_undegraded":"2024-04-24T08:10:20.497647+0000","last_fullsized":"2024-04-24T08:10:20.497647+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:32:26.311164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.503483+0000","last_change":"2024-04-24T08:08:51.250206+0000","last_active":"2024-04-24T08:10:20.503483+0000","last_peered":"2024-04-24T08:10:20.503483+0000","last_clean":"2024-04-24T08:10:20.503483+0000","last_became_active":"2024-04-24T08:08:51.249677+0000","last_became_peered":"2024-04-24T08:08:51.249677+0000","last_unstale":"2024-04-24T08:10:20.503483+0000","last_undegraded":"2024-04-24T08:10:20.503483+0000","last_fullsized":"2024-04-24T08:10:20.503483+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:06:26.655987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.442885+0000","last_change":"2024-04-24T08:07:52.078604+0000","last_active":"2024-04-24T08:10:20.442885+0000","last_peered":"2024-04-24T08:10:20.442885+0000","last_clean":"2024-04-24T08:10:20.442885+0000","last_became_active":"2024-04-24T08:07:52.078122+0000","last_became_peered":"2024-04-24T08:07:52.078122+0000","last_unstale":"2024-04-24T08:10:20.442885+0000","last_undegraded":"2024-04-24T08:10:20.442885+0000","last_fullsized":"2024-04-24T08:10:20.442885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:24:19.062821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.289182+0000","last_change":"2024-04-24T08:07:52.055986+0000","last_active":"2024-04-24T08:10:20.289182+0000","last_peered":"2024-04-24T08:10:20.289182+0000","last_clean":"2024-04-24T08:10:20.289182+0000","last_became_active":"2024-04-24T08:07:52.055609+0000","last_became_peered":"2024-04-24T08:07:52.055609+0000","last_unstale":"2024-04-24T08:10:20.289182+0000","last_undegraded":"2024-04-24T08:10:20.289182+0000","last_fullsized":"2024-04-24T08:10:20.289182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:38:04.731204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498055+0000","last_change":"2024-04-24T08:07:52.055605+0000","last_active":"2024-04-24T08:10:20.498055+0000","last_peered":"2024-04-24T08:10:20.498055+0000","last_clean":"2024-04-24T08:10:20.498055+0000","last_became_active":"2024-04-24T08:07:52.055268+0000","last_became_peered":"2024-04-24T08:07:52.055268+0000","last_unstale":"2024-04-24T08:10:20.498055+0000","last_undegraded":"2024-04-24T08:10:20.498055+0000","last_fullsized":"2024-04-24T08:10:20.498055+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:50:29.197484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288890+0000","last_change":"2024-04-24T08:07:52.064590+0000","last_active":"2024-04-24T08:10:20.288890+0000","last_peered":"2024-04-24T08:10:20.288890+0000","last_clean":"2024-04-24T08:10:20.288890+0000","last_became_active":"2024-04-24T08:07:52.064113+0000","last_became_peered":"2024-04-24T08:07:52.064113+0000","last_unstale":"2024-04-24T08:10:20.288890+0000","last_undegraded":"2024-04-24T08:10:20.288890+0000","last_fullsized":"2024-04-24T08:10:20.288890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:36:10.926145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297622+0000","last_change":"2024-04-24T08:08:51.250599+0000","last_active":"2024-04-24T08:10:20.297622+0000","last_peered":"2024-04-24T08:10:20.297622+0000","last_clean":"2024-04-24T08:10:20.297622+0000","last_became_active":"2024-04-24T08:08:51.250197+0000","last_became_peered":"2024-04-24T08:08:51.250197+0000","last_unstale":"2024-04-24T08:10:20.297622+0000","last_undegraded":"2024-04-24T08:10:20.297622+0000","last_fullsized":"2024-04-24T08:10:20.297622+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:31:30.729911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6],"acting":[5,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288304+0000","last_change":"2024-04-24T08:07:52.068308+0000","last_active":"2024-04-24T08:10:20.288304+0000","last_peered":"2024-04-24T08:10:20.288304+0000","last_clean":"2024-04-24T08:10:20.288304+0000","last_became_active":"2024-04-24T08:07:52.067786+0000","last_became_peered":"2024-04-24T08:07:52.067786+0000","last_unstale":"2024-04-24T08:10:20.288304+0000","last_undegraded":"2024-04-24T08:10:20.288304+0000","last_fullsized":"2024-04-24T08:10:20.288304+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:26:09.349120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498381+0000","last_change":"2024-04-24T08:07:52.053089+0000","last_active":"2024-04-24T08:10:20.498381+0000","last_peered":"2024-04-24T08:10:20.498381+0000","last_clean":"2024-04-24T08:10:20.498381+0000","last_became_active":"2024-04-24T08:07:52.052753+0000","last_became_peered":"2024-04-24T08:07:52.052753+0000","last_unstale":"2024-04-24T08:10:20.498381+0000","last_undegraded":"2024-04-24T08:10:20.498381+0000","last_fullsized":"2024-04-24T08:10:20.498381+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:02:14.583922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.442875+0000","last_change":"2024-04-24T08:07:52.076633+0000","last_active":"2024-04-24T08:10:20.442875+0000","last_peered":"2024-04-24T08:10:20.442875+0000","last_clean":"2024-04-24T08:10:20.442875+0000","last_became_active":"2024-04-24T08:07:52.076280+0000","last_became_peered":"2024-04-24T08:07:52.076280+0000","last_unstale":"2024-04-24T08:10:20.442875+0000","last_undegraded":"2024-04-24T08:10:20.442875+0000","last_fullsized":"2024-04-24T08:10:20.442875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:29:59.823772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288825+0000","last_change":"2024-04-24T08:07:52.053017+0000","last_active":"2024-04-24T08:10:20.288825+0000","last_peered":"2024-04-24T08:10:20.288825+0000","last_clean":"2024-04-24T08:10:20.288825+0000","last_became_active":"2024-04-24T08:07:52.052325+0000","last_became_peered":"2024-04-24T08:07:52.052325+0000","last_unstale":"2024-04-24T08:10:20.288825+0000","last_undegraded":"2024-04-24T08:10:20.288825+0000","last_fullsized":"2024-04-24T08:10:20.288825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:05:58.161360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108294,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":594726,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":987,"num_read_kb":595,"num_write":293,"num_write_kb":1437,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":616,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":565248,"data_stored":1784178,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":229,"ondisk_log_size":229,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351020,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1019904,"data_stored":881474,"data_compressed":33679,"data_compressed_allocated":45056,"data_compressed_original":434176,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612655,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1163264,"data_stored":1271276,"data_compressed":56686,"data_compressed_allocated":73728,"data_compressed_original":753664,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776181,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":881676,"data_compressed":33679,"data_compressed_allocated":45056,"data_compressed_original":434176,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037819,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1007616,"data_stored":488380,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332161,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":1072279,"data_compressed":47714,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495688,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":487849,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1060864,"data_stored":551786,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":489120,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":553753,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182488,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":488672,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920851,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":489791,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757326,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1089536,"data_stored":1212045,"data_compressed":56686,"data_compressed_allocated":73728,"data_compressed_original":753664,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":64144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":66111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2149,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134309,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":126976,"data_stored":198245,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":393832,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:10:25.832 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:10:25.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:25 smithi028 ceph-mon[27786]: Unable to set osd_memory_target on smithi194 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-24T08:10:25.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:25 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi117 to 176.2M 2024-04-24T08:10:25.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:25 smithi028 ceph-mon[27786]: Unable to set osd_memory_target on smithi117 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-24T08:10:25.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:25 smithi028 ceph-mon[27786]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:10:25.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:25 smithi028 ceph-mon[27786]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-04-24T08:10:25.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:25 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2234724785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:10:25.916 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:25 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/447577061' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:10:25.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:25 smithi194 ceph-mon[32836]: Unable to set osd_memory_target on smithi194 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-24T08:10:25.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:25 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi117 to 176.2M 2024-04-24T08:10:25.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:25 smithi194 ceph-mon[32836]: Unable to set osd_memory_target on smithi117 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-24T08:10:25.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:25 smithi194 ceph-mon[32836]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:10:25.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:25 smithi194 ceph-mon[32836]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-04-24T08:10:25.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:25 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2234724785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:10:25.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:25 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/447577061' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:10:26.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.069+0000 7fa6bce00640 1 -- 172.21.15.28:0/3885101750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6b8102900 msgr2=0x7fa6b8104cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.069+0000 7fa6bce00640 1 --2- 172.21.15.28:0/3885101750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6b8102900 0x7fa6b8104cf0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8009900 tx=0x7fa6a802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.069+0000 7fa6bce00640 1 -- 172.21.15.28:0/3885101750 shutdown_connections 2024-04-24T08:10:26.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.069+0000 7fa6bce00640 1 --2- 172.21.15.28:0/3885101750 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6b8105230 0x7fa6b8107620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.069+0000 7fa6bce00640 1 --2- 172.21.15.28:0/3885101750 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b8107b60 0x7fa6b8109fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.069+0000 7fa6bce00640 1 --2- 172.21.15.28:0/3885101750 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6b8102900 0x7fa6b8104cf0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.069+0000 7fa6bce00640 1 -- 172.21.15.28:0/3885101750 >> 172.21.15.28:0/3885101750 conn(0x7fa6b80fc2d0 msgr2=0x7fa6b80fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:26.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.069+0000 7fa6bce00640 1 -- 172.21.15.28:0/3885101750 shutdown_connections 2024-04-24T08:10:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.069+0000 7fa6bce00640 1 -- 172.21.15.28:0/3885101750 wait complete. 2024-04-24T08:10:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.071+0000 7fa6bce00640 1 Processor -- start 2024-04-24T08:10:26.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.071+0000 7fa6bce00640 1 -- start start 2024-04-24T08:10:26.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.071+0000 7fa6bce00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b8102900 0x7fa6b819a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.072+0000 7fa6bce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6b8105230 0x7fa6b819a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.072+0000 7fa6bce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6b8107b60 0x7fa6b81a1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.072+0000 7fa6bce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b810db50 con 0x7fa6b8105230 2024-04-24T08:10:26.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.072+0000 7fa6bce00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b810d9d0 con 0x7fa6b8102900 2024-04-24T08:10:26.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.072+0000 7fa6bce00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa6b810dcd0 con 0x7fa6b8107b60 2024-04-24T08:10:26.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.072+0000 7fa6b6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6b8105230 0x7fa6b819a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.072+0000 7fa6b6a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6b8105230 0x7fa6b819a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33934/0 (socket says 172.21.15.28:33934) 2024-04-24T08:10:26.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.072+0000 7fa6b6a00640 1 -- 172.21.15.28:0/2748779452 learned_addr learned my addr 172.21.15.28:0/2748779452 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:26.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.072+0000 7fa6b7400640 1 --2- 172.21.15.28:0/2748779452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b8102900 0x7fa6b819a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.072+0000 7fa6b7e00640 1 --2- 172.21.15.28:0/2748779452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6b8107b60 0x7fa6b81a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.072+0000 7fa6b6a00640 1 -- 172.21.15.28:0/2748779452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6b8107b60 msgr2=0x7fa6b81a1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.072+0000 7fa6b6a00640 1 --2- 172.21.15.28:0/2748779452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6b8107b60 0x7fa6b81a1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.072+0000 7fa6b6a00640 1 -- 172.21.15.28:0/2748779452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b8102900 msgr2=0x7fa6b819a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.072+0000 7fa6b6a00640 1 --2- 172.21.15.28:0/2748779452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b8102900 0x7fa6b819a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.072+0000 7fa6b6a00640 1 -- 172.21.15.28:0/2748779452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6b81a1fa0 con 0x7fa6b8105230 2024-04-24T08:10:26.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.073+0000 7fa6b7400640 1 --2- 172.21.15.28:0/2748779452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b8102900 0x7fa6b819a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:26.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.073+0000 7fa6b6a00640 1 --2- 172.21.15.28:0/2748779452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6b8105230 0x7fa6b819a8c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400e9f0 tx=0x7fa6a400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:26.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.073+0000 7fa6afe00640 1 -- 172.21.15.28:0/2748779452 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a400ccc0 con 0x7fa6b8105230 2024-04-24T08:10:26.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.073+0000 7fa6bce00640 1 -- 172.21.15.28:0/2748779452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6b81a2290 con 0x7fa6b8105230 2024-04-24T08:10:26.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.073+0000 7fa6afe00640 1 -- 172.21.15.28:0/2748779452 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6a4005500 con 0x7fa6b8105230 2024-04-24T08:10:26.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.073+0000 7fa6afe00640 1 -- 172.21.15.28:0/2748779452 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6a4018430 con 0x7fa6b8105230 2024-04-24T08:10:26.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.073+0000 7fa6bce00640 1 -- 172.21.15.28:0/2748779452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6b81a2790 con 0x7fa6b8105230 2024-04-24T08:10:26.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.074+0000 7fa6bce00640 1 -- 172.21.15.28:0/2748779452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6b81096b0 con 0x7fa6b8105230 2024-04-24T08:10:26.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.076+0000 7fa6afe00640 1 -- 172.21.15.28:0/2748779452 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7fa6a4005000 con 0x7fa6b8105230 2024-04-24T08:10:26.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.078+0000 7fa6afe00640 1 --2- 172.21.15.28:0/2748779452 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa69c0a3e40 0x7fa69c0a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.078+0000 7fa6afe00640 1 -- 172.21.15.28:0/2748779452 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fa6a40d0f40 con 0x7fa6b8105230 2024-04-24T08:10:26.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.078+0000 7fa6b7400640 1 --2- 172.21.15.28:0/2748779452 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa69c0a3e40 0x7fa69c0a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.079+0000 7fa6b7400640 1 --2- 172.21.15.28:0/2748779452 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa69c0a3e40 0x7fa69c0a6300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8031440 tx=0x7fa6a80315b0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:26.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.087+0000 7fa6afe00640 1 -- 172.21.15.28:0/2748779452 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa6a409d310 con 0x7fa6b8105230 2024-04-24T08:10:26.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:25 smithi117 ceph-mon[32438]: Unable to set osd_memory_target on smithi194 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-24T08:10:26.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:25 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi117 to 176.2M 2024-04-24T08:10:26.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:25 smithi117 ceph-mon[32438]: Unable to set osd_memory_target on smithi117 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-24T08:10:26.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:25 smithi117 ceph-mon[32438]: pgmap v5: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:10:26.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:25 smithi117 ceph-mon[32438]: mgrmap e25: x(active, since 4s), standbys: y, z 2024-04-24T08:10:26.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:25 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2234724785' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:10:26.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:25 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/447577061' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:10:26.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.305+0000 7fa6bce00640 1 -- 172.21.15.28:0/2748779452 --> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa6b8062440 con 0x7fa69c0a3e40 2024-04-24T08:10:26.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.315+0000 7fa6afe00640 1 -- 172.21.15.28:0/2748779452 <== mgr.25225 v2:172.21.15.28:6800/353036813 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258067 (secure 0 0 0) 0x7fa6b8062440 con 0x7fa69c0a3e40 2024-04-24T08:10:26.319 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:10:26.320 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-24T08:10:26.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.322+0000 7fa6bce00640 1 -- 172.21.15.28:0/2748779452 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa69c0a3e40 msgr2=0x7fa69c0a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.322+0000 7fa6bce00640 1 --2- 172.21.15.28:0/2748779452 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa69c0a3e40 0x7fa69c0a6300 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa6a8031440 tx=0x7fa6a80315b0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.322+0000 7fa6bce00640 1 -- 172.21.15.28:0/2748779452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6b8105230 msgr2=0x7fa6b819a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.322+0000 7fa6bce00640 1 --2- 172.21.15.28:0/2748779452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6b8105230 0x7fa6b819a8c0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fa6a400e9f0 tx=0x7fa6a400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.322+0000 7fa6bce00640 1 -- 172.21.15.28:0/2748779452 shutdown_connections 2024-04-24T08:10:26.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.322+0000 7fa6bce00640 1 --2- 172.21.15.28:0/2748779452 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa69c0a3e40 0x7fa69c0a6300 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.322+0000 7fa6bce00640 1 --2- 172.21.15.28:0/2748779452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa6b8105230 0x7fa6b819a8c0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.322+0000 7fa6bce00640 1 --2- 172.21.15.28:0/2748779452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa6b8107b60 0x7fa6b81a1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.322+0000 7fa6bce00640 1 --2- 172.21.15.28:0/2748779452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b8102900 0x7fa6b819a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.322+0000 7fa6bce00640 1 -- 172.21.15.28:0/2748779452 >> 172.21.15.28:0/2748779452 conn(0x7fa6b80fc2d0 msgr2=0x7fa6b8102010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:26.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.322+0000 7fa6bce00640 1 -- 172.21.15.28:0/2748779452 shutdown_connections 2024-04-24T08:10:26.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.322+0000 7fa6bce00640 1 -- 172.21.15.28:0/2748779452 wait complete. 2024-04-24T08:10:26.343 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-04-24T08:10:26.156645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188630,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1146,"num_read_kb":733,"num_write":479,"num_write_kb":4382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1181352,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":871,"ondisk_log_size":871,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365600,"kb_used_data":12240,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576224,"statfs":{"total":1151940427776,"available":1151566053376,"internally_reserved":0,"allocated":12533760,"data_stored":8876890,"data_compressed":330276,"data_compressed_allocated":442368,"data_compressed_original":3284992,"omap_allocated":19140,"internal_metadata":361674044},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3302,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":19,"num_write":3,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.979281"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.289087+0000","last_change":"2024-04-24T08:07:52.074319+0000","last_active":"2024-04-24T08:10:20.289087+0000","last_peered":"2024-04-24T08:10:20.289087+0000","last_clean":"2024-04-24T08:10:20.289087+0000","last_became_active":"2024-04-24T08:07:52.073923+0000","last_became_peered":"2024-04-24T08:07:52.073923+0000","last_unstale":"2024-04-24T08:10:20.289087+0000","last_undegraded":"2024-04-24T08:10:20.289087+0000","last_fullsized":"2024-04-24T08:10:20.289087+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:29.325904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.504231+0000","last_change":"2024-04-24T08:07:52.068316+0000","last_active":"2024-04-24T08:10:20.504231+0000","last_peered":"2024-04-24T08:10:20.504231+0000","last_clean":"2024-04-24T08:10:20.504231+0000","last_became_active":"2024-04-24T08:07:52.067961+0000","last_became_peered":"2024-04-24T08:07:52.067961+0000","last_unstale":"2024-04-24T08:10:20.504231+0000","last_undegraded":"2024-04-24T08:10:20.504231+0000","last_fullsized":"2024-04-24T08:10:20.504231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:35:14.079348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.503658+0000","last_change":"2024-04-24T08:07:52.066870+0000","last_active":"2024-04-24T08:10:20.503658+0000","last_peered":"2024-04-24T08:10:20.503658+0000","last_clean":"2024-04-24T08:10:20.503658+0000","last_became_active":"2024-04-24T08:07:52.066481+0000","last_became_peered":"2024-04-24T08:07:52.066481+0000","last_unstale":"2024-04-24T08:10:20.503658+0000","last_undegraded":"2024-04-24T08:10:20.503658+0000","last_fullsized":"2024-04-24T08:10:20.503658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:25:45.526667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.129237+0000","last_change":"2024-04-24T08:07:52.061548+0000","last_active":"2024-04-24T08:10:20.129237+0000","last_peered":"2024-04-24T08:10:20.129237+0000","last_clean":"2024-04-24T08:10:20.129237+0000","last_became_active":"2024-04-24T08:07:52.060519+0000","last_became_peered":"2024-04-24T08:07:52.060519+0000","last_unstale":"2024-04-24T08:10:20.129237+0000","last_undegraded":"2024-04-24T08:10:20.129237+0000","last_fullsized":"2024-04-24T08:10:20.129237+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:47:20.952035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128104+0000","last_change":"2024-04-24T08:07:52.063805+0000","last_active":"2024-04-24T08:10:20.128104+0000","last_peered":"2024-04-24T08:10:20.128104+0000","last_clean":"2024-04-24T08:10:20.128104+0000","last_became_active":"2024-04-24T08:07:52.063414+0000","last_became_peered":"2024-04-24T08:07:52.063414+0000","last_unstale":"2024-04-24T08:10:20.128104+0000","last_undegraded":"2024-04-24T08:10:20.128104+0000","last_fullsized":"2024-04-24T08:10:20.128104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:06:24.465035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.301125+0000","last_change":"2024-04-24T08:07:52.050152+0000","last_active":"2024-04-24T08:10:20.301125+0000","last_peered":"2024-04-24T08:10:20.301125+0000","last_clean":"2024-04-24T08:10:20.301125+0000","last_became_active":"2024-04-24T08:07:52.049805+0000","last_became_peered":"2024-04-24T08:07:52.049805+0000","last_unstale":"2024-04-24T08:10:20.301125+0000","last_undegraded":"2024-04-24T08:10:20.301125+0000","last_fullsized":"2024-04-24T08:10:20.301125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:47:34.644263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.258824+0000","last_change":"2024-04-24T08:07:52.055726+0000","last_active":"2024-04-24T08:10:20.258824+0000","last_peered":"2024-04-24T08:10:20.258824+0000","last_clean":"2024-04-24T08:10:20.258824+0000","last_became_active":"2024-04-24T08:07:52.054563+0000","last_became_peered":"2024-04-24T08:07:52.054563+0000","last_unstale":"2024-04-24T08:10:20.258824+0000","last_undegraded":"2024-04-24T08:10:20.258824+0000","last_fullsized":"2024-04-24T08:10:20.258824+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:34:57.430232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288218+0000","last_change":"2024-04-24T08:07:52.053672+0000","last_active":"2024-04-24T08:10:20.288218+0000","last_peered":"2024-04-24T08:10:20.288218+0000","last_clean":"2024-04-24T08:10:20.288218+0000","last_became_active":"2024-04-24T08:07:52.053150+0000","last_became_peered":"2024-04-24T08:07:52.053150+0000","last_unstale":"2024-04-24T08:10:20.288218+0000","last_undegraded":"2024-04-24T08:10:20.288218+0000","last_fullsized":"2024-04-24T08:10:20.288218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:00:12.272555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498877+0000","last_change":"2024-04-24T08:07:52.054648+0000","last_active":"2024-04-24T08:10:20.498877+0000","last_peered":"2024-04-24T08:10:20.498877+0000","last_clean":"2024-04-24T08:10:20.498877+0000","last_became_active":"2024-04-24T08:07:52.054301+0000","last_became_peered":"2024-04-24T08:07:52.054301+0000","last_unstale":"2024-04-24T08:10:20.498877+0000","last_undegraded":"2024-04-24T08:10:20.498877+0000","last_fullsized":"2024-04-24T08:10:20.498877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:52:20.533855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.301467+0000","last_change":"2024-04-24T08:07:52.052994+0000","last_active":"2024-04-24T08:10:20.301467+0000","last_peered":"2024-04-24T08:10:20.301467+0000","last_clean":"2024-04-24T08:10:20.301467+0000","last_became_active":"2024-04-24T08:07:52.052641+0000","last_became_peered":"2024-04-24T08:07:52.052641+0000","last_unstale":"2024-04-24T08:10:20.301467+0000","last_undegraded":"2024-04-24T08:10:20.301467+0000","last_fullsized":"2024-04-24T08:10:20.301467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:02.665622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259119+0000","last_change":"2024-04-24T08:07:52.055901+0000","last_active":"2024-04-24T08:10:20.259119+0000","last_peered":"2024-04-24T08:10:20.259119+0000","last_clean":"2024-04-24T08:10:20.259119+0000","last_became_active":"2024-04-24T08:07:52.055277+0000","last_became_peered":"2024-04-24T08:07:52.055277+0000","last_unstale":"2024-04-24T08:10:20.259119+0000","last_undegraded":"2024-04-24T08:10:20.259119+0000","last_fullsized":"2024-04-24T08:10:20.259119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:46.950578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128973+0000","last_change":"2024-04-24T08:07:52.060329+0000","last_active":"2024-04-24T08:10:20.128973+0000","last_peered":"2024-04-24T08:10:20.128973+0000","last_clean":"2024-04-24T08:10:20.128973+0000","last_became_active":"2024-04-24T08:07:52.059914+0000","last_became_peered":"2024-04-24T08:07:52.059914+0000","last_unstale":"2024-04-24T08:10:20.128973+0000","last_undegraded":"2024-04-24T08:10:20.128973+0000","last_fullsized":"2024-04-24T08:10:20.128973+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:15:20.263916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.443115+0000","last_change":"2024-04-24T08:07:52.062108+0000","last_active":"2024-04-24T08:10:20.443115+0000","last_peered":"2024-04-24T08:10:20.443115+0000","last_clean":"2024-04-24T08:10:20.443115+0000","last_became_active":"2024-04-24T08:07:52.061665+0000","last_became_peered":"2024-04-24T08:07:52.061665+0000","last_unstale":"2024-04-24T08:10:20.443115+0000","last_undegraded":"2024-04-24T08:10:20.443115+0000","last_fullsized":"2024-04-24T08:10:20.443115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:55:58.071807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.503324+0000","last_change":"2024-04-24T08:07:52.063033+0000","last_active":"2024-04-24T08:10:20.503324+0000","last_peered":"2024-04-24T08:10:20.503324+0000","last_clean":"2024-04-24T08:10:20.503324+0000","last_became_active":"2024-04-24T08:07:52.062610+0000","last_became_peered":"2024-04-24T08:07:52.062610+0000","last_unstale":"2024-04-24T08:10:20.503324+0000","last_undegraded":"2024-04-24T08:10:20.503324+0000","last_fullsized":"2024-04-24T08:10:20.503324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:59:35.906720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.503122+0000","last_change":"2024-04-24T08:07:52.064657+0000","last_active":"2024-04-24T08:10:20.503122+0000","last_peered":"2024-04-24T08:10:20.503122+0000","last_clean":"2024-04-24T08:10:20.503122+0000","last_became_active":"2024-04-24T08:07:52.064188+0000","last_became_peered":"2024-04-24T08:07:52.064188+0000","last_unstale":"2024-04-24T08:10:20.503122+0000","last_undegraded":"2024-04-24T08:10:20.503122+0000","last_fullsized":"2024-04-24T08:10:20.503122+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:56:32.139420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498229+0000","last_change":"2024-04-24T08:07:52.067227+0000","last_active":"2024-04-24T08:10:20.498229+0000","last_peered":"2024-04-24T08:10:20.498229+0000","last_clean":"2024-04-24T08:10:20.498229+0000","last_became_active":"2024-04-24T08:07:52.066854+0000","last_became_peered":"2024-04-24T08:07:52.066854+0000","last_unstale":"2024-04-24T08:10:20.498229+0000","last_undegraded":"2024-04-24T08:10:20.498229+0000","last_fullsized":"2024-04-24T08:10:20.498229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:25:17.580223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.289276+0000","last_change":"2024-04-24T08:07:52.074127+0000","last_active":"2024-04-24T08:10:20.289276+0000","last_peered":"2024-04-24T08:10:20.289276+0000","last_clean":"2024-04-24T08:10:20.289276+0000","last_became_active":"2024-04-24T08:07:52.073806+0000","last_became_peered":"2024-04-24T08:07:52.073806+0000","last_unstale":"2024-04-24T08:10:20.289276+0000","last_undegraded":"2024-04-24T08:10:20.289276+0000","last_fullsized":"2024-04-24T08:10:20.289276+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:25:22.068997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.131220+0000","last_change":"2024-04-24T08:07:52.051239+0000","last_active":"2024-04-24T08:10:20.131220+0000","last_peered":"2024-04-24T08:10:20.131220+0000","last_clean":"2024-04-24T08:10:20.131220+0000","last_became_active":"2024-04-24T08:07:52.049232+0000","last_became_peered":"2024-04-24T08:07:52.049232+0000","last_unstale":"2024-04-24T08:10:20.131220+0000","last_undegraded":"2024-04-24T08:10:20.131220+0000","last_fullsized":"2024-04-24T08:10:20.131220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:07:39.768537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.497883+0000","last_change":"2024-04-24T08:07:52.062119+0000","last_active":"2024-04-24T08:10:20.497883+0000","last_peered":"2024-04-24T08:10:20.497883+0000","last_clean":"2024-04-24T08:10:20.497883+0000","last_became_active":"2024-04-24T08:07:52.061781+0000","last_became_peered":"2024-04-24T08:07:52.061781+0000","last_unstale":"2024-04-24T08:10:20.497883+0000","last_undegraded":"2024-04-24T08:10:20.497883+0000","last_fullsized":"2024-04-24T08:10:20.497883+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:49:41.595368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.129438+0000","last_change":"2024-04-24T08:07:51.052411+0000","last_active":"2024-04-24T08:10:20.129438+0000","last_peered":"2024-04-24T08:10:20.129438+0000","last_clean":"2024-04-24T08:10:20.129438+0000","last_became_active":"2024-04-24T08:07:51.051704+0000","last_became_peered":"2024-04-24T08:07:51.051704+0000","last_unstale":"2024-04-24T08:10:20.129438+0000","last_undegraded":"2024-04-24T08:10:20.129438+0000","last_fullsized":"2024-04-24T08:10:20.129438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:34:23.001338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288658+0000","last_change":"2024-04-24T08:07:52.056508+0000","last_active":"2024-04-24T08:10:20.288658+0000","last_peered":"2024-04-24T08:10:20.288658+0000","last_clean":"2024-04-24T08:10:20.288658+0000","last_became_active":"2024-04-24T08:07:52.056158+0000","last_became_peered":"2024-04-24T08:07:52.056158+0000","last_unstale":"2024-04-24T08:10:20.288658+0000","last_undegraded":"2024-04-24T08:10:20.288658+0000","last_fullsized":"2024-04-24T08:10:20.288658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:52.267984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288783+0000","last_change":"2024-04-24T08:07:51.080403+0000","last_active":"2024-04-24T08:10:20.288783+0000","last_peered":"2024-04-24T08:10:20.288783+0000","last_clean":"2024-04-24T08:10:20.288783+0000","last_became_active":"2024-04-24T08:07:51.079158+0000","last_became_peered":"2024-04-24T08:07:51.079158+0000","last_unstale":"2024-04-24T08:10:20.288783+0000","last_undegraded":"2024-04-24T08:10:20.288783+0000","last_fullsized":"2024-04-24T08:10:20.288783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:49:59.491850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.504255+0000","last_change":"2024-04-24T08:07:52.066963+0000","last_active":"2024-04-24T08:10:20.504255+0000","last_peered":"2024-04-24T08:10:20.504255+0000","last_clean":"2024-04-24T08:10:20.504255+0000","last_became_active":"2024-04-24T08:07:52.065475+0000","last_became_peered":"2024-04-24T08:07:52.065475+0000","last_unstale":"2024-04-24T08:10:20.504255+0000","last_undegraded":"2024-04-24T08:10:20.504255+0000","last_fullsized":"2024-04-24T08:10:20.504255+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:28:34.220200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128324+0000","last_change":"2024-04-24T08:07:51.042772+0000","last_active":"2024-04-24T08:10:20.128324+0000","last_peered":"2024-04-24T08:10:20.128324+0000","last_clean":"2024-04-24T08:10:20.128324+0000","last_became_active":"2024-04-24T08:07:51.042128+0000","last_became_peered":"2024-04-24T08:07:51.042128+0000","last_unstale":"2024-04-24T08:10:20.128324+0000","last_undegraded":"2024-04-24T08:10:20.128324+0000","last_fullsized":"2024-04-24T08:10:20.128324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:10:29.207991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498210+0000","last_change":"2024-04-24T08:07:52.062411+0000","last_active":"2024-04-24T08:10:20.498210+0000","last_peered":"2024-04-24T08:10:20.498210+0000","last_clean":"2024-04-24T08:10:20.498210+0000","last_became_active":"2024-04-24T08:07:52.062037+0000","last_became_peered":"2024-04-24T08:07:52.062037+0000","last_unstale":"2024-04-24T08:10:20.498210+0000","last_undegraded":"2024-04-24T08:10:20.498210+0000","last_fullsized":"2024-04-24T08:10:20.498210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:16:17.463874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128356+0000","last_change":"2024-04-24T08:07:51.046046+0000","last_active":"2024-04-24T08:10:20.128356+0000","last_peered":"2024-04-24T08:10:20.128356+0000","last_clean":"2024-04-24T08:10:20.128356+0000","last_became_active":"2024-04-24T08:07:51.045633+0000","last_became_peered":"2024-04-24T08:07:51.045633+0000","last_unstale":"2024-04-24T08:10:20.128356+0000","last_undegraded":"2024-04-24T08:10:20.128356+0000","last_fullsized":"2024-04-24T08:10:20.128356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:18:20.529234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297863+0000","last_change":"2024-04-24T08:07:52.061167+0000","last_active":"2024-04-24T08:10:20.297863+0000","last_peered":"2024-04-24T08:10:20.297863+0000","last_clean":"2024-04-24T08:10:20.297863+0000","last_became_active":"2024-04-24T08:07:52.060841+0000","last_became_peered":"2024-04-24T08:07:52.060841+0000","last_unstale":"2024-04-24T08:10:20.297863+0000","last_undegraded":"2024-04-24T08:10:20.297863+0000","last_fullsized":"2024-04-24T08:10:20.297863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:04:27.048760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"91'9","reported_seq":184,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.502848+0000","last_change":"2024-04-24T08:07:51.044274+0000","last_active":"2024-04-24T08:10:20.502848+0000","last_peered":"2024-04-24T08:10:20.502848+0000","last_clean":"2024-04-24T08:10:20.502848+0000","last_became_active":"2024-04-24T08:07:51.043938+0000","last_became_peered":"2024-04-24T08:07:51.043938+0000","last_unstale":"2024-04-24T08:10:20.502848+0000","last_undegraded":"2024-04-24T08:10:20.502848+0000","last_fullsized":"2024-04-24T08:10:20.502848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:05:38.349981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1855,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128148+0000","last_change":"2024-04-24T08:07:52.055777+0000","last_active":"2024-04-24T08:10:20.128148+0000","last_peered":"2024-04-24T08:10:20.128148+0000","last_clean":"2024-04-24T08:10:20.128148+0000","last_became_active":"2024-04-24T08:07:52.055415+0000","last_became_peered":"2024-04-24T08:07:52.055415+0000","last_unstale":"2024-04-24T08:10:20.128148+0000","last_undegraded":"2024-04-24T08:10:20.128148+0000","last_fullsized":"2024-04-24T08:10:20.128148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:18:31.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.301075+0000","last_change":"2024-04-24T08:07:51.072513+0000","last_active":"2024-04-24T08:10:20.301075+0000","last_peered":"2024-04-24T08:10:20.301075+0000","last_clean":"2024-04-24T08:10:20.301075+0000","last_became_active":"2024-04-24T08:07:51.072249+0000","last_became_peered":"2024-04-24T08:07:51.072249+0000","last_unstale":"2024-04-24T08:10:20.301075+0000","last_undegraded":"2024-04-24T08:10:20.301075+0000","last_fullsized":"2024-04-24T08:10:20.301075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:45:07.112617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297787+0000","last_change":"2024-04-24T08:08:51.587130+0000","last_active":"2024-04-24T08:10:20.297787+0000","last_peered":"2024-04-24T08:10:20.297787+0000","last_clean":"2024-04-24T08:10:20.297787+0000","last_became_active":"2024-04-24T08:08:51.586645+0000","last_became_peered":"2024-04-24T08:08:51.586645+0000","last_unstale":"2024-04-24T08:10:20.297787+0000","last_undegraded":"2024-04-24T08:10:20.297787+0000","last_fullsized":"2024-04-24T08:10:20.297787+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:24:14.677548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.443164+0000","last_change":"2024-04-24T08:07:51.070901+0000","last_active":"2024-04-24T08:10:20.443164+0000","last_peered":"2024-04-24T08:10:20.443164+0000","last_clean":"2024-04-24T08:10:20.443164+0000","last_became_active":"2024-04-24T08:07:51.067178+0000","last_became_peered":"2024-04-24T08:07:51.067178+0000","last_unstale":"2024-04-24T08:10:20.443164+0000","last_undegraded":"2024-04-24T08:10:20.443164+0000","last_fullsized":"2024-04-24T08:10:20.443164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:24:26.807786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288202+0000","last_change":"2024-04-24T08:07:52.079493+0000","last_active":"2024-04-24T08:10:20.288202+0000","last_peered":"2024-04-24T08:10:20.288202+0000","last_clean":"2024-04-24T08:10:20.288202+0000","last_became_active":"2024-04-24T08:07:52.077266+0000","last_became_peered":"2024-04-24T08:07:52.077266+0000","last_unstale":"2024-04-24T08:10:20.288202+0000","last_undegraded":"2024-04-24T08:10:20.288202+0000","last_fullsized":"2024-04-24T08:10:20.288202+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:40:45.660146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297546+0000","last_change":"2024-04-24T08:07:51.045963+0000","last_active":"2024-04-24T08:10:20.297546+0000","last_peered":"2024-04-24T08:10:20.297546+0000","last_clean":"2024-04-24T08:10:20.297546+0000","last_became_active":"2024-04-24T08:07:51.045641+0000","last_became_peered":"2024-04-24T08:07:51.045641+0000","last_unstale":"2024-04-24T08:10:20.297546+0000","last_undegraded":"2024-04-24T08:10:20.297546+0000","last_fullsized":"2024-04-24T08:10:20.297546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:07:59.646155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128896+0000","last_change":"2024-04-24T08:07:52.057280+0000","last_active":"2024-04-24T08:10:20.128896+0000","last_peered":"2024-04-24T08:10:20.128896+0000","last_clean":"2024-04-24T08:10:20.128896+0000","last_became_active":"2024-04-24T08:07:52.054944+0000","last_became_peered":"2024-04-24T08:07:52.054944+0000","last_unstale":"2024-04-24T08:10:20.128896+0000","last_undegraded":"2024-04-24T08:10:20.128896+0000","last_fullsized":"2024-04-24T08:10:20.128896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:22:45.911472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":180,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:24.378138+0000","last_change":"2024-04-24T08:07:51.048898+0000","last_active":"2024-04-24T08:10:24.378138+0000","last_peered":"2024-04-24T08:10:24.378138+0000","last_clean":"2024-04-24T08:10:24.378138+0000","last_became_active":"2024-04-24T08:07:51.048585+0000","last_became_peered":"2024-04-24T08:07:51.048585+0000","last_unstale":"2024-04-24T08:10:24.378138+0000","last_undegraded":"2024-04-24T08:10:24.378138+0000","last_fullsized":"2024-04-24T08:10:24.378138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:40:30.413541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":149,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498865+0000","last_change":"2024-04-24T08:07:52.048093+0000","last_active":"2024-04-24T08:10:20.498865+0000","last_peered":"2024-04-24T08:10:20.498865+0000","last_clean":"2024-04-24T08:10:20.498865+0000","last_became_active":"2024-04-24T08:07:52.047727+0000","last_became_peered":"2024-04-24T08:07:52.047727+0000","last_unstale":"2024-04-24T08:10:20.498865+0000","last_undegraded":"2024-04-24T08:10:20.498865+0000","last_fullsized":"2024-04-24T08:10:20.498865+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:57:26.248355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259256+0000","last_change":"2024-04-24T08:07:51.047810+0000","last_active":"2024-04-24T08:10:20.259256+0000","last_peered":"2024-04-24T08:10:20.259256+0000","last_clean":"2024-04-24T08:10:20.259256+0000","last_became_active":"2024-04-24T08:07:51.047501+0000","last_became_peered":"2024-04-24T08:07:51.047501+0000","last_unstale":"2024-04-24T08:10:20.259256+0000","last_undegraded":"2024-04-24T08:10:20.259256+0000","last_fullsized":"2024-04-24T08:10:20.259256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:02:27.585202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.504166+0000","last_change":"2024-04-24T08:07:52.063782+0000","last_active":"2024-04-24T08:10:20.504166+0000","last_peered":"2024-04-24T08:10:20.504166+0000","last_clean":"2024-04-24T08:10:20.504166+0000","last_became_active":"2024-04-24T08:07:52.063056+0000","last_became_peered":"2024-04-24T08:07:52.063056+0000","last_unstale":"2024-04-24T08:10:20.504166+0000","last_undegraded":"2024-04-24T08:10:20.504166+0000","last_fullsized":"2024-04-24T08:10:20.504166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:41:30.966787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.298089+0000","last_change":"2024-04-24T08:07:51.076139+0000","last_active":"2024-04-24T08:10:20.298089+0000","last_peered":"2024-04-24T08:10:20.298089+0000","last_clean":"2024-04-24T08:10:20.298089+0000","last_became_active":"2024-04-24T08:07:51.075842+0000","last_became_peered":"2024-04-24T08:07:51.075842+0000","last_unstale":"2024-04-24T08:10:20.298089+0000","last_undegraded":"2024-04-24T08:10:20.298089+0000","last_fullsized":"2024-04-24T08:10:20.298089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:46:56.741726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498434+0000","last_change":"2024-04-24T08:08:51.250881+0000","last_active":"2024-04-24T08:10:20.498434+0000","last_peered":"2024-04-24T08:10:20.498434+0000","last_clean":"2024-04-24T08:10:20.498434+0000","last_became_active":"2024-04-24T08:08:51.250519+0000","last_became_peered":"2024-04-24T08:08:51.250519+0000","last_unstale":"2024-04-24T08:10:20.498434+0000","last_undegraded":"2024-04-24T08:10:20.498434+0000","last_fullsized":"2024-04-24T08:10:20.498434+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:23:04.805687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'36","reported_seq":371,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:25.379830+0000","last_change":"2024-04-24T08:07:51.048079+0000","last_active":"2024-04-24T08:10:25.379830+0000","last_peered":"2024-04-24T08:10:25.379830+0000","last_clean":"2024-04-24T08:10:25.379830+0000","last_became_active":"2024-04-24T08:07:51.047584+0000","last_became_peered":"2024-04-24T08:07:51.047584+0000","last_unstale":"2024-04-24T08:10:25.379830+0000","last_undegraded":"2024-04-24T08:10:25.379830+0000","last_fullsized":"2024-04-24T08:10:25.379830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:09:05.554984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67055,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":309,"num_read_kb":219,"num_write":37,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128091+0000","last_change":"2024-04-24T08:07:52.055894+0000","last_active":"2024-04-24T08:10:20.128091+0000","last_peered":"2024-04-24T08:10:20.128091+0000","last_clean":"2024-04-24T08:10:20.128091+0000","last_became_active":"2024-04-24T08:07:52.055548+0000","last_became_peered":"2024-04-24T08:07:52.055548+0000","last_unstale":"2024-04-24T08:10:20.128091+0000","last_undegraded":"2024-04-24T08:10:20.128091+0000","last_fullsized":"2024-04-24T08:10:20.128091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:33:22.609865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288830+0000","last_change":"2024-04-24T08:08:51.188459+0000","last_active":"2024-04-24T08:10:20.288830+0000","last_peered":"2024-04-24T08:10:20.288830+0000","last_clean":"2024-04-24T08:10:20.288830+0000","last_became_active":"2024-04-24T08:08:51.180387+0000","last_became_peered":"2024-04-24T08:08:51.180387+0000","last_unstale":"2024-04-24T08:10:20.288830+0000","last_undegraded":"2024-04-24T08:10:20.288830+0000","last_fullsized":"2024-04-24T08:10:20.288830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:34:56.592978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.443550+0000","last_change":"2024-04-24T08:07:52.069950+0000","last_active":"2024-04-24T08:10:20.443550+0000","last_peered":"2024-04-24T08:10:20.443550+0000","last_clean":"2024-04-24T08:10:20.443550+0000","last_became_active":"2024-04-24T08:07:52.069515+0000","last_became_peered":"2024-04-24T08:07:52.069515+0000","last_unstale":"2024-04-24T08:10:20.443550+0000","last_undegraded":"2024-04-24T08:10:20.443550+0000","last_fullsized":"2024-04-24T08:10:20.443550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:52:27.530541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297642+0000","last_change":"2024-04-24T08:07:51.067062+0000","last_active":"2024-04-24T08:10:20.297642+0000","last_peered":"2024-04-24T08:10:20.297642+0000","last_clean":"2024-04-24T08:10:20.297642+0000","last_became_active":"2024-04-24T08:07:51.066712+0000","last_became_peered":"2024-04-24T08:07:51.066712+0000","last_unstale":"2024-04-24T08:10:20.297642+0000","last_undegraded":"2024-04-24T08:10:20.297642+0000","last_fullsized":"2024-04-24T08:10:20.297642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:11:52.974470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259346+0000","last_change":"2024-04-24T08:07:52.064521+0000","last_active":"2024-04-24T08:10:20.259346+0000","last_peered":"2024-04-24T08:10:20.259346+0000","last_clean":"2024-04-24T08:10:20.259346+0000","last_became_active":"2024-04-24T08:07:52.064184+0000","last_became_peered":"2024-04-24T08:07:52.064184+0000","last_unstale":"2024-04-24T08:10:20.259346+0000","last_undegraded":"2024-04-24T08:10:20.259346+0000","last_fullsized":"2024-04-24T08:10:20.259346+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:00:27.294441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":187,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:24.451262+0000","last_change":"2024-04-24T08:07:51.043577+0000","last_active":"2024-04-24T08:10:24.451262+0000","last_peered":"2024-04-24T08:10:24.451262+0000","last_clean":"2024-04-24T08:10:24.451262+0000","last_became_active":"2024-04-24T08:07:51.043215+0000","last_became_peered":"2024-04-24T08:07:51.043215+0000","last_unstale":"2024-04-24T08:10:24.451262+0000","last_undegraded":"2024-04-24T08:10:24.451262+0000","last_fullsized":"2024-04-24T08:10:24.451262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:25:09.979236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":155,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.258879+0000","last_change":"2024-04-24T08:07:52.057633+0000","last_active":"2024-04-24T08:10:20.258879+0000","last_peered":"2024-04-24T08:10:20.258879+0000","last_clean":"2024-04-24T08:10:20.258879+0000","last_became_active":"2024-04-24T08:07:52.057309+0000","last_became_peered":"2024-04-24T08:07:52.057309+0000","last_unstale":"2024-04-24T08:10:20.258879+0000","last_undegraded":"2024-04-24T08:10:20.258879+0000","last_fullsized":"2024-04-24T08:10:20.258879+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:49:56.045030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.127851+0000","last_change":"2024-04-24T08:07:51.058957+0000","last_active":"2024-04-24T08:10:20.127851+0000","last_peered":"2024-04-24T08:10:20.127851+0000","last_clean":"2024-04-24T08:10:20.127851+0000","last_became_active":"2024-04-24T08:07:51.058654+0000","last_became_peered":"2024-04-24T08:07:51.058654+0000","last_unstale":"2024-04-24T08:10:20.127851+0000","last_undegraded":"2024-04-24T08:10:20.127851+0000","last_fullsized":"2024-04-24T08:10:20.127851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:13.354590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297541+0000","last_change":"2024-04-24T08:07:52.051052+0000","last_active":"2024-04-24T08:10:20.297541+0000","last_peered":"2024-04-24T08:10:20.297541+0000","last_clean":"2024-04-24T08:10:20.297541+0000","last_became_active":"2024-04-24T08:07:52.050685+0000","last_became_peered":"2024-04-24T08:07:52.050685+0000","last_unstale":"2024-04-24T08:10:20.297541+0000","last_undegraded":"2024-04-24T08:10:20.297541+0000","last_fullsized":"2024-04-24T08:10:20.297541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:34.491653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128105+0000","last_change":"2024-04-24T08:07:51.041088+0000","last_active":"2024-04-24T08:10:20.128105+0000","last_peered":"2024-04-24T08:10:20.128105+0000","last_clean":"2024-04-24T08:10:20.128105+0000","last_became_active":"2024-04-24T08:07:51.040752+0000","last_became_peered":"2024-04-24T08:07:51.040752+0000","last_unstale":"2024-04-24T08:10:20.128105+0000","last_undegraded":"2024-04-24T08:10:20.128105+0000","last_fullsized":"2024-04-24T08:10:20.128105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:48:23.405705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.289018+0000","last_change":"2024-04-24T08:08:51.173331+0000","last_active":"2024-04-24T08:10:20.289018+0000","last_peered":"2024-04-24T08:10:20.289018+0000","last_clean":"2024-04-24T08:10:20.289018+0000","last_became_active":"2024-04-24T08:08:51.172912+0000","last_became_peered":"2024-04-24T08:08:51.172912+0000","last_unstale":"2024-04-24T08:10:20.289018+0000","last_undegraded":"2024-04-24T08:10:20.289018+0000","last_fullsized":"2024-04-24T08:10:20.289018+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:54:35.241315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.134622+0000","last_change":"2024-04-24T08:07:51.075583+0000","last_active":"2024-04-24T08:10:20.134622+0000","last_peered":"2024-04-24T08:10:20.134622+0000","last_clean":"2024-04-24T08:10:20.134622+0000","last_became_active":"2024-04-24T08:07:51.075111+0000","last_became_peered":"2024-04-24T08:07:51.075111+0000","last_unstale":"2024-04-24T08:10:20.134622+0000","last_undegraded":"2024-04-24T08:10:20.134622+0000","last_fullsized":"2024-04-24T08:10:20.134622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:56:00.820057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.301341+0000","last_change":"2024-04-24T08:07:52.063232+0000","last_active":"2024-04-24T08:10:20.301341+0000","last_peered":"2024-04-24T08:10:20.301341+0000","last_clean":"2024-04-24T08:10:20.301341+0000","last_became_active":"2024-04-24T08:07:52.062868+0000","last_became_peered":"2024-04-24T08:07:52.062868+0000","last_unstale":"2024-04-24T08:10:20.301341+0000","last_undegraded":"2024-04-24T08:10:20.301341+0000","last_fullsized":"2024-04-24T08:10:20.301341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:00:40.333389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.258895+0000","last_change":"2024-04-24T08:07:51.070647+0000","last_active":"2024-04-24T08:10:20.258895+0000","last_peered":"2024-04-24T08:10:20.258895+0000","last_clean":"2024-04-24T08:10:20.258895+0000","last_became_active":"2024-04-24T08:07:51.070298+0000","last_became_peered":"2024-04-24T08:07:51.070298+0000","last_unstale":"2024-04-24T08:10:20.258895+0000","last_undegraded":"2024-04-24T08:10:20.258895+0000","last_fullsized":"2024-04-24T08:10:20.258895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:56:41.706796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.301283+0000","last_change":"2024-04-24T08:07:52.073417+0000","last_active":"2024-04-24T08:10:20.301283+0000","last_peered":"2024-04-24T08:10:20.301283+0000","last_clean":"2024-04-24T08:10:20.301283+0000","last_became_active":"2024-04-24T08:07:52.073051+0000","last_became_peered":"2024-04-24T08:07:52.073051+0000","last_unstale":"2024-04-24T08:10:20.301283+0000","last_undegraded":"2024-04-24T08:10:20.301283+0000","last_fullsized":"2024-04-24T08:10:20.301283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:54:58.338279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128064+0000","last_change":"2024-04-24T08:08:51.172525+0000","last_active":"2024-04-24T08:10:20.128064+0000","last_peered":"2024-04-24T08:10:20.128064+0000","last_clean":"2024-04-24T08:10:20.128064+0000","last_became_active":"2024-04-24T08:08:51.171924+0000","last_became_peered":"2024-04-24T08:08:51.171924+0000","last_unstale":"2024-04-24T08:10:20.128064+0000","last_undegraded":"2024-04-24T08:10:20.128064+0000","last_fullsized":"2024-04-24T08:10:20.128064+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:25:42.067150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":176,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.443285+0000","last_change":"2024-04-24T08:07:51.076937+0000","last_active":"2024-04-24T08:10:20.443285+0000","last_peered":"2024-04-24T08:10:20.443285+0000","last_clean":"2024-04-24T08:10:20.443285+0000","last_became_active":"2024-04-24T08:07:51.076621+0000","last_became_peered":"2024-04-24T08:07:51.076621+0000","last_unstale":"2024-04-24T08:10:20.443285+0000","last_undegraded":"2024-04-24T08:10:20.443285+0000","last_fullsized":"2024-04-24T08:10:20.443285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:09.966994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288284+0000","last_change":"2024-04-24T08:07:52.054150+0000","last_active":"2024-04-24T08:10:20.288284+0000","last_peered":"2024-04-24T08:10:20.288284+0000","last_clean":"2024-04-24T08:10:20.288284+0000","last_became_active":"2024-04-24T08:07:52.053817+0000","last_became_peered":"2024-04-24T08:07:52.053817+0000","last_unstale":"2024-04-24T08:10:20.288284+0000","last_undegraded":"2024-04-24T08:10:20.288284+0000","last_fullsized":"2024-04-24T08:10:20.288284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:46:16.570004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128899+0000","last_change":"2024-04-24T08:07:51.045431+0000","last_active":"2024-04-24T08:10:20.128899+0000","last_peered":"2024-04-24T08:10:20.128899+0000","last_clean":"2024-04-24T08:10:20.128899+0000","last_became_active":"2024-04-24T08:07:51.045112+0000","last_became_peered":"2024-04-24T08:07:51.045112+0000","last_unstale":"2024-04-24T08:10:20.128899+0000","last_undegraded":"2024-04-24T08:10:20.128899+0000","last_fullsized":"2024-04-24T08:10:20.128899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:52:05.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":358,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.258842+0000","last_change":"2024-04-24T08:06:31.583310+0000","last_active":"2024-04-24T08:10:20.258842+0000","last_peered":"2024-04-24T08:10:20.258842+0000","last_clean":"2024-04-24T08:10:20.258842+0000","last_became_active":"2024-04-24T08:06:31.582735+0000","last_became_peered":"2024-04-24T08:06:31.582735+0000","last_unstale":"2024-04-24T08:10:20.258842+0000","last_undegraded":"2024-04-24T08:10:20.258842+0000","last_fullsized":"2024-04-24T08:10:20.258842+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:02:34.411025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:02:34.411025+0000","last_clean_scrub_stamp":"2024-04-24T08:02:34.411025+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:48.816092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.442781+0000","last_change":"2024-04-24T08:07:52.063961+0000","last_active":"2024-04-24T08:10:20.442781+0000","last_peered":"2024-04-24T08:10:20.442781+0000","last_clean":"2024-04-24T08:10:20.442781+0000","last_became_active":"2024-04-24T08:07:52.063608+0000","last_became_peered":"2024-04-24T08:07:52.063608+0000","last_unstale":"2024-04-24T08:10:20.442781+0000","last_undegraded":"2024-04-24T08:10:20.442781+0000","last_fullsized":"2024-04-24T08:10:20.442781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:40:50.233349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259215+0000","last_change":"2024-04-24T08:07:51.045626+0000","last_active":"2024-04-24T08:10:20.259215+0000","last_peered":"2024-04-24T08:10:20.259215+0000","last_clean":"2024-04-24T08:10:20.259215+0000","last_became_active":"2024-04-24T08:07:51.045293+0000","last_became_peered":"2024-04-24T08:07:51.045293+0000","last_unstale":"2024-04-24T08:10:20.259215+0000","last_undegraded":"2024-04-24T08:10:20.259215+0000","last_fullsized":"2024-04-24T08:10:20.259215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:13:04.601331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128055+0000","last_change":"2024-04-24T08:07:51.079228+0000","last_active":"2024-04-24T08:10:20.128055+0000","last_peered":"2024-04-24T08:10:20.128055+0000","last_clean":"2024-04-24T08:10:20.128055+0000","last_became_active":"2024-04-24T08:07:51.078932+0000","last_became_peered":"2024-04-24T08:07:51.078932+0000","last_unstale":"2024-04-24T08:10:20.128055+0000","last_undegraded":"2024-04-24T08:10:20.128055+0000","last_fullsized":"2024-04-24T08:10:20.128055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:58:32.024133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.442826+0000","last_change":"2024-04-24T08:07:52.065943+0000","last_active":"2024-04-24T08:10:20.442826+0000","last_peered":"2024-04-24T08:10:20.442826+0000","last_clean":"2024-04-24T08:10:20.442826+0000","last_became_active":"2024-04-24T08:07:52.062810+0000","last_became_peered":"2024-04-24T08:07:52.062810+0000","last_unstale":"2024-04-24T08:10:20.442826+0000","last_undegraded":"2024-04-24T08:10:20.442826+0000","last_fullsized":"2024-04-24T08:10:20.442826+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:40:08.803652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'45","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.444289+0000","last_change":"2024-04-24T08:08:51.193009+0000","last_active":"2024-04-24T08:10:20.444289+0000","last_peered":"2024-04-24T08:10:20.444289+0000","last_clean":"2024-04-24T08:10:20.444289+0000","last_became_active":"2024-04-24T08:08:51.180025+0000","last_became_peered":"2024-04-24T08:08:51.180025+0000","last_unstale":"2024-04-24T08:10:20.444289+0000","last_undegraded":"2024-04-24T08:10:20.444289+0000","last_fullsized":"2024-04-24T08:10:20.444289+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:31:19.397470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":81,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288903+0000","last_change":"2024-04-24T08:07:52.058764+0000","last_active":"2024-04-24T08:10:20.288903+0000","last_peered":"2024-04-24T08:10:20.288903+0000","last_clean":"2024-04-24T08:10:20.288903+0000","last_became_active":"2024-04-24T08:07:52.058336+0000","last_became_peered":"2024-04-24T08:07:52.058336+0000","last_unstale":"2024-04-24T08:10:20.288903+0000","last_undegraded":"2024-04-24T08:10:20.288903+0000","last_fullsized":"2024-04-24T08:10:20.288903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:05:37.245521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.129060+0000","last_change":"2024-04-24T08:07:51.067154+0000","last_active":"2024-04-24T08:10:20.129060+0000","last_peered":"2024-04-24T08:10:20.129060+0000","last_clean":"2024-04-24T08:10:20.129060+0000","last_became_active":"2024-04-24T08:07:51.066769+0000","last_became_peered":"2024-04-24T08:07:51.066769+0000","last_unstale":"2024-04-24T08:10:20.129060+0000","last_undegraded":"2024-04-24T08:10:20.129060+0000","last_fullsized":"2024-04-24T08:10:20.129060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:39:25.908189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.504066+0000","last_change":"2024-04-24T08:07:52.072134+0000","last_active":"2024-04-24T08:10:20.504066+0000","last_peered":"2024-04-24T08:10:20.504066+0000","last_clean":"2024-04-24T08:10:20.504066+0000","last_became_active":"2024-04-24T08:07:52.071703+0000","last_became_peered":"2024-04-24T08:07:52.071703+0000","last_unstale":"2024-04-24T08:10:20.504066+0000","last_undegraded":"2024-04-24T08:10:20.504066+0000","last_fullsized":"2024-04-24T08:10:20.504066+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:59.829988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498172+0000","last_change":"2024-04-24T08:07:51.059151+0000","last_active":"2024-04-24T08:10:20.498172+0000","last_peered":"2024-04-24T08:10:20.498172+0000","last_clean":"2024-04-24T08:10:20.498172+0000","last_became_active":"2024-04-24T08:07:51.058872+0000","last_became_peered":"2024-04-24T08:07:51.058872+0000","last_unstale":"2024-04-24T08:10:20.498172+0000","last_undegraded":"2024-04-24T08:10:20.498172+0000","last_fullsized":"2024-04-24T08:10:20.498172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:36.023595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.127663+0000","last_change":"2024-04-24T08:07:52.056919+0000","last_active":"2024-04-24T08:10:20.127663+0000","last_peered":"2024-04-24T08:10:20.127663+0000","last_clean":"2024-04-24T08:10:20.127663+0000","last_became_active":"2024-04-24T08:07:52.056573+0000","last_became_peered":"2024-04-24T08:07:52.056573+0000","last_unstale":"2024-04-24T08:10:20.127663+0000","last_undegraded":"2024-04-24T08:10:20.127663+0000","last_fullsized":"2024-04-24T08:10:20.127663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:58:17.939465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288463+0000","last_change":"2024-04-24T08:07:51.056325+0000","last_active":"2024-04-24T08:10:20.288463+0000","last_peered":"2024-04-24T08:10:20.288463+0000","last_clean":"2024-04-24T08:10:20.288463+0000","last_became_active":"2024-04-24T08:07:51.055862+0000","last_became_peered":"2024-04-24T08:07:51.055862+0000","last_unstale":"2024-04-24T08:10:20.288463+0000","last_undegraded":"2024-04-24T08:10:20.288463+0000","last_fullsized":"2024-04-24T08:10:20.288463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:11:44.487530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288781+0000","last_change":"2024-04-24T08:07:52.065570+0000","last_active":"2024-04-24T08:10:20.288781+0000","last_peered":"2024-04-24T08:10:20.288781+0000","last_clean":"2024-04-24T08:10:20.288781+0000","last_became_active":"2024-04-24T08:07:52.065196+0000","last_became_peered":"2024-04-24T08:07:52.065196+0000","last_unstale":"2024-04-24T08:10:20.288781+0000","last_undegraded":"2024-04-24T08:10:20.288781+0000","last_fullsized":"2024-04-24T08:10:20.288781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:48:09.474159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.497754+0000","last_change":"2024-04-24T08:07:51.067187+0000","last_active":"2024-04-24T08:10:20.497754+0000","last_peered":"2024-04-24T08:10:20.497754+0000","last_clean":"2024-04-24T08:10:20.497754+0000","last_became_active":"2024-04-24T08:07:51.066808+0000","last_became_peered":"2024-04-24T08:07:51.066808+0000","last_unstale":"2024-04-24T08:10:20.497754+0000","last_undegraded":"2024-04-24T08:10:20.497754+0000","last_fullsized":"2024-04-24T08:10:20.497754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:32:11.852191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288593+0000","last_change":"2024-04-24T08:07:52.078426+0000","last_active":"2024-04-24T08:10:20.288593+0000","last_peered":"2024-04-24T08:10:20.288593+0000","last_clean":"2024-04-24T08:10:20.288593+0000","last_became_active":"2024-04-24T08:07:52.078076+0000","last_became_peered":"2024-04-24T08:07:52.078076+0000","last_unstale":"2024-04-24T08:10:20.288593+0000","last_undegraded":"2024-04-24T08:10:20.288593+0000","last_fullsized":"2024-04-24T08:10:20.288593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:55:29.885688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.127956+0000","last_change":"2024-04-24T08:07:51.042601+0000","last_active":"2024-04-24T08:10:20.127956+0000","last_peered":"2024-04-24T08:10:20.127956+0000","last_clean":"2024-04-24T08:10:20.127956+0000","last_became_active":"2024-04-24T08:07:51.042252+0000","last_became_peered":"2024-04-24T08:07:51.042252+0000","last_unstale":"2024-04-24T08:10:20.127956+0000","last_undegraded":"2024-04-24T08:10:20.127956+0000","last_fullsized":"2024-04-24T08:10:20.127956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:11:44.640303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259102+0000","last_change":"2024-04-24T08:07:52.063142+0000","last_active":"2024-04-24T08:10:20.259102+0000","last_peered":"2024-04-24T08:10:20.259102+0000","last_clean":"2024-04-24T08:10:20.259102+0000","last_became_active":"2024-04-24T08:07:52.062263+0000","last_became_peered":"2024-04-24T08:07:52.062263+0000","last_unstale":"2024-04-24T08:10:20.259102+0000","last_undegraded":"2024-04-24T08:10:20.259102+0000","last_fullsized":"2024-04-24T08:10:20.259102+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:38.879380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.129102+0000","last_change":"2024-04-24T08:08:51.184795+0000","last_active":"2024-04-24T08:10:20.129102+0000","last_peered":"2024-04-24T08:10:20.129102+0000","last_clean":"2024-04-24T08:10:20.129102+0000","last_became_active":"2024-04-24T08:08:51.179560+0000","last_became_peered":"2024-04-24T08:08:51.179560+0000","last_unstale":"2024-04-24T08:10:20.129102+0000","last_undegraded":"2024-04-24T08:10:20.129102+0000","last_fullsized":"2024-04-24T08:10:20.129102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:31:23.795126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259332+0000","last_change":"2024-04-24T08:07:52.055958+0000","last_active":"2024-04-24T08:10:20.259332+0000","last_peered":"2024-04-24T08:10:20.259332+0000","last_clean":"2024-04-24T08:10:20.259332+0000","last_became_active":"2024-04-24T08:07:52.055598+0000","last_became_peered":"2024-04-24T08:07:52.055598+0000","last_unstale":"2024-04-24T08:10:20.259332+0000","last_undegraded":"2024-04-24T08:10:20.259332+0000","last_fullsized":"2024-04-24T08:10:20.259332+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:12:22.452576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'39","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.439516+0000","last_change":"2024-04-24T08:07:51.073234+0000","last_active":"2024-04-24T08:10:20.439516+0000","last_peered":"2024-04-24T08:10:20.439516+0000","last_clean":"2024-04-24T08:10:20.439516+0000","last_became_active":"2024-04-24T08:07:51.072887+0000","last_became_peered":"2024-04-24T08:07:51.072887+0000","last_unstale":"2024-04-24T08:10:20.439516+0000","last_undegraded":"2024-04-24T08:10:20.439516+0000","last_fullsized":"2024-04-24T08:10:20.439516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:28:15.908859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259151+0000","last_change":"2024-04-24T08:07:52.061188+0000","last_active":"2024-04-24T08:10:20.259151+0000","last_peered":"2024-04-24T08:10:20.259151+0000","last_clean":"2024-04-24T08:10:20.259151+0000","last_became_active":"2024-04-24T08:07:52.059616+0000","last_became_peered":"2024-04-24T08:07:52.059616+0000","last_unstale":"2024-04-24T08:10:20.259151+0000","last_undegraded":"2024-04-24T08:10:20.259151+0000","last_fullsized":"2024-04-24T08:10:20.259151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:43.546091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.442965+0000","last_change":"2024-04-24T08:07:51.078300+0000","last_active":"2024-04-24T08:10:20.442965+0000","last_peered":"2024-04-24T08:10:20.442965+0000","last_clean":"2024-04-24T08:10:20.442965+0000","last_became_active":"2024-04-24T08:07:51.076026+0000","last_became_peered":"2024-04-24T08:07:51.076026+0000","last_unstale":"2024-04-24T08:10:20.442965+0000","last_undegraded":"2024-04-24T08:10:20.442965+0000","last_fullsized":"2024-04-24T08:10:20.442965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:45:31.494213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259123+0000","last_change":"2024-04-24T08:07:52.056148+0000","last_active":"2024-04-24T08:10:20.259123+0000","last_peered":"2024-04-24T08:10:20.259123+0000","last_clean":"2024-04-24T08:10:20.259123+0000","last_became_active":"2024-04-24T08:07:52.055768+0000","last_became_peered":"2024-04-24T08:07:52.055768+0000","last_unstale":"2024-04-24T08:10:20.259123+0000","last_undegraded":"2024-04-24T08:10:20.259123+0000","last_fullsized":"2024-04-24T08:10:20.259123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:52:48.503479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288880+0000","last_change":"2024-04-24T08:07:52.056611+0000","last_active":"2024-04-24T08:10:20.288880+0000","last_peered":"2024-04-24T08:10:20.288880+0000","last_clean":"2024-04-24T08:10:20.288880+0000","last_became_active":"2024-04-24T08:07:52.055788+0000","last_became_peered":"2024-04-24T08:07:52.055788+0000","last_unstale":"2024-04-24T08:10:20.288880+0000","last_undegraded":"2024-04-24T08:10:20.288880+0000","last_fullsized":"2024-04-24T08:10:20.288880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:21:28.997588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.497715+0000","last_change":"2024-04-24T08:08:51.177574+0000","last_active":"2024-04-24T08:10:20.497715+0000","last_peered":"2024-04-24T08:10:20.497715+0000","last_clean":"2024-04-24T08:10:20.497715+0000","last_became_active":"2024-04-24T08:08:51.177145+0000","last_became_peered":"2024-04-24T08:08:51.177145+0000","last_unstale":"2024-04-24T08:10:20.497715+0000","last_undegraded":"2024-04-24T08:10:20.497715+0000","last_fullsized":"2024-04-24T08:10:20.497715+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:49:19.438233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.497647+0000","last_change":"2024-04-24T08:09:51.441560+0000","last_active":"2024-04-24T08:10:20.497647+0000","last_peered":"2024-04-24T08:10:20.497647+0000","last_clean":"2024-04-24T08:10:20.497647+0000","last_became_active":"2024-04-24T08:09:51.441192+0000","last_became_peered":"2024-04-24T08:09:51.441192+0000","last_unstale":"2024-04-24T08:10:20.497647+0000","last_undegraded":"2024-04-24T08:10:20.497647+0000","last_fullsized":"2024-04-24T08:10:20.497647+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:32:26.311164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.503483+0000","last_change":"2024-04-24T08:08:51.250206+0000","last_active":"2024-04-24T08:10:20.503483+0000","last_peered":"2024-04-24T08:10:20.503483+0000","last_clean":"2024-04-24T08:10:20.503483+0000","last_became_active":"2024-04-24T08:08:51.249677+0000","last_became_peered":"2024-04-24T08:08:51.249677+0000","last_unstale":"2024-04-24T08:10:20.503483+0000","last_undegraded":"2024-04-24T08:10:20.503483+0000","last_fullsized":"2024-04-24T08:10:20.503483+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:06:26.655987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.442885+0000","last_change":"2024-04-24T08:07:52.078604+0000","last_active":"2024-04-24T08:10:20.442885+0000","last_peered":"2024-04-24T08:10:20.442885+0000","last_clean":"2024-04-24T08:10:20.442885+0000","last_became_active":"2024-04-24T08:07:52.078122+0000","last_became_peered":"2024-04-24T08:07:52.078122+0000","last_unstale":"2024-04-24T08:10:20.442885+0000","last_undegraded":"2024-04-24T08:10:20.442885+0000","last_fullsized":"2024-04-24T08:10:20.442885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:24:19.062821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.289182+0000","last_change":"2024-04-24T08:07:52.055986+0000","last_active":"2024-04-24T08:10:20.289182+0000","last_peered":"2024-04-24T08:10:20.289182+0000","last_clean":"2024-04-24T08:10:20.289182+0000","last_became_active":"2024-04-24T08:07:52.055609+0000","last_became_peered":"2024-04-24T08:07:52.055609+0000","last_unstale":"2024-04-24T08:10:20.289182+0000","last_undegraded":"2024-04-24T08:10:20.289182+0000","last_fullsized":"2024-04-24T08:10:20.289182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:38:04.731204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498055+0000","last_change":"2024-04-24T08:07:52.055605+0000","last_active":"2024-04-24T08:10:20.498055+0000","last_peered":"2024-04-24T08:10:20.498055+0000","last_clean":"2024-04-24T08:10:20.498055+0000","last_became_active":"2024-04-24T08:07:52.055268+0000","last_became_peered":"2024-04-24T08:07:52.055268+0000","last_unstale":"2024-04-24T08:10:20.498055+0000","last_undegraded":"2024-04-24T08:10:20.498055+0000","last_fullsized":"2024-04-24T08:10:20.498055+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:50:29.197484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288890+0000","last_change":"2024-04-24T08:07:52.064590+0000","last_active":"2024-04-24T08:10:20.288890+0000","last_peered":"2024-04-24T08:10:20.288890+0000","last_clean":"2024-04-24T08:10:20.288890+0000","last_became_active":"2024-04-24T08:07:52.064113+0000","last_became_peered":"2024-04-24T08:07:52.064113+0000","last_unstale":"2024-04-24T08:10:20.288890+0000","last_undegraded":"2024-04-24T08:10:20.288890+0000","last_fullsized":"2024-04-24T08:10:20.288890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:36:10.926145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297622+0000","last_change":"2024-04-24T08:08:51.250599+0000","last_active":"2024-04-24T08:10:20.297622+0000","last_peered":"2024-04-24T08:10:20.297622+0000","last_clean":"2024-04-24T08:10:20.297622+0000","last_became_active":"2024-04-24T08:08:51.250197+0000","last_became_peered":"2024-04-24T08:08:51.250197+0000","last_unstale":"2024-04-24T08:10:20.297622+0000","last_undegraded":"2024-04-24T08:10:20.297622+0000","last_fullsized":"2024-04-24T08:10:20.297622+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:31:30.729911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6],"acting":[5,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288304+0000","last_change":"2024-04-24T08:07:52.068308+0000","last_active":"2024-04-24T08:10:20.288304+0000","last_peered":"2024-04-24T08:10:20.288304+0000","last_clean":"2024-04-24T08:10:20.288304+0000","last_became_active":"2024-04-24T08:07:52.067786+0000","last_became_peered":"2024-04-24T08:07:52.067786+0000","last_unstale":"2024-04-24T08:10:20.288304+0000","last_undegraded":"2024-04-24T08:10:20.288304+0000","last_fullsized":"2024-04-24T08:10:20.288304+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:26:09.349120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498381+0000","last_change":"2024-04-24T08:07:52.053089+0000","last_active":"2024-04-24T08:10:20.498381+0000","last_peered":"2024-04-24T08:10:20.498381+0000","last_clean":"2024-04-24T08:10:20.498381+0000","last_became_active":"2024-04-24T08:07:52.052753+0000","last_became_peered":"2024-04-24T08:07:52.052753+0000","last_unstale":"2024-04-24T08:10:20.498381+0000","last_undegraded":"2024-04-24T08:10:20.498381+0000","last_fullsized":"2024-04-24T08:10:20.498381+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:02:14.583922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.442875+0000","last_change":"2024-04-24T08:07:52.076633+0000","last_active":"2024-04-24T08:10:20.442875+0000","last_peered":"2024-04-24T08:10:20.442875+0000","last_clean":"2024-04-24T08:10:20.442875+0000","last_became_active":"2024-04-24T08:07:52.076280+0000","last_became_peered":"2024-04-24T08:07:52.076280+0000","last_unstale":"2024-04-24T08:10:20.442875+0000","last_undegraded":"2024-04-24T08:10:20.442875+0000","last_fullsized":"2024-04-24T08:10:20.442875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:29:59.823772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288825+0000","last_change":"2024-04-24T08:07:52.053017+0000","last_active":"2024-04-24T08:10:20.288825+0000","last_peered":"2024-04-24T08:10:20.288825+0000","last_clean":"2024-04-24T08:10:20.288825+0000","last_became_active":"2024-04-24T08:07:52.052325+0000","last_became_peered":"2024-04-24T08:07:52.052325+0000","last_unstale":"2024-04-24T08:10:20.288825+0000","last_undegraded":"2024-04-24T08:10:20.288825+0000","last_fullsized":"2024-04-24T08:10:20.288825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:05:58.161360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108294,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598028,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1010,"num_read_kb":614,"num_write":296,"num_write_kb":1442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":616,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":569344,"data_stored":1787871,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351020,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1019904,"data_stored":881474,"data_compressed":33679,"data_compressed_allocated":45056,"data_compressed_original":434176,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612656,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1171456,"data_stored":1279674,"data_compressed":56787,"data_compressed_allocated":73728,"data_compressed_original":753664,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776181,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":881676,"data_compressed":33679,"data_compressed_allocated":45056,"data_compressed_original":434176,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037819,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1007616,"data_stored":488380,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332161,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":1072279,"data_compressed":47714,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495688,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":487849,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1060864,"data_stored":551786,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":489120,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":553753,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182488,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":488672,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920852,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":490182,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757326,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1089536,"data_stored":1212045,"data_compressed":56686,"data_compressed_allocated":73728,"data_compressed_original":753664,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":64144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":66111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134309,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":201547,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":393832,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:10:26.347 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-04-24T08:10:26.348 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-04-24T08:10:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.588+0000 7fd7ae800640 1 -- 172.21.15.28:0/2584704456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7a8101570 msgr2=0x7fd7a8103960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.588+0000 7fd7ae800640 1 --2- 172.21.15.28:0/2584704456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7a8101570 0x7fd7a8103960 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd79c009900 tx=0x7fd79c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.588+0000 7fd7ae800640 1 -- 172.21.15.28:0/2584704456 shutdown_connections 2024-04-24T08:10:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.588+0000 7fd7ae800640 1 --2- 172.21.15.28:0/2584704456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a8103f30 0x7fd7a8106340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.588+0000 7fd7ae800640 1 --2- 172.21.15.28:0/2584704456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7a81069d0 0x7fd7a8108e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.588+0000 7fd7ae800640 1 --2- 172.21.15.28:0/2584704456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7a8101570 0x7fd7a8103960 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.588+0000 7fd7ae800640 1 -- 172.21.15.28:0/2584704456 >> 172.21.15.28:0/2584704456 conn(0x7fd7a80fb630 msgr2=0x7fd7a80fda50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.588+0000 7fd7ae800640 1 -- 172.21.15.28:0/2584704456 shutdown_connections 2024-04-24T08:10:26.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.588+0000 7fd7ae800640 1 -- 172.21.15.28:0/2584704456 wait complete. 2024-04-24T08:10:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.589+0000 7fd7ae800640 1 Processor -- start 2024-04-24T08:10:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.589+0000 7fd7ae800640 1 -- start start 2024-04-24T08:10:26.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.590+0000 7fd7ae800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7a8101570 0x7fd7a810b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.590+0000 7fd7ae800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a8103f30 0x7fd7a8109cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.590+0000 7fd7ae800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7a81069d0 0x7fd7a810a230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.590+0000 7fd7ae800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a810db60 con 0x7fd7a8103f30 2024-04-24T08:10:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.590+0000 7fd7ae800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a810d9e0 con 0x7fd7a81069d0 2024-04-24T08:10:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.590+0000 7fd7ae800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd7a810dce0 con 0x7fd7a8101570 2024-04-24T08:10:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.590+0000 7fd7aca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a8103f30 0x7fd7a8109cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.590+0000 7fd7ad400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7a8101570 0x7fd7a810b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.590+0000 7fd7aca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a8103f30 0x7fd7a8109cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33952/0 (socket says 172.21.15.28:33952) 2024-04-24T08:10:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.590+0000 7fd7aca00640 1 -- 172.21.15.28:0/3696219119 learned_addr learned my addr 172.21.15.28:0/3696219119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.590+0000 7fd7ade00640 1 --2- 172.21.15.28:0/3696219119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7a81069d0 0x7fd7a810a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.590+0000 7fd7ad400640 1 -- 172.21.15.28:0/3696219119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7a81069d0 msgr2=0x7fd7a810a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.590+0000 7fd7ad400640 1 --2- 172.21.15.28:0/3696219119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7a81069d0 0x7fd7a810a230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.590+0000 7fd7ad400640 1 -- 172.21.15.28:0/3696219119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a8103f30 msgr2=0x7fd7a8109cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.590+0000 7fd7ad400640 1 --2- 172.21.15.28:0/3696219119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a8103f30 0x7fd7a8109cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.590+0000 7fd7ad400640 1 -- 172.21.15.28:0/3696219119 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7a810aac0 con 0x7fd7a8101570 2024-04-24T08:10:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.590+0000 7fd7ade00640 1 --2- 172.21.15.28:0/3696219119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7a81069d0 0x7fd7a810a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.591+0000 7fd7ad400640 1 --2- 172.21.15.28:0/3696219119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7a8101570 0x7fd7a810b6b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd79c02a910 tx=0x7fd79c032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.591+0000 7fd7a6000640 1 -- 172.21.15.28:0/3696219119 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c03f030 con 0x7fd7a8101570 2024-04-24T08:10:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.591+0000 7fd7a6000640 1 -- 172.21.15.28:0/3696219119 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd79c003ce0 con 0x7fd7a8101570 2024-04-24T08:10:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.591+0000 7fd7ae800640 1 -- 172.21.15.28:0/3696219119 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7a81a2230 con 0x7fd7a8101570 2024-04-24T08:10:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.591+0000 7fd7a6000640 1 -- 172.21.15.28:0/3696219119 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd79c035070 con 0x7fd7a8101570 2024-04-24T08:10:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.591+0000 7fd7ae800640 1 -- 172.21.15.28:0/3696219119 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7a81a2750 con 0x7fd7a8101570 2024-04-24T08:10:26.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.592+0000 7fd7ae800640 1 -- 172.21.15.28:0/3696219119 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7a80fd120 con 0x7fd7a8101570 2024-04-24T08:10:26.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.594+0000 7fd7a6000640 1 -- 172.21.15.28:0/3696219119 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7fd79c042030 con 0x7fd7a8101570 2024-04-24T08:10:26.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.596+0000 7fd7a6000640 1 --2- 172.21.15.28:0/3696219119 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd7900a3e40 0x7fd7900a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:26.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.596+0000 7fd7a6000640 1 -- 172.21.15.28:0/3696219119 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fd79c0f0e90 con 0x7fd7a8101570 2024-04-24T08:10:26.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.597+0000 7fd7aca00640 1 --2- 172.21.15.28:0/3696219119 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd7900a3e40 0x7fd7900a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:26.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.597+0000 7fd7aca00640 1 --2- 172.21.15.28:0/3696219119 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd7900a3e40 0x7fd7900a6300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd7a810b420 tx=0x7fd794005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:26.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.601+0000 7fd7a6000640 1 -- 172.21.15.28:0/3696219119 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd79c0bcce0 con 0x7fd7a8101570 2024-04-24T08:10:26.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.849+0000 7fd7ae800640 1 -- 172.21.15.28:0/3696219119 --> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd7a8062440 con 0x7fd7900a3e40 2024-04-24T08:10:26.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.859+0000 7fd7a6000640 1 -- 172.21.15.28:0/3696219119 <== mgr.25225 v2:172.21.15.28:6800/353036813 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+258067 (secure 0 0 0) 0x7fd7a8062440 con 0x7fd7900a3e40 2024-04-24T08:10:26.862 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:10:26.863 INFO:teuthology.orchestra.run.smithi028.stderr:dumped all 2024-04-24T08:10:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.863+0000 7fd7ae800640 1 -- 172.21.15.28:0/3696219119 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd7900a3e40 msgr2=0x7fd7900a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.863+0000 7fd7ae800640 1 --2- 172.21.15.28:0/3696219119 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd7900a3e40 0x7fd7900a6300 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd7a810b420 tx=0x7fd794005f90 comp rx=0 tx=0).stop 2024-04-24T08:10:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.863+0000 7fd7ae800640 1 -- 172.21.15.28:0/3696219119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7a8101570 msgr2=0x7fd7a810b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.863+0000 7fd7ae800640 1 --2- 172.21.15.28:0/3696219119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7a8101570 0x7fd7a810b6b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd79c02a910 tx=0x7fd79c032ce0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.864+0000 7fd7ae800640 1 -- 172.21.15.28:0/3696219119 shutdown_connections 2024-04-24T08:10:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.864+0000 7fd7ae800640 1 --2- 172.21.15.28:0/3696219119 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd7900a3e40 0x7fd7900a6300 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.864+0000 7fd7ae800640 1 --2- 172.21.15.28:0/3696219119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd7a8103f30 0x7fd7a8109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.864+0000 7fd7ae800640 1 --2- 172.21.15.28:0/3696219119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7a81069d0 0x7fd7a810a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.864+0000 7fd7ae800640 1 --2- 172.21.15.28:0/3696219119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd7a8101570 0x7fd7a810b6b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.864+0000 7fd7ae800640 1 -- 172.21.15.28:0/3696219119 >> 172.21.15.28:0/3696219119 conn(0x7fd7a80fb630 msgr2=0x7fd7a8101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.864+0000 7fd7ae800640 1 -- 172.21.15.28:0/3696219119 shutdown_connections 2024-04-24T08:10:26.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:26.864+0000 7fd7ae800640 1 -- 172.21.15.28:0/3696219119 wait complete. 2024-04-24T08:10:26.883 INFO:teuthology.orchestra.run.smithi028.stdout:{"pg_ready":true,"pg_map":{"version":6,"stamp":"2024-04-24T08:10:26.156645+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":1188630,"num_objects":66,"num_object_clones":0,"num_object_copies":198,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":66,"num_whiteouts":0,"num_read":1146,"num_read_kb":733,"num_write":479,"num_write_kb":4382,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12,"num_bytes_recovered":1181352,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":871,"ondisk_log_size":871,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":365600,"kb_used_data":12240,"kb_used_omap":18,"kb_used_meta":353197,"kb_avail":1124576224,"statfs":{"total":1151940427776,"available":1151566053376,"internally_reserved":0,"allocated":12533760,"data_stored":8876890,"data_compressed":330276,"data_compressed_allocated":442368,"data_compressed_original":3284992,"omap_allocated":19140,"internal_metadata":361674044},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3302,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":23,"num_read_kb":19,"num_write":3,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.979281"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.289087+0000","last_change":"2024-04-24T08:07:52.074319+0000","last_active":"2024-04-24T08:10:20.289087+0000","last_peered":"2024-04-24T08:10:20.289087+0000","last_clean":"2024-04-24T08:10:20.289087+0000","last_became_active":"2024-04-24T08:07:52.073923+0000","last_became_peered":"2024-04-24T08:07:52.073923+0000","last_unstale":"2024-04-24T08:10:20.289087+0000","last_undegraded":"2024-04-24T08:10:20.289087+0000","last_fullsized":"2024-04-24T08:10:20.289087+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:29.325904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.504231+0000","last_change":"2024-04-24T08:07:52.068316+0000","last_active":"2024-04-24T08:10:20.504231+0000","last_peered":"2024-04-24T08:10:20.504231+0000","last_clean":"2024-04-24T08:10:20.504231+0000","last_became_active":"2024-04-24T08:07:52.067961+0000","last_became_peered":"2024-04-24T08:07:52.067961+0000","last_unstale":"2024-04-24T08:10:20.504231+0000","last_undegraded":"2024-04-24T08:10:20.504231+0000","last_fullsized":"2024-04-24T08:10:20.504231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:35:14.079348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.503658+0000","last_change":"2024-04-24T08:07:52.066870+0000","last_active":"2024-04-24T08:10:20.503658+0000","last_peered":"2024-04-24T08:10:20.503658+0000","last_clean":"2024-04-24T08:10:20.503658+0000","last_became_active":"2024-04-24T08:07:52.066481+0000","last_became_peered":"2024-04-24T08:07:52.066481+0000","last_unstale":"2024-04-24T08:10:20.503658+0000","last_undegraded":"2024-04-24T08:10:20.503658+0000","last_fullsized":"2024-04-24T08:10:20.503658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:25:45.526667+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.129237+0000","last_change":"2024-04-24T08:07:52.061548+0000","last_active":"2024-04-24T08:10:20.129237+0000","last_peered":"2024-04-24T08:10:20.129237+0000","last_clean":"2024-04-24T08:10:20.129237+0000","last_became_active":"2024-04-24T08:07:52.060519+0000","last_became_peered":"2024-04-24T08:07:52.060519+0000","last_unstale":"2024-04-24T08:10:20.129237+0000","last_undegraded":"2024-04-24T08:10:20.129237+0000","last_fullsized":"2024-04-24T08:10:20.129237+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:47:20.952035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128104+0000","last_change":"2024-04-24T08:07:52.063805+0000","last_active":"2024-04-24T08:10:20.128104+0000","last_peered":"2024-04-24T08:10:20.128104+0000","last_clean":"2024-04-24T08:10:20.128104+0000","last_became_active":"2024-04-24T08:07:52.063414+0000","last_became_peered":"2024-04-24T08:07:52.063414+0000","last_unstale":"2024-04-24T08:10:20.128104+0000","last_undegraded":"2024-04-24T08:10:20.128104+0000","last_fullsized":"2024-04-24T08:10:20.128104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:06:24.465035+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.301125+0000","last_change":"2024-04-24T08:07:52.050152+0000","last_active":"2024-04-24T08:10:20.301125+0000","last_peered":"2024-04-24T08:10:20.301125+0000","last_clean":"2024-04-24T08:10:20.301125+0000","last_became_active":"2024-04-24T08:07:52.049805+0000","last_became_peered":"2024-04-24T08:07:52.049805+0000","last_unstale":"2024-04-24T08:10:20.301125+0000","last_undegraded":"2024-04-24T08:10:20.301125+0000","last_fullsized":"2024-04-24T08:10:20.301125+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:47:34.644263+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.258824+0000","last_change":"2024-04-24T08:07:52.055726+0000","last_active":"2024-04-24T08:10:20.258824+0000","last_peered":"2024-04-24T08:10:20.258824+0000","last_clean":"2024-04-24T08:10:20.258824+0000","last_became_active":"2024-04-24T08:07:52.054563+0000","last_became_peered":"2024-04-24T08:07:52.054563+0000","last_unstale":"2024-04-24T08:10:20.258824+0000","last_undegraded":"2024-04-24T08:10:20.258824+0000","last_fullsized":"2024-04-24T08:10:20.258824+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:34:57.430232+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288218+0000","last_change":"2024-04-24T08:07:52.053672+0000","last_active":"2024-04-24T08:10:20.288218+0000","last_peered":"2024-04-24T08:10:20.288218+0000","last_clean":"2024-04-24T08:10:20.288218+0000","last_became_active":"2024-04-24T08:07:52.053150+0000","last_became_peered":"2024-04-24T08:07:52.053150+0000","last_unstale":"2024-04-24T08:10:20.288218+0000","last_undegraded":"2024-04-24T08:10:20.288218+0000","last_fullsized":"2024-04-24T08:10:20.288218+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:00:12.272555+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498877+0000","last_change":"2024-04-24T08:07:52.054648+0000","last_active":"2024-04-24T08:10:20.498877+0000","last_peered":"2024-04-24T08:10:20.498877+0000","last_clean":"2024-04-24T08:10:20.498877+0000","last_became_active":"2024-04-24T08:07:52.054301+0000","last_became_peered":"2024-04-24T08:07:52.054301+0000","last_unstale":"2024-04-24T08:10:20.498877+0000","last_undegraded":"2024-04-24T08:10:20.498877+0000","last_fullsized":"2024-04-24T08:10:20.498877+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:52:20.533855+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.301467+0000","last_change":"2024-04-24T08:07:52.052994+0000","last_active":"2024-04-24T08:10:20.301467+0000","last_peered":"2024-04-24T08:10:20.301467+0000","last_clean":"2024-04-24T08:10:20.301467+0000","last_became_active":"2024-04-24T08:07:52.052641+0000","last_became_peered":"2024-04-24T08:07:52.052641+0000","last_unstale":"2024-04-24T08:10:20.301467+0000","last_undegraded":"2024-04-24T08:10:20.301467+0000","last_fullsized":"2024-04-24T08:10:20.301467+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:28:02.665622+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259119+0000","last_change":"2024-04-24T08:07:52.055901+0000","last_active":"2024-04-24T08:10:20.259119+0000","last_peered":"2024-04-24T08:10:20.259119+0000","last_clean":"2024-04-24T08:10:20.259119+0000","last_became_active":"2024-04-24T08:07:52.055277+0000","last_became_peered":"2024-04-24T08:07:52.055277+0000","last_unstale":"2024-04-24T08:10:20.259119+0000","last_undegraded":"2024-04-24T08:10:20.259119+0000","last_fullsized":"2024-04-24T08:10:20.259119+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:46.950578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128973+0000","last_change":"2024-04-24T08:07:52.060329+0000","last_active":"2024-04-24T08:10:20.128973+0000","last_peered":"2024-04-24T08:10:20.128973+0000","last_clean":"2024-04-24T08:10:20.128973+0000","last_became_active":"2024-04-24T08:07:52.059914+0000","last_became_peered":"2024-04-24T08:07:52.059914+0000","last_unstale":"2024-04-24T08:10:20.128973+0000","last_undegraded":"2024-04-24T08:10:20.128973+0000","last_fullsized":"2024-04-24T08:10:20.128973+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:15:20.263916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.443115+0000","last_change":"2024-04-24T08:07:52.062108+0000","last_active":"2024-04-24T08:10:20.443115+0000","last_peered":"2024-04-24T08:10:20.443115+0000","last_clean":"2024-04-24T08:10:20.443115+0000","last_became_active":"2024-04-24T08:07:52.061665+0000","last_became_peered":"2024-04-24T08:07:52.061665+0000","last_unstale":"2024-04-24T08:10:20.443115+0000","last_undegraded":"2024-04-24T08:10:20.443115+0000","last_fullsized":"2024-04-24T08:10:20.443115+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:55:58.071807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.503324+0000","last_change":"2024-04-24T08:07:52.063033+0000","last_active":"2024-04-24T08:10:20.503324+0000","last_peered":"2024-04-24T08:10:20.503324+0000","last_clean":"2024-04-24T08:10:20.503324+0000","last_became_active":"2024-04-24T08:07:52.062610+0000","last_became_peered":"2024-04-24T08:07:52.062610+0000","last_unstale":"2024-04-24T08:10:20.503324+0000","last_undegraded":"2024-04-24T08:10:20.503324+0000","last_fullsized":"2024-04-24T08:10:20.503324+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:59:35.906720+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.503122+0000","last_change":"2024-04-24T08:07:52.064657+0000","last_active":"2024-04-24T08:10:20.503122+0000","last_peered":"2024-04-24T08:10:20.503122+0000","last_clean":"2024-04-24T08:10:20.503122+0000","last_became_active":"2024-04-24T08:07:52.064188+0000","last_became_peered":"2024-04-24T08:07:52.064188+0000","last_unstale":"2024-04-24T08:10:20.503122+0000","last_undegraded":"2024-04-24T08:10:20.503122+0000","last_fullsized":"2024-04-24T08:10:20.503122+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:56:32.139420+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498229+0000","last_change":"2024-04-24T08:07:52.067227+0000","last_active":"2024-04-24T08:10:20.498229+0000","last_peered":"2024-04-24T08:10:20.498229+0000","last_clean":"2024-04-24T08:10:20.498229+0000","last_became_active":"2024-04-24T08:07:52.066854+0000","last_became_peered":"2024-04-24T08:07:52.066854+0000","last_unstale":"2024-04-24T08:10:20.498229+0000","last_undegraded":"2024-04-24T08:10:20.498229+0000","last_fullsized":"2024-04-24T08:10:20.498229+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:25:17.580223+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.289276+0000","last_change":"2024-04-24T08:07:52.074127+0000","last_active":"2024-04-24T08:10:20.289276+0000","last_peered":"2024-04-24T08:10:20.289276+0000","last_clean":"2024-04-24T08:10:20.289276+0000","last_became_active":"2024-04-24T08:07:52.073806+0000","last_became_peered":"2024-04-24T08:07:52.073806+0000","last_unstale":"2024-04-24T08:10:20.289276+0000","last_undegraded":"2024-04-24T08:10:20.289276+0000","last_fullsized":"2024-04-24T08:10:20.289276+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:25:22.068997+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.131220+0000","last_change":"2024-04-24T08:07:52.051239+0000","last_active":"2024-04-24T08:10:20.131220+0000","last_peered":"2024-04-24T08:10:20.131220+0000","last_clean":"2024-04-24T08:10:20.131220+0000","last_became_active":"2024-04-24T08:07:52.049232+0000","last_became_peered":"2024-04-24T08:07:52.049232+0000","last_unstale":"2024-04-24T08:10:20.131220+0000","last_undegraded":"2024-04-24T08:10:20.131220+0000","last_fullsized":"2024-04-24T08:10:20.131220+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:07:39.768537+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.497883+0000","last_change":"2024-04-24T08:07:52.062119+0000","last_active":"2024-04-24T08:10:20.497883+0000","last_peered":"2024-04-24T08:10:20.497883+0000","last_clean":"2024-04-24T08:10:20.497883+0000","last_became_active":"2024-04-24T08:07:52.061781+0000","last_became_peered":"2024-04-24T08:07:52.061781+0000","last_unstale":"2024-04-24T08:10:20.497883+0000","last_undegraded":"2024-04-24T08:10:20.497883+0000","last_fullsized":"2024-04-24T08:10:20.497883+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:49:41.595368+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.129438+0000","last_change":"2024-04-24T08:07:51.052411+0000","last_active":"2024-04-24T08:10:20.129438+0000","last_peered":"2024-04-24T08:10:20.129438+0000","last_clean":"2024-04-24T08:10:20.129438+0000","last_became_active":"2024-04-24T08:07:51.051704+0000","last_became_peered":"2024-04-24T08:07:51.051704+0000","last_unstale":"2024-04-24T08:10:20.129438+0000","last_undegraded":"2024-04-24T08:10:20.129438+0000","last_fullsized":"2024-04-24T08:10:20.129438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:34:23.001338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288658+0000","last_change":"2024-04-24T08:07:52.056508+0000","last_active":"2024-04-24T08:10:20.288658+0000","last_peered":"2024-04-24T08:10:20.288658+0000","last_clean":"2024-04-24T08:10:20.288658+0000","last_became_active":"2024-04-24T08:07:52.056158+0000","last_became_peered":"2024-04-24T08:07:52.056158+0000","last_unstale":"2024-04-24T08:10:20.288658+0000","last_undegraded":"2024-04-24T08:10:20.288658+0000","last_fullsized":"2024-04-24T08:10:20.288658+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:37:52.267984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288783+0000","last_change":"2024-04-24T08:07:51.080403+0000","last_active":"2024-04-24T08:10:20.288783+0000","last_peered":"2024-04-24T08:10:20.288783+0000","last_clean":"2024-04-24T08:10:20.288783+0000","last_became_active":"2024-04-24T08:07:51.079158+0000","last_became_peered":"2024-04-24T08:07:51.079158+0000","last_unstale":"2024-04-24T08:10:20.288783+0000","last_undegraded":"2024-04-24T08:10:20.288783+0000","last_fullsized":"2024-04-24T08:10:20.288783+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:49:59.491850+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.504255+0000","last_change":"2024-04-24T08:07:52.066963+0000","last_active":"2024-04-24T08:10:20.504255+0000","last_peered":"2024-04-24T08:10:20.504255+0000","last_clean":"2024-04-24T08:10:20.504255+0000","last_became_active":"2024-04-24T08:07:52.065475+0000","last_became_peered":"2024-04-24T08:07:52.065475+0000","last_unstale":"2024-04-24T08:10:20.504255+0000","last_undegraded":"2024-04-24T08:10:20.504255+0000","last_fullsized":"2024-04-24T08:10:20.504255+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:28:34.220200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128324+0000","last_change":"2024-04-24T08:07:51.042772+0000","last_active":"2024-04-24T08:10:20.128324+0000","last_peered":"2024-04-24T08:10:20.128324+0000","last_clean":"2024-04-24T08:10:20.128324+0000","last_became_active":"2024-04-24T08:07:51.042128+0000","last_became_peered":"2024-04-24T08:07:51.042128+0000","last_unstale":"2024-04-24T08:10:20.128324+0000","last_undegraded":"2024-04-24T08:10:20.128324+0000","last_fullsized":"2024-04-24T08:10:20.128324+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:10:29.207991+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498210+0000","last_change":"2024-04-24T08:07:52.062411+0000","last_active":"2024-04-24T08:10:20.498210+0000","last_peered":"2024-04-24T08:10:20.498210+0000","last_clean":"2024-04-24T08:10:20.498210+0000","last_became_active":"2024-04-24T08:07:52.062037+0000","last_became_peered":"2024-04-24T08:07:52.062037+0000","last_unstale":"2024-04-24T08:10:20.498210+0000","last_undegraded":"2024-04-24T08:10:20.498210+0000","last_fullsized":"2024-04-24T08:10:20.498210+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:16:17.463874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128356+0000","last_change":"2024-04-24T08:07:51.046046+0000","last_active":"2024-04-24T08:10:20.128356+0000","last_peered":"2024-04-24T08:10:20.128356+0000","last_clean":"2024-04-24T08:10:20.128356+0000","last_became_active":"2024-04-24T08:07:51.045633+0000","last_became_peered":"2024-04-24T08:07:51.045633+0000","last_unstale":"2024-04-24T08:10:20.128356+0000","last_undegraded":"2024-04-24T08:10:20.128356+0000","last_fullsized":"2024-04-24T08:10:20.128356+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:18:20.529234+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297863+0000","last_change":"2024-04-24T08:07:52.061167+0000","last_active":"2024-04-24T08:10:20.297863+0000","last_peered":"2024-04-24T08:10:20.297863+0000","last_clean":"2024-04-24T08:10:20.297863+0000","last_became_active":"2024-04-24T08:07:52.060841+0000","last_became_peered":"2024-04-24T08:07:52.060841+0000","last_unstale":"2024-04-24T08:10:20.297863+0000","last_undegraded":"2024-04-24T08:10:20.297863+0000","last_fullsized":"2024-04-24T08:10:20.297863+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:04:27.048760+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"91'9","reported_seq":184,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.502848+0000","last_change":"2024-04-24T08:07:51.044274+0000","last_active":"2024-04-24T08:10:20.502848+0000","last_peered":"2024-04-24T08:10:20.502848+0000","last_clean":"2024-04-24T08:10:20.502848+0000","last_became_active":"2024-04-24T08:07:51.043938+0000","last_became_peered":"2024-04-24T08:07:51.043938+0000","last_unstale":"2024-04-24T08:10:20.502848+0000","last_undegraded":"2024-04-24T08:10:20.502848+0000","last_fullsized":"2024-04-24T08:10:20.502848+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:05:38.349981+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1855,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":5,"num_write":6,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128148+0000","last_change":"2024-04-24T08:07:52.055777+0000","last_active":"2024-04-24T08:10:20.128148+0000","last_peered":"2024-04-24T08:10:20.128148+0000","last_clean":"2024-04-24T08:10:20.128148+0000","last_became_active":"2024-04-24T08:07:52.055415+0000","last_became_peered":"2024-04-24T08:07:52.055415+0000","last_unstale":"2024-04-24T08:10:20.128148+0000","last_undegraded":"2024-04-24T08:10:20.128148+0000","last_fullsized":"2024-04-24T08:10:20.128148+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:18:31.175390+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.301075+0000","last_change":"2024-04-24T08:07:51.072513+0000","last_active":"2024-04-24T08:10:20.301075+0000","last_peered":"2024-04-24T08:10:20.301075+0000","last_clean":"2024-04-24T08:10:20.301075+0000","last_became_active":"2024-04-24T08:07:51.072249+0000","last_became_peered":"2024-04-24T08:07:51.072249+0000","last_unstale":"2024-04-24T08:10:20.301075+0000","last_undegraded":"2024-04-24T08:10:20.301075+0000","last_fullsized":"2024-04-24T08:10:20.301075+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:45:07.112617+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297787+0000","last_change":"2024-04-24T08:08:51.587130+0000","last_active":"2024-04-24T08:10:20.297787+0000","last_peered":"2024-04-24T08:10:20.297787+0000","last_clean":"2024-04-24T08:10:20.297787+0000","last_became_active":"2024-04-24T08:08:51.586645+0000","last_became_peered":"2024-04-24T08:08:51.586645+0000","last_unstale":"2024-04-24T08:10:20.297787+0000","last_undegraded":"2024-04-24T08:10:20.297787+0000","last_fullsized":"2024-04-24T08:10:20.297787+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:24:14.677548+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,7,3],"acting":[5,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.443164+0000","last_change":"2024-04-24T08:07:51.070901+0000","last_active":"2024-04-24T08:10:20.443164+0000","last_peered":"2024-04-24T08:10:20.443164+0000","last_clean":"2024-04-24T08:10:20.443164+0000","last_became_active":"2024-04-24T08:07:51.067178+0000","last_became_peered":"2024-04-24T08:07:51.067178+0000","last_unstale":"2024-04-24T08:10:20.443164+0000","last_undegraded":"2024-04-24T08:10:20.443164+0000","last_fullsized":"2024-04-24T08:10:20.443164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:24:26.807786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288202+0000","last_change":"2024-04-24T08:07:52.079493+0000","last_active":"2024-04-24T08:10:20.288202+0000","last_peered":"2024-04-24T08:10:20.288202+0000","last_clean":"2024-04-24T08:10:20.288202+0000","last_became_active":"2024-04-24T08:07:52.077266+0000","last_became_peered":"2024-04-24T08:07:52.077266+0000","last_unstale":"2024-04-24T08:10:20.288202+0000","last_undegraded":"2024-04-24T08:10:20.288202+0000","last_fullsized":"2024-04-24T08:10:20.288202+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:40:45.660146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297546+0000","last_change":"2024-04-24T08:07:51.045963+0000","last_active":"2024-04-24T08:10:20.297546+0000","last_peered":"2024-04-24T08:10:20.297546+0000","last_clean":"2024-04-24T08:10:20.297546+0000","last_became_active":"2024-04-24T08:07:51.045641+0000","last_became_peered":"2024-04-24T08:07:51.045641+0000","last_unstale":"2024-04-24T08:10:20.297546+0000","last_undegraded":"2024-04-24T08:10:20.297546+0000","last_fullsized":"2024-04-24T08:10:20.297546+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:07:59.646155+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128896+0000","last_change":"2024-04-24T08:07:52.057280+0000","last_active":"2024-04-24T08:10:20.128896+0000","last_peered":"2024-04-24T08:10:20.128896+0000","last_clean":"2024-04-24T08:10:20.128896+0000","last_became_active":"2024-04-24T08:07:52.054944+0000","last_became_peered":"2024-04-24T08:07:52.054944+0000","last_unstale":"2024-04-24T08:10:20.128896+0000","last_undegraded":"2024-04-24T08:10:20.128896+0000","last_fullsized":"2024-04-24T08:10:20.128896+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:22:45.911472+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"90'5","reported_seq":180,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:24.378138+0000","last_change":"2024-04-24T08:07:51.048898+0000","last_active":"2024-04-24T08:10:24.378138+0000","last_peered":"2024-04-24T08:10:24.378138+0000","last_clean":"2024-04-24T08:10:24.378138+0000","last_became_active":"2024-04-24T08:07:51.048585+0000","last_became_peered":"2024-04-24T08:07:51.048585+0000","last_unstale":"2024-04-24T08:10:24.378138+0000","last_undegraded":"2024-04-24T08:10:24.378138+0000","last_fullsized":"2024-04-24T08:10:24.378138+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:40:30.413541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":149,"num_read_kb":149,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498865+0000","last_change":"2024-04-24T08:07:52.048093+0000","last_active":"2024-04-24T08:10:20.498865+0000","last_peered":"2024-04-24T08:10:20.498865+0000","last_clean":"2024-04-24T08:10:20.498865+0000","last_became_active":"2024-04-24T08:07:52.047727+0000","last_became_peered":"2024-04-24T08:07:52.047727+0000","last_unstale":"2024-04-24T08:10:20.498865+0000","last_undegraded":"2024-04-24T08:10:20.498865+0000","last_fullsized":"2024-04-24T08:10:20.498865+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:57:26.248355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259256+0000","last_change":"2024-04-24T08:07:51.047810+0000","last_active":"2024-04-24T08:10:20.259256+0000","last_peered":"2024-04-24T08:10:20.259256+0000","last_clean":"2024-04-24T08:10:20.259256+0000","last_became_active":"2024-04-24T08:07:51.047501+0000","last_became_peered":"2024-04-24T08:07:51.047501+0000","last_unstale":"2024-04-24T08:10:20.259256+0000","last_undegraded":"2024-04-24T08:10:20.259256+0000","last_fullsized":"2024-04-24T08:10:20.259256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T20:02:27.585202+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.504166+0000","last_change":"2024-04-24T08:07:52.063782+0000","last_active":"2024-04-24T08:10:20.504166+0000","last_peered":"2024-04-24T08:10:20.504166+0000","last_clean":"2024-04-24T08:10:20.504166+0000","last_became_active":"2024-04-24T08:07:52.063056+0000","last_became_peered":"2024-04-24T08:07:52.063056+0000","last_unstale":"2024-04-24T08:10:20.504166+0000","last_undegraded":"2024-04-24T08:10:20.504166+0000","last_fullsized":"2024-04-24T08:10:20.504166+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:41:30.966787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.298089+0000","last_change":"2024-04-24T08:07:51.076139+0000","last_active":"2024-04-24T08:10:20.298089+0000","last_peered":"2024-04-24T08:10:20.298089+0000","last_clean":"2024-04-24T08:10:20.298089+0000","last_became_active":"2024-04-24T08:07:51.075842+0000","last_became_peered":"2024-04-24T08:07:51.075842+0000","last_unstale":"2024-04-24T08:10:20.298089+0000","last_undegraded":"2024-04-24T08:10:20.298089+0000","last_fullsized":"2024-04-24T08:10:20.298089+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:46:56.741726+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498434+0000","last_change":"2024-04-24T08:08:51.250881+0000","last_active":"2024-04-24T08:10:20.498434+0000","last_peered":"2024-04-24T08:10:20.498434+0000","last_clean":"2024-04-24T08:10:20.498434+0000","last_became_active":"2024-04-24T08:08:51.250519+0000","last_became_peered":"2024-04-24T08:08:51.250519+0000","last_unstale":"2024-04-24T08:10:20.498434+0000","last_undegraded":"2024-04-24T08:10:20.498434+0000","last_fullsized":"2024-04-24T08:10:20.498434+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:23:04.805687+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"91'36","reported_seq":371,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:25.379830+0000","last_change":"2024-04-24T08:07:51.048079+0000","last_active":"2024-04-24T08:10:25.379830+0000","last_peered":"2024-04-24T08:10:25.379830+0000","last_clean":"2024-04-24T08:10:25.379830+0000","last_became_active":"2024-04-24T08:07:51.047584+0000","last_became_peered":"2024-04-24T08:07:51.047584+0000","last_unstale":"2024-04-24T08:10:25.379830+0000","last_undegraded":"2024-04-24T08:10:25.379830+0000","last_fullsized":"2024-04-24T08:10:25.379830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:09:05.554984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":67055,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":309,"num_read_kb":219,"num_write":37,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128091+0000","last_change":"2024-04-24T08:07:52.055894+0000","last_active":"2024-04-24T08:10:20.128091+0000","last_peered":"2024-04-24T08:10:20.128091+0000","last_clean":"2024-04-24T08:10:20.128091+0000","last_became_active":"2024-04-24T08:07:52.055548+0000","last_became_peered":"2024-04-24T08:07:52.055548+0000","last_unstale":"2024-04-24T08:10:20.128091+0000","last_undegraded":"2024-04-24T08:10:20.128091+0000","last_fullsized":"2024-04-24T08:10:20.128091+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:33:22.609865+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'6","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288830+0000","last_change":"2024-04-24T08:08:51.188459+0000","last_active":"2024-04-24T08:10:20.288830+0000","last_peered":"2024-04-24T08:10:20.288830+0000","last_clean":"2024-04-24T08:10:20.288830+0000","last_became_active":"2024-04-24T08:08:51.180387+0000","last_became_peered":"2024-04-24T08:08:51.180387+0000","last_unstale":"2024-04-24T08:10:20.288830+0000","last_undegraded":"2024-04-24T08:10:20.288830+0000","last_fullsized":"2024-04-24T08:10:20.288830+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:34:56.592978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":34,"num_keys_recovered":10,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,10,3],"acting":[11,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.443550+0000","last_change":"2024-04-24T08:07:52.069950+0000","last_active":"2024-04-24T08:10:20.443550+0000","last_peered":"2024-04-24T08:10:20.443550+0000","last_clean":"2024-04-24T08:10:20.443550+0000","last_became_active":"2024-04-24T08:07:52.069515+0000","last_became_peered":"2024-04-24T08:07:52.069515+0000","last_unstale":"2024-04-24T08:10:20.443550+0000","last_undegraded":"2024-04-24T08:10:20.443550+0000","last_fullsized":"2024-04-24T08:10:20.443550+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:52:27.530541+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297642+0000","last_change":"2024-04-24T08:07:51.067062+0000","last_active":"2024-04-24T08:10:20.297642+0000","last_peered":"2024-04-24T08:10:20.297642+0000","last_clean":"2024-04-24T08:10:20.297642+0000","last_became_active":"2024-04-24T08:07:51.066712+0000","last_became_peered":"2024-04-24T08:07:51.066712+0000","last_unstale":"2024-04-24T08:10:20.297642+0000","last_undegraded":"2024-04-24T08:10:20.297642+0000","last_fullsized":"2024-04-24T08:10:20.297642+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:11:52.974470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259346+0000","last_change":"2024-04-24T08:07:52.064521+0000","last_active":"2024-04-24T08:10:20.259346+0000","last_peered":"2024-04-24T08:10:20.259346+0000","last_clean":"2024-04-24T08:10:20.259346+0000","last_became_active":"2024-04-24T08:07:52.064184+0000","last_became_peered":"2024-04-24T08:07:52.064184+0000","last_unstale":"2024-04-24T08:10:20.259346+0000","last_undegraded":"2024-04-24T08:10:20.259346+0000","last_fullsized":"2024-04-24T08:10:20.259346+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:00:27.294441+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"91'7","reported_seq":187,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:24.451262+0000","last_change":"2024-04-24T08:07:51.043577+0000","last_active":"2024-04-24T08:10:24.451262+0000","last_peered":"2024-04-24T08:10:24.451262+0000","last_clean":"2024-04-24T08:10:24.451262+0000","last_became_active":"2024-04-24T08:07:51.043215+0000","last_became_peered":"2024-04-24T08:07:51.043215+0000","last_unstale":"2024-04-24T08:10:24.451262+0000","last_undegraded":"2024-04-24T08:10:24.451262+0000","last_fullsized":"2024-04-24T08:10:24.451262+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:25:09.979236+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1902,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":155,"num_read_kb":6,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.258879+0000","last_change":"2024-04-24T08:07:52.057633+0000","last_active":"2024-04-24T08:10:20.258879+0000","last_peered":"2024-04-24T08:10:20.258879+0000","last_clean":"2024-04-24T08:10:20.258879+0000","last_became_active":"2024-04-24T08:07:52.057309+0000","last_became_peered":"2024-04-24T08:07:52.057309+0000","last_unstale":"2024-04-24T08:10:20.258879+0000","last_undegraded":"2024-04-24T08:10:20.258879+0000","last_fullsized":"2024-04-24T08:10:20.258879+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:49:56.045030+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'15","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.127851+0000","last_change":"2024-04-24T08:07:51.058957+0000","last_active":"2024-04-24T08:10:20.127851+0000","last_peered":"2024-04-24T08:10:20.127851+0000","last_clean":"2024-04-24T08:10:20.127851+0000","last_became_active":"2024-04-24T08:07:51.058654+0000","last_became_peered":"2024-04-24T08:07:51.058654+0000","last_unstale":"2024-04-24T08:10:20.127851+0000","last_undegraded":"2024-04-24T08:10:20.127851+0000","last_fullsized":"2024-04-24T08:10:20.127851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:09:13.354590+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297541+0000","last_change":"2024-04-24T08:07:52.051052+0000","last_active":"2024-04-24T08:10:20.297541+0000","last_peered":"2024-04-24T08:10:20.297541+0000","last_clean":"2024-04-24T08:10:20.297541+0000","last_became_active":"2024-04-24T08:07:52.050685+0000","last_became_peered":"2024-04-24T08:07:52.050685+0000","last_unstale":"2024-04-24T08:10:20.297541+0000","last_undegraded":"2024-04-24T08:10:20.297541+0000","last_fullsized":"2024-04-24T08:10:20.297541+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:53:34.491653+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"91'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128105+0000","last_change":"2024-04-24T08:07:51.041088+0000","last_active":"2024-04-24T08:10:20.128105+0000","last_peered":"2024-04-24T08:10:20.128105+0000","last_clean":"2024-04-24T08:10:20.128105+0000","last_became_active":"2024-04-24T08:07:51.040752+0000","last_became_peered":"2024-04-24T08:07:51.040752+0000","last_unstale":"2024-04-24T08:10:20.128105+0000","last_undegraded":"2024-04-24T08:10:20.128105+0000","last_fullsized":"2024-04-24T08:10:20.128105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:48:23.405705+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.289018+0000","last_change":"2024-04-24T08:08:51.173331+0000","last_active":"2024-04-24T08:10:20.289018+0000","last_peered":"2024-04-24T08:10:20.289018+0000","last_clean":"2024-04-24T08:10:20.289018+0000","last_became_active":"2024-04-24T08:08:51.172912+0000","last_became_peered":"2024-04-24T08:08:51.172912+0000","last_unstale":"2024-04-24T08:10:20.289018+0000","last_undegraded":"2024-04-24T08:10:20.289018+0000","last_fullsized":"2024-04-24T08:10:20.289018+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:54:35.241315+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,8],"acting":[11,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.134622+0000","last_change":"2024-04-24T08:07:51.075583+0000","last_active":"2024-04-24T08:10:20.134622+0000","last_peered":"2024-04-24T08:10:20.134622+0000","last_clean":"2024-04-24T08:10:20.134622+0000","last_became_active":"2024-04-24T08:07:51.075111+0000","last_became_peered":"2024-04-24T08:07:51.075111+0000","last_unstale":"2024-04-24T08:10:20.134622+0000","last_undegraded":"2024-04-24T08:10:20.134622+0000","last_fullsized":"2024-04-24T08:10:20.134622+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:56:00.820057+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.301341+0000","last_change":"2024-04-24T08:07:52.063232+0000","last_active":"2024-04-24T08:10:20.301341+0000","last_peered":"2024-04-24T08:10:20.301341+0000","last_clean":"2024-04-24T08:10:20.301341+0000","last_became_active":"2024-04-24T08:07:52.062868+0000","last_became_peered":"2024-04-24T08:07:52.062868+0000","last_unstale":"2024-04-24T08:10:20.301341+0000","last_undegraded":"2024-04-24T08:10:20.301341+0000","last_fullsized":"2024-04-24T08:10:20.301341+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:00:40.333389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.258895+0000","last_change":"2024-04-24T08:07:51.070647+0000","last_active":"2024-04-24T08:10:20.258895+0000","last_peered":"2024-04-24T08:10:20.258895+0000","last_clean":"2024-04-24T08:10:20.258895+0000","last_became_active":"2024-04-24T08:07:51.070298+0000","last_became_peered":"2024-04-24T08:07:51.070298+0000","last_unstale":"2024-04-24T08:10:20.258895+0000","last_undegraded":"2024-04-24T08:10:20.258895+0000","last_fullsized":"2024-04-24T08:10:20.258895+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:56:41.706796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.301283+0000","last_change":"2024-04-24T08:07:52.073417+0000","last_active":"2024-04-24T08:10:20.301283+0000","last_peered":"2024-04-24T08:10:20.301283+0000","last_clean":"2024-04-24T08:10:20.301283+0000","last_became_active":"2024-04-24T08:07:52.073051+0000","last_became_peered":"2024-04-24T08:07:52.073051+0000","last_unstale":"2024-04-24T08:10:20.301283+0000","last_undegraded":"2024-04-24T08:10:20.301283+0000","last_fullsized":"2024-04-24T08:10:20.301283+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:54:58.338279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128064+0000","last_change":"2024-04-24T08:08:51.172525+0000","last_active":"2024-04-24T08:10:20.128064+0000","last_peered":"2024-04-24T08:10:20.128064+0000","last_clean":"2024-04-24T08:10:20.128064+0000","last_became_active":"2024-04-24T08:08:51.171924+0000","last_became_peered":"2024-04-24T08:08:51.171924+0000","last_unstale":"2024-04-24T08:10:20.128064+0000","last_undegraded":"2024-04-24T08:10:20.128064+0000","last_fullsized":"2024-04-24T08:10:20.128064+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:25:42.067150+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"91'6","reported_seq":176,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.443285+0000","last_change":"2024-04-24T08:07:51.076937+0000","last_active":"2024-04-24T08:10:20.443285+0000","last_peered":"2024-04-24T08:10:20.443285+0000","last_clean":"2024-04-24T08:10:20.443285+0000","last_became_active":"2024-04-24T08:07:51.076621+0000","last_became_peered":"2024-04-24T08:07:51.076621+0000","last_unstale":"2024-04-24T08:10:20.443285+0000","last_undegraded":"2024-04-24T08:10:20.443285+0000","last_fullsized":"2024-04-24T08:10:20.443285+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:09.966994+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":144,"num_read_kb":144,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288284+0000","last_change":"2024-04-24T08:07:52.054150+0000","last_active":"2024-04-24T08:10:20.288284+0000","last_peered":"2024-04-24T08:10:20.288284+0000","last_clean":"2024-04-24T08:10:20.288284+0000","last_became_active":"2024-04-24T08:07:52.053817+0000","last_became_peered":"2024-04-24T08:07:52.053817+0000","last_unstale":"2024-04-24T08:10:20.288284+0000","last_undegraded":"2024-04-24T08:10:20.288284+0000","last_fullsized":"2024-04-24T08:10:20.288284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:46:16.570004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128899+0000","last_change":"2024-04-24T08:07:51.045431+0000","last_active":"2024-04-24T08:10:20.128899+0000","last_peered":"2024-04-24T08:10:20.128899+0000","last_clean":"2024-04-24T08:10:20.128899+0000","last_became_active":"2024-04-24T08:07:51.045112+0000","last_became_peered":"2024-04-24T08:07:51.045112+0000","last_unstale":"2024-04-24T08:10:20.128899+0000","last_undegraded":"2024-04-24T08:10:20.128899+0000","last_fullsized":"2024-04-24T08:10:20.128899+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:52:05.342934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":358,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.258842+0000","last_change":"2024-04-24T08:06:31.583310+0000","last_active":"2024-04-24T08:10:20.258842+0000","last_peered":"2024-04-24T08:10:20.258842+0000","last_clean":"2024-04-24T08:10:20.258842+0000","last_became_active":"2024-04-24T08:06:31.582735+0000","last_became_peered":"2024-04-24T08:06:31.582735+0000","last_unstale":"2024-04-24T08:10:20.258842+0000","last_undegraded":"2024-04-24T08:10:20.258842+0000","last_fullsized":"2024-04-24T08:10:20.258842+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:02:34.411025+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:02:34.411025+0000","last_clean_scrub_stamp":"2024-04-24T08:02:34.411025+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:00:48.816092+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.442781+0000","last_change":"2024-04-24T08:07:52.063961+0000","last_active":"2024-04-24T08:10:20.442781+0000","last_peered":"2024-04-24T08:10:20.442781+0000","last_clean":"2024-04-24T08:10:20.442781+0000","last_became_active":"2024-04-24T08:07:52.063608+0000","last_became_peered":"2024-04-24T08:07:52.063608+0000","last_unstale":"2024-04-24T08:10:20.442781+0000","last_undegraded":"2024-04-24T08:10:20.442781+0000","last_fullsized":"2024-04-24T08:10:20.442781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:40:50.233349+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259215+0000","last_change":"2024-04-24T08:07:51.045626+0000","last_active":"2024-04-24T08:10:20.259215+0000","last_peered":"2024-04-24T08:10:20.259215+0000","last_clean":"2024-04-24T08:10:20.259215+0000","last_became_active":"2024-04-24T08:07:51.045293+0000","last_became_peered":"2024-04-24T08:07:51.045293+0000","last_unstale":"2024-04-24T08:10:20.259215+0000","last_undegraded":"2024-04-24T08:10:20.259215+0000","last_fullsized":"2024-04-24T08:10:20.259215+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:13:04.601331+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.128055+0000","last_change":"2024-04-24T08:07:51.079228+0000","last_active":"2024-04-24T08:10:20.128055+0000","last_peered":"2024-04-24T08:10:20.128055+0000","last_clean":"2024-04-24T08:10:20.128055+0000","last_became_active":"2024-04-24T08:07:51.078932+0000","last_became_peered":"2024-04-24T08:07:51.078932+0000","last_unstale":"2024-04-24T08:10:20.128055+0000","last_undegraded":"2024-04-24T08:10:20.128055+0000","last_fullsized":"2024-04-24T08:10:20.128055+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:58:32.024133+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.442826+0000","last_change":"2024-04-24T08:07:52.065943+0000","last_active":"2024-04-24T08:10:20.442826+0000","last_peered":"2024-04-24T08:10:20.442826+0000","last_clean":"2024-04-24T08:10:20.442826+0000","last_became_active":"2024-04-24T08:07:52.062810+0000","last_became_peered":"2024-04-24T08:07:52.062810+0000","last_unstale":"2024-04-24T08:10:20.442826+0000","last_undegraded":"2024-04-24T08:10:20.442826+0000","last_fullsized":"2024-04-24T08:10:20.442826+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:40:08.803652+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"91'45","reported_seq":106,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.444289+0000","last_change":"2024-04-24T08:08:51.193009+0000","last_active":"2024-04-24T08:10:20.444289+0000","last_peered":"2024-04-24T08:10:20.444289+0000","last_clean":"2024-04-24T08:10:20.444289+0000","last_became_active":"2024-04-24T08:08:51.180025+0000","last_became_peered":"2024-04-24T08:08:51.180025+0000","last_unstale":"2024-04-24T08:10:20.444289+0000","last_undegraded":"2024-04-24T08:10:20.444289+0000","last_fullsized":"2024-04-24T08:10:20.444289+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":45,"log_dups_size":0,"ondisk_log_size":45,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:31:19.397470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":393216,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":82,"num_read_kb":70,"num_write":81,"num_write_kb":837,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":7,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,11,3],"acting":[9,11,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288903+0000","last_change":"2024-04-24T08:07:52.058764+0000","last_active":"2024-04-24T08:10:20.288903+0000","last_peered":"2024-04-24T08:10:20.288903+0000","last_clean":"2024-04-24T08:10:20.288903+0000","last_became_active":"2024-04-24T08:07:52.058336+0000","last_became_peered":"2024-04-24T08:07:52.058336+0000","last_unstale":"2024-04-24T08:10:20.288903+0000","last_undegraded":"2024-04-24T08:10:20.288903+0000","last_fullsized":"2024-04-24T08:10:20.288903+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:05:37.245521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.129060+0000","last_change":"2024-04-24T08:07:51.067154+0000","last_active":"2024-04-24T08:10:20.129060+0000","last_peered":"2024-04-24T08:10:20.129060+0000","last_clean":"2024-04-24T08:10:20.129060+0000","last_became_active":"2024-04-24T08:07:51.066769+0000","last_became_peered":"2024-04-24T08:07:51.066769+0000","last_unstale":"2024-04-24T08:10:20.129060+0000","last_undegraded":"2024-04-24T08:10:20.129060+0000","last_fullsized":"2024-04-24T08:10:20.129060+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:39:25.908189+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.504066+0000","last_change":"2024-04-24T08:07:52.072134+0000","last_active":"2024-04-24T08:10:20.504066+0000","last_peered":"2024-04-24T08:10:20.504066+0000","last_clean":"2024-04-24T08:10:20.504066+0000","last_became_active":"2024-04-24T08:07:52.071703+0000","last_became_peered":"2024-04-24T08:07:52.071703+0000","last_unstale":"2024-04-24T08:10:20.504066+0000","last_undegraded":"2024-04-24T08:10:20.504066+0000","last_fullsized":"2024-04-24T08:10:20.504066+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:56:59.829988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498172+0000","last_change":"2024-04-24T08:07:51.059151+0000","last_active":"2024-04-24T08:10:20.498172+0000","last_peered":"2024-04-24T08:10:20.498172+0000","last_clean":"2024-04-24T08:10:20.498172+0000","last_became_active":"2024-04-24T08:07:51.058872+0000","last_became_peered":"2024-04-24T08:07:51.058872+0000","last_unstale":"2024-04-24T08:10:20.498172+0000","last_undegraded":"2024-04-24T08:10:20.498172+0000","last_fullsized":"2024-04-24T08:10:20.498172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:57:36.023595+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.127663+0000","last_change":"2024-04-24T08:07:52.056919+0000","last_active":"2024-04-24T08:10:20.127663+0000","last_peered":"2024-04-24T08:10:20.127663+0000","last_clean":"2024-04-24T08:10:20.127663+0000","last_became_active":"2024-04-24T08:07:52.056573+0000","last_became_peered":"2024-04-24T08:07:52.056573+0000","last_unstale":"2024-04-24T08:10:20.127663+0000","last_undegraded":"2024-04-24T08:10:20.127663+0000","last_fullsized":"2024-04-24T08:10:20.127663+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:58:17.939465+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288463+0000","last_change":"2024-04-24T08:07:51.056325+0000","last_active":"2024-04-24T08:10:20.288463+0000","last_peered":"2024-04-24T08:10:20.288463+0000","last_clean":"2024-04-24T08:10:20.288463+0000","last_became_active":"2024-04-24T08:07:51.055862+0000","last_became_peered":"2024-04-24T08:07:51.055862+0000","last_unstale":"2024-04-24T08:10:20.288463+0000","last_undegraded":"2024-04-24T08:10:20.288463+0000","last_fullsized":"2024-04-24T08:10:20.288463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T10:11:44.487530+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288781+0000","last_change":"2024-04-24T08:07:52.065570+0000","last_active":"2024-04-24T08:10:20.288781+0000","last_peered":"2024-04-24T08:10:20.288781+0000","last_clean":"2024-04-24T08:10:20.288781+0000","last_became_active":"2024-04-24T08:07:52.065196+0000","last_became_peered":"2024-04-24T08:07:52.065196+0000","last_unstale":"2024-04-24T08:10:20.288781+0000","last_undegraded":"2024-04-24T08:10:20.288781+0000","last_fullsized":"2024-04-24T08:10:20.288781+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:48:09.474159+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"91'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.497754+0000","last_change":"2024-04-24T08:07:51.067187+0000","last_active":"2024-04-24T08:10:20.497754+0000","last_peered":"2024-04-24T08:10:20.497754+0000","last_clean":"2024-04-24T08:10:20.497754+0000","last_became_active":"2024-04-24T08:07:51.066808+0000","last_became_peered":"2024-04-24T08:07:51.066808+0000","last_unstale":"2024-04-24T08:10:20.497754+0000","last_undegraded":"2024-04-24T08:10:20.497754+0000","last_fullsized":"2024-04-24T08:10:20.497754+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:32:11.852191+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288593+0000","last_change":"2024-04-24T08:07:52.078426+0000","last_active":"2024-04-24T08:10:20.288593+0000","last_peered":"2024-04-24T08:10:20.288593+0000","last_clean":"2024-04-24T08:10:20.288593+0000","last_became_active":"2024-04-24T08:07:52.078076+0000","last_became_peered":"2024-04-24T08:07:52.078076+0000","last_unstale":"2024-04-24T08:10:20.288593+0000","last_undegraded":"2024-04-24T08:10:20.288593+0000","last_fullsized":"2024-04-24T08:10:20.288593+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:55:29.885688+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.127956+0000","last_change":"2024-04-24T08:07:51.042601+0000","last_active":"2024-04-24T08:10:20.127956+0000","last_peered":"2024-04-24T08:10:20.127956+0000","last_clean":"2024-04-24T08:10:20.127956+0000","last_became_active":"2024-04-24T08:07:51.042252+0000","last_became_peered":"2024-04-24T08:07:51.042252+0000","last_unstale":"2024-04-24T08:10:20.127956+0000","last_undegraded":"2024-04-24T08:10:20.127956+0000","last_fullsized":"2024-04-24T08:10:20.127956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:11:44.640303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259102+0000","last_change":"2024-04-24T08:07:52.063142+0000","last_active":"2024-04-24T08:10:20.259102+0000","last_peered":"2024-04-24T08:10:20.259102+0000","last_clean":"2024-04-24T08:10:20.259102+0000","last_became_active":"2024-04-24T08:07:52.062263+0000","last_became_peered":"2024-04-24T08:07:52.062263+0000","last_unstale":"2024-04-24T08:10:20.259102+0000","last_undegraded":"2024-04-24T08:10:20.259102+0000","last_fullsized":"2024-04-24T08:10:20.259102+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:35:38.879380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'3","reported_seq":23,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.129102+0000","last_change":"2024-04-24T08:08:51.184795+0000","last_active":"2024-04-24T08:10:20.129102+0000","last_peered":"2024-04-24T08:10:20.129102+0000","last_clean":"2024-04-24T08:10:20.129102+0000","last_became_active":"2024-04-24T08:08:51.179560+0000","last_became_peered":"2024-04-24T08:08:51.179560+0000","last_unstale":"2024-04-24T08:10:20.129102+0000","last_undegraded":"2024-04-24T08:10:20.129102+0000","last_fullsized":"2024-04-24T08:10:20.129102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:31:23.795126+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":582,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,0],"acting":[4,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259332+0000","last_change":"2024-04-24T08:07:52.055958+0000","last_active":"2024-04-24T08:10:20.259332+0000","last_peered":"2024-04-24T08:10:20.259332+0000","last_clean":"2024-04-24T08:10:20.259332+0000","last_became_active":"2024-04-24T08:07:52.055598+0000","last_became_peered":"2024-04-24T08:07:52.055598+0000","last_unstale":"2024-04-24T08:10:20.259332+0000","last_undegraded":"2024-04-24T08:10:20.259332+0000","last_fullsized":"2024-04-24T08:10:20.259332+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:12:22.452576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"91'39","reported_seq":73,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.439516+0000","last_change":"2024-04-24T08:07:51.073234+0000","last_active":"2024-04-24T08:10:20.439516+0000","last_peered":"2024-04-24T08:10:20.439516+0000","last_clean":"2024-04-24T08:10:20.439516+0000","last_became_active":"2024-04-24T08:07:51.072887+0000","last_became_peered":"2024-04-24T08:07:51.072887+0000","last_unstale":"2024-04-24T08:10:20.439516+0000","last_undegraded":"2024-04-24T08:10:20.439516+0000","last_fullsized":"2024-04-24T08:10:20.439516+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":39,"log_dups_size":0,"ondisk_log_size":39,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:28:15.908859+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":131670,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":11,"num_read_kb":10,"num_write":49,"num_write_kb":475,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259151+0000","last_change":"2024-04-24T08:07:52.061188+0000","last_active":"2024-04-24T08:10:20.259151+0000","last_peered":"2024-04-24T08:10:20.259151+0000","last_clean":"2024-04-24T08:10:20.259151+0000","last_became_active":"2024-04-24T08:07:52.059616+0000","last_became_peered":"2024-04-24T08:07:52.059616+0000","last_unstale":"2024-04-24T08:10:20.259151+0000","last_undegraded":"2024-04-24T08:10:20.259151+0000","last_fullsized":"2024-04-24T08:10:20.259151+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:30:43.546091+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.442965+0000","last_change":"2024-04-24T08:07:51.078300+0000","last_active":"2024-04-24T08:10:20.442965+0000","last_peered":"2024-04-24T08:10:20.442965+0000","last_clean":"2024-04-24T08:10:20.442965+0000","last_became_active":"2024-04-24T08:07:51.076026+0000","last_became_peered":"2024-04-24T08:07:51.076026+0000","last_unstale":"2024-04-24T08:10:20.442965+0000","last_undegraded":"2024-04-24T08:10:20.442965+0000","last_fullsized":"2024-04-24T08:10:20.442965+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:50.009786+0000","last_clean_scrub_stamp":"2024-04-24T08:07:50.009786+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T19:45:31.494213+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.259123+0000","last_change":"2024-04-24T08:07:52.056148+0000","last_active":"2024-04-24T08:10:20.259123+0000","last_peered":"2024-04-24T08:10:20.259123+0000","last_clean":"2024-04-24T08:10:20.259123+0000","last_became_active":"2024-04-24T08:07:52.055768+0000","last_became_peered":"2024-04-24T08:07:52.055768+0000","last_unstale":"2024-04-24T08:10:20.259123+0000","last_undegraded":"2024-04-24T08:10:20.259123+0000","last_fullsized":"2024-04-24T08:10:20.259123+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:52:48.503479+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288880+0000","last_change":"2024-04-24T08:07:52.056611+0000","last_active":"2024-04-24T08:10:20.288880+0000","last_peered":"2024-04-24T08:10:20.288880+0000","last_clean":"2024-04-24T08:10:20.288880+0000","last_became_active":"2024-04-24T08:07:52.055788+0000","last_became_peered":"2024-04-24T08:07:52.055788+0000","last_unstale":"2024-04-24T08:10:20.288880+0000","last_undegraded":"2024-04-24T08:10:20.288880+0000","last_fullsized":"2024-04-24T08:10:20.288880+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:21:28.997588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.497715+0000","last_change":"2024-04-24T08:08:51.177574+0000","last_active":"2024-04-24T08:10:20.497715+0000","last_peered":"2024-04-24T08:10:20.497715+0000","last_clean":"2024-04-24T08:10:20.497715+0000","last_became_active":"2024-04-24T08:08:51.177145+0000","last_became_peered":"2024-04-24T08:08:51.177145+0000","last_unstale":"2024-04-24T08:10:20.497715+0000","last_undegraded":"2024-04-24T08:10:20.497715+0000","last_fullsized":"2024-04-24T08:10:20.497715+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T17:49:19.438233+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,4],"acting":[2,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.497647+0000","last_change":"2024-04-24T08:09:51.441560+0000","last_active":"2024-04-24T08:10:20.497647+0000","last_peered":"2024-04-24T08:10:20.497647+0000","last_clean":"2024-04-24T08:10:20.497647+0000","last_became_active":"2024-04-24T08:09:51.441192+0000","last_became_peered":"2024-04-24T08:09:51.441192+0000","last_unstale":"2024-04-24T08:10:20.497647+0000","last_undegraded":"2024-04-24T08:10:20.497647+0000","last_fullsized":"2024-04-24T08:10:20.497647+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T08:32:26.311164+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,9,0],"acting":[2,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.503483+0000","last_change":"2024-04-24T08:08:51.250206+0000","last_active":"2024-04-24T08:10:20.503483+0000","last_peered":"2024-04-24T08:10:20.503483+0000","last_clean":"2024-04-24T08:10:20.503483+0000","last_became_active":"2024-04-24T08:08:51.249677+0000","last_became_peered":"2024-04-24T08:08:51.249677+0000","last_unstale":"2024-04-24T08:10:20.503483+0000","last_undegraded":"2024-04-24T08:10:20.503483+0000","last_fullsized":"2024-04-24T08:10:20.503483+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T12:06:26.655987+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,8],"acting":[0,6,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.442885+0000","last_change":"2024-04-24T08:07:52.078604+0000","last_active":"2024-04-24T08:10:20.442885+0000","last_peered":"2024-04-24T08:10:20.442885+0000","last_clean":"2024-04-24T08:10:20.442885+0000","last_became_active":"2024-04-24T08:07:52.078122+0000","last_became_peered":"2024-04-24T08:07:52.078122+0000","last_unstale":"2024-04-24T08:10:20.442885+0000","last_undegraded":"2024-04-24T08:10:20.442885+0000","last_fullsized":"2024-04-24T08:10:20.442885+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T14:24:19.062821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.289182+0000","last_change":"2024-04-24T08:07:52.055986+0000","last_active":"2024-04-24T08:10:20.289182+0000","last_peered":"2024-04-24T08:10:20.289182+0000","last_clean":"2024-04-24T08:10:20.289182+0000","last_became_active":"2024-04-24T08:07:52.055609+0000","last_became_peered":"2024-04-24T08:07:52.055609+0000","last_unstale":"2024-04-24T08:10:20.289182+0000","last_undegraded":"2024-04-24T08:10:20.289182+0000","last_fullsized":"2024-04-24T08:10:20.289182+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T09:38:04.731204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498055+0000","last_change":"2024-04-24T08:07:52.055605+0000","last_active":"2024-04-24T08:10:20.498055+0000","last_peered":"2024-04-24T08:10:20.498055+0000","last_clean":"2024-04-24T08:10:20.498055+0000","last_became_active":"2024-04-24T08:07:52.055268+0000","last_became_peered":"2024-04-24T08:07:52.055268+0000","last_unstale":"2024-04-24T08:10:20.498055+0000","last_undegraded":"2024-04-24T08:10:20.498055+0000","last_fullsized":"2024-04-24T08:10:20.498055+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:50:29.197484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288890+0000","last_change":"2024-04-24T08:07:52.064590+0000","last_active":"2024-04-24T08:10:20.288890+0000","last_peered":"2024-04-24T08:10:20.288890+0000","last_clean":"2024-04-24T08:10:20.288890+0000","last_became_active":"2024-04-24T08:07:52.064113+0000","last_became_peered":"2024-04-24T08:07:52.064113+0000","last_unstale":"2024-04-24T08:10:20.288890+0000","last_undegraded":"2024-04-24T08:10:20.288890+0000","last_fullsized":"2024-04-24T08:10:20.288890+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:36:10.926145+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.297622+0000","last_change":"2024-04-24T08:08:51.250599+0000","last_active":"2024-04-24T08:10:20.297622+0000","last_peered":"2024-04-24T08:10:20.297622+0000","last_clean":"2024-04-24T08:10:20.297622+0000","last_became_active":"2024-04-24T08:08:51.250197+0000","last_became_peered":"2024-04-24T08:08:51.250197+0000","last_unstale":"2024-04-24T08:10:20.297622+0000","last_undegraded":"2024-04-24T08:10:20.297622+0000","last_fullsized":"2024-04-24T08:10:20.297622+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T15:31:30.729911+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6],"acting":[5,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288304+0000","last_change":"2024-04-24T08:07:52.068308+0000","last_active":"2024-04-24T08:10:20.288304+0000","last_peered":"2024-04-24T08:10:20.288304+0000","last_clean":"2024-04-24T08:10:20.288304+0000","last_became_active":"2024-04-24T08:07:52.067786+0000","last_became_peered":"2024-04-24T08:07:52.067786+0000","last_unstale":"2024-04-24T08:10:20.288304+0000","last_undegraded":"2024-04-24T08:10:20.288304+0000","last_fullsized":"2024-04-24T08:10:20.288304+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T18:26:09.349120+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.498381+0000","last_change":"2024-04-24T08:07:52.053089+0000","last_active":"2024-04-24T08:10:20.498381+0000","last_peered":"2024-04-24T08:10:20.498381+0000","last_clean":"2024-04-24T08:10:20.498381+0000","last_became_active":"2024-04-24T08:07:52.052753+0000","last_became_peered":"2024-04-24T08:07:52.052753+0000","last_unstale":"2024-04-24T08:10:20.498381+0000","last_undegraded":"2024-04-24T08:10:20.498381+0000","last_fullsized":"2024-04-24T08:10:20.498381+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T16:02:14.583922+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.442875+0000","last_change":"2024-04-24T08:07:52.076633+0000","last_active":"2024-04-24T08:10:20.442875+0000","last_peered":"2024-04-24T08:10:20.442875+0000","last_clean":"2024-04-24T08:10:20.442875+0000","last_became_active":"2024-04-24T08:07:52.076280+0000","last_became_peered":"2024-04-24T08:07:52.076280+0000","last_unstale":"2024-04-24T08:10:20.442875+0000","last_undegraded":"2024-04-24T08:10:20.442875+0000","last_fullsized":"2024-04-24T08:10:20.442875+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T11:29:59.823772+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-04-24T08:10:20.288825+0000","last_change":"2024-04-24T08:07:52.053017+0000","last_active":"2024-04-24T08:10:20.288825+0000","last_peered":"2024-04-24T08:10:20.288825+0000","last_clean":"2024-04-24T08:10:20.288825+0000","last_became_active":"2024-04-24T08:07:52.052325+0000","last_became_peered":"2024-04-24T08:07:52.052325+0000","last_unstale":"2024-04-24T08:10:20.288825+0000","last_undegraded":"2024-04-24T08:10:20.288825+0000","last_fullsized":"2024-04-24T08:10:20.288825+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-04-24T08:07:51.013734+0000","last_clean_scrub_stamp":"2024-04-24T08:07:51.013734+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-04-25T13:05:58.161360+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":147456,"data_stored":1771104,"data_compressed":108294,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":3},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":598028,"num_objects":62,"num_object_clones":0,"num_object_copies":186,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":62,"num_whiteouts":0,"num_read":1010,"num_read_kb":614,"num_write":296,"num_write_kb":1442,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":8,"num_bytes_recovered":616,"num_keys_recovered":17,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":569344,"data_stored":1787871,"data_compressed":75906,"data_compressed_allocated":98304,"data_compressed_original":1548288,"omap_allocated":0,"internal_metadata":0},"log_size":232,"ondisk_log_size":232,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351020,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36336,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708816,"statfs":{"total":95995035648,"available":95957827584,"internally_reserved":0,"allocated":1019904,"data_stored":881474,"data_compressed":33679,"data_compressed_allocated":45056,"data_compressed_original":434176,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612656,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28160,"kb_used_data":1144,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93716992,"statfs":{"total":95995035648,"available":95966199808,"internally_reserved":0,"allocated":1171456,"data_stored":1279674,"data_compressed":56787,"data_compressed_allocated":73728,"data_compressed_original":753664,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776181,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":881676,"data_compressed":33679,"data_compressed_allocated":45056,"data_compressed_original":434176,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037819,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32164,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712988,"statfs":{"total":95995035648,"available":95962099712,"internally_reserved":0,"allocated":1007616,"data_stored":488380,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332161,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32200,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712952,"statfs":{"total":95995035648,"available":95962062848,"internally_reserved":0,"allocated":1044480,"data_stored":1072279,"data_compressed":47714,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495688,"num_pgs":22,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":995328,"data_stored":487849,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574948,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1060864,"data_stored":551786,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738473,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":489120,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411423,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28072,"kb_used_data":1052,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717080,"statfs":{"total":95995035648,"available":95966289920,"internally_reserved":0,"allocated":1077248,"data_stored":553753,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182488,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1015808,"data_stored":488672,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920852,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1003520,"data_stored":490182,"data_compressed":14533,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757326,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32248,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712904,"statfs":{"total":95995035648,"available":95962013696,"internally_reserved":0,"allocated":1089536,"data_stored":1212045,"data_compressed":56686,"data_compressed_allocated":73728,"data_compressed_original":753664,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36098,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":64144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":66111,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":61440,"data_stored":525100,"data_compressed":25302,"data_compressed_allocated":32768,"data_compressed_original":516096,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2540,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":134309,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":3948,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":394034,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":131072,"data_stored":201547,"data_compressed":6156,"data_compressed_allocated":8192,"data_compressed_original":122880,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":393832,"data_compressed":19146,"data_compressed_allocated":24576,"data_compressed_original":393216,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-04-24T08:10:26.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:10:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.111+0000 7f2b20c00640 1 -- 172.21.15.28:0/1059248117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b1c1040a0 msgr2=0x7f2b1c1044c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.111+0000 7f2b20c00640 1 --2- 172.21.15.28:0/1059248117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b1c1040a0 0x7f2b1c1044c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c009860 tx=0x7f2b0c02a340 comp rx=0 tx=0).stop 2024-04-24T08:10:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.111+0000 7f2b20c00640 1 -- 172.21.15.28:0/1059248117 shutdown_connections 2024-04-24T08:10:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.111+0000 7f2b20c00640 1 --2- 172.21.15.28:0/1059248117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b1c105290 0x7f2b1c105710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.111+0000 7f2b20c00640 1 --2- 172.21.15.28:0/1059248117 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b1c105c50 0x7f2b1c10c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.111+0000 7f2b20c00640 1 --2- 172.21.15.28:0/1059248117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b1c1040a0 0x7f2b1c1044c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.111+0000 7f2b20c00640 1 -- 172.21.15.28:0/1059248117 >> 172.21.15.28:0/1059248117 conn(0x7f2b1c0ff610 msgr2=0x7f2b1c101a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.111+0000 7f2b20c00640 1 -- 172.21.15.28:0/1059248117 shutdown_connections 2024-04-24T08:10:27.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.111+0000 7f2b20c00640 1 -- 172.21.15.28:0/1059248117 wait complete. 2024-04-24T08:10:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.112+0000 7f2b20c00640 1 Processor -- start 2024-04-24T08:10:27.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.113+0000 7f2b20c00640 1 -- start start 2024-04-24T08:10:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.113+0000 7f2b20c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b1c1040a0 0x7f2b1c19a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.113+0000 7f2b20c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b1c105290 0x7f2b1c19a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.113+0000 7f2b20c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b1c105c50 0x7f2b1c1a1940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.113+0000 7f2b20c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c10fcf0 con 0x7f2b1c105290 2024-04-24T08:10:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.113+0000 7f2b20c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c10fb70 con 0x7f2b1c1040a0 2024-04-24T08:10:27.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.113+0000 7f2b20c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2b1c10fe70 con 0x7f2b1c105c50 2024-04-24T08:10:27.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.114+0000 7f2b1b400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b1c1040a0 0x7f2b1c19a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.114+0000 7f2b1aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b1c105290 0x7f2b1c19a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.114+0000 7f2b1b400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b1c1040a0 0x7f2b1c19a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:38162/0 (socket says 172.21.15.28:38162) 2024-04-24T08:10:27.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.114+0000 7f2b1b400640 1 -- 172.21.15.28:0/2617904138 learned_addr learned my addr 172.21.15.28:0/2617904138 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:27.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.114+0000 7f2b1be00640 1 --2- 172.21.15.28:0/2617904138 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b1c105c50 0x7f2b1c1a1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.114+0000 7f2b1b400640 1 -- 172.21.15.28:0/2617904138 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b1c105c50 msgr2=0x7f2b1c1a1940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.114+0000 7f2b1b400640 1 --2- 172.21.15.28:0/2617904138 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b1c105c50 0x7f2b1c1a1940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.114+0000 7f2b1b400640 1 -- 172.21.15.28:0/2617904138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b1c105290 msgr2=0x7f2b1c19a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.114+0000 7f2b1b400640 1 --2- 172.21.15.28:0/2617904138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b1c105290 0x7f2b1c19a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.114+0000 7f2b1b400640 1 -- 172.21.15.28:0/2617904138 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b1c1a1fb0 con 0x7f2b1c1040a0 2024-04-24T08:10:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.114+0000 7f2b1b400640 1 --2- 172.21.15.28:0/2617904138 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b1c1040a0 0x7f2b1c19a390 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c035040 tx=0x7f2b0c033d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.115+0000 7f2b13e00640 1 -- 172.21.15.28:0/2617904138 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c037040 con 0x7f2b1c1040a0 2024-04-24T08:10:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.115+0000 7f2b13e00640 1 -- 172.21.15.28:0/2617904138 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b0c030850 con 0x7f2b1c1040a0 2024-04-24T08:10:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.115+0000 7f2b13e00640 1 -- 172.21.15.28:0/2617904138 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b0c004920 con 0x7f2b1c1040a0 2024-04-24T08:10:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.115+0000 7f2b20c00640 1 -- 172.21.15.28:0/2617904138 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b1c1a2240 con 0x7f2b1c1040a0 2024-04-24T08:10:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.115+0000 7f2b20c00640 1 -- 172.21.15.28:0/2617904138 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b1c1a26e0 con 0x7f2b1c1040a0 2024-04-24T08:10:27.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.116+0000 7f2b20c00640 1 -- 172.21.15.28:0/2617904138 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2b1c10b440 con 0x7f2b1c1040a0 2024-04-24T08:10:27.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.119+0000 7f2b13e00640 1 -- 172.21.15.28:0/2617904138 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7f2b0c004050 con 0x7f2b1c1040a0 2024-04-24T08:10:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.120+0000 7f2b13e00640 1 --2- 172.21.15.28:0/2617904138 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2b000a3ec0 0x7f2b000a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.120+0000 7f2b13e00640 1 -- 172.21.15.28:0/2617904138 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f2b0c0f06b0 con 0x7f2b1c1040a0 2024-04-24T08:10:27.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.120+0000 7f2b1aa00640 1 --2- 172.21.15.28:0/2617904138 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2b000a3ec0 0x7f2b000a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.121+0000 7f2b1aa00640 1 --2- 172.21.15.28:0/2617904138 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2b000a3ec0 0x7f2b000a6380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c19b8a0 tx=0x7f2b08009260 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:27.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.122+0000 7f2b13e00640 1 -- 172.21.15.28:0/2617904138 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2b0c049060 con 0x7f2b1c1040a0 2024-04-24T08:10:27.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.333+0000 7f2b20c00640 1 -- 172.21.15.28:0/2617904138 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2b1c02d890 con 0x7f2b1c1040a0 2024-04-24T08:10:27.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.335+0000 7f2b13e00640 1 -- 172.21.15.28:0/2617904138 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23099 (secure 0 0 0) 0x7f2b0c0bcc20 con 0x7f2b1c1040a0 2024-04-24T08:10:27.337 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:10:27.337 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:10:20.116137+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1819419617":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6801/361125225":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/3758227512":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/874707228":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/2660639233":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/361125225":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:10:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.338+0000 7f2b20c00640 1 -- 172.21.15.28:0/2617904138 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2b000a3ec0 msgr2=0x7f2b000a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.338+0000 7f2b20c00640 1 --2- 172.21.15.28:0/2617904138 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2b000a3ec0 0x7f2b000a6380 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c19b8a0 tx=0x7f2b08009260 comp rx=0 tx=0).stop 2024-04-24T08:10:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.338+0000 7f2b20c00640 1 -- 172.21.15.28:0/2617904138 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b1c1040a0 msgr2=0x7f2b1c19a390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.338+0000 7f2b20c00640 1 --2- 172.21.15.28:0/2617904138 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b1c1040a0 0x7f2b1c19a390 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f2b0c035040 tx=0x7f2b0c033d90 comp rx=0 tx=0).stop 2024-04-24T08:10:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.338+0000 7f2b20c00640 1 -- 172.21.15.28:0/2617904138 shutdown_connections 2024-04-24T08:10:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.338+0000 7f2b20c00640 1 --2- 172.21.15.28:0/2617904138 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2b000a3ec0 0x7f2b000a6380 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.338+0000 7f2b20c00640 1 --2- 172.21.15.28:0/2617904138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b1c105290 0x7f2b1c19a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.338+0000 7f2b20c00640 1 --2- 172.21.15.28:0/2617904138 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b1c105c50 0x7f2b1c1a1940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.338+0000 7f2b20c00640 1 --2- 172.21.15.28:0/2617904138 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b1c1040a0 0x7f2b1c19a390 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.338+0000 7f2b20c00640 1 -- 172.21.15.28:0/2617904138 >> 172.21.15.28:0/2617904138 conn(0x7f2b1c0ff610 msgr2=0x7f2b1c108610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.338+0000 7f2b20c00640 1 -- 172.21.15.28:0/2617904138 shutdown_connections 2024-04-24T08:10:27.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.339+0000 7f2b20c00640 1 -- 172.21.15.28:0/2617904138 wait complete. 2024-04-24T08:10:27.359 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:02:34.086371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:07:49.819536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:07:50.493486+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:10:27.360 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:10:27.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: from='client.25366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:10:27.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.0 KiB/s wr, 5 op/s 2024-04-24T08:10:27.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:27.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:27.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-24T08:10:27.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-24T08:10:27.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:10:27.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:10:27.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-24T08:10:27.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: Updating smithi117:/etc/ceph/ceph.conf 2024-04-24T08:10:27.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-24T08:10:27.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:10:27.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: Updating smithi117:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:10:27.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:27 smithi028 ceph-mon[27786]: Updating smithi194:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:10:27.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.548+0000 7ffb31000640 1 -- 172.21.15.28:0/2435818360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb2c101f10 msgr2=0x7ffb2c102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.548+0000 7ffb31000640 1 --2- 172.21.15.28:0/2435818360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb2c101f10 0x7ffb2c102330 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c009900 tx=0x7ffb1c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.549+0000 7ffb31000640 1 -- 172.21.15.28:0/2435818360 shutdown_connections 2024-04-24T08:10:27.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.549+0000 7ffb31000640 1 --2- 172.21.15.28:0/2435818360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb2c103a20 0x7ffb2c10a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.549+0000 7ffb31000640 1 --2- 172.21.15.28:0/2435818360 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb2c103060 0x7ffb2c1034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.549+0000 7ffb31000640 1 --2- 172.21.15.28:0/2435818360 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb2c101f10 0x7ffb2c102330 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.549+0000 7ffb31000640 1 -- 172.21.15.28:0/2435818360 >> 172.21.15.28:0/2435818360 conn(0x7ffb2c0fd4a0 msgr2=0x7ffb2c0ff8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:27.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.549+0000 7ffb31000640 1 -- 172.21.15.28:0/2435818360 shutdown_connections 2024-04-24T08:10:27.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.549+0000 7ffb31000640 1 -- 172.21.15.28:0/2435818360 wait complete. 2024-04-24T08:10:27.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.550+0000 7ffb31000640 1 Processor -- start 2024-04-24T08:10:27.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.550+0000 7ffb31000640 1 -- start start 2024-04-24T08:10:27.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.550+0000 7ffb31000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb2c101f10 0x7ffb2c198220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.550+0000 7ffb31000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb2c103060 0x7ffb2c198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.550+0000 7ffb31000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb2c103a20 0x7ffb2c19f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.550+0000 7ffb31000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2c10dc10 con 0x7ffb2c103060 2024-04-24T08:10:27.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.550+0000 7ffb31000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2c10da90 con 0x7ffb2c103a20 2024-04-24T08:10:27.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.550+0000 7ffb31000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ffb2c10dd90 con 0x7ffb2c101f10 2024-04-24T08:10:27.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.551+0000 7ffb2aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb2c103060 0x7ffb2c198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.551+0000 7ffb2aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb2c103060 0x7ffb2c198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:33990/0 (socket says 172.21.15.28:33990) 2024-04-24T08:10:27.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.551+0000 7ffb2aa00640 1 -- 172.21.15.28:0/640889129 learned_addr learned my addr 172.21.15.28:0/640889129 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:27.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.551+0000 7ffb2b400640 1 --2- 172.21.15.28:0/640889129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb2c101f10 0x7ffb2c198220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.551+0000 7ffb2be00640 1 --2- 172.21.15.28:0/640889129 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb2c103a20 0x7ffb2c19f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.551+0000 7ffb2b400640 1 -- 172.21.15.28:0/640889129 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb2c103a20 msgr2=0x7ffb2c19f7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.551+0000 7ffb2b400640 1 --2- 172.21.15.28:0/640889129 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb2c103a20 0x7ffb2c19f7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.551+0000 7ffb2b400640 1 -- 172.21.15.28:0/640889129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb2c103060 msgr2=0x7ffb2c198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.551+0000 7ffb2b400640 1 --2- 172.21.15.28:0/640889129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb2c103060 0x7ffb2c198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.551+0000 7ffb2b400640 1 -- 172.21.15.28:0/640889129 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffb2c19fe40 con 0x7ffb2c101f10 2024-04-24T08:10:27.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.551+0000 7ffb2be00640 1 --2- 172.21.15.28:0/640889129 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb2c103a20 0x7ffb2c19f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:10:27.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.552+0000 7ffb2b400640 1 --2- 172.21.15.28:0/640889129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb2c101f10 0x7ffb2c198220 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c02a910 tx=0x7ffb1c032ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:27.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.552+0000 7ffb23e00640 1 -- 172.21.15.28:0/640889129 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb1c03f030 con 0x7ffb2c101f10 2024-04-24T08:10:27.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.552+0000 7ffb23e00640 1 -- 172.21.15.28:0/640889129 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffb1c003ce0 con 0x7ffb2c101f10 2024-04-24T08:10:27.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.552+0000 7ffb23e00640 1 -- 172.21.15.28:0/640889129 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffb1c035070 con 0x7ffb2c101f10 2024-04-24T08:10:27.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.552+0000 7ffb31000640 1 -- 172.21.15.28:0/640889129 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffb2c1a00d0 con 0x7ffb2c101f10 2024-04-24T08:10:27.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.552+0000 7ffb31000640 1 -- 172.21.15.28:0/640889129 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffb2c1a0540 con 0x7ffb2c101f10 2024-04-24T08:10:27.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.554+0000 7ffb31000640 1 -- 172.21.15.28:0/640889129 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffb2c109220 con 0x7ffb2c101f10 2024-04-24T08:10:27.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.555+0000 7ffb23e00640 1 -- 172.21.15.28:0/640889129 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7ffb1c042030 con 0x7ffb2c101f10 2024-04-24T08:10:27.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.557+0000 7ffb23e00640 1 --2- 172.21.15.28:0/640889129 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ffb100a3ec0 0x7ffb100a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:27.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.557+0000 7ffb23e00640 1 -- 172.21.15.28:0/640889129 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7ffb1c0f0de0 con 0x7ffb2c101f10 2024-04-24T08:10:27.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.557+0000 7ffb2aa00640 1 --2- 172.21.15.28:0/640889129 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ffb100a3ec0 0x7ffb100a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:27.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.558+0000 7ffb2aa00640 1 --2- 172.21.15.28:0/640889129 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ffb100a3ec0 0x7ffb100a6380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c199730 tx=0x7ffb18005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:27.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.560+0000 7ffb23e00640 1 -- 172.21.15.28:0/640889129 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffb1c0bd280 con 0x7ffb2c101f10 2024-04-24T08:10:27.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: from='client.25366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:10:27.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.0 KiB/s wr, 5 op/s 2024-04-24T08:10:27.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:27.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:27.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-24T08:10:27.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-24T08:10:27.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:10:27.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:10:27.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-24T08:10:27.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: Updating smithi117:/etc/ceph/ceph.conf 2024-04-24T08:10:27.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-24T08:10:27.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:10:27.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: Updating smithi117:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:10:27.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:27 smithi117 ceph-mon[32438]: Updating smithi194:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:10:27.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: from='client.25366 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:10:27.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: pgmap v6: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.0 KiB/s wr, 5 op/s 2024-04-24T08:10:27.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:27.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:27.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:10:27.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-24T08:10:27.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-24T08:10:27.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:10:27.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:10:27.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: Updating smithi028:/etc/ceph/ceph.conf 2024-04-24T08:10:27.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: Updating smithi117:/etc/ceph/ceph.conf 2024-04-24T08:10:27.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: Updating smithi194:/etc/ceph/ceph.conf 2024-04-24T08:10:27.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: from='client.15243 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:10:27.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: Updating smithi117:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:10:27.692 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:27 smithi194 ceph-mon[32836]: Updating smithi194:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:10:27.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.779+0000 7ffb31000640 1 -- 172.21.15.28:0/640889129 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7ffb2c062c00 con 0x7ffb2c101f10 2024-04-24T08:10:27.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.780+0000 7ffb23e00640 1 -- 172.21.15.28:0/640889129 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7ffb1c0bcb80 con 0x7ffb2c101f10 2024-04-24T08:10:27.782 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-24T08:10:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.783+0000 7ffb31000640 1 -- 172.21.15.28:0/640889129 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ffb100a3ec0 msgr2=0x7ffb100a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.783+0000 7ffb31000640 1 --2- 172.21.15.28:0/640889129 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ffb100a3ec0 0x7ffb100a6380 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c199730 tx=0x7ffb18005f90 comp rx=0 tx=0).stop 2024-04-24T08:10:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.783+0000 7ffb31000640 1 -- 172.21.15.28:0/640889129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb2c101f10 msgr2=0x7ffb2c198220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.783+0000 7ffb31000640 1 --2- 172.21.15.28:0/640889129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb2c101f10 0x7ffb2c198220 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7ffb1c02a910 tx=0x7ffb1c032ce0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.783+0000 7ffb31000640 1 -- 172.21.15.28:0/640889129 shutdown_connections 2024-04-24T08:10:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.783+0000 7ffb31000640 1 --2- 172.21.15.28:0/640889129 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ffb100a3ec0 0x7ffb100a6380 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.783+0000 7ffb31000640 1 --2- 172.21.15.28:0/640889129 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffb2c103060 0x7ffb2c198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.783+0000 7ffb31000640 1 --2- 172.21.15.28:0/640889129 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb2c103a20 0x7ffb2c19f7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.783+0000 7ffb31000640 1 --2- 172.21.15.28:0/640889129 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffb2c101f10 0x7ffb2c198220 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.783+0000 7ffb31000640 1 -- 172.21.15.28:0/640889129 >> 172.21.15.28:0/640889129 conn(0x7ffb2c0fd4a0 msgr2=0x7ffb2c1063f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.783+0000 7ffb31000640 1 -- 172.21.15.28:0/640889129 shutdown_connections 2024-04-24T08:10:27.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.783+0000 7ffb31000640 1 -- 172.21.15.28:0/640889129 wait complete. 2024-04-24T08:10:27.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:10:28.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.997+0000 7f2caa400640 1 -- 172.21.15.28:0/1624396916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca40749e0 msgr2=0x7f2ca4074e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.997+0000 7f2caa400640 1 --2- 172.21.15.28:0/1624396916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca40749e0 0x7f2ca4074e40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2c90009900 tx=0x7f2c9002a480 comp rx=0 tx=0).stop 2024-04-24T08:10:28.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.997+0000 7f2caa400640 1 -- 172.21.15.28:0/1624396916 shutdown_connections 2024-04-24T08:10:28.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.997+0000 7f2caa400640 1 --2- 172.21.15.28:0/1624396916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca40749e0 0x7f2ca4074e40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.997+0000 7f2caa400640 1 --2- 172.21.15.28:0/1624396916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ca4109f00 0x7f2ca410c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.997+0000 7f2caa400640 1 --2- 172.21.15.28:0/1624396916 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ca4075fc0 0x7f2ca4074410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.997+0000 7f2caa400640 1 -- 172.21.15.28:0/1624396916 >> 172.21.15.28:0/1624396916 conn(0x7f2ca40fd5e0 msgr2=0x7f2ca40ffa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:28.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.997+0000 7f2caa400640 1 -- 172.21.15.28:0/1624396916 shutdown_connections 2024-04-24T08:10:28.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.998+0000 7f2caa400640 1 -- 172.21.15.28:0/1624396916 wait complete. 2024-04-24T08:10:28.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.999+0000 7f2caa400640 1 Processor -- start 2024-04-24T08:10:28.004 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.999+0000 7f2caa400640 1 -- start start 2024-04-24T08:10:28.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.999+0000 7f2caa400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ca40749e0 0x7f2ca419a590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:28.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.999+0000 7f2caa400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca4075fc0 0x7f2ca419aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:28.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.999+0000 7f2caa400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ca4109f00 0x7f2ca41a1b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:28.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.999+0000 7f2caa400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca410ff40 con 0x7f2ca4075fc0 2024-04-24T08:10:28.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.999+0000 7f2caa400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca410fdc0 con 0x7f2ca4109f00 2024-04-24T08:10:28.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:27.999+0000 7f2caa400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2ca41100c0 con 0x7f2ca40749e0 2024-04-24T08:10:28.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.000+0000 7f2ca2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca4075fc0 0x7f2ca419aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:28.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.000+0000 7f2ca3400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ca40749e0 0x7f2ca419a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:28.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.000+0000 7f2ca3e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ca4109f00 0x7f2ca41a1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:28.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.000+0000 7f2ca2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca4075fc0 0x7f2ca419aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34014/0 (socket says 172.21.15.28:34014) 2024-04-24T08:10:28.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.000+0000 7f2ca2a00640 1 -- 172.21.15.28:0/1712749364 learned_addr learned my addr 172.21.15.28:0/1712749364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:28.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.000+0000 7f2ca3e00640 1 -- 172.21.15.28:0/1712749364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ca40749e0 msgr2=0x7f2ca419a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.000+0000 7f2ca3e00640 1 --2- 172.21.15.28:0/1712749364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ca40749e0 0x7f2ca419a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.000+0000 7f2ca3e00640 1 -- 172.21.15.28:0/1712749364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca4075fc0 msgr2=0x7f2ca419aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.000+0000 7f2ca3e00640 1 --2- 172.21.15.28:0/1712749364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca4075fc0 0x7f2ca419aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.000+0000 7f2ca3e00640 1 -- 172.21.15.28:0/1712749364 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ca41a21b0 con 0x7f2ca4109f00 2024-04-24T08:10:28.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.000+0000 7f2ca2a00640 1 --2- 172.21.15.28:0/1712749364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca4075fc0 0x7f2ca419aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:28.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.000+0000 7f2ca3e00640 1 --2- 172.21.15.28:0/1712749364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ca4109f00 0x7f2ca41a1b40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009fd0 tx=0x7f2c9c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:28.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.001+0000 7f2c9be00640 1 -- 172.21.15.28:0/1712749364 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c00ce20 con 0x7f2ca4109f00 2024-04-24T08:10:28.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.001+0000 7f2c9be00640 1 -- 172.21.15.28:0/1712749364 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2c9c005020 con 0x7f2ca4109f00 2024-04-24T08:10:28.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.001+0000 7f2caa400640 1 -- 172.21.15.28:0/1712749364 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ca41a24a0 con 0x7f2ca4109f00 2024-04-24T08:10:28.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.001+0000 7f2c9be00640 1 -- 172.21.15.28:0/1712749364 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2c9c010720 con 0x7f2ca4109f00 2024-04-24T08:10:28.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.001+0000 7f2caa400640 1 -- 172.21.15.28:0/1712749364 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ca41a29f0 con 0x7f2ca4109f00 2024-04-24T08:10:28.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.002+0000 7f2caa400640 1 -- 172.21.15.28:0/1712749364 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2ca4108cc0 con 0x7f2ca4109f00 2024-04-24T08:10:28.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.004+0000 7f2c9be00640 1 -- 172.21.15.28:0/1712749364 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7f2c9c005540 con 0x7f2ca4109f00 2024-04-24T08:10:28.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.005+0000 7f2c9be00640 1 --2- 172.21.15.28:0/1712749364 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2c880a3ec0 0x7f2c880a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:28.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.006+0000 7f2c9be00640 1 -- 172.21.15.28:0/1712749364 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f2c9c0d11c0 con 0x7f2ca4109f00 2024-04-24T08:10:28.007 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.006+0000 7f2ca3400640 1 --2- 172.21.15.28:0/1712749364 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2c880a3ec0 0x7f2c880a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:28.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.006+0000 7f2ca3400640 1 --2- 172.21.15.28:0/1712749364 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2c880a3ec0 0x7f2c880a6380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009bd0 tx=0x7f2c94005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:28.010 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.009+0000 7f2c9be00640 1 -- 172.21.15.28:0/1712749364 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f2c9c09d590 con 0x7f2ca4109f00 2024-04-24T08:10:28.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.237+0000 7f2caa400640 1 -- 172.21.15.28:0/1712749364 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2ca4062c00 con 0x7f2ca4109f00 2024-04-24T08:10:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.237+0000 7f2c9be00640 1 -- 172.21.15.28:0/1712749364 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f2c9c09ce90 con 0x7f2ca4109f00 2024-04-24T08:10:28.239 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-24T08:10:28.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.241+0000 7f2caa400640 1 -- 172.21.15.28:0/1712749364 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2c880a3ec0 msgr2=0x7f2c880a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.241+0000 7f2caa400640 1 --2- 172.21.15.28:0/1712749364 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2c880a3ec0 0x7f2c880a6380 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2c94009bd0 tx=0x7f2c94005f90 comp rx=0 tx=0).stop 2024-04-24T08:10:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.241+0000 7f2caa400640 1 -- 172.21.15.28:0/1712749364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ca4109f00 msgr2=0x7f2ca41a1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.241+0000 7f2caa400640 1 --2- 172.21.15.28:0/1712749364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ca4109f00 0x7f2ca41a1b40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f2c9c009fd0 tx=0x7f2c9c00ef10 comp rx=0 tx=0).stop 2024-04-24T08:10:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.241+0000 7f2caa400640 1 -- 172.21.15.28:0/1712749364 shutdown_connections 2024-04-24T08:10:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.241+0000 7f2caa400640 1 --2- 172.21.15.28:0/1712749364 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2c880a3ec0 0x7f2c880a6380 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.241+0000 7f2caa400640 1 --2- 172.21.15.28:0/1712749364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2ca4075fc0 0x7f2ca419aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.241+0000 7f2caa400640 1 --2- 172.21.15.28:0/1712749364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ca4109f00 0x7f2ca41a1b40 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.241+0000 7f2caa400640 1 --2- 172.21.15.28:0/1712749364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2ca40749e0 0x7f2ca419a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.241+0000 7f2caa400640 1 -- 172.21.15.28:0/1712749364 >> 172.21.15.28:0/1712749364 conn(0x7f2ca40fd5e0 msgr2=0x7f2ca4105e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.241+0000 7f2caa400640 1 -- 172.21.15.28:0/1712749364 shutdown_connections 2024-04-24T08:10:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.241+0000 7f2caa400640 1 -- 172.21.15.28:0/1712749364 wait complete. 2024-04-24T08:10:28.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:10:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.455+0000 7f8ac8200640 1 -- 172.21.15.28:0/2905039381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ac0106390 msgr2=0x7f8ac0106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.455+0000 7f8ac8200640 1 --2- 172.21.15.28:0/2905039381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ac0106390 0x7f8ac0106820 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8009ca0 tx=0x7f8ab802a9e0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.455+0000 7f8ac8200640 1 -- 172.21.15.28:0/2905039381 shutdown_connections 2024-04-24T08:10:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.455+0000 7f8ac8200640 1 --2- 172.21.15.28:0/2905039381 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ac0106390 0x7f8ac0106820 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.455+0000 7f8ac8200640 1 --2- 172.21.15.28:0/2905039381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ac0101990 0x7f8ac0105e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.455+0000 7f8ac8200640 1 --2- 172.21.15.28:0/2905039381 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ac0101070 0x7f8ac0101450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.455+0000 7f8ac8200640 1 -- 172.21.15.28:0/2905039381 >> 172.21.15.28:0/2905039381 conn(0x7f8ac0078000 msgr2=0x7f8ac0078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:28.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.456+0000 7f8ac8200640 1 -- 172.21.15.28:0/2905039381 shutdown_connections 2024-04-24T08:10:28.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.456+0000 7f8ac8200640 1 -- 172.21.15.28:0/2905039381 wait complete. 2024-04-24T08:10:28.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.457+0000 7f8ac8200640 1 Processor -- start 2024-04-24T08:10:28.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.457+0000 7f8ac8200640 1 -- start start 2024-04-24T08:10:28.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.457+0000 7f8ac8200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ac0101070 0x7f8ac019eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:28.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.457+0000 7f8ac8200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ac0101990 0x7f8ac019f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.457+0000 7f8ac8200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ac0106390 0x7f8ac01a37b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.457+0000 7f8ac8200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac0118a10 con 0x7f8ac0101990 2024-04-24T08:10:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.457+0000 7f8ac8200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac0118890 con 0x7f8ac0106390 2024-04-24T08:10:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.457+0000 7f8ac8200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8ac0118b90 con 0x7f8ac0101070 2024-04-24T08:10:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.457+0000 7f8ac6200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ac0106390 0x7f8ac01a37b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.457+0000 7f8ac4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ac0101990 0x7f8ac019f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.457+0000 7f8ac6200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ac0106390 0x7f8ac01a37b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:38210/0 (socket says 172.21.15.28:38210) 2024-04-24T08:10:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.457+0000 7f8ac6200640 1 -- 172.21.15.28:0/4061515171 learned_addr learned my addr 172.21.15.28:0/4061515171 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.457+0000 7f8ac5800640 1 --2- 172.21.15.28:0/4061515171 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ac0101070 0x7f8ac019eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.458+0000 7f8ac6200640 1 -- 172.21.15.28:0/4061515171 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ac0101070 msgr2=0x7f8ac019eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.458+0000 7f8ac6200640 1 --2- 172.21.15.28:0/4061515171 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ac0101070 0x7f8ac019eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.458+0000 7f8ac6200640 1 -- 172.21.15.28:0/4061515171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ac0101990 msgr2=0x7f8ac019f430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.458+0000 7f8ac6200640 1 --2- 172.21.15.28:0/4061515171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ac0101990 0x7f8ac019f430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.458+0000 7f8ac6200640 1 -- 172.21.15.28:0/4061515171 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ac01a3ee0 con 0x7f8ac0106390 2024-04-24T08:10:28.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.458+0000 7f8ac6200640 1 --2- 172.21.15.28:0/4061515171 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ac0106390 0x7f8ac01a37b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8005e30 tx=0x7f8ab8003ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.459+0000 7f8abe000640 1 -- 172.21.15.28:0/4061515171 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab8047070 con 0x7f8ac0106390 2024-04-24T08:10:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.459+0000 7f8abe000640 1 -- 172.21.15.28:0/4061515171 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ab802e870 con 0x7f8ac0106390 2024-04-24T08:10:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.459+0000 7f8abe000640 1 -- 172.21.15.28:0/4061515171 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ab8033050 con 0x7f8ac0106390 2024-04-24T08:10:28.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.459+0000 7f8ac8200640 1 -- 172.21.15.28:0/4061515171 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ac01a4170 con 0x7f8ac0106390 2024-04-24T08:10:28.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.459+0000 7f8ac8200640 1 -- 172.21.15.28:0/4061515171 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ac01aba60 con 0x7f8ac0106390 2024-04-24T08:10:28.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.460+0000 7f8ac8200640 1 -- 172.21.15.28:0/4061515171 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ac0068e20 con 0x7f8ac0106390 2024-04-24T08:10:28.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.462+0000 7f8abe000640 1 -- 172.21.15.28:0/4061515171 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7f8ab804a070 con 0x7f8ac0106390 2024-04-24T08:10:28.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.463+0000 7f8abe000640 1 --2- 172.21.15.28:0/4061515171 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8aa80a3ec0 0x7f8aa80a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:28.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.464+0000 7f8abe000640 1 -- 172.21.15.28:0/4061515171 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f8ab80f1920 con 0x7f8ac0106390 2024-04-24T08:10:28.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.464+0000 7f8ac5800640 1 --2- 172.21.15.28:0/4061515171 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8aa80a3ec0 0x7f8aa80a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:28.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.465+0000 7f8ac5800640 1 --2- 172.21.15.28:0/4061515171 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8aa80a3ec0 0x7f8aa80a6380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4009bd0 tx=0x7f8ab4005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:28.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.465+0000 7f8abe000640 1 -- 172.21.15.28:0/4061515171 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f8ab8051080 con 0x7f8ac0106390 2024-04-24T08:10:28.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:28 smithi028 ceph-mon[27786]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:10:28.668 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:28 smithi028 ceph-mon[27786]: from='client.25378 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:10:28.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:28 smithi028 ceph-mon[27786]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:10:28.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:28 smithi028 ceph-mon[27786]: Updating smithi028:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:10:28.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:28 smithi028 ceph-mon[27786]: Updating smithi117:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:10:28.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:28 smithi028 ceph-mon[27786]: Updating smithi194:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:10:28.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:28 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2617904138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:28.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:28 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:28.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:28 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:28.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:28 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:28.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:28 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:28.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:28 smithi028 ceph-mon[27786]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:10:28.669 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:28 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/640889129' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:10:28.670 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:28 smithi117 ceph-mon[32438]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:10:28.670 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:28 smithi117 ceph-mon[32438]: from='client.25378 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:10:28.670 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:28 smithi117 ceph-mon[32438]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:10:28.670 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:28 smithi117 ceph-mon[32438]: Updating smithi028:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:10:28.670 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:28 smithi117 ceph-mon[32438]: Updating smithi117:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:10:28.670 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:28 smithi117 ceph-mon[32438]: Updating smithi194:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:10:28.671 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:28 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2617904138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:28.671 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:28 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:28.671 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:28 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:28.671 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:28 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:28.671 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:28 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:28.671 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:28 smithi117 ceph-mon[32438]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:10:28.671 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:28 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/640889129' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:10:28.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.674+0000 7f8ac8200640 1 -- 172.21.15.28:0/4061515171 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8ac01abc30 con 0x7f8ac0106390 2024-04-24T08:10:28.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.675+0000 7f8abe000640 1 -- 172.21.15.28:0/4061515171 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f8ab80bd6a0 con 0x7f8ac0106390 2024-04-24T08:10:28.676 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-24T08:10:28.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.677+0000 7f8ac8200640 1 -- 172.21.15.28:0/4061515171 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8aa80a3ec0 msgr2=0x7f8aa80a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.677+0000 7f8ac8200640 1 --2- 172.21.15.28:0/4061515171 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8aa80a3ec0 0x7f8aa80a6380 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f8ab4009bd0 tx=0x7f8ab4005f90 comp rx=0 tx=0).stop 2024-04-24T08:10:28.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.677+0000 7f8ac8200640 1 -- 172.21.15.28:0/4061515171 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ac0106390 msgr2=0x7f8ac01a37b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.677+0000 7f8ac8200640 1 --2- 172.21.15.28:0/4061515171 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ac0106390 0x7f8ac01a37b0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f8ab8005e30 tx=0x7f8ab8003ea0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.677+0000 7f8ac8200640 1 -- 172.21.15.28:0/4061515171 shutdown_connections 2024-04-24T08:10:28.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.677+0000 7f8ac8200640 1 --2- 172.21.15.28:0/4061515171 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8aa80a3ec0 0x7f8aa80a6380 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.677+0000 7f8ac8200640 1 --2- 172.21.15.28:0/4061515171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8ac0101990 0x7f8ac019f430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.677+0000 7f8ac8200640 1 --2- 172.21.15.28:0/4061515171 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ac0106390 0x7f8ac01a37b0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.677+0000 7f8ac8200640 1 --2- 172.21.15.28:0/4061515171 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8ac0101070 0x7f8ac019eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.677+0000 7f8ac8200640 1 -- 172.21.15.28:0/4061515171 >> 172.21.15.28:0/4061515171 conn(0x7f8ac0078000 msgr2=0x7f8ac0103e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:28.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.678+0000 7f8ac8200640 1 -- 172.21.15.28:0/4061515171 shutdown_connections 2024-04-24T08:10:28.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.678+0000 7f8ac8200640 1 -- 172.21.15.28:0/4061515171 wait complete. 2024-04-24T08:10:28.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:28 smithi194 ceph-mon[32836]: Updating smithi117:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:10:28.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:28 smithi194 ceph-mon[32836]: from='client.25378 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-04-24T08:10:28.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:28 smithi194 ceph-mon[32836]: Updating smithi194:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:10:28.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:28 smithi194 ceph-mon[32836]: Updating smithi028:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.conf 2024-04-24T08:10:28.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:28 smithi194 ceph-mon[32836]: Updating smithi117:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:10:28.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:28 smithi194 ceph-mon[32836]: Updating smithi194:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:10:28.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:28 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2617904138' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:28.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:28 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:28.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:28 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:28.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:28 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:28.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:28 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:28.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:28 smithi194 ceph-mon[32836]: Updating smithi028:/etc/ceph/ceph.client.admin.keyring 2024-04-24T08:10:28.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:28 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/640889129' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:10:28.699 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:10:28.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.889+0000 7ff2f4c00640 1 -- 172.21.15.28:0/2004708153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2f0107160 msgr2=0x7ff2f010ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.889+0000 7ff2f4c00640 1 --2- 172.21.15.28:0/2004708153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2f0107160 0x7ff2f010ed30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff2e8009be0 tx=0x7ff2e802a6c0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.889+0000 7ff2f4c00640 1 -- 172.21.15.28:0/2004708153 shutdown_connections 2024-04-24T08:10:28.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.889+0000 7ff2f4c00640 1 --2- 172.21.15.28:0/2004708153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2f0107160 0x7ff2f010ed30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.889+0000 7ff2f4c00640 1 --2- 172.21.15.28:0/2004708153 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2f0102a40 0x7ff2f0106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.889+0000 7ff2f4c00640 1 --2- 172.21.15.28:0/2004708153 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f0102090 0x7ff2f0102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.889+0000 7ff2f4c00640 1 -- 172.21.15.28:0/2004708153 >> 172.21.15.28:0/2004708153 conn(0x7ff2f00fd930 msgr2=0x7ff2f00ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:28.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.889+0000 7ff2f4c00640 1 -- 172.21.15.28:0/2004708153 shutdown_connections 2024-04-24T08:10:28.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.889+0000 7ff2f4c00640 1 -- 172.21.15.28:0/2004708153 wait complete. 2024-04-24T08:10:28.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.890+0000 7ff2f4c00640 1 Processor -- start 2024-04-24T08:10:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.890+0000 7ff2f4c00640 1 -- start start 2024-04-24T08:10:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.890+0000 7ff2f4c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f0102090 0x7ff2f01a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.890+0000 7ff2f4c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2f0107160 0x7ff2f01a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.890+0000 7ff2f4c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2f019a720 0x7ff2f019aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.890+0000 7ff2f4c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f0112200 con 0x7ff2f0102090 2024-04-24T08:10:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.890+0000 7ff2f4c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f0112080 con 0x7ff2f019a720 2024-04-24T08:10:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.890+0000 7ff2f4c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff2f0112380 con 0x7ff2f0107160 2024-04-24T08:10:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.891+0000 7ff2ef400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f0102090 0x7ff2f01a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.891+0000 7ff2ef400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f0102090 0x7ff2f01a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34046/0 (socket says 172.21.15.28:34046) 2024-04-24T08:10:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.891+0000 7ff2ef400640 1 -- 172.21.15.28:0/2217751333 learned_addr learned my addr 172.21.15.28:0/2217751333 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.891+0000 7ff2e7e00640 1 --2- 172.21.15.28:0/2217751333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2f0107160 0x7ff2f01a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.891+0000 7ff2efe00640 1 --2- 172.21.15.28:0/2217751333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2f019a720 0x7ff2f019aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.891+0000 7ff2ef400640 1 -- 172.21.15.28:0/2217751333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2f0107160 msgr2=0x7ff2f01a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.891+0000 7ff2ef400640 1 --2- 172.21.15.28:0/2217751333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2f0107160 0x7ff2f01a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.891+0000 7ff2ef400640 1 -- 172.21.15.28:0/2217751333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2f019a720 msgr2=0x7ff2f019aba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:28.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.891+0000 7ff2ef400640 1 --2- 172.21.15.28:0/2217751333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2f019a720 0x7ff2f019aba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:28.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.891+0000 7ff2ef400640 1 -- 172.21.15.28:0/2217751333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff2f019b320 con 0x7ff2f0102090 2024-04-24T08:10:28.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.891+0000 7ff2ef400640 1 --2- 172.21.15.28:0/2217751333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f0102090 0x7ff2f01a0660 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff2e000e9c0 tx=0x7ff2e000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:28.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.892+0000 7ff2ecc00640 1 -- 172.21.15.28:0/2217751333 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e000ccc0 con 0x7ff2f0102090 2024-04-24T08:10:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.892+0000 7ff2ecc00640 1 -- 172.21.15.28:0/2217751333 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff2e0005500 con 0x7ff2f0102090 2024-04-24T08:10:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.892+0000 7ff2ecc00640 1 -- 172.21.15.28:0/2217751333 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2e0010430 con 0x7ff2f0102090 2024-04-24T08:10:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.892+0000 7ff2f4c00640 1 -- 172.21.15.28:0/2217751333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff2f01a7400 con 0x7ff2f0102090 2024-04-24T08:10:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.892+0000 7ff2f4c00640 1 -- 172.21.15.28:0/2217751333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff2f01a7860 con 0x7ff2f0102090 2024-04-24T08:10:28.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.893+0000 7ff2f4c00640 1 -- 172.21.15.28:0/2217751333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff2f0103560 con 0x7ff2f0102090 2024-04-24T08:10:28.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.895+0000 7ff2ecc00640 1 -- 172.21.15.28:0/2217751333 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7ff2e00046c0 con 0x7ff2f0102090 2024-04-24T08:10:28.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.897+0000 7ff2ecc00640 1 --2- 172.21.15.28:0/2217751333 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff2d40a3ec0 0x7ff2d40a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:28.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.898+0000 7ff2ecc00640 1 -- 172.21.15.28:0/2217751333 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7ff2e00d0d80 con 0x7ff2f0102090 2024-04-24T08:10:28.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.898+0000 7ff2e7e00640 1 --2- 172.21.15.28:0/2217751333 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff2d40a3ec0 0x7ff2d40a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:28.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.899+0000 7ff2e7e00640 1 --2- 172.21.15.28:0/2217751333 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff2d40a3ec0 0x7ff2d40a6380 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff2f019bf70 tx=0x7ff2d8005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:28.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:28.902+0000 7ff2ecc00640 1 -- 172.21.15.28:0/2217751333 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff2e009d150 con 0x7ff2f0102090 2024-04-24T08:10:29.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.155+0000 7ff2f4c00640 1 -- 172.21.15.28:0/2217751333 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff2f0102b20 con 0x7ff2f0102090 2024-04-24T08:10:29.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.155+0000 7ff2ecc00640 1 -- 172.21.15.28:0/2217751333 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11871 (secure 0 0 0) 0x7ff2e009ca50 con 0x7ff2f0102090 2024-04-24T08:10:29.157 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:10:29.157 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:10:19.714831+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24574,24556,24572]},"id":1}]} 2024-04-24T08:10:29.157 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-24T08:10:29.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.158+0000 7ff2f4c00640 1 -- 172.21.15.28:0/2217751333 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff2d40a3ec0 msgr2=0x7ff2d40a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.158+0000 7ff2f4c00640 1 --2- 172.21.15.28:0/2217751333 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff2d40a3ec0 0x7ff2d40a6380 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7ff2f019bf70 tx=0x7ff2d8005f90 comp rx=0 tx=0).stop 2024-04-24T08:10:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.158+0000 7ff2f4c00640 1 -- 172.21.15.28:0/2217751333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f0102090 msgr2=0x7ff2f01a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.158+0000 7ff2f4c00640 1 --2- 172.21.15.28:0/2217751333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f0102090 0x7ff2f01a0660 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7ff2e000e9c0 tx=0x7ff2e000ee80 comp rx=0 tx=0).stop 2024-04-24T08:10:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.158+0000 7ff2f4c00640 1 -- 172.21.15.28:0/2217751333 shutdown_connections 2024-04-24T08:10:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.158+0000 7ff2f4c00640 1 --2- 172.21.15.28:0/2217751333 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff2d40a3ec0 0x7ff2d40a6380 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.158+0000 7ff2f4c00640 1 --2- 172.21.15.28:0/2217751333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2f019a720 0x7ff2f019aba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.158+0000 7ff2f4c00640 1 --2- 172.21.15.28:0/2217751333 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff2f0107160 0x7ff2f01a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.158+0000 7ff2f4c00640 1 --2- 172.21.15.28:0/2217751333 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff2f0102090 0x7ff2f01a0660 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.158+0000 7ff2f4c00640 1 -- 172.21.15.28:0/2217751333 >> 172.21.15.28:0/2217751333 conn(0x7ff2f00fd930 msgr2=0x7ff2f00ffc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.158+0000 7ff2f4c00640 1 -- 172.21.15.28:0/2217751333 shutdown_connections 2024-04-24T08:10:29.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.158+0000 7ff2f4c00640 1 -- 172.21.15.28:0/2217751333 wait complete. 2024-04-24T08:10:29.177 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:10:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.365+0000 7fd2a9800640 1 -- 172.21.15.28:0/2918882957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a4104c10 msgr2=0x7fd2a4108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.365+0000 7fd2a9800640 1 --2- 172.21.15.28:0/2918882957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a4104c10 0x7fd2a4108c50 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fd290009900 tx=0x7fd29002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.365+0000 7fd2a9800640 1 -- 172.21.15.28:0/2918882957 shutdown_connections 2024-04-24T08:10:29.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.365+0000 7fd2a9800640 1 --2- 172.21.15.28:0/2918882957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a4104c10 0x7fd2a4108c50 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.365+0000 7fd2a9800640 1 --2- 172.21.15.28:0/2918882957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2a4109330 0x7fd2a4110f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.365+0000 7fd2a9800640 1 --2- 172.21.15.28:0/2918882957 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd2a4104260 0x7fd2a4104640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.365+0000 7fd2a9800640 1 -- 172.21.15.28:0/2918882957 >> 172.21.15.28:0/2918882957 conn(0x7fd2a40ffaa0 msgr2=0x7fd2a4101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:29.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.365+0000 7fd2a9800640 1 -- 172.21.15.28:0/2918882957 shutdown_connections 2024-04-24T08:10:29.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.365+0000 7fd2a9800640 1 -- 172.21.15.28:0/2918882957 wait complete. 2024-04-24T08:10:29.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.366+0000 7fd2a9800640 1 Processor -- start 2024-04-24T08:10:29.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.366+0000 7fd2a9800640 1 -- start start 2024-04-24T08:10:29.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.366+0000 7fd2a9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a4104260 0x7fd2a41a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.366+0000 7fd2a9800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2a4109330 0x7fd2a41a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.366+0000 7fd2a9800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd2a419a710 0x7fd2a419ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.366+0000 7fd2a9800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a4114410 con 0x7fd2a4104260 2024-04-24T08:10:29.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.366+0000 7fd2a9800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a4114290 con 0x7fd2a4109330 2024-04-24T08:10:29.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.366+0000 7fd2a9800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd2a4114590 con 0x7fd2a419a710 2024-04-24T08:10:29.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.367+0000 7fd2a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a4104260 0x7fd2a41a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.367+0000 7fd2a3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a4104260 0x7fd2a41a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:34072/0 (socket says 172.21.15.28:34072) 2024-04-24T08:10:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.367+0000 7fd2a3e00640 1 -- 172.21.15.28:0/1850954336 learned_addr learned my addr 172.21.15.28:0/1850954336 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.367+0000 7fd2a8e00640 1 --2- 172.21.15.28:0/1850954336 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd2a419a710 0x7fd2a419ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.367+0000 7fd2a3400640 1 --2- 172.21.15.28:0/1850954336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2a4109330 0x7fd2a41a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.367+0000 7fd2a3e00640 1 -- 172.21.15.28:0/1850954336 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd2a419a710 msgr2=0x7fd2a419ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.367+0000 7fd2a3e00640 1 --2- 172.21.15.28:0/1850954336 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd2a419a710 0x7fd2a419ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.367+0000 7fd2a3e00640 1 -- 172.21.15.28:0/1850954336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2a4109330 msgr2=0x7fd2a41a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.367+0000 7fd2a3e00640 1 --2- 172.21.15.28:0/1850954336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2a4109330 0x7fd2a41a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.367+0000 7fd2a3e00640 1 -- 172.21.15.28:0/1850954336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2a419b310 con 0x7fd2a4104260 2024-04-24T08:10:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.367+0000 7fd2a3e00640 1 --2- 172.21.15.28:0/1850954336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a4104260 0x7fd2a41a0650 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd29400e9f0 tx=0x7fd29400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.368+0000 7fd2a0c00640 1 -- 172.21.15.28:0/1850954336 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd29400cd80 con 0x7fd2a4104260 2024-04-24T08:10:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.368+0000 7fd2a9800640 1 -- 172.21.15.28:0/1850954336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2a41a73f0 con 0x7fd2a4104260 2024-04-24T08:10:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.368+0000 7fd2a0c00640 1 -- 172.21.15.28:0/1850954336 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd294005500 con 0x7fd2a4104260 2024-04-24T08:10:29.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.368+0000 7fd2a0c00640 1 -- 172.21.15.28:0/1850954336 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd294010690 con 0x7fd2a4104260 2024-04-24T08:10:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.368+0000 7fd2a9800640 1 -- 172.21.15.28:0/1850954336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2a41a77a0 con 0x7fd2a4104260 2024-04-24T08:10:29.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.369+0000 7fd2a9800640 1 -- 172.21.15.28:0/1850954336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd2a4105730 con 0x7fd2a4104260 2024-04-24T08:10:29.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.371+0000 7fd2a0c00640 1 -- 172.21.15.28:0/1850954336 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7fd2940046c0 con 0x7fd2a4104260 2024-04-24T08:10:29.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.372+0000 7fd2a0c00640 1 --2- 172.21.15.28:0/1850954336 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd2880a3e40 0x7fd2880a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.372+0000 7fd2a0c00640 1 -- 172.21.15.28:0/1850954336 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fd2940d0fd0 con 0x7fd2a4104260 2024-04-24T08:10:29.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.373+0000 7fd2a3400640 1 --2- 172.21.15.28:0/1850954336 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd2880a3e40 0x7fd2880a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.373+0000 7fd2a3400640 1 --2- 172.21.15.28:0/1850954336 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd2880a3e40 0x7fd2880a6300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd290006010 tx=0x7fd290005a20 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:29.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.377+0000 7fd2a0c00640 1 -- 172.21.15.28:0/1850954336 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd29409d3a0 con 0x7fd2a4104260 2024-04-24T08:10:29.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:29 smithi028 ceph-mon[27786]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-04-24T08:10:29.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:29 smithi028 ceph-mon[27786]: Updating smithi028:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:10:29.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:29 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1712749364' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:10:29.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:29 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/4061515171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:10:29.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:29 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:29.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:29 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:29.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:29 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:29.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:29 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:10:29.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:29 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:10:29.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:29 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:10:29.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:29 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:29.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:29 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2217751333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:10:29.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.591+0000 7fd2a9800640 1 -- 172.21.15.28:0/1850954336 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fd2a4069190 con 0x7fd2a4104260 2024-04-24T08:10:29.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.595+0000 7fd2a0c00640 1 -- 172.21.15.28:0/1850954336 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+23099 (secure 0 0 0) 0x7fd29409cca0 con 0x7fd2a4104260 2024-04-24T08:10:29.597 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:10:29.597 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":91,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:10:20.116137+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1819419617":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6801/361125225":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/3758227512":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/874707228":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/2660639233":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/361125225":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:10:29.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.600+0000 7fd2a9800640 1 -- 172.21.15.28:0/1850954336 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd2880a3e40 msgr2=0x7fd2880a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.600+0000 7fd2a9800640 1 --2- 172.21.15.28:0/1850954336 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd2880a3e40 0x7fd2880a6300 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd290006010 tx=0x7fd290005a20 comp rx=0 tx=0).stop 2024-04-24T08:10:29.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.600+0000 7fd2a9800640 1 -- 172.21.15.28:0/1850954336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a4104260 msgr2=0x7fd2a41a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.600+0000 7fd2a9800640 1 --2- 172.21.15.28:0/1850954336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a4104260 0x7fd2a41a0650 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fd29400e9f0 tx=0x7fd29400edf0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.600+0000 7fd2a9800640 1 -- 172.21.15.28:0/1850954336 shutdown_connections 2024-04-24T08:10:29.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.600+0000 7fd2a9800640 1 --2- 172.21.15.28:0/1850954336 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd2880a3e40 0x7fd2880a6300 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.600+0000 7fd2a9800640 1 --2- 172.21.15.28:0/1850954336 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd2a419a710 0x7fd2a419ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.600+0000 7fd2a9800640 1 --2- 172.21.15.28:0/1850954336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2a4109330 0x7fd2a41a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.600+0000 7fd2a9800640 1 --2- 172.21.15.28:0/1850954336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd2a4104260 0x7fd2a41a0650 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.600+0000 7fd2a9800640 1 -- 172.21.15.28:0/1850954336 >> 172.21.15.28:0/1850954336 conn(0x7fd2a40ffaa0 msgr2=0x7fd2a4101da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:29.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.600+0000 7fd2a9800640 1 -- 172.21.15.28:0/1850954336 shutdown_connections 2024-04-24T08:10:29.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.600+0000 7fd2a9800640 1 -- 172.21.15.28:0/1850954336 wait complete. 2024-04-24T08:10:29.621 DEBUG:tasks.fwd_scrub:Yielding 2024-04-24T08:10:29.621 INFO:teuthology.run_tasks:Running task exec... 2024-04-24T08:10:29.631 INFO:teuthology.task.exec:Executing custom commands... 2024-04-24T08:10:29.631 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-24T08:10:29.631 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-04-24T08:10:29.631 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-04-24T08:10:29.632 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:10:29.632 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:10:29.632 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:10:29.633 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:29 smithi117 ceph-mon[32438]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-04-24T08:10:29.633 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:29 smithi117 ceph-mon[32438]: Updating smithi028:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:10:29.633 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:29 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1712749364' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:10:29.633 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:29 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/4061515171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:10:29.633 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:29 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:29.633 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:29 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:29.633 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:29 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:29.634 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:29 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:10:29.634 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:29 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:10:29.634 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:29 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:10:29.634 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:29 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:29.634 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:29 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2217751333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:10:29.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:29 smithi194 ceph-mon[32836]: pgmap v7: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 1.1 KiB/s wr, 23 op/s 2024-04-24T08:10:29.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:29 smithi194 ceph-mon[32836]: Updating smithi028:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/config/ceph.client.admin.keyring 2024-04-24T08:10:29.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:29 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1712749364' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:10:29.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:29 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/4061515171' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:10:29.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:29 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:29.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:29 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:29.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:29 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:29.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:29 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:10:29.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:29 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:10:29.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:29 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:10:29.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:29 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:10:29.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:29 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2217751333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:10:29.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.822+0000 7f6a3d800640 1 -- 172.21.15.28:0/3521322316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a381038e0 msgr2=0x7f6a3810bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.822+0000 7f6a3d800640 1 --2- 172.21.15.28:0/3521322316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a381038e0 0x7f6a3810bdc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6a24009900 tx=0x7f6a2402a480 comp rx=0 tx=0).stop 2024-04-24T08:10:29.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.822+0000 7f6a3d800640 1 -- 172.21.15.28:0/3521322316 shutdown_connections 2024-04-24T08:10:29.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.822+0000 7f6a3d800640 1 --2- 172.21.15.28:0/3521322316 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a381038e0 0x7f6a3810bdc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.822+0000 7f6a3d800640 1 --2- 172.21.15.28:0/3521322316 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a3810c3e0 0x7f6a3810e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.822+0000 7f6a3d800640 1 --2- 172.21.15.28:0/3521322316 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a38102fc0 0x7f6a381033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.822+0000 7f6a3d800640 1 -- 172.21.15.28:0/3521322316 >> 172.21.15.28:0/3521322316 conn(0x7f6a380fc820 msgr2=0x7f6a380fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:29.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.822+0000 7f6a3d800640 1 -- 172.21.15.28:0/3521322316 shutdown_connections 2024-04-24T08:10:29.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.822+0000 7f6a3d800640 1 -- 172.21.15.28:0/3521322316 wait complete. 2024-04-24T08:10:29.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.823+0000 7f6a3d800640 1 Processor -- start 2024-04-24T08:10:29.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.823+0000 7f6a3d800640 1 -- start start 2024-04-24T08:10:29.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.824+0000 7f6a3d800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a38102fc0 0x7f6a38100c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.824+0000 7f6a3d800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a381038e0 0x7f6a38101150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.824+0000 7f6a3d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a3810c3e0 0x7f6a38101690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.824+0000 7f6a3d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a38112210 con 0x7f6a3810c3e0 2024-04-24T08:10:29.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.824+0000 7f6a3d800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6a38112090 con 0x7f6a381038e0 2024-04-24T08:10:29.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.824+0000 7f6a3d800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6a38112390 con 0x7f6a38102fc0 2024-04-24T08:10:29.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.824+0000 7f6a3ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a3810c3e0 0x7f6a38101690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.824+0000 7f6a37e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a38102fc0 0x7f6a38100c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.824+0000 7f6a37e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a38102fc0 0x7f6a38100c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:39672/0 (socket says 172.21.15.28:39672) 2024-04-24T08:10:29.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.824+0000 7f6a37e00640 1 -- 172.21.15.28:0/1813557152 learned_addr learned my addr 172.21.15.28:0/1813557152 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:29.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.824+0000 7f6a37400640 1 --2- 172.21.15.28:0/1813557152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a381038e0 0x7f6a38101150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f6a37e00640 1 -- 172.21.15.28:0/1813557152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a381038e0 msgr2=0x7f6a38101150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f6a37e00640 1 --2- 172.21.15.28:0/1813557152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a381038e0 0x7f6a38101150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f6a37e00640 1 -- 172.21.15.28:0/1813557152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a3810c3e0 msgr2=0x7f6a38101690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f6a37e00640 1 --2- 172.21.15.28:0/1813557152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a3810c3e0 0x7f6a38101690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f6a37e00640 1 -- 172.21.15.28:0/1813557152 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a381a6c60 con 0x7f6a38102fc0 2024-04-24T08:10:29.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f6a37400640 1 --2- 172.21.15.28:0/1813557152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a381038e0 0x7f6a38101150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:10:29.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f6a37e00640 1 --2- 172.21.15.28:0/1813557152 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a38102fc0 0x7f6a38100c10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6a2800e9f0 tx=0x7f6a2800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:29.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f6a34c00640 1 -- 172.21.15.28:0/1813557152 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2800cd80 con 0x7f6a38102fc0 2024-04-24T08:10:29.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f6a34c00640 1 -- 172.21.15.28:0/1813557152 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a28010910 con 0x7f6a38102fc0 2024-04-24T08:10:29.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f8622800640 1 -- 172.21.15.28:0/3452306336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f861c1044b0 msgr2=0x7f861c104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f8622800640 1 --2- 172.21.15.28:0/3452306336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f861c1044b0 0x7f861c104890 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f860c009900 tx=0x7f860c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f6a3d800640 1 -- 172.21.15.28:0/1813557152 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a381a6f50 con 0x7f6a38102fc0 2024-04-24T08:10:29.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f6a34c00640 1 -- 172.21.15.28:0/1813557152 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a28018990 con 0x7f6a38102fc0 2024-04-24T08:10:29.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f6a3d800640 1 -- 172.21.15.28:0/1813557152 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a381a73c0 con 0x7f6a38102fc0 2024-04-24T08:10:29.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f8622800640 1 -- 172.21.15.28:0/3452306336 shutdown_connections 2024-04-24T08:10:29.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f8622800640 1 --2- 172.21.15.28:0/3452306336 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f861c109580 0x7f861c111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f8622800640 1 --2- 172.21.15.28:0/3452306336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f861c104e60 0x7f861c108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f8622800640 1 --2- 172.21.15.28:0/3452306336 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f861c1044b0 0x7f861c104890 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.825+0000 7f8622800640 1 -- 172.21.15.28:0/3452306336 >> 172.21.15.28:0/3452306336 conn(0x7f861c0ffd90 msgr2=0x7f861c1021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:29.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.826+0000 7f8622800640 1 -- 172.21.15.28:0/3452306336 shutdown_connections 2024-04-24T08:10:29.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.826+0000 7f8622800640 1 -- 172.21.15.28:0/3452306336 wait complete. 2024-04-24T08:10:29.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.827+0000 7f6a3d800640 1 -- 172.21.15.28:0/1813557152 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6a380505e0 con 0x7f6a38102fc0 2024-04-24T08:10:29.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.827+0000 7f8622800640 1 Processor -- start 2024-04-24T08:10:29.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.827+0000 7f8622800640 1 -- start start 2024-04-24T08:10:29.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.827+0000 7f8622800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f861c1044b0 0x7f861c1a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.827+0000 7f8622800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f861c109580 0x7f861c1a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.827+0000 7f8622800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f861c19aa30 0x7f861c19aee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.827+0000 7f8622800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f861c114690 con 0x7f861c19aa30 2024-04-24T08:10:29.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.827+0000 7f8622800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f861c114510 con 0x7f861c1044b0 2024-04-24T08:10:29.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.827+0000 7f8622800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f861c114810 con 0x7f861c109580 2024-04-24T08:10:29.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.828+0000 7f861be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f861c19aa30 0x7f861c19aee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.828+0000 7f861aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f861c109580 0x7f861c1a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.828+0000 7f861b400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f861c1044b0 0x7f861c1a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.828+0000 7f861b400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f861c1044b0 0x7f861c1a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:38872/0 (socket says 172.21.15.28:38872) 2024-04-24T08:10:29.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.828+0000 7f861be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f861c19aa30 0x7f861c19aee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55466/0 (socket says 172.21.15.28:55466) 2024-04-24T08:10:29.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.828+0000 7f861b400640 1 -- 172.21.15.28:0/1054038540 learned_addr learned my addr 172.21.15.28:0/1054038540 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:29.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.828+0000 7f861aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f861c109580 0x7f861c1a0dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:39684/0 (socket says 172.21.15.28:39684) 2024-04-24T08:10:29.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.828+0000 7f861aa00640 1 -- 172.21.15.28:0/1054038540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f861c1044b0 msgr2=0x7f861c1a0880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.828+0000 7f861aa00640 1 --2- 172.21.15.28:0/1054038540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f861c1044b0 0x7f861c1a0880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.828+0000 7f861aa00640 1 -- 172.21.15.28:0/1054038540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f861c19aa30 msgr2=0x7f861c19aee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.828+0000 7f861aa00640 1 --2- 172.21.15.28:0/1054038540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f861c19aa30 0x7f861c19aee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.828+0000 7f861aa00640 1 -- 172.21.15.28:0/1054038540 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f861c19b550 con 0x7f861c109580 2024-04-24T08:10:29.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.828+0000 7f861aa00640 1 --2- 172.21.15.28:0/1054038540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f861c109580 0x7f861c1a0dc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f860800e9c0 tx=0x7f860800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:29.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.829+0000 7f6a34c00640 1 -- 172.21.15.28:0/1813557152 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7f6a28010ac0 con 0x7f6a38102fc0 2024-04-24T08:10:29.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.830+0000 7f6a34c00640 1 --2- 172.21.15.28:0/1813557152 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6a1c0a3e60 0x7f6a1c0a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.831+0000 7f6a34c00640 1 -- 172.21.15.28:0/1813557152 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f6a280d0fc0 con 0x7f6a38102fc0 2024-04-24T08:10:29.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.831+0000 7f6a37400640 1 --2- 172.21.15.28:0/1813557152 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6a1c0a3e60 0x7f6a1c0a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.831+0000 7f6a37400640 1 --2- 172.21.15.28:0/1813557152 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6a1c0a3e60 0x7f6a1c0a6320 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6a24000c00 tx=0x7f6a24005a40 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:29.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.832+0000 7f8613e00640 1 -- 172.21.15.28:0/1054038540 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860800cd80 con 0x7f861c109580 2024-04-24T08:10:29.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.832+0000 7f8613e00640 1 -- 172.21.15.28:0/1054038540 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86080035f0 con 0x7f861c109580 2024-04-24T08:10:29.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.832+0000 7f8613e00640 1 -- 172.21.15.28:0/1054038540 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8608010690 con 0x7f861c109580 2024-04-24T08:10:29.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.832+0000 7f8622800640 1 -- 172.21.15.28:0/1054038540 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f861c19b840 con 0x7f861c109580 2024-04-24T08:10:29.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.833+0000 7f8622800640 1 -- 172.21.15.28:0/1054038540 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f861c1a7ae0 con 0x7f861c109580 2024-04-24T08:10:29.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.833+0000 7f6a34c00640 1 -- 172.21.15.28:0/1813557152 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6a2809d4b0 con 0x7f6a38102fc0 2024-04-24T08:10:29.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.836+0000 7f86257b9740 1 Processor -- start 2024-04-24T08:10:29.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.836+0000 7f86257b9740 1 -- start start 2024-04-24T08:10:29.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.836+0000 7f86257b9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 0x5601f1172470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.836+0000 7f86257b9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f11729b0 0x5601f117cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.836+0000 7f86257b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f117d080 0x5601f117f470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.836+0000 7f86257b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5601f10d7e70 con 0x5601f117d080 2024-04-24T08:10:29.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.836+0000 7f86257b9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5601f107e380 con 0x5601f11729b0 2024-04-24T08:10:29.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.836+0000 7f86257b9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5601f10d3e30 con 0x5601f1172090 2024-04-24T08:10:29.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.837+0000 7f8613e00640 1 -- 172.21.15.28:0/1054038540 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7f86080037a0 con 0x7f861c109580 2024-04-24T08:10:29.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.837+0000 7f861be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 0x5601f1172470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.837+0000 7f861be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 0x5601f1172470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:39692/0 (socket says 172.21.15.28:39692) 2024-04-24T08:10:29.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.837+0000 7f861be00640 1 -- 172.21.15.28:0/1232120395 learned_addr learned my addr 172.21.15.28:0/1232120395 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:29.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.837+0000 7f861b400640 1 --2- 172.21.15.28:0/1232120395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f11729b0 0x5601f117cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.837+0000 7f861aa00640 1 --2- 172.21.15.28:0/1232120395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f117d080 0x5601f117f470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.837+0000 7f861be00640 1 -- 172.21.15.28:0/1232120395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f11729b0 msgr2=0x5601f117cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:29.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.837+0000 7f861be00640 1 --2- 172.21.15.28:0/1232120395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f11729b0 0x5601f117cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.837+0000 7f861be00640 1 -- 172.21.15.28:0/1232120395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f117d080 msgr2=0x5601f117f470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:29.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.837+0000 7f861be00640 1 --2- 172.21.15.28:0/1232120395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f117d080 0x5601f117f470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.837+0000 7f861be00640 1 -- 172.21.15.28:0/1232120395 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601f0eb27e0 con 0x5601f1172090 2024-04-24T08:10:29.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.837+0000 7f861aa00640 1 --2- 172.21.15.28:0/1232120395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f117d080 0x5601f117f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:10:29.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.838+0000 7f861be00640 1 --2- 172.21.15.28:0/1232120395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 0x5601f1172470 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8614009f50 tx=0x7f86140118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e7cfa62c4416a8b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:29.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.838+0000 7f8610c00640 1 -- 172.21.15.28:0/1232120395 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8614004d20 con 0x5601f1172090 2024-04-24T08:10:29.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.838+0000 7f8610c00640 1 -- 172.21.15.28:0/1232120395 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8614004050 con 0x5601f1172090 2024-04-24T08:10:29.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.838+0000 7f8610c00640 1 -- 172.21.15.28:0/1232120395 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8614005280 con 0x5601f1172090 2024-04-24T08:10:29.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.838+0000 7f8613e00640 1 --2- 172.21.15.28:0/1054038540 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f86000a3ec0 0x7f86000a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.839+0000 7f8613e00640 1 -- 172.21.15.28:0/1054038540 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f86080d0d60 con 0x7f861c109580 2024-04-24T08:10:29.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.839+0000 7f86257b9740 1 -- 172.21.15.28:0/1232120395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 msgr2=0x5601f1172470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.839+0000 7f86257b9740 1 --2- 172.21.15.28:0/1232120395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 0x5601f1172470 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f8614009f50 tx=0x7f86140118d0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.839+0000 7f861b400640 1 --2- 172.21.15.28:0/1054038540 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f86000a3ec0 0x7f86000a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.839+0000 7f86257b9740 1 -- 172.21.15.28:0/1232120395 shutdown_connections 2024-04-24T08:10:29.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.839+0000 7f86257b9740 1 --2- 172.21.15.28:0/1232120395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f117d080 0x5601f117f470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.839+0000 7f86257b9740 1 --2- 172.21.15.28:0/1232120395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f11729b0 0x5601f117cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.839+0000 7f86257b9740 1 --2- 172.21.15.28:0/1232120395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 0x5601f1172470 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.839+0000 7f86257b9740 1 -- 172.21.15.28:0/1232120395 >> 172.21.15.28:0/1232120395 conn(0x5601f116e3b0 msgr2=0x5601f116e790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:29.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.839+0000 7f86257b9740 1 -- 172.21.15.28:0/1232120395 shutdown_connections 2024-04-24T08:10:29.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.839+0000 7f86257b9740 1 -- 172.21.15.28:0/1232120395 wait complete. 2024-04-24T08:10:29.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.840+0000 7f86257b9740 1 -- start start 2024-04-24T08:10:29.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.840+0000 7f861b400640 1 --2- 172.21.15.28:0/1054038540 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f86000a3ec0 0x7f86000a6380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x5601f1173650 tx=0x7f860c0046c0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:29.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.840+0000 7f86257b9740 1 Processor -- start 2024-04-24T08:10:29.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.840+0000 7f86257b9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 0x5601f117a030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.840+0000 7f86257b9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f11729b0 0x5601f117a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.840+0000 7f86257b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f117d080 0x5601f117aab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.840+0000 7f86257b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5601f117fe70 con 0x5601f117d080 2024-04-24T08:10:29.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.841+0000 7f86257b9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5601f1091660 con 0x5601f11729b0 2024-04-24T08:10:29.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.841+0000 7f86257b9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5601f117fff0 con 0x5601f1172090 2024-04-24T08:10:29.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.841+0000 7f86257b9740 10 client.0 _ll_register_callbacks cb 0x5601f107faa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:10:29.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.841+0000 7f861be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f117d080 0x5601f117aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.841+0000 7f861b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 0x5601f117a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.841+0000 7f861be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f117d080 0x5601f117aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55486/0 (socket says 172.21.15.28:55486) 2024-04-24T08:10:29.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.841+0000 7f861be00640 1 -- 172.21.15.28:0/1627383205 learned_addr learned my addr 172.21.15.28:0/1627383205 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:29.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.841+0000 7f861aa00640 1 --2- 172.21.15.28:0/1627383205 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f11729b0 0x5601f117a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.841+0000 7f861be00640 1 -- 172.21.15.28:0/1627383205 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 msgr2=0x5601f117a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.841+0000 7f861be00640 1 --2- 172.21.15.28:0/1627383205 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 0x5601f117a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.841+0000 7f861be00640 1 -- 172.21.15.28:0/1627383205 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f11729b0 msgr2=0x5601f117a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.841+0000 7f861be00640 1 --2- 172.21.15.28:0/1627383205 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f11729b0 0x5601f117a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.841+0000 7f861be00640 1 -- 172.21.15.28:0/1627383205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601f0f12f70 con 0x5601f117d080 2024-04-24T08:10:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.842+0000 7f861be00640 1 --2- 172.21.15.28:0/1627383205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f117d080 0x5601f117aab0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8614012fd0 tx=0x7f8614004490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2181f46c2fa490e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.842+0000 7f8605600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:10:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.842+0000 7f8605600640 1 -- 172.21.15.28:0/1627383205 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8614029070 con 0x5601f117d080 2024-04-24T08:10:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.842+0000 7f8605600640 1 -- 172.21.15.28:0/1627383205 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f861400f480 con 0x5601f117d080 2024-04-24T08:10:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.842+0000 7f86257b9740 1 -- 172.21.15.28:0/1627383205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8614004050 con 0x5601f117d080 2024-04-24T08:10:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.842+0000 7f8605600640 1 -- 172.21.15.28:0/1627383205 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f8614035050 con 0x5601f117d080 2024-04-24T08:10:29.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.842+0000 7f86257b9740 10 client.15300 fetch_fsmap learned FSMap version 24 2024-04-24T08:10:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.842+0000 7f86257b9740 1 -- 172.21.15.28:0/1627383205 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5601f0daa170 con 0x5601f117d080 2024-04-24T08:10:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.843+0000 7f8605600640 1 -- 172.21.15.28:0/1627383205 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8614004050 con 0x5601f117d080 2024-04-24T08:10:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.843+0000 7f8605600640 1 -- 172.21.15.28:0/1627383205 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f86140195d0 con 0x5601f117d080 2024-04-24T08:10:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.843+0000 7f86257b9740 10 client.15300 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:10:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.843+0000 7f86257b9740 10 client.15300 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:10:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.843+0000 7f86257b9740 20 client.15300 populate_metadata read hostname 'smithi028' 2024-04-24T08:10:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.843+0000 7f86257b9740 1 --2- 172.21.15.28:0/1627383205 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5601f118d280 0x5601f118f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.843+0000 7f86257b9740 4 client.15300 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:10:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.843+0000 7f86257b9740 1 -- 172.21.15.28:0/1627383205 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5601f10e5890 con 0x5601f118d280 2024-04-24T08:10:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.843+0000 7f861b400640 1 --2- 172.21.15.28:0/1627383205 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5601f118d280 0x5601f118f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.844+0000 7f861b400640 1 --2- 172.21.15.28:0/1627383205 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5601f118d280 0x5601f118f670 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c995b54fe11a707 server_cookie=5807b567640fc750 in_seq=0 out_seq=0 2024-04-24T08:10:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.844+0000 7f8605600640 10 client.15300 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:10:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.845+0000 7f8605600640 1 -- 172.21.15.28:0/1627383205 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5601f10e5890 con 0x5601f118d280 2024-04-24T08:10:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.845+0000 7f8605600640 10 client.15300 handle_command_reply: tid=0 2024-04-24T08:10:29.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.845+0000 7f86257b9740 1 client.15300 shutdown 2024-04-24T08:10:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.845+0000 7f86257b9740 1 -- 172.21.15.28:0/1627383205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f117d080 msgr2=0x5601f117aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.845+0000 7f86257b9740 1 --2- 172.21.15.28:0/1627383205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f117d080 0x5601f117aab0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f8614012fd0 tx=0x7f8614004490 comp rx=0 tx=0).stop 2024-04-24T08:10:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.845+0000 7f86257b9740 1 -- 172.21.15.28:0/1627383205 shutdown_connections 2024-04-24T08:10:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.845+0000 7f86257b9740 1 --2- 172.21.15.28:0/1627383205 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5601f118d280 0x5601f118f670 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.845+0000 7f86257b9740 1 --2- 172.21.15.28:0/1627383205 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f117d080 0x5601f117aab0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.845+0000 7f86257b9740 1 --2- 172.21.15.28:0/1627383205 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f11729b0 0x5601f117a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.845+0000 7f86257b9740 1 --2- 172.21.15.28:0/1627383205 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 0x5601f117a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.845+0000 7f86257b9740 1 -- 172.21.15.28:0/1627383205 >> 172.21.15.28:0/1627383205 conn(0x5601f116e3b0 msgr2=0x5601f11741b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.845+0000 7f8605600640 0 client.15300 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:10:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.845+0000 7f86257b9740 1 -- 172.21.15.28:0/1627383205 shutdown_connections 2024-04-24T08:10:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.845+0000 7f86257b9740 1 -- 172.21.15.28:0/1627383205 wait complete. 2024-04-24T08:10:29.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.845+0000 7f86257b9740 20 client.15300 trim_cache size 0 max 16384 2024-04-24T08:10:29.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.865+0000 7f86257b9740 1 Processor -- start 2024-04-24T08:10:29.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.865+0000 7f86257b9740 1 -- start start 2024-04-24T08:10:29.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.865+0000 7f86257b9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 0x5601f1174830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.865+0000 7f86257b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f11729b0 0x5601f117a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.865+0000 7f86257b9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f117d080 0x5601f1194de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.865+0000 7f86257b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f861c1b28e0 con 0x5601f11729b0 2024-04-24T08:10:29.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.865+0000 7f86257b9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f861c1b01b0 con 0x5601f117d080 2024-04-24T08:10:29.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.865+0000 7f86257b9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f861c1ae000 con 0x5601f1172090 2024-04-24T08:10:29.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.866+0000 7f861b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 0x5601f1174830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.866+0000 7f861aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f11729b0 0x5601f117a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.866+0000 7f861b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 0x5601f1174830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:39708/0 (socket says 172.21.15.28:39708) 2024-04-24T08:10:29.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.866+0000 7f861b400640 1 -- 172.21.15.28:0/3467608374 learned_addr learned my addr 172.21.15.28:0/3467608374 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.866+0000 7f861be00640 1 --2- 172.21.15.28:0/3467608374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f117d080 0x5601f1194de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.866+0000 7f861b400640 1 -- 172.21.15.28:0/3467608374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f117d080 msgr2=0x5601f1194de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.866+0000 7f861b400640 1 --2- 172.21.15.28:0/3467608374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f117d080 0x5601f1194de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.866+0000 7f861b400640 1 -- 172.21.15.28:0/3467608374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f11729b0 msgr2=0x5601f117a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.866+0000 7f861b400640 1 --2- 172.21.15.28:0/3467608374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f11729b0 0x5601f117a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.866+0000 7f861b400640 1 -- 172.21.15.28:0/3467608374 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5601f1178580 con 0x5601f1172090 2024-04-24T08:10:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.866+0000 7f861be00640 1 --2- 172.21.15.28:0/3467608374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f117d080 0x5601f1194de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.866+0000 7f861aa00640 1 --2- 172.21.15.28:0/3467608374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f11729b0 0x5601f117a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.867+0000 7f861b400640 1 --2- 172.21.15.28:0/3467608374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 0x5601f1174830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f860c004770 tx=0x7f860c03f000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88df61ea5fafa53a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.867+0000 7f8607e00640 1 -- 172.21.15.28:0/3467608374 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c035040 con 0x5601f1172090 2024-04-24T08:10:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.867+0000 7f8607e00640 1 -- 172.21.15.28:0/3467608374 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f860c05a070 con 0x5601f1172090 2024-04-24T08:10:29.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.867+0000 7f8607e00640 1 -- 172.21.15.28:0/3467608374 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c003bd0 con 0x5601f1172090 2024-04-24T08:10:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.868+0000 7f86257b9740 1 -- 172.21.15.28:0/3467608374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 msgr2=0x5601f1174830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.868+0000 7f86257b9740 1 --2- 172.21.15.28:0/3467608374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 0x5601f1174830 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f860c004770 tx=0x7f860c03f000 comp rx=0 tx=0).stop 2024-04-24T08:10:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.868+0000 7f86257b9740 1 -- 172.21.15.28:0/3467608374 shutdown_connections 2024-04-24T08:10:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.868+0000 7f86257b9740 1 --2- 172.21.15.28:0/3467608374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f11729b0 0x5601f117a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.868+0000 7f86257b9740 1 --2- 172.21.15.28:0/3467608374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f117d080 0x5601f1194de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.868+0000 7f86257b9740 1 --2- 172.21.15.28:0/3467608374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f1172090 0x5601f1174830 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.868+0000 7f86257b9740 1 -- 172.21.15.28:0/3467608374 >> 172.21.15.28:0/3467608374 conn(0x5601f116e3b0 msgr2=0x5601f116d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.868+0000 7f86257b9740 1 -- 172.21.15.28:0/3467608374 shutdown_connections 2024-04-24T08:10:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.868+0000 7f86257b9740 1 -- 172.21.15.28:0/3467608374 wait complete. 2024-04-24T08:10:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.868+0000 7f86257b9740 1 -- start start 2024-04-24T08:10:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.868+0000 7f86257b9740 1 Processor -- start 2024-04-24T08:10:29.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.869+0000 7f86257b9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f1172090 0x5601f1185760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.869+0000 7f86257b9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f11729b0 0x5601f1180ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.869+0000 7f86257b9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f117d080 0x5601f1181210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.869+0000 7f86257b9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f861c1ae670 con 0x5601f1172090 2024-04-24T08:10:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.869+0000 7f86257b9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f861c1afae0 con 0x5601f11729b0 2024-04-24T08:10:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.869+0000 7f86257b9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f861c1aed40 con 0x5601f117d080 2024-04-24T08:10:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.869+0000 7f86257b9740 10 client.0 _ll_register_callbacks cb 0x7f86140093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:10:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.869+0000 7f861b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f1172090 0x5601f1185760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.869+0000 7f861b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f1172090 0x5601f1185760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55514/0 (socket says 172.21.15.28:55514) 2024-04-24T08:10:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.869+0000 7f861aa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f11729b0 0x5601f1180ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.869+0000 7f861b400640 1 -- 172.21.15.28:0/2418951489 learned_addr learned my addr 172.21.15.28:0/2418951489 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.869+0000 7f861be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f117d080 0x5601f1181210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.870+0000 7f861b400640 1 -- 172.21.15.28:0/2418951489 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f117d080 msgr2=0x5601f1181210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.870+0000 7f861b400640 1 --2- 172.21.15.28:0/2418951489 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f117d080 0x5601f1181210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.870+0000 7f861b400640 1 -- 172.21.15.28:0/2418951489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f11729b0 msgr2=0x5601f1180ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:29.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.870+0000 7f861b400640 1 --2- 172.21.15.28:0/2418951489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f11729b0 0x5601f1180ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.870+0000 7f861b400640 1 -- 172.21.15.28:0/2418951489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5601f11f2650 con 0x5601f1172090 2024-04-24T08:10:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.870+0000 7f861aa00640 1 --2- 172.21.15.28:0/2418951489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f11729b0 0x5601f1180ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.870+0000 7f861b400640 1 --2- 172.21.15.28:0/2418951489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f1172090 0x5601f1185760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f860c005cc0 tx=0x7f860c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9755fc06bf09eb1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.870+0000 7f8605600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:10:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.870+0000 7f8605600640 1 -- 172.21.15.28:0/2418951489 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f860c031cb0 con 0x5601f1172090 2024-04-24T08:10:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.870+0000 7f8605600640 1 -- 172.21.15.28:0/2418951489 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f860c0543f0 con 0x5601f1172090 2024-04-24T08:10:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.870+0000 7f86257b9740 1 -- 172.21.15.28:0/2418951489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f860c05a070 con 0x5601f1172090 2024-04-24T08:10:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.871+0000 7f8605600640 1 -- 172.21.15.28:0/2418951489 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f860c031e50 con 0x5601f1172090 2024-04-24T08:10:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.871+0000 7f86257b9740 10 client.15306 fetch_fsmap learned FSMap version 24 2024-04-24T08:10:29.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.871+0000 7f86257b9740 1 -- 172.21.15.28:0/2418951489 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5601f11f37f0 con 0x5601f1172090 2024-04-24T08:10:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.871+0000 7f8605600640 1 -- 172.21.15.28:0/2418951489 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f860c05a070 con 0x5601f1172090 2024-04-24T08:10:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.871+0000 7f8605600640 1 -- 172.21.15.28:0/2418951489 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f860c03d680 con 0x5601f1172090 2024-04-24T08:10:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.871+0000 7f86257b9740 10 client.15306 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:10:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.871+0000 7f86257b9740 10 client.15306 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:10:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.871+0000 7f86257b9740 20 client.15306 populate_metadata read hostname 'smithi028' 2024-04-24T08:10:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.871+0000 7f86257b9740 1 --2- 172.21.15.28:0/2418951489 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5601f11f3ee0 0x5601f11f6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.871+0000 7f86257b9740 4 client.15306 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:10:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.871+0000 7f86257b9740 1 -- 172.21.15.28:0/2418951489 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5601f10df700 con 0x5601f11f3ee0 2024-04-24T08:10:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.872+0000 7f861aa00640 1 --2- 172.21.15.28:0/2418951489 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5601f11f3ee0 0x5601f11f6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:29.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.872+0000 7f861aa00640 1 --2- 172.21.15.28:0/2418951489 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5601f11f3ee0 0x5601f11f6300 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c69756765900bc4d server_cookie=cd5a425b8c0f6414 in_seq=0 out_seq=0 2024-04-24T08:10:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.872+0000 7f8605600640 10 client.15306 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:10:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.873+0000 7f8605600640 1 -- 172.21.15.28:0/2418951489 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5601f10df700 con 0x5601f11f3ee0 2024-04-24T08:10:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.873+0000 7f8605600640 10 client.15306 handle_command_reply: tid=0 2024-04-24T08:10:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.873+0000 7f86257b9740 1 client.15306 shutdown 2024-04-24T08:10:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.874+0000 7f86257b9740 1 -- 172.21.15.28:0/2418951489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f1172090 msgr2=0x5601f1185760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.874+0000 7f86257b9740 1 --2- 172.21.15.28:0/2418951489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f1172090 0x5601f1185760 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f860c005cc0 tx=0x7f860c009260 comp rx=0 tx=0).stop 2024-04-24T08:10:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.874+0000 7f86257b9740 1 -- 172.21.15.28:0/2418951489 shutdown_connections 2024-04-24T08:10:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.874+0000 7f86257b9740 1 --2- 172.21.15.28:0/2418951489 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5601f11f3ee0 0x5601f11f6300 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.874+0000 7f86257b9740 1 --2- 172.21.15.28:0/2418951489 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5601f117d080 0x5601f1181210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.874+0000 7f86257b9740 1 --2- 172.21.15.28:0/2418951489 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5601f11729b0 0x5601f1180ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.874+0000 7f86257b9740 1 --2- 172.21.15.28:0/2418951489 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5601f1172090 0x5601f1185760 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.874+0000 7f86257b9740 1 -- 172.21.15.28:0/2418951489 >> 172.21.15.28:0/2418951489 conn(0x5601f116e3b0 msgr2=0x5601f117b860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.874+0000 7f8605600640 0 client.15306 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:10:29.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.874+0000 7f86257b9740 1 -- 172.21.15.28:0/2418951489 shutdown_connections 2024-04-24T08:10:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.874+0000 7f86257b9740 1 -- 172.21.15.28:0/2418951489 wait complete. 2024-04-24T08:10:29.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.874+0000 7f86257b9740 20 client.15306 trim_cache size 0 max 16384 2024-04-24T08:10:29.878 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:10:29.878 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-24T08:10:29.878 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "04ced641-0ea9-497b-a30e-ebbc1064dee3", 2024-04-24T08:10:29.878 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-24T08:10:29.878 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:10:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.875+0000 7f8622800640 1 -- 172.21.15.28:0/1054038540 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f86000a3ec0 msgr2=0x7f86000a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.875+0000 7f8622800640 1 --2- 172.21.15.28:0/1054038540 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f86000a3ec0 0x7f86000a6380 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x5601f1173650 tx=0x7f860c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.875+0000 7f8622800640 1 -- 172.21.15.28:0/1054038540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f861c109580 msgr2=0x7f861c1a0dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.875+0000 7f8622800640 1 --2- 172.21.15.28:0/1054038540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f861c109580 0x7f861c1a0dc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f860800e9c0 tx=0x7f860800ee80 comp rx=0 tx=0).stop 2024-04-24T08:10:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.875+0000 7f8622800640 1 -- 172.21.15.28:0/1054038540 shutdown_connections 2024-04-24T08:10:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.875+0000 7f8622800640 1 --2- 172.21.15.28:0/1054038540 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f86000a3ec0 0x7f86000a6380 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.875+0000 7f8622800640 1 --2- 172.21.15.28:0/1054038540 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f861c19aa30 0x7f861c19aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.875+0000 7f8622800640 1 --2- 172.21.15.28:0/1054038540 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f861c109580 0x7f861c1a0dc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.875+0000 7f8622800640 1 --2- 172.21.15.28:0/1054038540 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f861c1044b0 0x7f861c1a0880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.875+0000 7f8622800640 1 -- 172.21.15.28:0/1054038540 >> 172.21.15.28:0/1054038540 conn(0x7f861c0ffd90 msgr2=0x7f861c1020e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.876+0000 7f8622800640 1 -- 172.21.15.28:0/1054038540 shutdown_connections 2024-04-24T08:10:29.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:29.876+0000 7f8622800640 1 -- 172.21.15.28:0/1054038540 wait complete. 2024-04-24T08:10:29.895 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:10:29.895 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:10:30.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.073+0000 7f6a3d800640 1 -- 172.21.15.28:0/1813557152 --> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) -- 0x7f6a38062440 con 0x7f6a1c0a3e60 2024-04-24T08:10:30.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.090+0000 7f9495600640 1 -- 172.21.15.28:0/2757827916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94901093e0 msgr2=0x7f9490110fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.090+0000 7f9495600640 1 --2- 172.21.15.28:0/2757827916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94901093e0 0x7f9490110fb0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f9488009ca0 tx=0x7f948802a8c0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.090+0000 7f9495600640 1 -- 172.21.15.28:0/2757827916 shutdown_connections 2024-04-24T08:10:30.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.090+0000 7f9495600640 1 --2- 172.21.15.28:0/2757827916 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f94901093e0 0x7f9490110fb0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.090+0000 7f9495600640 1 --2- 172.21.15.28:0/2757827916 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9490104cc0 0x7f9490108d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.090+0000 7f9495600640 1 --2- 172.21.15.28:0/2757827916 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490104310 0x7f94901046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.090+0000 7f9495600640 1 -- 172.21.15.28:0/2757827916 >> 172.21.15.28:0/2757827916 conn(0x7f94900ffb70 msgr2=0x7f9490101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:30.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.090+0000 7f9495600640 1 -- 172.21.15.28:0/2757827916 shutdown_connections 2024-04-24T08:10:30.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.090+0000 7f9495600640 1 -- 172.21.15.28:0/2757827916 wait complete. 2024-04-24T08:10:30.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.091+0000 7f9495600640 1 Processor -- start 2024-04-24T08:10:30.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.091+0000 7f9495600640 1 -- start start 2024-04-24T08:10:30.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.092+0000 7f9495600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490104310 0x7f94901a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.092+0000 7f9495600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f94901093e0 0x7f94901a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.092+0000 7f9495600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f949019a750 0x7f949019abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.092+0000 7f9495600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94901144d0 con 0x7f9490104310 2024-04-24T08:10:30.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.092+0000 7f9495600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9490114350 con 0x7f949019a750 2024-04-24T08:10:30.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.092+0000 7f9495600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9490114650 con 0x7f94901093e0 2024-04-24T08:10:30.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.092+0000 7f948fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490104310 0x7f94901a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.092+0000 7f948fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490104310 0x7f94901a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55524/0 (socket says 172.21.15.28:55524) 2024-04-24T08:10:30.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.092+0000 7f948fe00640 1 -- 172.21.15.28:0/2306335844 learned_addr learned my addr 172.21.15.28:0/2306335844 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:30.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.092+0000 7f948f400640 1 --2- 172.21.15.28:0/2306335844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f94901093e0 0x7f94901a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.092+0000 7f9494c00640 1 --2- 172.21.15.28:0/2306335844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f949019a750 0x7f949019abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.093+0000 7f948fe00640 1 -- 172.21.15.28:0/2306335844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f94901093e0 msgr2=0x7f94901a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.093+0000 7f948fe00640 1 --2- 172.21.15.28:0/2306335844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f94901093e0 0x7f94901a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.093+0000 7f948fe00640 1 -- 172.21.15.28:0/2306335844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f949019a750 msgr2=0x7f949019abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.093+0000 7f948fe00640 1 --2- 172.21.15.28:0/2306335844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f949019a750 0x7f949019abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.093+0000 7f948fe00640 1 -- 172.21.15.28:0/2306335844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f949019b350 con 0x7f9490104310 2024-04-24T08:10:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.093+0000 7f948f400640 1 --2- 172.21.15.28:0/2306335844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f94901093e0 0x7f94901a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:30.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.093+0000 7f948fe00640 1 --2- 172.21.15.28:0/2306335844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490104310 0x7f94901a0690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f948000e9f0 tx=0x7f948000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:30.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.094+0000 7f948cc00640 1 -- 172.21.15.28:0/2306335844 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948000cd80 con 0x7f9490104310 2024-04-24T08:10:30.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.094+0000 7f948cc00640 1 -- 172.21.15.28:0/2306335844 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9480010910 con 0x7f9490104310 2024-04-24T08:10:30.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.094+0000 7f948cc00640 1 -- 172.21.15.28:0/2306335844 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480018990 con 0x7f9490104310 2024-04-24T08:10:30.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.094+0000 7f9495600640 1 -- 172.21.15.28:0/2306335844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f94901a7430 con 0x7f9490104310 2024-04-24T08:10:30.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.094+0000 7f9495600640 1 -- 172.21.15.28:0/2306335844 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f94901a77e0 con 0x7f9490104310 2024-04-24T08:10:30.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.101+0000 7f948cc00640 1 -- 172.21.15.28:0/2306335844 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7f9480010ac0 con 0x7f9490104310 2024-04-24T08:10:30.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.101+0000 7f9499bcf740 1 Processor -- start 2024-04-24T08:10:30.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.102+0000 7f9499bcf740 1 -- start start 2024-04-24T08:10:30.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.102+0000 7f9499bcf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c821a30 0x56427c821e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.102+0000 7f9499bcf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c822350 0x56427c82c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.102+0000 7f9499bcf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c82ca20 0x56427c82ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.102+0000 7f9499bcf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56427c7871f0 con 0x56427c82ca20 2024-04-24T08:10:30.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.102+0000 7f9499bcf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56427c72d350 con 0x56427c822350 2024-04-24T08:10:30.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.102+0000 7f9499bcf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56427c7831b0 con 0x56427c821a30 2024-04-24T08:10:30.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.102+0000 7f948fe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c822350 0x56427c82c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.102+0000 7f9494c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c821a30 0x56427c821e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.102+0000 7f948fe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c822350 0x56427c82c4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:38918/0 (socket says 172.21.15.28:38918) 2024-04-24T08:10:30.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.102+0000 7f948fe00640 1 -- 172.21.15.28:0/3216162220 learned_addr learned my addr 172.21.15.28:0/3216162220 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:30.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.102+0000 7f948f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c82ca20 0x56427c82ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.102+0000 7f9494c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c821a30 0x56427c821e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:39754/0 (socket says 172.21.15.28:39754) 2024-04-24T08:10:30.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.103+0000 7f948fe00640 1 -- 172.21.15.28:0/3216162220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c821a30 msgr2=0x56427c821e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:30.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.103+0000 7f948fe00640 1 --2- 172.21.15.28:0/3216162220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c821a30 0x56427c821e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.103+0000 7f948fe00640 1 -- 172.21.15.28:0/3216162220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c82ca20 msgr2=0x56427c82ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:30.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.103+0000 7f948fe00640 1 --2- 172.21.15.28:0/3216162220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c82ca20 0x56427c82ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.103+0000 7f948fe00640 1 -- 172.21.15.28:0/3216162220 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56427c781f50 con 0x56427c822350 2024-04-24T08:10:30.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.103+0000 7f948cc00640 1 --2- 172.21.15.28:0/2306335844 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f94740a3e40 0x7f94740a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.103+0000 7f948fe00640 1 --2- 172.21.15.28:0/3216162220 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c822350 0x56427c82c4e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f94800ce4b0 tx=0x7f94800d9570 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4069a4adadcfab52 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:30.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.104+0000 7f948cc00640 1 -- 172.21.15.28:0/2306335844 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f94800d0df0 con 0x7f9490104310 2024-04-24T08:10:30.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.104+0000 7f9485200640 1 -- 172.21.15.28:0/3216162220 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9480098e70 con 0x56427c822350 2024-04-24T08:10:30.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.104+0000 7f9485200640 1 -- 172.21.15.28:0/3216162220 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f948009de50 con 0x56427c822350 2024-04-24T08:10:30.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.104+0000 7f9485200640 1 -- 172.21.15.28:0/3216162220 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948009c530 con 0x56427c822350 2024-04-24T08:10:30.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.104+0000 7f9494c00640 1 --2- 172.21.15.28:0/2306335844 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f94740a3e40 0x7f94740a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.105+0000 7f9499bcf740 1 -- 172.21.15.28:0/3216162220 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c822350 msgr2=0x56427c82c4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.105+0000 7f9499bcf740 1 --2- 172.21.15.28:0/3216162220 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c822350 0x56427c82c4e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f94800ce4b0 tx=0x7f94800d9570 comp rx=0 tx=0).stop 2024-04-24T08:10:30.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.105+0000 7f9494c00640 1 --2- 172.21.15.28:0/2306335844 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f94740a3e40 0x7f94740a6300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9488006010 tx=0x7f9488005d80 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:30.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.105+0000 7f9499bcf740 1 -- 172.21.15.28:0/3216162220 shutdown_connections 2024-04-24T08:10:30.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.105+0000 7f9499bcf740 1 --2- 172.21.15.28:0/3216162220 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c82ca20 0x56427c82ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.105+0000 7f9499bcf740 1 --2- 172.21.15.28:0/3216162220 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c822350 0x56427c82c4e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.105+0000 7f9499bcf740 1 --2- 172.21.15.28:0/3216162220 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c821a30 0x56427c821e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.105+0000 7f9499bcf740 1 -- 172.21.15.28:0/3216162220 >> 172.21.15.28:0/3216162220 conn(0x56427c81dd50 msgr2=0x56427c81e130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:30.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.105+0000 7f9499bcf740 1 -- 172.21.15.28:0/3216162220 shutdown_connections 2024-04-24T08:10:30.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.105+0000 7f9499bcf740 1 -- 172.21.15.28:0/3216162220 wait complete. 2024-04-24T08:10:30.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.105+0000 7f9499bcf740 1 -- start start 2024-04-24T08:10:30.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.106+0000 7f9499bcf740 1 Processor -- start 2024-04-24T08:10:30.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.106+0000 7f9499bcf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c821a30 0x56427c829a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.106+0000 7f9499bcf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c822350 0x56427c829f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.106+0000 7f9499bcf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c82ca20 0x56427c82a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.106+0000 7f9499bcf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56427c82f810 con 0x56427c822350 2024-04-24T08:10:30.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.106+0000 7f9499bcf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56427c740690 con 0x56427c821a30 2024-04-24T08:10:30.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.106+0000 7f9499bcf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56427c82f990 con 0x56427c82ca20 2024-04-24T08:10:30.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.106+0000 7f9499bcf740 10 client.0 _ll_register_callbacks cb 0x56427c77c650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:10:30.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.107+0000 7f948f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c822350 0x56427c829f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.107+0000 7f9494c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c82ca20 0x56427c82a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.107+0000 7f948f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c822350 0x56427c829f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55540/0 (socket says 172.21.15.28:55540) 2024-04-24T08:10:30.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.107+0000 7f948f400640 1 -- 172.21.15.28:0/349899805 learned_addr learned my addr 172.21.15.28:0/349899805 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:30.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.107+0000 7f948fe00640 1 --2- 172.21.15.28:0/349899805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c821a30 0x56427c829a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.107+0000 7f948fe00640 1 -- 172.21.15.28:0/349899805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c82ca20 msgr2=0x56427c82a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:30.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.108+0000 7f948fe00640 1 --2- 172.21.15.28:0/349899805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c82ca20 0x56427c82a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.108+0000 7f948fe00640 1 -- 172.21.15.28:0/349899805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c822350 msgr2=0x56427c829f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:30.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.108+0000 7f948fe00640 1 --2- 172.21.15.28:0/349899805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c822350 0x56427c829f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.108+0000 7f948fe00640 1 -- 172.21.15.28:0/349899805 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56427c73f920 con 0x56427c821a30 2024-04-24T08:10:30.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.108+0000 7f948fe00640 1 --2- 172.21.15.28:0/349899805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c821a30 0x56427c829a30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f948000e9c0 tx=0x7f9480009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11534e4ea327219d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:30.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.108+0000 7f9479600640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:10:30.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.108+0000 7f9479600640 1 -- 172.21.15.28:0/349899805 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948009c710 con 0x56427c821a30 2024-04-24T08:10:30.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.108+0000 7f9479600640 1 -- 172.21.15.28:0/349899805 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f9480099240 con 0x56427c821a30 2024-04-24T08:10:30.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.108+0000 7f9499bcf740 1 -- 172.21.15.28:0/349899805 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f948009de50 con 0x56427c821a30 2024-04-24T08:10:30.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.109+0000 7f9479600640 1 -- 172.21.15.28:0/349899805 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f948009c9a0 con 0x56427c821a30 2024-04-24T08:10:30.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.109+0000 7f9499bcf740 10 client.25418 fetch_fsmap learned FSMap version 24 2024-04-24T08:10:30.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.109+0000 7f9499bcf740 1 -- 172.21.15.28:0/349899805 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56427c5637d0 con 0x56427c821a30 2024-04-24T08:10:30.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.109+0000 7f9479600640 1 -- 172.21.15.28:0/349899805 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f948009de50 con 0x56427c821a30 2024-04-24T08:10:30.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.109+0000 7f9479600640 1 -- 172.21.15.28:0/349899805 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f94800bbce0 con 0x56427c821a30 2024-04-24T08:10:30.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.109+0000 7f9499bcf740 10 client.25418 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:10:30.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.109+0000 7f9499bcf740 10 client.25418 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:10:30.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.109+0000 7f9499bcf740 20 client.25418 populate_metadata read hostname 'smithi028' 2024-04-24T08:10:30.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.109+0000 7f9499bcf740 1 --2- 172.21.15.28:0/349899805 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56427c8454d0 0x56427c8478c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.109+0000 7f9499bcf740 4 client.25418 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:10:30.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.109+0000 7f9499bcf740 1 -- 172.21.15.28:0/349899805 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56427c794c10 con 0x56427c8454d0 2024-04-24T08:10:30.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.110+0000 7f948f400640 1 --2- 172.21.15.28:0/349899805 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56427c8454d0 0x56427c8478c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.110+0000 7f948f400640 1 --2- 172.21.15.28:0/349899805 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56427c8454d0 0x56427c8478c0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a7ffe5de82b6712 server_cookie=b440f7e8d9a6aa70 in_seq=0 out_seq=0 2024-04-24T08:10:30.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.110+0000 7f9479600640 10 client.25418 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:10:30.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.111+0000 7f9479600640 1 -- 172.21.15.28:0/349899805 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56427c794c10 con 0x56427c8454d0 2024-04-24T08:10:30.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.111+0000 7f9479600640 10 client.25418 handle_command_reply: tid=0 2024-04-24T08:10:30.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.112+0000 7f9499bcf740 1 client.25418 shutdown 2024-04-24T08:10:30.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.112+0000 7f9499bcf740 1 -- 172.21.15.28:0/349899805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c821a30 msgr2=0x56427c829a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.112+0000 7f9499bcf740 1 --2- 172.21.15.28:0/349899805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c821a30 0x56427c829a30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f948000e9c0 tx=0x7f9480009310 comp rx=0 tx=0).stop 2024-04-24T08:10:30.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.112+0000 7f9499bcf740 1 -- 172.21.15.28:0/349899805 shutdown_connections 2024-04-24T08:10:30.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.112+0000 7f9499bcf740 1 --2- 172.21.15.28:0/349899805 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56427c8454d0 0x56427c8478c0 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.112+0000 7f9499bcf740 1 --2- 172.21.15.28:0/349899805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c822350 0x56427c829f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.112+0000 7f9499bcf740 1 --2- 172.21.15.28:0/349899805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c82ca20 0x56427c82a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.112+0000 7f9499bcf740 1 --2- 172.21.15.28:0/349899805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c821a30 0x56427c829a30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.112+0000 7f9499bcf740 1 -- 172.21.15.28:0/349899805 >> 172.21.15.28:0/349899805 conn(0x56427c81dd50 msgr2=0x56427c81cfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:30.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.112+0000 7f9479600640 0 client.25418 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:10:30.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.112+0000 7f9499bcf740 1 -- 172.21.15.28:0/349899805 shutdown_connections 2024-04-24T08:10:30.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.113+0000 7f9499bcf740 1 -- 172.21.15.28:0/349899805 wait complete. 2024-04-24T08:10:30.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.113+0000 7f9499bcf740 20 client.25418 trim_cache size 0 max 16384 2024-04-24T08:10:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.141+0000 7f9499bcf740 1 Processor -- start 2024-04-24T08:10:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.141+0000 7f9499bcf740 1 -- start start 2024-04-24T08:10:30.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.141+0000 7f9499bcf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c821a30 0x56427c828aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.141+0000 7f9499bcf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c822350 0x56427c824190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.141+0000 7f9499bcf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c82ca20 0x56427c81ce50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.141+0000 7f9499bcf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94901ae360 con 0x56427c821a30 2024-04-24T08:10:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.141+0000 7f9499bcf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f94901abc30 con 0x56427c82ca20 2024-04-24T08:10:30.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.141+0000 7f9499bcf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f94901a9ae0 con 0x56427c822350 2024-04-24T08:10:30.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.142+0000 7f948f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c822350 0x56427c824190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.142+0000 7f948fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c821a30 0x56427c828aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.142+0000 7f948f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c822350 0x56427c824190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:39766/0 (socket says 172.21.15.28:39766) 2024-04-24T08:10:30.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.142+0000 7f948f400640 1 -- 172.21.15.28:0/2967379285 learned_addr learned my addr 172.21.15.28:0/2967379285 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:30.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.142+0000 7f9494c00640 1 --2- 172.21.15.28:0/2967379285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c82ca20 0x56427c81ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.142+0000 7f948f400640 1 -- 172.21.15.28:0/2967379285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c82ca20 msgr2=0x56427c81ce50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:30.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.142+0000 7f948f400640 1 --2- 172.21.15.28:0/2967379285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c82ca20 0x56427c81ce50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.142+0000 7f948f400640 1 -- 172.21.15.28:0/2967379285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c821a30 msgr2=0x56427c828aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:30.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.142+0000 7f948f400640 1 --2- 172.21.15.28:0/2967379285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c821a30 0x56427c828aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.142+0000 7f948f400640 1 -- 172.21.15.28:0/2967379285 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56427c4993f0 con 0x56427c822350 2024-04-24T08:10:30.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.142+0000 7f948f400640 1 --2- 172.21.15.28:0/2967379285 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c822350 0x56427c824190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f947c012660 tx=0x7f947c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11b27769f0b21672 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:30.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.143+0000 7f947be00640 1 -- 172.21.15.28:0/2967379285 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c025020 con 0x56427c822350 2024-04-24T08:10:30.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.143+0000 7f947be00640 1 -- 172.21.15.28:0/2967379285 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f947c012c20 con 0x56427c822350 2024-04-24T08:10:30.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.143+0000 7f947be00640 1 -- 172.21.15.28:0/2967379285 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f947c0038e0 con 0x56427c822350 2024-04-24T08:10:30.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.144+0000 7f9499bcf740 1 -- 172.21.15.28:0/2967379285 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c822350 msgr2=0x56427c824190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.144+0000 7f9499bcf740 1 --2- 172.21.15.28:0/2967379285 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c822350 0x56427c824190 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f947c012660 tx=0x7f947c005f90 comp rx=0 tx=0).stop 2024-04-24T08:10:30.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.144+0000 7f9499bcf740 1 -- 172.21.15.28:0/2967379285 shutdown_connections 2024-04-24T08:10:30.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.144+0000 7f9499bcf740 1 --2- 172.21.15.28:0/2967379285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c82ca20 0x56427c81ce50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.144+0000 7f9499bcf740 1 --2- 172.21.15.28:0/2967379285 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c822350 0x56427c824190 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.144+0000 7f9499bcf740 1 --2- 172.21.15.28:0/2967379285 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c821a30 0x56427c828aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.144+0000 7f9499bcf740 1 -- 172.21.15.28:0/2967379285 >> 172.21.15.28:0/2967379285 conn(0x56427c81dd50 msgr2=0x56427c81c740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:30.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.144+0000 7f9499bcf740 1 -- 172.21.15.28:0/2967379285 shutdown_connections 2024-04-24T08:10:30.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.144+0000 7f9499bcf740 1 -- 172.21.15.28:0/2967379285 wait complete. 2024-04-24T08:10:30.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.144+0000 7f9499bcf740 1 -- start start 2024-04-24T08:10:30.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.145+0000 7f9499bcf740 1 Processor -- start 2024-04-24T08:10:30.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.145+0000 7f9499bcf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c821a30 0x56427c848610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.145+0000 7f9499bcf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c822350 0x56427c848b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.146+0000 7f9499bcf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c82ca20 0x56427c849090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.146+0000 7f9499bcf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f94901aa130 con 0x56427c82ca20 2024-04-24T08:10:30.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.146+0000 7f9499bcf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f94901ab560 con 0x56427c822350 2024-04-24T08:10:30.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.146+0000 7f9499bcf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f94901aa7c0 con 0x56427c821a30 2024-04-24T08:10:30.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.146+0000 7f9499bcf740 10 client.0 _ll_register_callbacks cb 0x7f9480094020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:10:30.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.146+0000 7f948fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c821a30 0x56427c848610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.146+0000 7f948fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c821a30 0x56427c848610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:39778/0 (socket says 172.21.15.28:39778) 2024-04-24T08:10:30.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.146+0000 7f948fe00640 1 -- 172.21.15.28:0/3641865086 learned_addr learned my addr 172.21.15.28:0/3641865086 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:30.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.146+0000 7f948f400640 1 --2- 172.21.15.28:0/3641865086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c822350 0x56427c848b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.146+0000 7f9494c00640 1 --2- 172.21.15.28:0/3641865086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c82ca20 0x56427c849090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.146+0000 7f948fe00640 1 -- 172.21.15.28:0/3641865086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c822350 msgr2=0x56427c848b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:30.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.146+0000 7f948fe00640 1 --2- 172.21.15.28:0/3641865086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c822350 0x56427c848b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.146+0000 7f948fe00640 1 -- 172.21.15.28:0/3641865086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c82ca20 msgr2=0x56427c849090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:10:30.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.146+0000 7f948fe00640 1 --2- 172.21.15.28:0/3641865086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c82ca20 0x56427c849090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.147+0000 7f948fe00640 1 -- 172.21.15.28:0/3641865086 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56427c4590f0 con 0x56427c821a30 2024-04-24T08:10:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.147+0000 7f948f400640 1 --2- 172.21.15.28:0/3641865086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c822350 0x56427c848b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.147+0000 7f948fe00640 1 --2- 172.21.15.28:0/3641865086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c821a30 0x56427c848610 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f948000c6a0 tx=0x7f94800d9570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3eefe2d4f26b546d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.147+0000 7f9479600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:10:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.148+0000 7f9479600640 1 -- 172.21.15.28:0/3641865086 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f948009c710 con 0x56427c821a30 2024-04-24T08:10:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.148+0000 7f9479600640 1 -- 172.21.15.28:0/3641865086 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f9480099240 con 0x56427c821a30 2024-04-24T08:10:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.148+0000 7f9499bcf740 1 -- 172.21.15.28:0/3641865086 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f947c012c20 con 0x56427c821a30 2024-04-24T08:10:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.148+0000 7f9479600640 1 -- 172.21.15.28:0/3641865086 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f948009c530 con 0x56427c821a30 2024-04-24T08:10:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.148+0000 7f9499bcf740 10 client.25456 fetch_fsmap learned FSMap version 24 2024-04-24T08:10:30.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.148+0000 7f9499bcf740 1 -- 172.21.15.28:0/3641865086 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56427c74c810 con 0x56427c821a30 2024-04-24T08:10:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.148+0000 7f9479600640 1 -- 172.21.15.28:0/3641865086 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f947c012c20 con 0x56427c821a30 2024-04-24T08:10:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.149+0000 7f9479600640 1 -- 172.21.15.28:0/3641865086 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f94800bbce0 con 0x56427c821a30 2024-04-24T08:10:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.149+0000 7f9499bcf740 10 client.25456 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:10:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.149+0000 7f9499bcf740 10 client.25456 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:10:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.149+0000 7f9499bcf740 20 client.25456 populate_metadata read hostname 'smithi028' 2024-04-24T08:10:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.149+0000 7f9499bcf740 1 --2- 172.21.15.28:0/3641865086 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56427c8abd90 0x56427c8ae180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.149+0000 7f9499bcf740 4 client.25456 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:10:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.149+0000 7f9499bcf740 1 -- 172.21.15.28:0/3641865086 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56427c78ea80 con 0x56427c8abd90 2024-04-24T08:10:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.149+0000 7f948f400640 1 --2- 172.21.15.28:0/3641865086 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56427c8abd90 0x56427c8ae180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.150+0000 7f948f400640 1 --2- 172.21.15.28:0/3641865086 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56427c8abd90 0x56427c8ae180 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3a8d4d103b4877a1 server_cookie=5443c62089fb43ed in_seq=0 out_seq=0 2024-04-24T08:10:30.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.150+0000 7f9479600640 10 client.25456 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:10:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.150+0000 7f9479600640 1 -- 172.21.15.28:0/3641865086 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56427c78ea80 con 0x56427c8abd90 2024-04-24T08:10:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.150+0000 7f9479600640 10 client.25456 handle_command_reply: tid=0 2024-04-24T08:10:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.150+0000 7f9499bcf740 1 client.25456 shutdown 2024-04-24T08:10:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.151+0000 7f9499bcf740 1 -- 172.21.15.28:0/3641865086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c821a30 msgr2=0x56427c848610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.151+0000 7f9499bcf740 1 --2- 172.21.15.28:0/3641865086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c821a30 0x56427c848610 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f948000c6a0 tx=0x7f94800d9570 comp rx=0 tx=0).stop 2024-04-24T08:10:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.151+0000 7f9499bcf740 1 -- 172.21.15.28:0/3641865086 shutdown_connections 2024-04-24T08:10:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.151+0000 7f9499bcf740 1 --2- 172.21.15.28:0/3641865086 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56427c8abd90 0x56427c8ae180 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.151+0000 7f9499bcf740 1 --2- 172.21.15.28:0/3641865086 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56427c82ca20 0x56427c849090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.151+0000 7f9499bcf740 1 --2- 172.21.15.28:0/3641865086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56427c822350 0x56427c848b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.151+0000 7f9479600640 0 client.25456 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:10:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.151+0000 7f9499bcf740 1 --2- 172.21.15.28:0/3641865086 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56427c821a30 0x56427c848610 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.151+0000 7f9499bcf740 1 -- 172.21.15.28:0/3641865086 >> 172.21.15.28:0/3641865086 conn(0x56427c81dd50 msgr2=0x56427c82baf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.151+0000 7f9499bcf740 1 -- 172.21.15.28:0/3641865086 shutdown_connections 2024-04-24T08:10:30.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.151+0000 7f9499bcf740 1 -- 172.21.15.28:0/3641865086 wait complete. 2024-04-24T08:10:30.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.151+0000 7f9499bcf740 20 client.25456 trim_cache size 0 max 16384 2024-04-24T08:10:30.155 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:10:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:10:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:10:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: "04ced641-0ea9-497b-a30e-ebbc1064dee3": { 2024-04-24T08:10:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:10:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "04ced641-0ea9-497b-a30e-ebbc1064dee3", 2024-04-24T08:10:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:10:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:10:30.155 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:10:30.155 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:10:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.152+0000 7f9495600640 1 -- 172.21.15.28:0/2306335844 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f94740a3e40 msgr2=0x7f94740a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.152+0000 7f9495600640 1 --2- 172.21.15.28:0/2306335844 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f94740a3e40 0x7f94740a6300 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f9488006010 tx=0x7f9488005d80 comp rx=0 tx=0).stop 2024-04-24T08:10:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.152+0000 7f9495600640 1 -- 172.21.15.28:0/2306335844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490104310 msgr2=0x7f94901a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.152+0000 7f9495600640 1 --2- 172.21.15.28:0/2306335844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490104310 0x7f94901a0690 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f948000e9f0 tx=0x7f948000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.152+0000 7f9495600640 1 -- 172.21.15.28:0/2306335844 shutdown_connections 2024-04-24T08:10:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.152+0000 7f9495600640 1 --2- 172.21.15.28:0/2306335844 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f94740a3e40 0x7f94740a6300 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.152+0000 7f9495600640 1 --2- 172.21.15.28:0/2306335844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f949019a750 0x7f949019abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.152+0000 7f9495600640 1 --2- 172.21.15.28:0/2306335844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f94901093e0 0x7f94901a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.152+0000 7f9495600640 1 --2- 172.21.15.28:0/2306335844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9490104310 0x7f94901a0690 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.152+0000 7f9495600640 1 -- 172.21.15.28:0/2306335844 >> 172.21.15.28:0/2306335844 conn(0x7f94900ffb70 msgr2=0x7f94901005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:30.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.152+0000 7f9495600640 1 -- 172.21.15.28:0/2306335844 shutdown_connections 2024-04-24T08:10:30.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.153+0000 7f9495600640 1 -- 172.21.15.28:0/2306335844 wait complete. 2024-04-24T08:10:30.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.159+0000 7f6a34c00640 1 -- 172.21.15.28:0/1813557152 <== mgr.25225 v2:172.21.15.28:6800/353036813 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7f6a38062440 con 0x7f6a1c0a3e60 2024-04-24T08:10:30.160 INFO:teuthology.orchestra.run.smithi028.stdout:Schedule set for path / 2024-04-24T08:10:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.161+0000 7f6a3d800640 1 -- 172.21.15.28:0/1813557152 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6a1c0a3e60 msgr2=0x7f6a1c0a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.162+0000 7f6a3d800640 1 --2- 172.21.15.28:0/1813557152 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6a1c0a3e60 0x7f6a1c0a6320 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f6a24000c00 tx=0x7f6a24005a40 comp rx=0 tx=0).stop 2024-04-24T08:10:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.162+0000 7f6a3d800640 1 -- 172.21.15.28:0/1813557152 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a38102fc0 msgr2=0x7f6a38100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.162+0000 7f6a3d800640 1 --2- 172.21.15.28:0/1813557152 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a38102fc0 0x7f6a38100c10 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6a2800e9f0 tx=0x7f6a2800eeb0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.162+0000 7f6a3d800640 1 -- 172.21.15.28:0/1813557152 shutdown_connections 2024-04-24T08:10:30.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.162+0000 7f6a3d800640 1 --2- 172.21.15.28:0/1813557152 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6a1c0a3e60 0x7f6a1c0a6320 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.162+0000 7f6a3d800640 1 --2- 172.21.15.28:0/1813557152 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a3810c3e0 0x7f6a38101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.162+0000 7f6a3d800640 1 --2- 172.21.15.28:0/1813557152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a381038e0 0x7f6a38101150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.162+0000 7f6a3d800640 1 --2- 172.21.15.28:0/1813557152 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a38102fc0 0x7f6a38100c10 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.162+0000 7f6a3d800640 1 -- 172.21.15.28:0/1813557152 >> 172.21.15.28:0/1813557152 conn(0x7f6a380fc820 msgr2=0x7f6a380fdfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:30.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.162+0000 7f6a3d800640 1 -- 172.21.15.28:0/1813557152 shutdown_connections 2024-04-24T08:10:30.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.162+0000 7f6a3d800640 1 -- 172.21.15.28:0/1813557152 wait complete. 2024-04-24T08:10:30.173 INFO:tasks.cephfs.filesystem:scrub status for tag:04ced641-0ea9-497b-a30e-ebbc1064dee3 - {'path': '/', 'tag': '04ced641-0ea9-497b-a30e-ebbc1064dee3', 'options': 'recursive,force'} 2024-04-24T08:10:30.184 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-04-24T08:10:30.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.432+0000 7ff367800640 1 -- 172.21.15.28:0/1997578454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff36010e690 msgr2=0x7ff36010ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.432+0000 7ff367800640 1 --2- 172.21.15.28:0/1997578454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff36010e690 0x7ff36010ea70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff350009cd0 tx=0x7ff35002a8f0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.433+0000 7ff367800640 1 -- 172.21.15.28:0/1997578454 shutdown_connections 2024-04-24T08:10:30.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.433+0000 7ff367800640 1 --2- 172.21.15.28:0/1997578454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff360108fb0 0x7ff3601154c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.433+0000 7ff367800640 1 --2- 172.21.15.28:0/1997578454 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3601085f0 0x7ff360108a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.433+0000 7ff367800640 1 --2- 172.21.15.28:0/1997578454 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff36010e690 0x7ff36010ea70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.433+0000 7ff367800640 1 -- 172.21.15.28:0/1997578454 >> 172.21.15.28:0/1997578454 conn(0x7ff360076590 msgr2=0x7ff3600789b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:30.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.433+0000 7ff367800640 1 -- 172.21.15.28:0/1997578454 shutdown_connections 2024-04-24T08:10:30.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.433+0000 7ff367800640 1 -- 172.21.15.28:0/1997578454 wait complete. 2024-04-24T08:10:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.434+0000 7ff367800640 1 Processor -- start 2024-04-24T08:10:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.434+0000 7ff367800640 1 -- start start 2024-04-24T08:10:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.434+0000 7ff367800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3601085f0 0x7ff36019ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.434+0000 7ff367800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff360108fb0 0x7ff36019f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.434+0000 7ff367800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff36010e690 0x7ff3601a36e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.434+0000 7ff367800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff360118a20 con 0x7ff360108fb0 2024-04-24T08:10:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.434+0000 7ff367800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff3601188a0 con 0x7ff3601085f0 2024-04-24T08:10:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.434+0000 7ff367800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff360118ba0 con 0x7ff36010e690 2024-04-24T08:10:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.435+0000 7ff364e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3601085f0 0x7ff36019ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.435+0000 7ff364e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3601085f0 0x7ff36019ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:38964/0 (socket says 172.21.15.28:38964) 2024-04-24T08:10:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.435+0000 7ff364e00640 1 -- 172.21.15.28:0/853504961 learned_addr learned my addr 172.21.15.28:0/853504961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:30.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.435+0000 7ff35fe00640 1 --2- 172.21.15.28:0/853504961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff360108fb0 0x7ff36019f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.435+0000 7ff365800640 1 --2- 172.21.15.28:0/853504961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff36010e690 0x7ff3601a36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.435+0000 7ff364e00640 1 -- 172.21.15.28:0/853504961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff36010e690 msgr2=0x7ff3601a36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.435+0000 7ff364e00640 1 --2- 172.21.15.28:0/853504961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff36010e690 0x7ff3601a36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.435+0000 7ff364e00640 1 -- 172.21.15.28:0/853504961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff360108fb0 msgr2=0x7ff36019f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.435+0000 7ff364e00640 1 --2- 172.21.15.28:0/853504961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff360108fb0 0x7ff36019f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.435+0000 7ff364e00640 1 -- 172.21.15.28:0/853504961 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3601a3e10 con 0x7ff3601085f0 2024-04-24T08:10:30.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.435+0000 7ff35fe00640 1 --2- 172.21.15.28:0/853504961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff360108fb0 0x7ff36019f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:30.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.436+0000 7ff364e00640 1 --2- 172.21.15.28:0/853504961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3601085f0 0x7ff36019ee20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff350000f80 tx=0x7ff350003ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.436+0000 7ff35d600640 1 -- 172.21.15.28:0/853504961 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff350030e40 con 0x7ff3601085f0 2024-04-24T08:10:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.436+0000 7ff367800640 1 -- 172.21.15.28:0/853504961 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3601a40a0 con 0x7ff3601085f0 2024-04-24T08:10:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.436+0000 7ff35d600640 1 -- 172.21.15.28:0/853504961 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff350040070 con 0x7ff3601085f0 2024-04-24T08:10:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.436+0000 7ff35d600640 1 -- 172.21.15.28:0/853504961 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff350031a30 con 0x7ff3601085f0 2024-04-24T08:10:30.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.436+0000 7ff367800640 1 -- 172.21.15.28:0/853504961 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3601ab990 con 0x7ff3601085f0 2024-04-24T08:10:30.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.438+0000 7ff367800640 1 -- 172.21.15.28:0/853504961 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff36004fd90 con 0x7ff3601085f0 2024-04-24T08:10:30.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.439+0000 7ff35d600640 1 -- 172.21.15.28:0/853504961 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7ff350004690 con 0x7ff3601085f0 2024-04-24T08:10:30.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.441+0000 7ff35d600640 1 --2- 172.21.15.28:0/853504961 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3440a3ee0 0x7ff3440a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:30.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.441+0000 7ff35d600640 1 -- 172.21.15.28:0/853504961 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7ff3500f19f0 con 0x7ff3601085f0 2024-04-24T08:10:30.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.441+0000 7ff35fe00640 1 --2- 172.21.15.28:0/853504961 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3440a3ee0 0x7ff3440a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:30.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.442+0000 7ff35fe00640 1 --2- 172.21.15.28:0/853504961 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3440a3ee0 0x7ff3440a63a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff3480095f0 tx=0x7ff348005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:30.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.445+0000 7ff35d600640 1 -- 172.21.15.28:0/853504961 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff3500bded0 con 0x7ff3601085f0 2024-04-24T08:10:30.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:30 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1850954336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:30.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:30 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1850954336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:30.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.679+0000 7ff367800640 1 -- 172.21.15.28:0/853504961 --> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) -- 0x7ff360078220 con 0x7ff3440a3ee0 2024-04-24T08:10:30.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:30 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1850954336' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:10:30.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.820+0000 7ff35d600640 1 -- 172.21.15.28:0/853504961 <== mgr.25225 v2:172.21.15.28:6800/353036813 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+26 (secure 0 0 0) 0x7ff360078220 con 0x7ff3440a3ee0 2024-04-24T08:10:30.821 INFO:teuthology.orchestra.run.smithi028.stdout:Retention added to path / 2024-04-24T08:10:30.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.823+0000 7ff367800640 1 -- 172.21.15.28:0/853504961 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3440a3ee0 msgr2=0x7ff3440a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.823+0000 7ff367800640 1 --2- 172.21.15.28:0/853504961 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3440a3ee0 0x7ff3440a63a0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7ff3480095f0 tx=0x7ff348005f90 comp rx=0 tx=0).stop 2024-04-24T08:10:30.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.823+0000 7ff367800640 1 -- 172.21.15.28:0/853504961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3601085f0 msgr2=0x7ff36019ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:30.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.823+0000 7ff367800640 1 --2- 172.21.15.28:0/853504961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3601085f0 0x7ff36019ee20 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff350000f80 tx=0x7ff350003ea0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.823+0000 7ff367800640 1 -- 172.21.15.28:0/853504961 shutdown_connections 2024-04-24T08:10:30.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.823+0000 7ff367800640 1 --2- 172.21.15.28:0/853504961 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3440a3ee0 0x7ff3440a63a0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.823+0000 7ff367800640 1 --2- 172.21.15.28:0/853504961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff360108fb0 0x7ff36019f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.823+0000 7ff367800640 1 --2- 172.21.15.28:0/853504961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff36010e690 0x7ff3601a36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.823+0000 7ff367800640 1 --2- 172.21.15.28:0/853504961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3601085f0 0x7ff36019ee20 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:30.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.823+0000 7ff367800640 1 -- 172.21.15.28:0/853504961 >> 172.21.15.28:0/853504961 conn(0x7ff360076590 msgr2=0x7ff360077b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:30.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.823+0000 7ff367800640 1 -- 172.21.15.28:0/853504961 shutdown_connections 2024-04-24T08:10:30.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:30.824+0000 7ff367800640 1 -- 172.21.15.28:0/853504961 wait complete. 2024-04-24T08:10:30.845 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-24T08:10:31.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.052+0000 7fa33cc00640 1 -- 172.21.15.28:0/1633590341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa338104c60 msgr2=0x7fa338108ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.052+0000 7fa33cc00640 1 --2- 172.21.15.28:0/1633590341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa338104c60 0x7fa338108ca0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa324009900 tx=0x7fa32402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.053+0000 7fa33cc00640 1 -- 172.21.15.28:0/1633590341 shutdown_connections 2024-04-24T08:10:31.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.053+0000 7fa33cc00640 1 --2- 172.21.15.28:0/1633590341 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa338109380 0x7fa338110f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.053+0000 7fa33cc00640 1 --2- 172.21.15.28:0/1633590341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa338104c60 0x7fa338108ca0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.053+0000 7fa33cc00640 1 --2- 172.21.15.28:0/1633590341 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3381042b0 0x7fa338104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.053+0000 7fa33cc00640 1 -- 172.21.15.28:0/1633590341 >> 172.21.15.28:0/1633590341 conn(0x7fa3380ffb10 msgr2=0x7fa338101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:31.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.053+0000 7fa33cc00640 1 -- 172.21.15.28:0/1633590341 shutdown_connections 2024-04-24T08:10:31.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.053+0000 7fa33cc00640 1 -- 172.21.15.28:0/1633590341 wait complete. 2024-04-24T08:10:31.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.054+0000 7fa33cc00640 1 Processor -- start 2024-04-24T08:10:31.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.054+0000 7fa33cc00640 1 -- start start 2024-04-24T08:10:31.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.055+0000 7fa33cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3381042b0 0x7fa3381a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.055+0000 7fa33cc00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa338109380 0x7fa3381a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.055+0000 7fa33cc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa33819a780 0x7fa33819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.055+0000 7fa33cc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa338114460 con 0x7fa3381042b0 2024-04-24T08:10:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.055+0000 7fa33cc00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa3381142e0 con 0x7fa338109380 2024-04-24T08:10:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.055+0000 7fa33cc00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa3381145e0 con 0x7fa33819a780 2024-04-24T08:10:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.055+0000 7fa337400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3381042b0 0x7fa3381a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.055+0000 7fa336a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa338109380 0x7fa3381a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.055+0000 7fa336a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa338109380 0x7fa3381a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:38986/0 (socket says 172.21.15.28:38986) 2024-04-24T08:10:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.055+0000 7fa336a00640 1 -- 172.21.15.28:0/3461570485 learned_addr learned my addr 172.21.15.28:0/3461570485 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:31.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.055+0000 7fa337e00640 1 --2- 172.21.15.28:0/3461570485 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa33819a780 0x7fa33819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.056+0000 7fa336a00640 1 -- 172.21.15.28:0/3461570485 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa33819a780 msgr2=0x7fa33819ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.056+0000 7fa336a00640 1 --2- 172.21.15.28:0/3461570485 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa33819a780 0x7fa33819ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.056+0000 7fa336a00640 1 -- 172.21.15.28:0/3461570485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3381042b0 msgr2=0x7fa3381a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.056+0000 7fa336a00640 1 --2- 172.21.15.28:0/3461570485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3381042b0 0x7fa3381a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.056+0000 7fa336a00640 1 -- 172.21.15.28:0/3461570485 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa33819b310 con 0x7fa338109380 2024-04-24T08:10:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.056+0000 7fa336a00640 1 --2- 172.21.15.28:0/3461570485 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa338109380 0x7fa3381a0b50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa32402a910 tx=0x7fa324003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.056+0000 7fa32fe00640 1 -- 172.21.15.28:0/3461570485 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa32403f070 con 0x7fa338109380 2024-04-24T08:10:31.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.056+0000 7fa33cc00640 1 -- 172.21.15.28:0/3461570485 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa33819b5a0 con 0x7fa338109380 2024-04-24T08:10:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.056+0000 7fa32fe00640 1 -- 172.21.15.28:0/3461570485 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa324031c90 con 0x7fa338109380 2024-04-24T08:10:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.056+0000 7fa32fe00640 1 -- 172.21.15.28:0/3461570485 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa324035030 con 0x7fa338109380 2024-04-24T08:10:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.056+0000 7fa33cc00640 1 -- 172.21.15.28:0/3461570485 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa3381a7780 con 0x7fa338109380 2024-04-24T08:10:31.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.058+0000 7fa33cc00640 1 -- 172.21.15.28:0/3461570485 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa338105780 con 0x7fa338109380 2024-04-24T08:10:31.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.060+0000 7fa32fe00640 1 -- 172.21.15.28:0/3461570485 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7fa324042070 con 0x7fa338109380 2024-04-24T08:10:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.062+0000 7fa32fe00640 1 --2- 172.21.15.28:0/3461570485 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa31c0a3eb0 0x7fa31c0a6370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.062+0000 7fa32fe00640 1 -- 172.21.15.28:0/3461570485 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fa3240f0c50 con 0x7fa338109380 2024-04-24T08:10:31.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.062+0000 7fa337400640 1 --2- 172.21.15.28:0/3461570485 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa31c0a3eb0 0x7fa31c0a6370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.063+0000 7fa337400640 1 --2- 172.21.15.28:0/3461570485 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa31c0a3eb0 0x7fa31c0a6370 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa328009bd0 tx=0x7fa328005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:31.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.066+0000 7fa32fe00640 1 -- 172.21.15.28:0/3461570485 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa3240bcaa0 con 0x7fa338109380 2024-04-24T08:10:31.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.302+0000 7fa33cc00640 1 -- 172.21.15.28:0/3461570485 --> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7fa33810c620 con 0x7fa31c0a3eb0 2024-04-24T08:10:31.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.319+0000 7fa32fe00640 1 -- 172.21.15.28:0/3461570485 <== mgr.25225 v2:172.21.15.28:6800/353036813 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+298 (secure 0 0 0) 0x7fa32801b3f0 con 0x7fa31c0a3eb0 2024-04-24T08:10:31.321 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-24T00:00:00", "created": "2024-04-24T08:10:30", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-04-24T08:10:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.323+0000 7fa33cc00640 1 -- 172.21.15.28:0/3461570485 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa31c0a3eb0 msgr2=0x7fa31c0a6370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.323+0000 7fa33cc00640 1 --2- 172.21.15.28:0/3461570485 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa31c0a3eb0 0x7fa31c0a6370 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fa328009bd0 tx=0x7fa328005f90 comp rx=0 tx=0).stop 2024-04-24T08:10:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.323+0000 7fa33cc00640 1 -- 172.21.15.28:0/3461570485 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa338109380 msgr2=0x7fa3381a0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.323+0000 7fa33cc00640 1 --2- 172.21.15.28:0/3461570485 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa338109380 0x7fa3381a0b50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fa32402a910 tx=0x7fa324003b20 comp rx=0 tx=0).stop 2024-04-24T08:10:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.323+0000 7fa33cc00640 1 -- 172.21.15.28:0/3461570485 shutdown_connections 2024-04-24T08:10:31.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.323+0000 7fa33cc00640 1 --2- 172.21.15.28:0/3461570485 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa31c0a3eb0 0x7fa31c0a6370 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.323+0000 7fa33cc00640 1 --2- 172.21.15.28:0/3461570485 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa33819a780 0x7fa33819ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.323+0000 7fa33cc00640 1 --2- 172.21.15.28:0/3461570485 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa338109380 0x7fa3381a0b50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.323+0000 7fa33cc00640 1 --2- 172.21.15.28:0/3461570485 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa3381042b0 0x7fa3381a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.323+0000 7fa33cc00640 1 -- 172.21.15.28:0/3461570485 >> 172.21.15.28:0/3461570485 conn(0x7fa3380ffb10 msgr2=0x7fa338101e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.323+0000 7fa33cc00640 1 -- 172.21.15.28:0/3461570485 shutdown_connections 2024-04-24T08:10:31.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.323+0000 7fa33cc00640 1 -- 172.21.15.28:0/3461570485 wait complete. 2024-04-24T08:10:31.344 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-24T08:10:31.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:31 smithi028 ceph-mon[27786]: scrub queued for path: / 2024-04-24T08:10:31.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:31 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:10:31.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:31 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:10:31.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:31 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:10:31.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:31 smithi028 ceph-mon[27786]: from='client.25423 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:31.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:31 smithi028 ceph-mon[27786]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 912 B/s wr, 18 op/s 2024-04-24T08:10:31.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.547+0000 7f20e5000640 1 -- 172.21.15.28:0/2342795805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20e0104460 msgr2=0x7f20e0104840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.547+0000 7f20e5000640 1 --2- 172.21.15.28:0/2342795805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20e0104460 0x7f20e0104840 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f20d0009900 tx=0x7f20d002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.548+0000 7f20e5000640 1 -- 172.21.15.28:0/2342795805 shutdown_connections 2024-04-24T08:10:31.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.548+0000 7f20e5000640 1 --2- 172.21.15.28:0/2342795805 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20e0104e10 0x7f20e0108e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.548+0000 7f20e5000640 1 --2- 172.21.15.28:0/2342795805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20e0109530 0x7f20e0111100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.548+0000 7f20e5000640 1 --2- 172.21.15.28:0/2342795805 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20e0104460 0x7f20e0104840 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.548+0000 7f20e5000640 1 -- 172.21.15.28:0/2342795805 >> 172.21.15.28:0/2342795805 conn(0x7f20e00ffd00 msgr2=0x7f20e0102120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:31.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.548+0000 7f20e5000640 1 -- 172.21.15.28:0/2342795805 shutdown_connections 2024-04-24T08:10:31.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.548+0000 7f20e5000640 1 -- 172.21.15.28:0/2342795805 wait complete. 2024-04-24T08:10:31.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.549+0000 7f20e5000640 1 Processor -- start 2024-04-24T08:10:31.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.549+0000 7f20e5000640 1 -- start start 2024-04-24T08:10:31.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.550+0000 7f20e5000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20e0104460 0x7f20e01a06c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.550+0000 7f20e5000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20e0109530 0x7f20e01a0c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.550+0000 7f20e5000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20e019a870 0x7f20e019ad20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.550+0000 7f20e5000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f20e0114610 con 0x7f20e0104460 2024-04-24T08:10:31.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.550+0000 7f20e5000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f20e0114490 con 0x7f20e019a870 2024-04-24T08:10:31.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.550+0000 7f20e5000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f20e0114790 con 0x7f20e0109530 2024-04-24T08:10:31.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.550+0000 7f20dde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20e0104460 0x7f20e01a06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.550+0000 7f20dde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20e0104460 0x7f20e01a06c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55622/0 (socket says 172.21.15.28:55622) 2024-04-24T08:10:31.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.550+0000 7f20dde00640 1 -- 172.21.15.28:0/3833547436 learned_addr learned my addr 172.21.15.28:0/3833547436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:10:31.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.550+0000 7f20de800640 1 --2- 172.21.15.28:0/3833547436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20e019a870 0x7f20e019ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.550+0000 7f20dd400640 1 --2- 172.21.15.28:0/3833547436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20e0109530 0x7f20e01a0c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.550+0000 7f20dde00640 1 -- 172.21.15.28:0/3833547436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20e0109530 msgr2=0x7f20e01a0c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.550+0000 7f20dde00640 1 --2- 172.21.15.28:0/3833547436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20e0109530 0x7f20e01a0c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.550+0000 7f20dde00640 1 -- 172.21.15.28:0/3833547436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20e019a870 msgr2=0x7f20e019ad20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.550+0000 7f20dde00640 1 --2- 172.21.15.28:0/3833547436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20e019a870 0x7f20e019ad20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.550+0000 7f20dde00640 1 -- 172.21.15.28:0/3833547436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20e019b390 con 0x7f20e0104460 2024-04-24T08:10:31.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.550+0000 7f20de800640 1 --2- 172.21.15.28:0/3833547436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20e019a870 0x7f20e019ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:10:31.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.551+0000 7f20dde00640 1 --2- 172.21.15.28:0/3833547436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20e0104460 0x7f20e01a06c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f20d0004770 tx=0x7f20d0003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:31.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.551+0000 7f20d6a00640 1 -- 172.21.15.28:0/3833547436 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d003f070 con 0x7f20e0104460 2024-04-24T08:10:31.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.551+0000 7f20e5000640 1 -- 172.21.15.28:0/3833547436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20e019b620 con 0x7f20e0104460 2024-04-24T08:10:31.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.551+0000 7f20d6a00640 1 -- 172.21.15.28:0/3833547436 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20d0031ca0 con 0x7f20e0104460 2024-04-24T08:10:31.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.551+0000 7f20d6a00640 1 -- 172.21.15.28:0/3833547436 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20d0035030 con 0x7f20e0104460 2024-04-24T08:10:31.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.551+0000 7f20e5000640 1 -- 172.21.15.28:0/3833547436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20e01a7820 con 0x7f20e0104460 2024-04-24T08:10:31.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.553+0000 7f20e5000640 1 -- 172.21.15.28:0/3833547436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20e004fd90 con 0x7f20e0104460 2024-04-24T08:10:31.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.555+0000 7f20d6a00640 1 -- 172.21.15.28:0/3833547436 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7f20d0042070 con 0x7f20e0104460 2024-04-24T08:10:31.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.557+0000 7f20d6a00640 1 --2- 172.21.15.28:0/3833547436 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f20c40a3ee0 0x7f20c40a63a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:10:31.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.557+0000 7f20d6a00640 1 -- 172.21.15.28:0/3833547436 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f20d00f1630 con 0x7f20e0104460 2024-04-24T08:10:31.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.557+0000 7f20dd400640 1 --2- 172.21.15.28:0/3833547436 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f20c40a3ee0 0x7f20c40a63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:10:31.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.558+0000 7f20dd400640 1 --2- 172.21.15.28:0/3833547436 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f20c40a3ee0 0x7f20c40a63a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f20c80099a0 tx=0x7f20c8009500 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:10:31.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.559+0000 7f20d6a00640 1 -- 172.21.15.28:0/3833547436 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f20d00bd540 con 0x7f20e0104460 2024-04-24T08:10:31.591 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:31 smithi194 ceph-mon[32836]: scrub queued for path: / 2024-04-24T08:10:31.591 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:31 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:10:31.592 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:31 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:10:31.592 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:31 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:10:31.592 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:31 smithi194 ceph-mon[32836]: from='client.25423 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:31.592 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:31 smithi194 ceph-mon[32836]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 912 B/s wr, 18 op/s 2024-04-24T08:10:31.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:31 smithi117 ceph-mon[32438]: scrub queued for path: / 2024-04-24T08:10:31.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:31 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:10:31.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:31 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:10:31.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:31 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:10:31.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:31 smithi117 ceph-mon[32438]: from='client.25423 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:31.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:31 smithi117 ceph-mon[32438]: pgmap v8: 97 pgs: 97 active+clean; 1.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 912 B/s wr, 18 op/s 2024-04-24T08:10:31.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.792+0000 7f20e5000640 1 -- 172.21.15.28:0/3833547436 --> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f20e0104e10 con 0x7f20c40a3ee0 2024-04-24T08:10:31.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.807+0000 7f20d6a00640 1 -- 172.21.15.28:0/3833547436 <== mgr.25225 v2:172.21.15.28:6800/353036813 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f20e0104e10 con 0x7f20c40a3ee0 2024-04-24T08:10:31.809 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-04-24T08:10:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.810+0000 7f20e5000640 1 -- 172.21.15.28:0/3833547436 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f20c40a3ee0 msgr2=0x7f20c40a63a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.810+0000 7f20e5000640 1 --2- 172.21.15.28:0/3833547436 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f20c40a3ee0 0x7f20c40a63a0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f20c80099a0 tx=0x7f20c8009500 comp rx=0 tx=0).stop 2024-04-24T08:10:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.810+0000 7f20e5000640 1 -- 172.21.15.28:0/3833547436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20e0104460 msgr2=0x7f20e01a06c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:10:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.810+0000 7f20e5000640 1 --2- 172.21.15.28:0/3833547436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20e0104460 0x7f20e01a06c0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f20d0004770 tx=0x7f20d0003c10 comp rx=0 tx=0).stop 2024-04-24T08:10:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.810+0000 7f20e5000640 1 -- 172.21.15.28:0/3833547436 shutdown_connections 2024-04-24T08:10:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.810+0000 7f20e5000640 1 --2- 172.21.15.28:0/3833547436 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f20c40a3ee0 0x7f20c40a63a0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.810+0000 7f20e5000640 1 --2- 172.21.15.28:0/3833547436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20e019a870 0x7f20e019ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.810+0000 7f20e5000640 1 --2- 172.21.15.28:0/3833547436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f20e0109530 0x7f20e01a0c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.810+0000 7f20e5000640 1 --2- 172.21.15.28:0/3833547436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f20e0104460 0x7f20e01a06c0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:10:31.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.810+0000 7f20e5000640 1 -- 172.21.15.28:0/3833547436 >> 172.21.15.28:0/3833547436 conn(0x7f20e00ffd00 msgr2=0x7f20e0100840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:10:31.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.811+0000 7f20e5000640 1 -- 172.21.15.28:0/3833547436 shutdown_connections 2024-04-24T08:10:31.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:10:31.811+0000 7f20e5000640 1 -- 172.21.15.28:0/3833547436 wait complete. 2024-04-24T08:10:31.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-04-24T08:10:31.925 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-04-24T08:10:31.962 INFO:teuthology.run_tasks:Running task workunit... 2024-04-24T08:10:31.975 INFO:tasks.workunit:Pulling workunits from ref 1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:10:31.976 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-04-24T08:10:31.976 INFO:tasks.workunit:timeout=3h 2024-04-24T08:10:31.976 INFO:tasks.workunit:cleanup=True 2024-04-24T08:10:31.976 DEBUG:teuthology.orchestra.run.smithi028:> stat -- /home/ubuntu/cephtest/mnt.0 2024-04-24T08:10:32.039 INFO:teuthology.orchestra.run.smithi028.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-04-24T08:10:32.039 INFO:teuthology.orchestra.run.smithi028.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-04-24T08:10:32.039 INFO:teuthology.orchestra.run.smithi028.stdout:Device: 72h/114d Inode: 1099511628283 Links: 2 2024-04-24T08:10:32.039 INFO:teuthology.orchestra.run.smithi028.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-04-24T08:10:32.039 INFO:teuthology.orchestra.run.smithi028.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-04-24T08:10:32.039 INFO:teuthology.orchestra.run.smithi028.stdout:Access: 2024-04-24 08:07:59.113137057 +0000 2024-04-24T08:10:32.039 INFO:teuthology.orchestra.run.smithi028.stdout:Modify: 2024-04-24 08:07:59.113137057 +0000 2024-04-24T08:10:32.039 INFO:teuthology.orchestra.run.smithi028.stdout:Change: 2024-04-24 08:10:18.663243012 +0000 2024-04-24T08:10:32.039 INFO:teuthology.orchestra.run.smithi028.stdout: Birth: 2024-04-24 08:07:59.113137057 +0000 2024-04-24T08:10:32.040 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-04-24T08:10:32.040 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-04-24T08:10:32.144 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 1f81fb397ae98da7563d451a78c61574c8f4e6e0 2024-04-24T08:10:32.203 INFO:tasks.workunit.client.0.smithi028.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-04-24T08:10:32.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:32 smithi028 ceph-mon[27786]: from='client.25430 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:32.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:32 smithi028 ceph-mon[27786]: from='client.25436 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:32.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:32 smithi117 ceph-mon[32438]: from='client.25430 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:32.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:32 smithi117 ceph-mon[32438]: from='client.25436 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:32.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:32 smithi194 ceph-mon[32836]: from='client.25430 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:32.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:32 smithi194 ceph-mon[32836]: from='client.25436 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:33.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:33 smithi028 ceph-mon[27786]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:33.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:33 smithi028 ceph-mon[27786]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 49 op/s 2024-04-24T08:10:33.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:33 smithi117 ceph-mon[32438]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:33.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:33 smithi117 ceph-mon[32438]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 49 op/s 2024-04-24T08:10:33.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:33 smithi194 ceph-mon[32836]: from='client.15342 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:10:33.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:33 smithi194 ceph-mon[32836]: pgmap v9: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 84 KiB/s wr, 49 op/s 2024-04-24T08:10:35.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:35 smithi028 ceph-mon[27786]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 45 op/s 2024-04-24T08:10:35.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:35 smithi117 ceph-mon[32438]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 45 op/s 2024-04-24T08:10:35.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:35 smithi194 ceph-mon[32836]: pgmap v10: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 77 KiB/s wr, 45 op/s 2024-04-24T08:10:37.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:37 smithi028 ceph-mon[27786]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-24T08:10:37.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:37 smithi117 ceph-mon[32438]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-24T08:10:37.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:37 smithi194 ceph-mon[32836]: pgmap v11: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 47 op/s 2024-04-24T08:10:39.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:39 smithi028 ceph-mon[27786]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-24T08:10:39.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:39 smithi117 ceph-mon[32438]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-24T08:10:39.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:39 smithi194 ceph-mon[32836]: pgmap v12: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 77 KiB/s wr, 48 op/s 2024-04-24T08:10:41.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:41 smithi028 ceph-mon[27786]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-24T08:10:41.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:41 smithi117 ceph-mon[32438]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-24T08:10:41.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:41 smithi194 ceph-mon[32836]: pgmap v13: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 22 KiB/s rd, 77 KiB/s wr, 37 op/s 2024-04-24T08:10:43.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:43 smithi028 ceph-mon[27786]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-24T08:10:43.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:43 smithi117 ceph-mon[32438]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-24T08:10:43.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:43 smithi194 ceph-mon[32836]: pgmap v14: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 24 KiB/s rd, 77 KiB/s wr, 39 op/s 2024-04-24T08:10:44.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:44 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:10:44.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:44 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:10:44.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:44 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:10:45.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:45 smithi028 ceph-mon[27786]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-04-24T08:10:45.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:45 smithi117 ceph-mon[32438]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-04-24T08:10:45.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:45 smithi194 ceph-mon[32836]: pgmap v15: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 8 op/s 2024-04-24T08:10:47.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:47 smithi028 ceph-mon[27786]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-24T08:10:47.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:47 smithi117 ceph-mon[32438]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-24T08:10:47.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:47 smithi194 ceph-mon[32836]: pgmap v16: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 767 B/s wr, 9 op/s 2024-04-24T08:10:49.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:49 smithi117 ceph-mon[32438]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-24T08:10:49.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:49 smithi194 ceph-mon[32836]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-24T08:10:49.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:49 smithi028 ceph-mon[27786]: pgmap v17: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 255 B/s wr, 8 op/s 2024-04-24T08:10:51.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:51 smithi117 ceph-mon[32438]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:10:51.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:51 smithi194 ceph-mon[32836]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:10:51.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:51 smithi028 ceph-mon[27786]: pgmap v18: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:10:53.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:53 smithi117 ceph-mon[32438]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:10:53.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:53 smithi194 ceph-mon[32836]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:10:53.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:53 smithi028 ceph-mon[27786]: pgmap v19: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:10:55.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:55 smithi117 ceph-mon[32438]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:10:55.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:55 smithi194 ceph-mon[32836]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:10:55.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:55 smithi028 ceph-mon[27786]: pgmap v20: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:10:57.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:57 smithi117 ceph-mon[32438]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-24T08:10:57.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:57 smithi194 ceph-mon[32836]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-24T08:10:57.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:57 smithi028 ceph-mon[27786]: pgmap v21: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-04-24T08:10:59.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:10:59 smithi117 ceph-mon[32438]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:10:59.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:10:59 smithi194 ceph-mon[32836]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:10:59.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:10:59 smithi028 ceph-mon[27786]: pgmap v22: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:11:00.175 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:11:00.175 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:11:00.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.409+0000 7f9701000640 1 -- 172.21.15.28:0/839262004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96fc076e50 msgr2=0x7f96fc1129a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:00.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.409+0000 7f9701000640 1 --2- 172.21.15.28:0/839262004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96fc076e50 0x7f96fc1129a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f96e4009930 tx=0x7f96e402a440 comp rx=0 tx=0).stop 2024-04-24T08:11:00.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.409+0000 7f9701000640 1 -- 172.21.15.28:0/839262004 shutdown_connections 2024-04-24T08:11:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.409+0000 7f9701000640 1 --2- 172.21.15.28:0/839262004 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96fc112ee0 0x7f96fc1152d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.409+0000 7f9701000640 1 --2- 172.21.15.28:0/839262004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96fc076e50 0x7f96fc1129a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.409+0000 7f9701000640 1 --2- 172.21.15.28:0/839262004 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96fc076530 0x7f96fc076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.409+0000 7f9701000640 1 -- 172.21.15.28:0/839262004 >> 172.21.15.28:0/839262004 conn(0x7f96fc0ffd90 msgr2=0x7f96fc1021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.410+0000 7f9701000640 1 -- 172.21.15.28:0/839262004 shutdown_connections 2024-04-24T08:11:00.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.410+0000 7f9701000640 1 -- 172.21.15.28:0/839262004 wait complete. 2024-04-24T08:11:00.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.411+0000 7f9701000640 1 Processor -- start 2024-04-24T08:11:00.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.411+0000 7f9701000640 1 -- start start 2024-04-24T08:11:00.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.411+0000 7f9701000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96fc076530 0x7f96fc19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.411+0000 7f9701000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96fc076e50 0x7f96fc19f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.411+0000 7f9701000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96fc112ee0 0x7f96fc1a3890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.411+0000 7f9701000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96fc118a90 con 0x7f96fc076530 2024-04-24T08:11:00.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.411+0000 7f9701000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f96fc118910 con 0x7f96fc112ee0 2024-04-24T08:11:00.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.411+0000 7f9701000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f96fc118c10 con 0x7f96fc076e50 2024-04-24T08:11:00.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.411+0000 7f96f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96fc076530 0x7f96fc19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.411+0000 7f96f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96fc076530 0x7f96fc19efd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55934/0 (socket says 172.21.15.28:55934) 2024-04-24T08:11:00.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.411+0000 7f96f9e00640 1 -- 172.21.15.28:0/1394215137 learned_addr learned my addr 172.21.15.28:0/1394215137 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:00.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.412+0000 7f96f1400640 1 --2- 172.21.15.28:0/1394215137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96fc076e50 0x7f96fc19f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.412+0000 7f96fa800640 1 --2- 172.21.15.28:0/1394215137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96fc112ee0 0x7f96fc1a3890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.412+0000 7f96f9e00640 1 -- 172.21.15.28:0/1394215137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96fc076e50 msgr2=0x7f96fc19f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.412+0000 7f96f9e00640 1 --2- 172.21.15.28:0/1394215137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96fc076e50 0x7f96fc19f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.412+0000 7f96f9e00640 1 -- 172.21.15.28:0/1394215137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96fc112ee0 msgr2=0x7f96fc1a3890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.412+0000 7f96f9e00640 1 --2- 172.21.15.28:0/1394215137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96fc112ee0 0x7f96fc1a3890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.412+0000 7f96f9e00640 1 -- 172.21.15.28:0/1394215137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f96fc1a3fc0 con 0x7f96fc076530 2024-04-24T08:11:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.412+0000 7f96fa800640 1 --2- 172.21.15.28:0/1394215137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96fc112ee0 0x7f96fc1a3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.412+0000 7f96f9e00640 1 --2- 172.21.15.28:0/1394215137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96fc076530 0x7f96fc19efd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f96ec00e9f0 tx=0x7f96ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.413+0000 7f96f3400640 1 -- 172.21.15.28:0/1394215137 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ec00cd80 con 0x7f96fc076530 2024-04-24T08:11:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.413+0000 7f96f3400640 1 -- 172.21.15.28:0/1394215137 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96ec010910 con 0x7f96fc076530 2024-04-24T08:11:00.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.413+0000 7f9701000640 1 -- 172.21.15.28:0/1394215137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f96fc1a42b0 con 0x7f96fc076530 2024-04-24T08:11:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.413+0000 7f96f3400640 1 -- 172.21.15.28:0/1394215137 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96ec0188f0 con 0x7f96fc076530 2024-04-24T08:11:00.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.413+0000 7f9701000640 1 -- 172.21.15.28:0/1394215137 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96fc1abb40 con 0x7f96fc076530 2024-04-24T08:11:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.416+0000 7f9703dc3740 1 Processor -- start 2024-04-24T08:11:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.416+0000 7f9703dc3740 1 -- start start 2024-04-24T08:11:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.416+0000 7f96f3400640 1 -- 172.21.15.28:0/1394215137 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7f96ec010ac0 con 0x7f96fc076530 2024-04-24T08:11:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.417+0000 7f9703dc3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916a860 0x55785916ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.417+0000 7f9703dc3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55785916b180 0x557859175310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.417+0000 7f9703dc3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557859175850 0x557859177c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.417+0000 7f9703dc3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5578590d01f0 con 0x557859175850 2024-04-24T08:11:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.417+0000 7f9703dc3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557859076350 con 0x55785916b180 2024-04-24T08:11:00.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.417+0000 7f9703dc3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5578590cc1b0 con 0x55785916a860 2024-04-24T08:11:00.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.417+0000 7f96fa800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916a860 0x55785916ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.417+0000 7f96f1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557859175850 0x557859177c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.417+0000 7f96fa800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916a860 0x55785916ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:52894/0 (socket says 172.21.15.28:52894) 2024-04-24T08:11:00.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.417+0000 7f96fa800640 1 -- 172.21.15.28:0/22856502 learned_addr learned my addr 172.21.15.28:0/22856502 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:00.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.417+0000 7f96f9e00640 1 --2- 172.21.15.28:0/22856502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55785916b180 0x557859175310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.417+0000 7f96fa800640 1 -- 172.21.15.28:0/22856502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55785916b180 msgr2=0x557859175310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:00.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.417+0000 7f96fa800640 1 --2- 172.21.15.28:0/22856502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55785916b180 0x557859175310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.417+0000 7f96fa800640 1 -- 172.21.15.28:0/22856502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557859175850 msgr2=0x557859177c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:00.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.417+0000 7f96fa800640 1 --2- 172.21.15.28:0/22856502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557859175850 0x557859177c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.417+0000 7f96fa800640 1 -- 172.21.15.28:0/22856502 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578590caf50 con 0x55785916a860 2024-04-24T08:11:00.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.417+0000 7f96f9e00640 1 --2- 172.21.15.28:0/22856502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55785916b180 0x557859175310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:11:00.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.418+0000 7f96fa800640 1 --2- 172.21.15.28:0/22856502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916a860 0x55785916ac40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f96f4011760 tx=0x7f96f4011c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=91f912ac268fcafe server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:00.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.418+0000 7f96eb400640 1 -- 172.21.15.28:0/22856502 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f4004d20 con 0x55785916a860 2024-04-24T08:11:00.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.418+0000 7f96eb400640 1 -- 172.21.15.28:0/22856502 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96f400f9a0 con 0x55785916a860 2024-04-24T08:11:00.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.418+0000 7f96eb400640 1 -- 172.21.15.28:0/22856502 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96f4004590 con 0x55785916a860 2024-04-24T08:11:00.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.418+0000 7f96f3400640 1 --2- 172.21.15.28:0/1394215137 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f96e00a3ec0 0x7f96e00a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.419+0000 7f96f3400640 1 -- 172.21.15.28:0/1394215137 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f96ec0d0e10 con 0x7f96fc076530 2024-04-24T08:11:00.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.419+0000 7f96f9e00640 1 --2- 172.21.15.28:0/1394215137 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f96e00a3ec0 0x7f96e00a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.419+0000 7f9703dc3740 1 -- 172.21.15.28:0/22856502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916a860 msgr2=0x55785916ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:00.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.419+0000 7f9703dc3740 1 --2- 172.21.15.28:0/22856502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916a860 0x55785916ac40 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f96f4011760 tx=0x7f96f4011c20 comp rx=0 tx=0).stop 2024-04-24T08:11:00.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.419+0000 7f9703dc3740 1 -- 172.21.15.28:0/22856502 shutdown_connections 2024-04-24T08:11:00.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.419+0000 7f9703dc3740 1 --2- 172.21.15.28:0/22856502 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557859175850 0x557859177c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.419+0000 7f9703dc3740 1 --2- 172.21.15.28:0/22856502 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55785916b180 0x557859175310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.419+0000 7f9703dc3740 1 --2- 172.21.15.28:0/22856502 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916a860 0x55785916ac40 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.419+0000 7f9703dc3740 1 -- 172.21.15.28:0/22856502 >> 172.21.15.28:0/22856502 conn(0x557859166b80 msgr2=0x557859166f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:00.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.419+0000 7f9703dc3740 1 -- 172.21.15.28:0/22856502 shutdown_connections 2024-04-24T08:11:00.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.419+0000 7f9703dc3740 1 -- 172.21.15.28:0/22856502 wait complete. 2024-04-24T08:11:00.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.419+0000 7f96f9e00640 1 --2- 172.21.15.28:0/1394215137 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f96e00a3ec0 0x7f96e00a6380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f96ec0ce3c0 tx=0x7f96ec0d45d0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:00.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.420+0000 7f9703dc3740 1 -- start start 2024-04-24T08:11:00.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.420+0000 7f9703dc3740 1 Processor -- start 2024-04-24T08:11:00.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.420+0000 7f9703dc3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916a860 0x5578591727f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.420+0000 7f9703dc3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55785916b180 0x557859172d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.420+0000 7f9703dc3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557859175850 0x557859173270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.420+0000 7f9703dc3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557859178640 con 0x55785916b180 2024-04-24T08:11:00.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.420+0000 7f9703dc3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557859089690 con 0x557859175850 2024-04-24T08:11:00.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.420+0000 7f9703dc3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5578591787c0 con 0x55785916a860 2024-04-24T08:11:00.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.420+0000 7f9703dc3740 10 client.0 _ll_register_callbacks cb 0x5578590c5650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:00.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.421+0000 7f96f1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55785916b180 0x557859172d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.421+0000 7f96f1400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55785916b180 0x557859172d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55944/0 (socket says 172.21.15.28:55944) 2024-04-24T08:11:00.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.421+0000 7f96f1400640 1 -- 172.21.15.28:0/1578598563 learned_addr learned my addr 172.21.15.28:0/1578598563 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:00.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.421+0000 7f96fa800640 1 --2- 172.21.15.28:0/1578598563 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916a860 0x5578591727f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.421+0000 7f96fa800640 1 --2- 172.21.15.28:0/1578598563 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557859175850 0x557859173270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.421+0000 7f96f1400640 1 -- 172.21.15.28:0/1578598563 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916a860 msgr2=0x5578591727f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:00.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.421+0000 7f96f1400640 1 --2- 172.21.15.28:0/1578598563 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916a860 0x5578591727f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.421+0000 7f96f1400640 1 -- 172.21.15.28:0/1578598563 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557859175850 msgr2=0x557859173270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:00.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.421+0000 7f96f1400640 1 --2- 172.21.15.28:0/1578598563 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557859175850 0x557859173270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.421+0000 7f96f1400640 1 -- 172.21.15.28:0/1578598563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557859088920 con 0x55785916b180 2024-04-24T08:11:00.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.421+0000 7f96fa800640 1 --2- 172.21.15.28:0/1578598563 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557859175850 0x557859173270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:00.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.421+0000 7f96f1400640 1 --2- 172.21.15.28:0/1578598563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55785916b180 0x557859172d30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f96e4033000 tx=0x7f96e40043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=585c4ee8c7116b96 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:00.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.421+0000 7f96dfe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:11:00.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.422+0000 7f96dfe00640 1 -- 172.21.15.28:0/1578598563 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e4035040 con 0x55785916b180 2024-04-24T08:11:00.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.422+0000 7f96dfe00640 1 -- 172.21.15.28:0/1578598563 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f96e4005450 con 0x55785916b180 2024-04-24T08:11:00.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.422+0000 7f9703dc3740 1 -- 172.21.15.28:0/1578598563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96f400f9a0 con 0x55785916b180 2024-04-24T08:11:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.422+0000 7f96dfe00640 1 -- 172.21.15.28:0/1578598563 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f96e4005650 con 0x55785916b180 2024-04-24T08:11:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.422+0000 7f9703dc3740 10 client.15354 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.422+0000 7f9703dc3740 1 -- 172.21.15.28:0/1578598563 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557858eac7d0 con 0x55785916b180 2024-04-24T08:11:00.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.422+0000 7f96dfe00640 1 -- 172.21.15.28:0/1578598563 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96f400f9a0 con 0x55785916b180 2024-04-24T08:11:00.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.423+0000 7f96dfe00640 1 -- 172.21.15.28:0/1578598563 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f96e4031a20 con 0x55785916b180 2024-04-24T08:11:00.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.423+0000 7f9703dc3740 10 client.15354 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:00.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.423+0000 7f9703dc3740 10 client.15354 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:11:00.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.423+0000 7f9703dc3740 20 client.15354 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:00.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.423+0000 7f9703dc3740 1 --2- 172.21.15.28:0/1578598563 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557859185c40 0x557859188080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.423+0000 7f9703dc3740 4 client.15354 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:11:00.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.423+0000 7f9703dc3740 1 -- 172.21.15.28:0/1578598563 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5578590ddc10 con 0x557859185c40 2024-04-24T08:11:00.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.423+0000 7f96fa800640 1 --2- 172.21.15.28:0/1578598563 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557859185c40 0x557859188080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.424+0000 7f96fa800640 1 --2- 172.21.15.28:0/1578598563 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557859185c40 0x557859188080 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2e6f85fd7dce26b5 server_cookie=8e7608779478874 in_seq=0 out_seq=0 2024-04-24T08:11:00.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.424+0000 7f96dfe00640 10 client.15354 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:00.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.424+0000 7f96dfe00640 1 -- 172.21.15.28:0/1578598563 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5578590ddc10 con 0x557859185c40 2024-04-24T08:11:00.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.424+0000 7f96dfe00640 10 client.15354 handle_command_reply: tid=0 2024-04-24T08:11:00.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.425+0000 7f9703dc3740 1 client.15354 shutdown 2024-04-24T08:11:00.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.425+0000 7f9703dc3740 1 -- 172.21.15.28:0/1578598563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55785916b180 msgr2=0x557859172d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:00.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.425+0000 7f9703dc3740 1 --2- 172.21.15.28:0/1578598563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55785916b180 0x557859172d30 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f96e4033000 tx=0x7f96e40043b0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.425+0000 7f9703dc3740 1 -- 172.21.15.28:0/1578598563 shutdown_connections 2024-04-24T08:11:00.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.425+0000 7f9703dc3740 1 --2- 172.21.15.28:0/1578598563 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557859185c40 0x557859188080 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.425+0000 7f9703dc3740 1 --2- 172.21.15.28:0/1578598563 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55785916b180 0x557859172d30 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.425+0000 7f9703dc3740 1 --2- 172.21.15.28:0/1578598563 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557859175850 0x557859173270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.425+0000 7f9703dc3740 1 --2- 172.21.15.28:0/1578598563 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916a860 0x5578591727f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.425+0000 7f96dfe00640 0 client.15354 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:00.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.425+0000 7f9703dc3740 1 -- 172.21.15.28:0/1578598563 >> 172.21.15.28:0/1578598563 conn(0x557859166b80 msgr2=0x55785916c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:00.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.425+0000 7f9703dc3740 1 -- 172.21.15.28:0/1578598563 shutdown_connections 2024-04-24T08:11:00.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.425+0000 7f9703dc3740 1 -- 172.21.15.28:0/1578598563 wait complete. 2024-04-24T08:11:00.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.425+0000 7f9703dc3740 20 client.15354 trim_cache size 0 max 16384 2024-04-24T08:11:00.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.444+0000 7f9703dc3740 1 Processor -- start 2024-04-24T08:11:00.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.444+0000 7f9703dc3740 1 -- start start 2024-04-24T08:11:00.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.445+0000 7f9703dc3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55785916a860 0x557859171770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.445+0000 7f9703dc3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916b180 0x557859181500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.445+0000 7f9703dc3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557859175850 0x55785916cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.445+0000 7f9703dc3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96fc1b6d40 con 0x55785916a860 2024-04-24T08:11:00.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.445+0000 7f9703dc3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f96fc1b4610 con 0x557859175850 2024-04-24T08:11:00.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.445+0000 7f9703dc3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f96fc1b2460 con 0x55785916b180 2024-04-24T08:11:00.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.445+0000 7f96fa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55785916a860 0x557859171770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.445+0000 7f96fa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55785916a860 0x557859171770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55956/0 (socket says 172.21.15.28:55956) 2024-04-24T08:11:00.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.445+0000 7f96f1400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916b180 0x557859181500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.445+0000 7f96fa800640 1 -- 172.21.15.28:0/501989231 learned_addr learned my addr 172.21.15.28:0/501989231 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.446+0000 7f96fa800640 1 --2- 172.21.15.28:0/501989231 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557859175850 0x55785916cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.446+0000 7f96f1400640 1 -- 172.21.15.28:0/501989231 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557859175850 msgr2=0x55785916cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.446+0000 7f96f1400640 1 --2- 172.21.15.28:0/501989231 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557859175850 0x55785916cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.446+0000 7f96f1400640 1 -- 172.21.15.28:0/501989231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55785916a860 msgr2=0x557859171770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.446+0000 7f96f1400640 1 --2- 172.21.15.28:0/501989231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55785916a860 0x557859171770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.446+0000 7f96f1400640 1 -- 172.21.15.28:0/501989231 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557858de23f0 con 0x55785916b180 2024-04-24T08:11:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.446+0000 7f96fa800640 1 --2- 172.21.15.28:0/501989231 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557859175850 0x55785916cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.446+0000 7f96f1400640 1 --2- 172.21.15.28:0/501989231 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916b180 0x557859181500 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f96e402aa30 tx=0x7f96e402a440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=442cb234141d6b8f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.446+0000 7f96eaa00640 1 -- 172.21.15.28:0/501989231 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e4053070 con 0x55785916b180 2024-04-24T08:11:00.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.447+0000 7f96eaa00640 1 -- 172.21.15.28:0/501989231 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96e4041070 con 0x55785916b180 2024-04-24T08:11:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.447+0000 7f96eaa00640 1 -- 172.21.15.28:0/501989231 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e4031930 con 0x55785916b180 2024-04-24T08:11:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.447+0000 7f9703dc3740 1 -- 172.21.15.28:0/501989231 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916b180 msgr2=0x557859181500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.447+0000 7f9703dc3740 1 --2- 172.21.15.28:0/501989231 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916b180 0x557859181500 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f96e402aa30 tx=0x7f96e402a440 comp rx=0 tx=0).stop 2024-04-24T08:11:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.447+0000 7f9703dc3740 1 -- 172.21.15.28:0/501989231 shutdown_connections 2024-04-24T08:11:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.447+0000 7f9703dc3740 1 --2- 172.21.15.28:0/501989231 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557859175850 0x55785916cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.447+0000 7f9703dc3740 1 --2- 172.21.15.28:0/501989231 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916b180 0x557859181500 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.447+0000 7f9703dc3740 1 --2- 172.21.15.28:0/501989231 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55785916a860 0x557859171770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.447+0000 7f9703dc3740 1 -- 172.21.15.28:0/501989231 >> 172.21.15.28:0/501989231 conn(0x557859166b80 msgr2=0x557859165c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.447+0000 7f9703dc3740 1 -- 172.21.15.28:0/501989231 shutdown_connections 2024-04-24T08:11:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.447+0000 7f9703dc3740 1 -- 172.21.15.28:0/501989231 wait complete. 2024-04-24T08:11:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.448+0000 7f9703dc3740 1 -- start start 2024-04-24T08:11:00.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.448+0000 7f9703dc3740 1 Processor -- start 2024-04-24T08:11:00.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.448+0000 7f9703dc3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916a860 0x55785917dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.448+0000 7f9703dc3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55785916b180 0x55785916f140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.449+0000 7f9703dc3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557859175850 0x55785916f680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.449+0000 7f9703dc3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f96fc1b2ad0 con 0x557859175850 2024-04-24T08:11:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.449+0000 7f9703dc3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f96fc1b3f40 con 0x55785916b180 2024-04-24T08:11:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.449+0000 7f9703dc3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f96fc1b31a0 con 0x55785916a860 2024-04-24T08:11:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.449+0000 7f9703dc3740 10 client.0 _ll_register_callbacks cb 0x7f96e4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.449+0000 7f96fa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557859175850 0x55785916f680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.449+0000 7f96f1400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55785916b180 0x55785916f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.449+0000 7f96fa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557859175850 0x55785916f680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55964/0 (socket says 172.21.15.28:55964) 2024-04-24T08:11:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.449+0000 7f96fa800640 1 -- 172.21.15.28:0/1777566846 learned_addr learned my addr 172.21.15.28:0/1777566846 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.449+0000 7f96fa800640 1 --2- 172.21.15.28:0/1777566846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916a860 0x55785917dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.449+0000 7f96f1400640 1 -- 172.21.15.28:0/1777566846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916a860 msgr2=0x55785917dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.449+0000 7f96f1400640 1 --2- 172.21.15.28:0/1777566846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916a860 0x55785917dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.449+0000 7f96f1400640 1 -- 172.21.15.28:0/1777566846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557859175850 msgr2=0x55785916f680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:00.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.449+0000 7f96f1400640 1 --2- 172.21.15.28:0/1777566846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557859175850 0x55785916f680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.449+0000 7f96f1400640 1 -- 172.21.15.28:0/1777566846 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557858da20f0 con 0x55785916b180 2024-04-24T08:11:00.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.450+0000 7f96f1400640 1 --2- 172.21.15.28:0/1777566846 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55785916b180 0x55785916f140 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x5578591785f0 tx=0x7f96e4003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f6dea3c34a2c3060 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:00.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.450+0000 7f96dfe00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:11:00.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.450+0000 7f96dfe00640 1 -- 172.21.15.28:0/1777566846 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96e405d060 con 0x55785916b180 2024-04-24T08:11:00.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.450+0000 7f96dfe00640 1 -- 172.21.15.28:0/1777566846 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f96e4003bc0 con 0x55785916b180 2024-04-24T08:11:00.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.450+0000 7f9703dc3740 1 -- 172.21.15.28:0/1777566846 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f96e4041070 con 0x55785916b180 2024-04-24T08:11:00.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.450+0000 7f96dfe00640 1 -- 172.21.15.28:0/1777566846 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f96e4035030 con 0x55785916b180 2024-04-24T08:11:00.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.450+0000 7f9703dc3740 10 client.25454 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:00.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.450+0000 7f9703dc3740 1 -- 172.21.15.28:0/1777566846 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557859095810 con 0x55785916b180 2024-04-24T08:11:00.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.451+0000 7f96dfe00640 1 -- 172.21.15.28:0/1777566846 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f96e4041070 con 0x55785916b180 2024-04-24T08:11:00.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.451+0000 7f96dfe00640 1 -- 172.21.15.28:0/1777566846 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f96e40047e0 con 0x55785916b180 2024-04-24T08:11:00.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.451+0000 7f9703dc3740 10 client.25454 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:00.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.451+0000 7f9703dc3740 10 client.25454 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:11:00.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.451+0000 7f9703dc3740 20 client.25454 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:00.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.451+0000 7f9703dc3740 1 --2- 172.21.15.28:0/1777566846 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5578591ec280 0x5578591ee6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.451+0000 7f9703dc3740 4 client.25454 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:11:00.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.451+0000 7f9703dc3740 1 -- 172.21.15.28:0/1777566846 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5578590d7a80 con 0x5578591ec280 2024-04-24T08:11:00.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.451+0000 7f96fa800640 1 --2- 172.21.15.28:0/1777566846 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5578591ec280 0x5578591ee6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.452+0000 7f96fa800640 1 --2- 172.21.15.28:0/1777566846 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5578591ec280 0x5578591ee6a0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e90dfc33861d0294 server_cookie=8ee827fa1b9422b in_seq=0 out_seq=0 2024-04-24T08:11:00.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.452+0000 7f96dfe00640 10 client.25454 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:00.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.452+0000 7f96dfe00640 1 -- 172.21.15.28:0/1777566846 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5578590d7a80 con 0x5578591ec280 2024-04-24T08:11:00.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.452+0000 7f96dfe00640 10 client.25454 handle_command_reply: tid=0 2024-04-24T08:11:00.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.452+0000 7f9703dc3740 1 client.25454 shutdown 2024-04-24T08:11:00.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.452+0000 7f9703dc3740 1 -- 172.21.15.28:0/1777566846 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55785916b180 msgr2=0x55785916f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:00.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.452+0000 7f9703dc3740 1 --2- 172.21.15.28:0/1777566846 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55785916b180 0x55785916f140 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x5578591785f0 tx=0x7f96e4003820 comp rx=0 tx=0).stop 2024-04-24T08:11:00.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.452+0000 7f9703dc3740 1 -- 172.21.15.28:0/1777566846 shutdown_connections 2024-04-24T08:11:00.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.452+0000 7f9703dc3740 1 --2- 172.21.15.28:0/1777566846 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5578591ec280 0x5578591ee6a0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.452+0000 7f9703dc3740 1 --2- 172.21.15.28:0/1777566846 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557859175850 0x55785916f680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.452+0000 7f9703dc3740 1 --2- 172.21.15.28:0/1777566846 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55785916b180 0x55785916f140 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.452+0000 7f96dfe00640 0 client.25454 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:00.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.452+0000 7f9703dc3740 1 --2- 172.21.15.28:0/1777566846 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55785916a860 0x55785917dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.452+0000 7f9703dc3740 1 -- 172.21.15.28:0/1777566846 >> 172.21.15.28:0/1777566846 conn(0x557859166b80 msgr2=0x5578591725e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:00.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.453+0000 7f9703dc3740 1 -- 172.21.15.28:0/1777566846 shutdown_connections 2024-04-24T08:11:00.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.453+0000 7f9703dc3740 1 -- 172.21.15.28:0/1777566846 wait complete. 2024-04-24T08:11:00.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.453+0000 7f9703dc3740 20 client.25454 trim_cache size 0 max 16384 2024-04-24T08:11:00.456 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:11:00.456 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-24T08:11:00.456 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-24T08:11:00.456 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:11:00.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.453+0000 7f9701000640 1 -- 172.21.15.28:0/1394215137 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f96e00a3ec0 msgr2=0x7f96e00a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:00.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.453+0000 7f9701000640 1 --2- 172.21.15.28:0/1394215137 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f96e00a3ec0 0x7f96e00a6380 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f96ec0ce3c0 tx=0x7f96ec0d45d0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.453+0000 7f9701000640 1 -- 172.21.15.28:0/1394215137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96fc076530 msgr2=0x7f96fc19efd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:00.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.453+0000 7f9701000640 1 --2- 172.21.15.28:0/1394215137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96fc076530 0x7f96fc19efd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f96ec00e9f0 tx=0x7f96ec00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.453+0000 7f9701000640 1 -- 172.21.15.28:0/1394215137 shutdown_connections 2024-04-24T08:11:00.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.453+0000 7f9701000640 1 --2- 172.21.15.28:0/1394215137 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f96e00a3ec0 0x7f96e00a6380 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.453+0000 7f9701000640 1 --2- 172.21.15.28:0/1394215137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96fc112ee0 0x7f96fc1a3890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.453+0000 7f9701000640 1 --2- 172.21.15.28:0/1394215137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f96fc076e50 0x7f96fc19f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.453+0000 7f9701000640 1 --2- 172.21.15.28:0/1394215137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f96fc076530 0x7f96fc19efd0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.453+0000 7f9701000640 1 -- 172.21.15.28:0/1394215137 >> 172.21.15.28:0/1394215137 conn(0x7f96fc0ffd90 msgr2=0x7f96fc102180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:00.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.454+0000 7f9701000640 1 -- 172.21.15.28:0/1394215137 shutdown_connections 2024-04-24T08:11:00.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.454+0000 7f9701000640 1 -- 172.21.15.28:0/1394215137 wait complete. 2024-04-24T08:11:00.472 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:11:00.472 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:11:00.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.668+0000 7faa47200640 1 -- 172.21.15.28:0/1762623866 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa4810d3c0 msgr2=0x7faa4810d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:00.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.668+0000 7faa47200640 1 --2- 172.21.15.28:0/1762623866 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa4810d3c0 0x7faa4810d7a0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7faa38009860 tx=0x7faa3802a3d0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.668+0000 7faa47200640 1 -- 172.21.15.28:0/1762623866 shutdown_connections 2024-04-24T08:11:00.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.668+0000 7faa47200640 1 --2- 172.21.15.28:0/1762623866 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa48108e40 0x7faa481152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.668+0000 7faa47200640 1 --2- 172.21.15.28:0/1762623866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa481084a0 0x7faa48108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.668+0000 7faa47200640 1 --2- 172.21.15.28:0/1762623866 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa4810d3c0 0x7faa4810d7a0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.668+0000 7faa47200640 1 -- 172.21.15.28:0/1762623866 >> 172.21.15.28:0/1762623866 conn(0x7faa48077b80 msgr2=0x7faa48079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:00.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.668+0000 7faa47200640 1 -- 172.21.15.28:0/1762623866 shutdown_connections 2024-04-24T08:11:00.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.669+0000 7faa47200640 1 -- 172.21.15.28:0/1762623866 wait complete. 2024-04-24T08:11:00.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.670+0000 7faa47200640 1 Processor -- start 2024-04-24T08:11:00.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.670+0000 7faa47200640 1 -- start start 2024-04-24T08:11:00.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.670+0000 7faa47200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa481084a0 0x7faa481a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.670+0000 7faa47200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa48108e40 0x7faa481a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.670+0000 7faa47200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4810d3c0 0x7faa4819eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.670+0000 7faa47200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7faa48118830 con 0x7faa4810d3c0 2024-04-24T08:11:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.670+0000 7faa47200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7faa481186b0 con 0x7faa48108e40 2024-04-24T08:11:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.670+0000 7faa47200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7faa481189b0 con 0x7faa481084a0 2024-04-24T08:11:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.670+0000 7faa46800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4810d3c0 0x7faa4819eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.670+0000 7faa3fe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa48108e40 0x7faa481a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.670+0000 7faa3fe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa48108e40 0x7faa481a4e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:45494/0 (socket says 172.21.15.28:45494) 2024-04-24T08:11:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.670+0000 7faa3fe00640 1 -- 172.21.15.28:0/1811413712 learned_addr learned my addr 172.21.15.28:0/1811413712 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.670+0000 7faa45e00640 1 --2- 172.21.15.28:0/1811413712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa481084a0 0x7faa481a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.671+0000 7faa3fe00640 1 -- 172.21.15.28:0/1811413712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa481084a0 msgr2=0x7faa481a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.671+0000 7faa3fe00640 1 --2- 172.21.15.28:0/1811413712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa481084a0 0x7faa481a4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.671+0000 7faa3fe00640 1 -- 172.21.15.28:0/1811413712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4810d3c0 msgr2=0x7faa4819eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:00.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.671+0000 7faa3fe00640 1 --2- 172.21.15.28:0/1811413712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4810d3c0 0x7faa4819eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.671+0000 7faa3fe00640 1 -- 172.21.15.28:0/1811413712 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4819f270 con 0x7faa48108e40 2024-04-24T08:11:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.671+0000 7faa45e00640 1 --2- 172.21.15.28:0/1811413712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa481084a0 0x7faa481a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.671+0000 7faa3fe00640 1 --2- 172.21.15.28:0/1811413712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa48108e40 0x7faa481a4e80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faa3000e9f0 tx=0x7faa3000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:00.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.671+0000 7faa3ea00640 1 -- 172.21.15.28:0/1811413712 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3000cd80 con 0x7faa48108e40 2024-04-24T08:11:00.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.672+0000 7faa47200640 1 -- 172.21.15.28:0/1811413712 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4819f560 con 0x7faa48108e40 2024-04-24T08:11:00.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.672+0000 7faa3ea00640 1 -- 172.21.15.28:0/1811413712 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa30010910 con 0x7faa48108e40 2024-04-24T08:11:00.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.672+0000 7faa3ea00640 1 -- 172.21.15.28:0/1811413712 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa300188f0 con 0x7faa48108e40 2024-04-24T08:11:00.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.672+0000 7faa47200640 1 -- 172.21.15.28:0/1811413712 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa481ab800 con 0x7faa48108e40 2024-04-24T08:11:00.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.673+0000 7faa47200640 1 -- 172.21.15.28:0/1811413712 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa48069190 con 0x7faa48108e40 2024-04-24T08:11:00.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.675+0000 7faa3ea00640 1 -- 172.21.15.28:0/1811413712 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7faa30010ac0 con 0x7faa48108e40 2024-04-24T08:11:00.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.676+0000 7faa3ea00640 1 --2- 172.21.15.28:0/1811413712 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7faa2c0a3ec0 0x7faa2c0a6380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:00.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.676+0000 7faa3ea00640 1 -- 172.21.15.28:0/1811413712 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7faa300d0fe0 con 0x7faa48108e40 2024-04-24T08:11:00.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.677+0000 7faa45e00640 1 --2- 172.21.15.28:0/1811413712 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7faa2c0a3ec0 0x7faa2c0a6380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:00.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.677+0000 7faa45e00640 1 --2- 172.21.15.28:0/1811413712 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7faa2c0a3ec0 0x7faa2c0a6380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa380095f0 tx=0x7faa38005c90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:00.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.679+0000 7faa3ea00640 1 -- 172.21.15.28:0/1811413712 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faa3009ce20 con 0x7faa48108e40 2024-04-24T08:11:00.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.962+0000 7faa47200640 1 -- 172.21.15.28:0/1811413712 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7faa4819ffe0 con 0x7faa48108e40 2024-04-24T08:11:00.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.963+0000 7faa3ea00640 1 -- 172.21.15.28:0/1811413712 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11871 (secure 0 0 0) 0x7faa300a1120 con 0x7faa48108e40 2024-04-24T08:11:00.964 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:11:00.964 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:10:19.714831+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24574,24556,24572]},"id":1}]} 2024-04-24T08:11:00.965 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-24T08:11:00.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.966+0000 7faa47200640 1 -- 172.21.15.28:0/1811413712 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7faa2c0a3ec0 msgr2=0x7faa2c0a6380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:00.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.966+0000 7faa47200640 1 --2- 172.21.15.28:0/1811413712 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7faa2c0a3ec0 0x7faa2c0a6380 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa380095f0 tx=0x7faa38005c90 comp rx=0 tx=0).stop 2024-04-24T08:11:00.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.967+0000 7faa47200640 1 -- 172.21.15.28:0/1811413712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa48108e40 msgr2=0x7faa481a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:00.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.967+0000 7faa47200640 1 --2- 172.21.15.28:0/1811413712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa48108e40 0x7faa481a4e80 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7faa3000e9f0 tx=0x7faa3000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.967+0000 7faa47200640 1 -- 172.21.15.28:0/1811413712 shutdown_connections 2024-04-24T08:11:00.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.967+0000 7faa47200640 1 --2- 172.21.15.28:0/1811413712 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7faa2c0a3ec0 0x7faa2c0a6380 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.967+0000 7faa47200640 1 --2- 172.21.15.28:0/1811413712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7faa4810d3c0 0x7faa4819eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.967+0000 7faa47200640 1 --2- 172.21.15.28:0/1811413712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa48108e40 0x7faa481a4e80 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.967+0000 7faa47200640 1 --2- 172.21.15.28:0/1811413712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7faa481084a0 0x7faa481a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:00.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.967+0000 7faa47200640 1 -- 172.21.15.28:0/1811413712 >> 172.21.15.28:0/1811413712 conn(0x7faa48077b80 msgr2=0x7faa48078430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:00.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.967+0000 7faa47200640 1 -- 172.21.15.28:0/1811413712 shutdown_connections 2024-04-24T08:11:00.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:00.967+0000 7faa47200640 1 -- 172.21.15.28:0/1811413712 wait complete. 2024-04-24T08:11:00.989 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:11:01.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.191+0000 7f6f5e000640 1 -- 172.21.15.28:0/2092602328 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5802f310 con 0x7f6f60112ee0 2024-04-24T08:11:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.191+0000 7f6f66200640 1 -- 172.21.15.28:0/2092602328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f60112ee0 msgr2=0x7f6f601152d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.191+0000 7f6f66200640 1 --2- 172.21.15.28:0/2092602328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f60112ee0 0x7f6f601152d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6f58009ca0 tx=0x7f6f5802a8c0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.191+0000 7f6f66200640 1 -- 172.21.15.28:0/2092602328 shutdown_connections 2024-04-24T08:11:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.191+0000 7f6f66200640 1 --2- 172.21.15.28:0/2092602328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f60112ee0 0x7f6f601152d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.191+0000 7f6f66200640 1 --2- 172.21.15.28:0/2092602328 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f60076e50 0x7f6f601129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.191+0000 7f6f66200640 1 --2- 172.21.15.28:0/2092602328 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f60076530 0x7f6f60076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.191+0000 7f6f66200640 1 -- 172.21.15.28:0/2092602328 >> 172.21.15.28:0/2092602328 conn(0x7f6f600ffd90 msgr2=0x7f6f601021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.192+0000 7f6f66200640 1 -- 172.21.15.28:0/2092602328 shutdown_connections 2024-04-24T08:11:01.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.192+0000 7f6f66200640 1 -- 172.21.15.28:0/2092602328 wait complete. 2024-04-24T08:11:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.193+0000 7f6f66200640 1 Processor -- start 2024-04-24T08:11:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.193+0000 7f6f66200640 1 -- start start 2024-04-24T08:11:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.193+0000 7f6f66200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f60076530 0x7f6f6019ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.193+0000 7f6f66200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f60076e50 0x7f6f6019f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.193+0000 7f6f66200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f60112ee0 0x7f6f601a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.193+0000 7f6f66200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6f60118a40 con 0x7f6f60112ee0 2024-04-24T08:11:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.193+0000 7f6f66200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6f601188c0 con 0x7f6f60076e50 2024-04-24T08:11:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.193+0000 7f6f66200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6f60118bc0 con 0x7f6f60076530 2024-04-24T08:11:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.194+0000 7f6f5fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f60112ee0 0x7f6f601a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.194+0000 7f6f5ea00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f60076e50 0x7f6f6019f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.194+0000 7f6f5fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f60112ee0 0x7f6f601a3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56014/0 (socket says 172.21.15.28:56014) 2024-04-24T08:11:01.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.194+0000 7f6f5fe00640 1 -- 172.21.15.28:0/3738521363 learned_addr learned my addr 172.21.15.28:0/3738521363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:01.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.194+0000 7f6f5f400640 1 --2- 172.21.15.28:0/3738521363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f60076530 0x7f6f6019ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.194+0000 7f6f5ea00640 1 -- 172.21.15.28:0/3738521363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f60076530 msgr2=0x7f6f6019ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.194+0000 7f6f5ea00640 1 --2- 172.21.15.28:0/3738521363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f60076530 0x7f6f6019ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.194+0000 7f6f5ea00640 1 -- 172.21.15.28:0/3738521363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f60112ee0 msgr2=0x7f6f601a3800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.194+0000 7f6f5ea00640 1 --2- 172.21.15.28:0/3738521363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f60112ee0 0x7f6f601a3800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.194+0000 7f6f5ea00640 1 -- 172.21.15.28:0/3738521363 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f601a3f30 con 0x7f6f60076e50 2024-04-24T08:11:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.194+0000 7f6f5fe00640 1 --2- 172.21.15.28:0/3738521363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f60112ee0 0x7f6f601a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.194+0000 7f6f5ea00640 1 --2- 172.21.15.28:0/3738521363 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f60076e50 0x7f6f6019f480 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800e9f0 tx=0x7f6f4800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.195+0000 7f6f57e00640 1 -- 172.21.15.28:0/3738521363 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f4800ccc0 con 0x7f6f60076e50 2024-04-24T08:11:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.195+0000 7f6f57e00640 1 -- 172.21.15.28:0/3738521363 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f48010910 con 0x7f6f60076e50 2024-04-24T08:11:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.195+0000 7f6f66200640 1 -- 172.21.15.28:0/3738521363 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f601a4220 con 0x7f6f60076e50 2024-04-24T08:11:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.195+0000 7f6f57e00640 1 -- 172.21.15.28:0/3738521363 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f48018960 con 0x7f6f60076e50 2024-04-24T08:11:01.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.195+0000 7f6f66200640 1 -- 172.21.15.28:0/3738521363 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f601abab0 con 0x7f6f60076e50 2024-04-24T08:11:01.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.198+0000 7f6f57e00640 1 -- 172.21.15.28:0/3738521363 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7f6f48010ac0 con 0x7f6f60076e50 2024-04-24T08:11:01.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.198+0000 7f6f691c3740 1 Processor -- start 2024-04-24T08:11:01.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.199+0000 7f6f691c3740 1 -- start start 2024-04-24T08:11:01.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.199+0000 7f6f691c3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 0x556a7131cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.199+0000 7f6f691c3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a7131d270 0x556a71327400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.199+0000 7f6f691c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a71327940 0x556a71329d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.199+0000 7f6f691c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556a712821d0 con 0x556a71327940 2024-04-24T08:11:01.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.199+0000 7f6f691c3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556a71228450 con 0x556a7131c950 2024-04-24T08:11:01.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.199+0000 7f6f691c3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556a7127e190 con 0x556a7131d270 2024-04-24T08:11:01.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.199+0000 7f6f5fe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 0x556a7131cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.199+0000 7f6f5f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a7131d270 0x556a71327400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.199+0000 7f6f5ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a71327940 0x556a71329d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.199+0000 7f6f5fe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 0x556a7131cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:45544/0 (socket says 172.21.15.28:45544) 2024-04-24T08:11:01.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.199+0000 7f6f5fe00640 1 -- 172.21.15.28:0/4249953406 learned_addr learned my addr 172.21.15.28:0/4249953406 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:01.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.199+0000 7f6f5f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a7131d270 0x556a71327400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:52992/0 (socket says 172.21.15.28:52992) 2024-04-24T08:11:01.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.200+0000 7f6f5fe00640 1 -- 172.21.15.28:0/4249953406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a7131d270 msgr2=0x556a71327400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.200+0000 7f6f5fe00640 1 --2- 172.21.15.28:0/4249953406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a7131d270 0x556a71327400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.200+0000 7f6f5fe00640 1 -- 172.21.15.28:0/4249953406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a71327940 msgr2=0x556a71329d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.200+0000 7f6f5fe00640 1 --2- 172.21.15.28:0/4249953406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a71327940 0x556a71329d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.200+0000 7f6f5fe00640 1 -- 172.21.15.28:0/4249953406 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a712e5310 con 0x556a7131c950 2024-04-24T08:11:01.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.200+0000 7f6f5fe00640 1 --2- 172.21.15.28:0/4249953406 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 0x556a7131cd30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6f5802f4a0 tx=0x7f6f5802f4d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cace22ef9b7a083f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.200+0000 7f6f57e00640 1 --2- 172.21.15.28:0/3738521363 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6f440a3e40 0x7f6f440a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.200+0000 7f6f57e00640 1 -- 172.21.15.28:0/3738521363 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f6f480d0e00 con 0x7f6f60076e50 2024-04-24T08:11:01.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.200+0000 7f6f4fe00640 1 -- 172.21.15.28:0/4249953406 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f58003b70 con 0x556a7131c950 2024-04-24T08:11:01.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.200+0000 7f6f4fe00640 1 -- 172.21.15.28:0/4249953406 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f58036040 con 0x556a7131c950 2024-04-24T08:11:01.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.200+0000 7f6f4fe00640 1 -- 172.21.15.28:0/4249953406 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f58004670 con 0x556a7131c950 2024-04-24T08:11:01.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.200+0000 7f6f5f400640 1 --2- 172.21.15.28:0/3738521363 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6f440a3e40 0x7f6f440a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.201+0000 7f6f691c3740 1 -- 172.21.15.28:0/4249953406 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 msgr2=0x556a7131cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.201+0000 7f6f691c3740 1 --2- 172.21.15.28:0/4249953406 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 0x556a7131cd30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f6f5802f4a0 tx=0x7f6f5802f4d0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.201+0000 7f6f5f400640 1 --2- 172.21.15.28:0/3738521363 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6f440a3e40 0x7f6f440a6300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x556a7131df10 tx=0x7f6f50005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.201+0000 7f6f691c3740 1 -- 172.21.15.28:0/4249953406 shutdown_connections 2024-04-24T08:11:01.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.201+0000 7f6f691c3740 1 --2- 172.21.15.28:0/4249953406 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a71327940 0x556a71329d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.201+0000 7f6f691c3740 1 --2- 172.21.15.28:0/4249953406 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a7131d270 0x556a71327400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.201+0000 7f6f691c3740 1 --2- 172.21.15.28:0/4249953406 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 0x556a7131cd30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.201+0000 7f6f691c3740 1 -- 172.21.15.28:0/4249953406 >> 172.21.15.28:0/4249953406 conn(0x556a71318c70 msgr2=0x556a71319050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.201+0000 7f6f691c3740 1 -- 172.21.15.28:0/4249953406 shutdown_connections 2024-04-24T08:11:01.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.201+0000 7f6f691c3740 1 -- 172.21.15.28:0/4249953406 wait complete. 2024-04-24T08:11:01.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.202+0000 7f6f691c3740 1 -- start start 2024-04-24T08:11:01.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.202+0000 7f6f691c3740 1 Processor -- start 2024-04-24T08:11:01.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.203+0000 7f6f691c3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 0x556a71327320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.203+0000 7f6f691c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a7131d270 0x556a71324680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.203+0000 7f6f691c3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a71327940 0x556a71324d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.203+0000 7f6f691c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556a7132a5f0 con 0x556a7131d270 2024-04-24T08:11:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.203+0000 7f6f691c3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556a7123b810 con 0x556a7131c950 2024-04-24T08:11:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.203+0000 7f6f691c3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556a7132a770 con 0x556a71327940 2024-04-24T08:11:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.203+0000 7f6f691c3740 10 client.0 _ll_register_callbacks cb 0x556a71209140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.203+0000 7f6f5ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a7131d270 0x556a71324680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.203+0000 7f6f5ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a7131d270 0x556a71324680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56034/0 (socket says 172.21.15.28:56034) 2024-04-24T08:11:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.203+0000 7f6f5ea00640 1 -- 172.21.15.28:0/1261099028 learned_addr learned my addr 172.21.15.28:0/1261099028 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.203+0000 7f6f5f400640 1 --2- 172.21.15.28:0/1261099028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 0x556a71327320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.203+0000 7f6f5fe00640 1 --2- 172.21.15.28:0/1261099028 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a71327940 0x556a71324d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.204+0000 7f6f5fe00640 1 -- 172.21.15.28:0/1261099028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 msgr2=0x556a71327320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.204+0000 7f6f5fe00640 1 --2- 172.21.15.28:0/1261099028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 0x556a71327320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.204+0000 7f6f5fe00640 1 -- 172.21.15.28:0/1261099028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a7131d270 msgr2=0x556a71324680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.204+0000 7f6f5fe00640 1 --2- 172.21.15.28:0/1261099028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a7131d270 0x556a71324680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.204+0000 7f6f5fe00640 1 -- 172.21.15.28:0/1261099028 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a7124c3f0 con 0x556a71327940 2024-04-24T08:11:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.204+0000 7f6f5ea00640 1 --2- 172.21.15.28:0/1261099028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a7131d270 0x556a71324680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.204+0000 7f6f5fe00640 1 --2- 172.21.15.28:0/1261099028 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a71327940 0x556a71324d00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6f58005e30 tx=0x7f6f58004150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9252b2c69da01a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.204+0000 7f6f4cc00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:11:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.204+0000 7f6f4cc00640 1 -- 172.21.15.28:0/1261099028 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f58003f70 con 0x556a71327940 2024-04-24T08:11:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.204+0000 7f6f4cc00640 1 -- 172.21.15.28:0/1261099028 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f6f58031450 con 0x556a71327940 2024-04-24T08:11:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.204+0000 7f6f691c3740 1 -- 172.21.15.28:0/1261099028 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f58036040 con 0x556a71327940 2024-04-24T08:11:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.204+0000 7f6f4cc00640 1 -- 172.21.15.28:0/1261099028 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6f5805e050 con 0x556a71327940 2024-04-24T08:11:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.204+0000 7f6f691c3740 10 client.25516 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.205+0000 7f6f691c3740 1 -- 172.21.15.28:0/1261099028 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556a70f943f0 con 0x556a71327940 2024-04-24T08:11:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.205+0000 7f6f4cc00640 1 -- 172.21.15.28:0/1261099028 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f58036040 con 0x556a71327940 2024-04-24T08:11:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.205+0000 7f6f4cc00640 1 -- 172.21.15.28:0/1261099028 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f6f5804f070 con 0x556a71327940 2024-04-24T08:11:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.205+0000 7f6f691c3740 10 client.25516 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.205+0000 7f6f691c3740 10 client.25516 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:11:01.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.205+0000 7f6f691c3740 20 client.25516 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:01.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.205+0000 7f6f691c3740 1 --2- 172.21.15.28:0/1261099028 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x556a71337800 0x556a71339bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.205+0000 7f6f691c3740 4 client.25516 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:11:01.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.205+0000 7f6f691c3740 1 -- 172.21.15.28:0/1261099028 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556a7128fbf0 con 0x556a71337800 2024-04-24T08:11:01.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.206+0000 7f6f5f400640 1 --2- 172.21.15.28:0/1261099028 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x556a71337800 0x556a71339bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.206+0000 7f6f5f400640 1 --2- 172.21.15.28:0/1261099028 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x556a71337800 0x556a71339bf0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=9ce7414ab144ad2b server_cookie=e67e24042ce833b8 in_seq=0 out_seq=0 2024-04-24T08:11:01.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.206+0000 7f6f4cc00640 10 client.25516 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:11:01.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.207+0000 7f6f4cc00640 1 -- 172.21.15.28:0/1261099028 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556a7128fbf0 con 0x556a71337800 2024-04-24T08:11:01.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.207+0000 7f6f4cc00640 10 client.25516 handle_command_reply: tid=0 2024-04-24T08:11:01.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.207+0000 7f6f691c3740 1 client.25516 shutdown 2024-04-24T08:11:01.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.207+0000 7f6f691c3740 1 -- 172.21.15.28:0/1261099028 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a71327940 msgr2=0x556a71324d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.207+0000 7f6f691c3740 1 --2- 172.21.15.28:0/1261099028 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a71327940 0x556a71324d00 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6f58005e30 tx=0x7f6f58004150 comp rx=0 tx=0).stop 2024-04-24T08:11:01.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.208+0000 7f6f691c3740 1 -- 172.21.15.28:0/1261099028 shutdown_connections 2024-04-24T08:11:01.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.208+0000 7f6f691c3740 1 --2- 172.21.15.28:0/1261099028 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x556a71337800 0x556a71339bf0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.208+0000 7f6f691c3740 1 --2- 172.21.15.28:0/1261099028 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a7131d270 0x556a71324680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.208+0000 7f6f4cc00640 0 client.25516 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:11:01.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.208+0000 7f6f691c3740 1 --2- 172.21.15.28:0/1261099028 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a71327940 0x556a71324d00 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.208+0000 7f6f691c3740 1 --2- 172.21.15.28:0/1261099028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 0x556a71327320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.208+0000 7f6f691c3740 1 -- 172.21.15.28:0/1261099028 >> 172.21.15.28:0/1261099028 conn(0x556a71318c70 msgr2=0x556a7131ea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.208+0000 7f6f691c3740 1 -- 172.21.15.28:0/1261099028 shutdown_connections 2024-04-24T08:11:01.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.208+0000 7f6f691c3740 1 -- 172.21.15.28:0/1261099028 wait complete. 2024-04-24T08:11:01.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.208+0000 7f6f691c3740 20 client.25516 trim_cache size 0 max 16384 2024-04-24T08:11:01.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.223+0000 7f6f691c3740 1 Processor -- start 2024-04-24T08:11:01.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.223+0000 7f6f691c3740 1 -- start start 2024-04-24T08:11:01.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.223+0000 7f6f691c3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 0x556a71326ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.223+0000 7f6f691c3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a7131d270 0x556a7133f3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.223+0000 7f6f691c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a71327940 0x556a7133f8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.223+0000 7f6f691c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6f601b6c00 con 0x556a71327940 2024-04-24T08:11:01.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.223+0000 7f6f691c3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6f601b44d0 con 0x556a7131c950 2024-04-24T08:11:01.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.223+0000 7f6f691c3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6f601b2320 con 0x556a7131d270 2024-04-24T08:11:01.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.224+0000 7f6f5fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a71327940 0x556a7133f8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.224+0000 7f6f5f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 0x556a71326ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.224+0000 7f6f5fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a71327940 0x556a7133f8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56040/0 (socket says 172.21.15.28:56040) 2024-04-24T08:11:01.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.224+0000 7f6f5fe00640 1 -- 172.21.15.28:0/3267098258 learned_addr learned my addr 172.21.15.28:0/3267098258 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:01.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.224+0000 7f6f5ea00640 1 --2- 172.21.15.28:0/3267098258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a7131d270 0x556a7133f3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.224+0000 7f6f5fe00640 1 -- 172.21.15.28:0/3267098258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a7131d270 msgr2=0x556a7133f3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.224+0000 7f6f5fe00640 1 --2- 172.21.15.28:0/3267098258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a7131d270 0x556a7133f3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.224+0000 7f6f5fe00640 1 -- 172.21.15.28:0/3267098258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 msgr2=0x556a71326ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.224+0000 7f6f5fe00640 1 --2- 172.21.15.28:0/3267098258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 0x556a71326ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.224+0000 7f6f5fe00640 1 -- 172.21.15.28:0/3267098258 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556a71339f60 con 0x556a71327940 2024-04-24T08:11:01.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.224+0000 7f6f5f400640 1 --2- 172.21.15.28:0/3267098258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 0x556a71326ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:01.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.225+0000 7f6f5fe00640 1 --2- 172.21.15.28:0/3267098258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a71327940 0x556a7133f8f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6f58009620 tx=0x7f6f5802a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af65c54fc765ba3b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.225+0000 7f6f4f400640 1 -- 172.21.15.28:0/3267098258 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f5802f310 con 0x556a71327940 2024-04-24T08:11:01.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.225+0000 7f6f4f400640 1 -- 172.21.15.28:0/3267098258 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f58040020 con 0x556a71327940 2024-04-24T08:11:01.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.225+0000 7f6f4f400640 1 -- 172.21.15.28:0/3267098258 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f58031450 con 0x556a71327940 2024-04-24T08:11:01.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.226+0000 7f6f691c3740 1 -- 172.21.15.28:0/3267098258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a71327940 msgr2=0x556a7133f8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.226+0000 7f6f691c3740 1 --2- 172.21.15.28:0/3267098258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a71327940 0x556a7133f8f0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f6f58009620 tx=0x7f6f5802a8c0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.226+0000 7f6f691c3740 1 -- 172.21.15.28:0/3267098258 shutdown_connections 2024-04-24T08:11:01.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.226+0000 7f6f691c3740 1 --2- 172.21.15.28:0/3267098258 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a71327940 0x556a7133f8f0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.226+0000 7f6f691c3740 1 --2- 172.21.15.28:0/3267098258 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a7131d270 0x556a7133f3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.226+0000 7f6f691c3740 1 --2- 172.21.15.28:0/3267098258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a7131c950 0x556a71326ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.226+0000 7f6f691c3740 1 -- 172.21.15.28:0/3267098258 >> 172.21.15.28:0/3267098258 conn(0x556a71318c70 msgr2=0x556a71317df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.226+0000 7f6f691c3740 1 -- 172.21.15.28:0/3267098258 shutdown_connections 2024-04-24T08:11:01.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.226+0000 7f6f691c3740 1 -- 172.21.15.28:0/3267098258 wait complete. 2024-04-24T08:11:01.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.226+0000 7f6f691c3740 1 -- start start 2024-04-24T08:11:01.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.226+0000 7f6f691c3740 1 Processor -- start 2024-04-24T08:11:01.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.227+0000 7f6f691c3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a7131c950 0x556a71321360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.227+0000 7f6f691c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a7131d270 0x556a713218a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.227+0000 7f6f691c3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a71327940 0x556a71321de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.227+0000 7f6f691c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6f601b2990 con 0x556a7131d270 2024-04-24T08:11:01.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.227+0000 7f6f691c3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6f601b3e00 con 0x556a71327940 2024-04-24T08:11:01.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.227+0000 7f6f691c3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6f601b3060 con 0x556a7131c950 2024-04-24T08:11:01.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.227+0000 7f6f691c3740 10 client.0 _ll_register_callbacks cb 0x7f6f480d47a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:01.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.227+0000 7f6f5ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a7131d270 0x556a713218a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.227+0000 7f6f5ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a7131d270 0x556a713218a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56050/0 (socket says 172.21.15.28:56050) 2024-04-24T08:11:01.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.227+0000 7f6f5ea00640 1 -- 172.21.15.28:0/2161727027 learned_addr learned my addr 172.21.15.28:0/2161727027 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:01.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.227+0000 7f6f5f400640 1 --2- 172.21.15.28:0/2161727027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a7131c950 0x556a71321360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.227+0000 7f6f5fe00640 1 --2- 172.21.15.28:0/2161727027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a71327940 0x556a71321de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.227+0000 7f6f5ea00640 1 -- 172.21.15.28:0/2161727027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a7131c950 msgr2=0x556a71321360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.228+0000 7f6f5ea00640 1 --2- 172.21.15.28:0/2161727027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a7131c950 0x556a71321360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.228+0000 7f6f5ea00640 1 -- 172.21.15.28:0/2161727027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a71327940 msgr2=0x556a71321de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.228+0000 7f6f5ea00640 1 --2- 172.21.15.28:0/2161727027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a71327940 0x556a71321de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.228+0000 7f6f5ea00640 1 -- 172.21.15.28:0/2161727027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556a7105e7d0 con 0x556a7131d270 2024-04-24T08:11:01.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.228+0000 7f6f5fe00640 1 --2- 172.21.15.28:0/2161727027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a71327940 0x556a71321de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:01.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.228+0000 7f6f5ea00640 1 --2- 172.21.15.28:0/2161727027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a7131d270 0x556a713218a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6f48099020 tx=0x7f6f480ce410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8ef3c41e8fd4aa7e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.228+0000 7f6f4cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:11:01.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.228+0000 7f6f4cc00640 1 -- 172.21.15.28:0/2161727027 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f48098e60 con 0x556a7131d270 2024-04-24T08:11:01.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.228+0000 7f6f4cc00640 1 -- 172.21.15.28:0/2161727027 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f6f480ce870 con 0x556a7131d270 2024-04-24T08:11:01.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.228+0000 7f6f691c3740 1 -- 172.21.15.28:0/2161727027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f58040020 con 0x556a7131d270 2024-04-24T08:11:01.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.228+0000 7f6f4cc00640 1 -- 172.21.15.28:0/2161727027 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6f4809c520 con 0x556a7131d270 2024-04-24T08:11:01.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.228+0000 7f6f691c3740 10 client.15387 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:01.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.228+0000 7f6f691c3740 1 -- 172.21.15.28:0/2161727027 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x556a7139de60 con 0x556a7131d270 2024-04-24T08:11:01.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.229+0000 7f6f4cc00640 1 -- 172.21.15.28:0/2161727027 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f58040020 con 0x556a7131d270 2024-04-24T08:11:01.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.229+0000 7f6f4cc00640 1 -- 172.21.15.28:0/2161727027 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f6f480b8070 con 0x556a7131d270 2024-04-24T08:11:01.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.229+0000 7f6f691c3740 10 client.15387 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:01.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.229+0000 7f6f691c3740 10 client.15387 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:11:01.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.229+0000 7f6f691c3740 20 client.15387 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:01.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.229+0000 7f6f691c3740 1 --2- 172.21.15.28:0/2161727027 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x556a7139e5c0 0x556a713a09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.229+0000 7f6f691c3740 4 client.15387 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:11:01.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.229+0000 7f6f691c3740 1 -- 172.21.15.28:0/2161727027 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "damage ls"}) -- 0x556a71289a60 con 0x556a7139e5c0 2024-04-24T08:11:01.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.229+0000 7f6f5f400640 1 --2- 172.21.15.28:0/2161727027 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x556a7139e5c0 0x556a713a09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.230+0000 7f6f5f400640 1 --2- 172.21.15.28:0/2161727027 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x556a7139e5c0 0x556a713a09e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c9c79e1b799ee45 server_cookie=e216c8a68346edad in_seq=0 out_seq=0 2024-04-24T08:11:01.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.230+0000 7f6f4cc00640 10 client.15387 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:11:01.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.230+0000 7f6f4cc00640 1 -- 172.21.15.28:0/2161727027 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x556a71289a60 con 0x556a7139e5c0 2024-04-24T08:11:01.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.230+0000 7f6f4cc00640 10 client.15387 handle_command_reply: tid=0 2024-04-24T08:11:01.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.230+0000 7f6f691c3740 1 client.15387 shutdown 2024-04-24T08:11:01.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.230+0000 7f6f691c3740 1 -- 172.21.15.28:0/2161727027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a7131d270 msgr2=0x556a713218a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.230+0000 7f6f691c3740 1 --2- 172.21.15.28:0/2161727027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a7131d270 0x556a713218a0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f6f48099020 tx=0x7f6f480ce410 comp rx=0 tx=0).stop 2024-04-24T08:11:01.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.230+0000 7f6f691c3740 1 -- 172.21.15.28:0/2161727027 shutdown_connections 2024-04-24T08:11:01.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.230+0000 7f6f691c3740 1 --2- 172.21.15.28:0/2161727027 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x556a7139e5c0 0x556a713a09e0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.230+0000 7f6f691c3740 1 --2- 172.21.15.28:0/2161727027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556a7131d270 0x556a713218a0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.230+0000 7f6f691c3740 1 --2- 172.21.15.28:0/2161727027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556a71327940 0x556a71321de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.230+0000 7f6f691c3740 1 --2- 172.21.15.28:0/2161727027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556a7131c950 0x556a71321360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.230+0000 7f6f691c3740 1 -- 172.21.15.28:0/2161727027 >> 172.21.15.28:0/2161727027 conn(0x556a71318c70 msgr2=0x556a7131e0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.230+0000 7f6f4cc00640 0 client.15387 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:11:01.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.230+0000 7f6f691c3740 1 -- 172.21.15.28:0/2161727027 shutdown_connections 2024-04-24T08:11:01.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.230+0000 7f6f691c3740 1 -- 172.21.15.28:0/2161727027 wait complete. 2024-04-24T08:11:01.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.231+0000 7f6f691c3740 20 client.15387 trim_cache size 0 max 16384 2024-04-24T08:11:01.235 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:11:01.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.231+0000 7f6f66200640 1 -- 172.21.15.28:0/3738521363 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6f440a3e40 msgr2=0x7f6f440a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.231+0000 7f6f66200640 1 --2- 172.21.15.28:0/3738521363 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6f440a3e40 0x7f6f440a6300 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x556a7131df10 tx=0x7f6f50005f90 comp rx=0 tx=0).stop 2024-04-24T08:11:01.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.231+0000 7f6f66200640 1 -- 172.21.15.28:0/3738521363 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f60076e50 msgr2=0x7f6f6019f480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.231+0000 7f6f66200640 1 --2- 172.21.15.28:0/3738521363 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f60076e50 0x7f6f6019f480 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f6f4800e9f0 tx=0x7f6f4800eeb0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.231+0000 7f6f66200640 1 -- 172.21.15.28:0/3738521363 shutdown_connections 2024-04-24T08:11:01.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.231+0000 7f6f66200640 1 --2- 172.21.15.28:0/3738521363 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6f440a3e40 0x7f6f440a6300 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.231+0000 7f6f66200640 1 --2- 172.21.15.28:0/3738521363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6f60112ee0 0x7f6f601a3800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.231+0000 7f6f66200640 1 --2- 172.21.15.28:0/3738521363 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f60076e50 0x7f6f6019f480 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.231+0000 7f6f66200640 1 --2- 172.21.15.28:0/3738521363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6f60076530 0x7f6f6019ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.231+0000 7f6f66200640 1 -- 172.21.15.28:0/3738521363 >> 172.21.15.28:0/3738521363 conn(0x7f6f600ffd90 msgr2=0x7f6f60114600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.232+0000 7f6f66200640 1 -- 172.21.15.28:0/3738521363 shutdown_connections 2024-04-24T08:11:01.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.232+0000 7f6f66200640 1 -- 172.21.15.28:0/3738521363 wait complete. 2024-04-24T08:11:01.247 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:11:00 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:11:00.841+0000 7f8b5ca00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:11:01.247 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:11:00 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:11:00.841+0000 7f8b5ca00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:11:01.247 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:11:00 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:11:00.841+0000 7f8b5ca00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:11:01.248 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:11:00 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:11:00.841+0000 7f8b5ca00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:11:01.248 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:11:00 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:11:00.841+0000 7f8b5ca00700 -1 client.0 error registering admin socket command: (17) File exists 2024-04-24T08:11:01.251 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:11:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.452+0000 7f6a6de00640 1 -- 172.21.15.28:0/997050166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a68107300 msgr2=0x7f6a6810eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.452+0000 7f6a6de00640 1 --2- 172.21.15.28:0/997050166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a68107300 0x7f6a6810eed0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009be0 tx=0x7f6a6002a6c0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.452+0000 7f6a6de00640 1 -- 172.21.15.28:0/997050166 shutdown_connections 2024-04-24T08:11:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.452+0000 7f6a6de00640 1 --2- 172.21.15.28:0/997050166 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a68102be0 0x7f6a68106c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.452+0000 7f6a6de00640 1 --2- 172.21.15.28:0/997050166 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a68107300 0x7f6a6810eed0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.452+0000 7f6a6de00640 1 --2- 172.21.15.28:0/997050166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a68102230 0x7f6a68102610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.452+0000 7f6a6de00640 1 -- 172.21.15.28:0/997050166 >> 172.21.15.28:0/997050166 conn(0x7f6a680fdab0 msgr2=0x7f6a680ffed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.452+0000 7f6a6de00640 1 -- 172.21.15.28:0/997050166 shutdown_connections 2024-04-24T08:11:01.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.452+0000 7f6a6de00640 1 -- 172.21.15.28:0/997050166 wait complete. 2024-04-24T08:11:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.453+0000 7f6a6de00640 1 Processor -- start 2024-04-24T08:11:01.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.454+0000 7f6a6de00640 1 -- start start 2024-04-24T08:11:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.454+0000 7f6a6de00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a68102230 0x7f6a68074300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.454+0000 7f6a6de00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a68107300 0x7f6a68074840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.454+0000 7f6a6de00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a68074d80 0x7f6a681a6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.454+0000 7f6a6de00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a681123a0 con 0x7f6a68102230 2024-04-24T08:11:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.454+0000 7f6a6de00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6a68112220 con 0x7f6a68074d80 2024-04-24T08:11:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.454+0000 7f6a6de00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6a68112520 con 0x7f6a68107300 2024-04-24T08:11:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.454+0000 7f6a66a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a68102230 0x7f6a68074300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.455+0000 7f6a66a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a68102230 0x7f6a68074300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56058/0 (socket says 172.21.15.28:56058) 2024-04-24T08:11:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.455+0000 7f6a66a00640 1 -- 172.21.15.28:0/1317229575 learned_addr learned my addr 172.21.15.28:0/1317229575 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.455+0000 7f6a67400640 1 --2- 172.21.15.28:0/1317229575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a68074d80 0x7f6a681a6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.455+0000 7f6a66000640 1 --2- 172.21.15.28:0/1317229575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a68107300 0x7f6a68074840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.455+0000 7f6a66a00640 1 -- 172.21.15.28:0/1317229575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a68107300 msgr2=0x7f6a68074840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.455+0000 7f6a66a00640 1 --2- 172.21.15.28:0/1317229575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a68107300 0x7f6a68074840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.455+0000 7f6a66a00640 1 -- 172.21.15.28:0/1317229575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a68074d80 msgr2=0x7f6a681a6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.455+0000 7f6a66a00640 1 --2- 172.21.15.28:0/1317229575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a68074d80 0x7f6a681a6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.455+0000 7f6a66a00640 1 -- 172.21.15.28:0/1317229575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a681a7170 con 0x7f6a68102230 2024-04-24T08:11:01.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.455+0000 7f6a66a00640 1 --2- 172.21.15.28:0/1317229575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a68102230 0x7f6a68074300 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800e9f0 tx=0x7f6a5800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.456+0000 7f6a6ca00640 1 -- 172.21.15.28:0/1317229575 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a5800cd80 con 0x7f6a68102230 2024-04-24T08:11:01.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.456+0000 7f6a6ca00640 1 -- 172.21.15.28:0/1317229575 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a58010910 con 0x7f6a68102230 2024-04-24T08:11:01.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.456+0000 7f6a6de00640 1 -- 172.21.15.28:0/1317229575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a681a7460 con 0x7f6a68102230 2024-04-24T08:11:01.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.456+0000 7f6a6ca00640 1 -- 172.21.15.28:0/1317229575 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a580188f0 con 0x7f6a68102230 2024-04-24T08:11:01.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.456+0000 7f6a6de00640 1 -- 172.21.15.28:0/1317229575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a681a7a00 con 0x7f6a68102230 2024-04-24T08:11:01.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.460+0000 7f6a70d99740 1 Processor -- start 2024-04-24T08:11:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.460+0000 7f6a70d99740 1 -- start start 2024-04-24T08:11:01.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.461+0000 7f6a6ca00640 1 -- 172.21.15.28:0/1317229575 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7f6a58010ac0 con 0x7f6a68102230 2024-04-24T08:11:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.461+0000 7f6a70d99740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcf3950 0x55ba5dcf3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.461+0000 7f6a70d99740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf4270 0x55ba5dcfe400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.461+0000 7f6a70d99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcfe940 0x55ba5dd00d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.461+0000 7f6a70d99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ba5dc591d0 con 0x55ba5dcfe940 2024-04-24T08:11:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.461+0000 7f6a70d99740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ba5dbff450 con 0x55ba5dcf4270 2024-04-24T08:11:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.461+0000 7f6a70d99740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ba5dc55190 con 0x55ba5dcf3950 2024-04-24T08:11:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.461+0000 7f6a66a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf4270 0x55ba5dcfe400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.461+0000 7f6a67400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcf3950 0x55ba5dcf3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.461+0000 7f6a66a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf4270 0x55ba5dcfe400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:45604/0 (socket says 172.21.15.28:45604) 2024-04-24T08:11:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.461+0000 7f6a66a00640 1 -- 172.21.15.28:0/3340712748 learned_addr learned my addr 172.21.15.28:0/3340712748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:01.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.461+0000 7f6a66000640 1 --2- 172.21.15.28:0/3340712748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcfe940 0x55ba5dd00d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.462+0000 7f6a67400640 1 -- 172.21.15.28:0/3340712748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf4270 msgr2=0x55ba5dcfe400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.462+0000 7f6a67400640 1 --2- 172.21.15.28:0/3340712748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf4270 0x55ba5dcfe400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.462+0000 7f6a67400640 1 -- 172.21.15.28:0/3340712748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcfe940 msgr2=0x55ba5dd00d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.462+0000 7f6a67400640 1 --2- 172.21.15.28:0/3340712748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcfe940 0x55ba5dd00d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.462+0000 7f6a67400640 1 -- 172.21.15.28:0/3340712748 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba5dcbc310 con 0x55ba5dcf3950 2024-04-24T08:11:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.462+0000 7f6a66a00640 1 --2- 172.21.15.28:0/3340712748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf4270 0x55ba5dcfe400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.462+0000 7f6a67400640 1 --2- 172.21.15.28:0/3340712748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcf3950 0x55ba5dcf3d30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009b30 tx=0x7f6a60030a30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=575b0af4d6edea06 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.463+0000 7f6a53e00640 1 -- 172.21.15.28:0/3340712748 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60041030 con 0x55ba5dcf3950 2024-04-24T08:11:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.463+0000 7f6a53e00640 1 -- 172.21.15.28:0/3340712748 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a60030c60 con 0x55ba5dcf3950 2024-04-24T08:11:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.463+0000 7f6a53e00640 1 -- 172.21.15.28:0/3340712748 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60036040 con 0x55ba5dcf3950 2024-04-24T08:11:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.463+0000 7f6a6ca00640 1 --2- 172.21.15.28:0/1317229575 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6a4c0a3ef0 0x7f6a4c0a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.463+0000 7f6a6ca00640 1 -- 172.21.15.28:0/1317229575 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f6a580d0de0 con 0x7f6a68102230 2024-04-24T08:11:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.463+0000 7f6a66a00640 1 --2- 172.21.15.28:0/1317229575 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6a4c0a3ef0 0x7f6a4c0a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.464+0000 7f6a70d99740 1 -- 172.21.15.28:0/3340712748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcf3950 msgr2=0x55ba5dcf3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.464+0000 7f6a70d99740 1 --2- 172.21.15.28:0/3340712748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcf3950 0x55ba5dcf3d30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009b30 tx=0x7f6a60030a30 comp rx=0 tx=0).stop 2024-04-24T08:11:01.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.464+0000 7f6a70d99740 1 -- 172.21.15.28:0/3340712748 shutdown_connections 2024-04-24T08:11:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.464+0000 7f6a70d99740 1 --2- 172.21.15.28:0/3340712748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcfe940 0x55ba5dd00d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.464+0000 7f6a70d99740 1 --2- 172.21.15.28:0/3340712748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf4270 0x55ba5dcfe400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.464+0000 7f6a70d99740 1 --2- 172.21.15.28:0/3340712748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcf3950 0x55ba5dcf3d30 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.464+0000 7f6a70d99740 1 -- 172.21.15.28:0/3340712748 >> 172.21.15.28:0/3340712748 conn(0x55ba5dcefc70 msgr2=0x55ba5dcf0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.464+0000 7f6a70d99740 1 -- 172.21.15.28:0/3340712748 shutdown_connections 2024-04-24T08:11:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.464+0000 7f6a70d99740 1 -- 172.21.15.28:0/3340712748 wait complete. 2024-04-24T08:11:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.464+0000 7f6a66a00640 1 --2- 172.21.15.28:0/1317229575 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6a4c0a3ef0 0x7f6a4c0a63b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6a580ceb10 tx=0x7f6a580d48f0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.464+0000 7f6a70d99740 1 -- start start 2024-04-24T08:11:01.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.465+0000 7f6a70d99740 1 Processor -- start 2024-04-24T08:11:01.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.465+0000 7f6a70d99740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dcfe320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.465+0000 7f6a70d99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcf4270 0x55ba5dcfb680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.465+0000 7f6a70d99740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcfe940 0x55ba5dcfbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.465+0000 7f6a70d99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ba5dd01770 con 0x55ba5dcf4270 2024-04-24T08:11:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.465+0000 7f6a70d99740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ba5dc12810 con 0x55ba5dcf3950 2024-04-24T08:11:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.465+0000 7f6a70d99740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ba5dd018f0 con 0x55ba5dcfe940 2024-04-24T08:11:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.465+0000 7f6a70d99740 10 client.0 _ll_register_callbacks cb 0x55ba5dbe0140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.465+0000 7f6a66000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcf4270 0x55ba5dcfb680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.465+0000 7f6a67400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dcfe320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.465+0000 7f6a67400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dcfe320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:45618/0 (socket says 172.21.15.28:45618) 2024-04-24T08:11:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.465+0000 7f6a67400640 1 -- 172.21.15.28:0/4250976120 learned_addr learned my addr 172.21.15.28:0/4250976120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.466+0000 7f6a67400640 1 --2- 172.21.15.28:0/4250976120 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcfe940 0x55ba5dcfbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.466+0000 7f6a67400640 1 -- 172.21.15.28:0/4250976120 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcfe940 msgr2=0x55ba5dcfbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.466+0000 7f6a67400640 1 --2- 172.21.15.28:0/4250976120 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcfe940 0x55ba5dcfbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.466+0000 7f6a67400640 1 -- 172.21.15.28:0/4250976120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcf4270 msgr2=0x55ba5dcfb680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.466+0000 7f6a67400640 1 --2- 172.21.15.28:0/4250976120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcf4270 0x55ba5dcfb680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.466+0000 7f6a67400640 1 -- 172.21.15.28:0/4250976120 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba5dc233f0 con 0x55ba5dcf3950 2024-04-24T08:11:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.466+0000 7f6a67400640 1 --2- 172.21.15.28:0/4250976120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dcfe320 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6a60034040 tx=0x7f6a60009300 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5f617eef7624a82a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.466+0000 7f6a50c00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:11:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.466+0000 7f6a50c00640 1 -- 172.21.15.28:0/4250976120 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60051060 con 0x55ba5dcf3950 2024-04-24T08:11:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.466+0000 7f6a50c00640 1 -- 172.21.15.28:0/4250976120 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f6a60003b80 con 0x55ba5dcf3950 2024-04-24T08:11:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.466+0000 7f6a70d99740 1 -- 172.21.15.28:0/4250976120 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a60030c60 con 0x55ba5dcf3950 2024-04-24T08:11:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.467+0000 7f6a50c00640 1 -- 172.21.15.28:0/4250976120 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6a60041030 con 0x55ba5dcf3950 2024-04-24T08:11:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.467+0000 7f6a70d99740 10 client.25496 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.467+0000 7f6a70d99740 1 -- 172.21.15.28:0/4250976120 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ba5d96b3f0 con 0x55ba5dcf3950 2024-04-24T08:11:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.467+0000 7f6a50c00640 1 -- 172.21.15.28:0/4250976120 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a60030c60 con 0x55ba5dcf3950 2024-04-24T08:11:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.467+0000 7f6a50c00640 1 -- 172.21.15.28:0/4250976120 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f6a60055480 con 0x55ba5dcf3950 2024-04-24T08:11:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.467+0000 7f6a70d99740 10 client.25496 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.467+0000 7f6a70d99740 10 client.25496 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:11:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.467+0000 7f6a70d99740 20 client.25496 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:01.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.467+0000 7f6a70d99740 1 --2- 172.21.15.28:0/4250976120 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55ba5dd0e950 0x55ba5dd10d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.467+0000 7f6a70d99740 4 client.25496 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:11:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.467+0000 7f6a70d99740 1 -- 172.21.15.28:0/4250976120 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ba5dc66bf0 con 0x55ba5dd0e950 2024-04-24T08:11:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.468+0000 7f6a66000640 1 --2- 172.21.15.28:0/4250976120 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55ba5dd0e950 0x55ba5dd10d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.468+0000 7f6a66000640 1 --2- 172.21.15.28:0/4250976120 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55ba5dd0e950 0x55ba5dd10d40 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a86ca850498e5e5 server_cookie=b389038a826daae2 in_seq=0 out_seq=0 2024-04-24T08:11:01.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.468+0000 7f6a50c00640 10 client.25496 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:11:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.469+0000 7f6a50c00640 1 -- 172.21.15.28:0/4250976120 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ba5dc66bf0 con 0x55ba5dd0e950 2024-04-24T08:11:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.469+0000 7f6a50c00640 10 client.25496 handle_command_reply: tid=0 2024-04-24T08:11:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.469+0000 7f6a70d99740 1 client.25496 shutdown 2024-04-24T08:11:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.470+0000 7f6a70d99740 1 -- 172.21.15.28:0/4250976120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 msgr2=0x55ba5dcfe320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.470+0000 7f6a70d99740 1 --2- 172.21.15.28:0/4250976120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dcfe320 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f6a60034040 tx=0x7f6a60009300 comp rx=0 tx=0).stop 2024-04-24T08:11:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.470+0000 7f6a70d99740 1 -- 172.21.15.28:0/4250976120 shutdown_connections 2024-04-24T08:11:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.470+0000 7f6a70d99740 1 --2- 172.21.15.28:0/4250976120 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55ba5dd0e950 0x55ba5dd10d40 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.470+0000 7f6a70d99740 1 --2- 172.21.15.28:0/4250976120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcf4270 0x55ba5dcfb680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.470+0000 7f6a70d99740 1 --2- 172.21.15.28:0/4250976120 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcfe940 0x55ba5dcfbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.470+0000 7f6a70d99740 1 --2- 172.21.15.28:0/4250976120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dcfe320 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.470+0000 7f6a70d99740 1 -- 172.21.15.28:0/4250976120 >> 172.21.15.28:0/4250976120 conn(0x55ba5dcefc70 msgr2=0x55ba5dcf5a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.470+0000 7f6a50c00640 0 client.25496 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:11:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.470+0000 7f6a70d99740 1 -- 172.21.15.28:0/4250976120 shutdown_connections 2024-04-24T08:11:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.470+0000 7f6a70d99740 1 -- 172.21.15.28:0/4250976120 wait complete. 2024-04-24T08:11:01.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.470+0000 7f6a70d99740 20 client.25496 trim_cache size 0 max 16384 2024-04-24T08:11:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.485+0000 7f6a70d99740 1 Processor -- start 2024-04-24T08:11:01.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.485+0000 7f6a70d99740 1 -- start start 2024-04-24T08:11:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.485+0000 7f6a70d99740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dcfae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.485+0000 7f6a70d99740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcf4270 0x55ba5dd0a3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.485+0000 7f6a70d99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcfe940 0x55ba5dcfdec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.485+0000 7f6a70d99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a681b28a0 con 0x55ba5dcfe940 2024-04-24T08:11:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.485+0000 7f6a70d99740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6a681b0170 con 0x55ba5dcf3950 2024-04-24T08:11:01.486 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.485+0000 7f6a70d99740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6a681adfc0 con 0x55ba5dcf4270 2024-04-24T08:11:01.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.486+0000 7f6a67400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcfe940 0x55ba5dcfdec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.486+0000 7f6a67400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dcfae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.486+0000 7f6a66000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcf4270 0x55ba5dd0a3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.486+0000 7f6a67400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dcfae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:45620/0 (socket says 172.21.15.28:45620) 2024-04-24T08:11:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.486+0000 7f6a67400640 1 -- 172.21.15.28:0/3808714598 learned_addr learned my addr 172.21.15.28:0/3808714598 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.486+0000 7f6a67400640 1 -- 172.21.15.28:0/3808714598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcf4270 msgr2=0x55ba5dd0a3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.486+0000 7f6a67400640 1 --2- 172.21.15.28:0/3808714598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcf4270 0x55ba5dd0a3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.486+0000 7f6a67400640 1 -- 172.21.15.28:0/3808714598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcfe940 msgr2=0x55ba5dcfdec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.486+0000 7f6a67400640 1 --2- 172.21.15.28:0/3808714598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcfe940 0x55ba5dcfdec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.486+0000 7f6a67400640 1 -- 172.21.15.28:0/3808714598 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ba5da357d0 con 0x55ba5dcf3950 2024-04-24T08:11:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.486+0000 7f6a67400640 1 --2- 172.21.15.28:0/3808714598 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dcfae30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x55ba5dcf0dd0 tx=0x7f6a60000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c17ec6f3ce98054 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.487+0000 7f6a53400640 1 -- 172.21.15.28:0/3808714598 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60036040 con 0x55ba5dcf3950 2024-04-24T08:11:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.487+0000 7f6a53400640 1 -- 172.21.15.28:0/3808714598 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a60030c60 con 0x55ba5dcf3950 2024-04-24T08:11:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.487+0000 7f6a53400640 1 -- 172.21.15.28:0/3808714598 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a600555c0 con 0x55ba5dcf3950 2024-04-24T08:11:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.487+0000 7f6a70d99740 1 -- 172.21.15.28:0/3808714598 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 msgr2=0x55ba5dcfae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.487+0000 7f6a70d99740 1 --2- 172.21.15.28:0/3808714598 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dcfae30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x55ba5dcf0dd0 tx=0x7f6a60000f40 comp rx=0 tx=0).stop 2024-04-24T08:11:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.487+0000 7f6a70d99740 1 -- 172.21.15.28:0/3808714598 shutdown_connections 2024-04-24T08:11:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.487+0000 7f6a70d99740 1 --2- 172.21.15.28:0/3808714598 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcfe940 0x55ba5dcfdec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.487+0000 7f6a70d99740 1 --2- 172.21.15.28:0/3808714598 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcf4270 0x55ba5dd0a3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.487+0000 7f6a70d99740 1 --2- 172.21.15.28:0/3808714598 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dcfae30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.487+0000 7f6a70d99740 1 -- 172.21.15.28:0/3808714598 >> 172.21.15.28:0/3808714598 conn(0x55ba5dcefc70 msgr2=0x55ba5dcf0050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.487+0000 7f6a70d99740 1 -- 172.21.15.28:0/3808714598 shutdown_connections 2024-04-24T08:11:01.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.487+0000 7f6a70d99740 1 -- 172.21.15.28:0/3808714598 wait complete. 2024-04-24T08:11:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.488+0000 7f6a70d99740 1 -- start start 2024-04-24T08:11:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.488+0000 7f6a70d99740 1 Processor -- start 2024-04-24T08:11:01.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.488+0000 7f6a70d99740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dd14160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.488+0000 7f6a70d99740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcf4270 0x55ba5dd0f4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.488+0000 7f6a70d99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcfe940 0x55ba5dd0fa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.488+0000 7f6a70d99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6a681ae630 con 0x55ba5dcfe940 2024-04-24T08:11:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.488+0000 7f6a70d99740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6a681afaa0 con 0x55ba5dcf3950 2024-04-24T08:11:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.488+0000 7f6a70d99740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6a681aed00 con 0x55ba5dcf4270 2024-04-24T08:11:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.489+0000 7f6a70d99740 10 client.0 _ll_register_callbacks cb 0x7f6a54009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.489+0000 7f6a67400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dd14160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.489+0000 7f6a67400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dd14160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:45622/0 (socket says 172.21.15.28:45622) 2024-04-24T08:11:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.489+0000 7f6a67400640 1 -- 172.21.15.28:0/986781221 learned_addr learned my addr 172.21.15.28:0/986781221 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.489+0000 7f6a67400640 1 --2- 172.21.15.28:0/986781221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcfe940 0x55ba5dd0fa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.489+0000 7f6a66000640 1 --2- 172.21.15.28:0/986781221 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcf4270 0x55ba5dd0f4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.489+0000 7f6a67400640 1 -- 172.21.15.28:0/986781221 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcf4270 msgr2=0x55ba5dd0f4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.489+0000 7f6a67400640 1 --2- 172.21.15.28:0/986781221 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcf4270 0x55ba5dd0f4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.489+0000 7f6a67400640 1 -- 172.21.15.28:0/986781221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcfe940 msgr2=0x55ba5dd0fa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.489+0000 7f6a67400640 1 --2- 172.21.15.28:0/986781221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcfe940 0x55ba5dd0fa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.489+0000 7f6a67400640 1 -- 172.21.15.28:0/986781221 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ba5dd10210 con 0x55ba5dcf3950 2024-04-24T08:11:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.490+0000 7f6a67400640 1 --2- 172.21.15.28:0/986781221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dd14160 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009920 tx=0x7f6a60005cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5415c9a3f47c9241 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.490+0000 7f6a50c00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:11:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.490+0000 7f6a50c00640 1 -- 172.21.15.28:0/986781221 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a60036070 con 0x55ba5dcf3950 2024-04-24T08:11:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.490+0000 7f6a70d99740 1 -- 172.21.15.28:0/986781221 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a60030c60 con 0x55ba5dcf3950 2024-04-24T08:11:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.490+0000 7f6a50c00640 1 -- 172.21.15.28:0/986781221 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f6a600043b0 con 0x55ba5dcf3950 2024-04-24T08:11:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.490+0000 7f6a50c00640 1 -- 172.21.15.28:0/986781221 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f6a60055cd0 con 0x55ba5dcf3950 2024-04-24T08:11:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.490+0000 7f6a70d99740 10 client.25502 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.490+0000 7f6a70d99740 1 -- 172.21.15.28:0/986781221 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55ba5dd74b50 con 0x55ba5dcf3950 2024-04-24T08:11:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.490+0000 7f6a50c00640 1 -- 172.21.15.28:0/986781221 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a60030c60 con 0x55ba5dcf3950 2024-04-24T08:11:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.491+0000 7f6a50c00640 1 -- 172.21.15.28:0/986781221 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f6a6003d580 con 0x55ba5dcf3950 2024-04-24T08:11:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.491+0000 7f6a70d99740 10 client.25502 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.491+0000 7f6a70d99740 10 client.25502 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:11:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.491+0000 7f6a70d99740 20 client.25502 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.491+0000 7f6a70d99740 1 --2- 172.21.15.28:0/986781221 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55ba5dd75110 0x55ba5dd77500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.491+0000 7f6a70d99740 4 client.25502 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:11:01.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.491+0000 7f6a70d99740 1 -- 172.21.15.28:0/986781221 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ba5dc60a60 con 0x55ba5dd75110 2024-04-24T08:11:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.491+0000 7f6a66000640 1 --2- 172.21.15.28:0/986781221 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55ba5dd75110 0x55ba5dd77500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.492+0000 7f6a66000640 1 --2- 172.21.15.28:0/986781221 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55ba5dd75110 0x55ba5dd77500 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8149f87e398c9e95 server_cookie=da4c9ff450073 in_seq=0 out_seq=0 2024-04-24T08:11:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.492+0000 7f6a50c00640 10 client.25502 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:11:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.492+0000 7f6a50c00640 1 -- 172.21.15.28:0/986781221 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ba5dc60a60 con 0x55ba5dd75110 2024-04-24T08:11:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.492+0000 7f6a50c00640 10 client.25502 handle_command_reply: tid=0 2024-04-24T08:11:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.492+0000 7f6a70d99740 1 client.25502 shutdown 2024-04-24T08:11:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.492+0000 7f6a70d99740 1 -- 172.21.15.28:0/986781221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 msgr2=0x55ba5dd14160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.492+0000 7f6a70d99740 1 --2- 172.21.15.28:0/986781221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dd14160 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f6a60009920 tx=0x7f6a60005cf0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.492+0000 7f6a70d99740 1 -- 172.21.15.28:0/986781221 shutdown_connections 2024-04-24T08:11:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.492+0000 7f6a70d99740 1 --2- 172.21.15.28:0/986781221 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55ba5dd75110 0x55ba5dd77500 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.492+0000 7f6a70d99740 1 --2- 172.21.15.28:0/986781221 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ba5dcfe940 0x55ba5dd0fa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.492+0000 7f6a50c00640 0 client.25502 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:11:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.492+0000 7f6a70d99740 1 --2- 172.21.15.28:0/986781221 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ba5dcf4270 0x55ba5dd0f4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.492+0000 7f6a70d99740 1 --2- 172.21.15.28:0/986781221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ba5dcf3950 0x55ba5dd14160 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.492+0000 7f6a70d99740 1 -- 172.21.15.28:0/986781221 >> 172.21.15.28:0/986781221 conn(0x55ba5dcefc70 msgr2=0x55ba5dd0a3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.492+0000 7f6a70d99740 1 -- 172.21.15.28:0/986781221 shutdown_connections 2024-04-24T08:11:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.493+0000 7f6a70d99740 1 -- 172.21.15.28:0/986781221 wait complete. 2024-04-24T08:11:01.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.493+0000 7f6a70d99740 20 client.25502 trim_cache size 0 max 16384 2024-04-24T08:11:01.496 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:11:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.493+0000 7f6a6de00640 1 -- 172.21.15.28:0/1317229575 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6a4c0a3ef0 msgr2=0x7f6a4c0a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.493+0000 7f6a6de00640 1 --2- 172.21.15.28:0/1317229575 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6a4c0a3ef0 0x7f6a4c0a63b0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f6a580ceb10 tx=0x7f6a580d48f0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.493+0000 7f6a6de00640 1 -- 172.21.15.28:0/1317229575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a68102230 msgr2=0x7f6a68074300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.493+0000 7f6a6de00640 1 --2- 172.21.15.28:0/1317229575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a68102230 0x7f6a68074300 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6a5800e9f0 tx=0x7f6a5800eeb0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.493+0000 7f6a6de00640 1 -- 172.21.15.28:0/1317229575 shutdown_connections 2024-04-24T08:11:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.493+0000 7f6a6de00640 1 --2- 172.21.15.28:0/1317229575 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6a4c0a3ef0 0x7f6a4c0a63b0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.493+0000 7f6a6de00640 1 --2- 172.21.15.28:0/1317229575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a68074d80 0x7f6a681a6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.493+0000 7f6a6de00640 1 --2- 172.21.15.28:0/1317229575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6a68107300 0x7f6a68074840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.493+0000 7f6a6de00640 1 --2- 172.21.15.28:0/1317229575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6a68102230 0x7f6a68074300 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.493+0000 7f6a6de00640 1 -- 172.21.15.28:0/1317229575 >> 172.21.15.28:0/1317229575 conn(0x7f6a680fdab0 msgr2=0x7f6a680ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.494+0000 7f6a6de00640 1 -- 172.21.15.28:0/1317229575 shutdown_connections 2024-04-24T08:11:01.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.494+0000 7f6a6de00640 1 -- 172.21.15.28:0/1317229575 wait complete. 2024-04-24T08:11:01.508 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:01 smithi028 ceph-mon[27786]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:01.509 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:01 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1811413712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:11:01.512 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:11:01.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:01 smithi117 ceph-mon[32438]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:01.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:01 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1811413712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:11:01.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:01 smithi194 ceph-mon[32836]: pgmap v23: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:01.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:01 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1811413712' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:11:01.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.732+0000 7f2f9b200640 1 -- 172.21.15.28:0/4241083492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f9c103060 msgr2=0x7f2f9c103440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.732+0000 7f2f9b200640 1 --2- 172.21.15.28:0/4241083492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f9c103060 0x7f2f9c103440 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c009860 tx=0x7f2f8c02a3d0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.732+0000 7f2f9b200640 1 -- 172.21.15.28:0/4241083492 shutdown_connections 2024-04-24T08:11:01.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.732+0000 7f2f9b200640 1 --2- 172.21.15.28:0/4241083492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f9c10c480 0x7f2f9c10e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.732+0000 7f2f9b200640 1 --2- 172.21.15.28:0/4241083492 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f9c103980 0x7f2f9c10be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.732+0000 7f2f9b200640 1 --2- 172.21.15.28:0/4241083492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f9c103060 0x7f2f9c103440 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.732+0000 7f2f9b200640 1 -- 172.21.15.28:0/4241083492 >> 172.21.15.28:0/4241083492 conn(0x7f2f9c0fc880 msgr2=0x7f2f9c0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.732+0000 7f2f9b200640 1 -- 172.21.15.28:0/4241083492 shutdown_connections 2024-04-24T08:11:01.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.733+0000 7f2f9b200640 1 -- 172.21.15.28:0/4241083492 wait complete. 2024-04-24T08:11:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.733+0000 7f2f9b200640 1 Processor -- start 2024-04-24T08:11:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.734+0000 7f2f9b200640 1 -- start start 2024-04-24T08:11:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.734+0000 7f2f9b200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f9c103060 0x7f2f9c1a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.734+0000 7f2f9b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f9c103980 0x7f2f9c1a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.734+0000 7f2f9b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f9c10c480 0x7f2f9c19a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.734+0000 7f2f9b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c112290 con 0x7f2f9c10c480 2024-04-24T08:11:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.734+0000 7f2f9b200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c112110 con 0x7f2f9c103060 2024-04-24T08:11:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.734+0000 7f2f9b200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c112410 con 0x7f2f9c103980 2024-04-24T08:11:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.734+0000 7f2f99e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f9c103060 0x7f2f9c1a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.734+0000 7f2f9a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f9c10c480 0x7f2f9c19a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.734+0000 7f2f99e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f9c103060 0x7f2f9c1a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:45634/0 (socket says 172.21.15.28:45634) 2024-04-24T08:11:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.735+0000 7f2f99e00640 1 -- 172.21.15.28:0/347636070 learned_addr learned my addr 172.21.15.28:0/347636070 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:01.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.734+0000 7f2f99400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f9c103980 0x7f2f9c1a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.735+0000 7f2f99e00640 1 -- 172.21.15.28:0/347636070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f9c103980 msgr2=0x7f2f9c1a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.735+0000 7f2f99e00640 1 --2- 172.21.15.28:0/347636070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f9c103980 0x7f2f9c1a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.735+0000 7f2f99e00640 1 -- 172.21.15.28:0/347636070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f9c10c480 msgr2=0x7f2f9c19a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.735+0000 7f2f99e00640 1 --2- 172.21.15.28:0/347636070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f9c10c480 0x7f2f9c19a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.735+0000 7f2f99e00640 1 -- 172.21.15.28:0/347636070 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2f9c19b0b0 con 0x7f2f9c103060 2024-04-24T08:11:01.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.735+0000 7f2f99400640 1 --2- 172.21.15.28:0/347636070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f9c103980 0x7f2f9c1a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:01.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.735+0000 7f2f9a800640 1 --2- 172.21.15.28:0/347636070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f9c10c480 0x7f2f9c19a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:01.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.735+0000 7f2f99e00640 1 --2- 172.21.15.28:0/347636070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f9c103060 0x7f2f9c1a0690 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c009980 tx=0x7f2f8c0309a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.736+0000 7f2f92a00640 1 -- 172.21.15.28:0/347636070 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c040070 con 0x7f2f9c103060 2024-04-24T08:11:01.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.736+0000 7f2f9b200640 1 -- 172.21.15.28:0/347636070 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2f9c19b340 con 0x7f2f9c103060 2024-04-24T08:11:01.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.736+0000 7f2f92a00640 1 -- 172.21.15.28:0/347636070 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f8c005620 con 0x7f2f9c103060 2024-04-24T08:11:01.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.736+0000 7f2f92a00640 1 -- 172.21.15.28:0/347636070 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f8c0049b0 con 0x7f2f9c103060 2024-04-24T08:11:01.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.736+0000 7f2f9b200640 1 -- 172.21.15.28:0/347636070 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2f9c1a7550 con 0x7f2f9c103060 2024-04-24T08:11:01.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.739+0000 7f2fa3fbe740 1 Processor -- start 2024-04-24T08:11:01.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.739+0000 7f2fa3fbe740 1 -- start start 2024-04-24T08:11:01.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.739+0000 7f2fa3fbe740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bab40 0x55dcd37baf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.739+0000 7f2fa3fbe740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37bb460 0x55dcd37c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.739+0000 7f2fa3fbe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37c5b30 0x55dcd37c7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.739+0000 7f2fa3fbe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dcd37201d0 con 0x55dcd37c5b30 2024-04-24T08:11:01.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.739+0000 7f2fa3fbe740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55dcd36c6450 con 0x55dcd37bb460 2024-04-24T08:11:01.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.739+0000 7f2fa3fbe740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55dcd371c190 con 0x55dcd37bab40 2024-04-24T08:11:01.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.739+0000 7f2f92a00640 1 -- 172.21.15.28:0/347636070 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 25) ==== 138873+0+0 (secure 0 0 0) 0x7f2f8c043070 con 0x7f2f9c103060 2024-04-24T08:11:01.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.740+0000 7f2f9a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bab40 0x55dcd37baf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.740+0000 7f2f99400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37c5b30 0x55dcd37c7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.740+0000 7f2f9a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bab40 0x55dcd37baf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53072/0 (socket says 172.21.15.28:53072) 2024-04-24T08:11:01.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.740+0000 7f2f9a800640 1 -- 172.21.15.28:0/3201594714 learned_addr learned my addr 172.21.15.28:0/3201594714 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:01.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.740+0000 7f2f99e00640 1 --2- 172.21.15.28:0/3201594714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37bb460 0x55dcd37c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.740+0000 7f2f9a800640 1 -- 172.21.15.28:0/3201594714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37bb460 msgr2=0x55dcd37c55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.740+0000 7f2f9a800640 1 --2- 172.21.15.28:0/3201594714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37bb460 0x55dcd37c55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.740+0000 7f2f9a800640 1 -- 172.21.15.28:0/3201594714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37c5b30 msgr2=0x55dcd37c7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.740+0000 7f2f9a800640 1 --2- 172.21.15.28:0/3201594714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37c5b30 0x55dcd37c7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.740+0000 7f2f9a800640 1 -- 172.21.15.28:0/3201594714 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcd3783310 con 0x55dcd37bab40 2024-04-24T08:11:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.741+0000 7f2f9a800640 1 --2- 172.21.15.28:0/3201594714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bab40 0x55dcd37baf20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400e870 tx=0x7f2f940095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2674968746bb42c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.741+0000 7f2f87400640 1 -- 172.21.15.28:0/3201594714 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f9400ddd0 con 0x55dcd37bab40 2024-04-24T08:11:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.741+0000 7f2f87400640 1 -- 172.21.15.28:0/3201594714 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f9401c070 con 0x55dcd37bab40 2024-04-24T08:11:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.741+0000 7f2f87400640 1 -- 172.21.15.28:0/3201594714 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f940165e0 con 0x55dcd37bab40 2024-04-24T08:11:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.741+0000 7f2f92a00640 1 --2- 172.21.15.28:0/347636070 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2f800a3e40 0x7f2f800a6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.741+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/3201594714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bab40 msgr2=0x55dcd37baf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.741+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/3201594714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bab40 0x55dcd37baf20 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f2f9400e870 tx=0x7f2f940095c0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.742+0000 7f2f92a00640 1 -- 172.21.15.28:0/347636070 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f2f8c0f1730 con 0x7f2f9c103060 2024-04-24T08:11:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.742+0000 7f2f99e00640 1 --2- 172.21.15.28:0/347636070 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2f800a3e40 0x7f2f800a6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.742+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/3201594714 shutdown_connections 2024-04-24T08:11:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.742+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/3201594714 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37c5b30 0x55dcd37c7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.742+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/3201594714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37bb460 0x55dcd37c55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.742+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/3201594714 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bab40 0x55dcd37baf20 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.742+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/3201594714 >> 172.21.15.28:0/3201594714 conn(0x55dcd37b6e60 msgr2=0x55dcd37b7240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.742+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/3201594714 shutdown_connections 2024-04-24T08:11:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.742+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/3201594714 wait complete. 2024-04-24T08:11:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.742+0000 7f2fa3fbe740 1 -- start start 2024-04-24T08:11:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.742+0000 7f2f99e00640 1 --2- 172.21.15.28:0/347636070 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2f800a3e40 0x7f2f800a6300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c0eece0 tx=0x7f2f8c02a3d0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.743+0000 7f2fa3fbe740 1 Processor -- start 2024-04-24T08:11:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.743+0000 7f2fa3fbe740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37bab40 0x55dcd37c5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.743+0000 7f2fa3fbe740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bb460 0x55dcd37c2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.743+0000 7f2fa3fbe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37c5b30 0x55dcd37c2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.743+0000 7f2fa3fbe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dcd37c89c0 con 0x55dcd37c5b30 2024-04-24T08:11:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.743+0000 7f2fa3fbe740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55dcd36d9810 con 0x55dcd37bab40 2024-04-24T08:11:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.743+0000 7f2fa3fbe740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55dcd37c8b40 con 0x55dcd37bb460 2024-04-24T08:11:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.743+0000 7f2fa3fbe740 10 client.0 _ll_register_callbacks cb 0x55dcd36a7140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.743+0000 7f2f9a800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37bab40 0x55dcd37c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.743+0000 7f2f99400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bb460 0x55dcd37c2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.743+0000 7f2f9a800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37bab40 0x55dcd37c5580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:45656/0 (socket says 172.21.15.28:45656) 2024-04-24T08:11:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.743+0000 7f2f99400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bb460 0x55dcd37c2870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53074/0 (socket says 172.21.15.28:53074) 2024-04-24T08:11:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.743+0000 7f2f9a800640 1 -- 172.21.15.28:0/783923191 learned_addr learned my addr 172.21.15.28:0/783923191 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.744+0000 7f2f9a800640 1 --2- 172.21.15.28:0/783923191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37c5b30 0x55dcd37c2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.744+0000 7f2f99400640 1 -- 172.21.15.28:0/783923191 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37bab40 msgr2=0x55dcd37c5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.744+0000 7f2f99400640 1 --2- 172.21.15.28:0/783923191 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37bab40 0x55dcd37c5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.744+0000 7f2f99400640 1 -- 172.21.15.28:0/783923191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37c5b30 msgr2=0x55dcd37c2fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.744+0000 7f2f99400640 1 --2- 172.21.15.28:0/783923191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37c5b30 0x55dcd37c2fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.744+0000 7f2f99400640 1 -- 172.21.15.28:0/783923191 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcd36ea3f0 con 0x55dcd37bb460 2024-04-24T08:11:01.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.744+0000 7f2f99400640 1 --2- 172.21.15.28:0/783923191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bb460 0x55dcd37c2870 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0fed50 tx=0x7f2f8800f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7e6346bd560e2fc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.744+0000 7f2f7be00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:11:01.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.745+0000 7f2f7be00640 1 -- 172.21.15.28:0/783923191 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f88012110 con 0x55dcd37bb460 2024-04-24T08:11:01.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.745+0000 7f2f7be00640 1 -- 172.21.15.28:0/783923191 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f2f88004050 con 0x55dcd37bb460 2024-04-24T08:11:01.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.745+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/783923191 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f9401c070 con 0x55dcd37bb460 2024-04-24T08:11:01.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.745+0000 7f2f7be00640 1 -- 172.21.15.28:0/783923191 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2f880122b0 con 0x55dcd37bb460 2024-04-24T08:11:01.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.745+0000 7f2fa3fbe740 10 client.25552 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.745+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/783923191 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dcd34323f0 con 0x55dcd37bb460 2024-04-24T08:11:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.745+0000 7f2f7be00640 1 -- 172.21.15.28:0/783923191 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f9401c070 con 0x55dcd37bb460 2024-04-24T08:11:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.746+0000 7f2f7be00640 1 -- 172.21.15.28:0/783923191 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f2f88010430 con 0x55dcd37bb460 2024-04-24T08:11:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.746+0000 7f2fa3fbe740 10 client.25552 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.746+0000 7f2fa3fbe740 10 client.25552 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:11:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.746+0000 7f2fa3fbe740 20 client.25552 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.746+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/783923191 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55dcd37d5db0 0x55dcd37d81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.746+0000 7f2fa3fbe740 4 client.25552 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:11:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.746+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/783923191 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcd372dbf0 con 0x55dcd37d5db0 2024-04-24T08:11:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.746+0000 7f2f9a800640 1 --2- 172.21.15.28:0/783923191 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55dcd37d5db0 0x55dcd37d81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.747+0000 7f2f9a800640 1 --2- 172.21.15.28:0/783923191 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55dcd37d5db0 0x55dcd37d81a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d5098c3b778866b6 server_cookie=dc0cd4c4ac99c227 in_seq=0 out_seq=0 2024-04-24T08:11:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.747+0000 7f2f7be00640 10 client.25552 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.747+0000 7f2f7be00640 1 -- 172.21.15.28:0/783923191 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dcd372dbf0 con 0x55dcd37d5db0 2024-04-24T08:11:01.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.748+0000 7f2f7be00640 10 client.25552 handle_command_reply: tid=0 2024-04-24T08:11:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.748+0000 7f2fa3fbe740 1 client.25552 shutdown 2024-04-24T08:11:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.748+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/783923191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bb460 msgr2=0x55dcd37c2870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.748+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/783923191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bb460 0x55dcd37c2870 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f2f9c0fed50 tx=0x7f2f8800f790 comp rx=0 tx=0).stop 2024-04-24T08:11:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.748+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/783923191 shutdown_connections 2024-04-24T08:11:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.748+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/783923191 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55dcd37d5db0 0x55dcd37d81a0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.748+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/783923191 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37c5b30 0x55dcd37c2fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.748+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/783923191 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bb460 0x55dcd37c2870 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.748+0000 7f2f7be00640 0 client.25552 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.748+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/783923191 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37bab40 0x55dcd37c5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.748+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/783923191 >> 172.21.15.28:0/783923191 conn(0x55dcd37b6e60 msgr2=0x55dcd37bcc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.748+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/783923191 shutdown_connections 2024-04-24T08:11:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.748+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/783923191 wait complete. 2024-04-24T08:11:01.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.748+0000 7f2fa3fbe740 20 client.25552 trim_cache size 0 max 16384 2024-04-24T08:11:01.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.763+0000 7f2fa3fbe740 1 Processor -- start 2024-04-24T08:11:01.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.763+0000 7f2fa3fbe740 1 -- start start 2024-04-24T08:11:01.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.764+0000 7f2fa3fbe740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37bab40 0x55dcd37d1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.764+0000 7f2fa3fbe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37bb460 0x55dcd37c50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.764+0000 7f2fa3fbe740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37c5b30 0x55dcd37c55f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.764+0000 7f2fa3fbe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c1b2700 con 0x55dcd37bb460 2024-04-24T08:11:01.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.764+0000 7f2fa3fbe740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c1affd0 con 0x55dcd37bab40 2024-04-24T08:11:01.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.764+0000 7f2fa3fbe740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c1ade20 con 0x55dcd37c5b30 2024-04-24T08:11:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.764+0000 7f2f99400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37bb460 0x55dcd37c50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.764+0000 7f2f9a800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37bab40 0x55dcd37d1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.764+0000 7f2f9a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37c5b30 0x55dcd37c55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.764+0000 7f2f9a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37c5b30 0x55dcd37c55f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53080/0 (socket says 172.21.15.28:53080) 2024-04-24T08:11:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.764+0000 7f2f9a800640 1 -- 172.21.15.28:0/1670917606 learned_addr learned my addr 172.21.15.28:0/1670917606 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.765+0000 7f2f9a800640 1 -- 172.21.15.28:0/1670917606 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37bab40 msgr2=0x55dcd37d1880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.765+0000 7f2f9a800640 1 --2- 172.21.15.28:0/1670917606 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37bab40 0x55dcd37d1880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.765+0000 7f2f9a800640 1 -- 172.21.15.28:0/1670917606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37bb460 msgr2=0x55dcd37c50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.765+0000 7f2f9a800640 1 --2- 172.21.15.28:0/1670917606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37bb460 0x55dcd37c50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.765+0000 7f2f9a800640 1 -- 172.21.15.28:0/1670917606 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcd37c3320 con 0x55dcd37c5b30 2024-04-24T08:11:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.765+0000 7f2f9a800640 1 --2- 172.21.15.28:0/1670917606 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37c5b30 0x55dcd37c55f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009400 tx=0x7f2f94000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df3035602f4344d2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.765+0000 7f2f86a00640 1 -- 172.21.15.28:0/1670917606 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f94003970 con 0x55dcd37c5b30 2024-04-24T08:11:01.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.765+0000 7f2f86a00640 1 -- 172.21.15.28:0/1670917606 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f9401c070 con 0x55dcd37c5b30 2024-04-24T08:11:01.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.765+0000 7f2f86a00640 1 -- 172.21.15.28:0/1670917606 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f94005100 con 0x55dcd37c5b30 2024-04-24T08:11:01.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.766+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/1670917606 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37c5b30 msgr2=0x55dcd37c55f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.766+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/1670917606 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37c5b30 0x55dcd37c55f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f2f94009400 tx=0x7f2f94000f40 comp rx=0 tx=0).stop 2024-04-24T08:11:01.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.766+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/1670917606 shutdown_connections 2024-04-24T08:11:01.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.766+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/1670917606 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37bb460 0x55dcd37c50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.766+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/1670917606 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37c5b30 0x55dcd37c55f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.766+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/1670917606 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37bab40 0x55dcd37d1880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.766+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/1670917606 >> 172.21.15.28:0/1670917606 conn(0x55dcd37b6e60 msgr2=0x55dcd37bdb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.766+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/1670917606 shutdown_connections 2024-04-24T08:11:01.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.766+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/1670917606 wait complete. 2024-04-24T08:11:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.766+0000 7f2fa3fbe740 1 -- start start 2024-04-24T08:11:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.767+0000 7f2fa3fbe740 1 Processor -- start 2024-04-24T08:11:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.767+0000 7f2fa3fbe740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bab40 0x55dcd37ce080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.767+0000 7f2fa3fbe740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37bb460 0x55dcd37bfaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.767+0000 7f2fa3fbe740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37c5b30 0x55dcd37c0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.767+0000 7f2fa3fbe740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c1ae490 con 0x55dcd37bb460 2024-04-24T08:11:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.767+0000 7f2fa3fbe740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c1af900 con 0x55dcd37c5b30 2024-04-24T08:11:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.767+0000 7f2fa3fbe740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2f9c1aeb60 con 0x55dcd37bab40 2024-04-24T08:11:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.767+0000 7f2fa3fbe740 10 client.0 _ll_register_callbacks cb 0x7f2f88009cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.767+0000 7f2f99400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37bb460 0x55dcd37bfaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.767+0000 7f2f99400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37bb460 0x55dcd37bfaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56146/0 (socket says 172.21.15.28:56146) 2024-04-24T08:11:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.768+0000 7f2f99400640 1 -- 172.21.15.28:0/2215206310 learned_addr learned my addr 172.21.15.28:0/2215206310 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.768+0000 7f2f9a800640 1 --2- 172.21.15.28:0/2215206310 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bab40 0x55dcd37ce080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.768+0000 7f2f9a800640 1 --2- 172.21.15.28:0/2215206310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37c5b30 0x55dcd37c0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.768+0000 7f2f99400640 1 -- 172.21.15.28:0/2215206310 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bab40 msgr2=0x55dcd37ce080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.768+0000 7f2f99400640 1 --2- 172.21.15.28:0/2215206310 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bab40 0x55dcd37ce080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.768+0000 7f2f99400640 1 -- 172.21.15.28:0/2215206310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37c5b30 msgr2=0x55dcd37c0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.768+0000 7f2f99400640 1 --2- 172.21.15.28:0/2215206310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37c5b30 0x55dcd37c0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.768+0000 7f2f99400640 1 -- 172.21.15.28:0/2215206310 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcd34fc7d0 con 0x55dcd37bb460 2024-04-24T08:11:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.768+0000 7f2f99400640 1 --2- 172.21.15.28:0/2215206310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37bb460 0x55dcd37bfaf0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55dcd37c20d0 tx=0x7f2f8800fd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b94a19fb7aba0073 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.768+0000 7f2f7be00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:11:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.768+0000 7f2f7be00640 1 -- 172.21.15.28:0/2215206310 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2f88032060 con 0x55dcd37bb460 2024-04-24T08:11:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.768+0000 7f2f7be00640 1 -- 172.21.15.28:0/2215206310 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f2f880104f0 con 0x55dcd37bb460 2024-04-24T08:11:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.768+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/2215206310 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2f9401c070 con 0x55dcd37bb460 2024-04-24T08:11:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.769+0000 7f2f7be00640 1 -- 172.21.15.28:0/2215206310 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2f88029040 con 0x55dcd37bb460 2024-04-24T08:11:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.769+0000 7f2fa3fbe740 10 client.15423 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.769+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/2215206310 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55dcd37c7340 con 0x55dcd37bb460 2024-04-24T08:11:01.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.769+0000 7f2f7be00640 1 -- 172.21.15.28:0/2215206310 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2f9401c070 con 0x55dcd37bb460 2024-04-24T08:11:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.769+0000 7f2f7be00640 1 -- 172.21.15.28:0/2215206310 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f2f8802c030 con 0x55dcd37bb460 2024-04-24T08:11:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.769+0000 7f2fa3fbe740 10 client.15423 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.769+0000 7f2fa3fbe740 10 client.15423 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:11:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.769+0000 7f2fa3fbe740 20 client.15423 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.769+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/2215206310 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55dcd383c8a0 0x55dcd383ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.769+0000 7f2fa3fbe740 4 client.15423 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:11:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.769+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/2215206310 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dcd3727a60 con 0x55dcd383c8a0 2024-04-24T08:11:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.770+0000 7f2f9a800640 1 --2- 172.21.15.28:0/2215206310 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55dcd383c8a0 0x55dcd383ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.770+0000 7f2f9a800640 1 --2- 172.21.15.28:0/2215206310 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55dcd383c8a0 0x55dcd383ecc0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=636087dc9f6ec96b server_cookie=e8837a49fd98f807 in_seq=0 out_seq=0 2024-04-24T08:11:01.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.770+0000 7f2f7be00640 10 client.15423 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.770+0000 7f2f7be00640 1 -- 172.21.15.28:0/2215206310 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dcd3727a60 con 0x55dcd383c8a0 2024-04-24T08:11:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.770+0000 7f2f7be00640 10 client.15423 handle_command_reply: tid=0 2024-04-24T08:11:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.770+0000 7f2fa3fbe740 1 client.15423 shutdown 2024-04-24T08:11:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.771+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/2215206310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37bb460 msgr2=0x55dcd37bfaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.771+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/2215206310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37bb460 0x55dcd37bfaf0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x55dcd37c20d0 tx=0x7f2f8800fd20 comp rx=0 tx=0).stop 2024-04-24T08:11:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.771+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/2215206310 shutdown_connections 2024-04-24T08:11:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.771+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/2215206310 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55dcd383c8a0 0x55dcd383ecc0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.771+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/2215206310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dcd37bb460 0x55dcd37bfaf0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.771+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/2215206310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dcd37c5b30 0x55dcd37c0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.771+0000 7f2fa3fbe740 1 --2- 172.21.15.28:0/2215206310 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dcd37bab40 0x55dcd37ce080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.771+0000 7f2f7be00640 0 client.15423 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.771+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/2215206310 >> 172.21.15.28:0/2215206310 conn(0x55dcd37b6e60 msgr2=0x55dcd37c2100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.771+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/2215206310 shutdown_connections 2024-04-24T08:11:01.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.771+0000 7f2fa3fbe740 1 -- 172.21.15.28:0/2215206310 wait complete. 2024-04-24T08:11:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.771+0000 7f2fa3fbe740 20 client.15423 trim_cache size 0 max 16384 2024-04-24T08:11:01.775 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:11:01.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.772+0000 7f2f9b200640 1 -- 172.21.15.28:0/347636070 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2f800a3e40 msgr2=0x7f2f800a6300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.772+0000 7f2f9b200640 1 --2- 172.21.15.28:0/347636070 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2f800a3e40 0x7f2f800a6300 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c0eece0 tx=0x7f2f8c02a3d0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.772+0000 7f2f9b200640 1 -- 172.21.15.28:0/347636070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f9c103060 msgr2=0x7f2f9c1a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.772+0000 7f2f9b200640 1 --2- 172.21.15.28:0/347636070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f9c103060 0x7f2f9c1a0690 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c009980 tx=0x7f2f8c0309a0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.772+0000 7f2f9b200640 1 -- 172.21.15.28:0/347636070 shutdown_connections 2024-04-24T08:11:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.772+0000 7f2f9b200640 1 --2- 172.21.15.28:0/347636070 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2f800a3e40 0x7f2f800a6300 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.772+0000 7f2f9b200640 1 --2- 172.21.15.28:0/347636070 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2f9c10c480 0x7f2f9c19a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.772+0000 7f2f9b200640 1 --2- 172.21.15.28:0/347636070 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2f9c103980 0x7f2f9c1a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.772+0000 7f2f9b200640 1 --2- 172.21.15.28:0/347636070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2f9c103060 0x7f2f9c1a0690 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.772+0000 7f2f9b200640 1 -- 172.21.15.28:0/347636070 >> 172.21.15.28:0/347636070 conn(0x7f2f9c0fc880 msgr2=0x7f2f9c101de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.772+0000 7f2f9b200640 1 -- 172.21.15.28:0/347636070 shutdown_connections 2024-04-24T08:11:01.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:01.772+0000 7f2f9b200640 1 -- 172.21.15.28:0/347636070 wait complete. 2024-04-24T08:11:02.791 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:11:02.792 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:02.792 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:11:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.019+0000 7f1efac00640 1 -- 172.21.15.28:0/3227416139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef4108f40 msgr2=0x7f1ef41153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.019+0000 7f1efac00640 1 --2- 172.21.15.28:0/3227416139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef4108f40 0x7f1ef41153b0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f1eec009cd0 tx=0x7f1eec02a8f0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.019+0000 7f1efac00640 1 -- 172.21.15.28:0/3227416139 shutdown_connections 2024-04-24T08:11:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.019+0000 7f1efac00640 1 --2- 172.21.15.28:0/3227416139 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef4108f40 0x7f1ef41153b0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.019+0000 7f1efac00640 1 --2- 172.21.15.28:0/3227416139 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ef4108580 0x7f1ef4108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.019+0000 7f1efac00640 1 --2- 172.21.15.28:0/3227416139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ef410d500 0x7f1ef410d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.019+0000 7f1efac00640 1 -- 172.21.15.28:0/3227416139 >> 172.21.15.28:0/3227416139 conn(0x7f1ef4077e20 msgr2=0x7f1ef4106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.019+0000 7f1efac00640 1 -- 172.21.15.28:0/3227416139 shutdown_connections 2024-04-24T08:11:03.021 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.019+0000 7f1efac00640 1 -- 172.21.15.28:0/3227416139 wait complete. 2024-04-24T08:11:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.020+0000 7f1efac00640 1 Processor -- start 2024-04-24T08:11:03.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.020+0000 7f1efac00640 1 -- start start 2024-04-24T08:11:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.021+0000 7f1efac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef4108580 0x7f1ef41a4930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.021+0000 7f1efac00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ef4108f40 0x7f1ef41a4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.021+0000 7f1efac00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ef410d500 0x7f1ef419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.021+0000 7f1efac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef41188b0 con 0x7f1ef4108580 2024-04-24T08:11:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.021+0000 7f1efac00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef4118730 con 0x7f1ef410d500 2024-04-24T08:11:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.021+0000 7f1efac00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef4118a30 con 0x7f1ef4108f40 2024-04-24T08:11:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.021+0000 7f1ef8e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ef4108f40 0x7f1ef41a4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.021+0000 7f1ef9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef4108580 0x7f1ef41a4930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.021+0000 7f1ef8e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ef4108f40 0x7f1ef41a4e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53100/0 (socket says 172.21.15.28:53100) 2024-04-24T08:11:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.021+0000 7f1ef8e00640 1 -- 172.21.15.28:0/1623879529 learned_addr learned my addr 172.21.15.28:0/1623879529 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.021+0000 7f1efa200640 1 --2- 172.21.15.28:0/1623879529 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ef410d500 0x7f1ef419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.021+0000 7f1ef8e00640 1 -- 172.21.15.28:0/1623879529 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ef410d500 msgr2=0x7f1ef419eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.021+0000 7f1ef8e00640 1 --2- 172.21.15.28:0/1623879529 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ef410d500 0x7f1ef419eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.021+0000 7f1ef8e00640 1 -- 172.21.15.28:0/1623879529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef4108580 msgr2=0x7f1ef41a4930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.024 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.021+0000 7f1ef8e00640 1 --2- 172.21.15.28:0/1623879529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef4108580 0x7f1ef41a4930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.021+0000 7f1ef8e00640 1 -- 172.21.15.28:0/1623879529 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1ef419f2b0 con 0x7f1ef4108f40 2024-04-24T08:11:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.022+0000 7f1ef8e00640 1 --2- 172.21.15.28:0/1623879529 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ef4108f40 0x7f1ef41a4e70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1ee400e9f0 tx=0x7f1ee400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.022+0000 7f1ef2000640 1 -- 172.21.15.28:0/1623879529 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee400cd80 con 0x7f1ef4108f40 2024-04-24T08:11:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.022+0000 7f1ef2000640 1 -- 172.21.15.28:0/1623879529 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ee4010910 con 0x7f1ef4108f40 2024-04-24T08:11:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.022+0000 7f1efac00640 1 -- 172.21.15.28:0/1623879529 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1ef419f5a0 con 0x7f1ef4108f40 2024-04-24T08:11:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.022+0000 7f1ef2000640 1 -- 172.21.15.28:0/1623879529 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee4018990 con 0x7f1ef4108f40 2024-04-24T08:11:03.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.022+0000 7f1efac00640 1 -- 172.21.15.28:0/1623879529 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1ef41ab7f0 con 0x7f1ef4108f40 2024-04-24T08:11:03.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.025+0000 7f1eff067740 1 Processor -- start 2024-04-24T08:11:03.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.025+0000 7f1eff067740 1 -- start start 2024-04-24T08:11:03.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.026+0000 7f1ef2000640 1 -- 172.21.15.28:0/1623879529 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 26) ==== 138930+0+0 (secure 0 0 0) 0x7f1ee4010ac0 con 0x7f1ef4108f40 2024-04-24T08:11:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.026+0000 7f1eff067740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 0x55929f24b650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.026+0000 7f1eff067740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24bb90 0x55929f255d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.026+0000 7f1eff067740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f256260 0x55929f258650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.026+0000 7f1eff067740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55929f1b0e70 con 0x55929f24bb90 2024-04-24T08:11:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.026+0000 7f1eff067740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55929f157380 con 0x55929f256260 2024-04-24T08:11:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.026+0000 7f1eff067740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55929f1ace30 con 0x55929f24b270 2024-04-24T08:11:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.026+0000 7f1efa200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 0x55929f24b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.026+0000 7f1ef9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24bb90 0x55929f255d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.026+0000 7f1efa200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 0x55929f24b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53106/0 (socket says 172.21.15.28:53106) 2024-04-24T08:11:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.026+0000 7f1efa200640 1 -- 172.21.15.28:0/1094752429 learned_addr learned my addr 172.21.15.28:0/1094752429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:03.028 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.026+0000 7f1ef8e00640 1 --2- 172.21.15.28:0/1094752429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f256260 0x55929f258650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.027+0000 7f1efa200640 1 -- 172.21.15.28:0/1094752429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f256260 msgr2=0x55929f258650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.027+0000 7f1efa200640 1 --2- 172.21.15.28:0/1094752429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f256260 0x55929f258650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.027+0000 7f1efa200640 1 -- 172.21.15.28:0/1094752429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24bb90 msgr2=0x55929f255d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.027+0000 7f1efa200640 1 --2- 172.21.15.28:0/1094752429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24bb90 0x55929f255d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.027+0000 7f1efa200640 1 -- 172.21.15.28:0/1094752429 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55929ef8b7e0 con 0x55929f24b270 2024-04-24T08:11:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.027+0000 7f1ef9800640 1 --2- 172.21.15.28:0/1094752429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24bb90 0x55929f255d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.027+0000 7f1ef8e00640 1 --2- 172.21.15.28:0/1094752429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f256260 0x55929f258650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.027+0000 7f1efa200640 1 --2- 172.21.15.28:0/1094752429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 0x55929f24b650 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1eec0093e0 tx=0x7f1eec009a20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5382bf540bb5ad3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.027+0000 7f1ef2000640 1 --2- 172.21.15.28:0/1623879529 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1edc0a3ed0 0x7f1edc0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.028+0000 7f1ee2a00640 1 -- 172.21.15.28:0/1094752429 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eec030620 con 0x55929f24b270 2024-04-24T08:11:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.028+0000 7f1ee2a00640 1 -- 172.21.15.28:0/1094752429 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eec036040 con 0x55929f24b270 2024-04-24T08:11:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.028+0000 7f1ef2000640 1 -- 172.21.15.28:0/1623879529 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f1ee40d0ed0 con 0x7f1ef4108f40 2024-04-24T08:11:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.028+0000 7f1ef9800640 1 --2- 172.21.15.28:0/1623879529 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1edc0a3ed0 0x7f1edc0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.028+0000 7f1ee2a00640 1 -- 172.21.15.28:0/1094752429 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1eec004460 con 0x55929f24b270 2024-04-24T08:11:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.028+0000 7f1eff067740 1 -- 172.21.15.28:0/1094752429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 msgr2=0x55929f24b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.028+0000 7f1eff067740 1 --2- 172.21.15.28:0/1094752429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 0x55929f24b650 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f1eec0093e0 tx=0x7f1eec009a20 comp rx=0 tx=0).stop 2024-04-24T08:11:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.028+0000 7f1ef9800640 1 --2- 172.21.15.28:0/1623879529 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1edc0a3ed0 0x7f1edc0a6390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x55929f24c830 tx=0x7f1ee8005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.029+0000 7f1eff067740 1 -- 172.21.15.28:0/1094752429 shutdown_connections 2024-04-24T08:11:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.029+0000 7f1eff067740 1 --2- 172.21.15.28:0/1094752429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24bb90 0x55929f255d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.029+0000 7f1eff067740 1 --2- 172.21.15.28:0/1094752429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f256260 0x55929f258650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.029+0000 7f1eff067740 1 --2- 172.21.15.28:0/1094752429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 0x55929f24b650 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.029+0000 7f1eff067740 1 -- 172.21.15.28:0/1094752429 >> 172.21.15.28:0/1094752429 conn(0x55929f247590 msgr2=0x55929f247970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.029+0000 7f1eff067740 1 -- 172.21.15.28:0/1094752429 shutdown_connections 2024-04-24T08:11:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.029+0000 7f1eff067740 1 -- 172.21.15.28:0/1094752429 wait complete. 2024-04-24T08:11:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.029+0000 7f1eff067740 1 -- start start 2024-04-24T08:11:03.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.029+0000 7f1eff067740 1 Processor -- start 2024-04-24T08:11:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.030+0000 7f1eff067740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24b270 0x55929f253210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.030+0000 7f1eff067740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f24bb90 0x55929f253750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.030+0000 7f1eff067740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f256260 0x55929f253c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.030+0000 7f1eff067740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55929f258ff0 con 0x55929f24b270 2024-04-24T08:11:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.030+0000 7f1eff067740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55929f16a660 con 0x55929f24bb90 2024-04-24T08:11:03.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.030+0000 7f1eff067740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55929f259170 con 0x55929f256260 2024-04-24T08:11:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.030+0000 7f1eff067740 10 client.0 _ll_register_callbacks cb 0x55929f158aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.030+0000 7f1ef9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24b270 0x55929f253210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.030+0000 7f1ef9800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24b270 0x55929f253210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56186/0 (socket says 172.21.15.28:56186) 2024-04-24T08:11:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.030+0000 7f1ef9800640 1 -- 172.21.15.28:0/1082890983 learned_addr learned my addr 172.21.15.28:0/1082890983 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.030+0000 7f1efa200640 1 --2- 172.21.15.28:0/1082890983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f256260 0x55929f253c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.030+0000 7f1ef8e00640 1 --2- 172.21.15.28:0/1082890983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f24bb90 0x55929f253750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.031+0000 7f1ef9800640 1 -- 172.21.15.28:0/1082890983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f256260 msgr2=0x55929f253c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.031+0000 7f1ef9800640 1 --2- 172.21.15.28:0/1082890983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f256260 0x55929f253c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.031+0000 7f1ef9800640 1 -- 172.21.15.28:0/1082890983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f24bb90 msgr2=0x55929f253750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.031+0000 7f1ef9800640 1 --2- 172.21.15.28:0/1082890983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f24bb90 0x55929f253750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.031+0000 7f1ef9800640 1 -- 172.21.15.28:0/1082890983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55929efebf70 con 0x55929f24b270 2024-04-24T08:11:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.031+0000 7f1efa200640 1 --2- 172.21.15.28:0/1082890983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f256260 0x55929f253c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:03.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.031+0000 7f1ef9800640 1 --2- 172.21.15.28:0/1082890983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24b270 0x55929f253210 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1ee801e9f0 tx=0x7f1ee8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f3198d2cd8a4a181 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.031+0000 7f1ed7400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:11:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.032+0000 7f1ed7400640 1 -- 172.21.15.28:0/1082890983 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee802e060 con 0x55929f24b270 2024-04-24T08:11:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.032+0000 7f1eff067740 1 -- 172.21.15.28:0/1082890983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1eec036040 con 0x55929f24b270 2024-04-24T08:11:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.032+0000 7f1ed7400640 1 -- 172.21.15.28:0/1082890983 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f1ee8018030 con 0x55929f24b270 2024-04-24T08:11:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.032+0000 7f1ed7400640 1 -- 172.21.15.28:0/1082890983 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1ee8032430 con 0x55929f24b270 2024-04-24T08:11:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.032+0000 7f1eff067740 10 client.15435 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.032+0000 7f1eff067740 1 -- 172.21.15.28:0/1082890983 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55929ee83170 con 0x55929f24b270 2024-04-24T08:11:03.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.032+0000 7f1ed7400640 1 -- 172.21.15.28:0/1082890983 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1eec036040 con 0x55929f24b270 2024-04-24T08:11:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.032+0000 7f1ed7400640 1 -- 172.21.15.28:0/1082890983 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f1ee800f690 con 0x55929f24b270 2024-04-24T08:11:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.033+0000 7f1eff067740 10 client.15435 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.033+0000 7f1eff067740 10 client.15435 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:11:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.033+0000 7f1eff067740 20 client.15435 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.033+0000 7f1eff067740 1 --2- 172.21.15.28:0/1082890983 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55929f266650 0x55929f268a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.033+0000 7f1eff067740 4 client.15435 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:11:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.033+0000 7f1eff067740 1 -- 172.21.15.28:0/1082890983 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55929f1be890 con 0x55929f266650 2024-04-24T08:11:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.033+0000 7f1ef8e00640 1 --2- 172.21.15.28:0/1082890983 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55929f266650 0x55929f268a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.034+0000 7f1ef8e00640 1 --2- 172.21.15.28:0/1082890983 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55929f266650 0x55929f268a60 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cac2c37e46595ce4 server_cookie=1c24d9137dbc18a8 in_seq=0 out_seq=0 2024-04-24T08:11:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.034+0000 7f1ed7400640 10 client.15435 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.034+0000 7f1ed7400640 1 -- 172.21.15.28:0/1082890983 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55929f1be890 con 0x55929f266650 2024-04-24T08:11:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.034+0000 7f1ed7400640 10 client.15435 handle_command_reply: tid=0 2024-04-24T08:11:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.034+0000 7f1eff067740 1 client.15435 shutdown 2024-04-24T08:11:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.035+0000 7f1eff067740 1 -- 172.21.15.28:0/1082890983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24b270 msgr2=0x55929f253210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.035+0000 7f1eff067740 1 --2- 172.21.15.28:0/1082890983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24b270 0x55929f253210 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f1ee801e9f0 tx=0x7f1ee8004730 comp rx=0 tx=0).stop 2024-04-24T08:11:03.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.035+0000 7f1eff067740 1 -- 172.21.15.28:0/1082890983 shutdown_connections 2024-04-24T08:11:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.035+0000 7f1eff067740 1 --2- 172.21.15.28:0/1082890983 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55929f266650 0x55929f268a60 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.035+0000 7f1eff067740 1 --2- 172.21.15.28:0/1082890983 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f256260 0x55929f253c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.035+0000 7f1eff067740 1 --2- 172.21.15.28:0/1082890983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f24bb90 0x55929f253750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.035+0000 7f1eff067740 1 --2- 172.21.15.28:0/1082890983 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24b270 0x55929f253210 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.035+0000 7f1eff067740 1 -- 172.21.15.28:0/1082890983 >> 172.21.15.28:0/1082890983 conn(0x55929f247590 msgr2=0x55929f24d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.035+0000 7f1ed7400640 0 client.15435 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.035+0000 7f1eff067740 1 -- 172.21.15.28:0/1082890983 shutdown_connections 2024-04-24T08:11:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.035+0000 7f1eff067740 1 -- 172.21.15.28:0/1082890983 wait complete. 2024-04-24T08:11:03.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.035+0000 7f1eff067740 20 client.15435 trim_cache size 0 max 16384 2024-04-24T08:11:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.055+0000 7f1eff067740 1 Processor -- start 2024-04-24T08:11:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.056+0000 7f1eff067740 1 -- start start 2024-04-24T08:11:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.056+0000 7f1eff067740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 0x55929f261eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.056+0000 7f1eff067740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f24bb90 0x55929f252af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.056+0000 7f1eff067740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f256260 0x55929f246570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.056+0000 7f1eff067740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef41b6ae0 con 0x55929f256260 2024-04-24T08:11:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.056+0000 7f1eff067740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef41b43b0 con 0x55929f24bb90 2024-04-24T08:11:03.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.056+0000 7f1eff067740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef41b2200 con 0x55929f24b270 2024-04-24T08:11:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.056+0000 7f1ef8e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f24bb90 0x55929f252af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.056+0000 7f1ef8e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f24bb90 0x55929f252af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:45722/0 (socket says 172.21.15.28:45722) 2024-04-24T08:11:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.056+0000 7f1ef8e00640 1 -- 172.21.15.28:0/1226253317 learned_addr learned my addr 172.21.15.28:0/1226253317 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.056+0000 7f1efa200640 1 --2- 172.21.15.28:0/1226253317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f256260 0x55929f246570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.056+0000 7f1ef9800640 1 --2- 172.21.15.28:0/1226253317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 0x55929f261eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.057+0000 7f1ef8e00640 1 -- 172.21.15.28:0/1226253317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 msgr2=0x55929f261eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.057+0000 7f1ef8e00640 1 --2- 172.21.15.28:0/1226253317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 0x55929f261eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.057+0000 7f1ef8e00640 1 -- 172.21.15.28:0/1226253317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f256260 msgr2=0x55929f246570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.057+0000 7f1ef8e00640 1 --2- 172.21.15.28:0/1226253317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f256260 0x55929f246570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.057+0000 7f1ef8e00640 1 -- 172.21.15.28:0/1226253317 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55929f2651f0 con 0x55929f24bb90 2024-04-24T08:11:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.057+0000 7f1ef9800640 1 --2- 172.21.15.28:0/1226253317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 0x55929f261eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:03.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.057+0000 7f1ef8e00640 1 --2- 172.21.15.28:0/1226253317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f24bb90 0x55929f252af0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1ee400c6a0 tx=0x7f1ee40d3360 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35f2429b901d8e15 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.057+0000 7f1ee2000640 1 -- 172.21.15.28:0/1226253317 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee409c570 con 0x55929f24bb90 2024-04-24T08:11:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.058+0000 7f1ee2000640 1 -- 172.21.15.28:0/1226253317 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ee409d310 con 0x55929f24bb90 2024-04-24T08:11:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.058+0000 7f1ee2000640 1 -- 172.21.15.28:0/1226253317 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee40bd0b0 con 0x55929f24bb90 2024-04-24T08:11:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.058+0000 7f1eff067740 1 -- 172.21.15.28:0/1226253317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f24bb90 msgr2=0x55929f252af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.058+0000 7f1eff067740 1 --2- 172.21.15.28:0/1226253317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f24bb90 0x55929f252af0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f1ee400c6a0 tx=0x7f1ee40d3360 comp rx=0 tx=0).stop 2024-04-24T08:11:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.058+0000 7f1eff067740 1 -- 172.21.15.28:0/1226253317 shutdown_connections 2024-04-24T08:11:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.058+0000 7f1eff067740 1 --2- 172.21.15.28:0/1226253317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f256260 0x55929f246570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.058+0000 7f1eff067740 1 --2- 172.21.15.28:0/1226253317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f24bb90 0x55929f252af0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.058+0000 7f1eff067740 1 --2- 172.21.15.28:0/1226253317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 0x55929f261eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.058+0000 7f1eff067740 1 -- 172.21.15.28:0/1226253317 >> 172.21.15.28:0/1226253317 conn(0x55929f247590 msgr2=0x55929f245e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.058+0000 7f1eff067740 1 -- 172.21.15.28:0/1226253317 shutdown_connections 2024-04-24T08:11:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.058+0000 7f1eff067740 1 -- 172.21.15.28:0/1226253317 wait complete. 2024-04-24T08:11:03.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.059+0000 7f1eff067740 1 -- start start 2024-04-24T08:11:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.059+0000 7f1eff067740 1 Processor -- start 2024-04-24T08:11:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1eff067740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 0x55929f25a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1eff067740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24bb90 0x55929f25aa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1eff067740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f256260 0x55929f2cbba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1eff067740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef41b2870 con 0x55929f24bb90 2024-04-24T08:11:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1eff067740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef41b3ce0 con 0x55929f256260 2024-04-24T08:11:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1eff067740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1ef41b2f40 con 0x55929f24b270 2024-04-24T08:11:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1eff067740 10 client.0 _ll_register_callbacks cb 0x7f1ee8009c80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1ef9800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 0x55929f25a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1ef8e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24bb90 0x55929f25aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1ef9800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 0x55929f25a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53140/0 (socket says 172.21.15.28:53140) 2024-04-24T08:11:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1ef9800640 1 -- 172.21.15.28:0/2948869052 learned_addr learned my addr 172.21.15.28:0/2948869052 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1efa200640 1 --2- 172.21.15.28:0/2948869052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f256260 0x55929f2cbba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1ef9800640 1 -- 172.21.15.28:0/2948869052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f256260 msgr2=0x55929f2cbba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:03.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1ef9800640 1 --2- 172.21.15.28:0/2948869052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f256260 0x55929f2cbba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1ef9800640 1 -- 172.21.15.28:0/2948869052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24bb90 msgr2=0x55929f25aa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1ef9800640 1 --2- 172.21.15.28:0/2948869052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24bb90 0x55929f25aa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1ef9800640 1 -- 172.21.15.28:0/2948869052 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55929f242e80 con 0x55929f24b270 2024-04-24T08:11:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1efa200640 1 --2- 172.21.15.28:0/2948869052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f256260 0x55929f2cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.060+0000 7f1ef8e00640 1 --2- 172.21.15.28:0/2948869052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24bb90 0x55929f25aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.061+0000 7f1ef9800640 1 --2- 172.21.15.28:0/2948869052 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 0x55929f25a4d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1ee800a010 tx=0x7f1ee8009d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6d13843a044f9ee5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.061+0000 7f1ed7400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:11:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.061+0000 7f1ed7400640 1 -- 172.21.15.28:0/2948869052 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ee801e7c0 con 0x55929f24b270 2024-04-24T08:11:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.061+0000 7f1ed7400640 1 -- 172.21.15.28:0/2948869052 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f1ee8005020 con 0x55929f24b270 2024-04-24T08:11:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.061+0000 7f1eff067740 1 -- 172.21.15.28:0/2948869052 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ee409d310 con 0x55929f24b270 2024-04-24T08:11:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.061+0000 7f1ed7400640 1 -- 172.21.15.28:0/2948869052 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1ee800f870 con 0x55929f24b270 2024-04-24T08:11:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.061+0000 7f1eff067740 10 client.25576 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.061+0000 7f1eff067740 1 -- 172.21.15.28:0/2948869052 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55929f2cd200 con 0x55929f24b270 2024-04-24T08:11:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.062+0000 7f1ed7400640 1 -- 172.21.15.28:0/2948869052 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ee409d310 con 0x55929f24b270 2024-04-24T08:11:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.062+0000 7f1ed7400640 1 -- 172.21.15.28:0/2948869052 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f1ee8004830 con 0x55929f24b270 2024-04-24T08:11:03.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.062+0000 7f1eff067740 10 client.25576 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:03.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.062+0000 7f1eff067740 10 client.25576 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:11:03.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.062+0000 7f1eff067740 20 client.25576 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:03.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.062+0000 7f1eff067740 1 --2- 172.21.15.28:0/2948869052 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55929f2cd910 0x55929f2cfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.062+0000 7f1eff067740 4 client.25576 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:11:03.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.062+0000 7f1eff067740 1 -- 172.21.15.28:0/2948869052 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55929f1b8700 con 0x55929f2cd910 2024-04-24T08:11:03.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.062+0000 7f1ef8e00640 1 --2- 172.21.15.28:0/2948869052 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55929f2cd910 0x55929f2cfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.063+0000 7f1ef8e00640 1 --2- 172.21.15.28:0/2948869052 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55929f2cd910 0x55929f2cfd30 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b7831651205a615 server_cookie=5fa5776fd1f43f15 in_seq=0 out_seq=0 2024-04-24T08:11:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.063+0000 7f1ed7400640 10 client.25576 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.064+0000 7f1ed7400640 1 -- 172.21.15.28:0/2948869052 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55929f1b8700 con 0x55929f2cd910 2024-04-24T08:11:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.064+0000 7f1ed7400640 10 client.25576 handle_command_reply: tid=0 2024-04-24T08:11:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.064+0000 7f1eff067740 1 client.25576 shutdown 2024-04-24T08:11:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.065+0000 7f1eff067740 1 -- 172.21.15.28:0/2948869052 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 msgr2=0x55929f25a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.065+0000 7f1eff067740 1 --2- 172.21.15.28:0/2948869052 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 0x55929f25a4d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f1ee800a010 tx=0x7f1ee8009d50 comp rx=0 tx=0).stop 2024-04-24T08:11:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.065+0000 7f1eff067740 1 -- 172.21.15.28:0/2948869052 shutdown_connections 2024-04-24T08:11:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.065+0000 7f1eff067740 1 --2- 172.21.15.28:0/2948869052 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55929f2cd910 0x55929f2cfd30 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.065+0000 7f1eff067740 1 --2- 172.21.15.28:0/2948869052 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55929f24bb90 0x55929f25aa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.065+0000 7f1eff067740 1 --2- 172.21.15.28:0/2948869052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55929f256260 0x55929f2cbba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.065+0000 7f1eff067740 1 --2- 172.21.15.28:0/2948869052 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55929f24b270 0x55929f25a4d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.065+0000 7f1eff067740 1 -- 172.21.15.28:0/2948869052 >> 172.21.15.28:0/2948869052 conn(0x55929f247590 msgr2=0x55929f24d250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.065+0000 7f1ed7400640 0 client.25576 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.065+0000 7f1eff067740 1 -- 172.21.15.28:0/2948869052 shutdown_connections 2024-04-24T08:11:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.065+0000 7f1eff067740 1 -- 172.21.15.28:0/2948869052 wait complete. 2024-04-24T08:11:03.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.065+0000 7f1eff067740 20 client.25576 trim_cache size 0 max 16384 2024-04-24T08:11:03.068 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:11:03.068 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-24T08:11:03.069 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "049460ae-a3a9-4ec2-a1f2-93e21a5e72dd", 2024-04-24T08:11:03.069 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-24T08:11:03.069 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:11:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.066+0000 7f1efac00640 1 -- 172.21.15.28:0/1623879529 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1edc0a3ed0 msgr2=0x7f1edc0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.066+0000 7f1efac00640 1 --2- 172.21.15.28:0/1623879529 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1edc0a3ed0 0x7f1edc0a6390 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x55929f24c830 tx=0x7f1ee8005f90 comp rx=0 tx=0).stop 2024-04-24T08:11:03.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.066+0000 7f1efac00640 1 -- 172.21.15.28:0/1623879529 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ef4108f40 msgr2=0x7f1ef41a4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.066+0000 7f1efac00640 1 --2- 172.21.15.28:0/1623879529 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ef4108f40 0x7f1ef41a4e70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f1ee400e9f0 tx=0x7f1ee400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.066+0000 7f1efac00640 1 -- 172.21.15.28:0/1623879529 shutdown_connections 2024-04-24T08:11:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.066+0000 7f1efac00640 1 --2- 172.21.15.28:0/1623879529 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1edc0a3ed0 0x7f1edc0a6390 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.066+0000 7f1efac00640 1 --2- 172.21.15.28:0/1623879529 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ef410d500 0x7f1ef419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.066+0000 7f1efac00640 1 --2- 172.21.15.28:0/1623879529 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1ef4108f40 0x7f1ef41a4e70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.066+0000 7f1efac00640 1 --2- 172.21.15.28:0/1623879529 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1ef4108580 0x7f1ef41a4930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.066+0000 7f1efac00640 1 -- 172.21.15.28:0/1623879529 >> 172.21.15.28:0/1623879529 conn(0x7f1ef4077e20 msgr2=0x7f1ef4078340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.066+0000 7f1efac00640 1 -- 172.21.15.28:0/1623879529 shutdown_connections 2024-04-24T08:11:03.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.066+0000 7f1efac00640 1 -- 172.21.15.28:0/1623879529 wait complete. 2024-04-24T08:11:03.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:11:03.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:11:03.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.290+0000 7fade4e00640 1 -- 172.21.15.28:0/2902650909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fade0104450 msgr2=0x7fade0104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.290+0000 7fade4e00640 1 --2- 172.21.15.28:0/2902650909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fade0104450 0x7fade0104830 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009ca0 tx=0x7fadd002a8c0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.290+0000 7fade4e00640 1 -- 172.21.15.28:0/2902650909 shutdown_connections 2024-04-24T08:11:03.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.290+0000 7fade4e00640 1 --2- 172.21.15.28:0/2902650909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fade0104e00 0x7fade0108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.290+0000 7fade4e00640 1 --2- 172.21.15.28:0/2902650909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fade0109520 0x7fade01110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.290+0000 7fade4e00640 1 --2- 172.21.15.28:0/2902650909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fade0104450 0x7fade0104830 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.290+0000 7fade4e00640 1 -- 172.21.15.28:0/2902650909 >> 172.21.15.28:0/2902650909 conn(0x7fade00ffc70 msgr2=0x7fade0102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:03.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.290+0000 7fade4e00640 1 -- 172.21.15.28:0/2902650909 shutdown_connections 2024-04-24T08:11:03.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.290+0000 7fade4e00640 1 -- 172.21.15.28:0/2902650909 wait complete. 2024-04-24T08:11:03.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.291+0000 7fade4e00640 1 Processor -- start 2024-04-24T08:11:03.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.291+0000 7fade4e00640 1 -- start start 2024-04-24T08:11:03.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.292+0000 7fade4e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fade0104450 0x7fade01a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.292+0000 7fade4e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fade0109520 0x7fade01a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.292+0000 7fade4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fade019a920 0x7fade019add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.292+0000 7fade4e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fade0114600 con 0x7fade019a920 2024-04-24T08:11:03.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.292+0000 7fade4e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fade0114480 con 0x7fade0104450 2024-04-24T08:11:03.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.292+0000 7fade4e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fade0114780 con 0x7fade0109520 2024-04-24T08:11:03.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.292+0000 7fadd5400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fade0109520 0x7fade01a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.292+0000 7fadd5400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fade0109520 0x7fade01a0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53150/0 (socket says 172.21.15.28:53150) 2024-04-24T08:11:03.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.292+0000 7fadd5400640 1 -- 172.21.15.28:0/102725117 learned_addr learned my addr 172.21.15.28:0/102725117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:03.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.292+0000 7fadde800640 1 --2- 172.21.15.28:0/102725117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fade019a920 0x7fade019add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.293+0000 7faddde00640 1 --2- 172.21.15.28:0/102725117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fade0104450 0x7fade01a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.293+0000 7fadd5400640 1 -- 172.21.15.28:0/102725117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fade0104450 msgr2=0x7fade01a0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.293+0000 7fadd5400640 1 --2- 172.21.15.28:0/102725117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fade0104450 0x7fade01a0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.293+0000 7fadd5400640 1 -- 172.21.15.28:0/102725117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fade019a920 msgr2=0x7fade019add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.293+0000 7fadd5400640 1 --2- 172.21.15.28:0/102725117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fade019a920 0x7fade019add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.293+0000 7fadd5400640 1 -- 172.21.15.28:0/102725117 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fade019b440 con 0x7fade0109520 2024-04-24T08:11:03.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.293+0000 7faddde00640 1 --2- 172.21.15.28:0/102725117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fade0104450 0x7fade01a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:11:03.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.293+0000 7fadde800640 1 --2- 172.21.15.28:0/102725117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fade019a920 0x7fade019add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:03.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.293+0000 7fadd5400640 1 --2- 172.21.15.28:0/102725117 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fade0109520 0x7fade01a0cb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fadc800e9c0 tx=0x7fadc800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:03.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.293+0000 7fadd7400640 1 -- 172.21.15.28:0/102725117 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc800cd80 con 0x7fade0109520 2024-04-24T08:11:03.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.294+0000 7fadd7400640 1 -- 172.21.15.28:0/102725117 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadc8005500 con 0x7fade0109520 2024-04-24T08:11:03.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.294+0000 7fade4e00640 1 -- 172.21.15.28:0/102725117 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fade019b730 con 0x7fade0109520 2024-04-24T08:11:03.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.294+0000 7fadd7400640 1 -- 172.21.15.28:0/102725117 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc8010690 con 0x7fade0109520 2024-04-24T08:11:03.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.294+0000 7fade4e00640 1 -- 172.21.15.28:0/102725117 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fade01a79d0 con 0x7fade0109520 2024-04-24T08:11:03.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.297+0000 7fade7d8f740 1 Processor -- start 2024-04-24T08:11:03.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.297+0000 7fade7d8f740 1 -- start start 2024-04-24T08:11:03.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.297+0000 7fadd7400640 1 -- 172.21.15.28:0/102725117 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 26) ==== 138930+0+0 (secure 0 0 0) 0x7fadc80046c0 con 0x7fade0109520 2024-04-24T08:11:03.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.297+0000 7fade7d8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fab860 0x5627f7fabc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.297+0000 7fade7d8f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fac180 0x5627f7fb6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.297+0000 7fade7d8f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fb6850 0x5627f7fb8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.297+0000 7fade7d8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5627f7f111f0 con 0x5627f7fab860 2024-04-24T08:11:03.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.297+0000 7fade7d8f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5627f7eb7350 con 0x5627f7fac180 2024-04-24T08:11:03.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.297+0000 7fade7d8f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5627f7f0d1b0 con 0x5627f7fb6850 2024-04-24T08:11:03.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.298+0000 7fadde800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fab860 0x5627f7fabc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.298+0000 7fadde800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fab860 0x5627f7fabc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56232/0 (socket says 172.21.15.28:56232) 2024-04-24T08:11:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.298+0000 7fadde800640 1 -- 172.21.15.28:0/3064123868 learned_addr learned my addr 172.21.15.28:0/3064123868 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.298+0000 7faddde00640 1 --2- 172.21.15.28:0/3064123868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fac180 0x5627f7fb6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.298+0000 7fadd5400640 1 --2- 172.21.15.28:0/3064123868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fb6850 0x5627f7fb8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.298+0000 7fadde800640 1 -- 172.21.15.28:0/3064123868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fb6850 msgr2=0x5627f7fb8c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.298+0000 7fadde800640 1 --2- 172.21.15.28:0/3064123868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fb6850 0x5627f7fb8c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.298+0000 7fadde800640 1 -- 172.21.15.28:0/3064123868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fac180 msgr2=0x5627f7fb6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.298+0000 7fadde800640 1 --2- 172.21.15.28:0/3064123868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fac180 0x5627f7fb6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.298+0000 7fadde800640 1 -- 172.21.15.28:0/3064123868 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627f7f0bf50 con 0x5627f7fab860 2024-04-24T08:11:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.298+0000 7fadde800640 1 --2- 172.21.15.28:0/3064123868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fab860 0x5627f7fabc40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fadd8011440 tx=0x7fadd8011900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a8adc283acec746 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.299+0000 7fadcf400640 1 -- 172.21.15.28:0/3064123868 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd8004020 con 0x5627f7fab860 2024-04-24T08:11:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.299+0000 7fadcf400640 1 -- 172.21.15.28:0/3064123868 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadd80041c0 con 0x5627f7fab860 2024-04-24T08:11:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.299+0000 7fadcf400640 1 -- 172.21.15.28:0/3064123868 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd800f870 con 0x5627f7fab860 2024-04-24T08:11:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.299+0000 7fadd7400640 1 --2- 172.21.15.28:0/102725117 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fadc40a3f50 0x7fadc40a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.299+0000 7fadd7400640 1 -- 172.21.15.28:0/102725117 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fadc80d0ea0 con 0x7fade0109520 2024-04-24T08:11:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.299+0000 7faddde00640 1 --2- 172.21.15.28:0/102725117 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fadc40a3f50 0x7fadc40a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.300+0000 7fade7d8f740 1 -- 172.21.15.28:0/3064123868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fab860 msgr2=0x5627f7fabc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.300+0000 7fade7d8f740 1 --2- 172.21.15.28:0/3064123868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fab860 0x5627f7fabc40 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fadd8011440 tx=0x7fadd8011900 comp rx=0 tx=0).stop 2024-04-24T08:11:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.300+0000 7fade7d8f740 1 -- 172.21.15.28:0/3064123868 shutdown_connections 2024-04-24T08:11:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.300+0000 7fade7d8f740 1 --2- 172.21.15.28:0/3064123868 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fb6850 0x5627f7fb8c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.300+0000 7fade7d8f740 1 --2- 172.21.15.28:0/3064123868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fac180 0x5627f7fb6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.300+0000 7fade7d8f740 1 --2- 172.21.15.28:0/3064123868 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fab860 0x5627f7fabc40 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.300+0000 7fade7d8f740 1 -- 172.21.15.28:0/3064123868 >> 172.21.15.28:0/3064123868 conn(0x5627f7fa7b80 msgr2=0x5627f7fa7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.300+0000 7fade7d8f740 1 -- 172.21.15.28:0/3064123868 shutdown_connections 2024-04-24T08:11:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.300+0000 7fade7d8f740 1 -- 172.21.15.28:0/3064123868 wait complete. 2024-04-24T08:11:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.300+0000 7faddde00640 1 --2- 172.21.15.28:0/102725117 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fadc40a3f50 0x7fadc40a6410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fadd0000f80 tx=0x7fadd0004a40 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.300+0000 7fade7d8f740 1 -- start start 2024-04-24T08:11:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.301+0000 7fade7d8f740 1 Processor -- start 2024-04-24T08:11:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.301+0000 7fade7d8f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fab860 0x5627f7fb3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.301+0000 7fade7d8f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fac180 0x5627f7fb3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.301+0000 7fade7d8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fb6850 0x5627f7fb4290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.301+0000 7fade7d8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5627f7fb9640 con 0x5627f7fb6850 2024-04-24T08:11:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.301+0000 7fade7d8f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5627f7eca690 con 0x5627f7fab860 2024-04-24T08:11:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.301+0000 7fade7d8f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5627f7fb97c0 con 0x5627f7fac180 2024-04-24T08:11:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.301+0000 7fade7d8f740 10 client.0 _ll_register_callbacks cb 0x5627f7f06650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.302+0000 7fadde800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fb6850 0x5627f7fb4290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.302+0000 7faddde00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fab860 0x5627f7fb3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.302+0000 7faddde00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fab860 0x5627f7fb3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:45776/0 (socket says 172.21.15.28:45776) 2024-04-24T08:11:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.302+0000 7fadde800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fb6850 0x5627f7fb4290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56246/0 (socket says 172.21.15.28:56246) 2024-04-24T08:11:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.302+0000 7faddde00640 1 -- 172.21.15.28:0/459503353 learned_addr learned my addr 172.21.15.28:0/459503353 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.302+0000 7fadd5400640 1 --2- 172.21.15.28:0/459503353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fac180 0x5627f7fb3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.302+0000 7faddde00640 1 -- 172.21.15.28:0/459503353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fac180 msgr2=0x5627f7fb3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.302+0000 7faddde00640 1 --2- 172.21.15.28:0/459503353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fac180 0x5627f7fb3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.302+0000 7faddde00640 1 -- 172.21.15.28:0/459503353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fb6850 msgr2=0x5627f7fb4290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.302+0000 7faddde00640 1 --2- 172.21.15.28:0/459503353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fb6850 0x5627f7fb4290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.302+0000 7faddde00640 1 -- 172.21.15.28:0/459503353 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627f7ec9920 con 0x5627f7fab860 2024-04-24T08:11:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.302+0000 7fadde800640 1 --2- 172.21.15.28:0/459503353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fb6850 0x5627f7fb4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.303+0000 7faddde00640 1 --2- 172.21.15.28:0/459503353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fab860 0x5627f7fb3810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fadd0035000 tx=0x7fadd002a8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7e45fd3e50fe8b0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.303+0000 7fadc3e00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:11:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.303+0000 7fadc3e00640 1 -- 172.21.15.28:0/459503353 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd00317e0 con 0x5627f7fab860 2024-04-24T08:11:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.303+0000 7fade7d8f740 1 -- 172.21.15.28:0/459503353 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadd80041c0 con 0x5627f7fab860 2024-04-24T08:11:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.303+0000 7fadc3e00640 1 -- 172.21.15.28:0/459503353 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fadd0031980 con 0x5627f7fab860 2024-04-24T08:11:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.303+0000 7fadc3e00640 1 -- 172.21.15.28:0/459503353 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fadd0003fb0 con 0x5627f7fab860 2024-04-24T08:11:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.303+0000 7fade7d8f740 10 client.25550 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:03.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.303+0000 7fade7d8f740 1 -- 172.21.15.28:0/459503353 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5627f7ced7d0 con 0x5627f7fab860 2024-04-24T08:11:03.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.304+0000 7fadc3e00640 1 -- 172.21.15.28:0/459503353 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadd80041c0 con 0x5627f7fab860 2024-04-24T08:11:03.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.304+0000 7fadc3e00640 1 -- 172.21.15.28:0/459503353 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fadd0051020 con 0x5627f7fab860 2024-04-24T08:11:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.304+0000 7fade7d8f740 10 client.25550 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.304+0000 7fade7d8f740 10 client.25550 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:11:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.304+0000 7fade7d8f740 20 client.25550 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.304+0000 7fade7d8f740 1 --2- 172.21.15.28:0/459503353 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5627f7fc6b50 0x5627f7fc8fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.304+0000 7fade7d8f740 4 client.25550 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:11:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.304+0000 7fade7d8f740 1 -- 172.21.15.28:0/459503353 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627f7f1ec10 con 0x5627f7fc6b50 2024-04-24T08:11:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.304+0000 7fadd5400640 1 --2- 172.21.15.28:0/459503353 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5627f7fc6b50 0x5627f7fc8fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.305+0000 7fadd5400640 1 --2- 172.21.15.28:0/459503353 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5627f7fc6b50 0x5627f7fc8fd0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40e522bb03030438 server_cookie=6a4630ee2355a40d in_seq=0 out_seq=0 2024-04-24T08:11:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.305+0000 7fadc3e00640 10 client.25550 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.306+0000 7fadc3e00640 1 -- 172.21.15.28:0/459503353 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5627f7f1ec10 con 0x5627f7fc6b50 2024-04-24T08:11:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.306+0000 7fadc3e00640 10 client.25550 handle_command_reply: tid=0 2024-04-24T08:11:03.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.306+0000 7fade7d8f740 1 client.25550 shutdown 2024-04-24T08:11:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.306+0000 7fade7d8f740 1 -- 172.21.15.28:0/459503353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fab860 msgr2=0x5627f7fb3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.306+0000 7fade7d8f740 1 --2- 172.21.15.28:0/459503353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fab860 0x5627f7fb3810 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fadd0035000 tx=0x7fadd002a8c0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.306+0000 7fade7d8f740 1 -- 172.21.15.28:0/459503353 shutdown_connections 2024-04-24T08:11:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.306+0000 7fade7d8f740 1 --2- 172.21.15.28:0/459503353 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5627f7fc6b50 0x5627f7fc8fd0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.306+0000 7fade7d8f740 1 --2- 172.21.15.28:0/459503353 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fb6850 0x5627f7fb4290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.306+0000 7fade7d8f740 1 --2- 172.21.15.28:0/459503353 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fac180 0x5627f7fb3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.306+0000 7fade7d8f740 1 --2- 172.21.15.28:0/459503353 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fab860 0x5627f7fb3810 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.306+0000 7fade7d8f740 1 -- 172.21.15.28:0/459503353 >> 172.21.15.28:0/459503353 conn(0x5627f7fa7b80 msgr2=0x5627f7fad980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.306+0000 7fadc3e00640 0 client.25550 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.306+0000 7fade7d8f740 1 -- 172.21.15.28:0/459503353 shutdown_connections 2024-04-24T08:11:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.306+0000 7fade7d8f740 1 -- 172.21.15.28:0/459503353 wait complete. 2024-04-24T08:11:03.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.306+0000 7fade7d8f740 20 client.25550 trim_cache size 0 max 16384 2024-04-24T08:11:03.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.326+0000 7fade7d8f740 1 Processor -- start 2024-04-24T08:11:03.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.326+0000 7fade7d8f740 1 -- start start 2024-04-24T08:11:03.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.326+0000 7fade7d8f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fab860 0x5627f7fc2500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.326+0000 7fade7d8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fac180 0x5627f7fb3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.326+0000 7fade7d8f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fb6850 0x5627f7fce8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.326+0000 7fade7d8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fade01b2a00 con 0x5627f7fac180 2024-04-24T08:11:03.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.326+0000 7fade7d8f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fade01b02d0 con 0x5627f7fab860 2024-04-24T08:11:03.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.326+0000 7fade7d8f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fade01ae120 con 0x5627f7fb6850 2024-04-24T08:11:03.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.327+0000 7fadd5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fac180 0x5627f7fb3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.327+0000 7fadd5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fac180 0x5627f7fb3ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56256/0 (socket says 172.21.15.28:56256) 2024-04-24T08:11:03.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.327+0000 7fadd5400640 1 -- 172.21.15.28:0/1153649417 learned_addr learned my addr 172.21.15.28:0/1153649417 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:03.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.327+0000 7fadde800640 1 --2- 172.21.15.28:0/1153649417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fb6850 0x5627f7fce8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.327+0000 7faddde00640 1 --2- 172.21.15.28:0/1153649417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fab860 0x5627f7fc2500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.327+0000 7fadd5400640 1 -- 172.21.15.28:0/1153649417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fb6850 msgr2=0x5627f7fce8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:03.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.327+0000 7fadd5400640 1 --2- 172.21.15.28:0/1153649417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fb6850 0x5627f7fce8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.327+0000 7fadd5400640 1 -- 172.21.15.28:0/1153649417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fab860 msgr2=0x5627f7fc2500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:03.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.327+0000 7fadd5400640 1 --2- 172.21.15.28:0/1153649417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fab860 0x5627f7fc2500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.327+0000 7fadd5400640 1 -- 172.21.15.28:0/1153649417 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627f7c233f0 con 0x5627f7fac180 2024-04-24T08:11:03.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.327+0000 7fadd5400640 1 --2- 172.21.15.28:0/1153649417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fac180 0x5627f7fb3ee0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fadc8099040 tx=0x7fadc80bd040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f23294e064d1b4f7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:03.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.328+0000 7fadcea00640 1 -- 172.21.15.28:0/1153649417 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc80b5030 con 0x5627f7fac180 2024-04-24T08:11:03.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.328+0000 7fadcea00640 1 -- 172.21.15.28:0/1153649417 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadc809d2e0 con 0x5627f7fac180 2024-04-24T08:11:03.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.328+0000 7fadcea00640 1 -- 172.21.15.28:0/1153649417 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc80a5850 con 0x5627f7fac180 2024-04-24T08:11:03.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.328+0000 7fade7d8f740 1 -- 172.21.15.28:0/1153649417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fac180 msgr2=0x5627f7fb3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.328+0000 7fade7d8f740 1 --2- 172.21.15.28:0/1153649417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fac180 0x5627f7fb3ee0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fadc8099040 tx=0x7fadc80bd040 comp rx=0 tx=0).stop 2024-04-24T08:11:03.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.328+0000 7fade7d8f740 1 -- 172.21.15.28:0/1153649417 shutdown_connections 2024-04-24T08:11:03.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.328+0000 7fade7d8f740 1 --2- 172.21.15.28:0/1153649417 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fac180 0x5627f7fb3ee0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.328+0000 7fade7d8f740 1 --2- 172.21.15.28:0/1153649417 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fb6850 0x5627f7fce8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.328+0000 7fade7d8f740 1 --2- 172.21.15.28:0/1153649417 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fab860 0x5627f7fc2500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.328+0000 7fade7d8f740 1 -- 172.21.15.28:0/1153649417 >> 172.21.15.28:0/1153649417 conn(0x5627f7fa7b80 msgr2=0x5627f7fb37d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:03.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.328+0000 7fade7d8f740 1 -- 172.21.15.28:0/1153649417 shutdown_connections 2024-04-24T08:11:03.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.328+0000 7fade7d8f740 1 -- 172.21.15.28:0/1153649417 wait complete. 2024-04-24T08:11:03.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.329+0000 7fade7d8f740 1 -- start start 2024-04-24T08:11:03.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.329+0000 7fade7d8f740 1 Processor -- start 2024-04-24T08:11:03.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.329+0000 7fade7d8f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fab860 0x5627f7fb0510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.329+0000 7fade7d8f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fac180 0x5627f7fb0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.329+0000 7fade7d8f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fb6850 0x5627f7fb0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.329+0000 7fade7d8f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fade01ae790 con 0x5627f7fac180 2024-04-24T08:11:03.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.329+0000 7fade7d8f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fade01afc00 con 0x5627f7fb6850 2024-04-24T08:11:03.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.329+0000 7fade7d8f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fade01aee60 con 0x5627f7fab860 2024-04-24T08:11:03.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.330+0000 7fade7d8f740 10 client.0 _ll_register_callbacks cb 0x7fadd8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:03.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.330+0000 7fadd5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fac180 0x5627f7fb0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.330+0000 7fadd5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fac180 0x5627f7fb0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56270/0 (socket says 172.21.15.28:56270) 2024-04-24T08:11:03.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.330+0000 7fadd5400640 1 -- 172.21.15.28:0/211135032 learned_addr learned my addr 172.21.15.28:0/211135032 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:03.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.330+0000 7fadde800640 1 --2- 172.21.15.28:0/211135032 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fb6850 0x5627f7fb0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.330+0000 7faddde00640 1 --2- 172.21.15.28:0/211135032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fab860 0x5627f7fb0510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.330+0000 7fadd5400640 1 -- 172.21.15.28:0/211135032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fab860 msgr2=0x5627f7fb0510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:03.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.330+0000 7fadd5400640 1 --2- 172.21.15.28:0/211135032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fab860 0x5627f7fb0510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.330+0000 7fadd5400640 1 -- 172.21.15.28:0/211135032 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fb6850 msgr2=0x5627f7fb0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:03.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.330+0000 7fadd5400640 1 --2- 172.21.15.28:0/211135032 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fb6850 0x5627f7fb0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.330+0000 7fadd5400640 1 -- 172.21.15.28:0/211135032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627f7be30f0 con 0x5627f7fac180 2024-04-24T08:11:03.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.330+0000 7fadd5400640 1 --2- 172.21.15.28:0/211135032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fac180 0x5627f7fb0a50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fadc80d0010 tx=0x7fadc809c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f952fa51881e8cd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:03.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.330+0000 7fadc3e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:11:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.331+0000 7fadc3e00640 1 -- 172.21.15.28:0/211135032 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc807f020 con 0x5627f7fac180 2024-04-24T08:11:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.331+0000 7fadc3e00640 1 -- 172.21.15.28:0/211135032 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fadc8099250 con 0x5627f7fac180 2024-04-24T08:11:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.331+0000 7fade7d8f740 1 -- 172.21.15.28:0/211135032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadc809d2e0 con 0x5627f7fac180 2024-04-24T08:11:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.331+0000 7fadc3e00640 1 -- 172.21.15.28:0/211135032 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fadc80b5030 con 0x5627f7fac180 2024-04-24T08:11:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.331+0000 7fade7d8f740 10 client.15459 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.331+0000 7fade7d8f740 1 -- 172.21.15.28:0/211135032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5627f7ed6810 con 0x5627f7fac180 2024-04-24T08:11:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.331+0000 7fadc3e00640 1 -- 172.21.15.28:0/211135032 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadc809d2e0 con 0x5627f7fac180 2024-04-24T08:11:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.331+0000 7fadc3e00640 1 -- 172.21.15.28:0/211135032 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fadc80a53c0 con 0x5627f7fac180 2024-04-24T08:11:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.332+0000 7fade7d8f740 10 client.15459 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.332+0000 7fade7d8f740 10 client.15459 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:11:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.332+0000 7fade7d8f740 20 client.15459 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.332+0000 7fade7d8f740 1 --2- 172.21.15.28:0/211135032 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5627f802d6c0 0x5627f802fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.332+0000 7fade7d8f740 4 client.15459 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:11:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.332+0000 7fade7d8f740 1 -- 172.21.15.28:0/211135032 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5627f7f18a80 con 0x5627f802d6c0 2024-04-24T08:11:03.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.332+0000 7faddde00640 1 --2- 172.21.15.28:0/211135032 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5627f802d6c0 0x5627f802fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.332+0000 7faddde00640 1 --2- 172.21.15.28:0/211135032 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5627f802d6c0 0x5627f802fab0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67dbc35ffa824c87 server_cookie=fa99b66618c90052 in_seq=0 out_seq=0 2024-04-24T08:11:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.332+0000 7fadc3e00640 10 client.15459 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.333+0000 7fadc3e00640 1 -- 172.21.15.28:0/211135032 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5627f7f18a80 con 0x5627f802d6c0 2024-04-24T08:11:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.333+0000 7fadc3e00640 10 client.15459 handle_command_reply: tid=0 2024-04-24T08:11:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.333+0000 7fade7d8f740 1 client.15459 shutdown 2024-04-24T08:11:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.333+0000 7fade7d8f740 1 -- 172.21.15.28:0/211135032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fac180 msgr2=0x5627f7fb0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.333+0000 7fade7d8f740 1 --2- 172.21.15.28:0/211135032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fac180 0x5627f7fb0a50 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fadc80d0010 tx=0x7fadc809c720 comp rx=0 tx=0).stop 2024-04-24T08:11:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.333+0000 7fade7d8f740 1 -- 172.21.15.28:0/211135032 shutdown_connections 2024-04-24T08:11:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.333+0000 7fade7d8f740 1 --2- 172.21.15.28:0/211135032 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5627f802d6c0 0x5627f802fab0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.333+0000 7fade7d8f740 1 --2- 172.21.15.28:0/211135032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5627f7fac180 0x5627f7fb0a50 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.333+0000 7fade7d8f740 1 --2- 172.21.15.28:0/211135032 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5627f7fb6850 0x5627f7fb0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.333+0000 7fadc3e00640 0 client.15459 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:03.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.333+0000 7fade7d8f740 1 --2- 172.21.15.28:0/211135032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5627f7fab860 0x5627f7fb0510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.333+0000 7fade7d8f740 1 -- 172.21.15.28:0/211135032 >> 172.21.15.28:0/211135032 conn(0x5627f7fa7b80 msgr2=0x5627f7fb4040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.333+0000 7fade7d8f740 1 -- 172.21.15.28:0/211135032 shutdown_connections 2024-04-24T08:11:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.333+0000 7fade7d8f740 1 -- 172.21.15.28:0/211135032 wait complete. 2024-04-24T08:11:03.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.333+0000 7fade7d8f740 20 client.15459 trim_cache size 0 max 16384 2024-04-24T08:11:03.339 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:11:03.339 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:11:03.339 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:11:03.339 INFO:teuthology.orchestra.run.smithi028.stdout: "049460ae-a3a9-4ec2-a1f2-93e21a5e72dd": { 2024-04-24T08:11:03.339 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:11:03.339 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "049460ae-a3a9-4ec2-a1f2-93e21a5e72dd", 2024-04-24T08:11:03.339 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:11:03.340 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:11:03.340 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:11:03.340 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:11:03.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.334+0000 7fade4e00640 1 -- 172.21.15.28:0/102725117 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fadc40a3f50 msgr2=0x7fadc40a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.334+0000 7fade4e00640 1 --2- 172.21.15.28:0/102725117 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fadc40a3f50 0x7fadc40a6410 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fadd0000f80 tx=0x7fadd0004a40 comp rx=0 tx=0).stop 2024-04-24T08:11:03.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.334+0000 7fade4e00640 1 -- 172.21.15.28:0/102725117 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fade0109520 msgr2=0x7fade01a0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.334+0000 7fade4e00640 1 --2- 172.21.15.28:0/102725117 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fade0109520 0x7fade01a0cb0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fadc800e9c0 tx=0x7fadc800ee80 comp rx=0 tx=0).stop 2024-04-24T08:11:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.334+0000 7fade4e00640 1 -- 172.21.15.28:0/102725117 shutdown_connections 2024-04-24T08:11:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.334+0000 7fade4e00640 1 --2- 172.21.15.28:0/102725117 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fadc40a3f50 0x7fadc40a6410 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.334+0000 7fade4e00640 1 --2- 172.21.15.28:0/102725117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fade019a920 0x7fade019add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.334+0000 7fade4e00640 1 --2- 172.21.15.28:0/102725117 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fade0109520 0x7fade01a0cb0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.334+0000 7fade4e00640 1 --2- 172.21.15.28:0/102725117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fade0104450 0x7fade01a0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.334+0000 7fade4e00640 1 -- 172.21.15.28:0/102725117 >> 172.21.15.28:0/102725117 conn(0x7fade00ffc70 msgr2=0x7fade0101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.334+0000 7fade4e00640 1 -- 172.21.15.28:0/102725117 shutdown_connections 2024-04-24T08:11:03.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:03.334+0000 7fade4e00640 1 -- 172.21.15.28:0/102725117 wait complete. 2024-04-24T08:11:03.355 INFO:tasks.cephfs.filesystem:scrub status for tag:049460ae-a3a9-4ec2-a1f2-93e21a5e72dd - {'path': '/', 'tag': '049460ae-a3a9-4ec2-a1f2-93e21a5e72dd', 'options': 'recursive,force'} 2024-04-24T08:11:03.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:03 smithi028 ceph-mon[27786]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:11:03.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:03 smithi028 ceph-mon[27786]: mgrmap e26: x(active, since 42s), standbys: y, z 2024-04-24T08:11:03.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:03 smithi117 ceph-mon[32438]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:11:03.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:03 smithi117 ceph-mon[32438]: mgrmap e26: x(active, since 42s), standbys: y, z 2024-04-24T08:11:03.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:03 smithi194 ceph-mon[32836]: pgmap v24: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:11:03.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:03 smithi194 ceph-mon[32836]: mgrmap e26: x(active, since 42s), standbys: y, z 2024-04-24T08:11:04.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:04 smithi117 ceph-mon[32438]: scrub queued for path: / 2024-04-24T08:11:04.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:04 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:11:04.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:04 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:11:04.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:04 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:11:04.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:04 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:11:04.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:04 smithi194 ceph-mon[32836]: scrub queued for path: / 2024-04-24T08:11:04.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:04 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:11:04.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:04 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:11:04.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:04 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:11:04.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:04 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:11:04.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:04 smithi028 ceph-mon[27786]: scrub queued for path: / 2024-04-24T08:11:04.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:04 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:11:04.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:04 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:11:04.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:04 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:11:04.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:04 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:11:05.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:05 smithi117 ceph-mon[32438]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:11:05.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:05 smithi194 ceph-mon[32836]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:11:05.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:05 smithi028 ceph-mon[27786]: pgmap v25: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:11:07.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:07 smithi117 ceph-mon[32438]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-24T08:11:07.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:07 smithi194 ceph-mon[32836]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-24T08:11:07.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:07 smithi028 ceph-mon[27786]: pgmap v26: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-24T08:11:09.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:09 smithi117 ceph-mon[32438]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-24T08:11:09.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:09 smithi194 ceph-mon[32836]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-24T08:11:09.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:09 smithi028 ceph-mon[27786]: pgmap v27: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 21 op/s 2024-04-24T08:11:11.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:11 smithi117 ceph-mon[32438]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-24T08:11:11.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:11 smithi194 ceph-mon[32836]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-24T08:11:11.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:11 smithi028 ceph-mon[27786]: pgmap v28: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 20 op/s 2024-04-24T08:11:13.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:13 smithi117 ceph-mon[32438]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-24T08:11:13.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:13 smithi194 ceph-mon[32836]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-24T08:11:13.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:13 smithi028 ceph-mon[27786]: pgmap v29: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 15 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-04-24T08:11:15.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:15 smithi194 ceph-mon[32836]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-24T08:11:15.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:15 smithi028 ceph-mon[27786]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-24T08:11:15.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:15 smithi117 ceph-mon[32438]: pgmap v30: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-04-24T08:11:17.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:17 smithi194 ceph-mon[32836]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-04-24T08:11:17.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:17 smithi028 ceph-mon[27786]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-04-24T08:11:17.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:17 smithi117 ceph-mon[32438]: pgmap v31: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 1.6 KiB/s wr, 8 op/s 2024-04-24T08:11:19.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:19 smithi194 ceph-mon[32836]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-24T08:11:19.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:19 smithi028 ceph-mon[27786]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-24T08:11:19.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:19 smithi117 ceph-mon[32438]: pgmap v32: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-24T08:11:21.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:21 smithi194 ceph-mon[32836]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-24T08:11:21.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:21 smithi028 ceph-mon[27786]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-24T08:11:21.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:21 smithi117 ceph-mon[32438]: pgmap v33: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 852 B/s wr, 5 op/s 2024-04-24T08:11:23.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:23 smithi194 ceph-mon[32836]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-24T08:11:23.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:23 smithi028 ceph-mon[27786]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-24T08:11:23.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:23 smithi117 ceph-mon[32438]: pgmap v34: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 852 B/s wr, 7 op/s 2024-04-24T08:11:25.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:25 smithi194 ceph-mon[32836]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:25.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:25 smithi028 ceph-mon[27786]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:25.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:25 smithi117 ceph-mon[32438]: pgmap v35: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:27.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:27 smithi194 ceph-mon[32836]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:11:27.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:27 smithi028 ceph-mon[27786]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:11:27.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:27 smithi117 ceph-mon[32438]: pgmap v36: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:11:29.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:29 smithi194 ceph-mon[32836]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:29.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:29 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:11:29.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:29 smithi028 ceph-mon[27786]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:29.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:29 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:11:29.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:29 smithi117 ceph-mon[32438]: pgmap v37: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:29.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:29 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:11:31.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:30 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:11:31.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:30 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:11:31.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:30 smithi194 ceph-mon[32836]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:31.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:30 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:11:31.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:30 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:11:31.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:30 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:11:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:30 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:11:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:30 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:11:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:30 smithi028 ceph-mon[27786]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:31.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:30 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:11:31.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:30 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:11:31.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:30 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:11:31.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:30 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:11:31.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:30 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:11:31.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:30 smithi117 ceph-mon[32438]: pgmap v38: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:31.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:30 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:11:31.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:30 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:11:31.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:30 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:11:33.356 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:11:33.356 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:11:33.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:33 smithi028 ceph-mon[27786]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:11:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.562+0000 7f3f10e00640 1 -- 172.21.15.28:0/1984675124 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f0c10e660 msgr2=0x7f3f0c10ea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.562+0000 7f3f10e00640 1 --2- 172.21.15.28:0/1984675124 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f0c10e660 0x7f3f0c10ea40 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3efc009890 tx=0x7f3efc02a360 comp rx=0 tx=0).stop 2024-04-24T08:11:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.562+0000 7f3f10e00640 1 -- 172.21.15.28:0/1984675124 shutdown_connections 2024-04-24T08:11:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.562+0000 7f3f10e00640 1 --2- 172.21.15.28:0/1984675124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f0c108610 0x7f3f0c108a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.562+0000 7f3f10e00640 1 --2- 172.21.15.28:0/1984675124 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f0c108fd0 0x7f3f0c115490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.562+0000 7f3f10e00640 1 --2- 172.21.15.28:0/1984675124 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f0c10e660 0x7f3f0c10ea40 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.562+0000 7f3f10e00640 1 -- 172.21.15.28:0/1984675124 >> 172.21.15.28:0/1984675124 conn(0x7f3f0c076540 msgr2=0x7f3f0c078960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.562+0000 7f3f10e00640 1 -- 172.21.15.28:0/1984675124 shutdown_connections 2024-04-24T08:11:33.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.562+0000 7f3f10e00640 1 -- 172.21.15.28:0/1984675124 wait complete. 2024-04-24T08:11:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.563+0000 7f3f10e00640 1 Processor -- start 2024-04-24T08:11:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.563+0000 7f3f10e00640 1 -- start start 2024-04-24T08:11:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.564+0000 7f3f10e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f0c108610 0x7f3f0c19ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.564+0000 7f3f10e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f0c108fd0 0x7f3f0c19f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.564+0000 7f3f10e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f0c10e660 0x7f3f0c1a3740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.564+0000 7f3f10e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c1189e0 con 0x7f3f0c108610 2024-04-24T08:11:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.564+0000 7f3f10e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c118860 con 0x7f3f0c108fd0 2024-04-24T08:11:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.564+0000 7f3f10e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c118b60 con 0x7f3f0c10e660 2024-04-24T08:11:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.564+0000 7f3f09e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f0c108610 0x7f3f0c19ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.564+0000 7f3f09400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f0c108fd0 0x7f3f0c19f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.564+0000 7f3f09400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f0c108fd0 0x7f3f0c19f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49336/0 (socket says 172.21.15.28:49336) 2024-04-24T08:11:33.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.564+0000 7f3f09400640 1 -- 172.21.15.28:0/3254281832 learned_addr learned my addr 172.21.15.28:0/3254281832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.564+0000 7f3f0a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f0c10e660 0x7f3f0c1a3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.564+0000 7f3f09400640 1 -- 172.21.15.28:0/3254281832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f0c10e660 msgr2=0x7f3f0c1a3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.564+0000 7f3f09400640 1 --2- 172.21.15.28:0/3254281832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f0c10e660 0x7f3f0c1a3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.564+0000 7f3f09400640 1 -- 172.21.15.28:0/3254281832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f0c108610 msgr2=0x7f3f0c19ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.564+0000 7f3f09400640 1 --2- 172.21.15.28:0/3254281832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f0c108610 0x7f3f0c19ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.564+0000 7f3f09400640 1 -- 172.21.15.28:0/3254281832 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f0c1a3e70 con 0x7f3f0c108fd0 2024-04-24T08:11:33.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.565+0000 7f3f09400640 1 --2- 172.21.15.28:0/3254281832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f0c108fd0 0x7f3f0c19f3c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800e9c0 tx=0x7f3ef800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.565+0000 7f3f02a00640 1 -- 172.21.15.28:0/3254281832 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef800cde0 con 0x7f3f0c108fd0 2024-04-24T08:11:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.565+0000 7f3f02a00640 1 -- 172.21.15.28:0/3254281832 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ef80035f0 con 0x7f3f0c108fd0 2024-04-24T08:11:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.565+0000 7f3f02a00640 1 -- 172.21.15.28:0/3254281832 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef8010690 con 0x7f3f0c108fd0 2024-04-24T08:11:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.565+0000 7f3f10e00640 1 -- 172.21.15.28:0/3254281832 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f0c1a4160 con 0x7f3f0c108fd0 2024-04-24T08:11:33.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.565+0000 7f3f10e00640 1 -- 172.21.15.28:0/3254281832 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f0c1ab9f0 con 0x7f3f0c108fd0 2024-04-24T08:11:33.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.569+0000 7f3f13c48740 1 Processor -- start 2024-04-24T08:11:33.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.569+0000 7f3f13c48740 1 -- start start 2024-04-24T08:11:33.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.569+0000 7f3f13c48740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330a860 0x56067330ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.569+0000 7f3f13c48740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330b180 0x560673315310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.569+0000 7f3f13c48740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560673315850 0x560673317c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.569+0000 7f3f13c48740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5606732701f0 con 0x56067330a860 2024-04-24T08:11:33.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.569+0000 7f3f13c48740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560673216350 con 0x56067330b180 2024-04-24T08:11:33.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.569+0000 7f3f13c48740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56067326c1b0 con 0x560673315850 2024-04-24T08:11:33.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.569+0000 7f3f02a00640 1 -- 172.21.15.28:0/3254281832 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 26) ==== 138930+0+0 (secure 0 0 0) 0x7f3ef80037a0 con 0x7f3f0c108fd0 2024-04-24T08:11:33.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.570+0000 7f3f0a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330a860 0x56067330ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.570+0000 7f3f09e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330b180 0x560673315310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.570+0000 7f3f09e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330b180 0x560673315310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49352/0 (socket says 172.21.15.28:49352) 2024-04-24T08:11:33.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.570+0000 7f3f0a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330a860 0x56067330ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57280/0 (socket says 172.21.15.28:57280) 2024-04-24T08:11:33.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.570+0000 7f3f09e00640 1 -- 172.21.15.28:0/1163052476 learned_addr learned my addr 172.21.15.28:0/1163052476 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:33.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.570+0000 7f3f09400640 1 --2- 172.21.15.28:0/1163052476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560673315850 0x560673317c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.570+0000 7f3f0a800640 1 -- 172.21.15.28:0/1163052476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560673315850 msgr2=0x560673317c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:33.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.570+0000 7f3f0a800640 1 --2- 172.21.15.28:0/1163052476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560673315850 0x560673317c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.570+0000 7f3f0a800640 1 -- 172.21.15.28:0/1163052476 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330b180 msgr2=0x560673315310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:33.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.570+0000 7f3f0a800640 1 --2- 172.21.15.28:0/1163052476 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330b180 0x560673315310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.570+0000 7f3f0a800640 1 -- 172.21.15.28:0/1163052476 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56067326af50 con 0x56067330a860 2024-04-24T08:11:33.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.570+0000 7f3f0a800640 1 --2- 172.21.15.28:0/1163052476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330a860 0x56067330ac40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009f00 tx=0x7f3f04011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b417c9d53b2b5e23 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:33.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.571+0000 7f3ef7400640 1 -- 172.21.15.28:0/1163052476 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f04004d20 con 0x56067330a860 2024-04-24T08:11:33.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.571+0000 7f3ef7400640 1 -- 172.21.15.28:0/1163052476 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f04004030 con 0x56067330a860 2024-04-24T08:11:33.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.571+0000 7f3ef7400640 1 -- 172.21.15.28:0/1163052476 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f04005280 con 0x56067330a860 2024-04-24T08:11:33.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.571+0000 7f3f02a00640 1 --2- 172.21.15.28:0/3254281832 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3ef00a3f50 0x7f3ef00a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.572+0000 7f3f13c48740 1 -- 172.21.15.28:0/1163052476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330a860 msgr2=0x56067330ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.572+0000 7f3f13c48740 1 --2- 172.21.15.28:0/1163052476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330a860 0x56067330ac40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f3f04009f00 tx=0x7f3f04011810 comp rx=0 tx=0).stop 2024-04-24T08:11:33.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.572+0000 7f3f02a00640 1 -- 172.21.15.28:0/3254281832 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f3ef80d0d60 con 0x7f3f0c108fd0 2024-04-24T08:11:33.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.572+0000 7f3f09e00640 1 --2- 172.21.15.28:0/3254281832 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3ef00a3f50 0x7f3ef00a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.572+0000 7f3f13c48740 1 -- 172.21.15.28:0/1163052476 shutdown_connections 2024-04-24T08:11:33.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.572+0000 7f3f13c48740 1 --2- 172.21.15.28:0/1163052476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560673315850 0x560673317c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.572+0000 7f3f13c48740 1 --2- 172.21.15.28:0/1163052476 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330b180 0x560673315310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.572+0000 7f3f13c48740 1 --2- 172.21.15.28:0/1163052476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330a860 0x56067330ac40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.572+0000 7f3f13c48740 1 -- 172.21.15.28:0/1163052476 >> 172.21.15.28:0/1163052476 conn(0x560673306b80 msgr2=0x560673306f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:33.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.572+0000 7f3f13c48740 1 -- 172.21.15.28:0/1163052476 shutdown_connections 2024-04-24T08:11:33.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.572+0000 7f3f13c48740 1 -- 172.21.15.28:0/1163052476 wait complete. 2024-04-24T08:11:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.572+0000 7f3f09e00640 1 --2- 172.21.15.28:0/3254281832 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3ef00a3f50 0x7f3ef00a6410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3efc005e00 tx=0x7f3efc0056d0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:33.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.573+0000 7f3f13c48740 1 -- start start 2024-04-24T08:11:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.573+0000 7f3f13c48740 1 Processor -- start 2024-04-24T08:11:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.573+0000 7f3f13c48740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330a860 0x560673312800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.573+0000 7f3f13c48740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330b180 0x560673312d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.573+0000 7f3f13c48740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560673315850 0x560673313280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.573+0000 7f3f13c48740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560673318640 con 0x56067330b180 2024-04-24T08:11:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.573+0000 7f3f13c48740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560673229690 con 0x56067330a860 2024-04-24T08:11:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.573+0000 7f3f13c48740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5606733187c0 con 0x560673315850 2024-04-24T08:11:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.574+0000 7f3f13c48740 10 client.0 _ll_register_callbacks cb 0x560673265650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.574+0000 7f3f09400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330b180 0x560673312d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.574+0000 7f3f09400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330b180 0x560673312d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57288/0 (socket says 172.21.15.28:57288) 2024-04-24T08:11:33.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.574+0000 7f3f09400640 1 -- 172.21.15.28:0/2843438468 learned_addr learned my addr 172.21.15.28:0/2843438468 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.574+0000 7f3f09e00640 1 --2- 172.21.15.28:0/2843438468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330a860 0x560673312800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.574+0000 7f3f0a800640 1 --2- 172.21.15.28:0/2843438468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560673315850 0x560673313280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.574+0000 7f3f09400640 1 -- 172.21.15.28:0/2843438468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560673315850 msgr2=0x560673313280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.574+0000 7f3f09400640 1 --2- 172.21.15.28:0/2843438468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560673315850 0x560673313280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.574+0000 7f3f09400640 1 -- 172.21.15.28:0/2843438468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330a860 msgr2=0x560673312800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.574+0000 7f3f09400640 1 --2- 172.21.15.28:0/2843438468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330a860 0x560673312800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.574+0000 7f3f09400640 1 -- 172.21.15.28:0/2843438468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560673228920 con 0x56067330b180 2024-04-24T08:11:33.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.575+0000 7f3f09400640 1 --2- 172.21.15.28:0/2843438468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330b180 0x560673312d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3ef80ce340 tx=0x7f3ef809c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8f22e36d3a78e2df server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.575+0000 7f3eebe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:11:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.575+0000 7f3eebe00640 1 -- 172.21.15.28:0/2843438468 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef80cede0 con 0x56067330b180 2024-04-24T08:11:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.575+0000 7f3eebe00640 1 -- 172.21.15.28:0/2843438468 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f3ef809bef0 con 0x56067330b180 2024-04-24T08:11:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.575+0000 7f3f13c48740 1 -- 172.21.15.28:0/2843438468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f04004030 con 0x56067330b180 2024-04-24T08:11:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.575+0000 7f3eebe00640 1 -- 172.21.15.28:0/2843438468 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3ef80a0e50 con 0x56067330b180 2024-04-24T08:11:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.575+0000 7f3f13c48740 10 client.15471 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.575+0000 7f3f13c48740 1 -- 172.21.15.28:0/2843438468 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56067304c7d0 con 0x56067330b180 2024-04-24T08:11:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.575+0000 7f3eebe00640 1 -- 172.21.15.28:0/2843438468 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f04004030 con 0x56067330b180 2024-04-24T08:11:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.576+0000 7f3eebe00640 1 -- 172.21.15.28:0/2843438468 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3ef80b9080 con 0x56067330b180 2024-04-24T08:11:33.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.576+0000 7f3f13c48740 10 client.15471 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.576+0000 7f3f13c48740 10 client.15471 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:11:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.576+0000 7f3f13c48740 20 client.15471 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.576+0000 7f3f13c48740 1 --2- 172.21.15.28:0/2843438468 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x560673325a80 0x560673327eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.576+0000 7f3f13c48740 4 client.15471 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:11:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.576+0000 7f3f13c48740 1 -- 172.21.15.28:0/2843438468 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56067327dc10 con 0x560673325a80 2024-04-24T08:11:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.576+0000 7f3f09e00640 1 --2- 172.21.15.28:0/2843438468 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x560673325a80 0x560673327eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.577+0000 7f3f09e00640 1 --2- 172.21.15.28:0/2843438468 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x560673325a80 0x560673327eb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37056fc52fc51b54 server_cookie=976b14841075e104 in_seq=0 out_seq=0 2024-04-24T08:11:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.577+0000 7f3eebe00640 10 client.15471 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.577+0000 7f3eebe00640 1 -- 172.21.15.28:0/2843438468 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56067327dc10 con 0x560673325a80 2024-04-24T08:11:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.577+0000 7f3eebe00640 10 client.15471 handle_command_reply: tid=0 2024-04-24T08:11:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.577+0000 7f3f13c48740 1 client.15471 shutdown 2024-04-24T08:11:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.578+0000 7f3f13c48740 1 -- 172.21.15.28:0/2843438468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330b180 msgr2=0x560673312d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.578+0000 7f3f13c48740 1 --2- 172.21.15.28:0/2843438468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330b180 0x560673312d40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3ef80ce340 tx=0x7f3ef809c400 comp rx=0 tx=0).stop 2024-04-24T08:11:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.578+0000 7f3f13c48740 1 -- 172.21.15.28:0/2843438468 shutdown_connections 2024-04-24T08:11:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.578+0000 7f3f13c48740 1 --2- 172.21.15.28:0/2843438468 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x560673325a80 0x560673327eb0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.578+0000 7f3f13c48740 1 --2- 172.21.15.28:0/2843438468 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330b180 0x560673312d40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.578+0000 7f3f13c48740 1 --2- 172.21.15.28:0/2843438468 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560673315850 0x560673313280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.578+0000 7f3f13c48740 1 --2- 172.21.15.28:0/2843438468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330a860 0x560673312800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.578+0000 7f3f13c48740 1 -- 172.21.15.28:0/2843438468 >> 172.21.15.28:0/2843438468 conn(0x560673306b80 msgr2=0x56067330c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.578+0000 7f3eebe00640 0 client.15471 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:33.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.578+0000 7f3f13c48740 1 -- 172.21.15.28:0/2843438468 shutdown_connections 2024-04-24T08:11:33.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.578+0000 7f3f13c48740 1 -- 172.21.15.28:0/2843438468 wait complete. 2024-04-24T08:11:33.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.578+0000 7f3f13c48740 20 client.15471 trim_cache size 0 max 16384 2024-04-24T08:11:33.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.601+0000 7f3f13c48740 1 Processor -- start 2024-04-24T08:11:33.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.601+0000 7f3f13c48740 1 -- start start 2024-04-24T08:11:33.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.601+0000 7f3f13c48740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56067330a860 0x560673321500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.602+0000 7f3f13c48740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330b180 0x560673311740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.602+0000 7f3f13c48740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560673315850 0x56067330d0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.602+0000 7f3f13c48740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c1b6c70 con 0x56067330b180 2024-04-24T08:11:33.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.602+0000 7f3f13c48740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c1b4540 con 0x560673315850 2024-04-24T08:11:33.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.602+0000 7f3f13c48740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c1b2390 con 0x56067330a860 2024-04-24T08:11:33.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.602+0000 7f3f09e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56067330a860 0x560673321500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.602+0000 7f3f09e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56067330a860 0x560673321500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60026/0 (socket says 172.21.15.28:60026) 2024-04-24T08:11:33.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.602+0000 7f3f09e00640 1 -- 172.21.15.28:0/1048134613 learned_addr learned my addr 172.21.15.28:0/1048134613 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:33.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.602+0000 7f3f0a800640 1 --2- 172.21.15.28:0/1048134613 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560673315850 0x56067330d0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.602+0000 7f3f09400640 1 --2- 172.21.15.28:0/1048134613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330b180 0x560673311740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.602+0000 7f3f09e00640 1 -- 172.21.15.28:0/1048134613 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560673315850 msgr2=0x56067330d0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:33.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.602+0000 7f3f09e00640 1 --2- 172.21.15.28:0/1048134613 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560673315850 0x56067330d0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.602+0000 7f3f09e00640 1 -- 172.21.15.28:0/1048134613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330b180 msgr2=0x560673311740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:33.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.602+0000 7f3f09e00640 1 --2- 172.21.15.28:0/1048134613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330b180 0x560673311740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.602+0000 7f3f09e00640 1 -- 172.21.15.28:0/1048134613 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560672f823f0 con 0x56067330a860 2024-04-24T08:11:33.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.602+0000 7f3f09400640 1 --2- 172.21.15.28:0/1048134613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330b180 0x560673311740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:11:33.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.603+0000 7f3f09e00640 1 --2- 172.21.15.28:0/1048134613 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56067330a860 0x560673321500 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3efc035040 tx=0x7f3efc03d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e217ed019952365 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:33.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.603+0000 7f3ef6a00640 1 -- 172.21.15.28:0/1048134613 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc037040 con 0x56067330a860 2024-04-24T08:11:33.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.603+0000 7f3ef6a00640 1 -- 172.21.15.28:0/1048134613 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3efc054070 con 0x56067330a860 2024-04-24T08:11:33.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.603+0000 7f3ef6a00640 1 -- 172.21.15.28:0/1048134613 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3efc003ab0 con 0x56067330a860 2024-04-24T08:11:33.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.604+0000 7f3f13c48740 1 -- 172.21.15.28:0/1048134613 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56067330a860 msgr2=0x560673321500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.604+0000 7f3f13c48740 1 --2- 172.21.15.28:0/1048134613 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56067330a860 0x560673321500 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f3efc035040 tx=0x7f3efc03d000 comp rx=0 tx=0).stop 2024-04-24T08:11:33.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.604+0000 7f3f13c48740 1 -- 172.21.15.28:0/1048134613 shutdown_connections 2024-04-24T08:11:33.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.604+0000 7f3f13c48740 1 --2- 172.21.15.28:0/1048134613 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330b180 0x560673311740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.604+0000 7f3f13c48740 1 --2- 172.21.15.28:0/1048134613 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560673315850 0x56067330d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.604+0000 7f3f13c48740 1 --2- 172.21.15.28:0/1048134613 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56067330a860 0x560673321500 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.604+0000 7f3f13c48740 1 -- 172.21.15.28:0/1048134613 >> 172.21.15.28:0/1048134613 conn(0x560673306b80 msgr2=0x560673305c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:33.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.604+0000 7f3f13c48740 1 -- 172.21.15.28:0/1048134613 shutdown_connections 2024-04-24T08:11:33.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.604+0000 7f3f13c48740 1 -- 172.21.15.28:0/1048134613 wait complete. 2024-04-24T08:11:33.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.604+0000 7f3f13c48740 1 -- start start 2024-04-24T08:11:33.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.605+0000 7f3f13c48740 1 Processor -- start 2024-04-24T08:11:33.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.605+0000 7f3f13c48740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330a860 0x5606733193b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.605+0000 7f3f13c48740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330b180 0x5606733198f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.605+0000 7f3f13c48740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560673315850 0x560673319e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.605+0000 7f3f13c48740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c1b2a00 con 0x56067330a860 2024-04-24T08:11:33.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.605+0000 7f3f13c48740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c1b3e70 con 0x56067330b180 2024-04-24T08:11:33.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.605+0000 7f3f13c48740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3f0c1b30d0 con 0x560673315850 2024-04-24T08:11:33.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.605+0000 7f3f13c48740 10 client.0 _ll_register_callbacks cb 0x7f3ef809c870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:33.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.605+0000 7f3f09e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330a860 0x5606733193b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.605+0000 7f3f09400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330b180 0x5606733198f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.605+0000 7f3f09e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330a860 0x5606733193b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57296/0 (socket says 172.21.15.28:57296) 2024-04-24T08:11:33.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.605+0000 7f3f09400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330b180 0x5606733198f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49372/0 (socket says 172.21.15.28:49372) 2024-04-24T08:11:33.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.605+0000 7f3f09e00640 1 -- 172.21.15.28:0/3587144255 learned_addr learned my addr 172.21.15.28:0/3587144255 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:33.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.606+0000 7f3f0a800640 1 --2- 172.21.15.28:0/3587144255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560673315850 0x560673319e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.606+0000 7f3f09400640 1 -- 172.21.15.28:0/3587144255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560673315850 msgr2=0x560673319e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.606+0000 7f3f09400640 1 --2- 172.21.15.28:0/3587144255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560673315850 0x560673319e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.606+0000 7f3f09400640 1 -- 172.21.15.28:0/3587144255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330a860 msgr2=0x5606733193b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.606+0000 7f3f09400640 1 --2- 172.21.15.28:0/3587144255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330a860 0x5606733193b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.606+0000 7f3f09400640 1 -- 172.21.15.28:0/3587144255 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560672f420f0 con 0x56067330b180 2024-04-24T08:11:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.606+0000 7f3f09e00640 1 --2- 172.21.15.28:0/3587144255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330a860 0x5606733193b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.606+0000 7f3f09400640 1 --2- 172.21.15.28:0/3587144255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330b180 0x5606733198f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x560673318720 tx=0x7f3ef809c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4ae9c8ec49662ba8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.606+0000 7f3eebe00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:11:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.606+0000 7f3eebe00640 1 -- 172.21.15.28:0/3587144255 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3ef80a5710 con 0x56067330b180 2024-04-24T08:11:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.606+0000 7f3eebe00640 1 -- 172.21.15.28:0/3587144255 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f3ef8004000 con 0x56067330b180 2024-04-24T08:11:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.606+0000 7f3f13c48740 1 -- 172.21.15.28:0/3587144255 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3efc054070 con 0x56067330b180 2024-04-24T08:11:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.607+0000 7f3eebe00640 1 -- 172.21.15.28:0/3587144255 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3ef8098d40 con 0x56067330b180 2024-04-24T08:11:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.607+0000 7f3f13c48740 10 client.25574 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.607+0000 7f3f13c48740 1 -- 172.21.15.28:0/3587144255 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560673235810 con 0x56067330b180 2024-04-24T08:11:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.607+0000 7f3eebe00640 1 -- 172.21.15.28:0/3587144255 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3efc054070 con 0x56067330b180 2024-04-24T08:11:33.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.607+0000 7f3eebe00640 1 -- 172.21.15.28:0/3587144255 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3ef80a1fe0 con 0x56067330b180 2024-04-24T08:11:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.607+0000 7f3f13c48740 10 client.25574 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.607+0000 7f3f13c48740 10 client.25574 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:11:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.607+0000 7f3f13c48740 20 client.25574 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.607+0000 7f3f13c48740 1 --2- 172.21.15.28:0/3587144255 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56067338c320 0x56067338e710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.607+0000 7f3f13c48740 4 client.25574 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:11:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.607+0000 7f3f13c48740 1 -- 172.21.15.28:0/3587144255 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x560673277a80 con 0x56067338c320 2024-04-24T08:11:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.607+0000 7f3f09e00640 1 --2- 172.21.15.28:0/3587144255 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56067338c320 0x56067338e710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.608+0000 7f3f09e00640 1 --2- 172.21.15.28:0/3587144255 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56067338c320 0x56067338e710 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78bad7bb0268d123 server_cookie=3ad587d04639b36b in_seq=0 out_seq=0 2024-04-24T08:11:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.608+0000 7f3eebe00640 10 client.25574 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.608+0000 7f3eebe00640 1 -- 172.21.15.28:0/3587144255 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x560673277a80 con 0x56067338c320 2024-04-24T08:11:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.608+0000 7f3eebe00640 10 client.25574 handle_command_reply: tid=0 2024-04-24T08:11:33.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.608+0000 7f3f13c48740 1 client.25574 shutdown 2024-04-24T08:11:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.608+0000 7f3f13c48740 1 -- 172.21.15.28:0/3587144255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330b180 msgr2=0x5606733198f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.608+0000 7f3f13c48740 1 --2- 172.21.15.28:0/3587144255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330b180 0x5606733198f0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x560673318720 tx=0x7f3ef809c530 comp rx=0 tx=0).stop 2024-04-24T08:11:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.609+0000 7f3f13c48740 1 -- 172.21.15.28:0/3587144255 shutdown_connections 2024-04-24T08:11:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.609+0000 7f3f13c48740 1 --2- 172.21.15.28:0/3587144255 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56067338c320 0x56067338e710 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.609+0000 7f3f13c48740 1 --2- 172.21.15.28:0/3587144255 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560673315850 0x560673319e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.609+0000 7f3f13c48740 1 --2- 172.21.15.28:0/3587144255 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56067330b180 0x5606733198f0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.609+0000 7f3eebe00640 0 client.25574 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.609+0000 7f3f13c48740 1 --2- 172.21.15.28:0/3587144255 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56067330a860 0x5606733193b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.609+0000 7f3f13c48740 1 -- 172.21.15.28:0/3587144255 >> 172.21.15.28:0/3587144255 conn(0x560673306b80 msgr2=0x5606733127c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.609+0000 7f3f13c48740 1 -- 172.21.15.28:0/3587144255 shutdown_connections 2024-04-24T08:11:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.609+0000 7f3f13c48740 1 -- 172.21.15.28:0/3587144255 wait complete. 2024-04-24T08:11:33.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.609+0000 7f3f13c48740 20 client.25574 trim_cache size 0 max 16384 2024-04-24T08:11:33.614 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:11:33.614 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-24T08:11:33.614 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-24T08:11:33.614 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:11:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.609+0000 7f3f10e00640 1 -- 172.21.15.28:0/3254281832 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3ef00a3f50 msgr2=0x7f3ef00a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.609+0000 7f3f10e00640 1 --2- 172.21.15.28:0/3254281832 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3ef00a3f50 0x7f3ef00a6410 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f3efc005e00 tx=0x7f3efc0056d0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.609+0000 7f3f10e00640 1 -- 172.21.15.28:0/3254281832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f0c108fd0 msgr2=0x7f3f0c19f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.609+0000 7f3f10e00640 1 --2- 172.21.15.28:0/3254281832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f0c108fd0 0x7f3f0c19f3c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f3ef800e9c0 tx=0x7f3ef800ee80 comp rx=0 tx=0).stop 2024-04-24T08:11:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.610+0000 7f3f10e00640 1 -- 172.21.15.28:0/3254281832 shutdown_connections 2024-04-24T08:11:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.610+0000 7f3f10e00640 1 --2- 172.21.15.28:0/3254281832 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3ef00a3f50 0x7f3ef00a6410 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.610+0000 7f3f10e00640 1 --2- 172.21.15.28:0/3254281832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f0c10e660 0x7f3f0c1a3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.610+0000 7f3f10e00640 1 --2- 172.21.15.28:0/3254281832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f0c108fd0 0x7f3f0c19f3c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.610+0000 7f3f10e00640 1 --2- 172.21.15.28:0/3254281832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f0c108610 0x7f3f0c19ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.610+0000 7f3f10e00640 1 -- 172.21.15.28:0/3254281832 >> 172.21.15.28:0/3254281832 conn(0x7f3f0c076540 msgr2=0x7f3f0c077c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:33.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.610+0000 7f3f10e00640 1 -- 172.21.15.28:0/3254281832 shutdown_connections 2024-04-24T08:11:33.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.610+0000 7f3f10e00640 1 -- 172.21.15.28:0/3254281832 wait complete. 2024-04-24T08:11:33.626 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:11:33.626 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:11:33.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:33 smithi117 ceph-mon[32438]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:11:33.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:33 smithi194 ceph-mon[32836]: pgmap v39: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:11:33.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.823+0000 7f88de400640 1 -- 172.21.15.28:0/4169977553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8069400 msgr2=0x7f88d80697e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.823+0000 7f88de400640 1 --2- 172.21.15.28:0/4169977553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8069400 0x7f88d80697e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f88cc009900 tx=0x7f88cc02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.823+0000 7f88de400640 1 -- 172.21.15.28:0/4169977553 shutdown_connections 2024-04-24T08:11:33.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.823+0000 7f88de400640 1 --2- 172.21.15.28:0/4169977553 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88d81071a0 0x7f88d810ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.823+0000 7f88de400640 1 --2- 172.21.15.28:0/4169977553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88d8069db0 0x7f88d8106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.823+0000 7f88de400640 1 --2- 172.21.15.28:0/4169977553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8069400 0x7f88d80697e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.823+0000 7f88de400640 1 -- 172.21.15.28:0/4169977553 >> 172.21.15.28:0/4169977553 conn(0x7f88d8076840 msgr2=0x7f88d8076c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:33.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.823+0000 7f88de400640 1 -- 172.21.15.28:0/4169977553 shutdown_connections 2024-04-24T08:11:33.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.823+0000 7f88de400640 1 -- 172.21.15.28:0/4169977553 wait complete. 2024-04-24T08:11:33.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.824+0000 7f88de400640 1 Processor -- start 2024-04-24T08:11:33.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.824+0000 7f88de400640 1 -- start start 2024-04-24T08:11:33.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.824+0000 7f88de400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8069400 0x7f88d81a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.825+0000 7f88de400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88d81071a0 0x7f88d81a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.825+0000 7f88de400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88d819a760 0x7f88d819abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.825+0000 7f88de400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88d8112280 con 0x7f88d8069400 2024-04-24T08:11:33.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.825+0000 7f88de400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f88d8112100 con 0x7f88d819a760 2024-04-24T08:11:33.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.825+0000 7f88de400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f88d8112400 con 0x7f88d81071a0 2024-04-24T08:11:33.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.825+0000 7f88dd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8069400 0x7f88d81a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.825+0000 7f88d7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88d81071a0 0x7f88d81a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.825+0000 7f88dd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8069400 0x7f88d81a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57316/0 (socket says 172.21.15.28:57316) 2024-04-24T08:11:33.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.825+0000 7f88dd000640 1 -- 172.21.15.28:0/2606626993 learned_addr learned my addr 172.21.15.28:0/2606626993 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:33.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.825+0000 7f88dda00640 1 --2- 172.21.15.28:0/2606626993 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88d819a760 0x7f88d819abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.825+0000 7f88d7e00640 1 -- 172.21.15.28:0/2606626993 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88d819a760 msgr2=0x7f88d819abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.825+0000 7f88d7e00640 1 --2- 172.21.15.28:0/2606626993 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88d819a760 0x7f88d819abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.825+0000 7f88d7e00640 1 -- 172.21.15.28:0/2606626993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8069400 msgr2=0x7f88d81a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:33.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.825+0000 7f88d7e00640 1 --2- 172.21.15.28:0/2606626993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8069400 0x7f88d81a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:33.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.825+0000 7f88d7e00640 1 -- 172.21.15.28:0/2606626993 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88d819b360 con 0x7f88d81071a0 2024-04-24T08:11:33.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.826+0000 7f88d7e00640 1 --2- 172.21.15.28:0/2606626993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88d81071a0 0x7f88d81a0be0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f88d8077ec0 tx=0x7f88c400ee50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:33.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.826+0000 7f88d5600640 1 -- 172.21.15.28:0/2606626993 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c400cc40 con 0x7f88d81071a0 2024-04-24T08:11:33.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.826+0000 7f88d5600640 1 -- 172.21.15.28:0/2606626993 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88c400cde0 con 0x7f88d81071a0 2024-04-24T08:11:33.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.826+0000 7f88de400640 1 -- 172.21.15.28:0/2606626993 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88d81a7440 con 0x7f88d81071a0 2024-04-24T08:11:33.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.826+0000 7f88d5600640 1 -- 172.21.15.28:0/2606626993 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c40188e0 con 0x7f88d81071a0 2024-04-24T08:11:33.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.826+0000 7f88de400640 1 -- 172.21.15.28:0/2606626993 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88d81a7840 con 0x7f88d81071a0 2024-04-24T08:11:33.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.827+0000 7f88de400640 1 -- 172.21.15.28:0/2606626993 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f88d8103550 con 0x7f88d81071a0 2024-04-24T08:11:33.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.831+0000 7f88d5600640 1 -- 172.21.15.28:0/2606626993 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 26) ==== 138930+0+0 (secure 0 0 0) 0x7f88c4018a80 con 0x7f88d81071a0 2024-04-24T08:11:33.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.832+0000 7f88d5600640 1 --2- 172.21.15.28:0/2606626993 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88c00a3ed0 0x7f88c00a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:33.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.832+0000 7f88d5600640 1 -- 172.21.15.28:0/2606626993 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f88c40d0e70 con 0x7f88d81071a0 2024-04-24T08:11:33.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.832+0000 7f88dd000640 1 --2- 172.21.15.28:0/2606626993 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88c00a3ed0 0x7f88c00a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:33.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.833+0000 7f88dd000640 1 --2- 172.21.15.28:0/2606626993 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88c00a3ed0 0x7f88c00a6390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f88cc0096c0 tx=0x7f88cc0046c0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:33.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:33.833+0000 7f88d5600640 1 -- 172.21.15.28:0/2606626993 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f88c409cc60 con 0x7f88d81071a0 2024-04-24T08:11:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.077+0000 7f88de400640 1 -- 172.21.15.28:0/2606626993 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f88d8068d40 con 0x7f88d81071a0 2024-04-24T08:11:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.078+0000 7f88d5600640 1 -- 172.21.15.28:0/2606626993 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11871 (secure 0 0 0) 0x7f88c40a0f60 con 0x7f88d81071a0 2024-04-24T08:11:34.078 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:11:34.078 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:10:19.714831+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24574,24556,24572]},"id":1}]} 2024-04-24T08:11:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-24T08:11:34.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.080+0000 7f88de400640 1 -- 172.21.15.28:0/2606626993 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88c00a3ed0 msgr2=0x7f88c00a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.081+0000 7f88de400640 1 --2- 172.21.15.28:0/2606626993 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88c00a3ed0 0x7f88c00a6390 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f88cc0096c0 tx=0x7f88cc0046c0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.081+0000 7f88de400640 1 -- 172.21.15.28:0/2606626993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88d81071a0 msgr2=0x7f88d81a0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.081+0000 7f88de400640 1 --2- 172.21.15.28:0/2606626993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88d81071a0 0x7f88d81a0be0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f88d8077ec0 tx=0x7f88c400ee50 comp rx=0 tx=0).stop 2024-04-24T08:11:34.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.081+0000 7f88de400640 1 -- 172.21.15.28:0/2606626993 shutdown_connections 2024-04-24T08:11:34.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.081+0000 7f88de400640 1 --2- 172.21.15.28:0/2606626993 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88c00a3ed0 0x7f88c00a6390 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.081+0000 7f88de400640 1 --2- 172.21.15.28:0/2606626993 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88d819a760 0x7f88d819abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.081+0000 7f88de400640 1 --2- 172.21.15.28:0/2606626993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88d81071a0 0x7f88d81a0be0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.081+0000 7f88de400640 1 --2- 172.21.15.28:0/2606626993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88d8069400 0x7f88d81a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.081+0000 7f88de400640 1 -- 172.21.15.28:0/2606626993 >> 172.21.15.28:0/2606626993 conn(0x7f88d8076840 msgr2=0x7f88d80fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.081+0000 7f88de400640 1 -- 172.21.15.28:0/2606626993 shutdown_connections 2024-04-24T08:11:34.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.081+0000 7f88de400640 1 -- 172.21.15.28:0/2606626993 wait complete. 2024-04-24T08:11:34.101 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:11:34.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.294+0000 7f1568e00640 1 -- 172.21.15.28:0/3979044984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15640743a0 msgr2=0x7f1564074780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.294+0000 7f1568e00640 1 --2- 172.21.15.28:0/3979044984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15640743a0 0x7f1564074780 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1554009900 tx=0x7f155402a480 comp rx=0 tx=0).stop 2024-04-24T08:11:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.295+0000 7f1568e00640 1 -- 172.21.15.28:0/3979044984 shutdown_connections 2024-04-24T08:11:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.295+0000 7f1568e00640 1 --2- 172.21.15.28:0/3979044984 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1564112eb0 0x7f15641152a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.295+0000 7f1568e00640 1 --2- 172.21.15.28:0/3979044984 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1564074cc0 0x7f1564112970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.295+0000 7f1568e00640 1 --2- 172.21.15.28:0/3979044984 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f15640743a0 0x7f1564074780 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.295+0000 7f1568e00640 1 -- 172.21.15.28:0/3979044984 >> 172.21.15.28:0/3979044984 conn(0x7f15640fd9d0 msgr2=0x7f15640ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.295+0000 7f1568e00640 1 -- 172.21.15.28:0/3979044984 shutdown_connections 2024-04-24T08:11:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.295+0000 7f1568e00640 1 -- 172.21.15.28:0/3979044984 wait complete. 2024-04-24T08:11:34.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.296+0000 7f1568e00640 1 Processor -- start 2024-04-24T08:11:34.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.296+0000 7f1568e00640 1 -- start start 2024-04-24T08:11:34.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.296+0000 7f1568e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f15640743a0 0x7f156419edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.296+0000 7f1568e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564074cc0 0x7f156419f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.296+0000 7f1568e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1564112eb0 0x7f15641a3690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.296+0000 7f1568e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15641188d0 con 0x7f1564074cc0 2024-04-24T08:11:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.296+0000 7f1568e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1564118750 con 0x7f1564112eb0 2024-04-24T08:11:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.296+0000 7f1568e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1564118a50 con 0x7f15640743a0 2024-04-24T08:11:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.296+0000 7f1562a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564074cc0 0x7f156419f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.297+0000 7f1562a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564074cc0 0x7f156419f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57342/0 (socket says 172.21.15.28:57342) 2024-04-24T08:11:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.297+0000 7f1562a00640 1 -- 172.21.15.28:0/1305499757 learned_addr learned my addr 172.21.15.28:0/1305499757 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.297+0000 7f1563400640 1 --2- 172.21.15.28:0/1305499757 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f15640743a0 0x7f156419edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.297+0000 7f1563e00640 1 --2- 172.21.15.28:0/1305499757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1564112eb0 0x7f15641a3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.297+0000 7f1562a00640 1 -- 172.21.15.28:0/1305499757 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f15640743a0 msgr2=0x7f156419edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.297+0000 7f1562a00640 1 --2- 172.21.15.28:0/1305499757 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f15640743a0 0x7f156419edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.297+0000 7f1562a00640 1 -- 172.21.15.28:0/1305499757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1564112eb0 msgr2=0x7f15641a3690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.297+0000 7f1562a00640 1 --2- 172.21.15.28:0/1305499757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1564112eb0 0x7f15641a3690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.297+0000 7f1562a00640 1 -- 172.21.15.28:0/1305499757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15641a3dc0 con 0x7f1564074cc0 2024-04-24T08:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.297+0000 7f1562a00640 1 --2- 172.21.15.28:0/1305499757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564074cc0 0x7f156419f310 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f155000ea60 tx=0x7f155000ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.298+0000 7f155be00640 1 -- 172.21.15.28:0/1305499757 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155000cde0 con 0x7f1564074cc0 2024-04-24T08:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.298+0000 7f155be00640 1 -- 172.21.15.28:0/1305499757 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15500035f0 con 0x7f1564074cc0 2024-04-24T08:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.298+0000 7f155be00640 1 -- 172.21.15.28:0/1305499757 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1550010670 con 0x7f1564074cc0 2024-04-24T08:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.298+0000 7f1568e00640 1 -- 172.21.15.28:0/1305499757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15641a40b0 con 0x7f1564074cc0 2024-04-24T08:11:34.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.298+0000 7f1568e00640 1 -- 172.21.15.28:0/1305499757 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15641ab940 con 0x7f1564074cc0 2024-04-24T08:11:34.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.302+0000 7f155be00640 1 -- 172.21.15.28:0/1305499757 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138930+0+0 (secure 0 0 0) 0x7f15500037a0 con 0x7f1564074cc0 2024-04-24T08:11:34.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.302+0000 7f156d20e740 1 Processor -- start 2024-04-24T08:11:34.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.303+0000 7f156d20e740 1 -- start start 2024-04-24T08:11:34.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.303+0000 7f156d20e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a92b40 0x55bde7a92f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.303+0000 7f156d20e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a93460 0x55bde7a9d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.303+0000 7f156d20e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a9db30 0x55bde7a9ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.303+0000 7f156d20e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bde79f81d0 con 0x55bde7a9db30 2024-04-24T08:11:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.303+0000 7f156d20e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bde799e450 con 0x55bde7a93460 2024-04-24T08:11:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.303+0000 7f156d20e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bde79f4190 con 0x55bde7a92b40 2024-04-24T08:11:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.304+0000 7f1562a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a9db30 0x55bde7a9ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.304+0000 7f1563400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a93460 0x55bde7a9d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.304+0000 7f1562a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a9db30 0x55bde7a9ff20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57346/0 (socket says 172.21.15.28:57346) 2024-04-24T08:11:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.304+0000 7f1562a00640 1 -- 172.21.15.28:0/3070760824 learned_addr learned my addr 172.21.15.28:0/3070760824 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.304+0000 7f1563400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a93460 0x55bde7a9d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49412/0 (socket says 172.21.15.28:49412) 2024-04-24T08:11:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.304+0000 7f1563e00640 1 --2- 172.21.15.28:0/3070760824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a92b40 0x55bde7a92f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.304+0000 7f155be00640 1 --2- 172.21.15.28:0/1305499757 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f15480a3f50 0x7f15480a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.304+0000 7f1563400640 1 -- 172.21.15.28:0/3070760824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a92b40 msgr2=0x55bde7a92f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.304+0000 7f1563400640 1 --2- 172.21.15.28:0/3070760824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a92b40 0x55bde7a92f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.304+0000 7f1563400640 1 -- 172.21.15.28:0/3070760824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a9db30 msgr2=0x55bde7a9ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.304+0000 7f1563400640 1 --2- 172.21.15.28:0/3070760824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a9db30 0x55bde7a9ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.304+0000 7f1563400640 1 -- 172.21.15.28:0/3070760824 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bde7a5b310 con 0x55bde7a93460 2024-04-24T08:11:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.304+0000 7f155be00640 1 -- 172.21.15.28:0/1305499757 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f15500d0e00 con 0x7f1564074cc0 2024-04-24T08:11:34.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.304+0000 7f1563e00640 1 --2- 172.21.15.28:0/1305499757 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f15480a3f50 0x7f15480a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.305+0000 7f1563400640 1 --2- 172.21.15.28:0/3070760824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a93460 0x55bde7a9d5f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f155402aa70 tx=0x7f1554005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=595af401df28f0d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.305+0000 7f1558c00640 1 -- 172.21.15.28:0/3070760824 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554035040 con 0x55bde7a93460 2024-04-24T08:11:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.305+0000 7f1558c00640 1 -- 172.21.15.28:0/3070760824 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1554004400 con 0x55bde7a93460 2024-04-24T08:11:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.305+0000 7f1558c00640 1 -- 172.21.15.28:0/3070760824 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15540035f0 con 0x55bde7a93460 2024-04-24T08:11:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.306+0000 7f156d20e740 1 -- 172.21.15.28:0/3070760824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a93460 msgr2=0x55bde7a9d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.306+0000 7f156d20e740 1 --2- 172.21.15.28:0/3070760824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a93460 0x55bde7a9d5f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f155402aa70 tx=0x7f1554005de0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.306+0000 7f1563e00640 1 --2- 172.21.15.28:0/1305499757 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f15480a3f50 0x7f15480a6410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f156406b560 tx=0x7f155c004a40 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.306+0000 7f156d20e740 1 -- 172.21.15.28:0/3070760824 shutdown_connections 2024-04-24T08:11:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.306+0000 7f156d20e740 1 --2- 172.21.15.28:0/3070760824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a9db30 0x55bde7a9ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.306+0000 7f156d20e740 1 --2- 172.21.15.28:0/3070760824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a93460 0x55bde7a9d5f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.306+0000 7f156d20e740 1 --2- 172.21.15.28:0/3070760824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a92b40 0x55bde7a92f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.306+0000 7f156d20e740 1 -- 172.21.15.28:0/3070760824 >> 172.21.15.28:0/3070760824 conn(0x55bde7a8ee60 msgr2=0x55bde7a8f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.306+0000 7f156d20e740 1 -- 172.21.15.28:0/3070760824 shutdown_connections 2024-04-24T08:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.306+0000 7f156d20e740 1 -- 172.21.15.28:0/3070760824 wait complete. 2024-04-24T08:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.307+0000 7f156d20e740 1 -- start start 2024-04-24T08:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.307+0000 7f156d20e740 1 Processor -- start 2024-04-24T08:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.308+0000 7f156d20e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a92b40 0x55bde7a9d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.308+0000 7f156d20e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a93460 0x55bde7a9a8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.308+0000 7f156d20e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a9db30 0x55bde7a9b010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.308+0000 7f156d20e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55bde7aa0960 con 0x55bde7a9db30 2024-04-24T08:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.308+0000 7f156d20e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55bde79b1810 con 0x55bde7a92b40 2024-04-24T08:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.308+0000 7f156d20e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55bde7aa0ae0 con 0x55bde7a93460 2024-04-24T08:11:34.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.308+0000 7f156d20e740 10 client.0 _ll_register_callbacks cb 0x55bde797f140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:34.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.308+0000 7f1563e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a9db30 0x55bde7a9b010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.308+0000 7f1563400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a92b40 0x55bde7a9d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.308+0000 7f1563400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a92b40 0x55bde7a9d5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49420/0 (socket says 172.21.15.28:49420) 2024-04-24T08:11:34.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.308+0000 7f1563400640 1 -- 172.21.15.28:0/3822487476 learned_addr learned my addr 172.21.15.28:0/3822487476 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:34.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.308+0000 7f1562a00640 1 --2- 172.21.15.28:0/3822487476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a93460 0x55bde7a9a8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.308+0000 7f1563400640 1 -- 172.21.15.28:0/3822487476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a93460 msgr2=0x55bde7a9a8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.308+0000 7f1563400640 1 --2- 172.21.15.28:0/3822487476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a93460 0x55bde7a9a8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.308+0000 7f1563400640 1 -- 172.21.15.28:0/3822487476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a9db30 msgr2=0x55bde7a9b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.308+0000 7f1563400640 1 --2- 172.21.15.28:0/3822487476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a9db30 0x55bde7a9b010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.308+0000 7f1563400640 1 -- 172.21.15.28:0/3822487476 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bde79c23f0 con 0x55bde7a92b40 2024-04-24T08:11:34.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.309+0000 7f1563400640 1 --2- 172.21.15.28:0/3822487476 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a92b40 0x55bde7a9d5e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1554031550 tx=0x7f15540048f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e82b97b76b59a36 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.309+0000 7f154d600640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:11:34.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.309+0000 7f154d600640 1 -- 172.21.15.28:0/3822487476 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554035040 con 0x55bde7a92b40 2024-04-24T08:11:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.309+0000 7f156d20e740 1 -- 172.21.15.28:0/3822487476 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1554004400 con 0x55bde7a92b40 2024-04-24T08:11:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.309+0000 7f154d600640 1 -- 172.21.15.28:0/3822487476 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f1554003800 con 0x55bde7a92b40 2024-04-24T08:11:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.310+0000 7f154d600640 1 -- 172.21.15.28:0/3822487476 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f155405a050 con 0x55bde7a92b40 2024-04-24T08:11:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.310+0000 7f156d20e740 10 client.25592 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.310+0000 7f156d20e740 1 -- 172.21.15.28:0/3822487476 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55bde770a3f0 con 0x55bde7a92b40 2024-04-24T08:11:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.310+0000 7f154d600640 1 -- 172.21.15.28:0/3822487476 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1554004400 con 0x55bde7a92b40 2024-04-24T08:11:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.310+0000 7f154d600640 1 -- 172.21.15.28:0/3822487476 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f1554055030 con 0x55bde7a92b40 2024-04-24T08:11:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.310+0000 7f156d20e740 10 client.25592 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.310+0000 7f156d20e740 10 client.25592 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:11:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.310+0000 7f156d20e740 20 client.25592 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.310+0000 7f156d20e740 1 --2- 172.21.15.28:0/3822487476 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55bde7aadb90 0x55bde7aaff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.310+0000 7f156d20e740 4 client.25592 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:11:34.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.310+0000 7f156d20e740 1 -- 172.21.15.28:0/3822487476 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bde7a05bf0 con 0x55bde7aadb90 2024-04-24T08:11:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.311+0000 7f1562a00640 1 --2- 172.21.15.28:0/3822487476 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55bde7aadb90 0x55bde7aaff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.311+0000 7f1562a00640 1 --2- 172.21.15.28:0/3822487476 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55bde7aadb90 0x55bde7aaff80 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b48f9f71b01f3a0 server_cookie=91ed2e2a8d317903 in_seq=0 out_seq=0 2024-04-24T08:11:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.311+0000 7f154d600640 10 client.25592 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:11:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.312+0000 7f154d600640 1 -- 172.21.15.28:0/3822487476 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bde7a05bf0 con 0x55bde7aadb90 2024-04-24T08:11:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.312+0000 7f154d600640 10 client.25592 handle_command_reply: tid=0 2024-04-24T08:11:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.312+0000 7f156d20e740 1 client.25592 shutdown 2024-04-24T08:11:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.312+0000 7f156d20e740 1 -- 172.21.15.28:0/3822487476 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a92b40 msgr2=0x55bde7a9d5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.312+0000 7f156d20e740 1 --2- 172.21.15.28:0/3822487476 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a92b40 0x55bde7a9d5e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f1554031550 tx=0x7f15540048f0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.312+0000 7f156d20e740 1 -- 172.21.15.28:0/3822487476 shutdown_connections 2024-04-24T08:11:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.312+0000 7f156d20e740 1 --2- 172.21.15.28:0/3822487476 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55bde7aadb90 0x55bde7aaff80 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.312+0000 7f156d20e740 1 --2- 172.21.15.28:0/3822487476 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a9db30 0x55bde7a9b010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.312+0000 7f156d20e740 1 --2- 172.21.15.28:0/3822487476 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a93460 0x55bde7a9a8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.312+0000 7f156d20e740 1 --2- 172.21.15.28:0/3822487476 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a92b40 0x55bde7a9d5e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.312+0000 7f154d600640 0 client.25592 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:11:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.312+0000 7f156d20e740 1 -- 172.21.15.28:0/3822487476 >> 172.21.15.28:0/3822487476 conn(0x55bde7a8ee60 msgr2=0x55bde7a94c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.313+0000 7f156d20e740 1 -- 172.21.15.28:0/3822487476 shutdown_connections 2024-04-24T08:11:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.313+0000 7f156d20e740 1 -- 172.21.15.28:0/3822487476 wait complete. 2024-04-24T08:11:34.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.313+0000 7f156d20e740 20 client.25592 trim_cache size 0 max 16384 2024-04-24T08:11:34.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.328+0000 7f156d20e740 1 Processor -- start 2024-04-24T08:11:34.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.328+0000 7f156d20e740 1 -- start start 2024-04-24T08:11:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.328+0000 7f156d20e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a92b40 0x55bde7aa9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.328+0000 7f156d20e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a93460 0x55bde7a99780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.328+0000 7f156d20e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a9db30 0x55bde7a9d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.328+0000 7f156d20e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15641b6b10 con 0x55bde7a92b40 2024-04-24T08:11:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.328+0000 7f156d20e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f15641b43e0 con 0x55bde7a9db30 2024-04-24T08:11:34.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.328+0000 7f156d20e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f15641b2230 con 0x55bde7a93460 2024-04-24T08:11:34.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.328+0000 7f1563400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a92b40 0x55bde7aa9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.328+0000 7f1562a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a93460 0x55bde7a99780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.328+0000 7f1563400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a92b40 0x55bde7aa9820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57366/0 (socket says 172.21.15.28:57366) 2024-04-24T08:11:34.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.328+0000 7f1563400640 1 -- 172.21.15.28:0/3521072044 learned_addr learned my addr 172.21.15.28:0/3521072044 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:34.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.328+0000 7f1562a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a93460 0x55bde7a99780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60074/0 (socket says 172.21.15.28:60074) 2024-04-24T08:11:34.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.329+0000 7f1563e00640 1 --2- 172.21.15.28:0/3521072044 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a9db30 0x55bde7a9d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.329+0000 7f1562a00640 1 -- 172.21.15.28:0/3521072044 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a9db30 msgr2=0x55bde7a9d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.329+0000 7f1562a00640 1 --2- 172.21.15.28:0/3521072044 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a9db30 0x55bde7a9d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.329+0000 7f1562a00640 1 -- 172.21.15.28:0/3521072044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a92b40 msgr2=0x55bde7aa9820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.329+0000 7f1562a00640 1 --2- 172.21.15.28:0/3521072044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a92b40 0x55bde7aa9820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.329+0000 7f1562a00640 1 -- 172.21.15.28:0/3521072044 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bde7a93290 con 0x55bde7a93460 2024-04-24T08:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.329+0000 7f1563400640 1 --2- 172.21.15.28:0/3521072044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a92b40 0x55bde7aa9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:34.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.329+0000 7f1562a00640 1 --2- 172.21.15.28:0/3521072044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a93460 0x55bde7a99780 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x55bde7a8fff0 tx=0x7f15500d3290 comp rx=0 tx=0).ready entity=mon.1 client_cookie=26259dbec698befe server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.329+0000 7f154fe00640 1 -- 172.21.15.28:0/3521072044 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155009c680 con 0x55bde7a93460 2024-04-24T08:11:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.329+0000 7f154fe00640 1 -- 172.21.15.28:0/3521072044 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f155009d240 con 0x55bde7a93460 2024-04-24T08:11:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.329+0000 7f154fe00640 1 -- 172.21.15.28:0/3521072044 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f155009c4a0 con 0x55bde7a93460 2024-04-24T08:11:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.330+0000 7f156d20e740 1 -- 172.21.15.28:0/3521072044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a93460 msgr2=0x55bde7a99780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.330+0000 7f156d20e740 1 --2- 172.21.15.28:0/3521072044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a93460 0x55bde7a99780 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x55bde7a8fff0 tx=0x7f15500d3290 comp rx=0 tx=0).stop 2024-04-24T08:11:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.330+0000 7f156d20e740 1 -- 172.21.15.28:0/3521072044 shutdown_connections 2024-04-24T08:11:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.330+0000 7f156d20e740 1 --2- 172.21.15.28:0/3521072044 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a9db30 0x55bde7a9d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.330+0000 7f156d20e740 1 --2- 172.21.15.28:0/3521072044 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a93460 0x55bde7a99780 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.330+0000 7f156d20e740 1 --2- 172.21.15.28:0/3521072044 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a92b40 0x55bde7aa9820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.330+0000 7f156d20e740 1 -- 172.21.15.28:0/3521072044 >> 172.21.15.28:0/3521072044 conn(0x55bde7a8ee60 msgr2=0x55bde7a99070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.330+0000 7f156d20e740 1 -- 172.21.15.28:0/3521072044 shutdown_connections 2024-04-24T08:11:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.330+0000 7f156d20e740 1 -- 172.21.15.28:0/3521072044 wait complete. 2024-04-24T08:11:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.331+0000 7f156d20e740 1 -- start start 2024-04-24T08:11:34.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.331+0000 7f156d20e740 1 Processor -- start 2024-04-24T08:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.331+0000 7f156d20e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a92b40 0x55bde7aa6000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.331+0000 7f156d20e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a93460 0x55bde7a97b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.331+0000 7f156d20e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a9db30 0x55bde7a98090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.332+0000 7f156d20e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f15641b28a0 con 0x55bde7a92b40 2024-04-24T08:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.332+0000 7f156d20e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f15641b3d10 con 0x55bde7a93460 2024-04-24T08:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.332+0000 7f156d20e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f15641b2f70 con 0x55bde7a9db30 2024-04-24T08:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.332+0000 7f156d20e740 10 client.0 _ll_register_callbacks cb 0x7f15500d47a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.332+0000 7f1563400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a92b40 0x55bde7aa6000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.332+0000 7f1563400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a92b40 0x55bde7aa6000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57368/0 (socket says 172.21.15.28:57368) 2024-04-24T08:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.332+0000 7f1563400640 1 -- 172.21.15.28:0/2028221819 learned_addr learned my addr 172.21.15.28:0/2028221819 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.332+0000 7f1563e00640 1 --2- 172.21.15.28:0/2028221819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a9db30 0x55bde7a98090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.332+0000 7f1562a00640 1 --2- 172.21.15.28:0/2028221819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a93460 0x55bde7a97b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.332+0000 7f1563400640 1 -- 172.21.15.28:0/2028221819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a9db30 msgr2=0x55bde7a98090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.332+0000 7f1563400640 1 --2- 172.21.15.28:0/2028221819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a9db30 0x55bde7a98090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.332+0000 7f1563400640 1 -- 172.21.15.28:0/2028221819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a93460 msgr2=0x55bde7a97b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.332+0000 7f1563400640 1 --2- 172.21.15.28:0/2028221819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a93460 0x55bde7a97b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.332+0000 7f1563400640 1 -- 172.21.15.28:0/2028221819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bde77d47d0 con 0x55bde7a92b40 2024-04-24T08:11:34.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.333+0000 7f1563400640 1 --2- 172.21.15.28:0/2028221819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a92b40 0x55bde7aa6000 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1554031cf0 tx=0x7f1554009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a997d977ba393b3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.333+0000 7f154d600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:11:34.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.333+0000 7f154d600640 1 -- 172.21.15.28:0/2028221819 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1554035070 con 0x55bde7a92b40 2024-04-24T08:11:34.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.333+0000 7f154d600640 1 -- 172.21.15.28:0/2028221819 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f1554004400 con 0x55bde7a92b40 2024-04-24T08:11:34.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.333+0000 7f156d20e740 1 -- 172.21.15.28:0/2028221819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f155009d240 con 0x55bde7a92b40 2024-04-24T08:11:34.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.333+0000 7f154d600640 1 -- 172.21.15.28:0/2028221819 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f155406c050 con 0x55bde7a92b40 2024-04-24T08:11:34.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.333+0000 7f156d20e740 10 client.15498 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:34.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.333+0000 7f156d20e740 1 -- 172.21.15.28:0/2028221819 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55bde7a9f2d0 con 0x55bde7a92b40 2024-04-24T08:11:34.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.334+0000 7f154d600640 1 -- 172.21.15.28:0/2028221819 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f155009d240 con 0x55bde7a92b40 2024-04-24T08:11:34.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.334+0000 7f154d600640 1 -- 172.21.15.28:0/2028221819 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f1554055040 con 0x55bde7a92b40 2024-04-24T08:11:34.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.334+0000 7f156d20e740 10 client.15498 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:34.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.334+0000 7f156d20e740 10 client.15498 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:11:34.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.334+0000 7f156d20e740 20 client.15498 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:34.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.334+0000 7f156d20e740 1 --2- 172.21.15.28:0/2028221819 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55bde7b142d0 0x55bde7b166f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.334+0000 7f156d20e740 4 client.15498 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:11:34.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.334+0000 7f156d20e740 1 -- 172.21.15.28:0/2028221819 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55bde79ffa60 con 0x55bde7b142d0 2024-04-24T08:11:34.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.334+0000 7f1562a00640 1 --2- 172.21.15.28:0/2028221819 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55bde7b142d0 0x55bde7b166f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.335+0000 7f1562a00640 1 --2- 172.21.15.28:0/2028221819 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55bde7b142d0 0x55bde7b166f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c28f960eb74e23c0 server_cookie=fe3855bfd02de067 in_seq=0 out_seq=0 2024-04-24T08:11:34.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.335+0000 7f154d600640 10 client.15498 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:11:34.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.335+0000 7f154d600640 1 -- 172.21.15.28:0/2028221819 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55bde79ffa60 con 0x55bde7b142d0 2024-04-24T08:11:34.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.335+0000 7f154d600640 10 client.15498 handle_command_reply: tid=0 2024-04-24T08:11:34.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.335+0000 7f156d20e740 1 client.15498 shutdown 2024-04-24T08:11:34.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.335+0000 7f156d20e740 1 -- 172.21.15.28:0/2028221819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a92b40 msgr2=0x55bde7aa6000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.335+0000 7f156d20e740 1 --2- 172.21.15.28:0/2028221819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a92b40 0x55bde7aa6000 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f1554031cf0 tx=0x7f1554009520 comp rx=0 tx=0).stop 2024-04-24T08:11:34.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.335+0000 7f156d20e740 1 -- 172.21.15.28:0/2028221819 shutdown_connections 2024-04-24T08:11:34.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.335+0000 7f156d20e740 1 --2- 172.21.15.28:0/2028221819 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55bde7b142d0 0x55bde7b166f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.335+0000 7f156d20e740 1 --2- 172.21.15.28:0/2028221819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55bde7a9db30 0x55bde7a98090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.335+0000 7f156d20e740 1 --2- 172.21.15.28:0/2028221819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55bde7a93460 0x55bde7a97b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.335+0000 7f154d600640 0 client.15498 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:11:34.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.335+0000 7f156d20e740 1 --2- 172.21.15.28:0/2028221819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55bde7a92b40 0x55bde7aa6000 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.335+0000 7f156d20e740 1 -- 172.21.15.28:0/2028221819 >> 172.21.15.28:0/2028221819 conn(0x55bde7a8ee60 msgr2=0x55bde7a8df50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.336+0000 7f156d20e740 1 -- 172.21.15.28:0/2028221819 shutdown_connections 2024-04-24T08:11:34.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.336+0000 7f156d20e740 1 -- 172.21.15.28:0/2028221819 wait complete. 2024-04-24T08:11:34.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.336+0000 7f156d20e740 20 client.15498 trim_cache size 0 max 16384 2024-04-24T08:11:34.341 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:11:34.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.336+0000 7f1568e00640 1 -- 172.21.15.28:0/1305499757 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f15480a3f50 msgr2=0x7f15480a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.336+0000 7f1568e00640 1 --2- 172.21.15.28:0/1305499757 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f15480a3f50 0x7f15480a6410 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f156406b560 tx=0x7f155c004a40 comp rx=0 tx=0).stop 2024-04-24T08:11:34.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.336+0000 7f1568e00640 1 -- 172.21.15.28:0/1305499757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564074cc0 msgr2=0x7f156419f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.336+0000 7f1568e00640 1 --2- 172.21.15.28:0/1305499757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564074cc0 0x7f156419f310 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f155000ea60 tx=0x7f155000ef20 comp rx=0 tx=0).stop 2024-04-24T08:11:34.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.337+0000 7f1568e00640 1 -- 172.21.15.28:0/1305499757 shutdown_connections 2024-04-24T08:11:34.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.337+0000 7f1568e00640 1 --2- 172.21.15.28:0/1305499757 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f15480a3f50 0x7f15480a6410 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.337+0000 7f1568e00640 1 --2- 172.21.15.28:0/1305499757 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1564074cc0 0x7f156419f310 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.337+0000 7f1568e00640 1 --2- 172.21.15.28:0/1305499757 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1564112eb0 0x7f15641a3690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.337+0000 7f1568e00640 1 --2- 172.21.15.28:0/1305499757 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f15640743a0 0x7f156419edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.337+0000 7f1568e00640 1 -- 172.21.15.28:0/1305499757 >> 172.21.15.28:0/1305499757 conn(0x7f15640fd9d0 msgr2=0x7f15640ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.337+0000 7f1568e00640 1 -- 172.21.15.28:0/1305499757 shutdown_connections 2024-04-24T08:11:34.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.337+0000 7f1568e00640 1 -- 172.21.15.28:0/1305499757 wait complete. 2024-04-24T08:11:34.352 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:34 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2606626993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:11:34.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:11:34.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.560+0000 7fb0a0a00640 1 -- 172.21.15.28:0/2040980804 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb09c1060e0 msgr2=0x7fb09c10dcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.560+0000 7fb0a0a00640 1 --2- 172.21.15.28:0/2040980804 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb09c1060e0 0x7fb09c10dcb0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fb094009860 tx=0x7fb09402a340 comp rx=0 tx=0).stop 2024-04-24T08:11:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.560+0000 7fb0a0a00640 1 -- 172.21.15.28:0/2040980804 shutdown_connections 2024-04-24T08:11:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.560+0000 7fb0a0a00640 1 --2- 172.21.15.28:0/2040980804 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb09c1019c0 0x7fb09c105a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.560+0000 7fb0a0a00640 1 --2- 172.21.15.28:0/2040980804 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb09c1060e0 0x7fb09c10dcb0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.560+0000 7fb0a0a00640 1 --2- 172.21.15.28:0/2040980804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb09c101010 0x7fb09c1013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.560+0000 7fb0a0a00640 1 -- 172.21.15.28:0/2040980804 >> 172.21.15.28:0/2040980804 conn(0x7fb09c0fc850 msgr2=0x7fb09c0fec70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.560+0000 7fb0a0a00640 1 -- 172.21.15.28:0/2040980804 shutdown_connections 2024-04-24T08:11:34.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.560+0000 7fb0a0a00640 1 -- 172.21.15.28:0/2040980804 wait complete. 2024-04-24T08:11:34.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.561+0000 7fb0a0a00640 1 Processor -- start 2024-04-24T08:11:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.561+0000 7fb0a0a00640 1 -- start start 2024-04-24T08:11:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.562+0000 7fb0a0a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb09c101010 0x7fb09c1a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.562+0000 7fb0a0a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb09c1060e0 0x7fb09c1a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.562+0000 7fb0a0a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb09c19a760 0x7fb09c19abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.562+0000 7fb0a0a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c111180 con 0x7fb09c101010 2024-04-24T08:11:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.562+0000 7fb0a0a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c111000 con 0x7fb09c1060e0 2024-04-24T08:11:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.562+0000 7fb0a0a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c111300 con 0x7fb09c19a760 2024-04-24T08:11:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.562+0000 7fb09b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb09c101010 0x7fb09c1a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.562+0000 7fb09b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb09c101010 0x7fb09c1a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57392/0 (socket says 172.21.15.28:57392) 2024-04-24T08:11:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.562+0000 7fb09b400640 1 -- 172.21.15.28:0/4209167395 learned_addr learned my addr 172.21.15.28:0/4209167395 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.562+0000 7fb092a00640 1 --2- 172.21.15.28:0/4209167395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb09c1060e0 0x7fb09c1a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.562+0000 7fb09be00640 1 --2- 172.21.15.28:0/4209167395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb09c19a760 0x7fb09c19abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.562+0000 7fb09b400640 1 -- 172.21.15.28:0/4209167395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb09c19a760 msgr2=0x7fb09c19abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.562+0000 7fb09b400640 1 --2- 172.21.15.28:0/4209167395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb09c19a760 0x7fb09c19abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.562+0000 7fb09b400640 1 -- 172.21.15.28:0/4209167395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb09c1060e0 msgr2=0x7fb09c1a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.562+0000 7fb09b400640 1 --2- 172.21.15.28:0/4209167395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb09c1060e0 0x7fb09c1a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.563+0000 7fb09b400640 1 -- 172.21.15.28:0/4209167395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb09c19b360 con 0x7fb09c101010 2024-04-24T08:11:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.563+0000 7fb092a00640 1 --2- 172.21.15.28:0/4209167395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb09c1060e0 0x7fb09c1a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.563+0000 7fb09b400640 1 --2- 172.21.15.28:0/4209167395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb09c101010 0x7fb09c1a06a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb08c00c420 tx=0x7fb08c009f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.563+0000 7fb098c00640 1 -- 172.21.15.28:0/4209167395 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c003970 con 0x7fb09c101010 2024-04-24T08:11:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.563+0000 7fb098c00640 1 -- 172.21.15.28:0/4209167395 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb08c004020 con 0x7fb09c101010 2024-04-24T08:11:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.563+0000 7fb0a0a00640 1 -- 172.21.15.28:0/4209167395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb09c1a7440 con 0x7fb09c101010 2024-04-24T08:11:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.563+0000 7fb098c00640 1 -- 172.21.15.28:0/4209167395 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c00bad0 con 0x7fb09c101010 2024-04-24T08:11:34.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.563+0000 7fb0a0a00640 1 -- 172.21.15.28:0/4209167395 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb09c1a7840 con 0x7fb09c101010 2024-04-24T08:11:34.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.567+0000 7fb098c00640 1 -- 172.21.15.28:0/4209167395 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138930+0+0 (secure 0 0 0) 0x7fb08c01d020 con 0x7fb09c101010 2024-04-24T08:11:34.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.567+0000 7fb0a4dbf740 1 Processor -- start 2024-04-24T08:11:34.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.567+0000 7fb0a4dbf740 1 -- start start 2024-04-24T08:11:34.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.567+0000 7fb0a4dbf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be9609b40 0x563be9609f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.567+0000 7fb0a4dbf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be960a460 0x563be96145f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.567+0000 7fb0a4dbf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9614b30 0x563be9616f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.567+0000 7fb0a4dbf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563be956f1d0 con 0x563be960a460 2024-04-24T08:11:34.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.567+0000 7fb0a4dbf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563be9515450 con 0x563be9609b40 2024-04-24T08:11:34.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.567+0000 7fb0a4dbf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563be956b190 con 0x563be9614b30 2024-04-24T08:11:34.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.568+0000 7fb09b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be960a460 0x563be96145f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.568+0000 7fb092a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9614b30 0x563be9616f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.568+0000 7fb09b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be960a460 0x563be96145f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57406/0 (socket says 172.21.15.28:57406) 2024-04-24T08:11:34.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.568+0000 7fb092a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9614b30 0x563be9616f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60124/0 (socket says 172.21.15.28:60124) 2024-04-24T08:11:34.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.568+0000 7fb09b400640 1 -- 172.21.15.28:0/1028816903 learned_addr learned my addr 172.21.15.28:0/1028816903 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:34.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.568+0000 7fb09be00640 1 --2- 172.21.15.28:0/1028816903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be9609b40 0x563be9609f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.568+0000 7fb092a00640 1 -- 172.21.15.28:0/1028816903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be9609b40 msgr2=0x563be9609f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.568+0000 7fb092a00640 1 --2- 172.21.15.28:0/1028816903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be9609b40 0x563be9609f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.568+0000 7fb092a00640 1 -- 172.21.15.28:0/1028816903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be960a460 msgr2=0x563be96145f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.568+0000 7fb092a00640 1 --2- 172.21.15.28:0/1028816903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be960a460 0x563be96145f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.568+0000 7fb092a00640 1 -- 172.21.15.28:0/1028816903 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563be95d2310 con 0x563be9614b30 2024-04-24T08:11:34.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.568+0000 7fb09b400640 1 --2- 172.21.15.28:0/1028816903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be960a460 0x563be96145f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:34.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.569+0000 7fb098c00640 1 --2- 172.21.15.28:0/4209167395 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb0800a3ed0 0x7fb0800a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.569+0000 7fb092a00640 1 --2- 172.21.15.28:0/1028816903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9614b30 0x563be9616f20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb09c06b6a0 tx=0x7fb084011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=44fc70151e54726e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.569+0000 7fb098c00640 1 -- 172.21.15.28:0/4209167395 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fb08c0d0960 con 0x7fb09c101010 2024-04-24T08:11:34.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.569+0000 7fb09be00640 1 --2- 172.21.15.28:0/4209167395 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb0800a3ed0 0x7fb0800a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.569+0000 7fb08be00640 1 -- 172.21.15.28:0/1028816903 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb084004230 con 0x563be9614b30 2024-04-24T08:11:34.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.569+0000 7fb08be00640 1 -- 172.21.15.28:0/1028816903 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0840043d0 con 0x563be9614b30 2024-04-24T08:11:34.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.569+0000 7fb08be00640 1 -- 172.21.15.28:0/1028816903 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0840035f0 con 0x563be9614b30 2024-04-24T08:11:34.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.570+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/1028816903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9614b30 msgr2=0x563be9616f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.570+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/1028816903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9614b30 0x563be9616f20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fb09c06b6a0 tx=0x7fb084011810 comp rx=0 tx=0).stop 2024-04-24T08:11:34.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.570+0000 7fb09be00640 1 --2- 172.21.15.28:0/4209167395 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb0800a3ed0 0x7fb0800a6390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb0940057c0 tx=0x7fb0940056d0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.570+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/1028816903 shutdown_connections 2024-04-24T08:11:34.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.570+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/1028816903 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be960a460 0x563be96145f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.570+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/1028816903 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9614b30 0x563be9616f20 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.570+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/1028816903 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be9609b40 0x563be9609f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.570+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/1028816903 >> 172.21.15.28:0/1028816903 conn(0x563be9605e60 msgr2=0x563be9606240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.570+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/1028816903 shutdown_connections 2024-04-24T08:11:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.570+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/1028816903 wait complete. 2024-04-24T08:11:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.571+0000 7fb0a4dbf740 1 -- start start 2024-04-24T08:11:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.571+0000 7fb0a4dbf740 1 Processor -- start 2024-04-24T08:11:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.571+0000 7fb0a4dbf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be9609b40 0x563be9614480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.571+0000 7fb0a4dbf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be960a460 0x563be9611770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.571+0000 7fb0a4dbf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be9614b30 0x563be9611cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.571+0000 7fb0a4dbf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563be96179c0 con 0x563be9609b40 2024-04-24T08:11:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.571+0000 7fb0a4dbf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563be9528810 con 0x563be9614b30 2024-04-24T08:11:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.571+0000 7fb0a4dbf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563be9617b40 con 0x563be960a460 2024-04-24T08:11:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.572+0000 7fb09b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be9609b40 0x563be9614480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.572+0000 7fb0a4dbf740 10 client.0 _ll_register_callbacks cb 0x563be94f6140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:34.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.572+0000 7fb09b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be9609b40 0x563be9614480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57418/0 (socket says 172.21.15.28:57418) 2024-04-24T08:11:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.572+0000 7fb09b400640 1 -- 172.21.15.28:0/2013006749 learned_addr learned my addr 172.21.15.28:0/2013006749 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.572+0000 7fb092a00640 1 --2- 172.21.15.28:0/2013006749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be960a460 0x563be9611770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.572+0000 7fb09be00640 1 --2- 172.21.15.28:0/2013006749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be9614b30 0x563be9611cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.572+0000 7fb09b400640 1 -- 172.21.15.28:0/2013006749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be960a460 msgr2=0x563be9611770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.572+0000 7fb09b400640 1 --2- 172.21.15.28:0/2013006749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be960a460 0x563be9611770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.572+0000 7fb09b400640 1 -- 172.21.15.28:0/2013006749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be9614b30 msgr2=0x563be9611cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.572+0000 7fb09b400640 1 --2- 172.21.15.28:0/2013006749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be9614b30 0x563be9611cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.572+0000 7fb09b400640 1 -- 172.21.15.28:0/2013006749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563be95393f0 con 0x563be9609b40 2024-04-24T08:11:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.572+0000 7fb09b400640 1 --2- 172.21.15.28:0/2013006749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be9609b40 0x563be9614480 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb08c000f80 tx=0x7fb08c009900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=841a118e06871018 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.572+0000 7fb088c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:11:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.573+0000 7fb088c00640 1 -- 172.21.15.28:0/2013006749 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb08c09c470 con 0x563be9609b40 2024-04-24T08:11:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.573+0000 7fb088c00640 1 -- 172.21.15.28:0/2013006749 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fb08c085de0 con 0x563be9609b40 2024-04-24T08:11:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.573+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/2013006749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb0840043d0 con 0x563be9609b40 2024-04-24T08:11:34.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.573+0000 7fb088c00640 1 -- 172.21.15.28:0/2013006749 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb08c0041d0 con 0x563be9609b40 2024-04-24T08:11:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.573+0000 7fb0a4dbf740 10 client.15510 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.573+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/2013006749 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563be92813f0 con 0x563be9609b40 2024-04-24T08:11:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.573+0000 7fb088c00640 1 -- 172.21.15.28:0/2013006749 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0840043d0 con 0x563be9609b40 2024-04-24T08:11:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.573+0000 7fb088c00640 1 -- 172.21.15.28:0/2013006749 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fb08c00b040 con 0x563be9609b40 2024-04-24T08:11:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.573+0000 7fb0a4dbf740 10 client.15510 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.573+0000 7fb0a4dbf740 10 client.15510 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:11:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.574+0000 7fb0a4dbf740 20 client.15510 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.574+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/2013006749 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x563be9624ce0 0x563be9627110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.574+0000 7fb0a4dbf740 4 client.15510 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:11:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.574+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/2013006749 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563be957cbf0 con 0x563be9624ce0 2024-04-24T08:11:34.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.574+0000 7fb092a00640 1 --2- 172.21.15.28:0/2013006749 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x563be9624ce0 0x563be9627110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.575+0000 7fb092a00640 1 --2- 172.21.15.28:0/2013006749 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x563be9624ce0 0x563be9627110 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=49dbc99ed1b5b5ac server_cookie=2b8b6654ebaaca3 in_seq=0 out_seq=0 2024-04-24T08:11:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.575+0000 7fb088c00640 10 client.15510 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:11:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.575+0000 7fb088c00640 1 -- 172.21.15.28:0/2013006749 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563be957cbf0 con 0x563be9624ce0 2024-04-24T08:11:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.575+0000 7fb088c00640 10 client.15510 handle_command_reply: tid=0 2024-04-24T08:11:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.575+0000 7fb0a4dbf740 1 client.15510 shutdown 2024-04-24T08:11:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.576+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/2013006749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be9609b40 msgr2=0x563be9614480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.576+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/2013006749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be9609b40 0x563be9614480 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fb08c000f80 tx=0x7fb08c009900 comp rx=0 tx=0).stop 2024-04-24T08:11:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.576+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/2013006749 shutdown_connections 2024-04-24T08:11:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.576+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/2013006749 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x563be9624ce0 0x563be9627110 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.576+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/2013006749 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be9614b30 0x563be9611cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.576+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/2013006749 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be960a460 0x563be9611770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.576+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/2013006749 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be9609b40 0x563be9614480 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.576+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/2013006749 >> 172.21.15.28:0/2013006749 conn(0x563be9605e60 msgr2=0x563be960bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.576+0000 7fb088c00640 0 client.15510 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:11:34.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.576+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/2013006749 shutdown_connections 2024-04-24T08:11:34.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.576+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/2013006749 wait complete. 2024-04-24T08:11:34.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.576+0000 7fb0a4dbf740 20 client.15510 trim_cache size 0 max 16384 2024-04-24T08:11:34.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.605+0000 7fb0a4dbf740 1 Processor -- start 2024-04-24T08:11:34.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.605+0000 7fb0a4dbf740 1 -- start start 2024-04-24T08:11:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.605+0000 7fb0a4dbf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9609b40 0x563be9604dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.605+0000 7fb0a4dbf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be960a460 0x563be9613fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.605+0000 7fb0a4dbf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be9614b30 0x563be962c9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.606+0000 7fb0a4dbf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c1b2730 con 0x563be9614b30 2024-04-24T08:11:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.606+0000 7fb0a4dbf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c1b0000 con 0x563be960a460 2024-04-24T08:11:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.606+0000 7fb0a4dbf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c1ade50 con 0x563be9609b40 2024-04-24T08:11:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.606+0000 7fb09be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be9614b30 0x563be962c9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.606+0000 7fb092a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be960a460 0x563be9613fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.606+0000 7fb09be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be9614b30 0x563be962c9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57426/0 (socket says 172.21.15.28:57426) 2024-04-24T08:11:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.606+0000 7fb09be00640 1 -- 172.21.15.28:0/1390948216 learned_addr learned my addr 172.21.15.28:0/1390948216 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.606+0000 7fb09b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9609b40 0x563be9604dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.606+0000 7fb092a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be960a460 0x563be9613fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49482/0 (socket says 172.21.15.28:49482) 2024-04-24T08:11:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.606+0000 7fb092a00640 1 -- 172.21.15.28:0/1390948216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9609b40 msgr2=0x563be9604dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.606+0000 7fb092a00640 1 --2- 172.21.15.28:0/1390948216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9609b40 0x563be9604dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.606+0000 7fb092a00640 1 -- 172.21.15.28:0/1390948216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be9614b30 msgr2=0x563be962c9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.606+0000 7fb092a00640 1 --2- 172.21.15.28:0/1390948216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be9614b30 0x563be962c9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.606+0000 7fb092a00640 1 -- 172.21.15.28:0/1390948216 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563be934b7d0 con 0x563be960a460 2024-04-24T08:11:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.606+0000 7fb09be00640 1 --2- 172.21.15.28:0/1390948216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be9614b30 0x563be962c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.606+0000 7fb09b400640 1 --2- 172.21.15.28:0/1390948216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9609b40 0x563be9604dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:34.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.607+0000 7fb092a00640 1 --2- 172.21.15.28:0/1390948216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be960a460 0x563be9613fb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb084000c00 tx=0x7fb084004a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbdf062a6ea35c47 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.607+0000 7fb08b400640 1 -- 172.21.15.28:0/1390948216 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb084004020 con 0x563be960a460 2024-04-24T08:11:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.607+0000 7fb08b400640 1 -- 172.21.15.28:0/1390948216 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb08401f040 con 0x563be960a460 2024-04-24T08:11:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.607+0000 7fb08b400640 1 -- 172.21.15.28:0/1390948216 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb084003df0 con 0x563be960a460 2024-04-24T08:11:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.608+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/1390948216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be960a460 msgr2=0x563be9613fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.608+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/1390948216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be960a460 0x563be9613fb0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fb084000c00 tx=0x7fb084004a20 comp rx=0 tx=0).stop 2024-04-24T08:11:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.608+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/1390948216 shutdown_connections 2024-04-24T08:11:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.608+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/1390948216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be9614b30 0x563be962c9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.608+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/1390948216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be960a460 0x563be9613fb0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.608+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/1390948216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9609b40 0x563be9604dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.608+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/1390948216 >> 172.21.15.28:0/1390948216 conn(0x563be9605e60 msgr2=0x563be96046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.608+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/1390948216 shutdown_connections 2024-04-24T08:11:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.608+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/1390948216 wait complete. 2024-04-24T08:11:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.609+0000 7fb0a4dbf740 1 -- start start 2024-04-24T08:11:34.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.609+0000 7fb0a4dbf740 1 Processor -- start 2024-04-24T08:11:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.610+0000 7fb0a4dbf740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9609b40 0x563be961d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.610+0000 7fb0a4dbf740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be960a460 0x563be960eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.610+0000 7fb0a4dbf740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be9614b30 0x563be960f230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.610+0000 7fb0a4dbf740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c1ae4c0 con 0x563be960a460 2024-04-24T08:11:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.610+0000 7fb0a4dbf740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c1af930 con 0x563be9614b30 2024-04-24T08:11:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.610+0000 7fb0a4dbf740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb09c1aeb90 con 0x563be9609b40 2024-04-24T08:11:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.610+0000 7fb0a4dbf740 10 client.0 _ll_register_callbacks cb 0x7fb08c00c370 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.610+0000 7fb092a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be960a460 0x563be960eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.610+0000 7fb092a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be960a460 0x563be960eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57430/0 (socket says 172.21.15.28:57430) 2024-04-24T08:11:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.610+0000 7fb092a00640 1 -- 172.21.15.28:0/850566801 learned_addr learned my addr 172.21.15.28:0/850566801 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.610+0000 7fb09be00640 1 --2- 172.21.15.28:0/850566801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be9614b30 0x563be960f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.610+0000 7fb09b400640 1 --2- 172.21.15.28:0/850566801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9609b40 0x563be961d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.610+0000 7fb092a00640 1 -- 172.21.15.28:0/850566801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9609b40 msgr2=0x563be961d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.610+0000 7fb092a00640 1 --2- 172.21.15.28:0/850566801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9609b40 0x563be961d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.610+0000 7fb092a00640 1 -- 172.21.15.28:0/850566801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be9614b30 msgr2=0x563be960f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.610+0000 7fb092a00640 1 --2- 172.21.15.28:0/850566801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be9614b30 0x563be960f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.610+0000 7fb092a00640 1 -- 172.21.15.28:0/850566801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563be96156d0 con 0x563be960a460 2024-04-24T08:11:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.611+0000 7fb092a00640 1 --2- 172.21.15.28:0/850566801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be960a460 0x563be960eaf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb084009bf0 tx=0x7fb084011dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc7fbcfd1aa89ea server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.611+0000 7fb088c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:11:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.611+0000 7fb088c00640 1 -- 172.21.15.28:0/850566801 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb084033070 con 0x563be960a460 2024-04-24T08:11:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.611+0000 7fb088c00640 1 -- 172.21.15.28:0/850566801 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fb08400f740 con 0x563be960a460 2024-04-24T08:11:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.611+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/850566801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb08401f040 con 0x563be960a460 2024-04-24T08:11:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.611+0000 7fb088c00640 1 -- 172.21.15.28:0/850566801 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb08403f050 con 0x563be960a460 2024-04-24T08:11:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.611+0000 7fb0a4dbf740 10 client.15516 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.611+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/850566801 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x563be968b430 con 0x563be960a460 2024-04-24T08:11:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.612+0000 7fb088c00640 1 -- 172.21.15.28:0/850566801 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb08401f040 con 0x563be960a460 2024-04-24T08:11:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.612+0000 7fb088c00640 1 -- 172.21.15.28:0/850566801 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fb084024040 con 0x563be960a460 2024-04-24T08:11:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.612+0000 7fb0a4dbf740 10 client.15516 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.612+0000 7fb0a4dbf740 10 client.15516 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:11:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.612+0000 7fb0a4dbf740 20 client.15516 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.612+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/850566801 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x563be968b8e0 0x563be968dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.612+0000 7fb0a4dbf740 4 client.15516 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:11:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.612+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/850566801 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563be9576a60 con 0x563be968b8e0 2024-04-24T08:11:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.613+0000 7fb09b400640 1 --2- 172.21.15.28:0/850566801 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x563be968b8e0 0x563be968dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.613+0000 7fb09b400640 1 --2- 172.21.15.28:0/850566801 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x563be968b8e0 0x563be968dd00 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c3e54da07704eb53 server_cookie=4d0fdd8ae4f580d3 in_seq=0 out_seq=0 2024-04-24T08:11:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.613+0000 7fb088c00640 10 client.15516 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:11:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.613+0000 7fb088c00640 1 -- 172.21.15.28:0/850566801 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563be9576a60 con 0x563be968b8e0 2024-04-24T08:11:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.613+0000 7fb088c00640 10 client.15516 handle_command_reply: tid=0 2024-04-24T08:11:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.613+0000 7fb0a4dbf740 1 client.15516 shutdown 2024-04-24T08:11:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.614+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/850566801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be960a460 msgr2=0x563be960eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.614+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/850566801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be960a460 0x563be960eaf0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fb084009bf0 tx=0x7fb084011dc0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.614+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/850566801 shutdown_connections 2024-04-24T08:11:34.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.614+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/850566801 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x563be968b8e0 0x563be968dd00 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.614+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/850566801 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563be960a460 0x563be960eaf0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.614+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/850566801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563be9614b30 0x563be960f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.614+0000 7fb0a4dbf740 1 --2- 172.21.15.28:0/850566801 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563be9609b40 0x563be961d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.614+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/850566801 >> 172.21.15.28:0/850566801 conn(0x563be9605e60 msgr2=0x563be9604f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.614+0000 7fb088c00640 0 client.15516 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:11:34.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.614+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/850566801 shutdown_connections 2024-04-24T08:11:34.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.614+0000 7fb0a4dbf740 1 -- 172.21.15.28:0/850566801 wait complete. 2024-04-24T08:11:34.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.614+0000 7fb0a4dbf740 20 client.15516 trim_cache size 0 max 16384 2024-04-24T08:11:34.617 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:11:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.615+0000 7fb0a0a00640 1 -- 172.21.15.28:0/4209167395 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb0800a3ed0 msgr2=0x7fb0800a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.615+0000 7fb0a0a00640 1 --2- 172.21.15.28:0/4209167395 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb0800a3ed0 0x7fb0800a6390 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fb0940057c0 tx=0x7fb0940056d0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.615+0000 7fb0a0a00640 1 -- 172.21.15.28:0/4209167395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb09c101010 msgr2=0x7fb09c1a06a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.615+0000 7fb0a0a00640 1 --2- 172.21.15.28:0/4209167395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb09c101010 0x7fb09c1a06a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fb08c00c420 tx=0x7fb08c009f90 comp rx=0 tx=0).stop 2024-04-24T08:11:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.615+0000 7fb0a0a00640 1 -- 172.21.15.28:0/4209167395 shutdown_connections 2024-04-24T08:11:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.615+0000 7fb0a0a00640 1 --2- 172.21.15.28:0/4209167395 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb0800a3ed0 0x7fb0800a6390 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.615+0000 7fb0a0a00640 1 --2- 172.21.15.28:0/4209167395 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb09c19a760 0x7fb09c19abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.615+0000 7fb0a0a00640 1 --2- 172.21.15.28:0/4209167395 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb09c1060e0 0x7fb09c1a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.615+0000 7fb0a0a00640 1 --2- 172.21.15.28:0/4209167395 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb09c101010 0x7fb09c1a06a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.615+0000 7fb0a0a00640 1 -- 172.21.15.28:0/4209167395 >> 172.21.15.28:0/4209167395 conn(0x7fb09c0fc850 msgr2=0x7fb09c0fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.615+0000 7fb0a0a00640 1 -- 172.21.15.28:0/4209167395 shutdown_connections 2024-04-24T08:11:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.615+0000 7fb0a0a00640 1 -- 172.21.15.28:0/4209167395 wait complete. 2024-04-24T08:11:34.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:34 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2606626993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:11:34.640 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:11:34.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:34 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2606626993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:11:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.837+0000 7f7ba7200640 1 -- 172.21.15.28:0/3378390669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba8103080 msgr2=0x7f7ba8103460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.837+0000 7f7ba7200640 1 --2- 172.21.15.28:0/3378390669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba8103080 0x7f7ba8103460 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f7b98009900 tx=0x7f7b9802a480 comp rx=0 tx=0).stop 2024-04-24T08:11:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.837+0000 7f7ba7200640 1 -- 172.21.15.28:0/3378390669 shutdown_connections 2024-04-24T08:11:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.837+0000 7f7ba7200640 1 --2- 172.21.15.28:0/3378390669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba810c4a0 0x7f7ba810e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.837+0000 7f7ba7200640 1 --2- 172.21.15.28:0/3378390669 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ba81039a0 0x7f7ba810be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.837+0000 7f7ba7200640 1 --2- 172.21.15.28:0/3378390669 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba8103080 0x7f7ba8103460 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.837+0000 7f7ba7200640 1 -- 172.21.15.28:0/3378390669 >> 172.21.15.28:0/3378390669 conn(0x7f7ba80fc880 msgr2=0x7f7ba80feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.837+0000 7f7ba7200640 1 -- 172.21.15.28:0/3378390669 shutdown_connections 2024-04-24T08:11:34.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.837+0000 7f7ba7200640 1 -- 172.21.15.28:0/3378390669 wait complete. 2024-04-24T08:11:34.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.839+0000 7f7ba7200640 1 Processor -- start 2024-04-24T08:11:34.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.839+0000 7f7ba7200640 1 -- start start 2024-04-24T08:11:34.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.839+0000 7f7ba7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba8103080 0x7f7ba8100cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.839+0000 7f7ba7200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ba81039a0 0x7f7ba8101210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.839+0000 7f7ba7200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba810c4a0 0x7f7ba8101750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.839+0000 7f7ba7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba81122d0 con 0x7f7ba8103080 2024-04-24T08:11:34.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.839+0000 7f7ba7200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba8112150 con 0x7f7ba810c4a0 2024-04-24T08:11:34.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.839+0000 7f7ba7200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba8112450 con 0x7f7ba81039a0 2024-04-24T08:11:34.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.839+0000 7f7ba5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba8103080 0x7f7ba8100cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.839+0000 7f7ba5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba8103080 0x7f7ba8100cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57460/0 (socket says 172.21.15.28:57460) 2024-04-24T08:11:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.839+0000 7f7ba5e00640 1 -- 172.21.15.28:0/3090719046 learned_addr learned my addr 172.21.15.28:0/3090719046 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.839+0000 7f7ba6800640 1 --2- 172.21.15.28:0/3090719046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba810c4a0 0x7f7ba8101750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.840+0000 7f7ba5400640 1 --2- 172.21.15.28:0/3090719046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ba81039a0 0x7f7ba8101210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.840+0000 7f7ba5e00640 1 -- 172.21.15.28:0/3090719046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ba81039a0 msgr2=0x7f7ba8101210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.840+0000 7f7ba5e00640 1 --2- 172.21.15.28:0/3090719046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ba81039a0 0x7f7ba8101210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.840+0000 7f7ba5e00640 1 -- 172.21.15.28:0/3090719046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba810c4a0 msgr2=0x7f7ba8101750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.840+0000 7f7ba5e00640 1 --2- 172.21.15.28:0/3090719046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba810c4a0 0x7f7ba8101750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.840+0000 7f7ba5e00640 1 -- 172.21.15.28:0/3090719046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ba81a6cf0 con 0x7f7ba8103080 2024-04-24T08:11:34.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.840+0000 7f7ba5e00640 1 --2- 172.21.15.28:0/3090719046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba8103080 0x7f7ba8100cd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7b98004770 tx=0x7f7b98003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.840+0000 7f7b9ea00640 1 -- 172.21.15.28:0/3090719046 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b9803f070 con 0x7f7ba8103080 2024-04-24T08:11:34.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.841+0000 7f7b9ea00640 1 -- 172.21.15.28:0/3090719046 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b98031ca0 con 0x7f7ba8103080 2024-04-24T08:11:34.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.841+0000 7f7ba7200640 1 -- 172.21.15.28:0/3090719046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ba81a6f80 con 0x7f7ba8103080 2024-04-24T08:11:34.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.841+0000 7f7b9ea00640 1 -- 172.21.15.28:0/3090719046 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b98035030 con 0x7f7ba8103080 2024-04-24T08:11:34.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.841+0000 7f7ba7200640 1 -- 172.21.15.28:0/3090719046 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ba81a73e0 con 0x7f7ba8103080 2024-04-24T08:11:34.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.844+0000 7f7b9ea00640 1 -- 172.21.15.28:0/3090719046 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138930+0+0 (secure 0 0 0) 0x7f7b98042070 con 0x7f7ba8103080 2024-04-24T08:11:34.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.845+0000 7f7baff84740 1 Processor -- start 2024-04-24T08:11:34.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.845+0000 7f7baff84740 1 -- start start 2024-04-24T08:11:34.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.845+0000 7f7baff84740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615ab40 0x55790615af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.845+0000 7f7baff84740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615b460 0x5579061655f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.845+0000 7f7baff84740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 0x557906167f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.845+0000 7f7baff84740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5579060c01d0 con 0x557906165b30 2024-04-24T08:11:34.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.845+0000 7f7baff84740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557906066450 con 0x55790615b460 2024-04-24T08:11:34.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.845+0000 7f7baff84740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5579060bc190 con 0x55790615ab40 2024-04-24T08:11:34.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.846+0000 7f7ba5e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615b460 0x5579061655f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.846+0000 7f7ba5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 0x557906167f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.846+0000 7f7ba5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 0x557906167f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57468/0 (socket says 172.21.15.28:57468) 2024-04-24T08:11:34.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.846+0000 7f7ba5400640 1 -- 172.21.15.28:0/1719529674 learned_addr learned my addr 172.21.15.28:0/1719529674 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:34.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.846+0000 7f7ba5e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615b460 0x5579061655f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49520/0 (socket says 172.21.15.28:49520) 2024-04-24T08:11:34.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.846+0000 7f7ba6800640 1 --2- 172.21.15.28:0/1719529674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615ab40 0x55790615af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.846+0000 7f7b9ea00640 1 --2- 172.21.15.28:0/3090719046 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7b8c0a3f00 0x7f7b8c0a63c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.846+0000 7f7ba5400640 1 -- 172.21.15.28:0/1719529674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615ab40 msgr2=0x55790615af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.846+0000 7f7b9ea00640 1 -- 172.21.15.28:0/3090719046 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f7b980f1110 con 0x7f7ba8103080 2024-04-24T08:11:34.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.846+0000 7f7ba5400640 1 --2- 172.21.15.28:0/1719529674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615ab40 0x55790615af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.846+0000 7f7ba5400640 1 -- 172.21.15.28:0/1719529674 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615b460 msgr2=0x5579061655f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.846+0000 7f7ba5400640 1 --2- 172.21.15.28:0/1719529674 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615b460 0x5579061655f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.846+0000 7f7ba5400640 1 -- 172.21.15.28:0/1719529674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557906123310 con 0x557906165b30 2024-04-24T08:11:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.846+0000 7f7ba6800640 1 --2- 172.21.15.28:0/1719529674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615ab40 0x55790615af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:11:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.846+0000 7f7ba5e00640 1 --2- 172.21.15.28:0/1719529674 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615b460 0x5579061655f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.846+0000 7f7ba6800640 1 --2- 172.21.15.28:0/3090719046 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7b8c0a3f00 0x7f7b8c0a63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.847+0000 7f7ba5400640 1 --2- 172.21.15.28:0/1719529674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 0x557906167f20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009c90 tx=0x7f7b94011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c23909695450cb10 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.847+0000 7f7b93400640 1 -- 172.21.15.28:0/1719529674 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b94005220 con 0x557906165b30 2024-04-24T08:11:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.847+0000 7f7ba6800640 1 --2- 172.21.15.28:0/3090719046 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7b8c0a3f00 0x7f7b8c0a63c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0005fd0 tx=0x7f7ba0004a40 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.847+0000 7f7b93400640 1 -- 172.21.15.28:0/1719529674 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b94004c80 con 0x557906165b30 2024-04-24T08:11:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.847+0000 7f7b93400640 1 -- 172.21.15.28:0/1719529674 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b940197b0 con 0x557906165b30 2024-04-24T08:11:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.848+0000 7f7baff84740 1 -- 172.21.15.28:0/1719529674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 msgr2=0x557906167f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.848+0000 7f7baff84740 1 --2- 172.21.15.28:0/1719529674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 0x557906167f20 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009c90 tx=0x7f7b94011810 comp rx=0 tx=0).stop 2024-04-24T08:11:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.848+0000 7f7baff84740 1 -- 172.21.15.28:0/1719529674 shutdown_connections 2024-04-24T08:11:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.848+0000 7f7baff84740 1 --2- 172.21.15.28:0/1719529674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 0x557906167f20 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.848+0000 7f7baff84740 1 --2- 172.21.15.28:0/1719529674 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615b460 0x5579061655f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.848+0000 7f7baff84740 1 --2- 172.21.15.28:0/1719529674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615ab40 0x55790615af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.848+0000 7f7baff84740 1 -- 172.21.15.28:0/1719529674 >> 172.21.15.28:0/1719529674 conn(0x557906156e60 msgr2=0x557906157240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.848+0000 7f7baff84740 1 -- 172.21.15.28:0/1719529674 shutdown_connections 2024-04-24T08:11:34.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.848+0000 7f7baff84740 1 -- 172.21.15.28:0/1719529674 wait complete. 2024-04-24T08:11:34.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.849+0000 7f7baff84740 1 -- start start 2024-04-24T08:11:34.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.849+0000 7f7baff84740 1 Processor -- start 2024-04-24T08:11:34.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.849+0000 7f7baff84740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615ab40 0x5579061655c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.849+0000 7f7baff84740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615b460 0x5579061628b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.849+0000 7f7baff84740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 0x557906162fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.849+0000 7f7baff84740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5579061689c0 con 0x557906165b30 2024-04-24T08:11:34.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.849+0000 7f7baff84740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557906079810 con 0x55790615ab40 2024-04-24T08:11:34.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.849+0000 7f7baff84740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557906168b40 con 0x55790615b460 2024-04-24T08:11:34.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.849+0000 7f7baff84740 10 client.0 _ll_register_callbacks cb 0x557906047140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:34.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.850+0000 7f7ba5400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615b460 0x5579061628b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.850+0000 7f7ba5400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615b460 0x5579061628b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60174/0 (socket says 172.21.15.28:60174) 2024-04-24T08:11:34.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.850+0000 7f7ba5400640 1 -- 172.21.15.28:0/1302420108 learned_addr learned my addr 172.21.15.28:0/1302420108 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:34.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.850+0000 7f7ba5e00640 1 --2- 172.21.15.28:0/1302420108 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615ab40 0x5579061655c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.850+0000 7f7ba6800640 1 --2- 172.21.15.28:0/1302420108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 0x557906162fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.850+0000 7f7ba5400640 1 -- 172.21.15.28:0/1302420108 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615ab40 msgr2=0x5579061655c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.850+0000 7f7ba5400640 1 --2- 172.21.15.28:0/1302420108 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615ab40 0x5579061655c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.850+0000 7f7ba5400640 1 -- 172.21.15.28:0/1302420108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 msgr2=0x557906162fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.850+0000 7f7ba5400640 1 --2- 172.21.15.28:0/1302420108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 0x557906162fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.850+0000 7f7ba5400640 1 -- 172.21.15.28:0/1302420108 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55790608a3f0 con 0x55790615b460 2024-04-24T08:11:34.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.850+0000 7f7ba6800640 1 --2- 172.21.15.28:0/1302420108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 0x557906162fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:11:34.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.850+0000 7f7ba5400640 1 --2- 172.21.15.28:0/1302420108 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615b460 0x5579061628b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009380 tx=0x7f7b94005db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79b1eaa1b8807de7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.850+0000 7f7b87e00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:11:34.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.851+0000 7f7b87e00640 1 -- 172.21.15.28:0/1302420108 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b94029070 con 0x55790615b460 2024-04-24T08:11:34.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.851+0000 7f7baff84740 1 -- 172.21.15.28:0/1302420108 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b94004c80 con 0x55790615b460 2024-04-24T08:11:34.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.851+0000 7f7b87e00640 1 -- 172.21.15.28:0/1302420108 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f7b94019510 con 0x55790615b460 2024-04-24T08:11:34.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.851+0000 7f7b87e00640 1 -- 172.21.15.28:0/1302420108 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7b94035050 con 0x55790615b460 2024-04-24T08:11:34.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.851+0000 7f7baff84740 10 client.25666 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:34.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.851+0000 7f7baff84740 1 -- 172.21.15.28:0/1302420108 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557905dd23f0 con 0x55790615b460 2024-04-24T08:11:34.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.852+0000 7f7b87e00640 1 -- 172.21.15.28:0/1302420108 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b94004c80 con 0x55790615b460 2024-04-24T08:11:34.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.852+0000 7f7b87e00640 1 -- 172.21.15.28:0/1302420108 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f7b94023690 con 0x55790615b460 2024-04-24T08:11:34.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.852+0000 7f7baff84740 10 client.25666 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:34.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.852+0000 7f7baff84740 10 client.25666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:11:34.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.852+0000 7f7baff84740 20 client.25666 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:34.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.852+0000 7f7baff84740 1 --2- 172.21.15.28:0/1302420108 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557906175c30 0x557906178020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.852+0000 7f7baff84740 4 client.25666 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:11:34.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.852+0000 7f7baff84740 1 -- 172.21.15.28:0/1302420108 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5579060cdbf0 con 0x557906175c30 2024-04-24T08:11:34.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.852+0000 7f7ba5e00640 1 --2- 172.21.15.28:0/1302420108 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557906175c30 0x557906178020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.853+0000 7f7ba5e00640 1 --2- 172.21.15.28:0/1302420108 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557906175c30 0x557906178020 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=975b19405e43b4f1 server_cookie=e977f5265f23523f in_seq=0 out_seq=0 2024-04-24T08:11:34.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.853+0000 7f7b87e00640 10 client.25666 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:34.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.854+0000 7f7b87e00640 1 -- 172.21.15.28:0/1302420108 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5579060cdbf0 con 0x557906175c30 2024-04-24T08:11:34.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.854+0000 7f7b87e00640 10 client.25666 handle_command_reply: tid=0 2024-04-24T08:11:34.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.854+0000 7f7baff84740 1 client.25666 shutdown 2024-04-24T08:11:34.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.854+0000 7f7baff84740 1 -- 172.21.15.28:0/1302420108 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615b460 msgr2=0x5579061628b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.854+0000 7f7baff84740 1 --2- 172.21.15.28:0/1302420108 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615b460 0x5579061628b0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f7b94009380 tx=0x7f7b94005db0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.854+0000 7f7baff84740 1 -- 172.21.15.28:0/1302420108 shutdown_connections 2024-04-24T08:11:34.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.854+0000 7f7baff84740 1 --2- 172.21.15.28:0/1302420108 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557906175c30 0x557906178020 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.854+0000 7f7baff84740 1 --2- 172.21.15.28:0/1302420108 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 0x557906162fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.854+0000 7f7baff84740 1 --2- 172.21.15.28:0/1302420108 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615b460 0x5579061628b0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.854+0000 7f7b87e00640 0 client.25666 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:34.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.854+0000 7f7baff84740 1 --2- 172.21.15.28:0/1302420108 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615ab40 0x5579061655c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.854+0000 7f7baff84740 1 -- 172.21.15.28:0/1302420108 >> 172.21.15.28:0/1302420108 conn(0x557906156e60 msgr2=0x55790615cc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.854+0000 7f7baff84740 1 -- 172.21.15.28:0/1302420108 shutdown_connections 2024-04-24T08:11:34.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.854+0000 7f7baff84740 1 -- 172.21.15.28:0/1302420108 wait complete. 2024-04-24T08:11:34.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.855+0000 7f7baff84740 20 client.25666 trim_cache size 0 max 16384 2024-04-24T08:11:34.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.870+0000 7f7baff84740 1 Processor -- start 2024-04-24T08:11:34.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.870+0000 7f7baff84740 1 -- start start 2024-04-24T08:11:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.870+0000 7f7baff84740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615ab40 0x557906171880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.870+0000 7f7baff84740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615b460 0x5579061650f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.870+0000 7f7baff84740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 0x55790615c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.870+0000 7f7baff84740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba81b2690 con 0x557906165b30 2024-04-24T08:11:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.870+0000 7f7baff84740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba81aff60 con 0x55790615ab40 2024-04-24T08:11:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.870+0000 7f7baff84740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba81addb0 con 0x55790615b460 2024-04-24T08:11:34.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.870+0000 7f7ba6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 0x55790615c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.871+0000 7f7ba6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 0x55790615c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57482/0 (socket says 172.21.15.28:57482) 2024-04-24T08:11:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.871+0000 7f7ba5e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615ab40 0x557906171880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.871+0000 7f7ba6800640 1 -- 172.21.15.28:0/1680570789 learned_addr learned my addr 172.21.15.28:0/1680570789 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.871+0000 7f7ba5400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615b460 0x5579061650f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.871+0000 7f7ba5e00640 1 -- 172.21.15.28:0/1680570789 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615b460 msgr2=0x5579061650f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.871+0000 7f7ba5e00640 1 --2- 172.21.15.28:0/1680570789 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615b460 0x5579061650f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.871+0000 7f7ba5e00640 1 -- 172.21.15.28:0/1680570789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 msgr2=0x55790615c9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.871+0000 7f7ba5e00640 1 --2- 172.21.15.28:0/1680570789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 0x55790615c9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.871+0000 7f7ba5e00640 1 -- 172.21.15.28:0/1680570789 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557905e9c7d0 con 0x55790615ab40 2024-04-24T08:11:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.871+0000 7f7ba5e00640 1 --2- 172.21.15.28:0/1680570789 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615ab40 0x557906171880 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7b980ee600 tx=0x7f7b980f3560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fb6bad9846f90b9d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.871+0000 7f7b92a00640 1 -- 172.21.15.28:0/1680570789 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b98049040 con 0x55790615ab40 2024-04-24T08:11:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.871+0000 7f7b92a00640 1 -- 172.21.15.28:0/1680570789 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b98031e50 con 0x55790615ab40 2024-04-24T08:11:34.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.871+0000 7f7b92a00640 1 -- 172.21.15.28:0/1680570789 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b980b9570 con 0x55790615ab40 2024-04-24T08:11:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.872+0000 7f7baff84740 1 -- 172.21.15.28:0/1680570789 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615ab40 msgr2=0x557906171880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.872+0000 7f7baff84740 1 --2- 172.21.15.28:0/1680570789 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615ab40 0x557906171880 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f7b980ee600 tx=0x7f7b980f3560 comp rx=0 tx=0).stop 2024-04-24T08:11:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.873+0000 7f7baff84740 1 -- 172.21.15.28:0/1680570789 shutdown_connections 2024-04-24T08:11:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.873+0000 7f7baff84740 1 --2- 172.21.15.28:0/1680570789 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557906165b30 0x55790615c9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.873+0000 7f7baff84740 1 --2- 172.21.15.28:0/1680570789 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615b460 0x5579061650f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.873+0000 7f7baff84740 1 --2- 172.21.15.28:0/1680570789 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55790615ab40 0x557906171880 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.873+0000 7f7baff84740 1 -- 172.21.15.28:0/1680570789 >> 172.21.15.28:0/1680570789 conn(0x557906156e60 msgr2=0x55790615c2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.873+0000 7f7baff84740 1 -- 172.21.15.28:0/1680570789 shutdown_connections 2024-04-24T08:11:34.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.873+0000 7f7baff84740 1 -- 172.21.15.28:0/1680570789 wait complete. 2024-04-24T08:11:34.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.873+0000 7f7baff84740 1 -- start start 2024-04-24T08:11:34.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.873+0000 7f7baff84740 1 Processor -- start 2024-04-24T08:11:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.874+0000 7f7baff84740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615ab40 0x557906166ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.874+0000 7f7baff84740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55790615b460 0x557906167410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.874+0000 7f7baff84740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557906165b30 0x55790616c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.874+0000 7f7baff84740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba81ae420 con 0x55790615b460 2024-04-24T08:11:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.874+0000 7f7baff84740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba81af890 con 0x557906165b30 2024-04-24T08:11:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.874+0000 7f7baff84740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7ba81aeaf0 con 0x55790615ab40 2024-04-24T08:11:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.874+0000 7f7baff84740 10 client.0 _ll_register_callbacks cb 0x7f7b980b9220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:34.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.874+0000 7f7ba5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55790615b460 0x557906167410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.874+0000 7f7ba5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55790615b460 0x557906167410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57490/0 (socket says 172.21.15.28:57490) 2024-04-24T08:11:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.875+0000 7f7ba5400640 1 -- 172.21.15.28:0/1965874594 learned_addr learned my addr 172.21.15.28:0/1965874594 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.875+0000 7f7ba5e00640 1 --2- 172.21.15.28:0/1965874594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615ab40 0x557906166ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.875+0000 7f7ba6800640 1 --2- 172.21.15.28:0/1965874594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557906165b30 0x55790616c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.875+0000 7f7ba5400640 1 -- 172.21.15.28:0/1965874594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615ab40 msgr2=0x557906166ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.875+0000 7f7ba5400640 1 --2- 172.21.15.28:0/1965874594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615ab40 0x557906166ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.875+0000 7f7ba5400640 1 -- 172.21.15.28:0/1965874594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557906165b30 msgr2=0x55790616c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.875+0000 7f7ba5400640 1 --2- 172.21.15.28:0/1965874594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557906165b30 0x55790616c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.875+0000 7f7ba5400640 1 -- 172.21.15.28:0/1965874594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55790616c910 con 0x55790615b460 2024-04-24T08:11:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.875+0000 7f7ba5400640 1 --2- 172.21.15.28:0/1965874594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55790615b460 0x557906167410 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x557906162110 tx=0x7f7b94005540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ba439521ecdc4e14 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.875+0000 7f7b87e00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:11:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.875+0000 7f7b87e00640 1 -- 172.21.15.28:0/1965874594 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b94029070 con 0x55790615b460 2024-04-24T08:11:34.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.876+0000 7f7b87e00640 1 -- 172.21.15.28:0/1965874594 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f7b94004ad0 con 0x55790615b460 2024-04-24T08:11:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.876+0000 7f7baff84740 1 -- 172.21.15.28:0/1965874594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7b98031e50 con 0x55790615b460 2024-04-24T08:11:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.876+0000 7f7b87e00640 1 -- 172.21.15.28:0/1965874594 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7b94023e40 con 0x55790615b460 2024-04-24T08:11:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.876+0000 7f7baff84740 10 client.15534 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.876+0000 7f7baff84740 1 -- 172.21.15.28:0/1965874594 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5579061dc760 con 0x55790615b460 2024-04-24T08:11:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.876+0000 7f7b87e00640 1 -- 172.21.15.28:0/1965874594 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b98031e50 con 0x55790615b460 2024-04-24T08:11:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.876+0000 7f7b87e00640 1 -- 172.21.15.28:0/1965874594 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f7b94004240 con 0x55790615b460 2024-04-24T08:11:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.876+0000 7f7baff84740 10 client.15534 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.876+0000 7f7baff84740 10 client.15534 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:11:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.876+0000 7f7baff84740 20 client.15534 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.876+0000 7f7baff84740 1 --2- 172.21.15.28:0/1965874594 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5579061dcea0 0x5579061df2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.876+0000 7f7baff84740 4 client.15534 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:11:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.876+0000 7f7baff84740 1 -- 172.21.15.28:0/1965874594 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5579060c7a60 con 0x5579061dcea0 2024-04-24T08:11:34.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.877+0000 7f7ba5e00640 1 --2- 172.21.15.28:0/1965874594 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5579061dcea0 0x5579061df2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.877+0000 7f7ba5e00640 1 --2- 172.21.15.28:0/1965874594 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5579061dcea0 0x5579061df2c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af50bb00fd0bdc4 server_cookie=32455ef287a44076 in_seq=0 out_seq=0 2024-04-24T08:11:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.877+0000 7f7b87e00640 10 client.15534 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.877+0000 7f7b87e00640 1 -- 172.21.15.28:0/1965874594 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5579060c7a60 con 0x5579061dcea0 2024-04-24T08:11:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.877+0000 7f7b87e00640 10 client.15534 handle_command_reply: tid=0 2024-04-24T08:11:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.878+0000 7f7baff84740 1 client.15534 shutdown 2024-04-24T08:11:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.878+0000 7f7baff84740 1 -- 172.21.15.28:0/1965874594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55790615b460 msgr2=0x557906167410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.878+0000 7f7baff84740 1 --2- 172.21.15.28:0/1965874594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55790615b460 0x557906167410 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x557906162110 tx=0x7f7b94005540 comp rx=0 tx=0).stop 2024-04-24T08:11:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.878+0000 7f7baff84740 1 -- 172.21.15.28:0/1965874594 shutdown_connections 2024-04-24T08:11:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.878+0000 7f7baff84740 1 --2- 172.21.15.28:0/1965874594 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5579061dcea0 0x5579061df2c0 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.878+0000 7f7baff84740 1 --2- 172.21.15.28:0/1965874594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55790615b460 0x557906167410 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.878+0000 7f7baff84740 1 --2- 172.21.15.28:0/1965874594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557906165b30 0x55790616c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.878+0000 7f7baff84740 1 --2- 172.21.15.28:0/1965874594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55790615ab40 0x557906166ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.878+0000 7f7baff84740 1 -- 172.21.15.28:0/1965874594 >> 172.21.15.28:0/1965874594 conn(0x557906156e60 msgr2=0x55790615cb20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.878+0000 7f7b87e00640 0 client.15534 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:34.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.878+0000 7f7baff84740 1 -- 172.21.15.28:0/1965874594 shutdown_connections 2024-04-24T08:11:34.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.878+0000 7f7baff84740 1 -- 172.21.15.28:0/1965874594 wait complete. 2024-04-24T08:11:34.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.878+0000 7f7baff84740 20 client.15534 trim_cache size 0 max 16384 2024-04-24T08:11:34.882 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:11:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.879+0000 7f7ba7200640 1 -- 172.21.15.28:0/3090719046 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7b8c0a3f00 msgr2=0x7f7b8c0a63c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.879+0000 7f7ba7200640 1 --2- 172.21.15.28:0/3090719046 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7b8c0a3f00 0x7f7b8c0a63c0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7ba0005fd0 tx=0x7f7ba0004a40 comp rx=0 tx=0).stop 2024-04-24T08:11:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.879+0000 7f7ba7200640 1 -- 172.21.15.28:0/3090719046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba8103080 msgr2=0x7f7ba8100cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.879+0000 7f7ba7200640 1 --2- 172.21.15.28:0/3090719046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba8103080 0x7f7ba8100cd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f7b98004770 tx=0x7f7b98003c10 comp rx=0 tx=0).stop 2024-04-24T08:11:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.879+0000 7f7ba7200640 1 -- 172.21.15.28:0/3090719046 shutdown_connections 2024-04-24T08:11:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.879+0000 7f7ba7200640 1 --2- 172.21.15.28:0/3090719046 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7b8c0a3f00 0x7f7b8c0a63c0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.879+0000 7f7ba7200640 1 --2- 172.21.15.28:0/3090719046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba810c4a0 0x7f7ba8101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.879+0000 7f7ba7200640 1 --2- 172.21.15.28:0/3090719046 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ba81039a0 0x7f7ba8101210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.879+0000 7f7ba7200640 1 --2- 172.21.15.28:0/3090719046 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ba8103080 0x7f7ba8100cd0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:34.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.879+0000 7f7ba7200640 1 -- 172.21.15.28:0/3090719046 >> 172.21.15.28:0/3090719046 conn(0x7f7ba80fc880 msgr2=0x7f7ba80fe040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.879+0000 7f7ba7200640 1 -- 172.21.15.28:0/3090719046 shutdown_connections 2024-04-24T08:11:34.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:34.879+0000 7f7ba7200640 1 -- 172.21.15.28:0/3090719046 wait complete. 2024-04-24T08:11:35.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:35 smithi028 ceph-mon[27786]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:35.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:35 smithi117 ceph-mon[32438]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:35.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:35 smithi194 ceph-mon[32836]: pgmap v40: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:35.897 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:11:35.897 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:11:35.897 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:11:36.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.120+0000 7f55dd600640 1 -- 172.21.15.28:0/1881745392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55d8102ac0 msgr2=0x7f55d8106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.120+0000 7f55dd600640 1 --2- 172.21.15.28:0/1881745392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55d8102ac0 0x7f55d8106b00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f55c0009900 tx=0x7f55c002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.120+0000 7f55dd600640 1 -- 172.21.15.28:0/1881745392 shutdown_connections 2024-04-24T08:11:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.120+0000 7f55dd600640 1 --2- 172.21.15.28:0/1881745392 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55d81071e0 0x7f55d810edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.120+0000 7f55dd600640 1 --2- 172.21.15.28:0/1881745392 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55d8102ac0 0x7f55d8106b00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.120+0000 7f55dd600640 1 --2- 172.21.15.28:0/1881745392 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d8102110 0x7f55d81024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.120+0000 7f55dd600640 1 -- 172.21.15.28:0/1881745392 >> 172.21.15.28:0/1881745392 conn(0x7f55d80fd9d0 msgr2=0x7f55d80ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.120+0000 7f55dd600640 1 -- 172.21.15.28:0/1881745392 shutdown_connections 2024-04-24T08:11:36.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.120+0000 7f55dd600640 1 -- 172.21.15.28:0/1881745392 wait complete. 2024-04-24T08:11:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.121+0000 7f55dd600640 1 Processor -- start 2024-04-24T08:11:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.121+0000 7f55dd600640 1 -- start start 2024-04-24T08:11:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.122+0000 7f55dd600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d8102110 0x7f55d81a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.122+0000 7f55dd600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55d81071e0 0x7f55d81a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.122+0000 7f55dd600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55d819a7d0 0x7f55d819ac80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.122+0000 7f55dd600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55d81122c0 con 0x7f55d8102110 2024-04-24T08:11:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.122+0000 7f55dd600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f55d8112140 con 0x7f55d819a7d0 2024-04-24T08:11:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.122+0000 7f55dd600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f55d8112440 con 0x7f55d81071e0 2024-04-24T08:11:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.122+0000 7f55d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d8102110 0x7f55d81a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.122+0000 7f55d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d8102110 0x7f55d81a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57508/0 (socket says 172.21.15.28:57508) 2024-04-24T08:11:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.122+0000 7f55d7e00640 1 -- 172.21.15.28:0/1931705700 learned_addr learned my addr 172.21.15.28:0/1931705700 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.122+0000 7f55d7400640 1 --2- 172.21.15.28:0/1931705700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55d81071e0 0x7f55d81a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.122+0000 7f55dcc00640 1 --2- 172.21.15.28:0/1931705700 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55d819a7d0 0x7f55d819ac80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.122+0000 7f55d7e00640 1 -- 172.21.15.28:0/1931705700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55d81071e0 msgr2=0x7f55d81a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.122+0000 7f55d7e00640 1 --2- 172.21.15.28:0/1931705700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55d81071e0 0x7f55d81a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.122+0000 7f55d7e00640 1 -- 172.21.15.28:0/1931705700 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55d819a7d0 msgr2=0x7f55d819ac80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.122+0000 7f55d7e00640 1 --2- 172.21.15.28:0/1931705700 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55d819a7d0 0x7f55d819ac80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.122+0000 7f55d7e00640 1 -- 172.21.15.28:0/1931705700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55d819b360 con 0x7f55d8102110 2024-04-24T08:11:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.123+0000 7f55d7e00640 1 --2- 172.21.15.28:0/1931705700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d8102110 0x7f55d81a0660 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f55c800ebc0 tx=0x7f55c800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.123+0000 7f55d4c00640 1 -- 172.21.15.28:0/1931705700 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8010630 con 0x7f55d8102110 2024-04-24T08:11:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.123+0000 7f55d4c00640 1 -- 172.21.15.28:0/1931705700 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55c8010cb0 con 0x7f55d8102110 2024-04-24T08:11:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.123+0000 7f55d4c00640 1 -- 172.21.15.28:0/1931705700 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8018bd0 con 0x7f55d8102110 2024-04-24T08:11:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.123+0000 7f55dd600640 1 -- 172.21.15.28:0/1931705700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55d819b650 con 0x7f55d8102110 2024-04-24T08:11:36.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.123+0000 7f55dd600640 1 -- 172.21.15.28:0/1931705700 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55d81a7850 con 0x7f55d8102110 2024-04-24T08:11:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.127+0000 7f55e1bed740 1 Processor -- start 2024-04-24T08:11:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.127+0000 7f55d4c00640 1 -- 172.21.15.28:0/1931705700 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 26) ==== 138930+0+0 (secure 0 0 0) 0x7f55c80107d0 con 0x7f55d8102110 2024-04-24T08:11:36.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.127+0000 7f55e1bed740 1 -- start start 2024-04-24T08:11:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.127+0000 7f55e1bed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22118270 0x55df22118650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.127+0000 7f55e1bed740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118b90 0x55df22122d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.127+0000 7f55e1bed740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22123260 0x55df22125650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.127+0000 7f55e1bed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55df2207de70 con 0x55df22118270 2024-04-24T08:11:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.127+0000 7f55e1bed740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55df22024380 con 0x55df22123260 2024-04-24T08:11:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.127+0000 7f55e1bed740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55df22079e30 con 0x55df22118b90 2024-04-24T08:11:36.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.127+0000 7f55d7400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22123260 0x55df22125650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.127+0000 7f55dcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22118270 0x55df22118650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.127+0000 7f55d7400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22123260 0x55df22125650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49582/0 (socket says 172.21.15.28:49582) 2024-04-24T08:11:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.127+0000 7f55d7400640 1 -- 172.21.15.28:0/2738422895 learned_addr learned my addr 172.21.15.28:0/2738422895 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.128+0000 7f55d7e00640 1 --2- 172.21.15.28:0/2738422895 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118b90 0x55df22122d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.128+0000 7f55d7400640 1 -- 172.21.15.28:0/2738422895 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118b90 msgr2=0x55df22122d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.128+0000 7f55d7400640 1 --2- 172.21.15.28:0/2738422895 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118b90 0x55df22122d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.128+0000 7f55d7400640 1 -- 172.21.15.28:0/2738422895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22118270 msgr2=0x55df22118650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.128+0000 7f55d7400640 1 --2- 172.21.15.28:0/2738422895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22118270 0x55df22118650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.128+0000 7f55d7400640 1 -- 172.21.15.28:0/2738422895 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df21e587e0 con 0x55df22123260 2024-04-24T08:11:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.128+0000 7f55d7400640 1 --2- 172.21.15.28:0/2738422895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22123260 0x55df22125650 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f55c0009900 tx=0x7f55c0005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c835ffa3816fed9e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.128+0000 7f55d4c00640 1 --2- 172.21.15.28:0/1931705700 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f55bc0a3f50 0x7f55bc0a6410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.128+0000 7f55cd200640 1 -- 172.21.15.28:0/2738422895 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c0035040 con 0x55df22123260 2024-04-24T08:11:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.129+0000 7f55cd200640 1 -- 172.21.15.28:0/2738422895 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55c0005520 con 0x55df22123260 2024-04-24T08:11:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.129+0000 7f55cd200640 1 -- 172.21.15.28:0/2738422895 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c00035f0 con 0x55df22123260 2024-04-24T08:11:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.129+0000 7f55d4c00640 1 -- 172.21.15.28:0/1931705700 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f55c80d10b0 con 0x7f55d8102110 2024-04-24T08:11:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.129+0000 7f55dcc00640 1 --2- 172.21.15.28:0/1931705700 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f55bc0a3f50 0x7f55bc0a6410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.129+0000 7f55e1bed740 1 -- 172.21.15.28:0/2738422895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22123260 msgr2=0x55df22125650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.129+0000 7f55e1bed740 1 --2- 172.21.15.28:0/2738422895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22123260 0x55df22125650 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f55c0009900 tx=0x7f55c0005de0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.130+0000 7f55dcc00640 1 --2- 172.21.15.28:0/1931705700 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f55bc0a3f50 0x7f55bc0a6410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f55d0009d20 tx=0x7f55d0005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.130+0000 7f55e1bed740 1 -- 172.21.15.28:0/2738422895 shutdown_connections 2024-04-24T08:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.130+0000 7f55e1bed740 1 --2- 172.21.15.28:0/2738422895 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22123260 0x55df22125650 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.130+0000 7f55e1bed740 1 --2- 172.21.15.28:0/2738422895 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118b90 0x55df22122d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.130+0000 7f55e1bed740 1 --2- 172.21.15.28:0/2738422895 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22118270 0x55df22118650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.130+0000 7f55e1bed740 1 -- 172.21.15.28:0/2738422895 >> 172.21.15.28:0/2738422895 conn(0x55df22114590 msgr2=0x55df22114970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.130+0000 7f55e1bed740 1 -- 172.21.15.28:0/2738422895 shutdown_connections 2024-04-24T08:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.130+0000 7f55e1bed740 1 -- 172.21.15.28:0/2738422895 wait complete. 2024-04-24T08:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.130+0000 7f55e1bed740 1 -- start start 2024-04-24T08:11:36.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.131+0000 7f55e1bed740 1 Processor -- start 2024-04-24T08:11:36.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.131+0000 7f55e1bed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22118270 0x55df221201a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.131+0000 7f55e1bed740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22118b90 0x55df221206e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.131+0000 7f55e1bed740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22123260 0x55df22120c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.131+0000 7f55e1bed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55df22125ff0 con 0x55df22118270 2024-04-24T08:11:36.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.131+0000 7f55e1bed740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55df22037660 con 0x55df22118b90 2024-04-24T08:11:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.131+0000 7f55e1bed740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55df22126170 con 0x55df22123260 2024-04-24T08:11:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.132+0000 7f55e1bed740 10 client.0 _ll_register_callbacks cb 0x55df22025aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.132+0000 7f55d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22118270 0x55df221201a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.132+0000 7f55d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22118270 0x55df221201a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57524/0 (socket says 172.21.15.28:57524) 2024-04-24T08:11:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.132+0000 7f55d7e00640 1 -- 172.21.15.28:0/2897167432 learned_addr learned my addr 172.21.15.28:0/2897167432 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.132+0000 7f55d7400640 1 --2- 172.21.15.28:0/2897167432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22118b90 0x55df221206e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.132+0000 7f55dcc00640 1 --2- 172.21.15.28:0/2897167432 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22123260 0x55df22120c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.132+0000 7f55d7e00640 1 -- 172.21.15.28:0/2897167432 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22123260 msgr2=0x55df22120c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.132+0000 7f55d7e00640 1 --2- 172.21.15.28:0/2897167432 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22123260 0x55df22120c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.132+0000 7f55d7e00640 1 -- 172.21.15.28:0/2897167432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22118b90 msgr2=0x55df221206e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.132+0000 7f55d7e00640 1 --2- 172.21.15.28:0/2897167432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22118b90 0x55df221206e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.132+0000 7f55d7e00640 1 -- 172.21.15.28:0/2897167432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df21eb8f70 con 0x55df22118270 2024-04-24T08:11:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.132+0000 7f55d7e00640 1 --2- 172.21.15.28:0/2897167432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22118270 0x55df221201a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f55c809cba0 tx=0x7f55c809cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d70c1df15ce7d7e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.132+0000 7f55c5600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:11:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.133+0000 7f55c5600640 1 -- 172.21.15.28:0/2897167432 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8010e60 con 0x55df22118270 2024-04-24T08:11:36.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.133+0000 7f55c5600640 1 -- 172.21.15.28:0/2897167432 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f55c80d3270 con 0x55df22118270 2024-04-24T08:11:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.133+0000 7f55e1bed740 1 -- 172.21.15.28:0/2897167432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55c0005520 con 0x55df22118270 2024-04-24T08:11:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.133+0000 7f55c5600640 1 -- 172.21.15.28:0/2897167432 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f55c809cea0 con 0x55df22118270 2024-04-24T08:11:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.133+0000 7f55e1bed740 10 client.15546 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.133+0000 7f55e1bed740 1 -- 172.21.15.28:0/2897167432 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55df21d50170 con 0x55df22118270 2024-04-24T08:11:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.133+0000 7f55c5600640 1 -- 172.21.15.28:0/2897167432 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55c0005520 con 0x55df22118270 2024-04-24T08:11:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.133+0000 7f55c5600640 1 -- 172.21.15.28:0/2897167432 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f55c809c2f0 con 0x55df22118270 2024-04-24T08:11:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.134+0000 7f55e1bed740 10 client.15546 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.134+0000 7f55e1bed740 10 client.15546 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:11:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.134+0000 7f55e1bed740 20 client.15546 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.134+0000 7f55e1bed740 1 --2- 172.21.15.28:0/2897167432 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55df22133480 0x55df221358f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.134+0000 7f55e1bed740 4 client.15546 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:11:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.134+0000 7f55e1bed740 1 -- 172.21.15.28:0/2897167432 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55df2208b890 con 0x55df22133480 2024-04-24T08:11:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.134+0000 7f55d7400640 1 --2- 172.21.15.28:0/2897167432 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55df22133480 0x55df221358f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.135+0000 7f55c5600640 10 client.15546 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:36.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.135+0000 7f55d7400640 1 --2- 172.21.15.28:0/2897167432 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55df22133480 0x55df221358f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdaec148416ec837 server_cookie=cea5c0a80d830d05 in_seq=0 out_seq=0 2024-04-24T08:11:36.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.135+0000 7f55c5600640 1 -- 172.21.15.28:0/2897167432 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55df2208b890 con 0x55df22133480 2024-04-24T08:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.135+0000 7f55c5600640 10 client.15546 handle_command_reply: tid=0 2024-04-24T08:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.136+0000 7f55e1bed740 1 client.15546 shutdown 2024-04-24T08:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.136+0000 7f55e1bed740 1 -- 172.21.15.28:0/2897167432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22118270 msgr2=0x55df221201a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.136+0000 7f55e1bed740 1 --2- 172.21.15.28:0/2897167432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22118270 0x55df221201a0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f55c809cba0 tx=0x7f55c809cc70 comp rx=0 tx=0).stop 2024-04-24T08:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.136+0000 7f55e1bed740 1 -- 172.21.15.28:0/2897167432 shutdown_connections 2024-04-24T08:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.136+0000 7f55e1bed740 1 --2- 172.21.15.28:0/2897167432 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55df22133480 0x55df221358f0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.136+0000 7f55e1bed740 1 --2- 172.21.15.28:0/2897167432 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22123260 0x55df22120c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.136+0000 7f55e1bed740 1 --2- 172.21.15.28:0/2897167432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22118b90 0x55df221206e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.136+0000 7f55e1bed740 1 --2- 172.21.15.28:0/2897167432 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22118270 0x55df221201a0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.136+0000 7f55c5600640 0 client.15546 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.136+0000 7f55e1bed740 1 -- 172.21.15.28:0/2897167432 >> 172.21.15.28:0/2897167432 conn(0x55df22114590 msgr2=0x55df2211a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.136+0000 7f55e1bed740 1 -- 172.21.15.28:0/2897167432 shutdown_connections 2024-04-24T08:11:36.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.136+0000 7f55e1bed740 1 -- 172.21.15.28:0/2897167432 wait complete. 2024-04-24T08:11:36.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.136+0000 7f55e1bed740 20 client.15546 trim_cache size 0 max 16384 2024-04-24T08:11:36.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.166+0000 7f55e1bed740 1 Processor -- start 2024-04-24T08:11:36.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.166+0000 7f55e1bed740 1 -- start start 2024-04-24T08:11:36.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.167+0000 7f55e1bed740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22118270 0x55df2212eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.167+0000 7f55e1bed740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118b90 0x55df2211fa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.167+0000 7f55e1bed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22123260 0x55df22113570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.167+0000 7f55e1bed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55d81ae380 con 0x55df22123260 2024-04-24T08:11:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.167+0000 7f55e1bed740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f55d81abc50 con 0x55df22118270 2024-04-24T08:11:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.167+0000 7f55e1bed740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f55d81a9ac0 con 0x55df22118b90 2024-04-24T08:11:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.167+0000 7f55dcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22123260 0x55df22113570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.167+0000 7f55d7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118b90 0x55df2211fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.167+0000 7f55dcc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22123260 0x55df22113570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57530/0 (socket says 172.21.15.28:57530) 2024-04-24T08:11:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.167+0000 7f55dcc00640 1 -- 172.21.15.28:0/443904242 learned_addr learned my addr 172.21.15.28:0/443904242 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:36.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.167+0000 7f55d7e00640 1 --2- 172.21.15.28:0/443904242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22118270 0x55df2212eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.167+0000 7f55d7400640 1 -- 172.21.15.28:0/443904242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22118270 msgr2=0x55df2212eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.167+0000 7f55d7400640 1 --2- 172.21.15.28:0/443904242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22118270 0x55df2212eeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.167+0000 7f55d7400640 1 -- 172.21.15.28:0/443904242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22123260 msgr2=0x55df22113570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.168+0000 7f55d7400640 1 --2- 172.21.15.28:0/443904242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22123260 0x55df22113570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.168+0000 7f55d7400640 1 -- 172.21.15.28:0/443904242 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55df2210ea60 con 0x55df22118b90 2024-04-24T08:11:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.168+0000 7f55d7e00640 1 --2- 172.21.15.28:0/443904242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22118270 0x55df2212eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.168+0000 7f55d7400640 1 --2- 172.21.15.28:0/443904242 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118b90 0x55df2211fa80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f55c0031550 tx=0x7f55c0004b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7562cdb87dd6a17c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.168+0000 7f55c7e00640 1 -- 172.21.15.28:0/443904242 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c0035040 con 0x55df22118b90 2024-04-24T08:11:36.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.168+0000 7f55c7e00640 1 -- 172.21.15.28:0/443904242 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55c0004380 con 0x55df22118b90 2024-04-24T08:11:36.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.168+0000 7f55c7e00640 1 -- 172.21.15.28:0/443904242 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c0031a80 con 0x55df22118b90 2024-04-24T08:11:36.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.169+0000 7f55e1bed740 1 -- 172.21.15.28:0/443904242 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118b90 msgr2=0x55df2211fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.169+0000 7f55e1bed740 1 --2- 172.21.15.28:0/443904242 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118b90 0x55df2211fa80 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f55c0031550 tx=0x7f55c0004b20 comp rx=0 tx=0).stop 2024-04-24T08:11:36.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.169+0000 7f55e1bed740 1 -- 172.21.15.28:0/443904242 shutdown_connections 2024-04-24T08:11:36.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.169+0000 7f55e1bed740 1 --2- 172.21.15.28:0/443904242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22123260 0x55df22113570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.169+0000 7f55e1bed740 1 --2- 172.21.15.28:0/443904242 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118b90 0x55df2211fa80 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.169+0000 7f55e1bed740 1 --2- 172.21.15.28:0/443904242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22118270 0x55df2212eeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.169+0000 7f55e1bed740 1 -- 172.21.15.28:0/443904242 >> 172.21.15.28:0/443904242 conn(0x55df22114590 msgr2=0x55df22112e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:36.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.169+0000 7f55e1bed740 1 -- 172.21.15.28:0/443904242 shutdown_connections 2024-04-24T08:11:36.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.169+0000 7f55e1bed740 1 -- 172.21.15.28:0/443904242 wait complete. 2024-04-24T08:11:36.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.170+0000 7f55e1bed740 1 -- start start 2024-04-24T08:11:36.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.170+0000 7f55e1bed740 1 Processor -- start 2024-04-24T08:11:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.171+0000 7f55e1bed740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118270 0x55df2212b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.171+0000 7f55e1bed740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22118b90 0x55df2211cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.171+0000 7f55e1bed740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22123260 0x55df2211d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.171+0000 7f55e1bed740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55d81aa110 con 0x55df22123260 2024-04-24T08:11:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.171+0000 7f55e1bed740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f55d81ab580 con 0x55df22118b90 2024-04-24T08:11:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.171+0000 7f55e1bed740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f55d81aa7e0 con 0x55df22118270 2024-04-24T08:11:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.171+0000 7f55e1bed740 10 client.0 _ll_register_callbacks cb 0x7f55c8098ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.171+0000 7f55d7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118270 0x55df2212b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.171+0000 7f55d7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118270 0x55df2212b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60244/0 (socket says 172.21.15.28:60244) 2024-04-24T08:11:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.171+0000 7f55d7e00640 1 -- 172.21.15.28:0/2098698861 learned_addr learned my addr 172.21.15.28:0/2098698861 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.171+0000 7f55dcc00640 1 --2- 172.21.15.28:0/2098698861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22123260 0x55df2211d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.171+0000 7f55d7400640 1 --2- 172.21.15.28:0/2098698861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22118b90 0x55df2211cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.171+0000 7f55d7e00640 1 -- 172.21.15.28:0/2098698861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22118b90 msgr2=0x55df2211cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.172+0000 7f55d7e00640 1 --2- 172.21.15.28:0/2098698861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22118b90 0x55df2211cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.172+0000 7f55d7e00640 1 -- 172.21.15.28:0/2098698861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22123260 msgr2=0x55df2211d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.172+0000 7f55d7e00640 1 --2- 172.21.15.28:0/2098698861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22123260 0x55df2211d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.172+0000 7f55d7e00640 1 -- 172.21.15.28:0/2098698861 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55df2211d9e0 con 0x55df22118270 2024-04-24T08:11:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.172+0000 7f55d7e00640 1 --2- 172.21.15.28:0/2098698861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118270 0x55df2212b850 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f55c8099300 tx=0x7f55c809cda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a480c178f03b8de8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.172+0000 7f55c5600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:11:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.172+0000 7f55c5600640 1 -- 172.21.15.28:0/2098698861 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55c8010e60 con 0x55df22118270 2024-04-24T08:11:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.172+0000 7f55c5600640 1 -- 172.21.15.28:0/2098698861 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f55c80b6250 con 0x55df22118270 2024-04-24T08:11:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.172+0000 7f55e1bed740 1 -- 172.21.15.28:0/2098698861 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55c0004380 con 0x55df22118270 2024-04-24T08:11:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.172+0000 7f55c5600640 1 -- 172.21.15.28:0/2098698861 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f55c80a11a0 con 0x55df22118270 2024-04-24T08:11:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.173+0000 7f55e1bed740 10 client.25690 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.173+0000 7f55e1bed740 1 -- 172.21.15.28:0/2098698861 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55df22199b40 con 0x55df22118270 2024-04-24T08:11:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.173+0000 7f55c5600640 1 -- 172.21.15.28:0/2098698861 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55c0004380 con 0x55df22118270 2024-04-24T08:11:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.173+0000 7f55c5600640 1 -- 172.21.15.28:0/2098698861 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f55c80ce8b0 con 0x55df22118270 2024-04-24T08:11:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.173+0000 7f55e1bed740 10 client.25690 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:36.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.173+0000 7f55e1bed740 10 client.25690 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:11:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.173+0000 7f55e1bed740 20 client.25690 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.173+0000 7f55e1bed740 1 --2- 172.21.15.28:0/2098698861 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55df2219a270 0x55df2219c690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.173+0000 7f55e1bed740 4 client.25690 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:11:36.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.173+0000 7f55e1bed740 1 -- 172.21.15.28:0/2098698861 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55df22085700 con 0x55df2219a270 2024-04-24T08:11:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.174+0000 7f55d7400640 1 --2- 172.21.15.28:0/2098698861 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55df2219a270 0x55df2219c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.174+0000 7f55d7400640 1 --2- 172.21.15.28:0/2098698861 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55df2219a270 0x55df2219c690 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac599390a86a573f server_cookie=7fc4672aef929ebb in_seq=0 out_seq=0 2024-04-24T08:11:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.174+0000 7f55c5600640 10 client.25690 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.175+0000 7f55c5600640 1 -- 172.21.15.28:0/2098698861 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55df22085700 con 0x55df2219a270 2024-04-24T08:11:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.175+0000 7f55c5600640 10 client.25690 handle_command_reply: tid=0 2024-04-24T08:11:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.176+0000 7f55e1bed740 1 client.25690 shutdown 2024-04-24T08:11:36.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.176+0000 7f55e1bed740 1 -- 172.21.15.28:0/2098698861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118270 msgr2=0x55df2212b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.176+0000 7f55e1bed740 1 --2- 172.21.15.28:0/2098698861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118270 0x55df2212b850 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f55c8099300 tx=0x7f55c809cda0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.176+0000 7f55e1bed740 1 -- 172.21.15.28:0/2098698861 shutdown_connections 2024-04-24T08:11:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.176+0000 7f55e1bed740 1 --2- 172.21.15.28:0/2098698861 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55df2219a270 0x55df2219c690 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.176+0000 7f55e1bed740 1 --2- 172.21.15.28:0/2098698861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55df22123260 0x55df2211d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.176+0000 7f55e1bed740 1 --2- 172.21.15.28:0/2098698861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55df22118b90 0x55df2211cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.176+0000 7f55e1bed740 1 --2- 172.21.15.28:0/2098698861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55df22118270 0x55df2212b850 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.176+0000 7f55e1bed740 1 -- 172.21.15.28:0/2098698861 >> 172.21.15.28:0/2098698861 conn(0x55df22114590 msgr2=0x55df221136f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.176+0000 7f55c5600640 0 client.25690 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.177+0000 7f55e1bed740 1 -- 172.21.15.28:0/2098698861 shutdown_connections 2024-04-24T08:11:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.177+0000 7f55e1bed740 1 -- 172.21.15.28:0/2098698861 wait complete. 2024-04-24T08:11:36.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.177+0000 7f55e1bed740 20 client.25690 trim_cache size 0 max 16384 2024-04-24T08:11:36.179 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:11:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-24T08:11:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "cea9db14-51df-4d18-b27b-8d8d49df0562", 2024-04-24T08:11:36.179 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-24T08:11:36.179 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.178+0000 7f55dd600640 1 -- 172.21.15.28:0/1931705700 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f55bc0a3f50 msgr2=0x7f55bc0a6410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.178+0000 7f55dd600640 1 --2- 172.21.15.28:0/1931705700 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f55bc0a3f50 0x7f55bc0a6410 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f55d0009d20 tx=0x7f55d0005f90 comp rx=0 tx=0).stop 2024-04-24T08:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.178+0000 7f55dd600640 1 -- 172.21.15.28:0/1931705700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d8102110 msgr2=0x7f55d81a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.178+0000 7f55dd600640 1 --2- 172.21.15.28:0/1931705700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d8102110 0x7f55d81a0660 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f55c800ebc0 tx=0x7f55c800c6a0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.178+0000 7f55dd600640 1 -- 172.21.15.28:0/1931705700 shutdown_connections 2024-04-24T08:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.178+0000 7f55dd600640 1 --2- 172.21.15.28:0/1931705700 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f55bc0a3f50 0x7f55bc0a6410 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.178+0000 7f55dd600640 1 --2- 172.21.15.28:0/1931705700 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55d819a7d0 0x7f55d819ac80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.178+0000 7f55dd600640 1 --2- 172.21.15.28:0/1931705700 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55d81071e0 0x7f55d81a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.178+0000 7f55dd600640 1 --2- 172.21.15.28:0/1931705700 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55d8102110 0x7f55d81a0660 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.178+0000 7f55dd600640 1 -- 172.21.15.28:0/1931705700 >> 172.21.15.28:0/1931705700 conn(0x7f55d80fd9d0 msgr2=0x7f55d80ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.179+0000 7f55dd600640 1 -- 172.21.15.28:0/1931705700 shutdown_connections 2024-04-24T08:11:36.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.179+0000 7f55dd600640 1 -- 172.21.15.28:0/1931705700 wait complete. 2024-04-24T08:11:36.195 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:11:36.196 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:11:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.390+0000 7f3f59e00640 1 -- 172.21.15.28:0/1922333994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54104cd0 msgr2=0x7f3f54108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.390+0000 7f3f59e00640 1 --2- 172.21.15.28:0/1922333994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54104cd0 0x7f3f54108d10 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3f40009900 tx=0x7f3f4002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.390+0000 7f3f59e00640 1 -- 172.21.15.28:0/1922333994 shutdown_connections 2024-04-24T08:11:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.390+0000 7f3f59e00640 1 --2- 172.21.15.28:0/1922333994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54104cd0 0x7f3f54108d10 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.390+0000 7f3f59e00640 1 --2- 172.21.15.28:0/1922333994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f541093f0 0x7f3f54110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.390+0000 7f3f59e00640 1 --2- 172.21.15.28:0/1922333994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f54104320 0x7f3f54104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.390+0000 7f3f59e00640 1 -- 172.21.15.28:0/1922333994 >> 172.21.15.28:0/1922333994 conn(0x7f3f540ffb40 msgr2=0x7f3f54101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.390+0000 7f3f59e00640 1 -- 172.21.15.28:0/1922333994 shutdown_connections 2024-04-24T08:11:36.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.390+0000 7f3f59e00640 1 -- 172.21.15.28:0/1922333994 wait complete. 2024-04-24T08:11:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.391+0000 7f3f59e00640 1 Processor -- start 2024-04-24T08:11:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.391+0000 7f3f59e00640 1 -- start start 2024-04-24T08:11:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.392+0000 7f3f59e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54104320 0x7f3f541a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.392+0000 7f3f59e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f541093f0 0x7f3f541a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.392+0000 7f3f59e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f5419a7f0 0x7f3f5419aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.392+0000 7f3f59e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541144d0 con 0x7f3f54104320 2024-04-24T08:11:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.392+0000 7f3f59e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3f54114350 con 0x7f3f5419a7f0 2024-04-24T08:11:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.392+0000 7f3f59e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3f54114650 con 0x7f3f541093f0 2024-04-24T08:11:36.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.392+0000 7f3f53e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f541093f0 0x7f3f541a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.392+0000 7f3f58a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54104320 0x7f3f541a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.392+0000 7f3f53e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f541093f0 0x7f3f541a0bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60266/0 (socket says 172.21.15.28:60266) 2024-04-24T08:11:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.392+0000 7f3f53e00640 1 -- 172.21.15.28:0/3368876226 learned_addr learned my addr 172.21.15.28:0/3368876226 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.392+0000 7f3f59400640 1 --2- 172.21.15.28:0/3368876226 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f5419a7f0 0x7f3f5419aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.393+0000 7f3f53e00640 1 -- 172.21.15.28:0/3368876226 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f5419a7f0 msgr2=0x7f3f5419aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.393+0000 7f3f53e00640 1 --2- 172.21.15.28:0/3368876226 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f5419a7f0 0x7f3f5419aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.393+0000 7f3f53e00640 1 -- 172.21.15.28:0/3368876226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54104320 msgr2=0x7f3f541a0680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.393+0000 7f3f53e00640 1 --2- 172.21.15.28:0/3368876226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54104320 0x7f3f541a0680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.393+0000 7f3f53e00640 1 -- 172.21.15.28:0/3368876226 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f5419b380 con 0x7f3f541093f0 2024-04-24T08:11:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.393+0000 7f3f58a00640 1 --2- 172.21.15.28:0/3368876226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54104320 0x7f3f541a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:36.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.393+0000 7f3f53e00640 1 --2- 172.21.15.28:0/3368876226 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f541093f0 0x7f3f541a0bc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3f4002a910 tx=0x7f3f40003b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.393+0000 7f3f51600640 1 -- 172.21.15.28:0/3368876226 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4003f070 con 0x7f3f541093f0 2024-04-24T08:11:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.393+0000 7f3f59e00640 1 -- 172.21.15.28:0/3368876226 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f5419b610 con 0x7f3f541093f0 2024-04-24T08:11:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.393+0000 7f3f51600640 1 -- 172.21.15.28:0/3368876226 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f40031c90 con 0x7f3f541093f0 2024-04-24T08:11:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.393+0000 7f3f51600640 1 -- 172.21.15.28:0/3368876226 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40035030 con 0x7f3f541093f0 2024-04-24T08:11:36.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.393+0000 7f3f59e00640 1 -- 172.21.15.28:0/3368876226 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f541a77a0 con 0x7f3f541093f0 2024-04-24T08:11:36.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.397+0000 7f3f51600640 1 -- 172.21.15.28:0/3368876226 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 26) ==== 138930+0+0 (secure 0 0 0) 0x7f3f40042070 con 0x7f3f541093f0 2024-04-24T08:11:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.397+0000 7f3f5e223740 1 Processor -- start 2024-04-24T08:11:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.397+0000 7f3f5e223740 1 -- start start 2024-04-24T08:11:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.398+0000 7f3f5e223740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe084a30 0x55a8fe084e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.398+0000 7f3f5e223740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe085350 0x55a8fe08f4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.398+0000 7f3f5e223740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe08fa20 0x55a8fe091e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.398+0000 7f3f5e223740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a8fdfea1f0 con 0x55a8fe08fa20 2024-04-24T08:11:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.398+0000 7f3f5e223740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a8fdf90350 con 0x55a8fe084a30 2024-04-24T08:11:36.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.398+0000 7f3f5e223740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a8fdfe61b0 con 0x55a8fe085350 2024-04-24T08:11:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.398+0000 7f3f58a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe085350 0x55a8fe08f4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.398+0000 7f3f59400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe084a30 0x55a8fe084e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.398+0000 7f3f58a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe085350 0x55a8fe08f4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60270/0 (socket says 172.21.15.28:60270) 2024-04-24T08:11:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.398+0000 7f3f53e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe08fa20 0x55a8fe091e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.398+0000 7f3f58a00640 1 -- 172.21.15.28:0/3545450410 learned_addr learned my addr 172.21.15.28:0/3545450410 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.398+0000 7f3f59400640 1 -- 172.21.15.28:0/3545450410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe085350 msgr2=0x55a8fe08f4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.398+0000 7f3f59400640 1 --2- 172.21.15.28:0/3545450410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe085350 0x55a8fe08f4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.398+0000 7f3f59400640 1 -- 172.21.15.28:0/3545450410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe08fa20 msgr2=0x55a8fe091e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.399+0000 7f3f59400640 1 --2- 172.21.15.28:0/3545450410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe08fa20 0x55a8fe091e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.399+0000 7f3f59400640 1 -- 172.21.15.28:0/3545450410 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8fdfe4f50 con 0x55a8fe084a30 2024-04-24T08:11:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.399+0000 7f3f51600640 1 --2- 172.21.15.28:0/3368876226 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3f3c0a3ed0 0x7f3f3c0a6390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.399+0000 7f3f51600640 1 -- 172.21.15.28:0/3368876226 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f3f400f1120 con 0x7f3f541093f0 2024-04-24T08:11:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.399+0000 7f3f59400640 1 --2- 172.21.15.28:0/3545450410 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe084a30 0x55a8fe084e10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c009f00 tx=0x7f3f4c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d915fcd580bfc561 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.399+0000 7f3f45c00640 1 -- 172.21.15.28:0/3545450410 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c004d20 con 0x55a8fe084a30 2024-04-24T08:11:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.399+0000 7f3f58a00640 1 --2- 172.21.15.28:0/3368876226 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3f3c0a3ed0 0x7f3f3c0a6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.399+0000 7f3f45c00640 1 -- 172.21.15.28:0/3545450410 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f4c004050 con 0x55a8fe084a30 2024-04-24T08:11:36.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.399+0000 7f3f45c00640 1 -- 172.21.15.28:0/3545450410 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4c005280 con 0x55a8fe084a30 2024-04-24T08:11:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.400+0000 7f3f5e223740 1 -- 172.21.15.28:0/3545450410 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe084a30 msgr2=0x55a8fe084e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.400+0000 7f3f5e223740 1 --2- 172.21.15.28:0/3545450410 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe084a30 0x55a8fe084e10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f3f4c009f00 tx=0x7f3f4c011810 comp rx=0 tx=0).stop 2024-04-24T08:11:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.400+0000 7f3f58a00640 1 --2- 172.21.15.28:0/3368876226 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3f3c0a3ed0 0x7f3f3c0a6390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x55a8fe085ff0 tx=0x7f3f48005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.400+0000 7f3f5e223740 1 -- 172.21.15.28:0/3545450410 shutdown_connections 2024-04-24T08:11:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.400+0000 7f3f5e223740 1 --2- 172.21.15.28:0/3545450410 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe08fa20 0x55a8fe091e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.400+0000 7f3f5e223740 1 --2- 172.21.15.28:0/3545450410 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe085350 0x55a8fe08f4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.400+0000 7f3f5e223740 1 --2- 172.21.15.28:0/3545450410 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe084a30 0x55a8fe084e10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.400+0000 7f3f5e223740 1 -- 172.21.15.28:0/3545450410 >> 172.21.15.28:0/3545450410 conn(0x55a8fe080d50 msgr2=0x55a8fe081130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.400+0000 7f3f5e223740 1 -- 172.21.15.28:0/3545450410 shutdown_connections 2024-04-24T08:11:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.400+0000 7f3f5e223740 1 -- 172.21.15.28:0/3545450410 wait complete. 2024-04-24T08:11:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.401+0000 7f3f5e223740 1 -- start start 2024-04-24T08:11:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.401+0000 7f3f5e223740 1 Processor -- start 2024-04-24T08:11:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.401+0000 7f3f5e223740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe084a30 0x55a8fe08ca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.401+0000 7f3f5e223740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe085350 0x55a8fe08cf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.401+0000 7f3f5e223740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe08fa20 0x55a8fe08d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.401+0000 7f3f5e223740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a8fe092810 con 0x55a8fe084a30 2024-04-24T08:11:36.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.401+0000 7f3f5e223740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a8fdfa3690 con 0x55a8fe085350 2024-04-24T08:11:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.401+0000 7f3f5e223740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a8fe092990 con 0x55a8fe08fa20 2024-04-24T08:11:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.402+0000 7f3f5e223740 10 client.0 _ll_register_callbacks cb 0x55a8fdfdf650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.402+0000 7f3f58a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe084a30 0x55a8fe08ca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.402+0000 7f3f58a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe084a30 0x55a8fe08ca40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57576/0 (socket says 172.21.15.28:57576) 2024-04-24T08:11:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.402+0000 7f3f53e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe085350 0x55a8fe08cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.402+0000 7f3f58a00640 1 -- 172.21.15.28:0/1263089852 learned_addr learned my addr 172.21.15.28:0/1263089852 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.402+0000 7f3f59400640 1 --2- 172.21.15.28:0/1263089852 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe08fa20 0x55a8fe08d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.402+0000 7f3f53e00640 1 -- 172.21.15.28:0/1263089852 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe08fa20 msgr2=0x55a8fe08d4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.402+0000 7f3f53e00640 1 --2- 172.21.15.28:0/1263089852 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe08fa20 0x55a8fe08d4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.402+0000 7f3f53e00640 1 -- 172.21.15.28:0/1263089852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe084a30 msgr2=0x55a8fe08ca40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.402+0000 7f3f53e00640 1 --2- 172.21.15.28:0/1263089852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe084a30 0x55a8fe08ca40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.402+0000 7f3f53e00640 1 -- 172.21.15.28:0/1263089852 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8fdfa2920 con 0x55a8fe085350 2024-04-24T08:11:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.403+0000 7f3f53e00640 1 --2- 172.21.15.28:0/1263089852 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe085350 0x55a8fe08cf80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3f400ee610 tx=0x7f3f40031f60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=79cc63dd4c30fbb5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.403+0000 7f3f3a000640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:11:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.403+0000 7f3f3a000640 1 -- 172.21.15.28:0/1263089852 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f40049070 con 0x55a8fe085350 2024-04-24T08:11:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.403+0000 7f3f3a000640 1 -- 172.21.15.28:0/1263089852 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f3f400bd170 con 0x55a8fe085350 2024-04-24T08:11:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.403+0000 7f3f5e223740 1 -- 172.21.15.28:0/1263089852 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f4c004050 con 0x55a8fe085350 2024-04-24T08:11:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.403+0000 7f3f3a000640 1 -- 172.21.15.28:0/1263089852 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3f400b91b0 con 0x55a8fe085350 2024-04-24T08:11:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.403+0000 7f3f5e223740 10 client.25664 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.403+0000 7f3f5e223740 1 -- 172.21.15.28:0/1263089852 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a8fddc67d0 con 0x55a8fe085350 2024-04-24T08:11:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.403+0000 7f3f3a000640 1 -- 172.21.15.28:0/1263089852 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f4c004050 con 0x55a8fe085350 2024-04-24T08:11:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.404+0000 7f3f3a000640 1 -- 172.21.15.28:0/1263089852 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3f400c5640 con 0x55a8fe085350 2024-04-24T08:11:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.404+0000 7f3f5e223740 10 client.25664 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.404+0000 7f3f5e223740 10 client.25664 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:11:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.404+0000 7f3f5e223740 20 client.25664 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.404+0000 7f3f5e223740 1 --2- 172.21.15.28:0/1263089852 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a8fe09fcc0 0x55a8fe0a20f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.404+0000 7f3f5e223740 4 client.25664 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:11:36.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.404+0000 7f3f5e223740 1 -- 172.21.15.28:0/1263089852 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8fdff7c10 con 0x55a8fe09fcc0 2024-04-24T08:11:36.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.404+0000 7f3f58a00640 1 --2- 172.21.15.28:0/1263089852 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a8fe09fcc0 0x55a8fe0a20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.405+0000 7f3f58a00640 1 --2- 172.21.15.28:0/1263089852 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a8fe09fcc0 0x55a8fe0a20f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be7a3262971e0a31 server_cookie=f4df2c40fc2bdb1a in_seq=0 out_seq=0 2024-04-24T08:11:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.405+0000 7f3f3a000640 10 client.25664 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.406+0000 7f3f3a000640 1 -- 172.21.15.28:0/1263089852 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a8fdff7c10 con 0x55a8fe09fcc0 2024-04-24T08:11:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.406+0000 7f3f3a000640 10 client.25664 handle_command_reply: tid=0 2024-04-24T08:11:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.406+0000 7f3f5e223740 1 client.25664 shutdown 2024-04-24T08:11:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.406+0000 7f3f5e223740 1 -- 172.21.15.28:0/1263089852 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe085350 msgr2=0x55a8fe08cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.406+0000 7f3f5e223740 1 --2- 172.21.15.28:0/1263089852 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe085350 0x55a8fe08cf80 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f3f400ee610 tx=0x7f3f40031f60 comp rx=0 tx=0).stop 2024-04-24T08:11:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.406+0000 7f3f5e223740 1 -- 172.21.15.28:0/1263089852 shutdown_connections 2024-04-24T08:11:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.406+0000 7f3f5e223740 1 --2- 172.21.15.28:0/1263089852 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a8fe09fcc0 0x55a8fe0a20f0 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.406+0000 7f3f5e223740 1 --2- 172.21.15.28:0/1263089852 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe08fa20 0x55a8fe08d4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.406+0000 7f3f5e223740 1 --2- 172.21.15.28:0/1263089852 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe085350 0x55a8fe08cf80 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.406+0000 7f3f3a000640 0 client.25664 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.406+0000 7f3f5e223740 1 --2- 172.21.15.28:0/1263089852 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe084a30 0x55a8fe08ca40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.406+0000 7f3f5e223740 1 -- 172.21.15.28:0/1263089852 >> 172.21.15.28:0/1263089852 conn(0x55a8fe080d50 msgr2=0x55a8fe086b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.406+0000 7f3f5e223740 1 -- 172.21.15.28:0/1263089852 shutdown_connections 2024-04-24T08:11:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.406+0000 7f3f5e223740 1 -- 172.21.15.28:0/1263089852 wait complete. 2024-04-24T08:11:36.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.406+0000 7f3f5e223740 20 client.25664 trim_cache size 0 max 16384 2024-04-24T08:11:36.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.434+0000 7f3f5e223740 1 Processor -- start 2024-04-24T08:11:36.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.434+0000 7f3f5e223740 1 -- start start 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.434+0000 7f3f5e223740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe084a30 0x55a8fe09b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.434+0000 7f3f5e223740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe085350 0x55a8fe0872a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.434+0000 7f3f5e223740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe08fa20 0x55a8fe0a79c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.434+0000 7f3f5e223740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541b2670 con 0x55a8fe084a30 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.434+0000 7f3f5e223740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541aff40 con 0x55a8fe08fa20 2024-04-24T08:11:36.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.434+0000 7f3f5e223740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541add90 con 0x55a8fe085350 2024-04-24T08:11:36.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.435+0000 7f3f58a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe084a30 0x55a8fe09b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.435+0000 7f3f58a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe084a30 0x55a8fe09b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57582/0 (socket says 172.21.15.28:57582) 2024-04-24T08:11:36.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.435+0000 7f3f58a00640 1 -- 172.21.15.28:0/2154489436 learned_addr learned my addr 172.21.15.28:0/2154489436 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:36.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.435+0000 7f3f53e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe085350 0x55a8fe0872a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.435+0000 7f3f59400640 1 --2- 172.21.15.28:0/2154489436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe08fa20 0x55a8fe0a79c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.435+0000 7f3f58a00640 1 -- 172.21.15.28:0/2154489436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe085350 msgr2=0x55a8fe0872a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.435+0000 7f3f58a00640 1 --2- 172.21.15.28:0/2154489436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe085350 0x55a8fe0872a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.435+0000 7f3f58a00640 1 -- 172.21.15.28:0/2154489436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe08fa20 msgr2=0x55a8fe0a79c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.435+0000 7f3f58a00640 1 --2- 172.21.15.28:0/2154489436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe08fa20 0x55a8fe0a79c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.435+0000 7f3f58a00640 1 -- 172.21.15.28:0/2154489436 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8fdcfc3f0 con 0x55a8fe084a30 2024-04-24T08:11:36.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.435+0000 7f3f58a00640 1 --2- 172.21.15.28:0/2154489436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe084a30 0x55a8fe09b6d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x55a8fe0a2900 tx=0x7f3f4801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa7590858ef3ea22 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.436+0000 7f3f45200640 1 -- 172.21.15.28:0/2154489436 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f4802b420 con 0x55a8fe084a30 2024-04-24T08:11:36.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.436+0000 7f3f45200640 1 -- 172.21.15.28:0/2154489436 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f48005020 con 0x55a8fe084a30 2024-04-24T08:11:36.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.436+0000 7f3f45200640 1 -- 172.21.15.28:0/2154489436 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f480048b0 con 0x55a8fe084a30 2024-04-24T08:11:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.436+0000 7f3f5e223740 1 -- 172.21.15.28:0/2154489436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe084a30 msgr2=0x55a8fe09b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.436+0000 7f3f5e223740 1 --2- 172.21.15.28:0/2154489436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe084a30 0x55a8fe09b6d0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x55a8fe0a2900 tx=0x7f3f4801a000 comp rx=0 tx=0).stop 2024-04-24T08:11:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.436+0000 7f3f5e223740 1 -- 172.21.15.28:0/2154489436 shutdown_connections 2024-04-24T08:11:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.436+0000 7f3f5e223740 1 --2- 172.21.15.28:0/2154489436 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe08fa20 0x55a8fe0a79c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.436+0000 7f3f5e223740 1 --2- 172.21.15.28:0/2154489436 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe085350 0x55a8fe0872a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.436+0000 7f3f5e223740 1 --2- 172.21.15.28:0/2154489436 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe084a30 0x55a8fe09b6d0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.436+0000 7f3f5e223740 1 -- 172.21.15.28:0/2154489436 >> 172.21.15.28:0/2154489436 conn(0x55a8fe080d50 msgr2=0x55a8fe08d290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.437+0000 7f3f5e223740 1 -- 172.21.15.28:0/2154489436 shutdown_connections 2024-04-24T08:11:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.437+0000 7f3f5e223740 1 -- 172.21.15.28:0/2154489436 wait complete. 2024-04-24T08:11:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.437+0000 7f3f5e223740 1 -- start start 2024-04-24T08:11:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.437+0000 7f3f5e223740 1 Processor -- start 2024-04-24T08:11:36.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f5e223740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe084a30 0x55a8fe097fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f5e223740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe085350 0x55a8fe093340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f5e223740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe08fa20 0x55a8fe093880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f5e223740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541ae400 con 0x55a8fe085350 2024-04-24T08:11:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f5e223740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541af870 con 0x55a8fe08fa20 2024-04-24T08:11:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f5e223740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3f541aead0 con 0x55a8fe084a30 2024-04-24T08:11:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f5e223740 10 client.0 _ll_register_callbacks cb 0x7f3f40005de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:11:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f53e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe085350 0x55a8fe093340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f58a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe084a30 0x55a8fe097fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f59400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe08fa20 0x55a8fe093880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f53e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe085350 0x55a8fe093340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57586/0 (socket says 172.21.15.28:57586) 2024-04-24T08:11:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f53e00640 1 -- 172.21.15.28:0/403210368 learned_addr learned my addr 172.21.15.28:0/403210368 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:11:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f58a00640 1 -- 172.21.15.28:0/403210368 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe08fa20 msgr2=0x55a8fe093880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f58a00640 1 --2- 172.21.15.28:0/403210368 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe08fa20 0x55a8fe093880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f58a00640 1 -- 172.21.15.28:0/403210368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe085350 msgr2=0x55a8fe093340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:11:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f58a00640 1 --2- 172.21.15.28:0/403210368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe085350 0x55a8fe093340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f58a00640 1 -- 172.21.15.28:0/403210368 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8fdcbc0f0 con 0x55a8fe084a30 2024-04-24T08:11:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f59400640 1 --2- 172.21.15.28:0/403210368 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe08fa20 0x55a8fe093880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.438+0000 7f3f53e00640 1 --2- 172.21.15.28:0/403210368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe085350 0x55a8fe093340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:11:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.439+0000 7f3f58a00640 1 --2- 172.21.15.28:0/403210368 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe084a30 0x55a8fe097fd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3f48012450 tx=0x7f3f480046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=57086bce779b3510 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:11:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.439+0000 7f3f3a000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:11:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.439+0000 7f3f3a000640 1 -- 172.21.15.28:0/403210368 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f48039070 con 0x55a8fe084a30 2024-04-24T08:11:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.439+0000 7f3f5e223740 1 -- 172.21.15.28:0/403210368 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3f48005020 con 0x55a8fe084a30 2024-04-24T08:11:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.439+0000 7f3f3a000640 1 -- 172.21.15.28:0/403210368 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f3f480051d0 con 0x55a8fe084a30 2024-04-24T08:11:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.440+0000 7f3f3a000640 1 -- 172.21.15.28:0/403210368 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f3f48045050 con 0x55a8fe084a30 2024-04-24T08:11:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.440+0000 7f3f5e223740 10 client.25708 fetch_fsmap learned FSMap version 24 2024-04-24T08:11:36.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.440+0000 7f3f5e223740 1 -- 172.21.15.28:0/403210368 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a8fdfaf810 con 0x55a8fe084a30 2024-04-24T08:11:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.440+0000 7f3f3a000640 1 -- 172.21.15.28:0/403210368 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f48005020 con 0x55a8fe084a30 2024-04-24T08:11:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.440+0000 7f3f3a000640 1 -- 172.21.15.28:0/403210368 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f3f4802b720 con 0x55a8fe084a30 2024-04-24T08:11:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.440+0000 7f3f5e223740 10 client.25708 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:11:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.440+0000 7f3f5e223740 10 client.25708 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:11:36.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.440+0000 7f3f5e223740 20 client.25708 populate_metadata read hostname 'smithi028' 2024-04-24T08:11:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.440+0000 7f3f5e223740 1 --2- 172.21.15.28:0/403210368 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a8fe1065d0 0x55a8fe1089f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:11:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.440+0000 7f3f5e223740 4 client.25708 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:11:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.440+0000 7f3f5e223740 1 -- 172.21.15.28:0/403210368 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a8fdff1a80 con 0x55a8fe1065d0 2024-04-24T08:11:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.441+0000 7f3f53e00640 1 --2- 172.21.15.28:0/403210368 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a8fe1065d0 0x55a8fe1089f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:11:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.441+0000 7f3f3a000640 10 client.25708 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.441+0000 7f3f53e00640 1 --2- 172.21.15.28:0/403210368 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a8fe1065d0 0x55a8fe1089f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c730fdb325ff4b8d server_cookie=925828b06a50762d in_seq=0 out_seq=0 2024-04-24T08:11:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.442+0000 7f3f3a000640 1 -- 172.21.15.28:0/403210368 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55a8fdff1a80 con 0x55a8fe1065d0 2024-04-24T08:11:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.442+0000 7f3f3a000640 10 client.25708 handle_command_reply: tid=0 2024-04-24T08:11:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.442+0000 7f3f5e223740 1 client.25708 shutdown 2024-04-24T08:11:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.442+0000 7f3f5e223740 1 -- 172.21.15.28:0/403210368 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe084a30 msgr2=0x55a8fe097fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.442+0000 7f3f5e223740 1 --2- 172.21.15.28:0/403210368 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe084a30 0x55a8fe097fd0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f3f48012450 tx=0x7f3f480046c0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.442+0000 7f3f5e223740 1 -- 172.21.15.28:0/403210368 shutdown_connections 2024-04-24T08:11:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.442+0000 7f3f5e223740 1 --2- 172.21.15.28:0/403210368 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a8fe1065d0 0x55a8fe1089f0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.442+0000 7f3f5e223740 1 --2- 172.21.15.28:0/403210368 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a8fe085350 0x55a8fe093340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.442+0000 7f3f5e223740 1 --2- 172.21.15.28:0/403210368 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a8fe08fa20 0x55a8fe093880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.442+0000 7f3f5e223740 1 --2- 172.21.15.28:0/403210368 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a8fe084a30 0x55a8fe097fd0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.442+0000 7f3f3a000640 0 client.25708 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:11:36.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.442+0000 7f3f5e223740 1 -- 172.21.15.28:0/403210368 >> 172.21.15.28:0/403210368 conn(0x55a8fe080d50 msgr2=0x55a8fe07f740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:36.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.442+0000 7f3f5e223740 1 -- 172.21.15.28:0/403210368 shutdown_connections 2024-04-24T08:11:36.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.442+0000 7f3f5e223740 1 -- 172.21.15.28:0/403210368 wait complete. 2024-04-24T08:11:36.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.442+0000 7f3f5e223740 20 client.25708 trim_cache size 0 max 16384 2024-04-24T08:11:36.446 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:11:36.446 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:11:36.446 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:11:36.446 INFO:teuthology.orchestra.run.smithi028.stdout: "cea9db14-51df-4d18-b27b-8d8d49df0562": { 2024-04-24T08:11:36.447 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:11:36.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "cea9db14-51df-4d18-b27b-8d8d49df0562", 2024-04-24T08:11:36.447 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:11:36.447 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:11:36.447 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:11:36.447 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:11:36.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.443+0000 7f3f59e00640 1 -- 172.21.15.28:0/3368876226 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3f3c0a3ed0 msgr2=0x7f3f3c0a6390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.443+0000 7f3f59e00640 1 --2- 172.21.15.28:0/3368876226 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3f3c0a3ed0 0x7f3f3c0a6390 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x55a8fe085ff0 tx=0x7f3f48005f90 comp rx=0 tx=0).stop 2024-04-24T08:11:36.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.443+0000 7f3f59e00640 1 -- 172.21.15.28:0/3368876226 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f541093f0 msgr2=0x7f3f541a0bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:11:36.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.443+0000 7f3f59e00640 1 --2- 172.21.15.28:0/3368876226 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f541093f0 0x7f3f541a0bc0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f3f4002a910 tx=0x7f3f40003b20 comp rx=0 tx=0).stop 2024-04-24T08:11:36.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.443+0000 7f3f59e00640 1 -- 172.21.15.28:0/3368876226 shutdown_connections 2024-04-24T08:11:36.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.443+0000 7f3f59e00640 1 --2- 172.21.15.28:0/3368876226 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3f3c0a3ed0 0x7f3f3c0a6390 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.443+0000 7f3f59e00640 1 --2- 172.21.15.28:0/3368876226 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f5419a7f0 0x7f3f5419aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.443+0000 7f3f59e00640 1 --2- 172.21.15.28:0/3368876226 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3f541093f0 0x7f3f541a0bc0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.443+0000 7f3f59e00640 1 --2- 172.21.15.28:0/3368876226 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3f54104320 0x7f3f541a0680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:11:36.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.443+0000 7f3f59e00640 1 -- 172.21.15.28:0/3368876226 >> 172.21.15.28:0/3368876226 conn(0x7f3f540ffb40 msgr2=0x7f3f54101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:11:36.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.443+0000 7f3f59e00640 1 -- 172.21.15.28:0/3368876226 shutdown_connections 2024-04-24T08:11:36.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:11:36.443+0000 7f3f59e00640 1 -- 172.21.15.28:0/3368876226 wait complete. 2024-04-24T08:11:36.461 INFO:tasks.cephfs.filesystem:scrub status for tag:cea9db14-51df-4d18-b27b-8d8d49df0562 - {'path': '/', 'tag': 'cea9db14-51df-4d18-b27b-8d8d49df0562', 'options': 'recursive,force'} 2024-04-24T08:11:37.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:37 smithi028 ceph-mon[27786]: scrub queued for path: / 2024-04-24T08:11:37.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:37 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:11:37.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:37 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:11:37.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:37 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:11:37.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:37 smithi028 ceph-mon[27786]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:11:37.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:37 smithi117 ceph-mon[32438]: scrub queued for path: / 2024-04-24T08:11:37.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:37 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:11:37.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:37 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:11:37.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:37 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:11:37.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:37 smithi117 ceph-mon[32438]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:11:37.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:37 smithi194 ceph-mon[32836]: scrub queued for path: / 2024-04-24T08:11:37.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:37 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:11:37.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:37 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:11:37.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:37 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:11:37.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:37 smithi194 ceph-mon[32836]: pgmap v41: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-04-24T08:11:39.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:39 smithi028 ceph-mon[27786]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:39.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:39 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:11:39.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:39 smithi117 ceph-mon[32438]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:39.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:39 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:11:39.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:39 smithi194 ceph-mon[32836]: pgmap v42: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:39.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:39 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:11:41.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:41 smithi028 ceph-mon[27786]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:41.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:41 smithi117 ceph-mon[32438]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:41.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:41 smithi194 ceph-mon[32836]: pgmap v43: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-04-24T08:11:43.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:43 smithi028 ceph-mon[27786]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:11:43.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:43 smithi117 ceph-mon[32438]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:11:43.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:43 smithi194 ceph-mon[32836]: pgmap v44: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:11:45.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:45 smithi028 ceph-mon[27786]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:11:45.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:45 smithi117 ceph-mon[32438]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:11:45.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:45 smithi194 ceph-mon[32836]: pgmap v45: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:11:47.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:47 smithi028 ceph-mon[27786]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:11:47.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:47 smithi117 ceph-mon[32438]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:11:47.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:47 smithi194 ceph-mon[32836]: pgmap v46: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-04-24T08:11:49.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:49 smithi028 ceph-mon[27786]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:11:49.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:49 smithi117 ceph-mon[32438]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:11:49.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:49 smithi194 ceph-mon[32836]: pgmap v47: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:11:51.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:51 smithi028 ceph-mon[27786]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:11:51.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:51 smithi117 ceph-mon[32438]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:11:51.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:51 smithi194 ceph-mon[32836]: pgmap v48: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:11:53.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:53 smithi117 ceph-mon[32438]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-24T08:11:53.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:53 smithi117 ceph-mon[32438]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-04-24T08:11:53.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:53 smithi194 ceph-mon[32836]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-24T08:11:53.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:53 smithi194 ceph-mon[32836]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-04-24T08:11:53.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:53 smithi028 ceph-mon[27786]: pgmap v49: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 938 B/s wr, 9 op/s 2024-04-24T08:11:53.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:53 smithi028 ceph-mon[27786]: mgrmap e27: x(active, since 92s), standbys: y, z 2024-04-24T08:11:55.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:55 smithi117 ceph-mon[32438]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-24T08:11:55.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:55 smithi194 ceph-mon[32836]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-24T08:11:55.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:55 smithi028 ceph-mon[27786]: pgmap v50: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-24T08:11:57.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:57 smithi117 ceph-mon[32438]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-24T08:11:57.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:57 smithi194 ceph-mon[32836]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-24T08:11:57.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:57 smithi028 ceph-mon[27786]: pgmap v51: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 938 B/s wr, 8 op/s 2024-04-24T08:11:59.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:11:59 smithi117 ceph-mon[32438]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-24T08:11:59.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:11:59 smithi194 ceph-mon[32836]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-24T08:11:59.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:11:59 smithi028 ceph-mon[27786]: pgmap v52: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-24T08:12:01.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:01 smithi117 ceph-mon[32438]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-24T08:12:01.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:01 smithi194 ceph-mon[32836]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-24T08:12:01.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:01 smithi028 ceph-mon[27786]: pgmap v53: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 938 B/s wr, 6 op/s 2024-04-24T08:12:03.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:03 smithi117 ceph-mon[32438]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-24T08:12:03.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:03 smithi194 ceph-mon[32836]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-24T08:12:03.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:03 smithi028 ceph-mon[27786]: pgmap v54: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 23 KiB/s wr, 23 op/s 2024-04-24T08:12:05.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:05 smithi117 ceph-mon[32438]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:12:05.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:05 smithi194 ceph-mon[32836]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:12:05.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:05 smithi028 ceph-mon[27786]: pgmap v55: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:12:06.462 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:12:06.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:12:06.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.736+0000 7f6631e00640 1 -- 172.21.15.28:0/1416644550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c103980 msgr2=0x7f662c10be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:06.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.736+0000 7f6631e00640 1 --2- 172.21.15.28:0/1416644550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c103980 0x7f662c10be60 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f661800fb10 tx=0x7f661802e600 comp rx=0 tx=0).stop 2024-04-24T08:12:06.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.736+0000 7f6631e00640 1 -- 172.21.15.28:0/1416644550 shutdown_connections 2024-04-24T08:12:06.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.736+0000 7f6631e00640 1 --2- 172.21.15.28:0/1416644550 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c103980 0x7f662c10be60 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.737+0000 7f6631e00640 1 --2- 172.21.15.28:0/1416644550 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f662c10c480 0x7f662c10e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.737+0000 7f6631e00640 1 --2- 172.21.15.28:0/1416644550 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f662c103060 0x7f662c103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.737+0000 7f6631e00640 1 -- 172.21.15.28:0/1416644550 >> 172.21.15.28:0/1416644550 conn(0x7f662c0fc880 msgr2=0x7f662c0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:06.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.737+0000 7f6631e00640 1 -- 172.21.15.28:0/1416644550 shutdown_connections 2024-04-24T08:12:06.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.737+0000 7f6631e00640 1 -- 172.21.15.28:0/1416644550 wait complete. 2024-04-24T08:12:06.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.738+0000 7f6631e00640 1 Processor -- start 2024-04-24T08:12:06.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.738+0000 7f6631e00640 1 -- start start 2024-04-24T08:12:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.739+0000 7f6631e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f662c103060 0x7f662c1965d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.739+0000 7f6631e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c103980 0x7f662c196b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.739+0000 7f6631e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f662c10c480 0x7f662c19ae40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.739+0000 7f6631e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f662c110070 con 0x7f662c103980 2024-04-24T08:12:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.739+0000 7f6631e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f662c10fef0 con 0x7f662c103060 2024-04-24T08:12:06.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.739+0000 7f6631e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f662c1101f0 con 0x7f662c10c480 2024-04-24T08:12:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.739+0000 7f6630a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f662c103060 0x7f662c1965d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.739+0000 7f662be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c103980 0x7f662c196b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.739+0000 7f6630a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f662c103060 0x7f662c1965d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40242/0 (socket says 172.21.15.28:40242) 2024-04-24T08:12:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.739+0000 7f6630a00640 1 -- 172.21.15.28:0/2421024146 learned_addr learned my addr 172.21.15.28:0/2421024146 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.740+0000 7f6631400640 1 --2- 172.21.15.28:0/2421024146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f662c10c480 0x7f662c19ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.740+0000 7f6630a00640 1 -- 172.21.15.28:0/2421024146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f662c10c480 msgr2=0x7f662c19ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.740+0000 7f6630a00640 1 --2- 172.21.15.28:0/2421024146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f662c10c480 0x7f662c19ae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.740+0000 7f6630a00640 1 -- 172.21.15.28:0/2421024146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c103980 msgr2=0x7f662c196b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.740+0000 7f6630a00640 1 --2- 172.21.15.28:0/2421024146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c103980 0x7f662c196b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.740+0000 7f6630a00640 1 -- 172.21.15.28:0/2421024146 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f662c19b570 con 0x7f662c103060 2024-04-24T08:12:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.740+0000 7f662be00640 1 --2- 172.21.15.28:0/2421024146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c103980 0x7f662c196b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.740+0000 7f6630a00640 1 --2- 172.21.15.28:0/2421024146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f662c103060 0x7f662c1965d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f662000ea60 tx=0x7f662000ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.741+0000 7f6629600640 1 -- 172.21.15.28:0/2421024146 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f662000cde0 con 0x7f662c103060 2024-04-24T08:12:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.741+0000 7f6631e00640 1 -- 172.21.15.28:0/2421024146 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f662c19b860 con 0x7f662c103060 2024-04-24T08:12:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.741+0000 7f6629600640 1 -- 172.21.15.28:0/2421024146 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f66200035f0 con 0x7f662c103060 2024-04-24T08:12:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.741+0000 7f6629600640 1 -- 172.21.15.28:0/2421024146 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6620010670 con 0x7f662c103060 2024-04-24T08:12:06.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.741+0000 7f6631e00640 1 -- 172.21.15.28:0/2421024146 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f662c1a30f0 con 0x7f662c103060 2024-04-24T08:12:06.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.745+0000 7f66361e2740 1 Processor -- start 2024-04-24T08:12:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.745+0000 7f66361e2740 1 -- start start 2024-04-24T08:12:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.746+0000 7f66361e2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 0x55a2bfe75e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.746+0000 7f66361e2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe76350 0x55a2bfe804e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.746+0000 7f66361e2740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe80a20 0x55a2bfe82e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.746+0000 7f66361e2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a2bfddb1f0 con 0x55a2bfe75a30 2024-04-24T08:12:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.746+0000 7f6629600640 1 -- 172.21.15.28:0/2421024146 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f66200037a0 con 0x7f662c103060 2024-04-24T08:12:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.746+0000 7f66361e2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a2bfd81350 con 0x55a2bfe76350 2024-04-24T08:12:06.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.746+0000 7f66361e2740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a2bfdd71b0 con 0x55a2bfe80a20 2024-04-24T08:12:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.747+0000 7f6630a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe76350 0x55a2bfe804e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.747+0000 7f6631400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 0x55a2bfe75e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.747+0000 7f662be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe80a20 0x55a2bfe82e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.747+0000 7f6630a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe76350 0x55a2bfe804e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40250/0 (socket says 172.21.15.28:40250) 2024-04-24T08:12:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.747+0000 7f6630a00640 1 -- 172.21.15.28:0/3990306074 learned_addr learned my addr 172.21.15.28:0/3990306074 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.747+0000 7f662be00640 1 -- 172.21.15.28:0/3990306074 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe76350 msgr2=0x55a2bfe804e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.747+0000 7f662be00640 1 --2- 172.21.15.28:0/3990306074 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe76350 0x55a2bfe804e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.747+0000 7f662be00640 1 -- 172.21.15.28:0/3990306074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 msgr2=0x55a2bfe75e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.747+0000 7f662be00640 1 --2- 172.21.15.28:0/3990306074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 0x55a2bfe75e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.747+0000 7f662be00640 1 -- 172.21.15.28:0/3990306074 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2bfdd5f50 con 0x55a2bfe80a20 2024-04-24T08:12:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.747+0000 7f6630a00640 1 --2- 172.21.15.28:0/3990306074 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe76350 0x55a2bfe804e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:06.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.747+0000 7f6631400640 1 --2- 172.21.15.28:0/3990306074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 0x55a2bfe75e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.748+0000 7f662be00640 1 --2- 172.21.15.28:0/3990306074 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe80a20 0x55a2bfe82e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6618009230 tx=0x7f66180046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=76d4031bd6e1fc3a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.748+0000 7f661dc00640 1 -- 172.21.15.28:0/3990306074 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6618004040 con 0x55a2bfe80a20 2024-04-24T08:12:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.748+0000 7f661dc00640 1 -- 172.21.15.28:0/3990306074 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f661800ee70 con 0x55a2bfe80a20 2024-04-24T08:12:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.748+0000 7f661dc00640 1 -- 172.21.15.28:0/3990306074 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661800f640 con 0x55a2bfe80a20 2024-04-24T08:12:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.748+0000 7f6629600640 1 --2- 172.21.15.28:0/2421024146 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f66140a3da0 0x7f66140a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.749+0000 7f6629600640 1 -- 172.21.15.28:0/2421024146 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f66200d0b90 con 0x7f662c103060 2024-04-24T08:12:06.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.749+0000 7f66361e2740 1 -- 172.21.15.28:0/3990306074 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe80a20 msgr2=0x55a2bfe82e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:06.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.749+0000 7f66361e2740 1 --2- 172.21.15.28:0/3990306074 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe80a20 0x55a2bfe82e10 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f6618009230 tx=0x7f66180046c0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.749+0000 7f6631400640 1 --2- 172.21.15.28:0/2421024146 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f66140a3da0 0x7f66140a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.751+0000 7f66361e2740 1 -- 172.21.15.28:0/3990306074 shutdown_connections 2024-04-24T08:12:06.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.751+0000 7f6631400640 1 --2- 172.21.15.28:0/2421024146 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f66140a3da0 0x7f66140a6260 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6624009fd0 tx=0x7f6624009500 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:06.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.751+0000 7f66361e2740 1 --2- 172.21.15.28:0/3990306074 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe80a20 0x55a2bfe82e10 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.751+0000 7f66361e2740 1 --2- 172.21.15.28:0/3990306074 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe76350 0x55a2bfe804e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.751+0000 7f66361e2740 1 --2- 172.21.15.28:0/3990306074 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 0x55a2bfe75e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.751+0000 7f66361e2740 1 -- 172.21.15.28:0/3990306074 >> 172.21.15.28:0/3990306074 conn(0x55a2bfe71d50 msgr2=0x55a2bfe72130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:06.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.751+0000 7f66361e2740 1 -- 172.21.15.28:0/3990306074 shutdown_connections 2024-04-24T08:12:06.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.751+0000 7f66361e2740 1 -- 172.21.15.28:0/3990306074 wait complete. 2024-04-24T08:12:06.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.752+0000 7f66361e2740 1 -- start start 2024-04-24T08:12:06.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.752+0000 7f66361e2740 1 Processor -- start 2024-04-24T08:12:06.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.753+0000 7f66361e2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe75a30 0x55a2bfe7d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.753+0000 7f66361e2740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe76350 0x55a2bfe7dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.753+0000 7f66361e2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe80a20 0x55a2bfe7e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.753+0000 7f66361e2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a2bfe837b0 con 0x55a2bfe80a20 2024-04-24T08:12:06.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.753+0000 7f66361e2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a2bfd94690 con 0x55a2bfe75a30 2024-04-24T08:12:06.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.753+0000 7f66361e2740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a2bfe83930 con 0x55a2bfe76350 2024-04-24T08:12:06.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.753+0000 7f66361e2740 10 client.0 _ll_register_callbacks cb 0x55a2bfdd0650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:06.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.753+0000 7f662be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe76350 0x55a2bfe7dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.753+0000 7f662be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe76350 0x55a2bfe7dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:33354/0 (socket says 172.21.15.28:33354) 2024-04-24T08:12:06.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.753+0000 7f662be00640 1 -- 172.21.15.28:0/2936658704 learned_addr learned my addr 172.21.15.28:0/2936658704 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:06.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.753+0000 7f6630a00640 1 --2- 172.21.15.28:0/2936658704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe75a30 0x55a2bfe7d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.754+0000 7f662be00640 1 -- 172.21.15.28:0/2936658704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe75a30 msgr2=0x55a2bfe7d960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:06.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.754+0000 7f662be00640 1 --2- 172.21.15.28:0/2936658704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe75a30 0x55a2bfe7d960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.753+0000 7f6631400640 1 --2- 172.21.15.28:0/2936658704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe80a20 0x55a2bfe7e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.754+0000 7f662be00640 1 -- 172.21.15.28:0/2936658704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe80a20 msgr2=0x55a2bfe7e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:06.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.754+0000 7f662be00640 1 --2- 172.21.15.28:0/2936658704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe80a20 0x55a2bfe7e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.754+0000 7f662be00640 1 -- 172.21.15.28:0/2936658704 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2bfd93920 con 0x55a2bfe76350 2024-04-24T08:12:06.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.754+0000 7f6630a00640 1 --2- 172.21.15.28:0/2936658704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe75a30 0x55a2bfe7d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:06.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.754+0000 7f662be00640 1 --2- 172.21.15.28:0/2936658704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe76350 0x55a2bfe7dea0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6618006010 tx=0x7f6618004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f46abd68c5cb4833 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:06.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.754+0000 7f6612000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:12:06.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.754+0000 7f6612000640 1 -- 172.21.15.28:0/2936658704 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f661800e990 con 0x55a2bfe76350 2024-04-24T08:12:06.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.754+0000 7f6612000640 1 -- 172.21.15.28:0/2936658704 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f661800eb30 con 0x55a2bfe76350 2024-04-24T08:12:06.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.755+0000 7f66361e2740 1 -- 172.21.15.28:0/2936658704 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f661800ee70 con 0x55a2bfe76350 2024-04-24T08:12:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.755+0000 7f6612000640 1 -- 172.21.15.28:0/2936658704 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f661800f3c0 con 0x55a2bfe76350 2024-04-24T08:12:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.755+0000 7f66361e2740 10 client.25720 fetch_fsmap learned FSMap version 24 2024-04-24T08:12:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.755+0000 7f66361e2740 1 -- 172.21.15.28:0/2936658704 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a2bfbb77d0 con 0x55a2bfe76350 2024-04-24T08:12:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.756+0000 7f6612000640 1 -- 172.21.15.28:0/2936658704 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f661800ee70 con 0x55a2bfe76350 2024-04-24T08:12:06.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.756+0000 7f6612000640 1 -- 172.21.15.28:0/2936658704 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f66180038c0 con 0x55a2bfe76350 2024-04-24T08:12:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.756+0000 7f66361e2740 10 client.25720 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:12:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.756+0000 7f66361e2740 10 client.25720 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:12:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.756+0000 7f66361e2740 20 client.25720 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.756+0000 7f66361e2740 1 --2- 172.21.15.28:0/2936658704 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a2bfe99390 0x55a2bfe9b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.756+0000 7f66361e2740 4 client.25720 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.756+0000 7f66361e2740 1 -- 172.21.15.28:0/2936658704 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a2bfde8c10 con 0x55a2bfe99390 2024-04-24T08:12:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.757+0000 7f6630a00640 1 --2- 172.21.15.28:0/2936658704 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a2bfe99390 0x55a2bfe9b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.757+0000 7f6630a00640 1 --2- 172.21.15.28:0/2936658704 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a2bfe99390 0x55a2bfe9b780 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ddfd568d4e88e199 server_cookie=6dcfa1707da2f4e7 in_seq=0 out_seq=0 2024-04-24T08:12:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.757+0000 7f6612000640 10 client.25720 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:06.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.758+0000 7f6612000640 1 -- 172.21.15.28:0/2936658704 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a2bfde8c10 con 0x55a2bfe99390 2024-04-24T08:12:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.758+0000 7f6612000640 10 client.25720 handle_command_reply: tid=0 2024-04-24T08:12:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.758+0000 7f66361e2740 1 client.25720 shutdown 2024-04-24T08:12:06.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.759+0000 7f66361e2740 1 -- 172.21.15.28:0/2936658704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe76350 msgr2=0x55a2bfe7dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.759+0000 7f66361e2740 1 --2- 172.21.15.28:0/2936658704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe76350 0x55a2bfe7dea0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f6618006010 tx=0x7f6618004ab0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.759+0000 7f66361e2740 1 -- 172.21.15.28:0/2936658704 shutdown_connections 2024-04-24T08:12:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.759+0000 7f66361e2740 1 --2- 172.21.15.28:0/2936658704 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a2bfe99390 0x55a2bfe9b780 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.759+0000 7f66361e2740 1 --2- 172.21.15.28:0/2936658704 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe80a20 0x55a2bfe7e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.759+0000 7f66361e2740 1 --2- 172.21.15.28:0/2936658704 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe76350 0x55a2bfe7dea0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.759+0000 7f6612000640 0 client.25720 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.759+0000 7f66361e2740 1 --2- 172.21.15.28:0/2936658704 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe75a30 0x55a2bfe7d960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.759+0000 7f66361e2740 1 -- 172.21.15.28:0/2936658704 >> 172.21.15.28:0/2936658704 conn(0x55a2bfe71d50 msgr2=0x55a2bfe70fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.759+0000 7f66361e2740 1 -- 172.21.15.28:0/2936658704 shutdown_connections 2024-04-24T08:12:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.759+0000 7f66361e2740 1 -- 172.21.15.28:0/2936658704 wait complete. 2024-04-24T08:12:06.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.760+0000 7f66361e2740 20 client.25720 trim_cache size 0 max 16384 2024-04-24T08:12:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.797+0000 7f66361e2740 1 Processor -- start 2024-04-24T08:12:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.797+0000 7f66361e2740 1 -- start start 2024-04-24T08:12:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.797+0000 7f66361e2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 0x55a2bfe71750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.797+0000 7f66361e2740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe76350 0x55a2bfe8c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.797+0000 7f66361e2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe80a20 0x55a2bfe7e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.797+0000 7f66361e2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f662c1ae2b0 con 0x55a2bfe75a30 2024-04-24T08:12:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.797+0000 7f66361e2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f662c1abb80 con 0x55a2bfe80a20 2024-04-24T08:12:06.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.797+0000 7f66361e2740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f662c1a99d0 con 0x55a2bfe76350 2024-04-24T08:12:06.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.797+0000 7f6630a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 0x55a2bfe71750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.798+0000 7f6631400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe80a20 0x55a2bfe7e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.798+0000 7f6630a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 0x55a2bfe71750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51568/0 (socket says 172.21.15.28:51568) 2024-04-24T08:12:06.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.798+0000 7f6630a00640 1 -- 172.21.15.28:0/1886970005 learned_addr learned my addr 172.21.15.28:0/1886970005 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:06.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.798+0000 7f662be00640 1 --2- 172.21.15.28:0/1886970005 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe76350 0x55a2bfe8c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.798+0000 7f6630a00640 1 -- 172.21.15.28:0/1886970005 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe76350 msgr2=0x55a2bfe8c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:06.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.798+0000 7f6630a00640 1 --2- 172.21.15.28:0/1886970005 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe76350 0x55a2bfe8c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.798+0000 7f6630a00640 1 -- 172.21.15.28:0/1886970005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe80a20 msgr2=0x55a2bfe7e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.798+0000 7f6630a00640 1 --2- 172.21.15.28:0/1886970005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe80a20 0x55a2bfe7e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.798+0000 7f6630a00640 1 -- 172.21.15.28:0/1886970005 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a2bfaed3f0 con 0x55a2bfe75a30 2024-04-24T08:12:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.798+0000 7f662be00640 1 --2- 172.21.15.28:0/1886970005 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe76350 0x55a2bfe8c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.798+0000 7f6631400640 1 --2- 172.21.15.28:0/1886970005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe80a20 0x55a2bfe7e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.799+0000 7f6630a00640 1 --2- 172.21.15.28:0/1886970005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 0x55a2bfe71750 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6620098ec0 tx=0x7f66200d2fe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27aeb1b1dd6dddcf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.799+0000 7f661d200640 1 -- 172.21.15.28:0/1886970005 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66200d5040 con 0x55a2bfe75a30 2024-04-24T08:12:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.799+0000 7f661d200640 1 -- 172.21.15.28:0/1886970005 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f662009cab0 con 0x55a2bfe75a30 2024-04-24T08:12:06.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.799+0000 7f661d200640 1 -- 172.21.15.28:0/1886970005 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6620099120 con 0x55a2bfe75a30 2024-04-24T08:12:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.800+0000 7f66361e2740 1 -- 172.21.15.28:0/1886970005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 msgr2=0x55a2bfe71750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.800+0000 7f66361e2740 1 --2- 172.21.15.28:0/1886970005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 0x55a2bfe71750 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f6620098ec0 tx=0x7f66200d2fe0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.800+0000 7f66361e2740 1 -- 172.21.15.28:0/1886970005 shutdown_connections 2024-04-24T08:12:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.800+0000 7f66361e2740 1 --2- 172.21.15.28:0/1886970005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe80a20 0x55a2bfe7e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.800+0000 7f66361e2740 1 --2- 172.21.15.28:0/1886970005 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe76350 0x55a2bfe8c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.800+0000 7f66361e2740 1 --2- 172.21.15.28:0/1886970005 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 0x55a2bfe71750 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.800+0000 7f66361e2740 1 -- 172.21.15.28:0/1886970005 >> 172.21.15.28:0/1886970005 conn(0x55a2bfe71d50 msgr2=0x55a2bfe7d920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:06.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.800+0000 7f66361e2740 1 -- 172.21.15.28:0/1886970005 shutdown_connections 2024-04-24T08:12:06.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.800+0000 7f66361e2740 1 -- 172.21.15.28:0/1886970005 wait complete. 2024-04-24T08:12:06.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.801+0000 7f66361e2740 1 -- start start 2024-04-24T08:12:06.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.801+0000 7f66361e2740 1 Processor -- start 2024-04-24T08:12:06.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.802+0000 7f66361e2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 0x55a2bfe890c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.802+0000 7f66361e2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe76350 0x55a2bfe7a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.802+0000 7f66361e2740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe80a20 0x55a2bfe7abb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.802+0000 7f66361e2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f662c1aa040 con 0x55a2bfe75a30 2024-04-24T08:12:06.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.802+0000 7f66361e2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f662c1ab4b0 con 0x55a2bfe76350 2024-04-24T08:12:06.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.802+0000 7f66361e2740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f662c1aa710 con 0x55a2bfe80a20 2024-04-24T08:12:06.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.802+0000 7f66361e2740 10 client.0 _ll_register_callbacks cb 0x7f6624009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:06.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.802+0000 7f6630a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 0x55a2bfe890c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.802+0000 7f6631400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe80a20 0x55a2bfe7abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.802+0000 7f6631400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe80a20 0x55a2bfe7abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:33380/0 (socket says 172.21.15.28:33380) 2024-04-24T08:12:06.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.802+0000 7f6631400640 1 -- 172.21.15.28:0/2667637850 learned_addr learned my addr 172.21.15.28:0/2667637850 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:06.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.802+0000 7f662be00640 1 --2- 172.21.15.28:0/2667637850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe76350 0x55a2bfe7a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.802+0000 7f6630a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 0x55a2bfe890c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51570/0 (socket says 172.21.15.28:51570) 2024-04-24T08:12:06.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.803+0000 7f6630a00640 1 -- 172.21.15.28:0/2667637850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe80a20 msgr2=0x55a2bfe7abb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:06.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.803+0000 7f6630a00640 1 --2- 172.21.15.28:0/2667637850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe80a20 0x55a2bfe7abb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.803+0000 7f6630a00640 1 -- 172.21.15.28:0/2667637850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe76350 msgr2=0x55a2bfe7a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:06.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.803+0000 7f6630a00640 1 --2- 172.21.15.28:0/2667637850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe76350 0x55a2bfe7a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.803+0000 7f6630a00640 1 -- 172.21.15.28:0/2667637850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a2bfaad0f0 con 0x55a2bfe75a30 2024-04-24T08:12:06.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.803+0000 7f662be00640 1 --2- 172.21.15.28:0/2667637850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe76350 0x55a2bfe7a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:06.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.804+0000 7f6630a00640 1 --2- 172.21.15.28:0/2667637850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 0x55a2bfe890c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f662009c7b0 tx=0x7f66200d0e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3cac2b5523f208c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:06.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.804+0000 7f6612000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:12:06.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.804+0000 7f6612000640 1 -- 172.21.15.28:0/2667637850 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66200b5020 con 0x55a2bfe75a30 2024-04-24T08:12:06.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.804+0000 7f6612000640 1 -- 172.21.15.28:0/2667637850 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f6620003b30 con 0x55a2bfe75a30 2024-04-24T08:12:06.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.804+0000 7f66361e2740 1 -- 172.21.15.28:0/2667637850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f662009cab0 con 0x55a2bfe75a30 2024-04-24T08:12:06.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.804+0000 7f6612000640 1 -- 172.21.15.28:0/2667637850 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f66200a5720 con 0x55a2bfe75a30 2024-04-24T08:12:06.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.804+0000 7f66361e2740 10 client.15585 fetch_fsmap learned FSMap version 24 2024-04-24T08:12:06.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.804+0000 7f66361e2740 1 -- 172.21.15.28:0/2667637850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55a2bfda0810 con 0x55a2bfe75a30 2024-04-24T08:12:06.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.805+0000 7f6612000640 1 -- 172.21.15.28:0/2667637850 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f662009cab0 con 0x55a2bfe75a30 2024-04-24T08:12:06.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.805+0000 7f6612000640 1 -- 172.21.15.28:0/2667637850 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f6620098800 con 0x55a2bfe75a30 2024-04-24T08:12:06.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.805+0000 7f66361e2740 10 client.15585 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:12:06.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.805+0000 7f66361e2740 10 client.15585 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:12:06.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.805+0000 7f66361e2740 20 client.15585 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:06.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.805+0000 7f66361e2740 1 --2- 172.21.15.28:0/2667637850 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a2bfeffa30 0x55a2bff01e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:06.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.805+0000 7f66361e2740 4 client.15585 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:12:06.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.805+0000 7f66361e2740 1 -- 172.21.15.28:0/2667637850 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55a2bfde2a80 con 0x55a2bfeffa30 2024-04-24T08:12:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.805+0000 7f662be00640 1 --2- 172.21.15.28:0/2667637850 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a2bfeffa30 0x55a2bff01e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.806+0000 7f662be00640 1 --2- 172.21.15.28:0/2667637850 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a2bfeffa30 0x55a2bff01e50 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9517da244dc08c server_cookie=b1a76aae85fee55c in_seq=0 out_seq=0 2024-04-24T08:12:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.806+0000 7f6612000640 10 client.15585 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.807+0000 7f6612000640 1 -- 172.21.15.28:0/2667637850 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55a2bfde2a80 con 0x55a2bfeffa30 2024-04-24T08:12:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.807+0000 7f6612000640 10 client.15585 handle_command_reply: tid=0 2024-04-24T08:12:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.807+0000 7f66361e2740 1 client.15585 shutdown 2024-04-24T08:12:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.807+0000 7f66361e2740 1 -- 172.21.15.28:0/2667637850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 msgr2=0x55a2bfe890c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.807+0000 7f66361e2740 1 --2- 172.21.15.28:0/2667637850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 0x55a2bfe890c0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f662009c7b0 tx=0x7f66200d0e10 comp rx=0 tx=0).stop 2024-04-24T08:12:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.807+0000 7f66361e2740 1 -- 172.21.15.28:0/2667637850 shutdown_connections 2024-04-24T08:12:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.807+0000 7f66361e2740 1 --2- 172.21.15.28:0/2667637850 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a2bfeffa30 0x55a2bff01e50 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.807+0000 7f66361e2740 1 --2- 172.21.15.28:0/2667637850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a2bfe80a20 0x55a2bfe7abb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.807+0000 7f66361e2740 1 --2- 172.21.15.28:0/2667637850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a2bfe76350 0x55a2bfe7a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.807+0000 7f6612000640 0 client.15585 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.807+0000 7f66361e2740 1 --2- 172.21.15.28:0/2667637850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a2bfe75a30 0x55a2bfe890c0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.807+0000 7f66361e2740 1 -- 172.21.15.28:0/2667637850 >> 172.21.15.28:0/2667637850 conn(0x55a2bfe71d50 msgr2=0x55a2bfe7e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.808+0000 7f66361e2740 1 -- 172.21.15.28:0/2667637850 shutdown_connections 2024-04-24T08:12:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.808+0000 7f66361e2740 1 -- 172.21.15.28:0/2667637850 wait complete. 2024-04-24T08:12:06.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.808+0000 7f66361e2740 20 client.15585 trim_cache size 0 max 16384 2024-04-24T08:12:06.811 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:12:06.811 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-24T08:12:06.811 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-24T08:12:06.811 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:12:06.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.809+0000 7f6631e00640 1 -- 172.21.15.28:0/2421024146 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f66140a3da0 msgr2=0x7f66140a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:06.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.809+0000 7f6631e00640 1 --2- 172.21.15.28:0/2421024146 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f66140a3da0 0x7f66140a6260 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f6624009fd0 tx=0x7f6624009500 comp rx=0 tx=0).stop 2024-04-24T08:12:06.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.809+0000 7f6631e00640 1 -- 172.21.15.28:0/2421024146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f662c103060 msgr2=0x7f662c1965d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:06.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.809+0000 7f6631e00640 1 --2- 172.21.15.28:0/2421024146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f662c103060 0x7f662c1965d0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f662000ea60 tx=0x7f662000ef20 comp rx=0 tx=0).stop 2024-04-24T08:12:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.809+0000 7f6631e00640 1 -- 172.21.15.28:0/2421024146 shutdown_connections 2024-04-24T08:12:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.809+0000 7f6631e00640 1 --2- 172.21.15.28:0/2421024146 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f66140a3da0 0x7f66140a6260 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.809+0000 7f6631e00640 1 --2- 172.21.15.28:0/2421024146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f662c103980 0x7f662c196b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.809+0000 7f6631e00640 1 --2- 172.21.15.28:0/2421024146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f662c10c480 0x7f662c19ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.809+0000 7f6631e00640 1 --2- 172.21.15.28:0/2421024146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f662c103060 0x7f662c1965d0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.809+0000 7f6631e00640 1 -- 172.21.15.28:0/2421024146 >> 172.21.15.28:0/2421024146 conn(0x7f662c0fc880 msgr2=0x7f662c0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.809+0000 7f6631e00640 1 -- 172.21.15.28:0/2421024146 shutdown_connections 2024-04-24T08:12:06.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:06.809+0000 7f6631e00640 1 -- 172.21.15.28:0/2421024146 wait complete. 2024-04-24T08:12:06.833 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:12:06.834 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:12:07.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.078+0000 7f45eee00640 1 -- 172.21.15.28:0/1399702298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45e810e6e0 msgr2=0x7f45e8110ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:07.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.078+0000 7f45eee00640 1 --2- 172.21.15.28:0/1399702298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45e810e6e0 0x7f45e8110ad0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f45e0009be0 tx=0x7f45e002a750 comp rx=0 tx=0).stop 2024-04-24T08:12:07.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.079+0000 7f45eee00640 1 -- 172.21.15.28:0/1399702298 shutdown_connections 2024-04-24T08:12:07.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.079+0000 7f45eee00640 1 --2- 172.21.15.28:0/1399702298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e8074cd0 0x7f45e810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.079+0000 7f45eee00640 1 --2- 172.21.15.28:0/1399702298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45e810e6e0 0x7f45e8110ad0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.079+0000 7f45eee00640 1 --2- 172.21.15.28:0/1399702298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45e80743b0 0x7f45e8074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.079+0000 7f45eee00640 1 -- 172.21.15.28:0/1399702298 >> 172.21.15.28:0/1399702298 conn(0x7f45e80fd930 msgr2=0x7f45e80ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:07.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.079+0000 7f45eee00640 1 -- 172.21.15.28:0/1399702298 shutdown_connections 2024-04-24T08:12:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.079+0000 7f45eee00640 1 -- 172.21.15.28:0/1399702298 wait complete. 2024-04-24T08:12:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.080+0000 7f45eee00640 1 Processor -- start 2024-04-24T08:12:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.080+0000 7f45eee00640 1 -- start start 2024-04-24T08:12:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.081+0000 7f45eee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e80743b0 0x7f45e8073840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.081+0000 7f45eee00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45e8074cd0 0x7f45e8073d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.081+0000 7f45eee00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45e810e6e0 0x7f45e806caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.081+0000 7f45eee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f45e81143e0 con 0x7f45e80743b0 2024-04-24T08:12:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.081+0000 7f45eee00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f45e8114260 con 0x7f45e8074cd0 2024-04-24T08:12:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.081+0000 7f45eee00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f45e8114560 con 0x7f45e810e6e0 2024-04-24T08:12:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.081+0000 7f45eda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e80743b0 0x7f45e8073840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.081+0000 7f45ee400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45e810e6e0 0x7f45e806caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.081+0000 7f45eda00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e80743b0 0x7f45e8073840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51592/0 (socket says 172.21.15.28:51592) 2024-04-24T08:12:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.081+0000 7f45eda00640 1 -- 172.21.15.28:0/2794660472 learned_addr learned my addr 172.21.15.28:0/2794660472 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.081+0000 7f45ed000640 1 --2- 172.21.15.28:0/2794660472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45e8074cd0 0x7f45e8073d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.082+0000 7f45ed000640 1 -- 172.21.15.28:0/2794660472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45e810e6e0 msgr2=0x7f45e806caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.082+0000 7f45ed000640 1 --2- 172.21.15.28:0/2794660472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45e810e6e0 0x7f45e806caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.082+0000 7f45ed000640 1 -- 172.21.15.28:0/2794660472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e80743b0 msgr2=0x7f45e8073840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.082+0000 7f45ed000640 1 --2- 172.21.15.28:0/2794660472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e80743b0 0x7f45e8073840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.082+0000 7f45ed000640 1 -- 172.21.15.28:0/2794660472 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45e806d360 con 0x7f45e8074cd0 2024-04-24T08:12:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.082+0000 7f45eda00640 1 --2- 172.21.15.28:0/2794660472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e80743b0 0x7f45e8073840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.082+0000 7f45ee400640 1 --2- 172.21.15.28:0/2794660472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45e810e6e0 0x7f45e806caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.082+0000 7f45ed000640 1 --2- 172.21.15.28:0/2794660472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45e8074cd0 0x7f45e8073d80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f45d4009e90 tx=0x7f45d400b690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:07.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.082+0000 7f45e6000640 1 -- 172.21.15.28:0/2794660472 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d4022070 con 0x7f45e8074cd0 2024-04-24T08:12:07.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.082+0000 7f45eee00640 1 -- 172.21.15.28:0/2794660472 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45e806d650 con 0x7f45e8074cd0 2024-04-24T08:12:07.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.083+0000 7f45eee00640 1 -- 172.21.15.28:0/2794660472 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45e8071680 con 0x7f45e8074cd0 2024-04-24T08:12:07.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.083+0000 7f45e6000640 1 -- 172.21.15.28:0/2794660472 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f45d400be40 con 0x7f45e8074cd0 2024-04-24T08:12:07.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.083+0000 7f45e6000640 1 -- 172.21.15.28:0/2794660472 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f45d401d8e0 con 0x7f45e8074cd0 2024-04-24T08:12:07.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.084+0000 7f45eee00640 1 -- 172.21.15.28:0/2794660472 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f45c8005290 con 0x7f45e8074cd0 2024-04-24T08:12:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.086+0000 7f45e6000640 1 -- 172.21.15.28:0/2794660472 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f45d4025070 con 0x7f45e8074cd0 2024-04-24T08:12:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.087+0000 7f45e6000640 1 --2- 172.21.15.28:0/2794660472 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f45d00a3da0 0x7f45d00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.087+0000 7f45e6000640 1 -- 172.21.15.28:0/2794660472 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f45d40d81b0 con 0x7f45e8074cd0 2024-04-24T08:12:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.088+0000 7f45eda00640 1 --2- 172.21.15.28:0/2794660472 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f45d00a3da0 0x7f45d00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.089+0000 7f45eda00640 1 --2- 172.21.15.28:0/2794660472 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f45d00a3da0 0x7f45d00a6260 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f45dc011500 tx=0x7f45dc005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:07.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.092+0000 7f45e6000640 1 -- 172.21.15.28:0/2794660472 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f45d40a4180 con 0x7f45e8074cd0 2024-04-24T08:12:07.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.567+0000 7f45eee00640 1 -- 172.21.15.28:0/2794660472 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f45c8005850 con 0x7f45e8074cd0 2024-04-24T08:12:07.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.568+0000 7f45e6000640 1 -- 172.21.15.28:0/2794660472 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11871 (secure 0 0 0) 0x7f45d40a8480 con 0x7f45e8074cd0 2024-04-24T08:12:07.569 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:12:07.569 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:10:19.714831+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24574,24556,24572]},"id":1}]} 2024-04-24T08:12:07.569 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 24 2024-04-24T08:12:07.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.571+0000 7f45eee00640 1 -- 172.21.15.28:0/2794660472 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f45d00a3da0 msgr2=0x7f45d00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:07.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.571+0000 7f45eee00640 1 --2- 172.21.15.28:0/2794660472 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f45d00a3da0 0x7f45d00a6260 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f45dc011500 tx=0x7f45dc005f90 comp rx=0 tx=0).stop 2024-04-24T08:12:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.571+0000 7f45eee00640 1 -- 172.21.15.28:0/2794660472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45e8074cd0 msgr2=0x7f45e8073d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.571+0000 7f45eee00640 1 --2- 172.21.15.28:0/2794660472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45e8074cd0 0x7f45e8073d80 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f45d4009e90 tx=0x7f45d400b690 comp rx=0 tx=0).stop 2024-04-24T08:12:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.572+0000 7f45eee00640 1 -- 172.21.15.28:0/2794660472 shutdown_connections 2024-04-24T08:12:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.572+0000 7f45eee00640 1 --2- 172.21.15.28:0/2794660472 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f45d00a3da0 0x7f45d00a6260 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.572+0000 7f45eee00640 1 --2- 172.21.15.28:0/2794660472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f45e810e6e0 0x7f45e806caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.572+0000 7f45eee00640 1 --2- 172.21.15.28:0/2794660472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45e8074cd0 0x7f45e8073d80 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.572+0000 7f45eee00640 1 --2- 172.21.15.28:0/2794660472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f45e80743b0 0x7f45e8073840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.572+0000 7f45eee00640 1 -- 172.21.15.28:0/2794660472 >> 172.21.15.28:0/2794660472 conn(0x7f45e80fd930 msgr2=0x7f45e80fe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.572+0000 7f45eee00640 1 -- 172.21.15.28:0/2794660472 shutdown_connections 2024-04-24T08:12:07.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.572+0000 7f45eee00640 1 -- 172.21.15.28:0/2794660472 wait complete. 2024-04-24T08:12:07.594 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:07 smithi028 ceph-mon[27786]: pgmap v56: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:12:07.598 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:12:07.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:07 smithi117 ceph-mon[32438]: pgmap v56: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:12:07.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:07 smithi194 ceph-mon[32836]: pgmap v56: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:12:07.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.776+0000 7f147b200640 1 -- 172.21.15.28:0/3908006107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f147c104320 msgr2=0x7f147c104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:07.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.776+0000 7f147b200640 1 --2- 172.21.15.28:0/3908006107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f147c104320 0x7f147c104700 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f146c009860 tx=0x7f146c02a340 comp rx=0 tx=0).stop 2024-04-24T08:12:07.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.776+0000 7f147b200640 1 -- 172.21.15.28:0/3908006107 shutdown_connections 2024-04-24T08:12:07.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.776+0000 7f147b200640 1 --2- 172.21.15.28:0/3908006107 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f147c104cd0 0x7f147c108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.776+0000 7f147b200640 1 --2- 172.21.15.28:0/3908006107 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f147c1093f0 0x7f147c110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.776+0000 7f147b200640 1 --2- 172.21.15.28:0/3908006107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f147c104320 0x7f147c104700 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.776+0000 7f147b200640 1 -- 172.21.15.28:0/3908006107 >> 172.21.15.28:0/3908006107 conn(0x7f147c0ffb40 msgr2=0x7f147c101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:07.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.776+0000 7f147b200640 1 -- 172.21.15.28:0/3908006107 shutdown_connections 2024-04-24T08:12:07.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.776+0000 7f147b200640 1 -- 172.21.15.28:0/3908006107 wait complete. 2024-04-24T08:12:07.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.778+0000 7f147b200640 1 Processor -- start 2024-04-24T08:12:07.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.778+0000 7f147b200640 1 -- start start 2024-04-24T08:12:07.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.778+0000 7f147b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f147c104320 0x7f147c1a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.778+0000 7f147b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f147c1093f0 0x7f147c1a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.778+0000 7f147b200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f147c19a800 0x7f147c19acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.778+0000 7f147b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f147c114480 con 0x7f147c104320 2024-04-24T08:12:07.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.778+0000 7f147b200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f147c114300 con 0x7f147c19a800 2024-04-24T08:12:07.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.778+0000 7f147b200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f147c114600 con 0x7f147c1093f0 2024-04-24T08:12:07.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.779+0000 7f1479400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f147c1093f0 0x7f147c1a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.779+0000 7f1479400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f147c1093f0 0x7f147c1a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:33420/0 (socket says 172.21.15.28:33420) 2024-04-24T08:12:07.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.779+0000 7f1479400640 1 -- 172.21.15.28:0/758226076 learned_addr learned my addr 172.21.15.28:0/758226076 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.779+0000 7f1479e00640 1 --2- 172.21.15.28:0/758226076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f147c104320 0x7f147c1a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.779+0000 7f147a800640 1 --2- 172.21.15.28:0/758226076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f147c19a800 0x7f147c19acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.779+0000 7f1479400640 1 -- 172.21.15.28:0/758226076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f147c19a800 msgr2=0x7f147c19acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.780+0000 7f1479400640 1 --2- 172.21.15.28:0/758226076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f147c19a800 0x7f147c19acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.780+0000 7f1479400640 1 -- 172.21.15.28:0/758226076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f147c104320 msgr2=0x7f147c1a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.780+0000 7f1479400640 1 --2- 172.21.15.28:0/758226076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f147c104320 0x7f147c1a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.780+0000 7f1479400640 1 -- 172.21.15.28:0/758226076 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f147c19b390 con 0x7f147c1093f0 2024-04-24T08:12:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.780+0000 7f1479e00640 1 --2- 172.21.15.28:0/758226076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f147c104320 0x7f147c1a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.780+0000 7f1479400640 1 --2- 172.21.15.28:0/758226076 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f147c1093f0 0x7f147c1a0bd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f146800e9f0 tx=0x7f146800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.780+0000 7f1472a00640 1 -- 172.21.15.28:0/758226076 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146800ccc0 con 0x7f147c1093f0 2024-04-24T08:12:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.780+0000 7f147b200640 1 -- 172.21.15.28:0/758226076 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f147c19b680 con 0x7f147c1093f0 2024-04-24T08:12:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.780+0000 7f147b200640 1 -- 172.21.15.28:0/758226076 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f147c1a78b0 con 0x7f147c1093f0 2024-04-24T08:12:07.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.781+0000 7f1472a00640 1 -- 172.21.15.28:0/758226076 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1468010910 con 0x7f147c1093f0 2024-04-24T08:12:07.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.781+0000 7f1472a00640 1 -- 172.21.15.28:0/758226076 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f14680188a0 con 0x7f147c1093f0 2024-04-24T08:12:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.784+0000 7f1483852740 1 Processor -- start 2024-04-24T08:12:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.784+0000 7f1483852740 1 -- start start 2024-04-24T08:12:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.784+0000 7f1483852740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db332b40 0x5632db332f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.784+0000 7f1483852740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db333460 0x5632db33d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.785+0000 7f1483852740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db33db30 0x5632db33ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.785+0000 7f1483852740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5632db2981d0 con 0x5632db333460 2024-04-24T08:12:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.785+0000 7f1483852740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5632db23e450 con 0x5632db33db30 2024-04-24T08:12:07.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.785+0000 7f1483852740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5632db294190 con 0x5632db332b40 2024-04-24T08:12:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.785+0000 7f1479e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db333460 0x5632db33d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.785+0000 7f1479e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db333460 0x5632db33d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51628/0 (socket says 172.21.15.28:51628) 2024-04-24T08:12:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.785+0000 7f1479e00640 1 -- 172.21.15.28:0/3394114135 learned_addr learned my addr 172.21.15.28:0/3394114135 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.785+0000 7f147a800640 1 --2- 172.21.15.28:0/3394114135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db332b40 0x5632db332f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.786+0000 7f1479e00640 1 -- 172.21.15.28:0/3394114135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db332b40 msgr2=0x5632db332f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:07.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.786+0000 7f1479e00640 1 --2- 172.21.15.28:0/3394114135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db332b40 0x5632db332f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.786+0000 7f1479e00640 1 -- 172.21.15.28:0/3394114135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db33db30 msgr2=0x5632db33ff20 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:12:07.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.786+0000 7f1479e00640 1 --2- 172.21.15.28:0/3394114135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db33db30 0x5632db33ff20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.786+0000 7f1479e00640 1 -- 172.21.15.28:0/3394114135 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632db2fb310 con 0x5632db333460 2024-04-24T08:12:07.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.786+0000 7f1472a00640 1 -- 172.21.15.28:0/758226076 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1468010ac0 con 0x7f147c1093f0 2024-04-24T08:12:07.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.786+0000 7f1479e00640 1 --2- 172.21.15.28:0/3394114135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db333460 0x5632db33d5f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f146c009780 tx=0x7f146c0303b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8872b7982558eaf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:07.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.787+0000 7f1467000640 1 -- 172.21.15.28:0/3394114135 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146c037040 con 0x5632db333460 2024-04-24T08:12:07.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.787+0000 7f1467000640 1 -- 172.21.15.28:0/3394114135 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f146c041070 con 0x5632db333460 2024-04-24T08:12:07.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.787+0000 7f1467000640 1 -- 172.21.15.28:0/3394114135 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146c005330 con 0x5632db333460 2024-04-24T08:12:07.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.788+0000 7f1483852740 1 -- 172.21.15.28:0/3394114135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db333460 msgr2=0x5632db33d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:07.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.788+0000 7f1483852740 1 --2- 172.21.15.28:0/3394114135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db333460 0x5632db33d5f0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f146c009780 tx=0x7f146c0303b0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.788+0000 7f1472a00640 1 --2- 172.21.15.28:0/758226076 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f14600a3e20 0x7f14600a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.788+0000 7f1472a00640 1 -- 172.21.15.28:0/758226076 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f14680d0b00 con 0x7f147c1093f0 2024-04-24T08:12:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.789+0000 7f1483852740 1 -- 172.21.15.28:0/3394114135 shutdown_connections 2024-04-24T08:12:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.789+0000 7f1483852740 1 --2- 172.21.15.28:0/3394114135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db333460 0x5632db33d5f0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.789+0000 7f1483852740 1 --2- 172.21.15.28:0/3394114135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db33db30 0x5632db33ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.789+0000 7f1483852740 1 --2- 172.21.15.28:0/3394114135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db332b40 0x5632db332f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.789+0000 7f1483852740 1 -- 172.21.15.28:0/3394114135 >> 172.21.15.28:0/3394114135 conn(0x5632db32ee60 msgr2=0x5632db32f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.789+0000 7f1483852740 1 -- 172.21.15.28:0/3394114135 shutdown_connections 2024-04-24T08:12:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.789+0000 7f147a800640 1 --2- 172.21.15.28:0/758226076 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f14600a3e20 0x7f14600a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.790+0000 7f1483852740 1 -- 172.21.15.28:0/3394114135 wait complete. 2024-04-24T08:12:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.790+0000 7f1483852740 1 -- start start 2024-04-24T08:12:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.790+0000 7f1483852740 1 Processor -- start 2024-04-24T08:12:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.791+0000 7f1483852740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db332b40 0x5632db33d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.791+0000 7f1483852740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db333460 0x5632db33a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.791+0000 7f1483852740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db33db30 0x5632db33afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.791+0000 7f1483852740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5632db340960 con 0x5632db333460 2024-04-24T08:12:07.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.791+0000 7f1483852740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5632db251810 con 0x5632db332b40 2024-04-24T08:12:07.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.791+0000 7f1483852740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5632db340ae0 con 0x5632db33db30 2024-04-24T08:12:07.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.791+0000 7f1483852740 10 client.0 _ll_register_callbacks cb 0x5632db21f140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:07.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.791+0000 7f147a800640 1 --2- 172.21.15.28:0/758226076 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f14600a3e20 0x7f14600a62e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1474005fd0 tx=0x7f1474009500 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:07.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.791+0000 7f1479e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db332b40 0x5632db33d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.792+0000 7f1479e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db332b40 0x5632db33d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40320/0 (socket says 172.21.15.28:40320) 2024-04-24T08:12:07.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.792+0000 7f1479e00640 1 -- 172.21.15.28:0/1022476317 learned_addr learned my addr 172.21.15.28:0/1022476317 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:07.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.792+0000 7f1479400640 1 --2- 172.21.15.28:0/1022476317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db333460 0x5632db33a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.792+0000 7f147a800640 1 --2- 172.21.15.28:0/1022476317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db33db30 0x5632db33afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.792+0000 7f1479e00640 1 -- 172.21.15.28:0/1022476317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db33db30 msgr2=0x5632db33afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:07.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.792+0000 7f1479e00640 1 --2- 172.21.15.28:0/1022476317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db33db30 0x5632db33afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.792+0000 7f1479e00640 1 -- 172.21.15.28:0/1022476317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db333460 msgr2=0x5632db33a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:07.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.792+0000 7f1479e00640 1 --2- 172.21.15.28:0/1022476317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db333460 0x5632db33a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.792+0000 7f1479e00640 1 -- 172.21.15.28:0/1022476317 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632db2623f0 con 0x5632db332b40 2024-04-24T08:12:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.792+0000 7f1479400640 1 --2- 172.21.15.28:0/1022476317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db333460 0x5632db33a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.793+0000 7f1479e00640 1 --2- 172.21.15.28:0/1022476317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db332b40 0x5632db33d590 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f146c02a380 tx=0x7f146c030ad0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e40901d5868e8442 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.793+0000 7f145b400640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:12:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.793+0000 7f145b400640 1 -- 172.21.15.28:0/1022476317 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146c049070 con 0x5632db332b40 2024-04-24T08:12:07.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.793+0000 7f145b400640 1 -- 172.21.15.28:0/1022476317 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f146c0050b0 con 0x5632db332b40 2024-04-24T08:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.793+0000 7f1483852740 1 -- 172.21.15.28:0/1022476317 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f146c041070 con 0x5632db332b40 2024-04-24T08:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.794+0000 7f145b400640 1 -- 172.21.15.28:0/1022476317 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f146c037070 con 0x5632db332b40 2024-04-24T08:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.794+0000 7f1483852740 10 client.25700 fetch_fsmap learned FSMap version 24 2024-04-24T08:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.794+0000 7f1483852740 1 -- 172.21.15.28:0/1022476317 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5632dafaa3f0 con 0x5632db332b40 2024-04-24T08:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.795+0000 7f145b400640 1 -- 172.21.15.28:0/1022476317 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f146c041070 con 0x5632db332b40 2024-04-24T08:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.795+0000 7f145b400640 1 -- 172.21.15.28:0/1022476317 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f146c004c20 con 0x5632db332b40 2024-04-24T08:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.795+0000 7f1483852740 10 client.25700 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.795+0000 7f1483852740 10 client.25700 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.795+0000 7f1483852740 20 client.25700 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:07.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.795+0000 7f1483852740 1 --2- 172.21.15.28:0/1022476317 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5632db34dbe0 0x5632db34ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.795+0000 7f1483852740 4 client.25700 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.795+0000 7f1483852740 1 -- 172.21.15.28:0/1022476317 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5632db2a5bf0 con 0x5632db34dbe0 2024-04-24T08:12:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.795+0000 7f1479400640 1 --2- 172.21.15.28:0/1022476317 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5632db34dbe0 0x5632db34ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.796+0000 7f145b400640 10 client.25700 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:12:07.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.796+0000 7f1479400640 1 --2- 172.21.15.28:0/1022476317 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5632db34dbe0 0x5632db34ffd0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=72ccbce8f3db945a server_cookie=a768fb529cf69642 in_seq=0 out_seq=0 2024-04-24T08:12:07.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.797+0000 7f145b400640 1 -- 172.21.15.28:0/1022476317 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5632db2a5bf0 con 0x5632db34dbe0 2024-04-24T08:12:07.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.797+0000 7f145b400640 10 client.25700 handle_command_reply: tid=0 2024-04-24T08:12:07.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.797+0000 7f1483852740 1 client.25700 shutdown 2024-04-24T08:12:07.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.797+0000 7f1483852740 1 -- 172.21.15.28:0/1022476317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db332b40 msgr2=0x5632db33d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:07.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.797+0000 7f1483852740 1 --2- 172.21.15.28:0/1022476317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db332b40 0x5632db33d590 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f146c02a380 tx=0x7f146c030ad0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.797+0000 7f1483852740 1 -- 172.21.15.28:0/1022476317 shutdown_connections 2024-04-24T08:12:07.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.797+0000 7f1483852740 1 --2- 172.21.15.28:0/1022476317 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5632db34dbe0 0x5632db34ffd0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.797+0000 7f1483852740 1 --2- 172.21.15.28:0/1022476317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db333460 0x5632db33a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.797+0000 7f1483852740 1 --2- 172.21.15.28:0/1022476317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db33db30 0x5632db33afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.797+0000 7f1483852740 1 --2- 172.21.15.28:0/1022476317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db332b40 0x5632db33d590 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.797+0000 7f1483852740 1 -- 172.21.15.28:0/1022476317 >> 172.21.15.28:0/1022476317 conn(0x5632db32ee60 msgr2=0x5632db334b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:07.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.797+0000 7f145b400640 0 client.25700 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:12:07.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.798+0000 7f1483852740 1 -- 172.21.15.28:0/1022476317 shutdown_connections 2024-04-24T08:12:07.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.798+0000 7f1483852740 1 -- 172.21.15.28:0/1022476317 wait complete. 2024-04-24T08:12:07.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.798+0000 7f1483852740 20 client.25700 trim_cache size 0 max 16384 2024-04-24T08:12:07.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.812+0000 7f1483852740 1 Processor -- start 2024-04-24T08:12:07.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.812+0000 7f1483852740 1 -- start start 2024-04-24T08:12:07.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.812+0000 7f1483852740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db332b40 0x5632db3498b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.812+0000 7f1483852740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db333460 0x5632db339860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.812+0000 7f1483852740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db33db30 0x5632db33d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.812+0000 7f1483852740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f145c011100 con 0x5632db332b40 2024-04-24T08:12:07.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.812+0000 7f1483852740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f145c00e9d0 con 0x5632db333460 2024-04-24T08:12:07.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.812+0000 7f1483852740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f145c00c820 con 0x5632db33db30 2024-04-24T08:12:07.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.813+0000 7f1479e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db332b40 0x5632db3498b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.813+0000 7f1479e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db332b40 0x5632db3498b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51648/0 (socket says 172.21.15.28:51648) 2024-04-24T08:12:07.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.813+0000 7f1479e00640 1 -- 172.21.15.28:0/3600814608 learned_addr learned my addr 172.21.15.28:0/3600814608 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:07.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.813+0000 7f147a800640 1 --2- 172.21.15.28:0/3600814608 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db33db30 0x5632db33d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.813+0000 7f1479400640 1 --2- 172.21.15.28:0/3600814608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db333460 0x5632db339860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.813+0000 7f1479e00640 1 -- 172.21.15.28:0/3600814608 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db33db30 msgr2=0x5632db33d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:07.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.813+0000 7f1479e00640 1 --2- 172.21.15.28:0/3600814608 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db33db30 0x5632db33d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.813+0000 7f1479e00640 1 -- 172.21.15.28:0/3600814608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db333460 msgr2=0x5632db339860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:07.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.813+0000 7f1479e00640 1 --2- 172.21.15.28:0/3600814608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db333460 0x5632db339860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.813+0000 7f1479e00640 1 -- 172.21.15.28:0/3600814608 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5632db333290 con 0x5632db332b40 2024-04-24T08:12:07.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.814+0000 7f147a800640 1 --2- 172.21.15.28:0/3600814608 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db33db30 0x5632db33d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:07.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.814+0000 7f1479e00640 1 --2- 172.21.15.28:0/3600814608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db332b40 0x5632db3498b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f146c02d020 tx=0x7f146c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7b5a16f6c1f71f3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:07.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.814+0000 7f1466600640 1 -- 172.21.15.28:0/3600814608 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146c049070 con 0x5632db332b40 2024-04-24T08:12:07.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.814+0000 7f1466600640 1 -- 172.21.15.28:0/3600814608 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f146c041040 con 0x5632db332b40 2024-04-24T08:12:07.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.814+0000 7f1466600640 1 -- 172.21.15.28:0/3600814608 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146c004ac0 con 0x5632db332b40 2024-04-24T08:12:07.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.815+0000 7f1483852740 1 -- 172.21.15.28:0/3600814608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db332b40 msgr2=0x5632db3498b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:07.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.815+0000 7f1483852740 1 --2- 172.21.15.28:0/3600814608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db332b40 0x5632db3498b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f146c02d020 tx=0x7f146c000bc0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.815+0000 7f1483852740 1 -- 172.21.15.28:0/3600814608 shutdown_connections 2024-04-24T08:12:07.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.815+0000 7f1483852740 1 --2- 172.21.15.28:0/3600814608 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db33db30 0x5632db33d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.815+0000 7f1483852740 1 --2- 172.21.15.28:0/3600814608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db333460 0x5632db339860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.815+0000 7f1483852740 1 --2- 172.21.15.28:0/3600814608 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db332b40 0x5632db3498b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.815+0000 7f1483852740 1 -- 172.21.15.28:0/3600814608 >> 172.21.15.28:0/3600814608 conn(0x5632db32ee60 msgr2=0x5632db339150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:07.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.815+0000 7f1483852740 1 -- 172.21.15.28:0/3600814608 shutdown_connections 2024-04-24T08:12:07.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.816+0000 7f1483852740 1 -- 172.21.15.28:0/3600814608 wait complete. 2024-04-24T08:12:07.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.816+0000 7f1483852740 1 -- start start 2024-04-24T08:12:07.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.816+0000 7f1483852740 1 Processor -- start 2024-04-24T08:12:07.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.817+0000 7f1483852740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db332b40 0x5632db33e110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.817+0000 7f1483852740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db333460 0x5632db33e650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.817+0000 7f1483852740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db33db30 0x5632db33eb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.817+0000 7f1483852740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f145c00ce90 con 0x5632db33db30 2024-04-24T08:12:07.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.817+0000 7f1483852740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f145c00e300 con 0x5632db333460 2024-04-24T08:12:07.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.817+0000 7f1483852740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f145c00d560 con 0x5632db332b40 2024-04-24T08:12:07.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.817+0000 7f1483852740 10 client.0 _ll_register_callbacks cb 0x7f1468005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:07.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.817+0000 7f1479e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db332b40 0x5632db33e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.817+0000 7f147a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db33db30 0x5632db33eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.817+0000 7f1479e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db332b40 0x5632db33e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:33446/0 (socket says 172.21.15.28:33446) 2024-04-24T08:12:07.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.817+0000 7f1479e00640 1 -- 172.21.15.28:0/876543531 learned_addr learned my addr 172.21.15.28:0/876543531 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:07.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.818+0000 7f1479e00640 1 -- 172.21.15.28:0/876543531 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db333460 msgr2=0x5632db33e650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:07.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.818+0000 7f1479e00640 1 --2- 172.21.15.28:0/876543531 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db333460 0x5632db33e650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.818+0000 7f1479e00640 1 -- 172.21.15.28:0/876543531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db33db30 msgr2=0x5632db33eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:07.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.818+0000 7f1479e00640 1 --2- 172.21.15.28:0/876543531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db33db30 0x5632db33eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.818+0000 7f1479e00640 1 -- 172.21.15.28:0/876543531 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5632db0747d0 con 0x5632db332b40 2024-04-24T08:12:07.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.818+0000 7f1479e00640 1 --2- 172.21.15.28:0/876543531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db332b40 0x5632db33e110 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f146c02a380 tx=0x7f146c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0d033b246a65db8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:07.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.818+0000 7f145b400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:12:07.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.819+0000 7f145b400640 1 -- 172.21.15.28:0/876543531 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f146c049070 con 0x5632db332b40 2024-04-24T08:12:07.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.819+0000 7f145b400640 1 -- 172.21.15.28:0/876543531 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f146c005250 con 0x5632db332b40 2024-04-24T08:12:07.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.819+0000 7f1483852740 1 -- 172.21.15.28:0/876543531 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f146c041040 con 0x5632db332b40 2024-04-24T08:12:07.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.819+0000 7f145b400640 1 -- 172.21.15.28:0/876543531 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f146c005450 con 0x5632db332b40 2024-04-24T08:12:07.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.819+0000 7f1483852740 10 client.25750 fetch_fsmap learned FSMap version 24 2024-04-24T08:12:07.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.819+0000 7f1483852740 1 -- 172.21.15.28:0/876543531 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5632db345f30 con 0x5632db332b40 2024-04-24T08:12:07.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.819+0000 7f145b400640 1 -- 172.21.15.28:0/876543531 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f146c041040 con 0x5632db332b40 2024-04-24T08:12:07.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.820+0000 7f145b400640 1 -- 172.21.15.28:0/876543531 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f146c058070 con 0x5632db332b40 2024-04-24T08:12:07.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.820+0000 7f1483852740 10 client.25750 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:12:07.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.820+0000 7f1483852740 10 client.25750 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:12:07.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.820+0000 7f1483852740 20 client.25750 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:07.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.820+0000 7f1483852740 1 --2- 172.21.15.28:0/876543531 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5632db346480 0x5632db3b5e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:07.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.820+0000 7f1483852740 4 client.25750 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:12:07.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.820+0000 7f1483852740 1 -- 172.21.15.28:0/876543531 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5632db29fa60 con 0x5632db346480 2024-04-24T08:12:07.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.820+0000 7f1479400640 1 --2- 172.21.15.28:0/876543531 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5632db346480 0x5632db3b5e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:07.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.821+0000 7f1479400640 1 --2- 172.21.15.28:0/876543531 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5632db346480 0x5632db3b5e50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=917c121c0b23f0de server_cookie=f38ebca971d11176 in_seq=0 out_seq=0 2024-04-24T08:12:07.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.821+0000 7f145b400640 10 client.25750 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:12:07.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.821+0000 7f145b400640 1 -- 172.21.15.28:0/876543531 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5632db29fa60 con 0x5632db346480 2024-04-24T08:12:07.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.821+0000 7f145b400640 10 client.25750 handle_command_reply: tid=0 2024-04-24T08:12:07.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.821+0000 7f1483852740 1 client.25750 shutdown 2024-04-24T08:12:07.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.821+0000 7f1483852740 1 -- 172.21.15.28:0/876543531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db332b40 msgr2=0x5632db33e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:07.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.821+0000 7f1483852740 1 --2- 172.21.15.28:0/876543531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db332b40 0x5632db33e110 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f146c02a380 tx=0x7f146c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.821+0000 7f1483852740 1 -- 172.21.15.28:0/876543531 shutdown_connections 2024-04-24T08:12:07.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.821+0000 7f1483852740 1 --2- 172.21.15.28:0/876543531 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5632db346480 0x5632db3b5e50 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.821+0000 7f1483852740 1 --2- 172.21.15.28:0/876543531 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5632db33db30 0x5632db33eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.821+0000 7f1483852740 1 --2- 172.21.15.28:0/876543531 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5632db333460 0x5632db33e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.821+0000 7f1483852740 1 --2- 172.21.15.28:0/876543531 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5632db332b40 0x5632db33e110 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.821+0000 7f1483852740 1 -- 172.21.15.28:0/876543531 >> 172.21.15.28:0/876543531 conn(0x5632db32ee60 msgr2=0x5632db3399e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:07.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.821+0000 7f145b400640 0 client.25750 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:12:07.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.821+0000 7f1483852740 1 -- 172.21.15.28:0/876543531 shutdown_connections 2024-04-24T08:12:07.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.821+0000 7f1483852740 1 -- 172.21.15.28:0/876543531 wait complete. 2024-04-24T08:12:07.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.822+0000 7f1483852740 20 client.25750 trim_cache size 0 max 16384 2024-04-24T08:12:07.825 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:12:07.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.822+0000 7f147b200640 1 -- 172.21.15.28:0/758226076 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f14600a3e20 msgr2=0x7f14600a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:07.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.822+0000 7f147b200640 1 --2- 172.21.15.28:0/758226076 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f14600a3e20 0x7f14600a62e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f1474005fd0 tx=0x7f1474009500 comp rx=0 tx=0).stop 2024-04-24T08:12:07.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.822+0000 7f147b200640 1 -- 172.21.15.28:0/758226076 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f147c1093f0 msgr2=0x7f147c1a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:07.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.822+0000 7f147b200640 1 --2- 172.21.15.28:0/758226076 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f147c1093f0 0x7f147c1a0bd0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f146800e9f0 tx=0x7f146800eeb0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.823+0000 7f147b200640 1 -- 172.21.15.28:0/758226076 shutdown_connections 2024-04-24T08:12:07.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.823+0000 7f147b200640 1 --2- 172.21.15.28:0/758226076 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f14600a3e20 0x7f14600a62e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.823+0000 7f147b200640 1 --2- 172.21.15.28:0/758226076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f147c19a800 0x7f147c19acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.823+0000 7f147b200640 1 --2- 172.21.15.28:0/758226076 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f147c1093f0 0x7f147c1a0bd0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.823+0000 7f147b200640 1 --2- 172.21.15.28:0/758226076 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f147c104320 0x7f147c1a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:07.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.823+0000 7f147b200640 1 -- 172.21.15.28:0/758226076 >> 172.21.15.28:0/758226076 conn(0x7f147c0ffb40 msgr2=0x7f147c100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:07.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.823+0000 7f147b200640 1 -- 172.21.15.28:0/758226076 shutdown_connections 2024-04-24T08:12:07.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:07.823+0000 7f147b200640 1 -- 172.21.15.28:0/758226076 wait complete. 2024-04-24T08:12:07.847 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:12:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.048+0000 7f5c6be00640 1 -- 172.21.15.28:0/99834831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c6c10e740 msgr2=0x7f5c6c110b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.048+0000 7f5c6be00640 1 --2- 172.21.15.28:0/99834831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c6c10e740 0x7f5c6c110b50 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f5c64009be0 tx=0x7f5c6402a750 comp rx=0 tx=0).stop 2024-04-24T08:12:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.048+0000 7f5c6be00640 1 -- 172.21.15.28:0/99834831 shutdown_connections 2024-04-24T08:12:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.048+0000 7f5c6be00640 1 --2- 172.21.15.28:0/99834831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c6c074cf0 0x7f5c6c10e200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.048+0000 7f5c6be00640 1 --2- 172.21.15.28:0/99834831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c6c10e740 0x7f5c6c110b50 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.048+0000 7f5c6be00640 1 --2- 172.21.15.28:0/99834831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c6c0743d0 0x7f5c6c0747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.048+0000 7f5c6be00640 1 -- 172.21.15.28:0/99834831 >> 172.21.15.28:0/99834831 conn(0x7f5c6c0fd9d0 msgr2=0x7f5c6c0ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.048+0000 7f5c6be00640 1 -- 172.21.15.28:0/99834831 shutdown_connections 2024-04-24T08:12:08.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.048+0000 7f5c6be00640 1 -- 172.21.15.28:0/99834831 wait complete. 2024-04-24T08:12:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.049+0000 7f5c6be00640 1 Processor -- start 2024-04-24T08:12:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.049+0000 7f5c6be00640 1 -- start start 2024-04-24T08:12:08.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.050+0000 7f5c6be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c6c0743d0 0x7f5c6c1a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.050+0000 7f5c6be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c6c074cf0 0x7f5c6c1a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.050+0000 7f5c6be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c6c10e740 0x7f5c6c19a820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.050+0000 7f5c6be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c114460 con 0x7f5c6c10e740 2024-04-24T08:12:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.050+0000 7f5c6be00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c1142e0 con 0x7f5c6c074cf0 2024-04-24T08:12:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.050+0000 7f5c6be00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c1145e0 con 0x7f5c6c0743d0 2024-04-24T08:12:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.050+0000 7f5c6aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c6c0743d0 0x7f5c6c1a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.050+0000 7f5c6aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c6c0743d0 0x7f5c6c1a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:33452/0 (socket says 172.21.15.28:33452) 2024-04-24T08:12:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.050+0000 7f5c6aa00640 1 -- 172.21.15.28:0/4007236878 learned_addr learned my addr 172.21.15.28:0/4007236878 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.050+0000 7f5c6a000640 1 --2- 172.21.15.28:0/4007236878 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c6c074cf0 0x7f5c6c1a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.050+0000 7f5c6b400640 1 --2- 172.21.15.28:0/4007236878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c6c10e740 0x7f5c6c19a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.051+0000 7f5c6a000640 1 -- 172.21.15.28:0/4007236878 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c6c0743d0 msgr2=0x7f5c6c1a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.051+0000 7f5c6a000640 1 --2- 172.21.15.28:0/4007236878 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c6c0743d0 0x7f5c6c1a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.051+0000 7f5c6a000640 1 -- 172.21.15.28:0/4007236878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c6c10e740 msgr2=0x7f5c6c19a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.051+0000 7f5c6a000640 1 --2- 172.21.15.28:0/4007236878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c6c10e740 0x7f5c6c19a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.051+0000 7f5c6a000640 1 -- 172.21.15.28:0/4007236878 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6c19aff0 con 0x7f5c6c074cf0 2024-04-24T08:12:08.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.051+0000 7f5c6a000640 1 --2- 172.21.15.28:0/4007236878 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c6c074cf0 0x7f5c6c1a0bf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5c5800e790 tx=0x7f5c5800ec50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:08.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.051+0000 7f5c63400640 1 -- 172.21.15.28:0/4007236878 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5800eea0 con 0x7f5c6c074cf0 2024-04-24T08:12:08.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.052+0000 7f5c63400640 1 -- 172.21.15.28:0/4007236878 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c5800ccc0 con 0x7f5c6c074cf0 2024-04-24T08:12:08.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.052+0000 7f5c6be00640 1 -- 172.21.15.28:0/4007236878 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c6c19b2e0 con 0x7f5c6c074cf0 2024-04-24T08:12:08.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.052+0000 7f5c63400640 1 -- 172.21.15.28:0/4007236878 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c58018780 con 0x7f5c6c074cf0 2024-04-24T08:12:08.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.052+0000 7f5c6be00640 1 -- 172.21.15.28:0/4007236878 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c6c1a7500 con 0x7f5c6c074cf0 2024-04-24T08:12:08.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.056+0000 7f5c63400640 1 -- 172.21.15.28:0/4007236878 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5c58010430 con 0x7f5c6c074cf0 2024-04-24T08:12:08.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.057+0000 7f5c743c1740 1 Processor -- start 2024-04-24T08:12:08.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.057+0000 7f5c743c1740 1 -- start start 2024-04-24T08:12:08.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.057+0000 7f5c743c1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d733891350 0x55d733891730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.058+0000 7f5c743c1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 0x55d73389be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.058+0000 7f5c743c1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d73389c340 0x55d73389e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.058+0000 7f5c743c1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d7337ee1d0 con 0x55d733891350 2024-04-24T08:12:08.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.058+0000 7f5c743c1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d733794450 con 0x55d73389c340 2024-04-24T08:12:08.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.058+0000 7f5c743c1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d7337ea190 con 0x55d733891c70 2024-04-24T08:12:08.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.058+0000 7f5c63400640 1 --2- 172.21.15.28:0/4007236878 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5c500a3da0 0x7f5c500a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.058+0000 7f5c63400640 1 -- 172.21.15.28:0/4007236878 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f5c580d08e0 con 0x7f5c6c074cf0 2024-04-24T08:12:08.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.058+0000 7f5c6a000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d73389c340 0x55d73389e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.058+0000 7f5c6a000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d73389c340 0x55d73389e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40374/0 (socket says 172.21.15.28:40374) 2024-04-24T08:12:08.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.058+0000 7f5c6a000640 1 -- 172.21.15.28:0/3702044751 learned_addr learned my addr 172.21.15.28:0/3702044751 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.058+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/3702044751 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 0x55d73389be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.059+0000 7f5c6b400640 1 --2- 172.21.15.28:0/3702044751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d733891350 0x55d733891730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.059+0000 7f5c6a000640 1 -- 172.21.15.28:0/3702044751 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 msgr2=0x55d73389be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.059+0000 7f5c6a000640 1 --2- 172.21.15.28:0/3702044751 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 0x55d73389be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.059+0000 7f5c6a000640 1 -- 172.21.15.28:0/3702044751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d733891350 msgr2=0x55d733891730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.059+0000 7f5c6a000640 1 --2- 172.21.15.28:0/3702044751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d733891350 0x55d733891730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.059+0000 7f5c6a000640 1 -- 172.21.15.28:0/3702044751 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d733851310 con 0x55d73389c340 2024-04-24T08:12:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.059+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/3702044751 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 0x55d73389be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.059+0000 7f5c6a000640 1 --2- 172.21.15.28:0/3702044751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d73389c340 0x55d73389e730 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5c58009230 tx=0x7f5c5809c910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c07a5dfe417e0be server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.059+0000 7f5c6b400640 1 --2- 172.21.15.28:0/3702044751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d733891350 0x55d733891730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:08.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.059+0000 7f5c57400640 1 -- 172.21.15.28:0/3702044751 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c58010a30 con 0x55d73389c340 2024-04-24T08:12:08.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.059+0000 7f5c57400640 1 -- 172.21.15.28:0/3702044751 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c58085e50 con 0x55d73389c340 2024-04-24T08:12:08.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.059+0000 7f5c6b400640 1 --2- 172.21.15.28:0/4007236878 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5c500a3da0 0x7f5c500a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.059+0000 7f5c57400640 1 -- 172.21.15.28:0/3702044751 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c580870e0 con 0x55d73389c340 2024-04-24T08:12:08.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.060+0000 7f5c743c1740 1 -- 172.21.15.28:0/3702044751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d73389c340 msgr2=0x55d73389e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.060+0000 7f5c743c1740 1 --2- 172.21.15.28:0/3702044751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d73389c340 0x55d73389e730 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5c58009230 tx=0x7f5c5809c910 comp rx=0 tx=0).stop 2024-04-24T08:12:08.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.061+0000 7f5c6b400640 1 --2- 172.21.15.28:0/4007236878 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5c500a3da0 0x7f5c500a6260 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5c64009b30 tx=0x7f5c64009500 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:08.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.061+0000 7f5c743c1740 1 -- 172.21.15.28:0/3702044751 shutdown_connections 2024-04-24T08:12:08.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.061+0000 7f5c743c1740 1 --2- 172.21.15.28:0/3702044751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d73389c340 0x55d73389e730 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.061+0000 7f5c743c1740 1 --2- 172.21.15.28:0/3702044751 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 0x55d73389be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.061+0000 7f5c743c1740 1 --2- 172.21.15.28:0/3702044751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d733891350 0x55d733891730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.061+0000 7f5c743c1740 1 -- 172.21.15.28:0/3702044751 >> 172.21.15.28:0/3702044751 conn(0x55d73388d670 msgr2=0x55d73388da50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:08.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.061+0000 7f5c743c1740 1 -- 172.21.15.28:0/3702044751 shutdown_connections 2024-04-24T08:12:08.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.061+0000 7f5c743c1740 1 -- 172.21.15.28:0/3702044751 wait complete. 2024-04-24T08:12:08.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.061+0000 7f5c743c1740 1 -- start start 2024-04-24T08:12:08.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.062+0000 7f5c743c1740 1 Processor -- start 2024-04-24T08:12:08.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.062+0000 7f5c743c1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d733891350 0x55d73389bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.062+0000 7f5c743c1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 0x55d733898fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.062+0000 7f5c743c1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d73389c340 0x55d733899580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.062+0000 7f5c743c1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d73389f1c0 con 0x55d73389c340 2024-04-24T08:12:08.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.062+0000 7f5c743c1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d7337a7810 con 0x55d733891350 2024-04-24T08:12:08.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.062+0000 7f5c743c1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d73389f340 con 0x55d733891c70 2024-04-24T08:12:08.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.062+0000 7f5c743c1740 10 client.0 _ll_register_callbacks cb 0x55d733775140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:08.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.062+0000 7f5c6a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 0x55d733898fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.062+0000 7f5c6a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 0x55d733898fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:33458/0 (socket says 172.21.15.28:33458) 2024-04-24T08:12:08.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.062+0000 7f5c6a000640 1 -- 172.21.15.28:0/3372728952 learned_addr learned my addr 172.21.15.28:0/3372728952 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:08.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.063+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/3372728952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d733891350 0x55d73389bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.063+0000 7f5c6b400640 1 --2- 172.21.15.28:0/3372728952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d73389c340 0x55d733899580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.063+0000 7f5c6a000640 1 -- 172.21.15.28:0/3372728952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d733891350 msgr2=0x55d73389bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:08.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.063+0000 7f5c6a000640 1 --2- 172.21.15.28:0/3372728952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d733891350 0x55d73389bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.063+0000 7f5c6a000640 1 -- 172.21.15.28:0/3372728952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d73389c340 msgr2=0x55d733899580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:08.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.063+0000 7f5c6a000640 1 --2- 172.21.15.28:0/3372728952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d73389c340 0x55d733899580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.063+0000 7f5c6a000640 1 -- 172.21.15.28:0/3372728952 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d7337b83f0 con 0x55d733891c70 2024-04-24T08:12:08.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.063+0000 7f5c6a000640 1 --2- 172.21.15.28:0/3372728952 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 0x55d733898fe0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5c5800c6a0 tx=0x7f5c58099220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8968a34cc3d1b21a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:08.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.063+0000 7f5c4be00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:12:08.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.063+0000 7f5c4be00640 1 -- 172.21.15.28:0/3372728952 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c580870e0 con 0x55d733891c70 2024-04-24T08:12:08.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.064+0000 7f5c743c1740 1 -- 172.21.15.28:0/3372728952 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c58085e50 con 0x55d733891c70 2024-04-24T08:12:08.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.064+0000 7f5c4be00640 1 -- 172.21.15.28:0/3372728952 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f5c58087280 con 0x55d733891c70 2024-04-24T08:12:08.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.064+0000 7f5c4be00640 1 -- 172.21.15.28:0/3372728952 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5c5809db90 con 0x55d733891c70 2024-04-24T08:12:08.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.064+0000 7f5c743c1740 10 client.25759 fetch_fsmap learned FSMap version 24 2024-04-24T08:12:08.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.064+0000 7f5c743c1740 1 -- 172.21.15.28:0/3372728952 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d7335003f0 con 0x55d733891c70 2024-04-24T08:12:08.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.064+0000 7f5c4be00640 1 -- 172.21.15.28:0/3372728952 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c58085e50 con 0x55d733891c70 2024-04-24T08:12:08.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.064+0000 7f5c4be00640 1 -- 172.21.15.28:0/3372728952 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f5c5809b860 con 0x55d733891c70 2024-04-24T08:12:08.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.064+0000 7f5c743c1740 10 client.25759 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:12:08.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.064+0000 7f5c743c1740 10 client.25759 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:12:08.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.064+0000 7f5c743c1740 20 client.25759 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:08.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.064+0000 7f5c743c1740 1 --2- 172.21.15.28:0/3372728952 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55d7338ac260 0x55d7338ae650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.064+0000 7f5c743c1740 4 client.25759 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:08.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.064+0000 7f5c743c1740 1 -- 172.21.15.28:0/3372728952 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d7337fbbf0 con 0x55d7338ac260 2024-04-24T08:12:08.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.065+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/3372728952 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55d7338ac260 0x55d7338ae650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.066+0000 7f5c4be00640 10 client.25759 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:12:08.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.066+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/3372728952 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55d7338ac260 0x55d7338ae650 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=dd7a808bb5344024 server_cookie=f553036664c03ce8 in_seq=0 out_seq=0 2024-04-24T08:12:08.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.067+0000 7f5c4be00640 1 -- 172.21.15.28:0/3372728952 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d7337fbbf0 con 0x55d7338ac260 2024-04-24T08:12:08.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.067+0000 7f5c4be00640 10 client.25759 handle_command_reply: tid=0 2024-04-24T08:12:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.067+0000 7f5c743c1740 1 client.25759 shutdown 2024-04-24T08:12:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.067+0000 7f5c743c1740 1 -- 172.21.15.28:0/3372728952 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 msgr2=0x55d733898fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.067+0000 7f5c743c1740 1 --2- 172.21.15.28:0/3372728952 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 0x55d733898fe0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f5c5800c6a0 tx=0x7f5c58099220 comp rx=0 tx=0).stop 2024-04-24T08:12:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.067+0000 7f5c743c1740 1 -- 172.21.15.28:0/3372728952 shutdown_connections 2024-04-24T08:12:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.067+0000 7f5c743c1740 1 --2- 172.21.15.28:0/3372728952 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55d7338ac260 0x55d7338ae650 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.067+0000 7f5c743c1740 1 --2- 172.21.15.28:0/3372728952 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d73389c340 0x55d733899580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.067+0000 7f5c743c1740 1 --2- 172.21.15.28:0/3372728952 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 0x55d733898fe0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.067+0000 7f5c743c1740 1 --2- 172.21.15.28:0/3372728952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d733891350 0x55d73389bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.067+0000 7f5c743c1740 1 -- 172.21.15.28:0/3372728952 >> 172.21.15.28:0/3372728952 conn(0x55d73388d670 msgr2=0x55d733893470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.067+0000 7f5c4be00640 0 client.25759 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:12:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.067+0000 7f5c743c1740 1 -- 172.21.15.28:0/3372728952 shutdown_connections 2024-04-24T08:12:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.067+0000 7f5c743c1740 1 -- 172.21.15.28:0/3372728952 wait complete. 2024-04-24T08:12:08.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.068+0000 7f5c743c1740 20 client.25759 trim_cache size 0 max 16384 2024-04-24T08:12:08.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.082+0000 7f5c743c1740 1 Processor -- start 2024-04-24T08:12:08.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.082+0000 7f5c743c1740 1 -- start start 2024-04-24T08:12:08.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.083+0000 7f5c743c1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891350 0x55d7338a8080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.083+0000 7f5c743c1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d733891c70 0x55d73389b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.083+0000 7f5c743c1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d73389c340 0x55d7338b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.083+0000 7f5c743c1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c1ae460 con 0x55d73389c340 2024-04-24T08:12:08.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.083+0000 7f5c743c1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c1abd30 con 0x55d733891c70 2024-04-24T08:12:08.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.083+0000 7f5c743c1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c1a9b80 con 0x55d733891350 2024-04-24T08:12:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.083+0000 7f5c6a000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d733891c70 0x55d73389b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.083+0000 7f5c6b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d73389c340 0x55d7338b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.083+0000 7f5c6a000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d733891c70 0x55d73389b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40388/0 (socket says 172.21.15.28:40388) 2024-04-24T08:12:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.083+0000 7f5c6a000640 1 -- 172.21.15.28:0/4072538530 learned_addr learned my addr 172.21.15.28:0/4072538530 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.083+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/4072538530 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891350 0x55d7338a8080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.084+0000 7f5c6a000640 1 -- 172.21.15.28:0/4072538530 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891350 msgr2=0x55d7338a8080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.084+0000 7f5c6a000640 1 --2- 172.21.15.28:0/4072538530 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891350 0x55d7338a8080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.084+0000 7f5c6a000640 1 -- 172.21.15.28:0/4072538530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d73389c340 msgr2=0x55d7338b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.084+0000 7f5c6a000640 1 --2- 172.21.15.28:0/4072538530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d73389c340 0x55d7338b3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.084+0000 7f5c6a000640 1 -- 172.21.15.28:0/4072538530 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d7335ca7d0 con 0x55d733891c70 2024-04-24T08:12:08.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.084+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/4072538530 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891350 0x55d7338a8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.084+0000 7f5c6b400640 1 --2- 172.21.15.28:0/4072538530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d73389c340 0x55d7338b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.084+0000 7f5c6a000640 1 --2- 172.21.15.28:0/4072538530 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d733891c70 0x55d73389b820 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5c5800cf90 tx=0x7f5c58098ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a6c7f5f908ae3d3e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.085+0000 7f5c56a00640 1 -- 172.21.15.28:0/4072538530 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5809b7d0 con 0x55d733891c70 2024-04-24T08:12:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.085+0000 7f5c56a00640 1 -- 172.21.15.28:0/4072538530 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c58085e50 con 0x55d733891c70 2024-04-24T08:12:08.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.085+0000 7f5c56a00640 1 -- 172.21.15.28:0/4072538530 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5809bd60 con 0x55d733891c70 2024-04-24T08:12:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.085+0000 7f5c743c1740 1 -- 172.21.15.28:0/4072538530 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d733891c70 msgr2=0x55d73389b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.085+0000 7f5c743c1740 1 --2- 172.21.15.28:0/4072538530 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d733891c70 0x55d73389b820 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f5c5800cf90 tx=0x7f5c58098ce0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.086+0000 7f5c743c1740 1 -- 172.21.15.28:0/4072538530 shutdown_connections 2024-04-24T08:12:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.086+0000 7f5c743c1740 1 --2- 172.21.15.28:0/4072538530 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d73389c340 0x55d7338b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.086+0000 7f5c743c1740 1 --2- 172.21.15.28:0/4072538530 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d733891c70 0x55d73389b820 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.086+0000 7f5c743c1740 1 --2- 172.21.15.28:0/4072538530 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891350 0x55d7338a8080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.086+0000 7f5c743c1740 1 -- 172.21.15.28:0/4072538530 >> 172.21.15.28:0/4072538530 conn(0x55d73388d670 msgr2=0x55d733894340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.086+0000 7f5c743c1740 1 -- 172.21.15.28:0/4072538530 shutdown_connections 2024-04-24T08:12:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.086+0000 7f5c743c1740 1 -- 172.21.15.28:0/4072538530 wait complete. 2024-04-24T08:12:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.086+0000 7f5c743c1740 1 -- start start 2024-04-24T08:12:08.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.087+0000 7f5c743c1740 1 Processor -- start 2024-04-24T08:12:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.087+0000 7f5c743c1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d733891350 0x55d7338a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.087+0000 7f5c743c1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 0x55d7338965b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.087+0000 7f5c743c1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d73389c340 0x55d733896bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.087+0000 7f5c743c1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c1aa1f0 con 0x55d73389c340 2024-04-24T08:12:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.087+0000 7f5c743c1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c1ab660 con 0x55d733891350 2024-04-24T08:12:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.087+0000 7f5c743c1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6c1aa8c0 con 0x55d733891c70 2024-04-24T08:12:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.087+0000 7f5c743c1740 10 client.0 _ll_register_callbacks cb 0x7f5c5809c540 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.088+0000 7f5c6aa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d733891350 0x55d7338a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.088+0000 7f5c6a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 0x55d7338965b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.088+0000 7f5c6a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 0x55d7338965b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:33476/0 (socket says 172.21.15.28:33476) 2024-04-24T08:12:08.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.088+0000 7f5c6a000640 1 -- 172.21.15.28:0/1815742590 learned_addr learned my addr 172.21.15.28:0/1815742590 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.088+0000 7f5c6b400640 1 --2- 172.21.15.28:0/1815742590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d73389c340 0x55d733896bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.089+0000 7f5c6a000640 1 -- 172.21.15.28:0/1815742590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d733891350 msgr2=0x55d7338a4a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.089+0000 7f5c6a000640 1 --2- 172.21.15.28:0/1815742590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d733891350 0x55d7338a4a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.089+0000 7f5c6a000640 1 -- 172.21.15.28:0/1815742590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d73389c340 msgr2=0x55d733896bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:08.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.089+0000 7f5c6a000640 1 --2- 172.21.15.28:0/1815742590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d73389c340 0x55d733896bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.089+0000 7f5c6a000640 1 -- 172.21.15.28:0/1815742590 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d73389d3b0 con 0x55d733891c70 2024-04-24T08:12:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.089+0000 7f5c6a000640 1 --2- 172.21.15.28:0/1815742590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 0x55d7338965b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5c5809abc0 tx=0x7f5c5809c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=28244be72d55d23f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.089+0000 7f5c4be00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:12:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.089+0000 7f5c4be00640 1 -- 172.21.15.28:0/1815742590 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5809bf60 con 0x55d733891c70 2024-04-24T08:12:08.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.089+0000 7f5c743c1740 1 -- 172.21.15.28:0/1815742590 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c58085e50 con 0x55d733891c70 2024-04-24T08:12:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.089+0000 7f5c4be00640 1 -- 172.21.15.28:0/1815742590 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f5c58098660 con 0x55d733891c70 2024-04-24T08:12:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.090+0000 7f5c4be00640 1 -- 172.21.15.28:0/1815742590 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f5c5809dd30 con 0x55d733891c70 2024-04-24T08:12:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.090+0000 7f5c743c1740 10 client.25765 fetch_fsmap learned FSMap version 24 2024-04-24T08:12:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.090+0000 7f5c743c1740 1 -- 172.21.15.28:0/1815742590 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55d73389d3b0 con 0x55d733891c70 2024-04-24T08:12:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.090+0000 7f5c4be00640 1 -- 172.21.15.28:0/1815742590 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c58085e50 con 0x55d733891c70 2024-04-24T08:12:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.091+0000 7f5c4be00640 1 -- 172.21.15.28:0/1815742590 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f5c58098860 con 0x55d733891c70 2024-04-24T08:12:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.091+0000 7f5c743c1740 10 client.25765 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:12:08.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.091+0000 7f5c743c1740 10 client.25765 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:12:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.091+0000 7f5c743c1740 20 client.25765 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.091+0000 7f5c743c1740 1 --2- 172.21.15.28:0/1815742590 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55d733912b60 0x55d733914f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.091+0000 7f5c743c1740 4 client.25765 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:12:08.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.091+0000 7f5c743c1740 1 -- 172.21.15.28:0/1815742590 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d7337f5a60 con 0x55d733912b60 2024-04-24T08:12:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.091+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/1815742590 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55d733912b60 0x55d733914f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.092+0000 7f5c6aa00640 1 --2- 172.21.15.28:0/1815742590 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55d733912b60 0x55d733914f50 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a91493af678839db server_cookie=608495bc3ef42797 in_seq=0 out_seq=0 2024-04-24T08:12:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.092+0000 7f5c4be00640 10 client.25765 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:12:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.092+0000 7f5c4be00640 1 -- 172.21.15.28:0/1815742590 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d7337f5a60 con 0x55d733912b60 2024-04-24T08:12:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.092+0000 7f5c4be00640 10 client.25765 handle_command_reply: tid=0 2024-04-24T08:12:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.092+0000 7f5c743c1740 1 client.25765 shutdown 2024-04-24T08:12:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.092+0000 7f5c743c1740 1 -- 172.21.15.28:0/1815742590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 msgr2=0x55d7338965b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.092+0000 7f5c743c1740 1 --2- 172.21.15.28:0/1815742590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 0x55d7338965b0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f5c5809abc0 tx=0x7f5c5809c910 comp rx=0 tx=0).stop 2024-04-24T08:12:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.093+0000 7f5c743c1740 1 -- 172.21.15.28:0/1815742590 shutdown_connections 2024-04-24T08:12:08.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.093+0000 7f5c743c1740 1 --2- 172.21.15.28:0/1815742590 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55d733912b60 0x55d733914f50 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.093+0000 7f5c743c1740 1 --2- 172.21.15.28:0/1815742590 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d73389c340 0x55d733896bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.093+0000 7f5c4be00640 0 client.25765 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:12:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.093+0000 7f5c743c1740 1 --2- 172.21.15.28:0/1815742590 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d733891c70 0x55d7338965b0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.093+0000 7f5c743c1740 1 --2- 172.21.15.28:0/1815742590 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d733891350 0x55d7338a4a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.093+0000 7f5c743c1740 1 -- 172.21.15.28:0/1815742590 >> 172.21.15.28:0/1815742590 conn(0x55d73388d670 msgr2=0x55d7338836c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.093+0000 7f5c743c1740 1 -- 172.21.15.28:0/1815742590 shutdown_connections 2024-04-24T08:12:08.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.093+0000 7f5c743c1740 1 -- 172.21.15.28:0/1815742590 wait complete. 2024-04-24T08:12:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.093+0000 7f5c743c1740 20 client.25765 trim_cache size 0 max 16384 2024-04-24T08:12:08.095 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:12:08.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.094+0000 7f5c6be00640 1 -- 172.21.15.28:0/4007236878 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5c500a3da0 msgr2=0x7f5c500a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.094+0000 7f5c6be00640 1 --2- 172.21.15.28:0/4007236878 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5c500a3da0 0x7f5c500a6260 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f5c64009b30 tx=0x7f5c64009500 comp rx=0 tx=0).stop 2024-04-24T08:12:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.094+0000 7f5c6be00640 1 -- 172.21.15.28:0/4007236878 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c6c074cf0 msgr2=0x7f5c6c1a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.094+0000 7f5c6be00640 1 --2- 172.21.15.28:0/4007236878 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c6c074cf0 0x7f5c6c1a0bf0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f5c5800e790 tx=0x7f5c5800ec50 comp rx=0 tx=0).stop 2024-04-24T08:12:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.094+0000 7f5c6be00640 1 -- 172.21.15.28:0/4007236878 shutdown_connections 2024-04-24T08:12:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.094+0000 7f5c6be00640 1 --2- 172.21.15.28:0/4007236878 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5c500a3da0 0x7f5c500a6260 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.094+0000 7f5c6be00640 1 --2- 172.21.15.28:0/4007236878 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c6c10e740 0x7f5c6c19a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.094+0000 7f5c6be00640 1 --2- 172.21.15.28:0/4007236878 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c6c074cf0 0x7f5c6c1a0bf0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.094+0000 7f5c6be00640 1 --2- 172.21.15.28:0/4007236878 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c6c0743d0 0x7f5c6c1a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.094+0000 7f5c6be00640 1 -- 172.21.15.28:0/4007236878 >> 172.21.15.28:0/4007236878 conn(0x7f5c6c0fd9d0 msgr2=0x7f5c6c0fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.095+0000 7f5c6be00640 1 -- 172.21.15.28:0/4007236878 shutdown_connections 2024-04-24T08:12:08.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.095+0000 7f5c6be00640 1 -- 172.21.15.28:0/4007236878 wait complete. 2024-04-24T08:12:08.117 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:12:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.293+0000 7f1f2ce00640 1 -- 172.21.15.28:0/316530855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f28102110 msgr2=0x7f1f281024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.293+0000 7f1f2ce00640 1 --2- 172.21.15.28:0/316530855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f28102110 0x7f1f281024f0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1f18009900 tx=0x7f1f1802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.293+0000 7f1f2ce00640 1 -- 172.21.15.28:0/316530855 shutdown_connections 2024-04-24T08:12:08.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.293+0000 7f1f2ce00640 1 --2- 172.21.15.28:0/316530855 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f281071e0 0x7f1f2810edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.293+0000 7f1f2ce00640 1 --2- 172.21.15.28:0/316530855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f28102ac0 0x7f1f28106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.293+0000 7f1f2ce00640 1 --2- 172.21.15.28:0/316530855 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f28102110 0x7f1f281024f0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.293+0000 7f1f2ce00640 1 -- 172.21.15.28:0/316530855 >> 172.21.15.28:0/316530855 conn(0x7f1f280fd9d0 msgr2=0x7f1f280ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:08.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.293+0000 7f1f2ce00640 1 -- 172.21.15.28:0/316530855 shutdown_connections 2024-04-24T08:12:08.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.293+0000 7f1f2ce00640 1 -- 172.21.15.28:0/316530855 wait complete. 2024-04-24T08:12:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.294+0000 7f1f2ce00640 1 Processor -- start 2024-04-24T08:12:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.294+0000 7f1f2ce00640 1 -- start start 2024-04-24T08:12:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.295+0000 7f1f2ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f28102110 0x7f1f28196590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.295+0000 7f1f2ce00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f281071e0 0x7f1f28196ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.295+0000 7f1f2ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f2819ae50 0x7f1f2819b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.295+0000 7f1f2ce00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f281100f0 con 0x7f1f2819ae50 2024-04-24T08:12:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.295+0000 7f1f2ce00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1f2810ff70 con 0x7f1f281071e0 2024-04-24T08:12:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.295+0000 7f1f2ce00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1f28110270 con 0x7f1f28102110 2024-04-24T08:12:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.295+0000 7f1f27400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f28102110 0x7f1f28196590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.295+0000 7f1f27400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f28102110 0x7f1f28196590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:33496/0 (socket says 172.21.15.28:33496) 2024-04-24T08:12:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.295+0000 7f1f27400640 1 -- 172.21.15.28:0/1930411135 learned_addr learned my addr 172.21.15.28:0/1930411135 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.295+0000 7f1f26a00640 1 --2- 172.21.15.28:0/1930411135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f281071e0 0x7f1f28196ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.295+0000 7f1f27e00640 1 --2- 172.21.15.28:0/1930411135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f2819ae50 0x7f1f2819b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.296+0000 7f1f27400640 1 -- 172.21.15.28:0/1930411135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f281071e0 msgr2=0x7f1f28196ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.296+0000 7f1f27400640 1 --2- 172.21.15.28:0/1930411135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f281071e0 0x7f1f28196ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.296+0000 7f1f27400640 1 -- 172.21.15.28:0/1930411135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f2819ae50 msgr2=0x7f1f2819b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.296+0000 7f1f27400640 1 --2- 172.21.15.28:0/1930411135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f2819ae50 0x7f1f2819b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.296+0000 7f1f27400640 1 -- 172.21.15.28:0/1930411135 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1f2819b940 con 0x7f1f28102110 2024-04-24T08:12:08.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.296+0000 7f1f27400640 1 --2- 172.21.15.28:0/1930411135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f28102110 0x7f1f28196590 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1f18004770 tx=0x7f1f18003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:08.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.296+0000 7f1f1fe00640 1 -- 172.21.15.28:0/1930411135 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f1803f070 con 0x7f1f28102110 2024-04-24T08:12:08.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.296+0000 7f1f1fe00640 1 -- 172.21.15.28:0/1930411135 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f18031ca0 con 0x7f1f28102110 2024-04-24T08:12:08.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.296+0000 7f1f1fe00640 1 -- 172.21.15.28:0/1930411135 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f18035030 con 0x7f1f28102110 2024-04-24T08:12:08.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.297+0000 7f1f2ce00640 1 -- 172.21.15.28:0/1930411135 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1f2819bbd0 con 0x7f1f28102110 2024-04-24T08:12:08.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.297+0000 7f1f2ce00640 1 -- 172.21.15.28:0/1930411135 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1f281a3340 con 0x7f1f28102110 2024-04-24T08:12:08.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.300+0000 7f1f1fe00640 1 -- 172.21.15.28:0/1930411135 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1f18042070 con 0x7f1f28102110 2024-04-24T08:12:08.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.301+0000 7f1f31208740 1 Processor -- start 2024-04-24T08:12:08.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.302+0000 7f1f31208740 1 -- start start 2024-04-24T08:12:08.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.302+0000 7f1f31208740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a7ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.302+0000 7f1f31208740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a895f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.302+0000 7f1f31208740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a89b30 0x55c4b8a8bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.302+0000 7f1f31208740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c4b89e41d0 con 0x55c4b8a7eb40 2024-04-24T08:12:08.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.302+0000 7f1f31208740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c4b898a450 con 0x55c4b8a89b30 2024-04-24T08:12:08.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.302+0000 7f1f31208740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c4b89e0190 con 0x55c4b8a7f460 2024-04-24T08:12:08.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.302+0000 7f1f27e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a7ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.303+0000 7f1f27e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a7ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51748/0 (socket says 172.21.15.28:51748) 2024-04-24T08:12:08.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.303+0000 7f1f27e00640 1 -- 172.21.15.28:0/288853157 learned_addr learned my addr 172.21.15.28:0/288853157 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:08.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.303+0000 7f1f1fe00640 1 --2- 172.21.15.28:0/1930411135 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1f0c0a8130 0x7f1f0c0aa5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.303+0000 7f1f27400640 1 --2- 172.21.15.28:0/288853157 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a895f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.303+0000 7f1f1fe00640 1 -- 172.21.15.28:0/1930411135 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f1f180f0ed0 con 0x7f1f28102110 2024-04-24T08:12:08.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.303+0000 7f1f26a00640 1 --2- 172.21.15.28:0/288853157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a89b30 0x55c4b8a8bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.303+0000 7f1f27e00640 1 -- 172.21.15.28:0/288853157 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 msgr2=0x55c4b8a895f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:08.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.303+0000 7f1f27e00640 1 --2- 172.21.15.28:0/288853157 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a895f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.303+0000 7f1f27e00640 1 -- 172.21.15.28:0/288853157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a89b30 msgr2=0x55c4b8a8bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:08.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.303+0000 7f1f27e00640 1 --2- 172.21.15.28:0/288853157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a89b30 0x55c4b8a8bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.303+0000 7f1f27e00640 1 -- 172.21.15.28:0/288853157 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4b8a47310 con 0x55c4b8a7eb40 2024-04-24T08:12:08.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.303+0000 7f1f26a00640 1 --2- 172.21.15.28:0/288853157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a89b30 0x55c4b8a8bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:08.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.304+0000 7f1f27e00640 1 --2- 172.21.15.28:0/1930411135 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1f0c0a8130 0x7f1f0c0aa5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.304+0000 7f1f27e00640 1 --2- 172.21.15.28:0/288853157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a7ef20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1f20010520 tx=0x7f1f200109e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3bae876e8fc171f5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:08.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.304+0000 7f1f1cc00640 1 -- 172.21.15.28:0/288853157 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f20019070 con 0x55c4b8a7eb40 2024-04-24T08:12:08.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.305+0000 7f1f27e00640 1 --2- 172.21.15.28:0/1930411135 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1f0c0a8130 0x7f1f0c0aa5f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1f20021000 tx=0x7f1f2001e000 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:08.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.305+0000 7f1f1cc00640 1 -- 172.21.15.28:0/288853157 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f20003e10 con 0x55c4b8a7eb40 2024-04-24T08:12:08.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.305+0000 7f1f1cc00640 1 -- 172.21.15.28:0/288853157 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f2001d630 con 0x55c4b8a7eb40 2024-04-24T08:12:08.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.306+0000 7f1f31208740 1 -- 172.21.15.28:0/288853157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a7eb40 msgr2=0x55c4b8a7ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.306+0000 7f1f31208740 1 --2- 172.21.15.28:0/288853157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a7ef20 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f1f20010520 tx=0x7f1f200109e0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.306+0000 7f1f31208740 1 -- 172.21.15.28:0/288853157 shutdown_connections 2024-04-24T08:12:08.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.306+0000 7f1f31208740 1 --2- 172.21.15.28:0/288853157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a89b30 0x55c4b8a8bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.306+0000 7f1f31208740 1 --2- 172.21.15.28:0/288853157 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a895f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.306+0000 7f1f31208740 1 --2- 172.21.15.28:0/288853157 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a7ef20 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.306+0000 7f1f31208740 1 -- 172.21.15.28:0/288853157 >> 172.21.15.28:0/288853157 conn(0x55c4b8a7ae60 msgr2=0x55c4b8a7b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.306+0000 7f1f31208740 1 -- 172.21.15.28:0/288853157 shutdown_connections 2024-04-24T08:12:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.306+0000 7f1f31208740 1 -- 172.21.15.28:0/288853157 wait complete. 2024-04-24T08:12:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.307+0000 7f1f31208740 1 -- start start 2024-04-24T08:12:08.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.307+0000 7f1f31208740 1 Processor -- start 2024-04-24T08:12:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.308+0000 7f1f31208740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a895f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.308+0000 7f1f31208740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a868e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.308+0000 7f1f31208740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a89b30 0x55c4b8a87020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.308+0000 7f1f31208740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c4b8a8c9c0 con 0x55c4b8a89b30 2024-04-24T08:12:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.308+0000 7f1f31208740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c4b899d810 con 0x55c4b8a7eb40 2024-04-24T08:12:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.308+0000 7f1f31208740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c4b8a8cb40 con 0x55c4b8a7f460 2024-04-24T08:12:08.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.308+0000 7f1f31208740 10 client.0 _ll_register_callbacks cb 0x55c4b896b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:08.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.308+0000 7f1f27400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a895f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.308+0000 7f1f27400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a895f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40424/0 (socket says 172.21.15.28:40424) 2024-04-24T08:12:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.308+0000 7f1f27400640 1 -- 172.21.15.28:0/490497267 learned_addr learned my addr 172.21.15.28:0/490497267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.308+0000 7f1f26a00640 1 --2- 172.21.15.28:0/490497267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a868e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.308+0000 7f1f27e00640 1 --2- 172.21.15.28:0/490497267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a89b30 0x55c4b8a87020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.309+0000 7f1f27400640 1 -- 172.21.15.28:0/490497267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 msgr2=0x55c4b8a868e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.309+0000 7f1f27400640 1 --2- 172.21.15.28:0/490497267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a868e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.309+0000 7f1f27400640 1 -- 172.21.15.28:0/490497267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a89b30 msgr2=0x55c4b8a87020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.309+0000 7f1f27400640 1 --2- 172.21.15.28:0/490497267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a89b30 0x55c4b8a87020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.309+0000 7f1f27400640 1 -- 172.21.15.28:0/490497267 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4b89ae3f0 con 0x55c4b8a7eb40 2024-04-24T08:12:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.309+0000 7f1f27e00640 1 --2- 172.21.15.28:0/490497267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a89b30 0x55c4b8a87020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.309+0000 7f1f26a00640 1 --2- 172.21.15.28:0/490497267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a868e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.309+0000 7f1f27400640 1 --2- 172.21.15.28:0/490497267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a895f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1f180b9880 tx=0x7f1f1808b920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c28ad9dd49a268ec server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.309+0000 7f1f11600640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:12:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.309+0000 7f1f11600640 1 -- 172.21.15.28:0/490497267 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f18049070 con 0x55c4b8a7eb40 2024-04-24T08:12:08.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.309+0000 7f1f11600640 1 -- 172.21.15.28:0/490497267 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f1f1803b4f0 con 0x55c4b8a7eb40 2024-04-24T08:12:08.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.310+0000 7f1f31208740 1 -- 172.21.15.28:0/490497267 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f20003e10 con 0x55c4b8a7eb40 2024-04-24T08:12:08.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.310+0000 7f1f11600640 1 -- 172.21.15.28:0/490497267 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1f180b9150 con 0x55c4b8a7eb40 2024-04-24T08:12:08.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.310+0000 7f1f31208740 10 client.25733 fetch_fsmap learned FSMap version 24 2024-04-24T08:12:08.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.310+0000 7f1f31208740 1 -- 172.21.15.28:0/490497267 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c4b86f63f0 con 0x55c4b8a7eb40 2024-04-24T08:12:08.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.310+0000 7f1f11600640 1 -- 172.21.15.28:0/490497267 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f20003e10 con 0x55c4b8a7eb40 2024-04-24T08:12:08.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.310+0000 7f1f11600640 1 -- 172.21.15.28:0/490497267 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f1f180ee820 con 0x55c4b8a7eb40 2024-04-24T08:12:08.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.311+0000 7f1f31208740 10 client.25733 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:12:08.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.311+0000 7f1f31208740 10 client.25733 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:12:08.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.311+0000 7f1f31208740 20 client.25733 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:08.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.311+0000 7f1f31208740 1 --2- 172.21.15.28:0/490497267 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55c4b8a99db0 0x55c4b8a9c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.311+0000 7f1f31208740 4 client.25733 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:08.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.311+0000 7f1f31208740 1 -- 172.21.15.28:0/490497267 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4b89f1bf0 con 0x55c4b8a99db0 2024-04-24T08:12:08.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.311+0000 7f1f26a00640 1 --2- 172.21.15.28:0/490497267 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55c4b8a99db0 0x55c4b8a9c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.312+0000 7f1f26a00640 1 --2- 172.21.15.28:0/490497267 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55c4b8a99db0 0x55c4b8a9c1d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a4e34f19fb4872f server_cookie=8e6f5c91a8919434 in_seq=0 out_seq=0 2024-04-24T08:12:08.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.312+0000 7f1f11600640 10 client.25733 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:08.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.313+0000 7f1f11600640 1 -- 172.21.15.28:0/490497267 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c4b89f1bf0 con 0x55c4b8a99db0 2024-04-24T08:12:08.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.313+0000 7f1f11600640 10 client.25733 handle_command_reply: tid=0 2024-04-24T08:12:08.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.314+0000 7f1f31208740 1 client.25733 shutdown 2024-04-24T08:12:08.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.314+0000 7f1f31208740 1 -- 172.21.15.28:0/490497267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7eb40 msgr2=0x55c4b8a895f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.314+0000 7f1f31208740 1 --2- 172.21.15.28:0/490497267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a895f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f1f180b9880 tx=0x7f1f1808b920 comp rx=0 tx=0).stop 2024-04-24T08:12:08.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.314+0000 7f1f31208740 1 -- 172.21.15.28:0/490497267 shutdown_connections 2024-04-24T08:12:08.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.314+0000 7f1f31208740 1 --2- 172.21.15.28:0/490497267 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55c4b8a99db0 0x55c4b8a9c1d0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.314+0000 7f1f31208740 1 --2- 172.21.15.28:0/490497267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a89b30 0x55c4b8a87020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.314+0000 7f1f31208740 1 --2- 172.21.15.28:0/490497267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a868e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.314+0000 7f1f31208740 1 --2- 172.21.15.28:0/490497267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a895f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.314+0000 7f1f31208740 1 -- 172.21.15.28:0/490497267 >> 172.21.15.28:0/490497267 conn(0x55c4b8a7ae60 msgr2=0x55c4b8a80b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:08.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.314+0000 7f1f11600640 0 client.25733 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:08.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.315+0000 7f1f31208740 1 -- 172.21.15.28:0/490497267 shutdown_connections 2024-04-24T08:12:08.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.315+0000 7f1f31208740 1 -- 172.21.15.28:0/490497267 wait complete. 2024-04-24T08:12:08.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.315+0000 7f1f31208740 20 client.25733 trim_cache size 0 max 16384 2024-04-24T08:12:08.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.344+0000 7f1f31208740 1 Processor -- start 2024-04-24T08:12:08.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.344+0000 7f1f31208740 1 -- start start 2024-04-24T08:12:08.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.344+0000 7f1f31208740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7ae60 0x55c4b8a86880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.344+0000 7f1f31208740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a958b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.345+0000 7f1f31208740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a89120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.345+0000 7f1f31208740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f281ae2f0 con 0x55c4b8a7eb40 2024-04-24T08:12:08.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.345+0000 7f1f31208740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1f281abbc0 con 0x55c4b8a7ae60 2024-04-24T08:12:08.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.345+0000 7f1f31208740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1f281a9a10 con 0x55c4b8a7f460 2024-04-24T08:12:08.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.345+0000 7f1f27400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7ae60 0x55c4b8a86880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.345+0000 7f1f27400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7ae60 0x55c4b8a86880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40428/0 (socket says 172.21.15.28:40428) 2024-04-24T08:12:08.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.345+0000 7f1f26a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a958b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.345+0000 7f1f27400640 1 -- 172.21.15.28:0/4104410372 learned_addr learned my addr 172.21.15.28:0/4104410372 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:08.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.345+0000 7f1f27400640 1 -- 172.21.15.28:0/4104410372 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 msgr2=0x55c4b8a89120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:08.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.345+0000 7f1f27400640 1 --2- 172.21.15.28:0/4104410372 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a89120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.345+0000 7f1f27400640 1 -- 172.21.15.28:0/4104410372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a7eb40 msgr2=0x55c4b8a958b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:08.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.345+0000 7f1f27400640 1 --2- 172.21.15.28:0/4104410372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a958b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.345+0000 7f1f27400640 1 -- 172.21.15.28:0/4104410372 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4b87c07d0 con 0x55c4b8a7ae60 2024-04-24T08:12:08.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.346+0000 7f1f26a00640 1 --2- 172.21.15.28:0/4104410372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:08.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.346+0000 7f1f27400640 1 --2- 172.21.15.28:0/4104410372 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7ae60 0x55c4b8a86880 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1f180b9310 tx=0x7f1f1803b460 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ea88638d966bb0c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:08.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.346+0000 7f1f13e00640 1 -- 172.21.15.28:0/4104410372 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f18049070 con 0x55c4b8a7ae60 2024-04-24T08:12:08.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.346+0000 7f1f13e00640 1 -- 172.21.15.28:0/4104410372 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f180bd5b0 con 0x55c4b8a7ae60 2024-04-24T08:12:08.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.346+0000 7f1f13e00640 1 -- 172.21.15.28:0/4104410372 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f180c5690 con 0x55c4b8a7ae60 2024-04-24T08:12:08.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.347+0000 7f1f31208740 1 -- 172.21.15.28:0/4104410372 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7ae60 msgr2=0x55c4b8a86880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.347+0000 7f1f31208740 1 --2- 172.21.15.28:0/4104410372 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7ae60 0x55c4b8a86880 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f1f180b9310 tx=0x7f1f1803b460 comp rx=0 tx=0).stop 2024-04-24T08:12:08.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.347+0000 7f1f31208740 1 -- 172.21.15.28:0/4104410372 shutdown_connections 2024-04-24T08:12:08.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.347+0000 7f1f31208740 1 --2- 172.21.15.28:0/4104410372 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a958b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.347+0000 7f1f31208740 1 --2- 172.21.15.28:0/4104410372 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a89120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.347+0000 7f1f31208740 1 --2- 172.21.15.28:0/4104410372 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7ae60 0x55c4b8a86880 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.347+0000 7f1f31208740 1 -- 172.21.15.28:0/4104410372 >> 172.21.15.28:0/4104410372 conn(0x55c4b8a822b0 msgr2=0x55c4b8a86170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:08.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.348+0000 7f1f31208740 1 -- 172.21.15.28:0/4104410372 shutdown_connections 2024-04-24T08:12:08.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.348+0000 7f1f31208740 1 -- 172.21.15.28:0/4104410372 wait complete. 2024-04-24T08:12:08.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.348+0000 7f1f31208740 1 -- start start 2024-04-24T08:12:08.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.348+0000 7f1f31208740 1 Processor -- start 2024-04-24T08:12:08.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.349+0000 7f1f31208740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a8aa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.349+0000 7f1f31208740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a8af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.349+0000 7f1f31208740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a89b30 0x55c4b8a8b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.349+0000 7f1f31208740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1f281aa080 con 0x55c4b8a89b30 2024-04-24T08:12:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.349+0000 7f1f31208740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1f281ab4f0 con 0x55c4b8a7eb40 2024-04-24T08:12:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.349+0000 7f1f31208740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1f281aa750 con 0x55c4b8a7f460 2024-04-24T08:12:08.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.349+0000 7f1f31208740 10 client.0 _ll_register_callbacks cb 0x7f1f18005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.350+0000 7f1f26a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a8af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.350+0000 7f1f26a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a8af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:33532/0 (socket says 172.21.15.28:33532) 2024-04-24T08:12:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.350+0000 7f1f27e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a89b30 0x55c4b8a8b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.350+0000 7f1f26a00640 1 -- 172.21.15.28:0/3428162120 learned_addr learned my addr 172.21.15.28:0/3428162120 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.350+0000 7f1f27400640 1 --2- 172.21.15.28:0/3428162120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a8aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.350+0000 7f1f26a00640 1 -- 172.21.15.28:0/3428162120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7eb40 msgr2=0x55c4b8a8aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.350+0000 7f1f26a00640 1 --2- 172.21.15.28:0/3428162120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a8aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.350+0000 7f1f26a00640 1 -- 172.21.15.28:0/3428162120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a89b30 msgr2=0x55c4b8a8b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.350+0000 7f1f26a00640 1 --2- 172.21.15.28:0/3428162120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a89b30 0x55c4b8a8b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.350+0000 7f1f26a00640 1 -- 172.21.15.28:0/3428162120 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4b8a90300 con 0x55c4b8a7f460 2024-04-24T08:12:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.350+0000 7f1f27e00640 1 --2- 172.21.15.28:0/3428162120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a89b30 0x55c4b8a8b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.351+0000 7f1f26a00640 1 --2- 172.21.15.28:0/3428162120 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a8af60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1f14009b80 tx=0x7f1f140108c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a7af18351202d783 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.351+0000 7f1f11600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:12:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.351+0000 7f1f11600640 1 -- 172.21.15.28:0/3428162120 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1f14020300 con 0x55c4b8a7f460 2024-04-24T08:12:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.351+0000 7f1f31208740 1 -- 172.21.15.28:0/3428162120 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1f180bd5b0 con 0x55c4b8a7f460 2024-04-24T08:12:08.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.351+0000 7f1f11600640 1 -- 172.21.15.28:0/3428162120 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f1f140204a0 con 0x55c4b8a7f460 2024-04-24T08:12:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.351+0000 7f1f11600640 1 -- 172.21.15.28:0/3428162120 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1f1402f050 con 0x55c4b8a7f460 2024-04-24T08:12:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.351+0000 7f1f31208740 10 client.25780 fetch_fsmap learned FSMap version 24 2024-04-24T08:12:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.351+0000 7f1f31208740 1 -- 172.21.15.28:0/3428162120 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55c4b8b09090 con 0x55c4b8a7f460 2024-04-24T08:12:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.352+0000 7f1f11600640 1 -- 172.21.15.28:0/3428162120 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1f180bd5b0 con 0x55c4b8a7f460 2024-04-24T08:12:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.352+0000 7f1f11600640 1 -- 172.21.15.28:0/3428162120 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7f1f140283b0 con 0x55c4b8a7f460 2024-04-24T08:12:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.352+0000 7f1f31208740 10 client.25780 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:12:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.352+0000 7f1f31208740 10 client.25780 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:12:08.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.352+0000 7f1f31208740 20 client.25780 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.352+0000 7f1f31208740 1 --2- 172.21.15.28:0/3428162120 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55c4b8b09840 0x55c4b8b0bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.352+0000 7f1f31208740 4 client.25780 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:12:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.352+0000 7f1f31208740 1 -- 172.21.15.28:0/3428162120 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c4b89eba60 con 0x55c4b8b09840 2024-04-24T08:12:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.353+0000 7f1f27400640 1 --2- 172.21.15.28:0/3428162120 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55c4b8b09840 0x55c4b8b0bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.353+0000 7f1f27400640 1 --2- 172.21.15.28:0/3428162120 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55c4b8b09840 0x55c4b8b0bc60 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10a79aba94a30cb server_cookie=48bc7f487ba035aa in_seq=0 out_seq=0 2024-04-24T08:12:08.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.353+0000 7f1f11600640 10 client.25780 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.353+0000 7f1f11600640 1 -- 172.21.15.28:0/3428162120 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c4b89eba60 con 0x55c4b8b09840 2024-04-24T08:12:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.353+0000 7f1f11600640 10 client.25780 handle_command_reply: tid=0 2024-04-24T08:12:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.353+0000 7f1f31208740 1 client.25780 shutdown 2024-04-24T08:12:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.354+0000 7f1f31208740 1 -- 172.21.15.28:0/3428162120 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 msgr2=0x55c4b8a8af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.354+0000 7f1f31208740 1 --2- 172.21.15.28:0/3428162120 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a8af60 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f1f14009b80 tx=0x7f1f140108c0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.354+0000 7f1f31208740 1 -- 172.21.15.28:0/3428162120 shutdown_connections 2024-04-24T08:12:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.354+0000 7f1f31208740 1 --2- 172.21.15.28:0/3428162120 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55c4b8b09840 0x55c4b8b0bc60 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.354+0000 7f1f31208740 1 --2- 172.21.15.28:0/3428162120 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4b8a89b30 0x55c4b8a8b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.354+0000 7f1f31208740 1 --2- 172.21.15.28:0/3428162120 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4b8a7f460 0x55c4b8a8af60 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.354+0000 7f1f31208740 1 --2- 172.21.15.28:0/3428162120 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4b8a7eb40 0x55c4b8a8aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.354+0000 7f1f31208740 1 -- 172.21.15.28:0/3428162120 >> 172.21.15.28:0/3428162120 conn(0x55c4b8a822b0 msgr2=0x55c4b8a796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.354+0000 7f1f11600640 0 client.25780 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.354+0000 7f1f31208740 1 -- 172.21.15.28:0/3428162120 shutdown_connections 2024-04-24T08:12:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.354+0000 7f1f31208740 1 -- 172.21.15.28:0/3428162120 wait complete. 2024-04-24T08:12:08.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.354+0000 7f1f31208740 20 client.25780 trim_cache size 0 max 16384 2024-04-24T08:12:08.356 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:12:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.355+0000 7f1f2ce00640 1 -- 172.21.15.28:0/1930411135 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1f0c0a8130 msgr2=0x7f1f0c0aa5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.355+0000 7f1f2ce00640 1 --2- 172.21.15.28:0/1930411135 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1f0c0a8130 0x7f1f0c0aa5f0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1f20021000 tx=0x7f1f2001e000 comp rx=0 tx=0).stop 2024-04-24T08:12:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.355+0000 7f1f2ce00640 1 -- 172.21.15.28:0/1930411135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f28102110 msgr2=0x7f1f28196590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.355+0000 7f1f2ce00640 1 --2- 172.21.15.28:0/1930411135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f28102110 0x7f1f28196590 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f1f18004770 tx=0x7f1f18003c10 comp rx=0 tx=0).stop 2024-04-24T08:12:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.356+0000 7f1f2ce00640 1 -- 172.21.15.28:0/1930411135 shutdown_connections 2024-04-24T08:12:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.356+0000 7f1f2ce00640 1 --2- 172.21.15.28:0/1930411135 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1f0c0a8130 0x7f1f0c0aa5f0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.356+0000 7f1f2ce00640 1 --2- 172.21.15.28:0/1930411135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1f2819ae50 0x7f1f2819b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.356+0000 7f1f2ce00640 1 --2- 172.21.15.28:0/1930411135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1f281071e0 0x7f1f28196ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.356+0000 7f1f2ce00640 1 --2- 172.21.15.28:0/1930411135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1f28102110 0x7f1f28196590 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.356+0000 7f1f2ce00640 1 -- 172.21.15.28:0/1930411135 >> 172.21.15.28:0/1930411135 conn(0x7f1f280fd9d0 msgr2=0x7f1f280ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.356+0000 7f1f2ce00640 1 -- 172.21.15.28:0/1930411135 shutdown_connections 2024-04-24T08:12:08.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:08.356+0000 7f1f2ce00640 1 -- 172.21.15.28:0/1930411135 wait complete. 2024-04-24T08:12:08.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:08 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2794660472' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:12:08.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:08 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2794660472' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:12:08.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:08 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2794660472' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:12:09.381 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:12:09.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:12:09.381 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:12:09.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.593+0000 7fb504c00640 1 -- 172.21.15.28:0/1541587492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb500101140 msgr2=0x7fb500101520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.593+0000 7fb504c00640 1 --2- 172.21.15.28:0/1541587492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb500101140 0x7fb500101520 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0009900 tx=0x7fb4f002a410 comp rx=0 tx=0).stop 2024-04-24T08:12:09.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.593+0000 7fb504c00640 1 -- 172.21.15.28:0/1541587492 shutdown_connections 2024-04-24T08:12:09.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.594+0000 7fb504c00640 1 --2- 172.21.15.28:0/1541587492 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb500106210 0x7fb50010dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.594+0000 7fb504c00640 1 --2- 172.21.15.28:0/1541587492 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb500101af0 0x7fb500105b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.594+0000 7fb504c00640 1 --2- 172.21.15.28:0/1541587492 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb500101140 0x7fb500101520 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.594+0000 7fb504c00640 1 -- 172.21.15.28:0/1541587492 >> 172.21.15.28:0/1541587492 conn(0x7fb500078000 msgr2=0x7fb500078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:09.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.594+0000 7fb504c00640 1 -- 172.21.15.28:0/1541587492 shutdown_connections 2024-04-24T08:12:09.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.594+0000 7fb504c00640 1 -- 172.21.15.28:0/1541587492 wait complete. 2024-04-24T08:12:09.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.595+0000 7fb504c00640 1 Processor -- start 2024-04-24T08:12:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.595+0000 7fb504c00640 1 -- start start 2024-04-24T08:12:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.595+0000 7fb504c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb500101140 0x7fb5001a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.595+0000 7fb504c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb500106210 0x7fb5001a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.595+0000 7fb504c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb50019a950 0x7fb50019ae00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.595+0000 7fb504c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5001112d0 con 0x7fb50019a950 2024-04-24T08:12:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.595+0000 7fb504c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb500111150 con 0x7fb500101140 2024-04-24T08:12:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.595+0000 7fb504c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb500111450 con 0x7fb500106210 2024-04-24T08:12:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.595+0000 7fb4fde00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb500101140 0x7fb5001a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.595+0000 7fb4fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb50019a950 0x7fb50019ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.595+0000 7fb4fde00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb500101140 0x7fb5001a07a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40458/0 (socket says 172.21.15.28:40458) 2024-04-24T08:12:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.595+0000 7fb4fde00640 1 -- 172.21.15.28:0/2208558054 learned_addr learned my addr 172.21.15.28:0/2208558054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.595+0000 7fb4fd400640 1 --2- 172.21.15.28:0/2208558054 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb500106210 0x7fb5001a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.596+0000 7fb4fde00640 1 -- 172.21.15.28:0/2208558054 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb500106210 msgr2=0x7fb5001a0ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.596+0000 7fb4fde00640 1 --2- 172.21.15.28:0/2208558054 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb500106210 0x7fb5001a0ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.596+0000 7fb4fde00640 1 -- 172.21.15.28:0/2208558054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb50019a950 msgr2=0x7fb50019ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.596+0000 7fb4fde00640 1 --2- 172.21.15.28:0/2208558054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb50019a950 0x7fb50019ae00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.596+0000 7fb4fde00640 1 -- 172.21.15.28:0/2208558054 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb50019b470 con 0x7fb500101140 2024-04-24T08:12:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.596+0000 7fb4fde00640 1 --2- 172.21.15.28:0/2208558054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb500101140 0x7fb5001a07a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0006010 tx=0x7fb4f0004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.596+0000 7fb4f6a00640 1 -- 172.21.15.28:0/2208558054 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f003f030 con 0x7fb500101140 2024-04-24T08:12:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.596+0000 7fb504c00640 1 -- 172.21.15.28:0/2208558054 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb50019b700 con 0x7fb500101140 2024-04-24T08:12:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.596+0000 7fb4f6a00640 1 -- 172.21.15.28:0/2208558054 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4f0004440 con 0x7fb500101140 2024-04-24T08:12:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.596+0000 7fb4f6a00640 1 -- 172.21.15.28:0/2208558054 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0035070 con 0x7fb500101140 2024-04-24T08:12:09.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.597+0000 7fb504c00640 1 -- 172.21.15.28:0/2208558054 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb5001a7950 con 0x7fb500101140 2024-04-24T08:12:09.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.599+0000 7fb507a71740 1 Processor -- start 2024-04-24T08:12:09.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.600+0000 7fb4f6a00640 1 -- 172.21.15.28:0/2208558054 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb4f0042030 con 0x7fb500101140 2024-04-24T08:12:09.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.600+0000 7fb507a71740 1 -- start start 2024-04-24T08:12:09.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.600+0000 7fb507a71740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 0x557877ae7470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.600+0000 7fb507a71740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae79b0 0x557877af1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.600+0000 7fb507a71740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877af2080 0x557877af4470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.600+0000 7fb507a71740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557877a4ce70 con 0x557877ae7090 2024-04-24T08:12:09.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.600+0000 7fb507a71740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5578779f3380 con 0x557877ae79b0 2024-04-24T08:12:09.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.600+0000 7fb507a71740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557877a48e30 con 0x557877af2080 2024-04-24T08:12:09.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.600+0000 7fb4fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 0x557877ae7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.601+0000 7fb4fde00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae79b0 0x557877af1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.601+0000 7fb4fde00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae79b0 0x557877af1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40460/0 (socket says 172.21.15.28:40460) 2024-04-24T08:12:09.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.601+0000 7fb4fde00640 1 -- 172.21.15.28:0/321694733 learned_addr learned my addr 172.21.15.28:0/321694733 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:09.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.601+0000 7fb4fd400640 1 --2- 172.21.15.28:0/321694733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877af2080 0x557877af4470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.601+0000 7fb4fde00640 1 -- 172.21.15.28:0/321694733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877af2080 msgr2=0x557877af4470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:09.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.601+0000 7fb4fde00640 1 --2- 172.21.15.28:0/321694733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877af2080 0x557877af4470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.601+0000 7fb4fde00640 1 -- 172.21.15.28:0/321694733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 msgr2=0x557877ae7470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:09.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.601+0000 7fb4fde00640 1 --2- 172.21.15.28:0/321694733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 0x557877ae7470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.601+0000 7fb4fde00640 1 -- 172.21.15.28:0/321694733 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5578778277e0 con 0x557877ae79b0 2024-04-24T08:12:09.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.601+0000 7fb4fde00640 1 --2- 172.21.15.28:0/321694733 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae79b0 0x557877af1b40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb4f00ee1a0 tx=0x7fb4f00f4dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b3bc8adf6f969e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.601+0000 7fb4f6a00640 1 --2- 172.21.15.28:0/2208558054 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb4e40a3d20 0x7fb4e40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.602+0000 7fb4eb000640 1 -- 172.21.15.28:0/321694733 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0100020 con 0x557877ae79b0 2024-04-24T08:12:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.602+0000 7fb4eb000640 1 -- 172.21.15.28:0/321694733 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4f00bcb60 con 0x557877ae79b0 2024-04-24T08:12:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.602+0000 7fb4eb000640 1 -- 172.21.15.28:0/321694733 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f00f5040 con 0x557877ae79b0 2024-04-24T08:12:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.602+0000 7fb4f6a00640 1 -- 172.21.15.28:0/2208558054 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fb4f00f0b90 con 0x7fb500101140 2024-04-24T08:12:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.602+0000 7fb4fe800640 1 --2- 172.21.15.28:0/2208558054 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb4e40a3d20 0x7fb4e40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.603+0000 7fb507a71740 1 -- 172.21.15.28:0/321694733 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae79b0 msgr2=0x557877af1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.603+0000 7fb4fe800640 1 --2- 172.21.15.28:0/2208558054 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb4e40a3d20 0x7fb4e40a61e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8005fd0 tx=0x7fb4f8009500 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.603+0000 7fb507a71740 1 --2- 172.21.15.28:0/321694733 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae79b0 0x557877af1b40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fb4f00ee1a0 tx=0x7fb4f00f4dc0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.603+0000 7fb507a71740 1 -- 172.21.15.28:0/321694733 shutdown_connections 2024-04-24T08:12:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.603+0000 7fb507a71740 1 --2- 172.21.15.28:0/321694733 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877af2080 0x557877af4470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.603+0000 7fb507a71740 1 --2- 172.21.15.28:0/321694733 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae79b0 0x557877af1b40 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.603+0000 7fb507a71740 1 --2- 172.21.15.28:0/321694733 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 0x557877ae7470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.603+0000 7fb507a71740 1 -- 172.21.15.28:0/321694733 >> 172.21.15.28:0/321694733 conn(0x557877ae33b0 msgr2=0x557877ae3790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.603+0000 7fb507a71740 1 -- 172.21.15.28:0/321694733 shutdown_connections 2024-04-24T08:12:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.603+0000 7fb507a71740 1 -- 172.21.15.28:0/321694733 wait complete. 2024-04-24T08:12:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.603+0000 7fb507a71740 1 -- start start 2024-04-24T08:12:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.603+0000 7fb507a71740 1 Processor -- start 2024-04-24T08:12:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.604+0000 7fb507a71740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 0x557877aeef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.604+0000 7fb507a71740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae79b0 0x557877aef4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.604+0000 7fb507a71740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877af2080 0x557877aefa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.604+0000 7fb507a71740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x557877af4c30 con 0x557877ae7090 2024-04-24T08:12:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.604+0000 7fb507a71740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x557877a06660 con 0x557877ae79b0 2024-04-24T08:12:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.604+0000 7fb507a71740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x557877af4db0 con 0x557877af2080 2024-04-24T08:12:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.604+0000 7fb507a71740 10 client.0 _ll_register_callbacks cb 0x5578779f4aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:09.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.604+0000 7fb4fde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 0x557877aeef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.604+0000 7fb4fde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 0x557877aeef80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51796/0 (socket says 172.21.15.28:51796) 2024-04-24T08:12:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.604+0000 7fb4fde00640 1 -- 172.21.15.28:0/1286092689 learned_addr learned my addr 172.21.15.28:0/1286092689 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.604+0000 7fb4fe800640 1 --2- 172.21.15.28:0/1286092689 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877af2080 0x557877aefa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.604+0000 7fb4fd400640 1 --2- 172.21.15.28:0/1286092689 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae79b0 0x557877aef4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.605+0000 7fb4fde00640 1 -- 172.21.15.28:0/1286092689 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877af2080 msgr2=0x557877aefa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.605+0000 7fb4fde00640 1 --2- 172.21.15.28:0/1286092689 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877af2080 0x557877aefa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.605+0000 7fb4fde00640 1 -- 172.21.15.28:0/1286092689 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae79b0 msgr2=0x557877aef4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.605+0000 7fb4fde00640 1 --2- 172.21.15.28:0/1286092689 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae79b0 0x557877aef4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.605+0000 7fb4fde00640 1 -- 172.21.15.28:0/1286092689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557877887f70 con 0x557877ae7090 2024-04-24T08:12:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.605+0000 7fb4fde00640 1 --2- 172.21.15.28:0/1286092689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 0x557877aeef80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb4f00c51f0 tx=0x7fb4f003b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfa6ea8e50498a78 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.605+0000 7fb4df400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:12:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.605+0000 7fb4df400640 1 -- 172.21.15.28:0/1286092689 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0049030 con 0x557877ae7090 2024-04-24T08:12:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.605+0000 7fb4df400640 1 -- 172.21.15.28:0/1286092689 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fb4f00f47e0 con 0x557877ae7090 2024-04-24T08:12:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.605+0000 7fb507a71740 1 -- 172.21.15.28:0/1286092689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4f00bcb60 con 0x557877ae7090 2024-04-24T08:12:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.605+0000 7fb4df400640 1 -- 172.21.15.28:0/1286092689 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb4f0100070 con 0x557877ae7090 2024-04-24T08:12:09.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.605+0000 7fb507a71740 10 client.15651 fetch_fsmap learned FSMap version 24 2024-04-24T08:12:09.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.605+0000 7fb507a71740 1 -- 172.21.15.28:0/1286092689 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55787771f170 con 0x557877ae7090 2024-04-24T08:12:09.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.606+0000 7fb4df400640 1 -- 172.21.15.28:0/1286092689 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4f00bcb60 con 0x557877ae7090 2024-04-24T08:12:09.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.606+0000 7fb4df400640 1 -- 172.21.15.28:0/1286092689 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fb4f00fb040 con 0x557877ae7090 2024-04-24T08:12:09.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.606+0000 7fb507a71740 10 client.15651 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:12:09.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.606+0000 7fb507a71740 10 client.15651 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:12:09.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.606+0000 7fb507a71740 20 client.15651 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:09.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.606+0000 7fb507a71740 1 --2- 172.21.15.28:0/1286092689 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557877b01f20 0x557877b04310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.606+0000 7fb507a71740 4 client.15651 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:09.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.606+0000 7fb507a71740 1 -- 172.21.15.28:0/1286092689 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557877a5a890 con 0x557877b01f20 2024-04-24T08:12:09.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.606+0000 7fb4fd400640 1 --2- 172.21.15.28:0/1286092689 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557877b01f20 0x557877b04310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.607+0000 7fb4fd400640 1 --2- 172.21.15.28:0/1286092689 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557877b01f20 0x557877b04310 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b3b0d6ed6f26f24 server_cookie=32853353c3c58967 in_seq=0 out_seq=0 2024-04-24T08:12:09.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.607+0000 7fb4df400640 10 client.15651 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:09.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.608+0000 7fb4df400640 1 -- 172.21.15.28:0/1286092689 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557877a5a890 con 0x557877b01f20 2024-04-24T08:12:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.608+0000 7fb4df400640 10 client.15651 handle_command_reply: tid=0 2024-04-24T08:12:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.608+0000 7fb507a71740 1 client.15651 shutdown 2024-04-24T08:12:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.608+0000 7fb507a71740 1 -- 172.21.15.28:0/1286092689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 msgr2=0x557877aeef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.608+0000 7fb507a71740 1 --2- 172.21.15.28:0/1286092689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 0x557877aeef80 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fb4f00c51f0 tx=0x7fb4f003b4d0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.608+0000 7fb507a71740 1 -- 172.21.15.28:0/1286092689 shutdown_connections 2024-04-24T08:12:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.608+0000 7fb507a71740 1 --2- 172.21.15.28:0/1286092689 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557877b01f20 0x557877b04310 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.608+0000 7fb507a71740 1 --2- 172.21.15.28:0/1286092689 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877af2080 0x557877aefa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.608+0000 7fb507a71740 1 --2- 172.21.15.28:0/1286092689 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae79b0 0x557877aef4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.608+0000 7fb4df400640 0 client.15651 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.608+0000 7fb507a71740 1 --2- 172.21.15.28:0/1286092689 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 0x557877aeef80 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.608+0000 7fb507a71740 1 -- 172.21.15.28:0/1286092689 >> 172.21.15.28:0/1286092689 conn(0x557877ae33b0 msgr2=0x557877ae9070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.608+0000 7fb507a71740 1 -- 172.21.15.28:0/1286092689 shutdown_connections 2024-04-24T08:12:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.608+0000 7fb507a71740 1 -- 172.21.15.28:0/1286092689 wait complete. 2024-04-24T08:12:09.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.608+0000 7fb507a71740 20 client.15651 trim_cache size 0 max 16384 2024-04-24T08:12:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.627+0000 7fb507a71740 1 Processor -- start 2024-04-24T08:12:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.627+0000 7fb507a71740 1 -- start start 2024-04-24T08:12:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.627+0000 7fb507a71740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae7090 0x557877afda60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.627+0000 7fb507a71740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877ae79b0 0x557877ae9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.627+0000 7fb507a71740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877af2080 0x557877ae2390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.627+0000 7fb507a71740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5001b29d0 con 0x557877af2080 2024-04-24T08:12:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.627+0000 7fb507a71740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb5001b02a0 con 0x557877ae7090 2024-04-24T08:12:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.627+0000 7fb507a71740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb5001ae0f0 con 0x557877ae79b0 2024-04-24T08:12:09.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.628+0000 7fb4fde00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae7090 0x557877afda60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.628+0000 7fb4fe800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877af2080 0x557877ae2390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.628+0000 7fb4fde00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae7090 0x557877afda60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40484/0 (socket says 172.21.15.28:40484) 2024-04-24T08:12:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.628+0000 7fb4fde00640 1 -- 172.21.15.28:0/1908933314 learned_addr learned my addr 172.21.15.28:0/1908933314 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:09.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.628+0000 7fb4fd400640 1 --2- 172.21.15.28:0/1908933314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877ae79b0 0x557877ae9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.629 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:09 smithi117 ceph-mon[32438]: pgmap v57: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:12:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.628+0000 7fb4fde00640 1 -- 172.21.15.28:0/1908933314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877ae79b0 msgr2=0x557877ae9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.628+0000 7fb4fde00640 1 --2- 172.21.15.28:0/1908933314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877ae79b0 0x557877ae9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.628+0000 7fb4fde00640 1 -- 172.21.15.28:0/1908933314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877af2080 msgr2=0x557877ae2390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.628+0000 7fb4fde00640 1 --2- 172.21.15.28:0/1908933314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877af2080 0x557877ae2390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.628+0000 7fb4fde00640 1 -- 172.21.15.28:0/1908933314 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557877ae9d10 con 0x557877ae7090 2024-04-24T08:12:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.628+0000 7fb4fe800640 1 --2- 172.21.15.28:0/1908933314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877af2080 0x557877ae2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:09.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.629+0000 7fb4fde00640 1 --2- 172.21.15.28:0/1908933314 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae7090 0x557877afda60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb4f00b9530 tx=0x7fb4f00bb070 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d8dca2679707f5f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.629+0000 7fb4ea600640 1 -- 172.21.15.28:0/1908933314 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0049070 con 0x557877ae7090 2024-04-24T08:12:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.629+0000 7fb4ea600640 1 -- 172.21.15.28:0/1908933314 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4f00f4170 con 0x557877ae7090 2024-04-24T08:12:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.629+0000 7fb4ea600640 1 -- 172.21.15.28:0/1908933314 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f00bcb60 con 0x557877ae7090 2024-04-24T08:12:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.630+0000 7fb507a71740 1 -- 172.21.15.28:0/1908933314 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae7090 msgr2=0x557877afda60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.630+0000 7fb507a71740 1 --2- 172.21.15.28:0/1908933314 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae7090 0x557877afda60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fb4f00b9530 tx=0x7fb4f00bb070 comp rx=0 tx=0).stop 2024-04-24T08:12:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.630+0000 7fb507a71740 1 -- 172.21.15.28:0/1908933314 shutdown_connections 2024-04-24T08:12:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.630+0000 7fb507a71740 1 --2- 172.21.15.28:0/1908933314 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877af2080 0x557877ae2390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.630+0000 7fb507a71740 1 --2- 172.21.15.28:0/1908933314 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877ae79b0 0x557877ae9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.630+0000 7fb507a71740 1 --2- 172.21.15.28:0/1908933314 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877ae7090 0x557877afda60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.630+0000 7fb507a71740 1 -- 172.21.15.28:0/1908933314 >> 172.21.15.28:0/1908933314 conn(0x557877ae33b0 msgr2=0x557877ae1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.630+0000 7fb507a71740 1 -- 172.21.15.28:0/1908933314 shutdown_connections 2024-04-24T08:12:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.630+0000 7fb507a71740 1 -- 172.21.15.28:0/1908933314 wait complete. 2024-04-24T08:12:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.630+0000 7fb507a71740 1 -- start start 2024-04-24T08:12:09.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.630+0000 7fb507a71740 1 Processor -- start 2024-04-24T08:12:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb507a71740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 0x557877afa220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb507a71740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877ae79b0 0x557877af5630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb507a71740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877af2080 0x557877af5b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb507a71740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb5001ae760 con 0x557877ae7090 2024-04-24T08:12:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb507a71740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb5001afbd0 con 0x557877af2080 2024-04-24T08:12:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb507a71740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb5001aee30 con 0x557877ae79b0 2024-04-24T08:12:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb507a71740 10 client.0 _ll_register_callbacks cb 0x7fb4f8009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb4fde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 0x557877afa220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb4fde00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 0x557877afa220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:51814/0 (socket says 172.21.15.28:51814) 2024-04-24T08:12:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb4fde00640 1 -- 172.21.15.28:0/3236229011 learned_addr learned my addr 172.21.15.28:0/3236229011 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb4fe800640 1 --2- 172.21.15.28:0/3236229011 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877af2080 0x557877af5b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb4fd400640 1 --2- 172.21.15.28:0/3236229011 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877ae79b0 0x557877af5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb4fde00640 1 -- 172.21.15.28:0/3236229011 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877ae79b0 msgr2=0x557877af5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:09.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb4fde00640 1 --2- 172.21.15.28:0/3236229011 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877ae79b0 0x557877af5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb4fde00640 1 -- 172.21.15.28:0/3236229011 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877af2080 msgr2=0x557877af5b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb4fde00640 1 --2- 172.21.15.28:0/3236229011 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877af2080 0x557877af5b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb4fde00640 1 -- 172.21.15.28:0/3236229011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557877af6260 con 0x557877ae7090 2024-04-24T08:12:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.631+0000 7fb4fd400640 1 --2- 172.21.15.28:0/3236229011 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877ae79b0 0x557877af5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.632+0000 7fb4fde00640 1 --2- 172.21.15.28:0/3236229011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 0x557877afa220 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb4f00bae10 tx=0x7fb4f003b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88e84d9cf2e01e5c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.632+0000 7fb4df400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:12:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.632+0000 7fb4df400640 1 -- 172.21.15.28:0/3236229011 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4f0049050 con 0x557877ae7090 2024-04-24T08:12:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.632+0000 7fb507a71740 1 -- 172.21.15.28:0/3236229011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4f00f4170 con 0x557877ae7090 2024-04-24T08:12:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.632+0000 7fb4df400640 1 -- 172.21.15.28:0/3236229011 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fb4f00c5290 con 0x557877ae7090 2024-04-24T08:12:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.632+0000 7fb4df400640 1 -- 172.21.15.28:0/3236229011 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fb4f00b8e00 con 0x557877ae7090 2024-04-24T08:12:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.632+0000 7fb507a71740 10 client.15657 fetch_fsmap learned FSMap version 24 2024-04-24T08:12:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.632+0000 7fb507a71740 1 -- 172.21.15.28:0/3236229011 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x557877af6260 con 0x557877ae7090 2024-04-24T08:12:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.632+0000 7fb4df400640 1 -- 172.21.15.28:0/3236229011 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4f00f4170 con 0x557877ae7090 2024-04-24T08:12:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.633+0000 7fb4df400640 1 -- 172.21.15.28:0/3236229011 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fb4f00fb070 con 0x557877ae7090 2024-04-24T08:12:09.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.633+0000 7fb507a71740 10 client.15657 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:12:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.633+0000 7fb507a71740 10 client.15657 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:12:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.633+0000 7fb507a71740 20 client.15657 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.633+0000 7fb507a71740 1 --2- 172.21.15.28:0/3236229011 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557877b68600 0x557877b6a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.633+0000 7fb507a71740 4 client.15657 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:12:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.633+0000 7fb507a71740 1 -- 172.21.15.28:0/3236229011 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x557877a54700 con 0x557877b68600 2024-04-24T08:12:09.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.633+0000 7fb4fd400640 1 --2- 172.21.15.28:0/3236229011 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557877b68600 0x557877b6a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.634+0000 7fb4fd400640 1 --2- 172.21.15.28:0/3236229011 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557877b68600 0x557877b6a9f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a575ee682d18d8f9 server_cookie=e18536d3db93cd19 in_seq=0 out_seq=0 2024-04-24T08:12:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.634+0000 7fb4df400640 10 client.15657 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.635+0000 7fb4df400640 1 -- 172.21.15.28:0/3236229011 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x557877a54700 con 0x557877b68600 2024-04-24T08:12:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.635+0000 7fb4df400640 10 client.15657 handle_command_reply: tid=0 2024-04-24T08:12:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.635+0000 7fb507a71740 1 client.15657 shutdown 2024-04-24T08:12:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.635+0000 7fb507a71740 1 -- 172.21.15.28:0/3236229011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 msgr2=0x557877afa220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.635+0000 7fb507a71740 1 --2- 172.21.15.28:0/3236229011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 0x557877afa220 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fb4f00bae10 tx=0x7fb4f003b4d0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.635+0000 7fb507a71740 1 -- 172.21.15.28:0/3236229011 shutdown_connections 2024-04-24T08:12:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.635+0000 7fb507a71740 1 --2- 172.21.15.28:0/3236229011 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557877b68600 0x557877b6a9f0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.635+0000 7fb507a71740 1 --2- 172.21.15.28:0/3236229011 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x557877af2080 0x557877af5b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.635+0000 7fb507a71740 1 --2- 172.21.15.28:0/3236229011 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x557877ae79b0 0x557877af5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.635+0000 7fb507a71740 1 --2- 172.21.15.28:0/3236229011 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x557877ae7090 0x557877afa220 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.635+0000 7fb507a71740 1 -- 172.21.15.28:0/3236229011 >> 172.21.15.28:0/3236229011 conn(0x557877ae33b0 msgr2=0x557877ae2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.635+0000 7fb4df400640 0 client.15657 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.636+0000 7fb507a71740 1 -- 172.21.15.28:0/3236229011 shutdown_connections 2024-04-24T08:12:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.636+0000 7fb507a71740 1 -- 172.21.15.28:0/3236229011 wait complete. 2024-04-24T08:12:09.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.636+0000 7fb507a71740 20 client.15657 trim_cache size 0 max 16384 2024-04-24T08:12:09.639 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:12:09.639 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-24T08:12:09.640 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "4dae45ee-e120-4e4b-afab-d37db50bca01", 2024-04-24T08:12:09.640 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-24T08:12:09.640 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:12:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.637+0000 7fb504c00640 1 -- 172.21.15.28:0/2208558054 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb4e40a3d20 msgr2=0x7fb4e40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.637+0000 7fb504c00640 1 --2- 172.21.15.28:0/2208558054 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb4e40a3d20 0x7fb4e40a61e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb4f8005fd0 tx=0x7fb4f8009500 comp rx=0 tx=0).stop 2024-04-24T08:12:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.637+0000 7fb504c00640 1 -- 172.21.15.28:0/2208558054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb500101140 msgr2=0x7fb5001a07a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.637+0000 7fb504c00640 1 --2- 172.21.15.28:0/2208558054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb500101140 0x7fb5001a07a0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fb4f0006010 tx=0x7fb4f0004040 comp rx=0 tx=0).stop 2024-04-24T08:12:09.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.637+0000 7fb504c00640 1 -- 172.21.15.28:0/2208558054 shutdown_connections 2024-04-24T08:12:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.637+0000 7fb504c00640 1 --2- 172.21.15.28:0/2208558054 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb4e40a3d20 0x7fb4e40a61e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.637+0000 7fb504c00640 1 --2- 172.21.15.28:0/2208558054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb50019a950 0x7fb50019ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.637+0000 7fb504c00640 1 --2- 172.21.15.28:0/2208558054 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb500106210 0x7fb5001a0ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.637+0000 7fb504c00640 1 --2- 172.21.15.28:0/2208558054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb500101140 0x7fb5001a07a0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.637+0000 7fb504c00640 1 -- 172.21.15.28:0/2208558054 >> 172.21.15.28:0/2208558054 conn(0x7fb500078000 msgr2=0x7fb500069750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.637+0000 7fb504c00640 1 -- 172.21.15.28:0/2208558054 shutdown_connections 2024-04-24T08:12:09.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.637+0000 7fb504c00640 1 -- 172.21.15.28:0/2208558054 wait complete. 2024-04-24T08:12:09.651 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:09 smithi028 ceph-mon[27786]: pgmap v57: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:12:09.653 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:12:09.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:12:09.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:09 smithi194 ceph-mon[32836]: pgmap v57: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:12:09.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.844+0000 7fe7e6c00640 1 -- 172.21.15.28:0/3291066093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7e0103980 msgr2=0x7fe7e010be60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.844+0000 7fe7e6c00640 1 --2- 172.21.15.28:0/3291066093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7e0103980 0x7fe7e010be60 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fe7cc009900 tx=0x7fe7cc02a480 comp rx=0 tx=0).stop 2024-04-24T08:12:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.844+0000 7fe7e6c00640 1 -- 172.21.15.28:0/3291066093 shutdown_connections 2024-04-24T08:12:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.844+0000 7fe7e6c00640 1 --2- 172.21.15.28:0/3291066093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7e0103980 0x7fe7e010be60 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.844+0000 7fe7e6c00640 1 --2- 172.21.15.28:0/3291066093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7e010c480 0x7fe7e010e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.844+0000 7fe7e6c00640 1 --2- 172.21.15.28:0/3291066093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7e0103060 0x7fe7e0103440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.844+0000 7fe7e6c00640 1 -- 172.21.15.28:0/3291066093 >> 172.21.15.28:0/3291066093 conn(0x7fe7e00fc880 msgr2=0x7fe7e00feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.844+0000 7fe7e6c00640 1 -- 172.21.15.28:0/3291066093 shutdown_connections 2024-04-24T08:12:09.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.844+0000 7fe7e6c00640 1 -- 172.21.15.28:0/3291066093 wait complete. 2024-04-24T08:12:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.845+0000 7fe7e6c00640 1 Processor -- start 2024-04-24T08:12:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.845+0000 7fe7e6c00640 1 -- start start 2024-04-24T08:12:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.845+0000 7fe7e6c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7e0103060 0x7fe7e01a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.845+0000 7fe7e6c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7e0103980 0x7fe7e01a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.845+0000 7fe7e6c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7e010c480 0x7fe7e019a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.845+0000 7fe7e6c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e0112290 con 0x7fe7e010c480 2024-04-24T08:12:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.845+0000 7fe7e6c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e0112110 con 0x7fe7e0103980 2024-04-24T08:12:09.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.845+0000 7fe7e6c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e0112410 con 0x7fe7e0103060 2024-04-24T08:12:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.846+0000 7fe7e6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7e010c480 0x7fe7e019a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.846+0000 7fe7e6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7e010c480 0x7fe7e019a930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36324/0 (socket says 172.21.15.28:36324) 2024-04-24T08:12:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.846+0000 7fe7e6200640 1 -- 172.21.15.28:0/498934228 learned_addr learned my addr 172.21.15.28:0/498934228 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.846+0000 7fe7e4e00640 1 --2- 172.21.15.28:0/498934228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7e0103980 0x7fe7e01a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.846+0000 7fe7e5800640 1 --2- 172.21.15.28:0/498934228 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7e0103060 0x7fe7e01a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.846+0000 7fe7e6200640 1 -- 172.21.15.28:0/498934228 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7e0103060 msgr2=0x7fe7e01a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.846+0000 7fe7e6200640 1 --2- 172.21.15.28:0/498934228 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7e0103060 0x7fe7e01a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.846+0000 7fe7e6200640 1 -- 172.21.15.28:0/498934228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7e0103980 msgr2=0x7fe7e01a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.846+0000 7fe7e6200640 1 --2- 172.21.15.28:0/498934228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7e0103980 0x7fe7e01a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.846+0000 7fe7e6200640 1 -- 172.21.15.28:0/498934228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7e019b0b0 con 0x7fe7e010c480 2024-04-24T08:12:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.846+0000 7fe7e4e00640 1 --2- 172.21.15.28:0/498934228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7e0103980 0x7fe7e01a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.846+0000 7fe7e6200640 1 --2- 172.21.15.28:0/498934228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7e010c480 0x7fe7e019a930 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe7d800eb10 tx=0x7fe7d800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.847+0000 7fe7de000640 1 -- 172.21.15.28:0/498934228 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d8004b20 con 0x7fe7e010c480 2024-04-24T08:12:09.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.847+0000 7fe7de000640 1 -- 172.21.15.28:0/498934228 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7d8004cc0 con 0x7fe7e010c480 2024-04-24T08:12:09.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.847+0000 7fe7e6c00640 1 -- 172.21.15.28:0/498934228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7e019b3a0 con 0x7fe7e010c480 2024-04-24T08:12:09.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.847+0000 7fe7de000640 1 -- 172.21.15.28:0/498934228 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d8010760 con 0x7fe7e010c480 2024-04-24T08:12:09.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.847+0000 7fe7e6c00640 1 -- 172.21.15.28:0/498934228 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7e01a7550 con 0x7fe7e010c480 2024-04-24T08:12:09.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.850+0000 7fe7eaff3740 1 Processor -- start 2024-04-24T08:12:09.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.850+0000 7fe7de000640 1 -- 172.21.15.28:0/498934228 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe7d8005020 con 0x7fe7e010c480 2024-04-24T08:12:09.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.850+0000 7fe7eaff3740 1 -- start start 2024-04-24T08:12:09.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.850+0000 7fe7eaff3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f615a30 0x56446f615e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.850+0000 7fe7eaff3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f616350 0x56446f6204e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.850+0000 7fe7eaff3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f620a20 0x56446f622e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.850+0000 7fe7eaff3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56446f57b1f0 con 0x56446f620a20 2024-04-24T08:12:09.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.850+0000 7fe7eaff3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56446f521350 con 0x56446f616350 2024-04-24T08:12:09.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.850+0000 7fe7eaff3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56446f5771b0 con 0x56446f615a30 2024-04-24T08:12:09.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.851+0000 7fe7e4e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f615a30 0x56446f615e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.851+0000 7fe7e4e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f615a30 0x56446f615e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:45510/0 (socket says 172.21.15.28:45510) 2024-04-24T08:12:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.851+0000 7fe7e4e00640 1 -- 172.21.15.28:0/3684068618 learned_addr learned my addr 172.21.15.28:0/3684068618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.851+0000 7fe7e5800640 1 --2- 172.21.15.28:0/3684068618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f616350 0x56446f6204e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.851+0000 7fe7e4e00640 1 --2- 172.21.15.28:0/3684068618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f620a20 0x56446f622e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.851+0000 7fe7e5800640 1 -- 172.21.15.28:0/3684068618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f615a30 msgr2=0x56446f615e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.851+0000 7fe7e5800640 1 --2- 172.21.15.28:0/3684068618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f615a30 0x56446f615e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.851+0000 7fe7e5800640 1 -- 172.21.15.28:0/3684068618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f620a20 msgr2=0x56446f622e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.851+0000 7fe7e5800640 1 --2- 172.21.15.28:0/3684068618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f620a20 0x56446f622e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.851+0000 7fe7e5800640 1 -- 172.21.15.28:0/3684068618 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56446f575f50 con 0x56446f616350 2024-04-24T08:12:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.851+0000 7fe7e4e00640 1 --2- 172.21.15.28:0/3684068618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f620a20 0x56446f622e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.851+0000 7fe7e5800640 1 --2- 172.21.15.28:0/3684068618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f616350 0x56446f6204e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4009b40 tx=0x7fe7d40113e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8829e079c9fb66d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.851+0000 7fe7de000640 1 --2- 172.21.15.28:0/498934228 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe7c80a3da0 0x7fe7c80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.852+0000 7fe7d2a00640 1 -- 172.21.15.28:0/3684068618 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d4011e10 con 0x56446f616350 2024-04-24T08:12:09.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.852+0000 7fe7de000640 1 -- 172.21.15.28:0/498934228 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fe7d80d0e90 con 0x7fe7e010c480 2024-04-24T08:12:09.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.852+0000 7fe7d2a00640 1 -- 172.21.15.28:0/3684068618 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7d4012930 con 0x56446f616350 2024-04-24T08:12:09.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.852+0000 7fe7d2a00640 1 -- 172.21.15.28:0/3684068618 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d400fa40 con 0x56446f616350 2024-04-24T08:12:09.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.852+0000 7fe7e4e00640 1 --2- 172.21.15.28:0/498934228 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe7c80a3da0 0x7fe7c80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.852+0000 7fe7eaff3740 1 -- 172.21.15.28:0/3684068618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f616350 msgr2=0x56446f6204e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.852+0000 7fe7e4e00640 1 --2- 172.21.15.28:0/498934228 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe7c80a3da0 0x7fe7c80a6260 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x56446f612ee0 tx=0x7fe7cc005a20 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:09.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.852+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/3684068618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f616350 0x56446f6204e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4009b40 tx=0x7fe7d40113e0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.853+0000 7fe7eaff3740 1 -- 172.21.15.28:0/3684068618 shutdown_connections 2024-04-24T08:12:09.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.853+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/3684068618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f620a20 0x56446f622e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.853+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/3684068618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f616350 0x56446f6204e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.853+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/3684068618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f615a30 0x56446f615e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.853+0000 7fe7eaff3740 1 -- 172.21.15.28:0/3684068618 >> 172.21.15.28:0/3684068618 conn(0x56446f611d50 msgr2=0x56446f612130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:09.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.853+0000 7fe7eaff3740 1 -- 172.21.15.28:0/3684068618 shutdown_connections 2024-04-24T08:12:09.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.853+0000 7fe7eaff3740 1 -- 172.21.15.28:0/3684068618 wait complete. 2024-04-24T08:12:09.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.853+0000 7fe7eaff3740 1 -- start start 2024-04-24T08:12:09.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.853+0000 7fe7eaff3740 1 Processor -- start 2024-04-24T08:12:09.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.854+0000 7fe7eaff3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f615a30 0x56446f61d9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.854+0000 7fe7eaff3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f616350 0x56446f61dee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.854+0000 7fe7eaff3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f620a20 0x56446f61e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.854+0000 7fe7eaff3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56446f623810 con 0x56446f616350 2024-04-24T08:12:09.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.854+0000 7fe7eaff3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56446f534690 con 0x56446f615a30 2024-04-24T08:12:09.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.854+0000 7fe7eaff3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56446f623990 con 0x56446f620a20 2024-04-24T08:12:09.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.854+0000 7fe7eaff3740 10 client.0 _ll_register_callbacks cb 0x56446f570650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:09.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.854+0000 7fe7e4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f616350 0x56446f61dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.854+0000 7fe7e4e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f616350 0x56446f61dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36356/0 (socket says 172.21.15.28:36356) 2024-04-24T08:12:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.854+0000 7fe7e4e00640 1 -- 172.21.15.28:0/1628696600 learned_addr learned my addr 172.21.15.28:0/1628696600 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.854+0000 7fe7e6200640 1 --2- 172.21.15.28:0/1628696600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f620a20 0x56446f61e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.854+0000 7fe7e5800640 1 --2- 172.21.15.28:0/1628696600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f615a30 0x56446f61d9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.854+0000 7fe7e6200640 1 -- 172.21.15.28:0/1628696600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f615a30 msgr2=0x56446f61d9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.854+0000 7fe7e6200640 1 --2- 172.21.15.28:0/1628696600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f615a30 0x56446f61d9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.854+0000 7fe7e6200640 1 -- 172.21.15.28:0/1628696600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f616350 msgr2=0x56446f61dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.854+0000 7fe7e6200640 1 --2- 172.21.15.28:0/1628696600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f616350 0x56446f61dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.854+0000 7fe7e6200640 1 -- 172.21.15.28:0/1628696600 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56446f533920 con 0x56446f620a20 2024-04-24T08:12:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.855+0000 7fe7e6200640 1 --2- 172.21.15.28:0/1628696600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f620a20 0x56446f61e420 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe7d80ce560 tx=0x7fe7d809c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da60286ed47fed93 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.855+0000 7fe7c7400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:12:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.855+0000 7fe7c7400640 1 -- 172.21.15.28:0/1628696600 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d80a1160 con 0x56446f620a20 2024-04-24T08:12:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.855+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1628696600 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7d4012930 con 0x56446f620a20 2024-04-24T08:12:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.855+0000 7fe7c7400640 1 -- 172.21.15.28:0/1628696600 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fe7d80a5590 con 0x56446f620a20 2024-04-24T08:12:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.855+0000 7fe7c7400640 1 -- 172.21.15.28:0/1628696600 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe7d80a5a20 con 0x56446f620a20 2024-04-24T08:12:09.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.855+0000 7fe7eaff3740 10 client.25810 fetch_fsmap learned FSMap version 24 2024-04-24T08:12:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.855+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1628696600 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56446f3577d0 con 0x56446f620a20 2024-04-24T08:12:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.855+0000 7fe7c7400640 1 -- 172.21.15.28:0/1628696600 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7d4012930 con 0x56446f620a20 2024-04-24T08:12:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.856+0000 7fe7c7400640 1 -- 172.21.15.28:0/1628696600 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fe7d8087040 con 0x56446f620a20 2024-04-24T08:12:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.856+0000 7fe7eaff3740 10 client.25810 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:12:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.856+0000 7fe7eaff3740 10 client.25810 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:12:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.856+0000 7fe7eaff3740 20 client.25810 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.856+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/1628696600 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56446f630dd0 0x56446f6331e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.856+0000 7fe7eaff3740 4 client.25810 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.856+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1628696600 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56446f588c10 con 0x56446f630dd0 2024-04-24T08:12:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.856+0000 7fe7e5800640 1 --2- 172.21.15.28:0/1628696600 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56446f630dd0 0x56446f6331e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.857+0000 7fe7e5800640 1 --2- 172.21.15.28:0/1628696600 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56446f630dd0 0x56446f6331e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=673e288aa6b9d3a server_cookie=2a4bd0118563c83e in_seq=0 out_seq=0 2024-04-24T08:12:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.857+0000 7fe7c7400640 10 client.25810 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.857+0000 7fe7c7400640 1 -- 172.21.15.28:0/1628696600 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56446f588c10 con 0x56446f630dd0 2024-04-24T08:12:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.857+0000 7fe7c7400640 10 client.25810 handle_command_reply: tid=0 2024-04-24T08:12:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.857+0000 7fe7eaff3740 1 client.25810 shutdown 2024-04-24T08:12:09.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.857+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1628696600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f620a20 msgr2=0x56446f61e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.858+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/1628696600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f620a20 0x56446f61e420 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe7d80ce560 tx=0x7fe7d809c7d0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.858+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1628696600 shutdown_connections 2024-04-24T08:12:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.858+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/1628696600 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56446f630dd0 0x56446f6331e0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.858+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/1628696600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f616350 0x56446f61dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.858+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/1628696600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f620a20 0x56446f61e420 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.858+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/1628696600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f615a30 0x56446f61d9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.858+0000 7fe7c7400640 0 client.25810 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.858+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1628696600 >> 172.21.15.28:0/1628696600 conn(0x56446f611d50 msgr2=0x56446f617b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.858+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1628696600 shutdown_connections 2024-04-24T08:12:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.858+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1628696600 wait complete. 2024-04-24T08:12:09.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.858+0000 7fe7eaff3740 20 client.25810 trim_cache size 0 max 16384 2024-04-24T08:12:09.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.876+0000 7fe7eaff3740 1 Processor -- start 2024-04-24T08:12:09.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.876+0000 7fe7eaff3740 1 -- start start 2024-04-24T08:12:09.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7eaff3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f615a30 0x56446f62c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7eaff3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f616350 0x56446f6181d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7eaff3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f620a20 0x56446f638a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7eaff3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e01b2660 con 0x56446f620a20 2024-04-24T08:12:09.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7eaff3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e01aff30 con 0x56446f615a30 2024-04-24T08:12:09.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7eaff3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e01add80 con 0x56446f616350 2024-04-24T08:12:09.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7e5800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f615a30 0x56446f62c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7e5800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f615a30 0x56446f62c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:48598/0 (socket says 172.21.15.28:48598) 2024-04-24T08:12:09.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7e6200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f620a20 0x56446f638a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7e5800640 1 -- 172.21.15.28:0/1481215057 learned_addr learned my addr 172.21.15.28:0/1481215057 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:09.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7e4e00640 1 --2- 172.21.15.28:0/1481215057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f616350 0x56446f6181d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7e5800640 1 -- 172.21.15.28:0/1481215057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f616350 msgr2=0x56446f6181d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:09.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7e5800640 1 --2- 172.21.15.28:0/1481215057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f616350 0x56446f6181d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7e5800640 1 -- 172.21.15.28:0/1481215057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f620a20 msgr2=0x56446f638a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:09.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7e5800640 1 --2- 172.21.15.28:0/1481215057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f620a20 0x56446f638a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7e5800640 1 -- 172.21.15.28:0/1481215057 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56446f28d3f0 con 0x56446f615a30 2024-04-24T08:12:09.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7e4e00640 1 --2- 172.21.15.28:0/1481215057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f616350 0x56446f6181d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:09.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.877+0000 7fe7e6200640 1 --2- 172.21.15.28:0/1481215057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f620a20 0x56446f638a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:09.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.878+0000 7fe7e5800640 1 --2- 172.21.15.28:0/1481215057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f615a30 0x56446f62c730 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4009b40 tx=0x7fe7d4005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86115707a08e53bb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:09.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.878+0000 7fe7d2000640 1 -- 172.21.15.28:0/1481215057 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d4032070 con 0x56446f615a30 2024-04-24T08:12:09.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.878+0000 7fe7d2000640 1 -- 172.21.15.28:0/1481215057 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7d402c870 con 0x56446f615a30 2024-04-24T08:12:09.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.878+0000 7fe7d2000640 1 -- 172.21.15.28:0/1481215057 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d4005580 con 0x56446f615a30 2024-04-24T08:12:09.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.879+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1481215057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f615a30 msgr2=0x56446f62c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.879+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/1481215057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f615a30 0x56446f62c730 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4009b40 tx=0x7fe7d4005f90 comp rx=0 tx=0).stop 2024-04-24T08:12:09.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.879+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1481215057 shutdown_connections 2024-04-24T08:12:09.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.879+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/1481215057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f620a20 0x56446f638a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.879+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/1481215057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f616350 0x56446f6181d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.879+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/1481215057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f615a30 0x56446f62c730 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.879+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1481215057 >> 172.21.15.28:0/1481215057 conn(0x56446f611d50 msgr2=0x56446f610fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:09.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.879+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1481215057 shutdown_connections 2024-04-24T08:12:09.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.879+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1481215057 wait complete. 2024-04-24T08:12:09.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.879+0000 7fe7eaff3740 1 -- start start 2024-04-24T08:12:09.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.879+0000 7fe7eaff3740 1 Processor -- start 2024-04-24T08:12:09.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.880+0000 7fe7eaff3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f615a30 0x56446f628e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.880+0000 7fe7eaff3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f616350 0x56446f61a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.880+0000 7fe7eaff3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f620a20 0x56446f61a710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.880+0000 7fe7eaff3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e01ae3f0 con 0x56446f615a30 2024-04-24T08:12:09.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.880+0000 7fe7eaff3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e01af860 con 0x56446f620a20 2024-04-24T08:12:09.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.880+0000 7fe7eaff3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe7e01aeac0 con 0x56446f616350 2024-04-24T08:12:09.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.880+0000 7fe7eaff3740 10 client.0 _ll_register_callbacks cb 0x7fe7d809cbf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:09.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.880+0000 7fe7e5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f615a30 0x56446f628e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.880+0000 7fe7e5800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f615a30 0x56446f628e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36380/0 (socket says 172.21.15.28:36380) 2024-04-24T08:12:09.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.880+0000 7fe7e5800640 1 -- 172.21.15.28:0/1875631256 learned_addr learned my addr 172.21.15.28:0/1875631256 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:09.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.880+0000 7fe7e4e00640 1 --2- 172.21.15.28:0/1875631256 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f616350 0x56446f61a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.880+0000 7fe7e6200640 1 --2- 172.21.15.28:0/1875631256 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f620a20 0x56446f61a710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.880+0000 7fe7e5800640 1 -- 172.21.15.28:0/1875631256 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f616350 msgr2=0x56446f61a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:09.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.880+0000 7fe7e5800640 1 --2- 172.21.15.28:0/1875631256 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f616350 0x56446f61a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.880+0000 7fe7e5800640 1 -- 172.21.15.28:0/1875631256 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f620a20 msgr2=0x56446f61a710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.880+0000 7fe7e5800640 1 --2- 172.21.15.28:0/1875631256 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f620a20 0x56446f61a710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.881+0000 7fe7e5800640 1 -- 172.21.15.28:0/1875631256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56446f24d0f0 con 0x56446f615a30 2024-04-24T08:12:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.881+0000 7fe7e6200640 1 --2- 172.21.15.28:0/1875631256 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f620a20 0x56446f61a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.881+0000 7fe7e5800640 1 --2- 172.21.15.28:0/1875631256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f615a30 0x56446f628e30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4005120 tx=0x7fe7d4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d485796fbbcfcee server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.881+0000 7fe7c7400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:12:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.881+0000 7fe7c7400640 1 -- 172.21.15.28:0/1875631256 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7d403b060 con 0x56446f615a30 2024-04-24T08:12:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.881+0000 7fe7c7400640 1 -- 172.21.15.28:0/1875631256 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fe7d402ca20 con 0x56446f615a30 2024-04-24T08:12:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.881+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1875631256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7d402c870 con 0x56446f615a30 2024-04-24T08:12:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.881+0000 7fe7c7400640 1 -- 172.21.15.28:0/1875631256 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7fe7d4032070 con 0x56446f615a30 2024-04-24T08:12:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.881+0000 7fe7eaff3740 10 client.15675 fetch_fsmap learned FSMap version 24 2024-04-24T08:12:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.881+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1875631256 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x56446f540810 con 0x56446f615a30 2024-04-24T08:12:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.882+0000 7fe7c7400640 1 -- 172.21.15.28:0/1875631256 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7d402c870 con 0x56446f615a30 2024-04-24T08:12:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.882+0000 7fe7c7400640 1 -- 172.21.15.28:0/1875631256 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 24) ==== 8338+0+0 (secure 0 0 0) 0x7fe7d402c070 con 0x56446f615a30 2024-04-24T08:12:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.882+0000 7fe7eaff3740 10 client.15675 fetch_fsmap finished waiting for FSMap version 24 2024-04-24T08:12:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.882+0000 7fe7eaff3740 10 client.15675 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:12:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.882+0000 7fe7eaff3740 20 client.15675 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.882+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/1875631256 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56446f6972b0 0x56446f6996d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.882+0000 7fe7eaff3740 4 client.15675 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:12:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.882+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1875631256 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56446f582a80 con 0x56446f6972b0 2024-04-24T08:12:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.882+0000 7fe7e4e00640 1 --2- 172.21.15.28:0/1875631256 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56446f6972b0 0x56446f6996d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.883+0000 7fe7e4e00640 1 --2- 172.21.15.28:0/1875631256 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56446f6972b0 0x56446f6996d0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dcd676c714c9df2f server_cookie=973edb38e25bf029 in_seq=0 out_seq=0 2024-04-24T08:12:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.883+0000 7fe7c7400640 10 client.15675 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.883+0000 7fe7c7400640 1 -- 172.21.15.28:0/1875631256 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x56446f582a80 con 0x56446f6972b0 2024-04-24T08:12:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.883+0000 7fe7c7400640 10 client.15675 handle_command_reply: tid=0 2024-04-24T08:12:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.883+0000 7fe7eaff3740 1 client.15675 shutdown 2024-04-24T08:12:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.883+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1875631256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f615a30 msgr2=0x56446f628e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.883+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/1875631256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f615a30 0x56446f628e30 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fe7d4005120 tx=0x7fe7d4011c10 comp rx=0 tx=0).stop 2024-04-24T08:12:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.883+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1875631256 shutdown_connections 2024-04-24T08:12:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.883+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/1875631256 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56446f6972b0 0x56446f6996d0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.883+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/1875631256 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56446f620a20 0x56446f61a710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.883+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/1875631256 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56446f616350 0x56446f61a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.883+0000 7fe7eaff3740 1 --2- 172.21.15.28:0/1875631256 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56446f615a30 0x56446f628e30 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.883+0000 7fe7c7400640 0 client.15675 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.883+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1875631256 >> 172.21.15.28:0/1875631256 conn(0x56446f611d50 msgr2=0x56446f61d960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.883+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1875631256 shutdown_connections 2024-04-24T08:12:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.883+0000 7fe7eaff3740 1 -- 172.21.15.28:0/1875631256 wait complete. 2024-04-24T08:12:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.884+0000 7fe7eaff3740 20 client.15675 trim_cache size 0 max 16384 2024-04-24T08:12:09.888 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:12:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:12:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:12:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: "4dae45ee-e120-4e4b-afab-d37db50bca01": { 2024-04-24T08:12:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:12:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "4dae45ee-e120-4e4b-afab-d37db50bca01", 2024-04-24T08:12:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:12:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:12:09.889 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:12:09.889 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:12:09.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.884+0000 7fe7e6c00640 1 -- 172.21.15.28:0/498934228 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe7c80a3da0 msgr2=0x7fe7c80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.884+0000 7fe7e6c00640 1 --2- 172.21.15.28:0/498934228 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe7c80a3da0 0x7fe7c80a6260 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x56446f612ee0 tx=0x7fe7cc005a20 comp rx=0 tx=0).stop 2024-04-24T08:12:09.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.884+0000 7fe7e6c00640 1 -- 172.21.15.28:0/498934228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7e010c480 msgr2=0x7fe7e019a930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:09.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.884+0000 7fe7e6c00640 1 --2- 172.21.15.28:0/498934228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7e010c480 0x7fe7e019a930 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fe7d800eb10 tx=0x7fe7d800c2d0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.884+0000 7fe7e6c00640 1 -- 172.21.15.28:0/498934228 shutdown_connections 2024-04-24T08:12:09.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.884+0000 7fe7e6c00640 1 --2- 172.21.15.28:0/498934228 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe7c80a3da0 0x7fe7c80a6260 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.884+0000 7fe7e6c00640 1 --2- 172.21.15.28:0/498934228 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe7e010c480 0x7fe7e019a930 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.884+0000 7fe7e6c00640 1 --2- 172.21.15.28:0/498934228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7e0103980 0x7fe7e01a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.884+0000 7fe7e6c00640 1 --2- 172.21.15.28:0/498934228 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe7e0103060 0x7fe7e01a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.884+0000 7fe7e6c00640 1 -- 172.21.15.28:0/498934228 >> 172.21.15.28:0/498934228 conn(0x7fe7e00fc880 msgr2=0x7fe7e0101de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.884+0000 7fe7e6c00640 1 -- 172.21.15.28:0/498934228 shutdown_connections 2024-04-24T08:12:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:09.884+0000 7fe7e6c00640 1 -- 172.21.15.28:0/498934228 wait complete. 2024-04-24T08:12:09.900 INFO:tasks.cephfs.filesystem:scrub status for tag:4dae45ee-e120-4e4b-afab-d37db50bca01 - {'path': '/', 'tag': '4dae45ee-e120-4e4b-afab-d37db50bca01', 'options': 'recursive,force'} 2024-04-24T08:12:10.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:10 smithi117 ceph-mon[32438]: scrub queued for path: / 2024-04-24T08:12:10.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:10 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:10.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:10 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:12:10.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:10 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:10.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:10 smithi194 ceph-mon[32836]: scrub queued for path: / 2024-04-24T08:12:10.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:10 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:10.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:10 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:12:10.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:10 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:10.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:10 smithi028 ceph-mon[27786]: scrub queued for path: / 2024-04-24T08:12:10.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:10 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:10.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:10 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:12:10.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:10 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:11.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:11 smithi117 ceph-mon[32438]: pgmap v58: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:12:11.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:11 smithi194 ceph-mon[32836]: pgmap v58: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:12:11.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:11 smithi028 ceph-mon[27786]: pgmap v58: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 19 op/s 2024-04-24T08:12:13.257 INFO:tasks.workunit.client.0.smithi028.stderr:Updating files: 57% (7111/12271) Updating files: 58% (7118/12271) Updating files: 59% (7240/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 95% (11663/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-04-24T08:12:13.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:13 smithi117 ceph-mon[32438]: pgmap v59: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:12:13.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:13 smithi194 ceph-mon[32836]: pgmap v59: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:12:13.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:13 smithi028 ceph-mon[27786]: pgmap v59: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 22 op/s 2024-04-24T08:12:13.914 INFO:tasks.workunit.client.0.smithi028.stderr:Note: switching to '1f81fb397ae98da7563d451a78c61574c8f4e6e0'. 2024-04-24T08:12:13.914 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-24T08:12:13.914 INFO:tasks.workunit.client.0.smithi028.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-04-24T08:12:13.915 INFO:tasks.workunit.client.0.smithi028.stderr:changes and commit them, and you can discard any commits you make in this 2024-04-24T08:12:13.915 INFO:tasks.workunit.client.0.smithi028.stderr:state without impacting any branches by switching back to a branch. 2024-04-24T08:12:13.915 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-24T08:12:13.915 INFO:tasks.workunit.client.0.smithi028.stderr:If you want to create a new branch to retain commits you create, you may 2024-04-24T08:12:13.915 INFO:tasks.workunit.client.0.smithi028.stderr:do so (now or later) by using -c with the switch command. Example: 2024-04-24T08:12:13.915 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-24T08:12:13.915 INFO:tasks.workunit.client.0.smithi028.stderr: git switch -c 2024-04-24T08:12:13.915 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-24T08:12:13.915 INFO:tasks.workunit.client.0.smithi028.stderr:Or undo this operation with: 2024-04-24T08:12:13.915 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-24T08:12:13.915 INFO:tasks.workunit.client.0.smithi028.stderr: git switch - 2024-04-24T08:12:13.915 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-24T08:12:13.915 INFO:tasks.workunit.client.0.smithi028.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-04-24T08:12:13.915 INFO:tasks.workunit.client.0.smithi028.stderr: 2024-04-24T08:12:13.916 INFO:tasks.workunit.client.0.smithi028.stderr:HEAD is now at 1f81fb397ae DNM: trigger builds 2024-04-24T08:12:13.929 DEBUG:teuthology.orchestra.run.smithi028:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-04-24T08:12:13.992 INFO:tasks.workunit.client.0.smithi028.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-04-24T08:12:14.000 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-24T08:12:14.000 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-04-24T08:12:14.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:14 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:12:14.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:14 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:12:14.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:14 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:12:14.882 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-04-24T08:12:15.012 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-04-24T08:12:15.101 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-04-24T08:12:15.107 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-24T08:12:15.107 INFO:tasks.workunit.client.0.smithi028.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-04-24T08:12:15.199 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-04-24T08:12:15.209 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T08:12:15.209 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-04-24T08:12:15.264 INFO:tasks.workunit:Running workunits matching suites/ffsb.sh on client.0... 2024-04-24T08:12:15.265 INFO:tasks.workunit:Running workunit suites/ffsb.sh... 2024-04-24T08:12:15.265 DEBUG:teuthology.orchestra.run.smithi028:workunit test suites/ffsb.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1f81fb397ae98da7563d451a78c61574c8f4e6e0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-04-24T08:12:15.342 INFO:tasks.workunit.client.0.smithi028.stderr:++ dirname /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh 2024-04-24T08:12:15.344 INFO:tasks.workunit.client.0.smithi028.stderr:+ mydir=/home/ubuntu/cephtest/clone.client.0/qa/workunits/suites 2024-04-24T08:12:15.345 INFO:tasks.workunit.client.0.smithi028.stderr:+ trap -- retry EXIT 2024-04-24T08:12:15.345 INFO:tasks.workunit.client.0.smithi028.stderr:+ rm -rf ffsb 2024-04-24T08:12:15.348 INFO:tasks.workunit.client.0.smithi028.stderr:+ timeout 1800 git clone https://git.ceph.com/ffsb.git --depth 1 2024-04-24T08:12:15.361 INFO:tasks.workunit.client.0.smithi028.stderr:Cloning into 'ffsb'... 2024-04-24T08:12:15.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:15 smithi117 ceph-mon[32438]: pgmap v60: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-24T08:12:15.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:15 smithi194 ceph-mon[32836]: pgmap v60: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-24T08:12:15.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:15 smithi028 ceph-mon[27786]: pgmap v60: 97 pgs: 97 active+clean; 1.3 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-04-24T08:12:17.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:17 smithi117 ceph-mon[32438]: pgmap v61: 97 pgs: 97 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-04-24T08:12:17.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:17 smithi194 ceph-mon[32836]: pgmap v61: 97 pgs: 97 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-04-24T08:12:17.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:17 smithi028 ceph-mon[27786]: pgmap v61: 97 pgs: 97 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 13 KiB/s wr, 8 op/s 2024-04-24T08:12:18.167 INFO:tasks.workunit.client.0.smithi028.stderr:+ trap - EXIT 2024-04-24T08:12:18.167 INFO:tasks.workunit.client.0.smithi028.stderr:+ cd ffsb 2024-04-24T08:12:18.167 INFO:tasks.workunit.client.0.smithi028.stderr:+ ./configure 2024-04-24T08:12:18.637 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-24T08:12:19.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:19 smithi117 ceph-mon[32438]: pgmap v62: 97 pgs: 97 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-24T08:12:19.664 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether build environment is sane... yes 2024-04-24T08:12:19.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:19 smithi194 ceph-mon[32836]: pgmap v62: 97 pgs: 97 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-24T08:12:19.695 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-04-24T08:12:19.695 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gawk... gawk 2024-04-24T08:12:19.731 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether make sets $(MAKE)... yes 2024-04-24T08:12:19.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:19 smithi028 ceph-mon[27786]: pgmap v62: 97 pgs: 97 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-24T08:12:19.843 INFO:tasks.workunit.client.0.smithi028.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-24T08:12:19.844 INFO:tasks.workunit.client.0.smithi028.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-24T08:12:19.871 INFO:tasks.workunit.client.0.smithi028.stdout:checking for style of include used by make... GNU 2024-04-24T08:12:19.879 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... gcc 2024-04-24T08:12:20.019 INFO:tasks.workunit.client.0.smithi028.stdout:checking for C compiler default output file name... a.out 2024-04-24T08:12:20.026 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether the C compiler works... yes 2024-04-24T08:12:20.032 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are cross compiling... no 2024-04-24T08:12:20.119 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of executables... 2024-04-24T08:12:20.206 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of object files... o 2024-04-24T08:12:20.295 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... yes 2024-04-24T08:12:20.394 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... yes 2024-04-24T08:12:20.559 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... none needed 2024-04-24T08:12:20.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:20 smithi194 ceph-mon[32836]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:12:20.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:20 smithi028 ceph-mon[27786]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:12:20.844 INFO:tasks.workunit.client.0.smithi028.stdout:checking dependency style of gcc... gcc3 2024-04-24T08:12:20.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:20 smithi117 ceph-mon[32438]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:12:20.990 INFO:tasks.workunit.client.0.smithi028.stdout:checking how to run the C preprocessor... gcc -E 2024-04-24T08:12:21.135 INFO:tasks.workunit.client.0.smithi028.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-24T08:12:21.141 INFO:tasks.workunit.client.0.smithi028.stdout:checking for egrep... /usr/bin/grep -E 2024-04-24T08:12:21.174 INFO:tasks.workunit.client.0.smithi028.stdout:checking for AIX... no 2024-04-24T08:12:21.220 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... (cached) gcc 2024-04-24T08:12:21.251 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-04-24T08:12:21.253 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-24T08:12:21.253 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-04-24T08:12:21.254 INFO:tasks.workunit.client.0.smithi028.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-24T08:12:21.401 INFO:tasks.workunit.client.0.smithi028.stdout:checking for main in -lm... yes 2024-04-24T08:12:21.520 INFO:tasks.workunit.client.0.smithi028.stdout:checking for main in -lpthread... yes 2024-04-24T08:12:21.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:21 smithi194 ceph-mon[32836]: pgmap v63: 97 pgs: 97 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-24T08:12:21.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:21 smithi028 ceph-mon[27786]: pgmap v63: 97 pgs: 97 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-24T08:12:21.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:21 smithi117 ceph-mon[32438]: pgmap v63: 97 pgs: 97 active+clean; 1.5 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 13 KiB/s wr, 6 op/s 2024-04-24T08:12:21.944 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ANSI C header files... yes 2024-04-24T08:12:22.083 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-24T08:12:22.202 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/types.h... yes 2024-04-24T08:12:22.331 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/stat.h... yes 2024-04-24T08:12:22.451 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdlib.h... yes 2024-04-24T08:12:22.566 INFO:tasks.workunit.client.0.smithi028.stdout:checking for string.h... yes 2024-04-24T08:12:22.703 INFO:tasks.workunit.client.0.smithi028.stdout:checking for memory.h... yes 2024-04-24T08:12:22.828 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strings.h... yes 2024-04-24T08:12:22.936 INFO:tasks.workunit.client.0.smithi028.stdout:checking for inttypes.h... yes 2024-04-24T08:12:23.047 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... yes 2024-04-24T08:12:23.172 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... yes 2024-04-24T08:12:23.329 INFO:tasks.workunit.client.0.smithi028.stdout:checking pthread.h usability... yes 2024-04-24T08:12:23.403 INFO:tasks.workunit.client.0.smithi028.stdout:checking pthread.h presence... yes 2024-04-24T08:12:23.405 INFO:tasks.workunit.client.0.smithi028.stdout:checking for pthread.h... yes 2024-04-24T08:12:23.531 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h usability... yes 2024-04-24T08:12:23.603 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h presence... yes 2024-04-24T08:12:23.606 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fcntl.h... yes 2024-04-24T08:12:23.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:23 smithi194 ceph-mon[32836]: pgmap v64: 97 pgs: 97 active+clean; 4.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 267 KiB/s wr, 32 op/s 2024-04-24T08:12:23.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:23 smithi028 ceph-mon[27786]: pgmap v64: 97 pgs: 97 active+clean; 4.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 267 KiB/s wr, 32 op/s 2024-04-24T08:12:23.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:23 smithi117 ceph-mon[32438]: pgmap v64: 97 pgs: 97 active+clean; 4.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 267 KiB/s wr, 32 op/s 2024-04-24T08:12:24.100 INFO:tasks.workunit.client.0.smithi028.stdout:checking limits.h usability... yes 2024-04-24T08:12:24.172 INFO:tasks.workunit.client.0.smithi028.stdout:checking limits.h presence... yes 2024-04-24T08:12:24.175 INFO:tasks.workunit.client.0.smithi028.stdout:checking for limits.h... yes 2024-04-24T08:12:24.219 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... (cached) yes 2024-04-24T08:12:24.331 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h usability... yes 2024-04-24T08:12:24.393 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h presence... yes 2024-04-24T08:12:24.396 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/time.h... yes 2024-04-24T08:12:24.440 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... (cached) yes 2024-04-24T08:12:24.548 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/vfs.h usability... yes 2024-04-24T08:12:24.607 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/vfs.h presence... yes 2024-04-24T08:12:24.610 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/vfs.h... yes 2024-04-24T08:12:24.743 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/limits.h usability... no 2024-04-24T08:12:24.829 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/limits.h presence... no 2024-04-24T08:12:24.830 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/limits.h... no 2024-04-24T08:12:24.931 INFO:tasks.workunit.client.0.smithi028.stdout:checking for an ANSI C-conforming const... yes 2024-04-24T08:12:25.031 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-04-24T08:12:25.034 INFO:tasks.workunit.client.0.smithi028.stdout:checking for function prototypes... yes 2024-04-24T08:12:25.136 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether setvbuf arguments are reversed... no 2024-04-24T08:12:25.320 INFO:tasks.workunit.client.0.smithi028.stdout:checking for system... yes 2024-04-24T08:12:25.477 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gettimeofday... yes 2024-04-24T08:12:25.645 INFO:tasks.workunit.client.0.smithi028.stdout:checking for mkdir... yes 2024-04-24T08:12:25.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:25 smithi194 ceph-mon[32836]: pgmap v65: 97 pgs: 97 active+clean; 4.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 267 KiB/s wr, 29 op/s 2024-04-24T08:12:25.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:25 smithi028 ceph-mon[27786]: pgmap v65: 97 pgs: 97 active+clean; 4.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 267 KiB/s wr, 29 op/s 2024-04-24T08:12:25.814 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strerror... yes 2024-04-24T08:12:25.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:25 smithi117 ceph-mon[32438]: pgmap v65: 97 pgs: 97 active+clean; 4.3 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 267 KiB/s wr, 29 op/s 2024-04-24T08:12:26.169 INFO:tasks.workunit.client.0.smithi028.stdout:checking for open64... yes 2024-04-24T08:12:26.326 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stat64... yes 2024-04-24T08:12:26.503 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fseeko64... yes 2024-04-24T08:12:26.660 INFO:tasks.workunit.client.0.smithi028.stdout:checking for lrand48_r... yes 2024-04-24T08:12:26.808 INFO:tasks.workunit.client.0.smithi028.stdout:checking for srand48_r... yes 2024-04-24T08:12:26.872 INFO:tasks.workunit.client.0.smithi028.stdout:configure: creating ./config.status 2024-04-24T08:12:27.225 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating Makefile 2024-04-24T08:12:27.273 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating config.h 2024-04-24T08:12:27.336 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing depfiles commands 2024-04-24T08:12:27.508 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing default commands 2024-04-24T08:12:27.650 INFO:tasks.workunit.client.0.smithi028.stderr:+ make 2024-04-24T08:12:27.662 INFO:tasks.workunit.client.0.smithi028.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run aclocal-1.10 2024-04-24T08:12:27.670 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: aclocal-1.10: command not found 2024-04-24T08:12:27.673 INFO:tasks.workunit.client.0.smithi028.stderr:WARNING: `aclocal-1.10' is missing on your system. You should only need it if 2024-04-24T08:12:27.673 INFO:tasks.workunit.client.0.smithi028.stderr: you modified `acinclude.m4' or `configure.in'. You might want 2024-04-24T08:12:27.673 INFO:tasks.workunit.client.0.smithi028.stderr: to install the `Automake' and `Perl' packages. Grab them from 2024-04-24T08:12:27.673 INFO:tasks.workunit.client.0.smithi028.stderr: any GNU archive site. 2024-04-24T08:12:27.680 INFO:tasks.workunit.client.0.smithi028.stdout: cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run automake-1.10 --gnu 2024-04-24T08:12:27.686 INFO:tasks.workunit.client.0.smithi028.stderr:/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing: line 46: automake-1.10: command not found 2024-04-24T08:12:27.688 INFO:tasks.workunit.client.0.smithi028.stderr:WARNING: `automake-1.10' is missing on your system. You should only need it if 2024-04-24T08:12:27.689 INFO:tasks.workunit.client.0.smithi028.stderr: you modified `Makefile.am', `acinclude.m4' or `configure.in'. 2024-04-24T08:12:27.689 INFO:tasks.workunit.client.0.smithi028.stderr: You might want to install the `Automake' and `Perl' packages. 2024-04-24T08:12:27.689 INFO:tasks.workunit.client.0.smithi028.stderr: Grab them from any GNU archive site. 2024-04-24T08:12:27.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:27 smithi194 ceph-mon[32836]: pgmap v66: 97 pgs: 97 active+clean; 5.5 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 382 KiB/s wr, 45 op/s 2024-04-24T08:12:27.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:27 smithi028 ceph-mon[27786]: pgmap v66: 97 pgs: 97 active+clean; 5.5 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 382 KiB/s wr, 45 op/s 2024-04-24T08:12:27.822 INFO:tasks.workunit.client.0.smithi028.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoconf 2024-04-24T08:12:27.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:27 smithi117 ceph-mon[32438]: pgmap v66: 97 pgs: 97 active+clean; 5.5 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 382 KiB/s wr, 45 op/s 2024-04-24T08:12:28.417 INFO:tasks.workunit.client.0.smithi028.stderr:aclocal.m4:16: warning: this file was generated for autoconf 2.61. 2024-04-24T08:12:28.417 INFO:tasks.workunit.client.0.smithi028.stderr:You have another version of autoconf. It may work, but is not guaranteed to. 2024-04-24T08:12:28.417 INFO:tasks.workunit.client.0.smithi028.stderr:If you have problems, you may need to regenerate the build system entirely. 2024-04-24T08:12:28.417 INFO:tasks.workunit.client.0.smithi028.stderr:To do so, use the procedure documented by the package, typically `autoreconf'. 2024-04-24T08:12:28.645 INFO:tasks.workunit.client.0.smithi028.stdout:/bin/sh ./config.status --recheck 2024-04-24T08:12:28.825 INFO:tasks.workunit.client.0.smithi028.stdout:running CONFIG_SHELL=/bin/sh /bin/sh ./configure --no-create --no-recursion 2024-04-24T08:12:29.143 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-04-24T08:12:29.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:29 smithi194 ceph-mon[32836]: pgmap v67: 97 pgs: 97 active+clean; 5.5 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 369 KiB/s wr, 42 op/s 2024-04-24T08:12:29.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:29 smithi028 ceph-mon[27786]: pgmap v67: 97 pgs: 97 active+clean; 5.5 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 369 KiB/s wr, 42 op/s 2024-04-24T08:12:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:29 smithi117 ceph-mon[32438]: pgmap v67: 97 pgs: 97 active+clean; 5.5 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 369 KiB/s wr, 42 op/s 2024-04-24T08:12:30.168 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether build environment is sane... yes 2024-04-24T08:12:30.188 INFO:tasks.workunit.client.0.smithi028.stdout:checking for a thread-safe mkdir -p... /usr/bin/mkdir -p 2024-04-24T08:12:30.188 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gawk... gawk 2024-04-24T08:12:30.220 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether make sets $(MAKE)... yes 2024-04-24T08:12:30.322 INFO:tasks.workunit.client.0.smithi028.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-04-24T08:12:30.322 INFO:tasks.workunit.client.0.smithi028.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-04-24T08:12:30.345 INFO:tasks.workunit.client.0.smithi028.stdout:checking for style of include used by make... GNU 2024-04-24T08:12:30.353 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... gcc 2024-04-24T08:12:30.552 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether the C compiler works... yes 2024-04-24T08:12:30.553 INFO:tasks.workunit.client.0.smithi028.stdout:checking for C compiler default output file name... a.out 2024-04-24T08:12:30.646 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of executables... 2024-04-24T08:12:30.739 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are cross compiling... no 2024-04-24T08:12:30.878 INFO:tasks.workunit.client.0.smithi028.stdout:checking for suffix of object files... o 2024-04-24T08:12:30.923 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... yes 2024-04-24T08:12:30.976 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... yes 2024-04-24T08:12:31.065 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... none needed 2024-04-24T08:12:31.237 INFO:tasks.workunit.client.0.smithi028.stdout:checking dependency style of gcc... gcc3 2024-04-24T08:12:31.365 INFO:tasks.workunit.client.0.smithi028.stdout:checking how to run the C preprocessor... gcc -E 2024-04-24T08:12:31.481 INFO:tasks.workunit.client.0.smithi028.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-04-24T08:12:31.486 INFO:tasks.workunit.client.0.smithi028.stdout:checking for egrep... /usr/bin/grep -E 2024-04-24T08:12:31.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:31 smithi194 ceph-mon[32836]: pgmap v68: 97 pgs: 97 active+clean; 5.5 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 369 KiB/s wr, 42 op/s 2024-04-24T08:12:31.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:31 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:31.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:31 smithi028 ceph-mon[27786]: pgmap v68: 97 pgs: 97 active+clean; 5.5 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 369 KiB/s wr, 42 op/s 2024-04-24T08:12:31.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:31 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:31.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:31 smithi117 ceph-mon[32438]: pgmap v68: 97 pgs: 97 active+clean; 5.5 MiB data, 364 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 369 KiB/s wr, 42 op/s 2024-04-24T08:12:31.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:31 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:12:32.250 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ANSI C header files... yes 2024-04-24T08:12:32.340 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/types.h... yes 2024-04-24T08:12:32.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:32 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:32.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:32 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:32.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:32 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:12:32.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:32 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:32.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:32 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:32.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:32 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:12:32.794 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/stat.h... yes 2024-04-24T08:12:32.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:32 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:12:32.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:32 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:12:32.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:32 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:12:32.896 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdlib.h... yes 2024-04-24T08:12:32.994 INFO:tasks.workunit.client.0.smithi028.stdout:checking for string.h... yes 2024-04-24T08:12:33.104 INFO:tasks.workunit.client.0.smithi028.stdout:checking for memory.h... yes 2024-04-24T08:12:33.208 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strings.h... yes 2024-04-24T08:12:33.311 INFO:tasks.workunit.client.0.smithi028.stdout:checking for inttypes.h... yes 2024-04-24T08:12:33.429 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... yes 2024-04-24T08:12:33.535 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... yes 2024-04-24T08:12:33.641 INFO:tasks.workunit.client.0.smithi028.stdout:checking minix/config.h usability... no 2024-04-24T08:12:33.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:33 smithi194 ceph-mon[32836]: pgmap v69: 97 pgs: 97 active+clean; 6.7 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 507 KiB/s wr, 62 op/s 2024-04-24T08:12:33.723 INFO:tasks.workunit.client.0.smithi028.stdout:checking minix/config.h presence... no 2024-04-24T08:12:33.723 INFO:tasks.workunit.client.0.smithi028.stdout:checking for minix/config.h... no 2024-04-24T08:12:33.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:33 smithi028 ceph-mon[27786]: pgmap v69: 97 pgs: 97 active+clean; 6.7 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 507 KiB/s wr, 62 op/s 2024-04-24T08:12:33.806 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether it is safe to define __EXTENSIONS__... yes 2024-04-24T08:12:33.806 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc... (cached) gcc 2024-04-24T08:12:33.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:33 smithi117 ceph-mon[32438]: pgmap v69: 97 pgs: 97 active+clean; 6.7 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 507 KiB/s wr, 62 op/s 2024-04-24T08:12:33.906 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-04-24T08:12:33.906 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether gcc accepts -g... (cached) yes 2024-04-24T08:12:33.906 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-04-24T08:12:33.907 INFO:tasks.workunit.client.0.smithi028.stdout:checking dependency style of gcc... (cached) gcc3 2024-04-24T08:12:34.021 INFO:tasks.workunit.client.0.smithi028.stdout:checking for main in -lm... yes 2024-04-24T08:12:34.132 INFO:tasks.workunit.client.0.smithi028.stdout:checking for main in -lpthread... yes 2024-04-24T08:12:34.136 INFO:tasks.workunit.client.0.smithi028.stdout:checking for ANSI C header files... (cached) yes 2024-04-24T08:12:34.222 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/wait.h that is POSIX.1 compatible... yes 2024-04-24T08:12:34.341 INFO:tasks.workunit.client.0.smithi028.stdout:checking pthread.h usability... yes 2024-04-24T08:12:34.399 INFO:tasks.workunit.client.0.smithi028.stdout:checking pthread.h presence... yes 2024-04-24T08:12:34.399 INFO:tasks.workunit.client.0.smithi028.stdout:checking for pthread.h... yes 2024-04-24T08:12:34.511 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h usability... yes 2024-04-24T08:12:34.559 INFO:tasks.workunit.client.0.smithi028.stdout:checking fcntl.h presence... yes 2024-04-24T08:12:34.559 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fcntl.h... yes 2024-04-24T08:12:34.655 INFO:tasks.workunit.client.0.smithi028.stdout:checking limits.h usability... yes 2024-04-24T08:12:34.705 INFO:tasks.workunit.client.0.smithi028.stdout:checking limits.h presence... yes 2024-04-24T08:12:34.705 INFO:tasks.workunit.client.0.smithi028.stdout:checking for limits.h... yes 2024-04-24T08:12:34.742 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stdint.h... (cached) yes 2024-04-24T08:12:34.831 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h usability... yes 2024-04-24T08:12:34.875 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/time.h presence... yes 2024-04-24T08:12:34.875 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/time.h... yes 2024-04-24T08:12:34.914 INFO:tasks.workunit.client.0.smithi028.stdout:checking for unistd.h... (cached) yes 2024-04-24T08:12:35.005 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/vfs.h usability... yes 2024-04-24T08:12:35.058 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/vfs.h presence... yes 2024-04-24T08:12:35.058 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/vfs.h... yes 2024-04-24T08:12:35.177 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/limits.h usability... no 2024-04-24T08:12:35.263 INFO:tasks.workunit.client.0.smithi028.stdout:checking sys/limits.h presence... no 2024-04-24T08:12:35.263 INFO:tasks.workunit.client.0.smithi028.stdout:checking for sys/limits.h... no 2024-04-24T08:12:35.342 INFO:tasks.workunit.client.0.smithi028.stdout:checking for an ANSI C-conforming const... yes 2024-04-24T08:12:35.424 INFO:tasks.workunit.client.0.smithi028.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-04-24T08:12:35.580 INFO:tasks.workunit.client.0.smithi028.stdout:checking for system... yes 2024-04-24T08:12:35.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:35 smithi194 ceph-mon[32836]: pgmap v70: 97 pgs: 97 active+clean; 6.7 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 253 KiB/s wr, 35 op/s 2024-04-24T08:12:35.714 INFO:tasks.workunit.client.0.smithi028.stdout:checking for gettimeofday... yes 2024-04-24T08:12:35.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:35 smithi028 ceph-mon[27786]: pgmap v70: 97 pgs: 97 active+clean; 6.7 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 253 KiB/s wr, 35 op/s 2024-04-24T08:12:35.849 INFO:tasks.workunit.client.0.smithi028.stdout:checking for mkdir... yes 2024-04-24T08:12:35.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:35 smithi117 ceph-mon[32438]: pgmap v70: 97 pgs: 97 active+clean; 6.7 MiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 184 KiB/s rd, 253 KiB/s wr, 35 op/s 2024-04-24T08:12:35.996 INFO:tasks.workunit.client.0.smithi028.stdout:checking for strerror... yes 2024-04-24T08:12:36.095 INFO:tasks.workunit.client.0.smithi028.stdout:checking for open64... yes 2024-04-24T08:12:36.219 INFO:tasks.workunit.client.0.smithi028.stdout:checking for stat64... yes 2024-04-24T08:12:36.345 INFO:tasks.workunit.client.0.smithi028.stdout:checking for fseeko64... yes 2024-04-24T08:12:36.478 INFO:tasks.workunit.client.0.smithi028.stdout:checking for lrand48_r... yes 2024-04-24T08:12:36.608 INFO:tasks.workunit.client.0.smithi028.stdout:checking for srand48_r... yes 2024-04-24T08:12:36.660 INFO:tasks.workunit.client.0.smithi028.stdout:configure: creating ./config.status 2024-04-24T08:12:36.829 INFO:tasks.workunit.client.0.smithi028.stdout: /bin/sh ./config.status 2024-04-24T08:12:36.989 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating Makefile 2024-04-24T08:12:37.027 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating config.h 2024-04-24T08:12:37.080 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing depfiles commands 2024-04-24T08:12:37.125 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: executing default commands 2024-04-24T08:12:37.181 INFO:tasks.workunit.client.0.smithi028.stdout:cd . && /bin/sh /home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb/missing --run autoheader 2024-04-24T08:12:37.448 INFO:tasks.workunit.client.0.smithi028.stdout:rm -f stamp-h1 2024-04-24T08:12:37.455 INFO:tasks.workunit.client.0.smithi028.stdout:touch config.h.in 2024-04-24T08:12:37.461 INFO:tasks.workunit.client.0.smithi028.stdout:cd . && /bin/sh ./config.status config.h 2024-04-24T08:12:37.578 INFO:tasks.workunit.client.0.smithi028.stdout:config.status: creating config.h 2024-04-24T08:12:37.655 INFO:tasks.workunit.client.0.smithi028.stdout:make all-am 2024-04-24T08:12:37.670 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-04-24T08:12:37.670 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fileops.o -MD -MP -MF .deps/fileops.Tpo -c -o fileops.o fileops.c 2024-04-24T08:12:37.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:37 smithi194 ceph-mon[32836]: pgmap v71: 97 pgs: 97 active+clean; 8.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 393 KiB/s wr, 57 op/s 2024-04-24T08:12:37.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:37 smithi028 ceph-mon[27786]: pgmap v71: 97 pgs: 97 active+clean; 8.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 393 KiB/s wr, 57 op/s 2024-04-24T08:12:37.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:37 smithi117 ceph-mon[32438]: pgmap v71: 97 pgs: 97 active+clean; 8.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 302 KiB/s rd, 393 KiB/s wr, 57 op/s 2024-04-24T08:12:37.990 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/fileops.Tpo .deps/fileops.Po 2024-04-24T08:12:37.997 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rand.o -MD -MP -MF .deps/rand.Tpo -c -o rand.o rand.c 2024-04-24T08:12:38.157 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/rand.Tpo .deps/rand.Po 2024-04-24T08:12:38.166 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT main.o -MD -MP -MF .deps/main.Tpo -c -o main.o main.c 2024-04-24T08:12:38.306 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/main.Tpo .deps/main.Po 2024-04-24T08:12:38.313 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT fh.o -MD -MP -MF .deps/fh.Tpo -c -o fh.o fh.c 2024-04-24T08:12:38.455 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/fh.Tpo .deps/fh.Po 2024-04-24T08:12:38.465 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT filelist.o -MD -MP -MF .deps/filelist.Tpo -c -o filelist.o filelist.c 2024-04-24T08:12:38.639 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/filelist.Tpo .deps/filelist.Po 2024-04-24T08:12:38.647 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT metaops.o -MD -MP -MF .deps/metaops.Tpo -c -o metaops.o metaops.c 2024-04-24T08:12:38.744 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/metaops.Tpo .deps/metaops.Po 2024-04-24T08:12:38.753 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rwlock.o -MD -MP -MF .deps/rwlock.Tpo -c -o rwlock.o rwlock.c 2024-04-24T08:12:38.836 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/rwlock.Tpo .deps/rwlock.Po 2024-04-24T08:12:38.843 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT cirlist.o -MD -MP -MF .deps/cirlist.Tpo -c -o cirlist.o cirlist.c 2024-04-24T08:12:38.931 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/cirlist.Tpo .deps/cirlist.Po 2024-04-24T08:12:38.940 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT rbt.o -MD -MP -MF .deps/rbt.Tpo -c -o rbt.o rbt.c 2024-04-24T08:12:39.220 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/rbt.Tpo .deps/rbt.Po 2024-04-24T08:12:39.229 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_tg.o -MD -MP -MF .deps/ffsb_tg.Tpo -c -o ffsb_tg.o ffsb_tg.c 2024-04-24T08:12:39.416 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_tg.Tpo .deps/ffsb_tg.Po 2024-04-24T08:12:39.425 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fs.o -MD -MP -MF .deps/ffsb_fs.Tpo -c -o ffsb_fs.o ffsb_fs.c 2024-04-24T08:12:39.645 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_fs.Tpo .deps/ffsb_fs.Po 2024-04-24T08:12:39.655 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_thread.o -MD -MP -MF .deps/ffsb_thread.Tpo -c -o ffsb_thread.o ffsb_thread.c 2024-04-24T08:12:39.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:39 smithi194 ceph-mon[32836]: pgmap v72: 97 pgs: 97 active+clean; 8.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 278 KiB/s wr, 41 op/s 2024-04-24T08:12:39.774 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_thread.Tpo .deps/ffsb_thread.Po 2024-04-24T08:12:39.786 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_op.o -MD -MP -MF .deps/ffsb_op.Tpo -c -o ffsb_op.o ffsb_op.c 2024-04-24T08:12:39.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:39 smithi028 ceph-mon[27786]: pgmap v72: 97 pgs: 97 active+clean; 8.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 278 KiB/s wr, 41 op/s 2024-04-24T08:12:39.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:39 smithi117 ceph-mon[32438]: pgmap v72: 97 pgs: 97 active+clean; 8.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 278 KiB/s wr, 41 op/s 2024-04-24T08:12:39.901 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:12:39.901 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:12:39.909 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_op.Tpo .deps/ffsb_op.Po 2024-04-24T08:12:39.911 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT util.o -MD -MP -MF .deps/util.Tpo -c -o util.o util.c 2024-04-24T08:12:40.076 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/util.Tpo .deps/util.Po 2024-04-24T08:12:40.084 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT parser.o -MD -MP -MF .deps/parser.Tpo -c -o parser.o parser.c 2024-04-24T08:12:40.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.106+0000 7fd5d9c00640 1 -- 172.21.15.28:0/3642971303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5d4103ba0 msgr2=0x7fd5d410c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.106+0000 7fd5d9c00640 1 --2- 172.21.15.28:0/3642971303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5d4103ba0 0x7fd5d410c080 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc009900 tx=0x7fd5bc02a510 comp rx=0 tx=0).stop 2024-04-24T08:12:40.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.106+0000 7fd5d9c00640 1 -- 172.21.15.28:0/3642971303 shutdown_connections 2024-04-24T08:12:40.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.106+0000 7fd5d9c00640 1 --2- 172.21.15.28:0/3642971303 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5d4103ba0 0x7fd5d410c080 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.106+0000 7fd5d9c00640 1 --2- 172.21.15.28:0/3642971303 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5d410c5c0 0x7fd5d410ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.106+0000 7fd5d9c00640 1 --2- 172.21.15.28:0/3642971303 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5d4103280 0x7fd5d4103660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.106+0000 7fd5d9c00640 1 -- 172.21.15.28:0/3642971303 >> 172.21.15.28:0/3642971303 conn(0x7fd5d4078000 msgr2=0x7fd5d4078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:40.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.106+0000 7fd5d9c00640 1 -- 172.21.15.28:0/3642971303 shutdown_connections 2024-04-24T08:12:40.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.106+0000 7fd5d9c00640 1 -- 172.21.15.28:0/3642971303 wait complete. 2024-04-24T08:12:40.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.107+0000 7fd5d9c00640 1 Processor -- start 2024-04-24T08:12:40.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.107+0000 7fd5d9c00640 1 -- start start 2024-04-24T08:12:40.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.108+0000 7fd5d9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5d4103280 0x7fd5d41a0710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.108+0000 7fd5d9c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5d4103ba0 0x7fd5d41a0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.108+0000 7fd5d9c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5d410c5c0 0x7fd5d419a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.108+0000 7fd5d9c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d41123d0 con 0x7fd5d4103280 2024-04-24T08:12:40.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.108+0000 7fd5d9c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d4112250 con 0x7fd5d410c5c0 2024-04-24T08:12:40.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.108+0000 7fd5d9c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d4112550 con 0x7fd5d4103ba0 2024-04-24T08:12:40.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.108+0000 7fd5d2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5d4103280 0x7fd5d41a0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.108+0000 7fd5ca000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5d4103ba0 0x7fd5d41a0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.108+0000 7fd5ca000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5d4103ba0 0x7fd5d41a0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44762/0 (socket says 172.21.15.28:44762) 2024-04-24T08:12:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.108+0000 7fd5ca000640 1 -- 172.21.15.28:0/3949043066 learned_addr learned my addr 172.21.15.28:0/3949043066 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.108+0000 7fd5d3400640 1 --2- 172.21.15.28:0/3949043066 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5d410c5c0 0x7fd5d419a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.109+0000 7fd5ca000640 1 -- 172.21.15.28:0/3949043066 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5d410c5c0 msgr2=0x7fd5d419a910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.109+0000 7fd5ca000640 1 --2- 172.21.15.28:0/3949043066 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5d410c5c0 0x7fd5d419a910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.109+0000 7fd5ca000640 1 -- 172.21.15.28:0/3949043066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5d4103280 msgr2=0x7fd5d41a0710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.109+0000 7fd5ca000640 1 --2- 172.21.15.28:0/3949043066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5d4103280 0x7fd5d41a0710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.109+0000 7fd5ca000640 1 -- 172.21.15.28:0/3949043066 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5d419b090 con 0x7fd5d4103ba0 2024-04-24T08:12:40.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.109+0000 7fd5d2a00640 1 --2- 172.21.15.28:0/3949043066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5d4103280 0x7fd5d41a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:40.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.109+0000 7fd5ca000640 1 --2- 172.21.15.28:0/3949043066 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5d4103ba0 0x7fd5d41a0c50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc004770 tx=0x7fd5bc003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:40.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.109+0000 7fd5cbe00640 1 -- 172.21.15.28:0/3949043066 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc03f070 con 0x7fd5d4103ba0 2024-04-24T08:12:40.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.109+0000 7fd5d9c00640 1 -- 172.21.15.28:0/3949043066 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5d419b320 con 0x7fd5d4103ba0 2024-04-24T08:12:40.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.109+0000 7fd5cbe00640 1 -- 172.21.15.28:0/3949043066 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5bc031ca0 con 0x7fd5d4103ba0 2024-04-24T08:12:40.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.109+0000 7fd5cbe00640 1 -- 172.21.15.28:0/3949043066 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc035030 con 0x7fd5d4103ba0 2024-04-24T08:12:40.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.109+0000 7fd5d9c00640 1 -- 172.21.15.28:0/3949043066 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5d41a75d0 con 0x7fd5d4103ba0 2024-04-24T08:12:40.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.113+0000 7fd5dca2d740 1 Processor -- start 2024-04-24T08:12:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.113+0000 7fd5dca2d740 1 -- start start 2024-04-24T08:12:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.113+0000 7fd5cbe00640 1 -- 172.21.15.28:0/3949043066 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fd5bc042070 con 0x7fd5d4103ba0 2024-04-24T08:12:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.113+0000 7fd5dca2d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa731b860 0x55daa731bc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.113+0000 7fd5dca2d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa731c180 0x55daa7326310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.113+0000 7fd5dca2d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa7326850 0x55daa7328c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.113+0000 7fd5dca2d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55daa72811f0 con 0x55daa7326850 2024-04-24T08:12:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.113+0000 7fd5dca2d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55daa7227350 con 0x55daa731c180 2024-04-24T08:12:40.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.113+0000 7fd5dca2d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55daa727d1b0 con 0x55daa731b860 2024-04-24T08:12:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.114+0000 7fd5d3400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa731b860 0x55daa731bc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.114+0000 7fd5ca000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa7326850 0x55daa7328c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.114+0000 7fd5d3400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa731b860 0x55daa731bc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44768/0 (socket says 172.21.15.28:44768) 2024-04-24T08:12:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.114+0000 7fd5d3400640 1 -- 172.21.15.28:0/3377524130 learned_addr learned my addr 172.21.15.28:0/3377524130 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.114+0000 7fd5d2a00640 1 --2- 172.21.15.28:0/3377524130 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa731c180 0x55daa7326310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.114+0000 7fd5d3400640 1 -- 172.21.15.28:0/3377524130 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa731c180 msgr2=0x55daa7326310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.114+0000 7fd5d3400640 1 --2- 172.21.15.28:0/3377524130 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa731c180 0x55daa7326310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.114+0000 7fd5d3400640 1 -- 172.21.15.28:0/3377524130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa7326850 msgr2=0x55daa7328c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.114+0000 7fd5d3400640 1 --2- 172.21.15.28:0/3377524130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa7326850 0x55daa7328c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.114+0000 7fd5d3400640 1 -- 172.21.15.28:0/3377524130 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daa727bf50 con 0x55daa731b860 2024-04-24T08:12:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.115+0000 7fd5d3400640 1 --2- 172.21.15.28:0/3377524130 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa731b860 0x55daa731bc40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc009c20 tx=0x7fd5cc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7a3ccae8716bf6e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:40.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.115+0000 7fd5c3e00640 1 -- 172.21.15.28:0/3377524130 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc01e070 con 0x55daa731b860 2024-04-24T08:12:40.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.115+0000 7fd5c3e00640 1 -- 172.21.15.28:0/3377524130 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5cc019b10 con 0x55daa731b860 2024-04-24T08:12:40.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.115+0000 7fd5c3e00640 1 -- 172.21.15.28:0/3377524130 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc00e980 con 0x55daa731b860 2024-04-24T08:12:40.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.115+0000 7fd5cbe00640 1 --2- 172.21.15.28:0/3949043066 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd5b80a3d20 0x7fd5b80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.115+0000 7fd5cbe00640 1 -- 172.21.15.28:0/3949043066 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fd5bc0f0af0 con 0x7fd5d4103ba0 2024-04-24T08:12:40.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.115+0000 7fd5d2a00640 1 --2- 172.21.15.28:0/3949043066 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd5b80a3d20 0x7fd5b80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.116+0000 7fd5dca2d740 1 -- 172.21.15.28:0/3377524130 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa731b860 msgr2=0x55daa731bc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.116+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/3377524130 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa731b860 0x55daa731bc40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc009c20 tx=0x7fd5cc011810 comp rx=0 tx=0).stop 2024-04-24T08:12:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.116+0000 7fd5dca2d740 1 -- 172.21.15.28:0/3377524130 shutdown_connections 2024-04-24T08:12:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.116+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/3377524130 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa7326850 0x55daa7328c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.116+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/3377524130 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa731c180 0x55daa7326310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.116+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/3377524130 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa731b860 0x55daa731bc40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.116+0000 7fd5dca2d740 1 -- 172.21.15.28:0/3377524130 >> 172.21.15.28:0/3377524130 conn(0x55daa7317b80 msgr2=0x55daa7317f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:40.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.116+0000 7fd5dca2d740 1 -- 172.21.15.28:0/3377524130 shutdown_connections 2024-04-24T08:12:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.116+0000 7fd5d2a00640 1 --2- 172.21.15.28:0/3949043066 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd5b80a3d20 0x7fd5b80a61e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009fd0 tx=0x7fd5c4009500 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.116+0000 7fd5dca2d740 1 -- 172.21.15.28:0/3377524130 wait complete. 2024-04-24T08:12:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.117+0000 7fd5dca2d740 1 -- start start 2024-04-24T08:12:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.117+0000 7fd5dca2d740 1 Processor -- start 2024-04-24T08:12:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.117+0000 7fd5dca2d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa731b860 0x55daa7323860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.117+0000 7fd5dca2d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731c180 0x55daa7323da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.117+0000 7fd5dca2d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa7326850 0x55daa73242e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.117+0000 7fd5dca2d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55daa7329640 con 0x55daa731c180 2024-04-24T08:12:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.117+0000 7fd5dca2d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55daa723a690 con 0x55daa7326850 2024-04-24T08:12:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.117+0000 7fd5dca2d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55daa73297c0 con 0x55daa731b860 2024-04-24T08:12:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.118+0000 7fd5dca2d740 10 client.0 _ll_register_callbacks cb 0x55daa7276650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.118+0000 7fd5ca000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731c180 0x55daa7323da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.118+0000 7fd5ca000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731c180 0x55daa7323da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58896/0 (socket says 172.21.15.28:58896) 2024-04-24T08:12:40.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.118+0000 7fd5ca000640 1 -- 172.21.15.28:0/292206945 learned_addr learned my addr 172.21.15.28:0/292206945 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.118+0000 7fd5d2a00640 1 --2- 172.21.15.28:0/292206945 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa731b860 0x55daa7323860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.118+0000 7fd5d3400640 1 --2- 172.21.15.28:0/292206945 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa7326850 0x55daa73242e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.118+0000 7fd5ca000640 1 -- 172.21.15.28:0/292206945 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa731b860 msgr2=0x55daa7323860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.118+0000 7fd5ca000640 1 --2- 172.21.15.28:0/292206945 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa731b860 0x55daa7323860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.118+0000 7fd5ca000640 1 -- 172.21.15.28:0/292206945 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa7326850 msgr2=0x55daa73242e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.118+0000 7fd5ca000640 1 --2- 172.21.15.28:0/292206945 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa7326850 0x55daa73242e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.118+0000 7fd5ca000640 1 -- 172.21.15.28:0/292206945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daa7239920 con 0x55daa731c180 2024-04-24T08:12:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.119+0000 7fd5ca000640 1 --2- 172.21.15.28:0/292206945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731c180 0x55daa7323da0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc000c00 tx=0x7fd5bc009620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88a99c8acc247742 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.119+0000 7fd5c0c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:12:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.119+0000 7fd5c0c00640 1 -- 172.21.15.28:0/292206945 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc049070 con 0x55daa731c180 2024-04-24T08:12:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.119+0000 7fd5c0c00640 1 -- 172.21.15.28:0/292206945 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fd5bc0bae80 con 0x55daa731c180 2024-04-24T08:12:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.119+0000 7fd5dca2d740 1 -- 172.21.15.28:0/292206945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5cc019b10 con 0x55daa731c180 2024-04-24T08:12:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.119+0000 7fd5c0c00640 1 -- 172.21.15.28:0/292206945 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd5bc0bc6c0 con 0x55daa731c180 2024-04-24T08:12:40.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.119+0000 7fd5dca2d740 10 client.15687 fetch_fsmap learned FSMap version 25 2024-04-24T08:12:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.119+0000 7fd5dca2d740 1 -- 172.21.15.28:0/292206945 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55daa705d7d0 con 0x55daa731c180 2024-04-24T08:12:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.119+0000 7fd5c0c00640 1 -- 172.21.15.28:0/292206945 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5cc019b10 con 0x55daa731c180 2024-04-24T08:12:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.120+0000 7fd5c0c00640 1 -- 172.21.15.28:0/292206945 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fd5bc04c060 con 0x55daa731c180 2024-04-24T08:12:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.120+0000 7fd5dca2d740 10 client.15687 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:12:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.120+0000 7fd5dca2d740 10 client.15687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:12:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.120+0000 7fd5dca2d740 20 client.15687 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.120+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/292206945 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55daa7336b90 0x55daa7338fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.120+0000 7fd5dca2d740 4 client.15687 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:40.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.120+0000 7fd5dca2d740 1 -- 172.21.15.28:0/292206945 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55daa728ec10 con 0x55daa7336b90 2024-04-24T08:12:40.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.120+0000 7fd5d2a00640 1 --2- 172.21.15.28:0/292206945 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55daa7336b90 0x55daa7338fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.121+0000 7fd5d2a00640 1 --2- 172.21.15.28:0/292206945 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55daa7336b90 0x55daa7338fe0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3da34248dc1d46a0 server_cookie=a1c28d0d506c9ca in_seq=0 out_seq=0 2024-04-24T08:12:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.121+0000 7fd5c0c00640 10 client.15687 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.122+0000 7fd5c0c00640 1 -- 172.21.15.28:0/292206945 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55daa728ec10 con 0x55daa7336b90 2024-04-24T08:12:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.122+0000 7fd5c0c00640 10 client.15687 handle_command_reply: tid=0 2024-04-24T08:12:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.122+0000 7fd5dca2d740 1 client.15687 shutdown 2024-04-24T08:12:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.122+0000 7fd5dca2d740 1 -- 172.21.15.28:0/292206945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731c180 msgr2=0x55daa7323da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.122+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/292206945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731c180 0x55daa7323da0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc000c00 tx=0x7fd5bc009620 comp rx=0 tx=0).stop 2024-04-24T08:12:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.122+0000 7fd5dca2d740 1 -- 172.21.15.28:0/292206945 shutdown_connections 2024-04-24T08:12:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.122+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/292206945 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55daa7336b90 0x55daa7338fe0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.122+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/292206945 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731c180 0x55daa7323da0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.122+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/292206945 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa7326850 0x55daa73242e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.122+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/292206945 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa731b860 0x55daa7323860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.122+0000 7fd5c0c00640 0 client.15687 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.122+0000 7fd5dca2d740 1 -- 172.21.15.28:0/292206945 >> 172.21.15.28:0/292206945 conn(0x55daa7317b80 msgr2=0x55daa731d980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.122+0000 7fd5dca2d740 1 -- 172.21.15.28:0/292206945 shutdown_connections 2024-04-24T08:12:40.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.122+0000 7fd5dca2d740 1 -- 172.21.15.28:0/292206945 wait complete. 2024-04-24T08:12:40.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.122+0000 7fd5dca2d740 20 client.15687 trim_cache size 0 max 16384 2024-04-24T08:12:40.141 INFO:tasks.workunit.client.0.smithi028.stderr:In file included from parser.h:22, 2024-04-24T08:12:40.141 INFO:tasks.workunit.client.0.smithi028.stderr: from parser.c:25: 2024-04-24T08:12:40.141 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c: In function ‘init_filesys’: 2024-04-24T08:12:40.141 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-24T08:12:40.141 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-24T08:12:40.141 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-04-24T08:12:40.141 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-24T08:12:40.141 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-24T08:12:40.141 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-04-24T08:12:40.141 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-24T08:12:40.141 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-04-24T08:12:40.142 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-04-24T08:12:40.142 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-24T08:12:40.142 INFO:tasks.workunit.client.0.smithi028.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-24T08:12:40.142 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-04-24T08:12:40.142 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-24T08:12:40.142 INFO:tasks.workunit.client.0.smithi028.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-24T08:12:40.142 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T08:12:40.142 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-24T08:12:40.142 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-24T08:12:40.142 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-04-24T08:12:40.142 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-24T08:12:40.142 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-24T08:12:40.142 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-04-24T08:12:40.143 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-24T08:12:40.143 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-04-24T08:12:40.143 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-04-24T08:12:40.143 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-24T08:12:40.143 INFO:tasks.workunit.client.0.smithi028.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-24T08:12:40.143 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-04-24T08:12:40.143 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:934:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-24T08:12:40.143 INFO:tasks.workunit.client.0.smithi028.stderr: 934 | list_for_each_entry(tmp_list, &list_head->list, list) 2024-04-24T08:12:40.143 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T08:12:40.143 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-24T08:12:40.143 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-24T08:12:40.143 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-04-24T08:12:40.144 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-24T08:12:40.144 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-24T08:12:40.144 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-04-24T08:12:40.144 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-24T08:12:40.144 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-04-24T08:12:40.144 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-04-24T08:12:40.144 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-24T08:12:40.144 INFO:tasks.workunit.client.0.smithi028.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-24T08:12:40.144 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-04-24T08:12:40.144 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-24T08:12:40.144 INFO:tasks.workunit.client.0.smithi028.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-24T08:12:40.144 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T08:12:40.144 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-24T08:12:40.145 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-24T08:12:40.145 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-04-24T08:12:40.145 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-24T08:12:40.145 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-24T08:12:40.145 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-04-24T08:12:40.145 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-24T08:12:40.145 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-04-24T08:12:40.145 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-04-24T08:12:40.145 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-24T08:12:40.145 INFO:tasks.workunit.client.0.smithi028.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-24T08:12:40.145 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-04-24T08:12:40.145 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:941:17: note: in expansion of macro ‘list_for_each_entry’ 2024-04-24T08:12:40.145 INFO:tasks.workunit.client.0.smithi028.stderr: 941 | list_for_each_entry(tmp_list, &list_head->list, list) { 2024-04-24T08:12:40.145 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T08:12:40.146 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c: In function ‘init_tg_stats’: 2024-04-24T08:12:40.146 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-24T08:12:40.146 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-24T08:12:40.146 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-04-24T08:12:40.146 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-24T08:12:40.146 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-24T08:12:40.146 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-04-24T08:12:40.146 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-24T08:12:40.146 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-04-24T08:12:40.146 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-04-24T08:12:40.146 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-24T08:12:40.146 INFO:tasks.workunit.client.0.smithi028.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-24T08:12:40.146 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-04-24T08:12:40.147 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-24T08:12:40.147 INFO:tasks.workunit.client.0.smithi028.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-24T08:12:40.147 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T08:12:40.147 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-24T08:12:40.147 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-24T08:12:40.147 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-04-24T08:12:40.147 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-24T08:12:40.147 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-24T08:12:40.147 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-04-24T08:12:40.147 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-24T08:12:40.147 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-04-24T08:12:40.147 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-04-24T08:12:40.147 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-24T08:12:40.147 INFO:tasks.workunit.client.0.smithi028.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-24T08:12:40.148 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-04-24T08:12:40.148 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:971:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-24T08:12:40.148 INFO:tasks.workunit.client.0.smithi028.stderr: 971 | list_for_each_entry(tmp_list, 2024-04-24T08:12:40.148 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T08:12:40.148 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-24T08:12:40.148 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-24T08:12:40.148 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-04-24T08:12:40.148 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-24T08:12:40.148 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-24T08:12:40.148 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-04-24T08:12:40.148 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-24T08:12:40.148 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-04-24T08:12:40.148 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-04-24T08:12:40.148 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:39:20: note: in expansion of macro ‘list_entry’ 2024-04-24T08:12:40.149 INFO:tasks.workunit.client.0.smithi028.stderr: 39 | for (pos = list_entry((head)->next, typeof(*pos), member); \ 2024-04-24T08:12:40.149 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-04-24T08:12:40.149 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-24T08:12:40.149 INFO:tasks.workunit.client.0.smithi028.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-24T08:12:40.149 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T08:12:40.149 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:29:33: warning: cast from pointer to integer of different size [-Wpointer-to-int-cast] 2024-04-24T08:12:40.149 INFO:tasks.workunit.client.0.smithi028.stderr: 29 | #define offsetof(type, member) ((int) &((type *)0)->member) 2024-04-24T08:12:40.149 INFO:tasks.workunit.client.0.smithi028.stderr: | ^ 2024-04-24T08:12:40.149 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:33:35: note: in expansion of macro ‘offsetof’ 2024-04-24T08:12:40.149 INFO:tasks.workunit.client.0.smithi028.stderr: 33 | (type *)((char *)__mptr - offsetof(type, member)); }) 2024-04-24T08:12:40.149 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~ 2024-04-24T08:12:40.149 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:36:9: note: in expansion of macro ‘container_of’ 2024-04-24T08:12:40.149 INFO:tasks.workunit.client.0.smithi028.stderr: 36 | container_of(ptr, type, member) 2024-04-24T08:12:40.149 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~ 2024-04-24T08:12:40.150 INFO:tasks.workunit.client.0.smithi028.stderr:list.h:41:20: note: in expansion of macro ‘list_entry’ 2024-04-24T08:12:40.150 INFO:tasks.workunit.client.0.smithi028.stderr: 41 | pos = list_entry(pos->member.next, typeof(*pos), member)) 2024-04-24T08:12:40.150 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~ 2024-04-24T08:12:40.150 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:980:41: note: in expansion of macro ‘list_for_each_entry’ 2024-04-24T08:12:40.150 INFO:tasks.workunit.client.0.smithi028.stderr: 980 | list_for_each_entry(tmp_list, 2024-04-24T08:12:40.150 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T08:12:40.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.142+0000 7fd5dca2d740 1 Processor -- start 2024-04-24T08:12:40.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.142+0000 7fd5dca2d740 1 -- start start 2024-04-24T08:12:40.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.142+0000 7fd5dca2d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731b860 0x55daa7332500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.142+0000 7fd5dca2d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa731c180 0x55daa73227a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.142+0000 7fd5dca2d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa7326850 0x55daa731e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.142+0000 7fd5dca2d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d41b28e0 con 0x55daa731b860 2024-04-24T08:12:40.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.142+0000 7fd5dca2d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d41b01b0 con 0x55daa731c180 2024-04-24T08:12:40.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.142+0000 7fd5dca2d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d41ae000 con 0x55daa7326850 2024-04-24T08:12:40.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.143+0000 7fd5d2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731b860 0x55daa7332500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.143+0000 7fd5d3400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa7326850 0x55daa731e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.143+0000 7fd5d2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731b860 0x55daa7332500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58902/0 (socket says 172.21.15.28:58902) 2024-04-24T08:12:40.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.143+0000 7fd5d2a00640 1 -- 172.21.15.28:0/51332282 learned_addr learned my addr 172.21.15.28:0/51332282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:40.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.143+0000 7fd5ca000640 1 --2- 172.21.15.28:0/51332282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa731c180 0x55daa73227a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.143+0000 7fd5d3400640 1 -- 172.21.15.28:0/51332282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa731c180 msgr2=0x55daa73227a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:40.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.143+0000 7fd5d3400640 1 --2- 172.21.15.28:0/51332282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa731c180 0x55daa73227a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.143+0000 7fd5d3400640 1 -- 172.21.15.28:0/51332282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731b860 msgr2=0x55daa7332500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:40.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.143+0000 7fd5d3400640 1 --2- 172.21.15.28:0/51332282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731b860 0x55daa7332500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.143+0000 7fd5d3400640 1 -- 172.21.15.28:0/51332282 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daa6f933f0 con 0x55daa7326850 2024-04-24T08:12:40.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.143+0000 7fd5d2a00640 1 --2- 172.21.15.28:0/51332282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731b860 0x55daa7332500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:40.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.143+0000 7fd5ca000640 1 --2- 172.21.15.28:0/51332282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa731c180 0x55daa73227a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:40.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.144+0000 7fd5d3400640 1 --2- 172.21.15.28:0/51332282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa7326850 0x55daa731e000 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc00e500 tx=0x7fd5cc00e3d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3c17d80d929bdce server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:40.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.144+0000 7fd5c3400640 1 -- 172.21.15.28:0/51332282 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc029020 con 0x55daa7326850 2024-04-24T08:12:40.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.144+0000 7fd5c3400640 1 -- 172.21.15.28:0/51332282 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5cc005520 con 0x55daa7326850 2024-04-24T08:12:40.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.144+0000 7fd5c3400640 1 -- 172.21.15.28:0/51332282 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5cc01e040 con 0x55daa7326850 2024-04-24T08:12:40.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.145+0000 7fd5dca2d740 1 -- 172.21.15.28:0/51332282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa7326850 msgr2=0x55daa731e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.145+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/51332282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa7326850 0x55daa731e000 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fd5cc00e500 tx=0x7fd5cc00e3d0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.145+0000 7fd5dca2d740 1 -- 172.21.15.28:0/51332282 shutdown_connections 2024-04-24T08:12:40.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.145+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/51332282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa7326850 0x55daa731e000 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.145+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/51332282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa731c180 0x55daa73227a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.145+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/51332282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731b860 0x55daa7332500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.145+0000 7fd5dca2d740 1 -- 172.21.15.28:0/51332282 >> 172.21.15.28:0/51332282 conn(0x55daa7317b80 msgr2=0x55daa7316c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.145+0000 7fd5dca2d740 1 -- 172.21.15.28:0/51332282 shutdown_connections 2024-04-24T08:12:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.145+0000 7fd5dca2d740 1 -- 172.21.15.28:0/51332282 wait complete. 2024-04-24T08:12:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.145+0000 7fd5dca2d740 1 -- start start 2024-04-24T08:12:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.146+0000 7fd5dca2d740 1 Processor -- start 2024-04-24T08:12:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.146+0000 7fd5dca2d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731b860 0x55daa732ec70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.146+0000 7fd5dca2d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa731c180 0x55daa73200d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.146+0000 7fd5dca2d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa7326850 0x55daa7320610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.146+0000 7fd5dca2d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d41ae670 con 0x55daa731b860 2024-04-24T08:12:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.146+0000 7fd5dca2d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d41afae0 con 0x55daa731c180 2024-04-24T08:12:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.146+0000 7fd5dca2d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd5d41aed40 con 0x55daa7326850 2024-04-24T08:12:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.146+0000 7fd5dca2d740 10 client.0 _ll_register_callbacks cb 0x7fd5bc0f4290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.147+0000 7fd5d2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731b860 0x55daa732ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.147+0000 7fd5d2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731b860 0x55daa732ec70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58916/0 (socket says 172.21.15.28:58916) 2024-04-24T08:12:40.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.147+0000 7fd5d2a00640 1 -- 172.21.15.28:0/2031485970 learned_addr learned my addr 172.21.15.28:0/2031485970 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:40.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.147+0000 7fd5ca000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa731c180 0x55daa73200d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.147+0000 7fd5d3400640 1 --2- 172.21.15.28:0/2031485970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa7326850 0x55daa7320610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.147+0000 7fd5ca000640 1 -- 172.21.15.28:0/2031485970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa7326850 msgr2=0x55daa7320610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:40.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.147+0000 7fd5ca000640 1 --2- 172.21.15.28:0/2031485970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa7326850 0x55daa7320610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.147+0000 7fd5ca000640 1 -- 172.21.15.28:0/2031485970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731b860 msgr2=0x55daa732ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:40.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.147+0000 7fd5ca000640 1 --2- 172.21.15.28:0/2031485970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731b860 0x55daa732ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.147+0000 7fd5ca000640 1 -- 172.21.15.28:0/2031485970 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daa6f530f0 con 0x55daa731c180 2024-04-24T08:12:40.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.147+0000 7fd5ca000640 1 --2- 172.21.15.28:0/2031485970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa731c180 0x55daa73200d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc031550 tx=0x7fd5bc03b4f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3c1a4c84d4fa528 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:40.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.147+0000 7fd5c0c00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:12:40.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.148+0000 7fd5c0c00640 1 -- 172.21.15.28:0/2031485970 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5bc049040 con 0x55daa731c180 2024-04-24T08:12:40.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.148+0000 7fd5c0c00640 1 -- 172.21.15.28:0/2031485970 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fd5bc0dbbd0 con 0x55daa731c180 2024-04-24T08:12:40.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.148+0000 7fd5dca2d740 1 -- 172.21.15.28:0/2031485970 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5cc005520 con 0x55daa731c180 2024-04-24T08:12:40.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.148+0000 7fd5c0c00640 1 -- 172.21.15.28:0/2031485970 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fd5bc005d50 con 0x55daa731c180 2024-04-24T08:12:40.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.148+0000 7fd5dca2d740 10 client.25793 fetch_fsmap learned FSMap version 25 2024-04-24T08:12:40.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.148+0000 7fd5dca2d740 1 -- 172.21.15.28:0/2031485970 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55daa7246810 con 0x55daa731c180 2024-04-24T08:12:40.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.148+0000 7fd5c0c00640 1 -- 172.21.15.28:0/2031485970 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5cc005520 con 0x55daa731c180 2024-04-24T08:12:40.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.148+0000 7fd5c0c00640 1 -- 172.21.15.28:0/2031485970 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fd5bc0f9020 con 0x55daa731c180 2024-04-24T08:12:40.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.149+0000 7fd5dca2d740 10 client.25793 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:12:40.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.149+0000 7fd5dca2d740 10 client.25793 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:12:40.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.149+0000 7fd5dca2d740 20 client.25793 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:40.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.149+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/2031485970 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55daa739d2c0 0x55daa739f6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.149+0000 7fd5dca2d740 4 client.25793 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:12:40.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.149+0000 7fd5dca2d740 1 -- 172.21.15.28:0/2031485970 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55daa7288a80 con 0x55daa739d2c0 2024-04-24T08:12:40.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.149+0000 7fd5d2a00640 1 --2- 172.21.15.28:0/2031485970 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55daa739d2c0 0x55daa739f6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.150+0000 7fd5d2a00640 1 --2- 172.21.15.28:0/2031485970 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55daa739d2c0 0x55daa739f6e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc4aacd3240a0490 server_cookie=5255a5aca08a846a in_seq=0 out_seq=0 2024-04-24T08:12:40.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.150+0000 7fd5c0c00640 10 client.25793 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:40.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.150+0000 7fd5c0c00640 1 -- 172.21.15.28:0/2031485970 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55daa7288a80 con 0x55daa739d2c0 2024-04-24T08:12:40.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.150+0000 7fd5c0c00640 10 client.25793 handle_command_reply: tid=0 2024-04-24T08:12:40.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.150+0000 7fd5dca2d740 1 client.25793 shutdown 2024-04-24T08:12:40.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.150+0000 7fd5dca2d740 1 -- 172.21.15.28:0/2031485970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa731c180 msgr2=0x55daa73200d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.150+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/2031485970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa731c180 0x55daa73200d0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc031550 tx=0x7fd5bc03b4f0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.150+0000 7fd5dca2d740 1 -- 172.21.15.28:0/2031485970 shutdown_connections 2024-04-24T08:12:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.150+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/2031485970 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55daa739d2c0 0x55daa739f6e0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.150+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/2031485970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55daa7326850 0x55daa7320610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.150+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/2031485970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55daa731c180 0x55daa73200d0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.150+0000 7fd5c0c00640 0 client.25793 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.150+0000 7fd5dca2d740 1 --2- 172.21.15.28:0/2031485970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55daa731b860 0x55daa732ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.150+0000 7fd5dca2d740 1 -- 172.21.15.28:0/2031485970 >> 172.21.15.28:0/2031485970 conn(0x55daa7317b80 msgr2=0x55daa7323820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.150+0000 7fd5dca2d740 1 -- 172.21.15.28:0/2031485970 shutdown_connections 2024-04-24T08:12:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.150+0000 7fd5dca2d740 1 -- 172.21.15.28:0/2031485970 wait complete. 2024-04-24T08:12:40.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.151+0000 7fd5dca2d740 20 client.25793 trim_cache size 0 max 16384 2024-04-24T08:12:40.161 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:12:40.162 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-24T08:12:40.162 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-24T08:12:40.162 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:12:40.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.151+0000 7fd5d9c00640 1 -- 172.21.15.28:0/3949043066 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd5b80a3d20 msgr2=0x7fd5b80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.151+0000 7fd5d9c00640 1 --2- 172.21.15.28:0/3949043066 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd5b80a3d20 0x7fd5b80a61e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fd5c4009fd0 tx=0x7fd5c4009500 comp rx=0 tx=0).stop 2024-04-24T08:12:40.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.151+0000 7fd5d9c00640 1 -- 172.21.15.28:0/3949043066 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5d4103ba0 msgr2=0x7fd5d41a0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.151+0000 7fd5d9c00640 1 --2- 172.21.15.28:0/3949043066 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5d4103ba0 0x7fd5d41a0c50 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc004770 tx=0x7fd5bc003c10 comp rx=0 tx=0).stop 2024-04-24T08:12:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.151+0000 7fd5d9c00640 1 -- 172.21.15.28:0/3949043066 shutdown_connections 2024-04-24T08:12:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.151+0000 7fd5d9c00640 1 --2- 172.21.15.28:0/3949043066 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd5b80a3d20 0x7fd5b80a61e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.151+0000 7fd5d9c00640 1 --2- 172.21.15.28:0/3949043066 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5d410c5c0 0x7fd5d419a910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.151+0000 7fd5d9c00640 1 --2- 172.21.15.28:0/3949043066 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd5d4103ba0 0x7fd5d41a0c50 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.151+0000 7fd5d9c00640 1 --2- 172.21.15.28:0/3949043066 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd5d4103280 0x7fd5d41a0710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.151+0000 7fd5d9c00640 1 -- 172.21.15.28:0/3949043066 >> 172.21.15.28:0/3949043066 conn(0x7fd5d4078000 msgr2=0x7fd5d41084d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.152+0000 7fd5d9c00640 1 -- 172.21.15.28:0/3949043066 shutdown_connections 2024-04-24T08:12:40.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.152+0000 7fd5d9c00640 1 -- 172.21.15.28:0/3949043066 wait complete. 2024-04-24T08:12:40.169 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:12:40.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:12:40.202 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c: In function ‘get_optstr’: 2024-04-24T08:12:40.202 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:206:23: warning: ‘strnlen’ specified bound 4095 exceeds source size 1024 [-Wstringop-overread] 2024-04-24T08:12:40.202 INFO:tasks.workunit.client.0.smithi028.stderr: 206 | len = strnlen(temp, 4095) + 1; 2024-04-24T08:12:40.202 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~ 2024-04-24T08:12:40.202 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:200:14: note: source object declared here 2024-04-24T08:12:40.202 INFO:tasks.workunit.client.0.smithi028.stderr: 200 | char temp[BUFSIZE]; 2024-04-24T08:12:40.202 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~ 2024-04-24T08:12:40.210 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c: In function ‘size64_convert’: 2024-04-24T08:12:40.210 INFO:tasks.workunit.client.0.smithi028.stderr:parser.c:119:9: warning: ‘memcpy’ reading 3 bytes from a region of size 2 [-Wstringop-overread] 2024-04-24T08:12:40.210 INFO:tasks.workunit.client.0.smithi028.stderr: 119 | memcpy(unit, "\0", 3); 2024-04-24T08:12:40.210 INFO:tasks.workunit.client.0.smithi028.stderr: | ^~~~~~~~~~~~~~~~~~~~~ 2024-04-24T08:12:40.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.407+0000 7f6663200640 1 -- 172.21.15.28:0/1559236093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6664107160 msgr2=0x7f666410ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.407+0000 7f6663200640 1 --2- 172.21.15.28:0/1559236093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6664107160 0x7f666410ed30 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f665c009ca0 tx=0x7f665c02a8b0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.407+0000 7f6663200640 1 -- 172.21.15.28:0/1559236093 shutdown_connections 2024-04-24T08:12:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.407+0000 7f6663200640 1 --2- 172.21.15.28:0/1559236093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6664102a40 0x7f6664106a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.407+0000 7f6663200640 1 --2- 172.21.15.28:0/1559236093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6664107160 0x7f666410ed30 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.407+0000 7f6663200640 1 --2- 172.21.15.28:0/1559236093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6664102090 0x7f6664102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.407+0000 7f6663200640 1 -- 172.21.15.28:0/1559236093 >> 172.21.15.28:0/1559236093 conn(0x7f66640fd930 msgr2=0x7f66640ffd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.407+0000 7f6663200640 1 -- 172.21.15.28:0/1559236093 shutdown_connections 2024-04-24T08:12:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.407+0000 7f6663200640 1 -- 172.21.15.28:0/1559236093 wait complete. 2024-04-24T08:12:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.408+0000 7f6663200640 1 Processor -- start 2024-04-24T08:12:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.408+0000 7f6663200640 1 -- start start 2024-04-24T08:12:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.409+0000 7f6663200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6664102090 0x7f66641986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.409+0000 7f6663200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6664107160 0x7f6664198c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.409+0000 7f6663200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f666419cf80 0x7f666419d430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.409+0000 7f6663200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6664112220 con 0x7f6664102090 2024-04-24T08:12:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.409+0000 7f6663200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f66641120a0 con 0x7f666419cf80 2024-04-24T08:12:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.409+0000 7f6663200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f66641123a0 con 0x7f6664107160 2024-04-24T08:12:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.409+0000 7f6661e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6664102090 0x7f66641986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.409+0000 7f6661e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6664102090 0x7f66641986c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58930/0 (socket says 172.21.15.28:58930) 2024-04-24T08:12:40.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.409+0000 7f6661e00640 1 -- 172.21.15.28:0/2400309582 learned_addr learned my addr 172.21.15.28:0/2400309582 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.409+0000 7f6662800640 1 --2- 172.21.15.28:0/2400309582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f666419cf80 0x7f666419d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.409+0000 7f6661400640 1 --2- 172.21.15.28:0/2400309582 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6664107160 0x7f6664198c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.409+0000 7f6661e00640 1 -- 172.21.15.28:0/2400309582 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6664107160 msgr2=0x7f6664198c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.409+0000 7f6661e00640 1 --2- 172.21.15.28:0/2400309582 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6664107160 0x7f6664198c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.409+0000 7f6661e00640 1 -- 172.21.15.28:0/2400309582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f666419cf80 msgr2=0x7f666419d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.409+0000 7f6661e00640 1 --2- 172.21.15.28:0/2400309582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f666419cf80 0x7f666419d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.409+0000 7f6661e00640 1 -- 172.21.15.28:0/2400309582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f666419da70 con 0x7f6664102090 2024-04-24T08:12:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.409+0000 7f6662800640 1 --2- 172.21.15.28:0/2400309582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f666419cf80 0x7f666419d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:40.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.410+0000 7f6661e00640 1 --2- 172.21.15.28:0/2400309582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6664102090 0x7f66641986c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f665400e9f0 tx=0x7f665400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.410+0000 7f665aa00640 1 -- 172.21.15.28:0/2400309582 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f665400cd80 con 0x7f6664102090 2024-04-24T08:12:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.410+0000 7f6663200640 1 -- 172.21.15.28:0/2400309582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f666419dd60 con 0x7f6664102090 2024-04-24T08:12:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.410+0000 7f665aa00640 1 -- 172.21.15.28:0/2400309582 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6654005500 con 0x7f6664102090 2024-04-24T08:12:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.410+0000 7f6663200640 1 -- 172.21.15.28:0/2400309582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66641a5550 con 0x7f6664102090 2024-04-24T08:12:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.411+0000 7f665aa00640 1 -- 172.21.15.28:0/2400309582 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6654018630 con 0x7f6664102090 2024-04-24T08:12:40.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.412+0000 7f6663200640 1 -- 172.21.15.28:0/2400309582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6664068d40 con 0x7f6664102090 2024-04-24T08:12:40.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.414+0000 7f665aa00640 1 -- 172.21.15.28:0/2400309582 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f66540046c0 con 0x7f6664102090 2024-04-24T08:12:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.415+0000 7f665aa00640 1 --2- 172.21.15.28:0/2400309582 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f66480a3d20 0x7f66480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.416+0000 7f665aa00640 1 -- 172.21.15.28:0/2400309582 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f66540d1340 con 0x7f6664102090 2024-04-24T08:12:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.416+0000 7f6661400640 1 --2- 172.21.15.28:0/2400309582 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f66480a3d20 0x7f66480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.416+0000 7f6661400640 1 --2- 172.21.15.28:0/2400309582 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f66480a3d20 0x7f66480a61e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6664108700 tx=0x7f665000b390 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:40.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.419+0000 7f665aa00640 1 -- 172.21.15.28:0/2400309582 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f665409d9c0 con 0x7f6664102090 2024-04-24T08:12:40.581 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/parser.Tpo .deps/parser.Po 2024-04-24T08:12:40.589 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_fc.o -MD -MP -MF .deps/ffsb_fc.Tpo -c -o ffsb_fc.o ffsb_fc.c 2024-04-24T08:12:40.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.670+0000 7f6663200640 1 -- 172.21.15.28:0/2400309582 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f666410a450 con 0x7f6664102090 2024-04-24T08:12:40.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.672+0000 7f665aa00640 1 -- 172.21.15.28:0/2400309582 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11872 (secure 0 0 0) 0x7f665409d2c0 con 0x7f6664102090 2024-04-24T08:12:40.672 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:12:40.672 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:12:18.621376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24572,24574,24556]},"id":1}]} 2024-04-24T08:12:40.673 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-04-24T08:12:40.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.674+0000 7f6663200640 1 -- 172.21.15.28:0/2400309582 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f66480a3d20 msgr2=0x7f66480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.674+0000 7f6663200640 1 --2- 172.21.15.28:0/2400309582 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f66480a3d20 0x7f66480a61e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f6664108700 tx=0x7f665000b390 comp rx=0 tx=0).stop 2024-04-24T08:12:40.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.674+0000 7f6663200640 1 -- 172.21.15.28:0/2400309582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6664102090 msgr2=0x7f66641986c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.674+0000 7f6663200640 1 --2- 172.21.15.28:0/2400309582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6664102090 0x7f66641986c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f665400e9f0 tx=0x7f665400edf0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.674+0000 7f6663200640 1 -- 172.21.15.28:0/2400309582 shutdown_connections 2024-04-24T08:12:40.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.675+0000 7f6663200640 1 --2- 172.21.15.28:0/2400309582 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f66480a3d20 0x7f66480a61e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.675+0000 7f6663200640 1 --2- 172.21.15.28:0/2400309582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f666419cf80 0x7f666419d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.675+0000 7f6663200640 1 --2- 172.21.15.28:0/2400309582 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6664107160 0x7f6664198c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.675+0000 7f6663200640 1 --2- 172.21.15.28:0/2400309582 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6664102090 0x7f66641986c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.675+0000 7f6663200640 1 -- 172.21.15.28:0/2400309582 >> 172.21.15.28:0/2400309582 conn(0x7f66640fd930 msgr2=0x7f66640fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:40.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.675+0000 7f6663200640 1 -- 172.21.15.28:0/2400309582 shutdown_connections 2024-04-24T08:12:40.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.675+0000 7f6663200640 1 -- 172.21.15.28:0/2400309582 wait complete. 2024-04-24T08:12:40.694 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_fc.Tpo .deps/ffsb_fc.Po 2024-04-24T08:12:40.696 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:12:40.703 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT ffsb_stats.o -MD -MP -MF .deps/ffsb_stats.Tpo -c -o ffsb_stats.o ffsb_stats.c 2024-04-24T08:12:40.835 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/ffsb_stats.Tpo .deps/ffsb_stats.Po 2024-04-24T08:12:40.841 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -DHAVE_CONFIG_H -I. -D_GNU_SOURCE -g -O2 -D_REENTRANT -MT list.o -MD -MP -MF .deps/list.Tpo -c -o list.o list.c 2024-04-24T08:12:40.904 INFO:tasks.workunit.client.0.smithi028.stdout:mv -f .deps/list.Tpo .deps/list.Po 2024-04-24T08:12:40.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.911+0000 7fdc1be00640 1 -- 172.21.15.28:0/2667107620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc1c0743a0 msgr2=0x7fdc1c074780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.911+0000 7fdc1be00640 1 --2- 172.21.15.28:0/2667107620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc1c0743a0 0x7fdc1c074780 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009be0 tx=0x7fdc0c02a750 comp rx=0 tx=0).stop 2024-04-24T08:12:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.911+0000 7fdc1be00640 1 -- 172.21.15.28:0/2667107620 shutdown_connections 2024-04-24T08:12:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.911+0000 7fdc1be00640 1 --2- 172.21.15.28:0/2667107620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c074cc0 0x7fdc1c112970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.911+0000 7fdc1be00640 1 --2- 172.21.15.28:0/2667107620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc1c112eb0 0x7fdc1c1152a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.911+0000 7fdc1be00640 1 --2- 172.21.15.28:0/2667107620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc1c0743a0 0x7fdc1c074780 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.911+0000 7fdc1be00640 1 -- 172.21.15.28:0/2667107620 >> 172.21.15.28:0/2667107620 conn(0x7fdc1c0fd9d0 msgr2=0x7fdc1c0ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.911+0000 7fdc1be00640 1 -- 172.21.15.28:0/2667107620 shutdown_connections 2024-04-24T08:12:40.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.911+0000 7fdc1be00640 1 -- 172.21.15.28:0/2667107620 wait complete. 2024-04-24T08:12:40.913 INFO:tasks.workunit.client.0.smithi028.stdout:gcc -D_GNU_SOURCE -g -O2 -D_REENTRANT -o ffsb fileops.o rand.o main.o fh.o filelist.o metaops.o rwlock.o cirlist.o rbt.o ffsb_tg.o ffsb_fs.o ffsb_thread.o ffsb_op.o util.o parser.o ffsb_fc.o ffsb_stats.o list.o -lpthread -lm 2024-04-24T08:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.912+0000 7fdc1be00640 1 Processor -- start 2024-04-24T08:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.912+0000 7fdc1be00640 1 -- start start 2024-04-24T08:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.913+0000 7fdc1be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c0743a0 0x7fdc1c19edb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.913+0000 7fdc1be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc1c074cc0 0x7fdc1c19f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.913+0000 7fdc1be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc1c112eb0 0x7fdc1c1a3670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.913+0000 7fdc1be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c118830 con 0x7fdc1c0743a0 2024-04-24T08:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.913+0000 7fdc1be00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1186b0 con 0x7fdc1c074cc0 2024-04-24T08:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.913+0000 7fdc1be00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1189b0 con 0x7fdc1c112eb0 2024-04-24T08:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.913+0000 7fdc1aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c0743a0 0x7fdc1c19edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.913+0000 7fdc1aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c0743a0 0x7fdc1c19edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58948/0 (socket says 172.21.15.28:58948) 2024-04-24T08:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.913+0000 7fdc1aa00640 1 -- 172.21.15.28:0/2707275159 learned_addr learned my addr 172.21.15.28:0/2707275159 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:40.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.913+0000 7fdc1b400640 1 --2- 172.21.15.28:0/2707275159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc1c112eb0 0x7fdc1c1a3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.913+0000 7fdc12000640 1 --2- 172.21.15.28:0/2707275159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc1c074cc0 0x7fdc1c19f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.914+0000 7fdc1aa00640 1 -- 172.21.15.28:0/2707275159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc1c112eb0 msgr2=0x7fdc1c1a3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.914+0000 7fdc1aa00640 1 --2- 172.21.15.28:0/2707275159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc1c112eb0 0x7fdc1c1a3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.914+0000 7fdc1aa00640 1 -- 172.21.15.28:0/2707275159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc1c074cc0 msgr2=0x7fdc1c19f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.914+0000 7fdc1aa00640 1 --2- 172.21.15.28:0/2707275159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc1c074cc0 0x7fdc1c19f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.914+0000 7fdc1aa00640 1 -- 172.21.15.28:0/2707275159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc1c1a3da0 con 0x7fdc1c0743a0 2024-04-24T08:12:40.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.914+0000 7fdc1b400640 1 --2- 172.21.15.28:0/2707275159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc1c112eb0 0x7fdc1c1a3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:40.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.914+0000 7fdc1aa00640 1 --2- 172.21.15.28:0/2707275159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c0743a0 0x7fdc1c19edb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c000f80 tx=0x7fdc0c030630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:40.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.914+0000 7fdc20a00640 1 -- 172.21.15.28:0/2707275159 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c041020 con 0x7fdc1c0743a0 2024-04-24T08:12:40.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.914+0000 7fdc1be00640 1 -- 172.21.15.28:0/2707275159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc1c1a4030 con 0x7fdc1c0743a0 2024-04-24T08:12:40.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.914+0000 7fdc20a00640 1 -- 172.21.15.28:0/2707275159 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc0c037040 con 0x7fdc1c0743a0 2024-04-24T08:12:40.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.914+0000 7fdc20a00640 1 -- 172.21.15.28:0/2707275159 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c036070 con 0x7fdc1c0743a0 2024-04-24T08:12:40.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.914+0000 7fdc1be00640 1 -- 172.21.15.28:0/2707275159 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc1c1ab920 con 0x7fdc1c0743a0 2024-04-24T08:12:40.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.918+0000 7fdc24c1a740 1 Processor -- start 2024-04-24T08:12:40.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.918+0000 7fdc20a00640 1 -- 172.21.15.28:0/2707275159 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fdc0c044020 con 0x7fdc1c0743a0 2024-04-24T08:12:40.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.918+0000 7fdc24c1a740 1 -- start start 2024-04-24T08:12:40.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.919+0000 7fdc24c1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c28b40 0x563443c28f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.919+0000 7fdc24c1a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c29460 0x563443c335f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.919+0000 7fdc24c1a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c33b30 0x563443c35f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.919+0000 7fdc24c1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563443b8e1d0 con 0x563443c28b40 2024-04-24T08:12:40.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.919+0000 7fdc24c1a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563443b34450 con 0x563443c29460 2024-04-24T08:12:40.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.919+0000 7fdc24c1a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563443b8a190 con 0x563443c33b30 2024-04-24T08:12:40.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.919+0000 7fdc1b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c28b40 0x563443c28f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.919+0000 7fdc12000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c33b30 0x563443c35f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.919+0000 7fdc1b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c28b40 0x563443c28f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58964/0 (socket says 172.21.15.28:58964) 2024-04-24T08:12:40.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.919+0000 7fdc12000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c33b30 0x563443c35f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44834/0 (socket says 172.21.15.28:44834) 2024-04-24T08:12:40.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.919+0000 7fdc1b400640 1 -- 172.21.15.28:0/3268394748 learned_addr learned my addr 172.21.15.28:0/3268394748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:40.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.919+0000 7fdc1aa00640 1 --2- 172.21.15.28:0/3268394748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c29460 0x563443c335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.920+0000 7fdc12000640 1 -- 172.21.15.28:0/3268394748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c29460 msgr2=0x563443c335f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:40.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.920+0000 7fdc12000640 1 --2- 172.21.15.28:0/3268394748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c29460 0x563443c335f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.920+0000 7fdc12000640 1 -- 172.21.15.28:0/3268394748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c28b40 msgr2=0x563443c28f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:40.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.920+0000 7fdc12000640 1 --2- 172.21.15.28:0/3268394748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c28b40 0x563443c28f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.920+0000 7fdc12000640 1 -- 172.21.15.28:0/3268394748 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563443bf1310 con 0x563443c33b30 2024-04-24T08:12:40.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.920+0000 7fdc1b400640 1 --2- 172.21.15.28:0/3268394748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c28b40 0x563443c28f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:40.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.920+0000 7fdc20a00640 1 --2- 172.21.15.28:0/2707275159 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdc000a3d20 0x7fdc000a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.920+0000 7fdc12000640 1 --2- 172.21.15.28:0/3268394748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c33b30 0x563443c35f20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdc04005fd0 tx=0x7fdc04011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3675b5c9fa3730a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:40.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.920+0000 7fdc20a00640 1 -- 172.21.15.28:0/2707275159 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fdc0c0f1400 con 0x7fdc1c0743a0 2024-04-24T08:12:40.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.920+0000 7fdc1b400640 1 --2- 172.21.15.28:0/2707275159 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdc000a3d20 0x7fdc000a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.920+0000 7fdc0be00640 1 -- 172.21.15.28:0/3268394748 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0401e070 con 0x563443c33b30 2024-04-24T08:12:40.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.920+0000 7fdc0be00640 1 -- 172.21.15.28:0/3268394748 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc040035f0 con 0x563443c33b30 2024-04-24T08:12:40.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.920+0000 7fdc0be00640 1 -- 172.21.15.28:0/3268394748 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc04019690 con 0x563443c33b30 2024-04-24T08:12:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.921+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3268394748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c33b30 msgr2=0x563443c35f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.921+0000 7fdc1b400640 1 --2- 172.21.15.28:0/2707275159 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdc000a3d20 0x7fdc000a61e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdc140058c0 tx=0x7fdc14005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.921+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/3268394748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c33b30 0x563443c35f20 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdc04005fd0 tx=0x7fdc04011810 comp rx=0 tx=0).stop 2024-04-24T08:12:40.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.921+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3268394748 shutdown_connections 2024-04-24T08:12:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.921+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/3268394748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c33b30 0x563443c35f20 secure :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fdc04005fd0 tx=0x7fdc04011810 comp rx=0 tx=0).stop 2024-04-24T08:12:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.921+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/3268394748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c29460 0x563443c335f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.921+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/3268394748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c28b40 0x563443c28f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.921+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3268394748 >> 172.21.15.28:0/3268394748 conn(0x563443c24e60 msgr2=0x563443c25240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.922+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3268394748 shutdown_connections 2024-04-24T08:12:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.922+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3268394748 wait complete. 2024-04-24T08:12:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.922+0000 7fdc24c1a740 1 -- start start 2024-04-24T08:12:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.922+0000 7fdc24c1a740 1 Processor -- start 2024-04-24T08:12:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.923+0000 7fdc24c1a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c28b40 0x563443c30a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.923+0000 7fdc24c1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c29460 0x563443c30fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.923+0000 7fdc24c1a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c31500 0x563443c3fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.923+0000 7fdc24c1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563443c369c0 con 0x563443c29460 2024-04-24T08:12:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.923+0000 7fdc24c1a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563443b47810 con 0x563443c31500 2024-04-24T08:12:40.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.923+0000 7fdc24c1a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563443c36b40 con 0x563443c28b40 2024-04-24T08:12:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.923+0000 7fdc24c1a740 10 client.0 _ll_register_callbacks cb 0x563443b15140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.923+0000 7fdc1aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c28b40 0x563443c30a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.923+0000 7fdc1aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c28b40 0x563443c30a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44850/0 (socket says 172.21.15.28:44850) 2024-04-24T08:12:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.923+0000 7fdc1aa00640 1 -- 172.21.15.28:0/2816287125 learned_addr learned my addr 172.21.15.28:0/2816287125 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.923+0000 7fdc1b400640 1 --2- 172.21.15.28:0/2816287125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c31500 0x563443c3fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.923+0000 7fdc12000640 1 --2- 172.21.15.28:0/2816287125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c29460 0x563443c30fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.924+0000 7fdc1aa00640 1 -- 172.21.15.28:0/2816287125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c31500 msgr2=0x563443c3fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.924+0000 7fdc1aa00640 1 --2- 172.21.15.28:0/2816287125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c31500 0x563443c3fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.924+0000 7fdc1aa00640 1 -- 172.21.15.28:0/2816287125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c29460 msgr2=0x563443c30fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.924+0000 7fdc1aa00640 1 --2- 172.21.15.28:0/2816287125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c29460 0x563443c30fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.924+0000 7fdc1aa00640 1 -- 172.21.15.28:0/2816287125 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563443b583f0 con 0x563443c28b40 2024-04-24T08:12:40.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.924+0000 7fdc1aa00640 1 --2- 172.21.15.28:0/2816287125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c28b40 0x563443c30a80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c003fd0 tx=0x7fdc0c0b9680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=174485e0ca512f8f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.924+0000 7fdc08c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:12:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.924+0000 7fdc08c00640 1 -- 172.21.15.28:0/2816287125 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c030820 con 0x563443c28b40 2024-04-24T08:12:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.924+0000 7fdc08c00640 1 -- 172.21.15.28:0/2816287125 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fdc0c0eed50 con 0x563443c28b40 2024-04-24T08:12:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.924+0000 7fdc24c1a740 1 -- 172.21.15.28:0/2816287125 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc040035f0 con 0x563443c28b40 2024-04-24T08:12:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.924+0000 7fdc08c00640 1 -- 172.21.15.28:0/2816287125 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdc0c0eef50 con 0x563443c28b40 2024-04-24T08:12:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.924+0000 7fdc24c1a740 10 client.25852 fetch_fsmap learned FSMap version 25 2024-04-24T08:12:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.924+0000 7fdc24c1a740 1 -- 172.21.15.28:0/2816287125 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5634438a03f0 con 0x563443c28b40 2024-04-24T08:12:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.925+0000 7fdc08c00640 1 -- 172.21.15.28:0/2816287125 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc040035f0 con 0x563443c28b40 2024-04-24T08:12:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.925+0000 7fdc08c00640 1 -- 172.21.15.28:0/2816287125 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fdc0c0c5b00 con 0x563443c28b40 2024-04-24T08:12:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.925+0000 7fdc24c1a740 10 client.25852 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:12:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.925+0000 7fdc24c1a740 10 client.25852 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:12:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.925+0000 7fdc24c1a740 20 client.25852 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.925+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/2816287125 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563443c44360 0x563443c467b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.925+0000 7fdc24c1a740 4 client.25852 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.925+0000 7fdc24c1a740 1 -- 172.21.15.28:0/2816287125 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563443b9bbf0 con 0x563443c44360 2024-04-24T08:12:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.925+0000 7fdc12000640 1 --2- 172.21.15.28:0/2816287125 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563443c44360 0x563443c467b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.926+0000 7fdc12000640 1 --2- 172.21.15.28:0/2816287125 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563443c44360 0x563443c467b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1a03634ec0cc7e2c server_cookie=599eb2f81d7647dc in_seq=0 out_seq=0 2024-04-24T08:12:40.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.926+0000 7fdc08c00640 10 client.25852 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:12:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.927+0000 7fdc08c00640 1 -- 172.21.15.28:0/2816287125 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563443b9bbf0 con 0x563443c44360 2024-04-24T08:12:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.927+0000 7fdc08c00640 10 client.25852 handle_command_reply: tid=0 2024-04-24T08:12:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.927+0000 7fdc24c1a740 1 client.25852 shutdown 2024-04-24T08:12:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.927+0000 7fdc24c1a740 1 -- 172.21.15.28:0/2816287125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c28b40 msgr2=0x563443c30a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.927+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/2816287125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c28b40 0x563443c30a80 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c003fd0 tx=0x7fdc0c0b9680 comp rx=0 tx=0).stop 2024-04-24T08:12:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.928+0000 7fdc24c1a740 1 -- 172.21.15.28:0/2816287125 shutdown_connections 2024-04-24T08:12:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.928+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/2816287125 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563443c44360 0x563443c467b0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.928+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/2816287125 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c29460 0x563443c30fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.928+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/2816287125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c31500 0x563443c3fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.928+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/2816287125 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c28b40 0x563443c30a80 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.928+0000 7fdc08c00640 0 client.25852 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:12:40.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.928+0000 7fdc24c1a740 1 -- 172.21.15.28:0/2816287125 >> 172.21.15.28:0/2816287125 conn(0x563443c24e60 msgr2=0x563443c2abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:40.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.928+0000 7fdc24c1a740 1 -- 172.21.15.28:0/2816287125 shutdown_connections 2024-04-24T08:12:40.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.928+0000 7fdc24c1a740 1 -- 172.21.15.28:0/2816287125 wait complete. 2024-04-24T08:12:40.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.928+0000 7fdc24c1a740 20 client.25852 trim_cache size 0 max 16384 2024-04-24T08:12:40.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.943+0000 7fdc24c1a740 1 Processor -- start 2024-04-24T08:12:40.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.943+0000 7fdc24c1a740 1 -- start start 2024-04-24T08:12:40.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.943+0000 7fdc24c1a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c28b40 0x563443c2b360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.943+0000 7fdc24c1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c29460 0x563443c2f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.943+0000 7fdc24c1a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c31500 0x563443c4c050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.943+0000 7fdc24c1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1b6ae0 con 0x563443c29460 2024-04-24T08:12:40.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.943+0000 7fdc24c1a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1b43b0 con 0x563443c31500 2024-04-24T08:12:40.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.943+0000 7fdc24c1a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1b2200 con 0x563443c28b40 2024-04-24T08:12:40.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.944+0000 7fdc12000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c29460 0x563443c2f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.944+0000 7fdc12000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c29460 0x563443c2f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58978/0 (socket says 172.21.15.28:58978) 2024-04-24T08:12:40.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.944+0000 7fdc12000640 1 -- 172.21.15.28:0/3752531795 learned_addr learned my addr 172.21.15.28:0/3752531795 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.944+0000 7fdc1aa00640 1 --2- 172.21.15.28:0/3752531795 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c28b40 0x563443c2b360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.944+0000 7fdc1b400640 1 --2- 172.21.15.28:0/3752531795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c31500 0x563443c4c050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.944+0000 7fdc12000640 1 -- 172.21.15.28:0/3752531795 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c28b40 msgr2=0x563443c2b360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.944+0000 7fdc12000640 1 --2- 172.21.15.28:0/3752531795 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c28b40 0x563443c2b360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.944+0000 7fdc12000640 1 -- 172.21.15.28:0/3752531795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c31500 msgr2=0x563443c4c050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.944+0000 7fdc12000640 1 --2- 172.21.15.28:0/3752531795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c31500 0x563443c4c050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.944+0000 7fdc12000640 1 -- 172.21.15.28:0/3752531795 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56344396a7d0 con 0x563443c29460 2024-04-24T08:12:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.945+0000 7fdc12000640 1 --2- 172.21.15.28:0/3752531795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c29460 0x563443c2f930 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdc04009690 tx=0x7fdc040092e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40c66b724257adfe server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.945+0000 7fdc0b400640 1 -- 172.21.15.28:0/3752531795 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0402f070 con 0x563443c29460 2024-04-24T08:12:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.945+0000 7fdc0b400640 1 -- 172.21.15.28:0/3752531795 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc04029030 con 0x563443c29460 2024-04-24T08:12:40.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.945+0000 7fdc0b400640 1 -- 172.21.15.28:0/3752531795 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0401e040 con 0x563443c29460 2024-04-24T08:12:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.946+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3752531795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c29460 msgr2=0x563443c2f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.946+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/3752531795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c29460 0x563443c2f930 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdc04009690 tx=0x7fdc040092e0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.946+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3752531795 shutdown_connections 2024-04-24T08:12:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.946+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/3752531795 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c29460 0x563443c2f930 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.946+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/3752531795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c31500 0x563443c4c050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.946+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/3752531795 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c28b40 0x563443c2b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.946+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3752531795 >> 172.21.15.28:0/3752531795 conn(0x563443c24e60 msgr2=0x563443c2f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.946+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3752531795 shutdown_connections 2024-04-24T08:12:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.946+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3752531795 wait complete. 2024-04-24T08:12:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.946+0000 7fdc24c1a740 1 -- start start 2024-04-24T08:12:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.947+0000 7fdc24c1a740 1 Processor -- start 2024-04-24T08:12:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.947+0000 7fdc24c1a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c28b40 0x563443c3be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.947+0000 7fdc24c1a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c29460 0x563443c37360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.947+0000 7fdc24c1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c31500 0x563443c378a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.947+0000 7fdc24c1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1b2870 con 0x563443c31500 2024-04-24T08:12:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.947+0000 7fdc24c1a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1b3ce0 con 0x563443c28b40 2024-04-24T08:12:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.947+0000 7fdc24c1a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdc1c1b2f40 con 0x563443c29460 2024-04-24T08:12:40.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.947+0000 7fdc24c1a740 10 client.0 _ll_register_callbacks cb 0x7fdc0c0f4ba0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:40.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.947+0000 7fdc1b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c31500 0x563443c378a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.947+0000 7fdc12000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c29460 0x563443c37360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.947+0000 7fdc12000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c29460 0x563443c37360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44862/0 (socket says 172.21.15.28:44862) 2024-04-24T08:12:40.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.947+0000 7fdc12000640 1 -- 172.21.15.28:0/3643040294 learned_addr learned my addr 172.21.15.28:0/3643040294 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:40.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.947+0000 7fdc1aa00640 1 --2- 172.21.15.28:0/3643040294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c28b40 0x563443c3be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.948+0000 7fdc12000640 1 -- 172.21.15.28:0/3643040294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c28b40 msgr2=0x563443c3be80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:40.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.948+0000 7fdc12000640 1 --2- 172.21.15.28:0/3643040294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c28b40 0x563443c3be80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.948+0000 7fdc12000640 1 -- 172.21.15.28:0/3643040294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c31500 msgr2=0x563443c378a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:40.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.948+0000 7fdc12000640 1 --2- 172.21.15.28:0/3643040294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c31500 0x563443c378a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.948+0000 7fdc12000640 1 -- 172.21.15.28:0/3643040294 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563443c37de0 con 0x563443c29460 2024-04-24T08:12:40.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.948+0000 7fdc12000640 1 --2- 172.21.15.28:0/3643040294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c29460 0x563443c37360 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdc04011d10 tx=0x7fdc04009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=162e95483fdf300 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:40.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.948+0000 7fdc08c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:12:40.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.948+0000 7fdc08c00640 1 -- 172.21.15.28:0/3643040294 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc0402f070 con 0x563443c29460 2024-04-24T08:12:40.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.949+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3643040294 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdc04029030 con 0x563443c29460 2024-04-24T08:12:40.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.949+0000 7fdc08c00640 1 -- 172.21.15.28:0/3643040294 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fdc040035f0 con 0x563443c29460 2024-04-24T08:12:40.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.949+0000 7fdc08c00640 1 -- 172.21.15.28:0/3643040294 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fdc0401e040 con 0x563443c29460 2024-04-24T08:12:40.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.949+0000 7fdc24c1a740 10 client.25858 fetch_fsmap learned FSMap version 25 2024-04-24T08:12:40.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.949+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3643040294 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x563443caa5d0 con 0x563443c29460 2024-04-24T08:12:40.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.949+0000 7fdc08c00640 1 -- 172.21.15.28:0/3643040294 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc04029030 con 0x563443c29460 2024-04-24T08:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.950+0000 7fdc08c00640 1 -- 172.21.15.28:0/3643040294 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fdc0402a6a0 con 0x563443c29460 2024-04-24T08:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.950+0000 7fdc24c1a740 10 client.25858 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.950+0000 7fdc24c1a740 10 client.25858 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.950+0000 7fdc24c1a740 20 client.25858 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.950+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/3643040294 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563443caaa60 0x563443cace50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.950+0000 7fdc24c1a740 4 client.25858 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.950+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3643040294 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563443b95a60 con 0x563443caaa60 2024-04-24T08:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.950+0000 7fdc1aa00640 1 --2- 172.21.15.28:0/3643040294 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563443caaa60 0x563443cace50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.951+0000 7fdc1aa00640 1 --2- 172.21.15.28:0/3643040294 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563443caaa60 0x563443cace50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=60243c56555b1db8 server_cookie=bc8f2476410bfc5c in_seq=0 out_seq=0 2024-04-24T08:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.951+0000 7fdc08c00640 10 client.25858 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:12:40.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.951+0000 7fdc08c00640 1 -- 172.21.15.28:0/3643040294 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563443b95a60 con 0x563443caaa60 2024-04-24T08:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.951+0000 7fdc08c00640 10 client.25858 handle_command_reply: tid=0 2024-04-24T08:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.951+0000 7fdc24c1a740 1 client.25858 shutdown 2024-04-24T08:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.951+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3643040294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c29460 msgr2=0x563443c37360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.951+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/3643040294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c29460 0x563443c37360 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdc04011d10 tx=0x7fdc04009f80 comp rx=0 tx=0).stop 2024-04-24T08:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.951+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3643040294 shutdown_connections 2024-04-24T08:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.951+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/3643040294 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563443caaa60 0x563443cace50 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.951+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/3643040294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563443c31500 0x563443c378a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.951+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/3643040294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563443c29460 0x563443c37360 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.951+0000 7fdc24c1a740 1 --2- 172.21.15.28:0/3643040294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563443c28b40 0x563443c3be80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.951+0000 7fdc08c00640 0 client.25858 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.951+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3643040294 >> 172.21.15.28:0/3643040294 conn(0x563443c24e60 msgr2=0x563443c2fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.951+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3643040294 shutdown_connections 2024-04-24T08:12:40.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.951+0000 7fdc24c1a740 1 -- 172.21.15.28:0/3643040294 wait complete. 2024-04-24T08:12:40.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.951+0000 7fdc24c1a740 20 client.25858 trim_cache size 0 max 16384 2024-04-24T08:12:40.956 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.952+0000 7fdc1be00640 1 -- 172.21.15.28:0/2707275159 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdc000a3d20 msgr2=0x7fdc000a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.952+0000 7fdc1be00640 1 --2- 172.21.15.28:0/2707275159 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdc000a3d20 0x7fdc000a61e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fdc140058c0 tx=0x7fdc14005f90 comp rx=0 tx=0).stop 2024-04-24T08:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.952+0000 7fdc1be00640 1 -- 172.21.15.28:0/2707275159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c0743a0 msgr2=0x7fdc1c19edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.952+0000 7fdc1be00640 1 --2- 172.21.15.28:0/2707275159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c0743a0 0x7fdc1c19edb0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c000f80 tx=0x7fdc0c030630 comp rx=0 tx=0).stop 2024-04-24T08:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.952+0000 7fdc1be00640 1 -- 172.21.15.28:0/2707275159 shutdown_connections 2024-04-24T08:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.952+0000 7fdc1be00640 1 --2- 172.21.15.28:0/2707275159 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdc000a3d20 0x7fdc000a61e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.952+0000 7fdc1be00640 1 --2- 172.21.15.28:0/2707275159 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc1c112eb0 0x7fdc1c1a3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.952+0000 7fdc1be00640 1 --2- 172.21.15.28:0/2707275159 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc1c074cc0 0x7fdc1c19f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.952+0000 7fdc1be00640 1 --2- 172.21.15.28:0/2707275159 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc1c0743a0 0x7fdc1c19edb0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:40.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.952+0000 7fdc1be00640 1 -- 172.21.15.28:0/2707275159 >> 172.21.15.28:0/2707275159 conn(0x7fdc1c0fd9d0 msgr2=0x7fdc1c0ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:40.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.952+0000 7fdc1be00640 1 -- 172.21.15.28:0/2707275159 shutdown_connections 2024-04-24T08:12:40.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:40.953+0000 7fdc1be00640 1 -- 172.21.15.28:0/2707275159 wait complete. 2024-04-24T08:12:40.971 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:12:40.972 INFO:tasks.workunit.client.0.smithi028.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/ffsb' 2024-04-24T08:12:40.973 INFO:tasks.workunit.client.0.smithi028.stderr:+ cd .. 2024-04-24T08:12:40.979 INFO:tasks.workunit.client.0.smithi028.stderr:+ mkdir tmp 2024-04-24T08:12:40.979 INFO:tasks.workunit.client.0.smithi028.stderr:+ cd tmp 2024-04-24T08:12:40.979 INFO:tasks.workunit.client.0.smithi028.stderr:+ for f in $mydir/*.ffsb 2024-04-24T08:12:40.979 INFO:tasks.workunit.client.0.smithi028.stderr:+ ../ffsb/ffsb /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/random_write.32.ffsb 2024-04-24T08:12:40.979 INFO:tasks.workunit.client.0.smithi028.stdout:FFSB version 6.0-RC2 started 2024-04-24T08:12:40.979 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-24T08:12:40.985 INFO:tasks.workunit.client.0.smithi028.stdout:benchmark time = 300 2024-04-24T08:12:40.985 INFO:tasks.workunit.client.0.smithi028.stdout:ThreadGroup 0 2024-04-24T08:12:40.985 INFO:tasks.workunit.client.0.smithi028.stdout:================ 2024-04-24T08:12:40.985 INFO:tasks.workunit.client.0.smithi028.stdout: num_threads = 32 2024-04-24T08:12:40.985 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-24T08:12:40.985 INFO:tasks.workunit.client.0.smithi028.stdout: read_random = off 2024-04-24T08:12:40.985 INFO:tasks.workunit.client.0.smithi028.stdout: read_size = 0 (0B) 2024-04-24T08:12:40.985 INFO:tasks.workunit.client.0.smithi028.stdout: read_blocksize = 0 (0B) 2024-04-24T08:12:40.985 INFO:tasks.workunit.client.0.smithi028.stdout: read_skip = off 2024-04-24T08:12:40.985 INFO:tasks.workunit.client.0.smithi028.stdout: read_skipsize = 0 (0B) 2024-04-24T08:12:40.985 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-24T08:12:40.986 INFO:tasks.workunit.client.0.smithi028.stdout: write_random = on 2024-04-24T08:12:40.986 INFO:tasks.workunit.client.0.smithi028.stdout: write_size = 5242880 (5MB) 2024-04-24T08:12:40.986 INFO:tasks.workunit.client.0.smithi028.stdout: fsync_file = 0 2024-04-24T08:12:40.986 INFO:tasks.workunit.client.0.smithi028.stdout: write_blocksize = 4096 (4KB) 2024-04-24T08:12:40.986 INFO:tasks.workunit.client.0.smithi028.stdout: wait time = 0 2024-04-24T08:12:40.986 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-24T08:12:40.986 INFO:tasks.workunit.client.0.smithi028.stdout: op weights 2024-04-24T08:12:40.986 INFO:tasks.workunit.client.0.smithi028.stdout: read = 0 (0.00%) 2024-04-24T08:12:40.986 INFO:tasks.workunit.client.0.smithi028.stdout: readall = 0 (0.00%) 2024-04-24T08:12:40.986 INFO:tasks.workunit.client.0.smithi028.stdout: write = 1 (100.00%) 2024-04-24T08:12:40.986 INFO:tasks.workunit.client.0.smithi028.stdout: create = 0 (0.00%) 2024-04-24T08:12:40.986 INFO:tasks.workunit.client.0.smithi028.stdout: append = 0 (0.00%) 2024-04-24T08:12:40.987 INFO:tasks.workunit.client.0.smithi028.stdout: delete = 0 (0.00%) 2024-04-24T08:12:40.987 INFO:tasks.workunit.client.0.smithi028.stdout: metaop = 0 (0.00%) 2024-04-24T08:12:40.987 INFO:tasks.workunit.client.0.smithi028.stdout: createdir = 0 (0.00%) 2024-04-24T08:12:40.987 INFO:tasks.workunit.client.0.smithi028.stdout: stat = 0 (0.00%) 2024-04-24T08:12:40.987 INFO:tasks.workunit.client.0.smithi028.stdout: writeall = 0 (0.00%) 2024-04-24T08:12:40.987 INFO:tasks.workunit.client.0.smithi028.stdout: writeall_fsync = 0 (0.00%) 2024-04-24T08:12:40.987 INFO:tasks.workunit.client.0.smithi028.stdout: open_close = 0 (0.00%) 2024-04-24T08:12:40.987 INFO:tasks.workunit.client.0.smithi028.stdout: write_fsync = 0 (0.00%) 2024-04-24T08:12:40.987 INFO:tasks.workunit.client.0.smithi028.stdout: create_fsync = 0 (0.00%) 2024-04-24T08:12:40.987 INFO:tasks.workunit.client.0.smithi028.stdout: append_fsync = 0 (0.00%) 2024-04-24T08:12:40.987 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-24T08:12:40.987 INFO:tasks.workunit.client.0.smithi028.stdout:FileSystem . 2024-04-24T08:12:40.987 INFO:tasks.workunit.client.0.smithi028.stdout:========== 2024-04-24T08:12:40.987 INFO:tasks.workunit.client.0.smithi028.stdout: num_dirs = 0 2024-04-24T08:12:40.988 INFO:tasks.workunit.client.0.smithi028.stdout: starting files = 128 2024-04-24T08:12:40.988 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-24T08:12:40.988 INFO:tasks.workunit.client.0.smithi028.stdout: min file size = 104857600 (100MB) 2024-04-24T08:12:40.988 INFO:tasks.workunit.client.0.smithi028.stdout: max file size = 104857600 (100MB) 2024-04-24T08:12:40.988 INFO:tasks.workunit.client.0.smithi028.stdout: directio = off 2024-04-24T08:12:40.988 INFO:tasks.workunit.client.0.smithi028.stdout: alignedio = on 2024-04-24T08:12:40.988 INFO:tasks.workunit.client.0.smithi028.stdout: bufferedio = off 2024-04-24T08:12:40.988 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-24T08:12:40.988 INFO:tasks.workunit.client.0.smithi028.stdout: aging is off 2024-04-24T08:12:40.989 INFO:tasks.workunit.client.0.smithi028.stdout: current utilization = 0.00% 2024-04-24T08:12:40.989 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-24T08:12:40.989 INFO:tasks.workunit.client.0.smithi028.stdout:checking existing fs: . 2024-04-24T08:12:40.989 INFO:tasks.workunit.client.0.smithi028.stdout:recreating new fileset 2024-04-24T08:12:40.989 INFO:tasks.workunit.client.0.smithi028.stderr:opendir: No such file or directory 2024-04-24T08:12:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.211+0000 7f13ebc00640 1 -- 172.21.15.28:0/2684371772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13e41044b0 msgr2=0x7f13e4104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.211+0000 7f13ebc00640 1 --2- 172.21.15.28:0/2684371772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13e41044b0 0x7f13e4104890 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f13d8009900 tx=0x7f13d802a410 comp rx=0 tx=0).stop 2024-04-24T08:12:41.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.211+0000 7f13ebc00640 1 -- 172.21.15.28:0/2684371772 shutdown_connections 2024-04-24T08:12:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.211+0000 7f13ebc00640 1 --2- 172.21.15.28:0/2684371772 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e4109580 0x7f13e4111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.211+0000 7f13ebc00640 1 --2- 172.21.15.28:0/2684371772 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13e4104e60 0x7f13e4108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.211+0000 7f13ebc00640 1 --2- 172.21.15.28:0/2684371772 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13e41044b0 0x7f13e4104890 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.211+0000 7f13ebc00640 1 -- 172.21.15.28:0/2684371772 >> 172.21.15.28:0/2684371772 conn(0x7f13e40ffd90 msgr2=0x7f13e41021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.211+0000 7f13ebc00640 1 -- 172.21.15.28:0/2684371772 shutdown_connections 2024-04-24T08:12:41.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.211+0000 7f13ebc00640 1 -- 172.21.15.28:0/2684371772 wait complete. 2024-04-24T08:12:41.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.212+0000 7f13ebc00640 1 Processor -- start 2024-04-24T08:12:41.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.212+0000 7f13ebc00640 1 -- start start 2024-04-24T08:12:41.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.212+0000 7f13ebc00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13e41044b0 0x7f13e41a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.212+0000 7f13ebc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13e4109580 0x7f13e41a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.212+0000 7f13ebc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e419aa60 0x7f13e419af10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.212+0000 7f13ebc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13e4114630 con 0x7f13e419aa60 2024-04-24T08:12:41.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.212+0000 7f13ebc00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f13e41144b0 con 0x7f13e41044b0 2024-04-24T08:12:41.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.212+0000 7f13ebc00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f13e41147b0 con 0x7f13e4109580 2024-04-24T08:12:41.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.213+0000 7f13e3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13e4109580 0x7f13e41a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.213+0000 7f13e9c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e419aa60 0x7f13e419af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.213+0000 7f13e3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13e4109580 0x7f13e41a0da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44880/0 (socket says 172.21.15.28:44880) 2024-04-24T08:12:41.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.213+0000 7f13e9200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13e41044b0 0x7f13e41a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.213+0000 7f13e3e00640 1 -- 172.21.15.28:0/4207019171 learned_addr learned my addr 172.21.15.28:0/4207019171 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:41.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.213+0000 7f13e3e00640 1 -- 172.21.15.28:0/4207019171 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13e41044b0 msgr2=0x7f13e41a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.213+0000 7f13e3e00640 1 --2- 172.21.15.28:0/4207019171 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13e41044b0 0x7f13e41a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.213+0000 7f13e3e00640 1 -- 172.21.15.28:0/4207019171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e419aa60 msgr2=0x7f13e419af10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.213+0000 7f13e3e00640 1 --2- 172.21.15.28:0/4207019171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e419aa60 0x7f13e419af10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.213+0000 7f13e3e00640 1 -- 172.21.15.28:0/4207019171 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f13e419b580 con 0x7f13e4109580 2024-04-24T08:12:41.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.213+0000 7f13e9200640 1 --2- 172.21.15.28:0/4207019171 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13e41044b0 0x7f13e41a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:41.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.213+0000 7f13e9c00640 1 --2- 172.21.15.28:0/4207019171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e419aa60 0x7f13e419af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:41.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.214+0000 7f13e3e00640 1 --2- 172.21.15.28:0/4207019171 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13e4109580 0x7f13e41a0da0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f13d000e9f0 tx=0x7f13d000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:41.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.214+0000 7f13e1600640 1 -- 172.21.15.28:0/4207019171 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d000ccc0 con 0x7f13e4109580 2024-04-24T08:12:41.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.214+0000 7f13e1600640 1 -- 172.21.15.28:0/4207019171 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13d0005500 con 0x7f13e4109580 2024-04-24T08:12:41.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.214+0000 7f13e1600640 1 -- 172.21.15.28:0/4207019171 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0010690 con 0x7f13e4109580 2024-04-24T08:12:41.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.214+0000 7f13ebc00640 1 -- 172.21.15.28:0/4207019171 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f13e419b870 con 0x7f13e4109580 2024-04-24T08:12:41.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.214+0000 7f13ebc00640 1 -- 172.21.15.28:0/4207019171 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13e41a7a20 con 0x7f13e4109580 2024-04-24T08:12:41.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.217+0000 7f13eeb8e740 1 Processor -- start 2024-04-24T08:12:41.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.217+0000 7f13eeb8e740 1 -- start start 2024-04-24T08:12:41.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.218+0000 7f13eeb8e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441bf950 0x560d441bfd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.218+0000 7f13eeb8e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441c0270 0x560d441ca400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.218+0000 7f13eeb8e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441ca940 0x560d441ccd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.218+0000 7f13eeb8e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560d441251d0 con 0x560d441ca940 2024-04-24T08:12:41.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.218+0000 7f13eeb8e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560d440cb450 con 0x560d441bf950 2024-04-24T08:12:41.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.218+0000 7f13eeb8e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560d44121190 con 0x560d441c0270 2024-04-24T08:12:41.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.218+0000 7f13e9c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441bf950 0x560d441bfd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.218+0000 7f13e9200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441c0270 0x560d441ca400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.218+0000 7f13e9c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441bf950 0x560d441bfd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41538/0 (socket says 172.21.15.28:41538) 2024-04-24T08:12:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.218+0000 7f13e9c00640 1 -- 172.21.15.28:0/1020344024 learned_addr learned my addr 172.21.15.28:0/1020344024 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.219+0000 7f13e9200640 1 -- 172.21.15.28:0/1020344024 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441bf950 msgr2=0x560d441bfd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.219+0000 7f13e9200640 1 --2- 172.21.15.28:0/1020344024 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441bf950 0x560d441bfd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.219+0000 7f13e9200640 1 -- 172.21.15.28:0/1020344024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441ca940 msgr2=0x560d441ccd30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:12:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.219+0000 7f13e9200640 1 --2- 172.21.15.28:0/1020344024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441ca940 0x560d441ccd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.219+0000 7f13e9200640 1 -- 172.21.15.28:0/1020344024 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d44188310 con 0x560d441c0270 2024-04-24T08:12:41.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.219+0000 7f13e9c00640 1 --2- 172.21.15.28:0/1020344024 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441bf950 0x560d441bfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:41.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.219+0000 7f13e9200640 1 --2- 172.21.15.28:0/1020344024 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441c0270 0x560d441ca400 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f13d8005940 tx=0x7f13d8005c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=152329ba1a849519 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.219+0000 7f13d6000640 1 -- 172.21.15.28:0/1020344024 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d8035040 con 0x560d441c0270 2024-04-24T08:12:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.219+0000 7f13d6000640 1 -- 172.21.15.28:0/1020344024 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13d8031870 con 0x560d441c0270 2024-04-24T08:12:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.219+0000 7f13d6000640 1 -- 172.21.15.28:0/1020344024 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d80049d0 con 0x560d441c0270 2024-04-24T08:12:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.219+0000 7f13e1600640 1 -- 172.21.15.28:0/4207019171 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f13d00046c0 con 0x7f13e4109580 2024-04-24T08:12:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.220+0000 7f13eeb8e740 1 -- 172.21.15.28:0/1020344024 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441c0270 msgr2=0x560d441ca400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.220+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/1020344024 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441c0270 0x560d441ca400 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f13d8005940 tx=0x7f13d8005c00 comp rx=0 tx=0).stop 2024-04-24T08:12:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.220+0000 7f13eeb8e740 1 -- 172.21.15.28:0/1020344024 shutdown_connections 2024-04-24T08:12:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.220+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/1020344024 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441ca940 0x560d441ccd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.220+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/1020344024 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441c0270 0x560d441ca400 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.220+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/1020344024 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441bf950 0x560d441bfd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.220+0000 7f13eeb8e740 1 -- 172.21.15.28:0/1020344024 >> 172.21.15.28:0/1020344024 conn(0x560d441bbc70 msgr2=0x560d441bc050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.220+0000 7f13eeb8e740 1 -- 172.21.15.28:0/1020344024 shutdown_connections 2024-04-24T08:12:41.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.220+0000 7f13eeb8e740 1 -- 172.21.15.28:0/1020344024 wait complete. 2024-04-24T08:12:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.221+0000 7f13e1600640 1 --2- 172.21.15.28:0/4207019171 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f13cc0a3c30 0x7f13cc0a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.221+0000 7f13eeb8e740 1 -- start start 2024-04-24T08:12:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.221+0000 7f13e1600640 1 -- 172.21.15.28:0/4207019171 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f13d00d0b70 con 0x7f13e4109580 2024-04-24T08:12:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.221+0000 7f13e9c00640 1 --2- 172.21.15.28:0/4207019171 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f13cc0a3c30 0x7f13cc0a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.221+0000 7f13eeb8e740 1 Processor -- start 2024-04-24T08:12:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.222+0000 7f13e9c00640 1 --2- 172.21.15.28:0/4207019171 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f13cc0a3c30 0x7f13cc0a60f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f13dc009fd0 tx=0x7f13dc009500 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.222+0000 7f13eeb8e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441bf950 0x560d441c7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.222+0000 7f13eeb8e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441c0270 0x560d441c7c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.222+0000 7f13eeb8e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441ca940 0x560d441c81a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.222+0000 7f13eeb8e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560d441cd770 con 0x560d441ca940 2024-04-24T08:12:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.222+0000 7f13eeb8e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560d440de810 con 0x560d441c0270 2024-04-24T08:12:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.222+0000 7f13eeb8e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560d441cd8f0 con 0x560d441bf950 2024-04-24T08:12:41.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.222+0000 7f13eeb8e740 10 client.0 _ll_register_callbacks cb 0x560d440ac140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.222+0000 7f13e9200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441bf950 0x560d441c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.222+0000 7f13e9200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441bf950 0x560d441c7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44900/0 (socket says 172.21.15.28:44900) 2024-04-24T08:12:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.222+0000 7f13e9200640 1 -- 172.21.15.28:0/1010744934 learned_addr learned my addr 172.21.15.28:0/1010744934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.222+0000 7f13e9c00640 1 --2- 172.21.15.28:0/1010744934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441ca940 0x560d441c81a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.223+0000 7f13e3e00640 1 --2- 172.21.15.28:0/1010744934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441c0270 0x560d441c7c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.223+0000 7f13e9200640 1 -- 172.21.15.28:0/1010744934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441c0270 msgr2=0x560d441c7c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.223+0000 7f13e9200640 1 --2- 172.21.15.28:0/1010744934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441c0270 0x560d441c7c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.223+0000 7f13e9200640 1 -- 172.21.15.28:0/1010744934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441ca940 msgr2=0x560d441c81a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.223+0000 7f13e9200640 1 --2- 172.21.15.28:0/1010744934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441ca940 0x560d441c81a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.223+0000 7f13e9200640 1 -- 172.21.15.28:0/1010744934 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d440ef3f0 con 0x560d441bf950 2024-04-24T08:12:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.223+0000 7f13e9200640 1 --2- 172.21.15.28:0/1010744934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441bf950 0x560d441c7720 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f13d8031a60 tx=0x7f13d802a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=da013d94a6e27c2e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.223+0000 7f13caa00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:12:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.223+0000 7f13caa00640 1 -- 172.21.15.28:0/1010744934 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d8035070 con 0x560d441bf950 2024-04-24T08:12:41.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.224+0000 7f13caa00640 1 -- 172.21.15.28:0/1010744934 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f13d8031670 con 0x560d441bf950 2024-04-24T08:12:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.224+0000 7f13eeb8e740 1 -- 172.21.15.28:0/1010744934 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13d8031870 con 0x560d441bf950 2024-04-24T08:12:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.224+0000 7f13caa00640 1 -- 172.21.15.28:0/1010744934 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f13d8004240 con 0x560d441bf950 2024-04-24T08:12:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.224+0000 7f13eeb8e740 10 client.25870 fetch_fsmap learned FSMap version 25 2024-04-24T08:12:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.224+0000 7f13eeb8e740 1 -- 172.21.15.28:0/1010744934 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560d43e373f0 con 0x560d441bf950 2024-04-24T08:12:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.224+0000 7f13caa00640 1 -- 172.21.15.28:0/1010744934 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13d8031870 con 0x560d441bf950 2024-04-24T08:12:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.224+0000 7f13caa00640 1 -- 172.21.15.28:0/1010744934 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f13d8046490 con 0x560d441bf950 2024-04-24T08:12:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.224+0000 7f13eeb8e740 10 client.25870 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:12:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.224+0000 7f13eeb8e740 10 client.25870 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:12:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.224+0000 7f13eeb8e740 20 client.25870 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.225+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/1010744934 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560d441da9e0 0x560d441dcdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.225+0000 7f13eeb8e740 4 client.25870 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.225+0000 7f13eeb8e740 1 -- 172.21.15.28:0/1010744934 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560d44132bf0 con 0x560d441da9e0 2024-04-24T08:12:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.225+0000 7f13e3e00640 1 --2- 172.21.15.28:0/1010744934 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560d441da9e0 0x560d441dcdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.225+0000 7f13e3e00640 1 --2- 172.21.15.28:0/1010744934 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560d441da9e0 0x560d441dcdd0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b178aeae8e488d2c server_cookie=291a5cac5f0bbcd3 in_seq=0 out_seq=0 2024-04-24T08:12:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.225+0000 7f13caa00640 10 client.25870 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:12:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.226+0000 7f13caa00640 1 -- 172.21.15.28:0/1010744934 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560d44132bf0 con 0x560d441da9e0 2024-04-24T08:12:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.226+0000 7f13caa00640 10 client.25870 handle_command_reply: tid=0 2024-04-24T08:12:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.226+0000 7f13eeb8e740 1 client.25870 shutdown 2024-04-24T08:12:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.226+0000 7f13eeb8e740 1 -- 172.21.15.28:0/1010744934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441bf950 msgr2=0x560d441c7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.226+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/1010744934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441bf950 0x560d441c7720 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f13d8031a60 tx=0x7f13d802a410 comp rx=0 tx=0).stop 2024-04-24T08:12:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.226+0000 7f13eeb8e740 1 -- 172.21.15.28:0/1010744934 shutdown_connections 2024-04-24T08:12:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.226+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/1010744934 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560d441da9e0 0x560d441dcdd0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.227+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/1010744934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441ca940 0x560d441c81a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.227+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/1010744934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441c0270 0x560d441c7c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.227+0000 7f13caa00640 0 client.25870 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:12:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.227+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/1010744934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441bf950 0x560d441c7720 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.227+0000 7f13eeb8e740 1 -- 172.21.15.28:0/1010744934 >> 172.21.15.28:0/1010744934 conn(0x560d441bbc70 msgr2=0x560d441c1a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.227+0000 7f13eeb8e740 1 -- 172.21.15.28:0/1010744934 shutdown_connections 2024-04-24T08:12:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.227+0000 7f13eeb8e740 1 -- 172.21.15.28:0/1010744934 wait complete. 2024-04-24T08:12:41.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.227+0000 7f13eeb8e740 20 client.25870 trim_cache size 0 max 16384 2024-04-24T08:12:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.242+0000 7f13eeb8e740 1 Processor -- start 2024-04-24T08:12:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.242+0000 7f13eeb8e740 1 -- start start 2024-04-24T08:12:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.242+0000 7f13eeb8e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441bf950 0x560d441c2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.242+0000 7f13eeb8e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441c0270 0x560d441c6ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.242+0000 7f13eeb8e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441ca940 0x560d441bac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.242+0000 7f13eeb8e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13e41b2850 con 0x560d441bf950 2024-04-24T08:12:41.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.242+0000 7f13eeb8e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f13e41b0120 con 0x560d441ca940 2024-04-24T08:12:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.242+0000 7f13eeb8e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f13e41adf70 con 0x560d441c0270 2024-04-24T08:12:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.243+0000 7f13e9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441bf950 0x560d441c2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.243+0000 7f13e3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441c0270 0x560d441c6ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.243+0000 7f13e9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441bf950 0x560d441c2190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59016/0 (socket says 172.21.15.28:59016) 2024-04-24T08:12:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.243+0000 7f13e9c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441ca940 0x560d441bac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.243+0000 7f13e9200640 1 -- 172.21.15.28:0/4200508893 learned_addr learned my addr 172.21.15.28:0/4200508893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:41.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.243+0000 7f13e3e00640 1 -- 172.21.15.28:0/4200508893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441ca940 msgr2=0x560d441bac70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.243+0000 7f13e3e00640 1 --2- 172.21.15.28:0/4200508893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441ca940 0x560d441bac70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.243+0000 7f13e3e00640 1 -- 172.21.15.28:0/4200508893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441bf950 msgr2=0x560d441c2190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.243+0000 7f13e3e00640 1 --2- 172.21.15.28:0/4200508893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441bf950 0x560d441c2190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.243+0000 7f13e3e00640 1 -- 172.21.15.28:0/4200508893 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560d43f017d0 con 0x560d441c0270 2024-04-24T08:12:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.243+0000 7f13e9200640 1 --2- 172.21.15.28:0/4200508893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441bf950 0x560d441c2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.243+0000 7f13e9c00640 1 --2- 172.21.15.28:0/4200508893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441ca940 0x560d441bac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.243+0000 7f13e3e00640 1 --2- 172.21.15.28:0/4200508893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441c0270 0x560d441c6ef0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f13d00ce2a0 tx=0x7f13d00bd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b01763a4dec2fe83 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.244+0000 7f13d5600640 1 -- 172.21.15.28:0/4200508893 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d00b6070 con 0x560d441c0270 2024-04-24T08:12:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.244+0000 7f13d5600640 1 -- 172.21.15.28:0/4200508893 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13d009caf0 con 0x560d441c0270 2024-04-24T08:12:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.244+0000 7f13d5600640 1 -- 172.21.15.28:0/4200508893 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d0099160 con 0x560d441c0270 2024-04-24T08:12:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.245+0000 7f13eeb8e740 1 -- 172.21.15.28:0/4200508893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441c0270 msgr2=0x560d441c6ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.245+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/4200508893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441c0270 0x560d441c6ef0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f13d00ce2a0 tx=0x7f13d00bd040 comp rx=0 tx=0).stop 2024-04-24T08:12:41.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.245+0000 7f13eeb8e740 1 -- 172.21.15.28:0/4200508893 shutdown_connections 2024-04-24T08:12:41.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.245+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/4200508893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441ca940 0x560d441bac70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.245+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/4200508893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441c0270 0x560d441c6ef0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.245+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/4200508893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441bf950 0x560d441c2190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.245+0000 7f13eeb8e740 1 -- 172.21.15.28:0/4200508893 >> 172.21.15.28:0/4200508893 conn(0x560d441bbc70 msgr2=0x560d441ba560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:41.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.245+0000 7f13eeb8e740 1 -- 172.21.15.28:0/4200508893 shutdown_connections 2024-04-24T08:12:41.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.245+0000 7f13eeb8e740 1 -- 172.21.15.28:0/4200508893 wait complete. 2024-04-24T08:12:41.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.245+0000 7f13eeb8e740 1 -- start start 2024-04-24T08:12:41.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.246+0000 7f13eeb8e740 1 Processor -- start 2024-04-24T08:12:41.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.246+0000 7f13eeb8e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441bf950 0x560d441e01c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.246+0000 7f13eeb8e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441c0270 0x560d441db530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.246+0000 7f13eeb8e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441ca940 0x560d441dba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.246+0000 7f13eeb8e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f13e41ae5e0 con 0x560d441c0270 2024-04-24T08:12:41.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.246+0000 7f13eeb8e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f13e41afa50 con 0x560d441bf950 2024-04-24T08:12:41.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.246+0000 7f13eeb8e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f13e41aecb0 con 0x560d441ca940 2024-04-24T08:12:41.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.246+0000 7f13eeb8e740 10 client.0 _ll_register_callbacks cb 0x7f13d8040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:41.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.246+0000 7f13e3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441c0270 0x560d441db530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.246+0000 7f13e3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441c0270 0x560d441db530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59028/0 (socket says 172.21.15.28:59028) 2024-04-24T08:12:41.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.246+0000 7f13e3e00640 1 -- 172.21.15.28:0/2457338747 learned_addr learned my addr 172.21.15.28:0/2457338747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:41.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.246+0000 7f13e9200640 1 --2- 172.21.15.28:0/2457338747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441bf950 0x560d441e01c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.246+0000 7f13e9c00640 1 --2- 172.21.15.28:0/2457338747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441ca940 0x560d441dba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.247+0000 7f13e3e00640 1 -- 172.21.15.28:0/2457338747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441ca940 msgr2=0x560d441dba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:41.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.247+0000 7f13e3e00640 1 --2- 172.21.15.28:0/2457338747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441ca940 0x560d441dba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.247+0000 7f13e3e00640 1 -- 172.21.15.28:0/2457338747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441bf950 msgr2=0x560d441e01c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:41.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.247+0000 7f13e3e00640 1 --2- 172.21.15.28:0/2457338747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441bf950 0x560d441e01c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.247+0000 7f13e3e00640 1 -- 172.21.15.28:0/2457338747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560d441dc250 con 0x560d441c0270 2024-04-24T08:12:41.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.247+0000 7f13e9200640 1 --2- 172.21.15.28:0/2457338747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441bf950 0x560d441e01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:41.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.247+0000 7f13e3e00640 1 --2- 172.21.15.28:0/2457338747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441c0270 0x560d441db530 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f13d00ce8a0 tx=0x7f13d00ce2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a672ac244b3ee654 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:41.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.247+0000 7f13caa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:12:41.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.247+0000 7f13caa00640 1 -- 172.21.15.28:0/2457338747 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13d00b6040 con 0x560d441c0270 2024-04-24T08:12:41.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.247+0000 7f13caa00640 1 -- 172.21.15.28:0/2457338747 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f13d0099160 con 0x560d441c0270 2024-04-24T08:12:41.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.247+0000 7f13eeb8e740 1 -- 172.21.15.28:0/2457338747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f13d009caf0 con 0x560d441c0270 2024-04-24T08:12:41.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.247+0000 7f13caa00640 1 -- 172.21.15.28:0/2457338747 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f13d00bd0b0 con 0x560d441c0270 2024-04-24T08:12:41.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.248+0000 7f13eeb8e740 10 client.15732 fetch_fsmap learned FSMap version 25 2024-04-24T08:12:41.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.248+0000 7f13eeb8e740 1 -- 172.21.15.28:0/2457338747 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560d44240bc0 con 0x560d441c0270 2024-04-24T08:12:41.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.248+0000 7f13caa00640 1 -- 172.21.15.28:0/2457338747 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13d009caf0 con 0x560d441c0270 2024-04-24T08:12:41.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.248+0000 7f13caa00640 1 -- 172.21.15.28:0/2457338747 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f13d009bfd0 con 0x560d441c0270 2024-04-24T08:12:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.248+0000 7f13eeb8e740 10 client.15732 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:12:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.248+0000 7f13eeb8e740 10 client.15732 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:12:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.248+0000 7f13eeb8e740 20 client.15732 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.248+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/2457338747 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560d44241270 0x560d44243690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.248+0000 7f13eeb8e740 4 client.15732 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:12:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.248+0000 7f13eeb8e740 1 -- 172.21.15.28:0/2457338747 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560d4412ca60 con 0x560d44241270 2024-04-24T08:12:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.249+0000 7f13e9200640 1 --2- 172.21.15.28:0/2457338747 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560d44241270 0x560d44243690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.249+0000 7f13e9200640 1 --2- 172.21.15.28:0/2457338747 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560d44241270 0x560d44243690 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=e91c0fb33e28e3fd server_cookie=3cb261c3d3985037 in_seq=0 out_seq=0 2024-04-24T08:12:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.249+0000 7f13caa00640 10 client.15732 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:12:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.249+0000 7f13caa00640 1 -- 172.21.15.28:0/2457338747 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560d4412ca60 con 0x560d44241270 2024-04-24T08:12:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.249+0000 7f13caa00640 10 client.15732 handle_command_reply: tid=0 2024-04-24T08:12:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.249+0000 7f13eeb8e740 1 client.15732 shutdown 2024-04-24T08:12:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.249+0000 7f13eeb8e740 1 -- 172.21.15.28:0/2457338747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441c0270 msgr2=0x560d441db530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.249+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/2457338747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441c0270 0x560d441db530 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f13d00ce8a0 tx=0x7f13d00ce2c0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.250+0000 7f13eeb8e740 1 -- 172.21.15.28:0/2457338747 shutdown_connections 2024-04-24T08:12:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.250+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/2457338747 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560d44241270 0x560d44243690 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.250+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/2457338747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560d441c0270 0x560d441db530 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.250+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/2457338747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560d441ca940 0x560d441dba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.250+0000 7f13eeb8e740 1 --2- 172.21.15.28:0/2457338747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560d441bf950 0x560d441e01c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.250+0000 7f13eeb8e740 1 -- 172.21.15.28:0/2457338747 >> 172.21.15.28:0/2457338747 conn(0x560d441bbc70 msgr2=0x560d441badf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.250+0000 7f13caa00640 0 client.15732 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:12:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.250+0000 7f13eeb8e740 1 -- 172.21.15.28:0/2457338747 shutdown_connections 2024-04-24T08:12:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.250+0000 7f13eeb8e740 1 -- 172.21.15.28:0/2457338747 wait complete. 2024-04-24T08:12:41.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.250+0000 7f13eeb8e740 20 client.15732 trim_cache size 0 max 16384 2024-04-24T08:12:41.254 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:12:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.251+0000 7f13ebc00640 1 -- 172.21.15.28:0/4207019171 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f13cc0a3c30 msgr2=0x7f13cc0a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.251+0000 7f13ebc00640 1 --2- 172.21.15.28:0/4207019171 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f13cc0a3c30 0x7f13cc0a60f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f13dc009fd0 tx=0x7f13dc009500 comp rx=0 tx=0).stop 2024-04-24T08:12:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.251+0000 7f13ebc00640 1 -- 172.21.15.28:0/4207019171 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13e4109580 msgr2=0x7f13e41a0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.251+0000 7f13ebc00640 1 --2- 172.21.15.28:0/4207019171 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13e4109580 0x7f13e41a0da0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f13d000e9f0 tx=0x7f13d000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.251+0000 7f13ebc00640 1 -- 172.21.15.28:0/4207019171 shutdown_connections 2024-04-24T08:12:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.251+0000 7f13ebc00640 1 --2- 172.21.15.28:0/4207019171 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f13cc0a3c30 0x7f13cc0a60f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.251+0000 7f13ebc00640 1 --2- 172.21.15.28:0/4207019171 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f13e419aa60 0x7f13e419af10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.251+0000 7f13ebc00640 1 --2- 172.21.15.28:0/4207019171 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f13e4109580 0x7f13e41a0da0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.251+0000 7f13ebc00640 1 --2- 172.21.15.28:0/4207019171 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13e41044b0 0x7f13e41a0860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.251+0000 7f13ebc00640 1 -- 172.21.15.28:0/4207019171 >> 172.21.15.28:0/4207019171 conn(0x7f13e40ffd90 msgr2=0x7f13e41010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:41.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.251+0000 7f13ebc00640 1 -- 172.21.15.28:0/4207019171 shutdown_connections 2024-04-24T08:12:41.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.251+0000 7f13ebc00640 1 -- 172.21.15.28:0/4207019171 wait complete. 2024-04-24T08:12:41.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:12:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.521+0000 7f866dc00640 1 -- 172.21.15.28:0/3922803496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8668109270 msgr2=0x7f8668109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.521+0000 7f866dc00640 1 --2- 172.21.15.28:0/3922803496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8668109270 0x7f8668109650 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f865800bd70 tx=0x7f865802c460 comp rx=0 tx=0).stop 2024-04-24T08:12:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.522+0000 7f866dc00640 1 -- 172.21.15.28:0/3922803496 shutdown_connections 2024-04-24T08:12:41.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.522+0000 7f866dc00640 1 --2- 172.21.15.28:0/3922803496 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f866810de20 0x7f86681159f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.522+0000 7f866dc00640 1 --2- 172.21.15.28:0/3922803496 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8668109c20 0x7f866810d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.522+0000 7f866dc00640 1 --2- 172.21.15.28:0/3922803496 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8668109270 0x7f8668109650 secure :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f865800bd70 tx=0x7f865802c460 comp rx=0 tx=0).stop 2024-04-24T08:12:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.522+0000 7f866dc00640 1 -- 172.21.15.28:0/3922803496 >> 172.21.15.28:0/3922803496 conn(0x7f866806d3e0 msgr2=0x7f866806d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:41.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.524+0000 7f866dc00640 1 -- 172.21.15.28:0/3922803496 shutdown_connections 2024-04-24T08:12:41.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.524+0000 7f866dc00640 1 -- 172.21.15.28:0/3922803496 wait complete. 2024-04-24T08:12:41.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.526+0000 7f866dc00640 1 Processor -- start 2024-04-24T08:12:41.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.526+0000 7f866dc00640 1 -- start start 2024-04-24T08:12:41.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.526+0000 7f866dc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f866810de20 0x7f866819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.527+0000 7f866dc00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f866819f6d0 0x7f86681a3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.527+0000 7f866dc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86681a3f90 0x7f86681a4440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.527+0000 7f866dc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8668118a80 con 0x7f866810de20 2024-04-24T08:12:41.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.527+0000 7f866dc00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8668118900 con 0x7f866819f6d0 2024-04-24T08:12:41.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.527+0000 7f866dc00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8668118c00 con 0x7f86681a3f90 2024-04-24T08:12:41.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.527+0000 7f8667400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f866819f6d0 0x7f86681a3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.527+0000 7f8667400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f866819f6d0 0x7f86681a3a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41566/0 (socket says 172.21.15.28:41566) 2024-04-24T08:12:41.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.527+0000 7f8667400640 1 -- 172.21.15.28:0/887335065 learned_addr learned my addr 172.21.15.28:0/887335065 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:41.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.527+0000 7f8667e00640 1 --2- 172.21.15.28:0/887335065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f866810de20 0x7f866819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.528+0000 7f866d200640 1 --2- 172.21.15.28:0/887335065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86681a3f90 0x7f86681a4440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.528+0000 7f8667400640 1 -- 172.21.15.28:0/887335065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86681a3f90 msgr2=0x7f86681a4440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.528+0000 7f8667400640 1 --2- 172.21.15.28:0/887335065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86681a3f90 0x7f86681a4440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.528+0000 7f8667400640 1 -- 172.21.15.28:0/887335065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f866810de20 msgr2=0x7f866819f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.528+0000 7f8667400640 1 --2- 172.21.15.28:0/887335065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f866810de20 0x7f866819f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.528+0000 7f8667400640 1 -- 172.21.15.28:0/887335065 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f86681abd00 con 0x7f866819f6d0 2024-04-24T08:12:41.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.529+0000 7f866d200640 1 --2- 172.21.15.28:0/887335065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86681a3f90 0x7f86681a4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:41.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.529+0000 7f8667e00640 1 --2- 172.21.15.28:0/887335065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f866810de20 0x7f866819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:41.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.529+0000 7f8667400640 1 --2- 172.21.15.28:0/887335065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f866819f6d0 0x7f86681a3a50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f86540078d0 tx=0x7f8654007d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:41.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.531+0000 7f8664c00640 1 -- 172.21.15.28:0/887335065 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865400ebe0 con 0x7f866819f6d0 2024-04-24T08:12:41.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.531+0000 7f8664c00640 1 -- 172.21.15.28:0/887335065 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8654019070 con 0x7f866819f6d0 2024-04-24T08:12:41.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.531+0000 7f866dc00640 1 -- 172.21.15.28:0/887335065 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f86681abf30 con 0x7f866819f6d0 2024-04-24T08:12:41.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.531+0000 7f8664c00640 1 -- 172.21.15.28:0/887335065 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86540136c0 con 0x7f866819f6d0 2024-04-24T08:12:41.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.531+0000 7f866dc00640 1 -- 172.21.15.28:0/887335065 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f86681ac440 con 0x7f866819f6d0 2024-04-24T08:12:41.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.537+0000 7f8671d22740 1 Processor -- start 2024-04-24T08:12:41.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:41 smithi028 ceph-mon[27786]: pgmap v73: 97 pgs: 97 active+clean; 8.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 278 KiB/s wr, 41 op/s 2024-04-24T08:12:41.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:41 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2400309582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:12:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.538+0000 7f8671d22740 1 -- start start 2024-04-24T08:12:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.538+0000 7f8671d22740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa75dc00 0x564daa75dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.538+0000 7f8664c00640 1 -- 172.21.15.28:0/887335065 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f8654013860 con 0x7f866819f6d0 2024-04-24T08:12:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.538+0000 7f8671d22740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa759460 0x564daa7598e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.538+0000 7f8671d22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759e20 0x564daa765b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.538+0000 7f8671d22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564daa6b51d0 con 0x564daa759e20 2024-04-24T08:12:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.538+0000 7f8671d22740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564daa65b450 con 0x564daa759460 2024-04-24T08:12:41.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.538+0000 7f8671d22740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564daa6b1190 con 0x564daa75dc00 2024-04-24T08:12:41.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.539+0000 7f866d200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa75dc00 0x564daa75dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.539+0000 7f866d200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa75dc00 0x564daa75dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44952/0 (socket says 172.21.15.28:44952) 2024-04-24T08:12:41.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.539+0000 7f866d200640 1 -- 172.21.15.28:0/1130967238 learned_addr learned my addr 172.21.15.28:0/1130967238 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:41.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.539+0000 7f8667400640 1 --2- 172.21.15.28:0/1130967238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759e20 0x564daa765b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.539+0000 7f8667e00640 1 --2- 172.21.15.28:0/1130967238 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa759460 0x564daa7598e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.539+0000 7f866d200640 1 -- 172.21.15.28:0/1130967238 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa759460 msgr2=0x564daa7598e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:41.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.541+0000 7f866d200640 1 --2- 172.21.15.28:0/1130967238 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa759460 0x564daa7598e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.541+0000 7f866d200640 1 -- 172.21.15.28:0/1130967238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759e20 msgr2=0x564daa765b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:41.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.541+0000 7f866d200640 1 --2- 172.21.15.28:0/1130967238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759e20 0x564daa765b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.541+0000 7f866d200640 1 -- 172.21.15.28:0/1130967238 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564daa718310 con 0x564daa75dc00 2024-04-24T08:12:41.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.541+0000 7f8667e00640 1 --2- 172.21.15.28:0/1130967238 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa759460 0x564daa7598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:12:41.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.541+0000 7f8667400640 1 --2- 172.21.15.28:0/1130967238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759e20 0x564daa765b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.542+0000 7f866d200640 1 --2- 172.21.15.28:0/1130967238 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa75dc00 0x564daa75dfe0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8660009c00 tx=0x7f8660010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6ca12ed5e190c388 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.542+0000 7f8664c00640 1 --2- 172.21.15.28:0/887335065 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f864c0a3e20 0x7f864c0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.543+0000 7f865d600640 1 -- 172.21.15.28:0/1130967238 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8660004030 con 0x564daa75dc00 2024-04-24T08:12:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.543+0000 7f865d600640 1 -- 172.21.15.28:0/1130967238 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86600041d0 con 0x564daa75dc00 2024-04-24T08:12:41.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.543+0000 7f865d600640 1 -- 172.21.15.28:0/1130967238 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8660012690 con 0x564daa75dc00 2024-04-24T08:12:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.543+0000 7f8664c00640 1 -- 172.21.15.28:0/887335065 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f86540d0bb0 con 0x7f866819f6d0 2024-04-24T08:12:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.544+0000 7f8667e00640 1 --2- 172.21.15.28:0/887335065 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f864c0a3e20 0x7f864c0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.544+0000 7f8671d22740 1 -- 172.21.15.28:0/1130967238 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa75dc00 msgr2=0x564daa75dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.544+0000 7f8671d22740 1 --2- 172.21.15.28:0/1130967238 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa75dc00 0x564daa75dfe0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f8660009c00 tx=0x7f8660010740 comp rx=0 tx=0).stop 2024-04-24T08:12:41.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.545+0000 7f8667e00640 1 --2- 172.21.15.28:0/887335065 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f864c0a3e20 0x7f864c0a62e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f865800bd40 tx=0x7f8658000f40 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.546+0000 7f8671d22740 1 -- 172.21.15.28:0/1130967238 shutdown_connections 2024-04-24T08:12:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.546+0000 7f8671d22740 1 --2- 172.21.15.28:0/1130967238 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759e20 0x564daa765b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.546+0000 7f8671d22740 1 --2- 172.21.15.28:0/1130967238 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa759460 0x564daa7598e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.546+0000 7f8671d22740 1 --2- 172.21.15.28:0/1130967238 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa75dc00 0x564daa75dfe0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.546+0000 7f8671d22740 1 -- 172.21.15.28:0/1130967238 >> 172.21.15.28:0/1130967238 conn(0x564daa74b910 msgr2=0x564daa74bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.546+0000 7f8671d22740 1 -- 172.21.15.28:0/1130967238 shutdown_connections 2024-04-24T08:12:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.546+0000 7f8671d22740 1 -- 172.21.15.28:0/1130967238 wait complete. 2024-04-24T08:12:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.547+0000 7f8671d22740 1 -- start start 2024-04-24T08:12:41.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.547+0000 7f8671d22740 1 Processor -- start 2024-04-24T08:12:41.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.549+0000 7f8671d22740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa759460 0x564daa765b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.549+0000 7f8671d22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759e20 0x564daa762e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.549+0000 7f8671d22740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa75dc00 0x564daa763540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.549+0000 7f8671d22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564daa766190 con 0x564daa759e20 2024-04-24T08:12:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.549+0000 7f8671d22740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564daa66e810 con 0x564daa75dc00 2024-04-24T08:12:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.549+0000 7f8671d22740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564daa766310 con 0x564daa759460 2024-04-24T08:12:41.550 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.549+0000 7f8671d22740 10 client.0 _ll_register_callbacks cb 0x564daa63c140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.550+0000 7f8667e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa759460 0x564daa765b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.550+0000 7f8667400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759e20 0x564daa762e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.550+0000 7f8667400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759e20 0x564daa762e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59046/0 (socket says 172.21.15.28:59046) 2024-04-24T08:12:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.550+0000 7f8667400640 1 -- 172.21.15.28:0/1465748472 learned_addr learned my addr 172.21.15.28:0/1465748472 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.550+0000 7f866d200640 1 --2- 172.21.15.28:0/1465748472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa75dc00 0x564daa763540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.551+0000 7f8667400640 1 -- 172.21.15.28:0/1465748472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa759460 msgr2=0x564daa765b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.551+0000 7f8667400640 1 --2- 172.21.15.28:0/1465748472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa759460 0x564daa765b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.551+0000 7f8667400640 1 -- 172.21.15.28:0/1465748472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa75dc00 msgr2=0x564daa763540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.551+0000 7f8667400640 1 --2- 172.21.15.28:0/1465748472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa75dc00 0x564daa763540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.551+0000 7f8667400640 1 -- 172.21.15.28:0/1465748472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564daa67f3f0 con 0x564daa759e20 2024-04-24T08:12:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.552+0000 7f8667400640 1 --2- 172.21.15.28:0/1465748472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759e20 0x564daa762e00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8654000c00 tx=0x7f865400ae70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59e0f4e21857d21e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:41.552 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.552+0000 7f8652000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:12:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.553+0000 7f8652000640 1 -- 172.21.15.28:0/1465748472 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f865409c430 con 0x564daa759e20 2024-04-24T08:12:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.553+0000 7f8652000640 1 -- 172.21.15.28:0/1465748472 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f8654099140 con 0x564daa759e20 2024-04-24T08:12:41.553 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.553+0000 7f8671d22740 1 -- 172.21.15.28:0/1465748472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f86600041d0 con 0x564daa759e20 2024-04-24T08:12:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.554+0000 7f8652000640 1 -- 172.21.15.28:0/1465748472 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f865409d1d0 con 0x564daa759e20 2024-04-24T08:12:41.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.554+0000 7f8671d22740 10 client.15741 fetch_fsmap learned FSMap version 25 2024-04-24T08:12:41.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.554+0000 7f8671d22740 1 -- 172.21.15.28:0/1465748472 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564daa3c73f0 con 0x564daa759e20 2024-04-24T08:12:41.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.555+0000 7f8652000640 1 -- 172.21.15.28:0/1465748472 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f86600041d0 con 0x564daa759e20 2024-04-24T08:12:41.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.555+0000 7f8652000640 1 -- 172.21.15.28:0/1465748472 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f865409bf20 con 0x564daa759e20 2024-04-24T08:12:41.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.555+0000 7f8671d22740 10 client.15741 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:12:41.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.555+0000 7f8671d22740 10 client.15741 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:12:41.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.555+0000 7f8671d22740 20 client.15741 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:41.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.555+0000 7f8671d22740 1 --2- 172.21.15.28:0/1465748472 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x564daa773540 0x564daa775930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.555+0000 7f8671d22740 4 client.15741 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:41.556 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.555+0000 7f8671d22740 1 -- 172.21.15.28:0/1465748472 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564daa6c2bf0 con 0x564daa773540 2024-04-24T08:12:41.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.556+0000 7f8667e00640 1 --2- 172.21.15.28:0/1465748472 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x564daa773540 0x564daa775930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.557+0000 7f8652000640 10 client.15741 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:41.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.557+0000 7f8667e00640 1 --2- 172.21.15.28:0/1465748472 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x564daa773540 0x564daa775930 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85891c5bc3275977 server_cookie=b3de90c9e26ffe15 in_seq=0 out_seq=0 2024-04-24T08:12:41.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.559+0000 7f8652000640 1 -- 172.21.15.28:0/1465748472 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564daa6c2bf0 con 0x564daa773540 2024-04-24T08:12:41.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.559+0000 7f8652000640 10 client.15741 handle_command_reply: tid=0 2024-04-24T08:12:41.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.559+0000 7f8671d22740 1 client.15741 shutdown 2024-04-24T08:12:41.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.560+0000 7f8671d22740 1 -- 172.21.15.28:0/1465748472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759e20 msgr2=0x564daa762e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.560+0000 7f8671d22740 1 --2- 172.21.15.28:0/1465748472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759e20 0x564daa762e00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f8654000c00 tx=0x7f865400ae70 comp rx=0 tx=0).stop 2024-04-24T08:12:41.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.560+0000 7f8671d22740 1 -- 172.21.15.28:0/1465748472 shutdown_connections 2024-04-24T08:12:41.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.561+0000 7f8671d22740 1 --2- 172.21.15.28:0/1465748472 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x564daa773540 0x564daa775930 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.561+0000 7f8671d22740 1 --2- 172.21.15.28:0/1465748472 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759e20 0x564daa762e00 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.561+0000 7f8671d22740 1 --2- 172.21.15.28:0/1465748472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa75dc00 0x564daa763540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.561+0000 7f8671d22740 1 --2- 172.21.15.28:0/1465748472 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa759460 0x564daa765b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.561+0000 7f8671d22740 1 -- 172.21.15.28:0/1465748472 >> 172.21.15.28:0/1465748472 conn(0x564daa74b910 msgr2=0x564daa758ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:41.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.561+0000 7f8652000640 0 client.15741 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:41.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.561+0000 7f8671d22740 1 -- 172.21.15.28:0/1465748472 shutdown_connections 2024-04-24T08:12:41.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.561+0000 7f8671d22740 1 -- 172.21.15.28:0/1465748472 wait complete. 2024-04-24T08:12:41.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.561+0000 7f8671d22740 20 client.15741 trim_cache size 0 max 16384 2024-04-24T08:12:41.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.594+0000 7f8671d22740 1 Processor -- start 2024-04-24T08:12:41.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.594+0000 7f8671d22740 1 -- start start 2024-04-24T08:12:41.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.594+0000 7f8671d22740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa759460 0x564daa76f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.594+0000 7f8671d22740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa759e20 0x564daa75b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.594+0000 7f8671d22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa75dc00 0x564daa761e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.594+0000 7f8671d22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f86681b7280 con 0x564daa75dc00 2024-04-24T08:12:41.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.594+0000 7f8671d22740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f86681b4b50 con 0x564daa759460 2024-04-24T08:12:41.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.594+0000 7f8671d22740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f86681b29a0 con 0x564daa759e20 2024-04-24T08:12:41.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.595+0000 7f8667e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa759460 0x564daa76f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.595+0000 7f8667e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa759460 0x564daa76f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41592/0 (socket says 172.21.15.28:41592) 2024-04-24T08:12:41.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.595+0000 7f8667e00640 1 -- 172.21.15.28:0/280845977 learned_addr learned my addr 172.21.15.28:0/280845977 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:41.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.595+0000 7f866d200640 1 --2- 172.21.15.28:0/280845977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa75dc00 0x564daa761e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.595+0000 7f8667400640 1 --2- 172.21.15.28:0/280845977 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa759e20 0x564daa75b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.595+0000 7f8667e00640 1 -- 172.21.15.28:0/280845977 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa759e20 msgr2=0x564daa75b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:41.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.595+0000 7f8667e00640 1 --2- 172.21.15.28:0/280845977 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa759e20 0x564daa75b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.595+0000 7f8667e00640 1 -- 172.21.15.28:0/280845977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa75dc00 msgr2=0x564daa761e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:41.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.595+0000 7f8667e00640 1 --2- 172.21.15.28:0/280845977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa75dc00 0x564daa761e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.595+0000 7f8667e00640 1 -- 172.21.15.28:0/280845977 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564daa4917d0 con 0x564daa759460 2024-04-24T08:12:41.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.595+0000 7f866d200640 1 --2- 172.21.15.28:0/280845977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa75dc00 0x564daa761e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:41.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.595+0000 7f8667e00640 1 --2- 172.21.15.28:0/280845977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa759460 0x564daa76f0c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f865800e090 tx=0x7f865803b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=35ab8e0a4d4728e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.596+0000 7f865cc00640 1 -- 172.21.15.28:0/280845977 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658057070 con 0x564daa759460 2024-04-24T08:12:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.596+0000 7f865cc00640 1 -- 172.21.15.28:0/280845977 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8658039040 con 0x564daa759460 2024-04-24T08:12:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.596+0000 7f865cc00640 1 -- 172.21.15.28:0/280845977 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8658051520 con 0x564daa759460 2024-04-24T08:12:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.597+0000 7f8671d22740 1 -- 172.21.15.28:0/280845977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa759460 msgr2=0x564daa76f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.597+0000 7f8671d22740 1 --2- 172.21.15.28:0/280845977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa759460 0x564daa76f0c0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f865800e090 tx=0x7f865803b000 comp rx=0 tx=0).stop 2024-04-24T08:12:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.597+0000 7f8671d22740 1 -- 172.21.15.28:0/280845977 shutdown_connections 2024-04-24T08:12:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.597+0000 7f8671d22740 1 --2- 172.21.15.28:0/280845977 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa75dc00 0x564daa761e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.597+0000 7f8671d22740 1 --2- 172.21.15.28:0/280845977 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa759e20 0x564daa75b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.597+0000 7f8671d22740 1 --2- 172.21.15.28:0/280845977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa759460 0x564daa76f0c0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.597+0000 7f8671d22740 1 -- 172.21.15.28:0/280845977 >> 172.21.15.28:0/280845977 conn(0x564daa74b910 msgr2=0x564daa7616f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.597+0000 7f8671d22740 1 -- 172.21.15.28:0/280845977 shutdown_connections 2024-04-24T08:12:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.597+0000 7f8671d22740 1 -- 172.21.15.28:0/280845977 wait complete. 2024-04-24T08:12:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.597+0000 7f8671d22740 1 -- start start 2024-04-24T08:12:41.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.598+0000 7f8671d22740 1 Processor -- start 2024-04-24T08:12:41.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.598+0000 7f8671d22740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759460 0x564daa76b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.598+0000 7f8671d22740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa759e20 0x564daa76bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.598+0000 7f8671d22740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa75dc00 0x564daa77a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.598+0000 7f8671d22740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f86681b3010 con 0x564daa759460 2024-04-24T08:12:41.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.598+0000 7f8671d22740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f86681b4480 con 0x564daa75dc00 2024-04-24T08:12:41.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.598+0000 7f8671d22740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f86681b36e0 con 0x564daa759e20 2024-04-24T08:12:41.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.599+0000 7f8671d22740 10 client.0 _ll_register_callbacks cb 0x7f865400ada0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:41.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.599+0000 7f8667e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759460 0x564daa76b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.599+0000 7f8667e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759460 0x564daa76b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59060/0 (socket says 172.21.15.28:59060) 2024-04-24T08:12:41.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.599+0000 7f866d200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa75dc00 0x564daa77a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.599+0000 7f8667e00640 1 -- 172.21.15.28:0/820791164 learned_addr learned my addr 172.21.15.28:0/820791164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:41.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.599+0000 7f8667400640 1 --2- 172.21.15.28:0/820791164 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa759e20 0x564daa76bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.599+0000 7f866d200640 1 -- 172.21.15.28:0/820791164 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa759e20 msgr2=0x564daa76bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.599+0000 7f866d200640 1 --2- 172.21.15.28:0/820791164 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa759e20 0x564daa76bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.599+0000 7f866d200640 1 -- 172.21.15.28:0/820791164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759460 msgr2=0x564daa76b5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.599+0000 7f866d200640 1 --2- 172.21.15.28:0/820791164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759460 0x564daa76b5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.599+0000 7f866d200640 1 -- 172.21.15.28:0/820791164 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564daa77a8b0 con 0x564daa75dc00 2024-04-24T08:12:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.599+0000 7f866d200640 1 --2- 172.21.15.28:0/820791164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa75dc00 0x564daa77a240 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8660010c40 tx=0x7f8660009690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82152fba718e4abd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.600+0000 7f8652000640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:12:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.600+0000 7f8652000640 1 -- 172.21.15.28:0/820791164 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f86600052b0 con 0x564daa75dc00 2024-04-24T08:12:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.600+0000 7f8671d22740 1 -- 172.21.15.28:0/820791164 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8658039040 con 0x564daa75dc00 2024-04-24T08:12:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.600+0000 7f8652000640 1 -- 172.21.15.28:0/820791164 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f8660005450 con 0x564daa75dc00 2024-04-24T08:12:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.600+0000 7f8652000640 1 -- 172.21.15.28:0/820791164 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f86600035f0 con 0x564daa75dc00 2024-04-24T08:12:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.600+0000 7f8671d22740 10 client.25850 fetch_fsmap learned FSMap version 25 2024-04-24T08:12:41.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.600+0000 7f8671d22740 1 -- 172.21.15.28:0/820791164 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564daa7da1c0 con 0x564daa75dc00 2024-04-24T08:12:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.600+0000 7f8652000640 1 -- 172.21.15.28:0/820791164 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8658039040 con 0x564daa75dc00 2024-04-24T08:12:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.601+0000 7f8652000640 1 -- 172.21.15.28:0/820791164 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f866002f030 con 0x564daa75dc00 2024-04-24T08:12:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.601+0000 7f8671d22740 10 client.25850 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:12:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.601+0000 7f8671d22740 10 client.25850 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:12:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.601+0000 7f8671d22740 20 client.25850 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.601+0000 7f8671d22740 1 --2- 172.21.15.28:0/820791164 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x564daa7da7d0 0x564daa7dcbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.601+0000 7f8671d22740 4 client.25850 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:12:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.601+0000 7f8671d22740 1 -- 172.21.15.28:0/820791164 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564daa6bca60 con 0x564daa7da7d0 2024-04-24T08:12:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.601+0000 7f8667e00640 1 --2- 172.21.15.28:0/820791164 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x564daa7da7d0 0x564daa7dcbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.602+0000 7f8667e00640 1 --2- 172.21.15.28:0/820791164 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x564daa7da7d0 0x564daa7dcbf0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9cda2b1b7b21f4e2 server_cookie=c1e2e4ff26ca135 in_seq=0 out_seq=0 2024-04-24T08:12:41.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.602+0000 7f8652000640 10 client.25850 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.602+0000 7f8652000640 1 -- 172.21.15.28:0/820791164 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564daa6bca60 con 0x564daa7da7d0 2024-04-24T08:12:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.602+0000 7f8652000640 10 client.25850 handle_command_reply: tid=0 2024-04-24T08:12:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.602+0000 7f8671d22740 1 client.25850 shutdown 2024-04-24T08:12:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.602+0000 7f8671d22740 1 -- 172.21.15.28:0/820791164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa75dc00 msgr2=0x564daa77a240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.602+0000 7f8671d22740 1 --2- 172.21.15.28:0/820791164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa75dc00 0x564daa77a240 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f8660010c40 tx=0x7f8660009690 comp rx=0 tx=0).stop 2024-04-24T08:12:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.602+0000 7f8671d22740 1 -- 172.21.15.28:0/820791164 shutdown_connections 2024-04-24T08:12:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.602+0000 7f8671d22740 1 --2- 172.21.15.28:0/820791164 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x564daa7da7d0 0x564daa7dcbf0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.602+0000 7f8671d22740 1 --2- 172.21.15.28:0/820791164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564daa75dc00 0x564daa77a240 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.602+0000 7f8671d22740 1 --2- 172.21.15.28:0/820791164 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564daa759e20 0x564daa76bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.602+0000 7f8652000640 0 client.25850 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:41.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.602+0000 7f8671d22740 1 --2- 172.21.15.28:0/820791164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564daa759460 0x564daa76b5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.602+0000 7f8671d22740 1 -- 172.21.15.28:0/820791164 >> 172.21.15.28:0/820791164 conn(0x564daa74b910 msgr2=0x564daa758450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:41.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.602+0000 7f8671d22740 1 -- 172.21.15.28:0/820791164 shutdown_connections 2024-04-24T08:12:41.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.602+0000 7f8671d22740 1 -- 172.21.15.28:0/820791164 wait complete. 2024-04-24T08:12:41.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.603+0000 7f8671d22740 20 client.25850 trim_cache size 0 max 16384 2024-04-24T08:12:41.607 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:12:41.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.603+0000 7f866dc00640 1 -- 172.21.15.28:0/887335065 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f864c0a3e20 msgr2=0x7f864c0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.603+0000 7f866dc00640 1 --2- 172.21.15.28:0/887335065 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f864c0a3e20 0x7f864c0a62e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f865800bd40 tx=0x7f8658000f40 comp rx=0 tx=0).stop 2024-04-24T08:12:41.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.603+0000 7f866dc00640 1 -- 172.21.15.28:0/887335065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f866819f6d0 msgr2=0x7f86681a3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:41.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.603+0000 7f866dc00640 1 --2- 172.21.15.28:0/887335065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f866819f6d0 0x7f86681a3a50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f86540078d0 tx=0x7f8654007d90 comp rx=0 tx=0).stop 2024-04-24T08:12:41.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.603+0000 7f866dc00640 1 -- 172.21.15.28:0/887335065 shutdown_connections 2024-04-24T08:12:41.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.603+0000 7f866dc00640 1 --2- 172.21.15.28:0/887335065 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f864c0a3e20 0x7f864c0a62e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.603+0000 7f866dc00640 1 --2- 172.21.15.28:0/887335065 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f86681a3f90 0x7f86681a4440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.603+0000 7f866dc00640 1 --2- 172.21.15.28:0/887335065 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f866819f6d0 0x7f86681a3a50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.603+0000 7f866dc00640 1 --2- 172.21.15.28:0/887335065 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f866810de20 0x7f866819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.603+0000 7f866dc00640 1 -- 172.21.15.28:0/887335065 >> 172.21.15.28:0/887335065 conn(0x7f866806d3e0 msgr2=0x7f8668071cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.604+0000 7f866dc00640 1 -- 172.21.15.28:0/887335065 shutdown_connections 2024-04-24T08:12:41.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:41.604+0000 7f866dc00640 1 -- 172.21.15.28:0/887335065 wait complete. 2024-04-24T08:12:41.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:41 smithi194 ceph-mon[32836]: pgmap v73: 97 pgs: 97 active+clean; 8.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 278 KiB/s wr, 41 op/s 2024-04-24T08:12:41.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:41 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2400309582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:12:41.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:41 smithi117 ceph-mon[32438]: pgmap v73: 97 pgs: 97 active+clean; 8.0 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 180 KiB/s rd, 278 KiB/s wr, 41 op/s 2024-04-24T08:12:41.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:41 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2400309582' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:12:42.622 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:12:42.622 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:12:42.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:12:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.834+0000 7f2971e00640 1 -- 172.21.15.28:0/2336177106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f296c102ac0 msgr2=0x7f296c106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.834+0000 7f2971e00640 1 --2- 172.21.15.28:0/2336177106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f296c102ac0 0x7f296c106b00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2958009860 tx=0x7f295802a340 comp rx=0 tx=0).stop 2024-04-24T08:12:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.835+0000 7f2971e00640 1 -- 172.21.15.28:0/2336177106 shutdown_connections 2024-04-24T08:12:42.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.835+0000 7f2971e00640 1 --2- 172.21.15.28:0/2336177106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f296c1071e0 0x7f296c10edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.835+0000 7f2971e00640 1 --2- 172.21.15.28:0/2336177106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f296c102ac0 0x7f296c106b00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.835+0000 7f2971e00640 1 --2- 172.21.15.28:0/2336177106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f296c102110 0x7f296c1024f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.835+0000 7f2971e00640 1 -- 172.21.15.28:0/2336177106 >> 172.21.15.28:0/2336177106 conn(0x7f296c0fd9d0 msgr2=0x7f296c0ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:42.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.835+0000 7f2971e00640 1 -- 172.21.15.28:0/2336177106 shutdown_connections 2024-04-24T08:12:42.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.835+0000 7f2971e00640 1 -- 172.21.15.28:0/2336177106 wait complete. 2024-04-24T08:12:42.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.836+0000 7f2971e00640 1 Processor -- start 2024-04-24T08:12:42.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.836+0000 7f2971e00640 1 -- start start 2024-04-24T08:12:42.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.837+0000 7f2971e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f296c102110 0x7f296c19ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.837+0000 7f2971e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f296c1071e0 0x7f296c19b050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.837+0000 7f2971e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f296c19f400 0x7f296c19f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.837+0000 7f2971e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f296c112270 con 0x7f296c102110 2024-04-24T08:12:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.837+0000 7f2971e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f296c1120f0 con 0x7f296c1071e0 2024-04-24T08:12:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.837+0000 7f2971e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f296c1123f0 con 0x7f296c19f400 2024-04-24T08:12:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.837+0000 7f2970a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f296c102110 0x7f296c19ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.837+0000 7f2970a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f296c102110 0x7f296c19ab10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59072/0 (socket says 172.21.15.28:59072) 2024-04-24T08:12:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.837+0000 7f2970a00640 1 -- 172.21.15.28:0/800108715 learned_addr learned my addr 172.21.15.28:0/800108715 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.837+0000 7f2971400640 1 --2- 172.21.15.28:0/800108715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f296c19f400 0x7f296c19f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.837+0000 7f296be00640 1 --2- 172.21.15.28:0/800108715 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f296c1071e0 0x7f296c19b050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.838+0000 7f2970a00640 1 -- 172.21.15.28:0/800108715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f296c19f400 msgr2=0x7f296c19f8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.838+0000 7f2970a00640 1 --2- 172.21.15.28:0/800108715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f296c19f400 0x7f296c19f8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.838+0000 7f2970a00640 1 -- 172.21.15.28:0/800108715 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f296c1071e0 msgr2=0x7f296c19b050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.838+0000 7f2970a00640 1 --2- 172.21.15.28:0/800108715 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f296c1071e0 0x7f296c19b050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.838+0000 7f2970a00640 1 -- 172.21.15.28:0/800108715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f296c19fef0 con 0x7f296c102110 2024-04-24T08:12:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.838+0000 7f2970a00640 1 --2- 172.21.15.28:0/800108715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f296c102110 0x7f296c19ab10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f296000e9f0 tx=0x7f296000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.838+0000 7f2969600640 1 -- 172.21.15.28:0/800108715 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296000cd80 con 0x7f296c102110 2024-04-24T08:12:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.838+0000 7f2969600640 1 -- 172.21.15.28:0/800108715 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2960005500 con 0x7f296c102110 2024-04-24T08:12:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.838+0000 7f2971e00640 1 -- 172.21.15.28:0/800108715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f296c1a01e0 con 0x7f296c102110 2024-04-24T08:12:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.838+0000 7f2969600640 1 -- 172.21.15.28:0/800108715 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2960010690 con 0x7f296c102110 2024-04-24T08:12:42.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.838+0000 7f2971e00640 1 -- 172.21.15.28:0/800108715 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f296c1a7950 con 0x7f296c102110 2024-04-24T08:12:42.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.842+0000 7f29761e0740 1 Processor -- start 2024-04-24T08:12:42.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.842+0000 7f29761e0740 1 -- start start 2024-04-24T08:12:42.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.842+0000 7f2969600640 1 -- 172.21.15.28:0/800108715 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f29600046c0 con 0x7f296c102110 2024-04-24T08:12:42.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.842+0000 7f29761e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 0x55842c866650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.842+0000 7f29761e0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c866b90 0x55842c870d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.842+0000 7f29761e0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c871260 0x55842c873650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.842+0000 7f29761e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55842c7cbe70 con 0x55842c866270 2024-04-24T08:12:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.842+0000 7f29761e0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55842c772380 con 0x55842c866b90 2024-04-24T08:12:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.842+0000 7f29761e0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55842c7c7e30 con 0x55842c871260 2024-04-24T08:12:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.843+0000 7f2971400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 0x55842c866650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.843+0000 7f2971400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 0x55842c866650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59084/0 (socket says 172.21.15.28:59084) 2024-04-24T08:12:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.843+0000 7f2971400640 1 -- 172.21.15.28:0/2619260967 learned_addr learned my addr 172.21.15.28:0/2619260967 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.843+0000 7f296be00640 1 --2- 172.21.15.28:0/2619260967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c871260 0x55842c873650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.843+0000 7f2970a00640 1 --2- 172.21.15.28:0/2619260967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c866b90 0x55842c870d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.843+0000 7f2971400640 1 -- 172.21.15.28:0/2619260967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c871260 msgr2=0x55842c873650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.843+0000 7f2971400640 1 --2- 172.21.15.28:0/2619260967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c871260 0x55842c873650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.843+0000 7f2971400640 1 -- 172.21.15.28:0/2619260967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c866b90 msgr2=0x55842c870d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.843+0000 7f2971400640 1 --2- 172.21.15.28:0/2619260967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c866b90 0x55842c870d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.843+0000 7f2971400640 1 -- 172.21.15.28:0/2619260967 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55842c5a67e0 con 0x55842c866270 2024-04-24T08:12:42.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.843+0000 7f2971400640 1 --2- 172.21.15.28:0/2619260967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 0x55842c866650 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f29640099f0 tx=0x7f29640095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8396559df77c84c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.844+0000 7f295dc00640 1 -- 172.21.15.28:0/2619260967 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296401d070 con 0x55842c866270 2024-04-24T08:12:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.844+0000 7f295dc00640 1 -- 172.21.15.28:0/2619260967 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2964003e40 con 0x55842c866270 2024-04-24T08:12:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.844+0000 7f295dc00640 1 -- 172.21.15.28:0/2619260967 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296400db40 con 0x55842c866270 2024-04-24T08:12:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.844+0000 7f29761e0740 1 -- 172.21.15.28:0/2619260967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 msgr2=0x55842c866650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.844+0000 7f29761e0740 1 --2- 172.21.15.28:0/2619260967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 0x55842c866650 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f29640099f0 tx=0x7f29640095c0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.845+0000 7f2969600640 1 --2- 172.21.15.28:0/800108715 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29540a3d20 0x7f29540a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.845+0000 7f2969600640 1 -- 172.21.15.28:0/800108715 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f29600d0c90 con 0x7f296c102110 2024-04-24T08:12:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.845+0000 7f29761e0740 1 -- 172.21.15.28:0/2619260967 shutdown_connections 2024-04-24T08:12:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.845+0000 7f29761e0740 1 --2- 172.21.15.28:0/2619260967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c871260 0x55842c873650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.845+0000 7f29761e0740 1 --2- 172.21.15.28:0/2619260967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c866b90 0x55842c870d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.845+0000 7f29761e0740 1 --2- 172.21.15.28:0/2619260967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 0x55842c866650 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.845+0000 7f29761e0740 1 -- 172.21.15.28:0/2619260967 >> 172.21.15.28:0/2619260967 conn(0x55842c862590 msgr2=0x55842c862970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.845+0000 7f2971400640 1 --2- 172.21.15.28:0/800108715 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29540a3d20 0x7f29540a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.845+0000 7f29761e0740 1 -- 172.21.15.28:0/2619260967 shutdown_connections 2024-04-24T08:12:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.845+0000 7f29761e0740 1 -- 172.21.15.28:0/2619260967 wait complete. 2024-04-24T08:12:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.846+0000 7f29761e0740 1 -- start start 2024-04-24T08:12:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.846+0000 7f29761e0740 1 Processor -- start 2024-04-24T08:12:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.846+0000 7f2971400640 1 --2- 172.21.15.28:0/800108715 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29540a3d20 0x7f29540a61e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2964005c10 tx=0x7f2964009a20 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:42.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.846+0000 7f29761e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 0x55842c86e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.846+0000 7f29761e0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c866b90 0x55842c86e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.846+0000 7f29761e0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c871260 0x55842c86ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.846+0000 7f29761e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55842c874050 con 0x55842c866270 2024-04-24T08:12:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.846+0000 7f29761e0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55842c785660 con 0x55842c871260 2024-04-24T08:12:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.846+0000 7f29761e0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55842c8741d0 con 0x55842c866b90 2024-04-24T08:12:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.846+0000 7f29761e0740 10 client.0 _ll_register_callbacks cb 0x55842c773aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.847+0000 7f2970a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 0x55842c86e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.847+0000 7f2970a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 0x55842c86e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59086/0 (socket says 172.21.15.28:59086) 2024-04-24T08:12:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.847+0000 7f2970a00640 1 -- 172.21.15.28:0/946198545 learned_addr learned my addr 172.21.15.28:0/946198545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.847+0000 7f296be00640 1 --2- 172.21.15.28:0/946198545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c866b90 0x55842c86e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.847+0000 7f2971400640 1 --2- 172.21.15.28:0/946198545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c871260 0x55842c86ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.847+0000 7f2970a00640 1 -- 172.21.15.28:0/946198545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c866b90 msgr2=0x55842c86e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.847+0000 7f2970a00640 1 --2- 172.21.15.28:0/946198545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c866b90 0x55842c86e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.847+0000 7f2970a00640 1 -- 172.21.15.28:0/946198545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c871260 msgr2=0x55842c86ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.847+0000 7f2970a00640 1 --2- 172.21.15.28:0/946198545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c871260 0x55842c86ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.847+0000 7f2970a00640 1 -- 172.21.15.28:0/946198545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55842c606f70 con 0x55842c866270 2024-04-24T08:12:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.847+0000 7f2970a00640 1 --2- 172.21.15.28:0/946198545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 0x55842c86e1b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f296000c6a0 tx=0x7f2960004ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef5b55e06d5c7c0d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.847+0000 7f2952000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:12:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.847+0000 7f2952000640 1 -- 172.21.15.28:0/946198545 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f296009c510 con 0x55842c866270 2024-04-24T08:12:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.848+0000 7f2952000640 1 -- 172.21.15.28:0/946198545 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f29600a5390 con 0x55842c866270 2024-04-24T08:12:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.848+0000 7f29761e0740 1 -- 172.21.15.28:0/946198545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55842c7d9890 con 0x55842c866270 2024-04-24T08:12:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.848+0000 7f2952000640 1 -- 172.21.15.28:0/946198545 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f29600ce7e0 con 0x55842c866270 2024-04-24T08:12:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.848+0000 7f29761e0740 10 client.15759 fetch_fsmap learned FSMap version 25 2024-04-24T08:12:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.848+0000 7f29761e0740 1 -- 172.21.15.28:0/946198545 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55842c49e170 con 0x55842c866270 2024-04-24T08:12:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.848+0000 7f2952000640 1 -- 172.21.15.28:0/946198545 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55842c7d9890 con 0x55842c866270 2024-04-24T08:12:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.848+0000 7f2952000640 1 -- 172.21.15.28:0/946198545 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f296009c000 con 0x55842c866270 2024-04-24T08:12:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.848+0000 7f29761e0740 10 client.15759 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:12:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.848+0000 7f29761e0740 10 client.15759 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:12:42.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.848+0000 7f29761e0740 20 client.15759 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.848+0000 7f29761e0740 1 --2- 172.21.15.28:0/946198545 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55842c881590 0x55842c883a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.848+0000 7f29761e0740 4 client.15759 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.848+0000 7f29761e0740 1 -- 172.21.15.28:0/946198545 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55842c7d3700 con 0x55842c881590 2024-04-24T08:12:42.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.849+0000 7f296be00640 1 --2- 172.21.15.28:0/946198545 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55842c881590 0x55842c883a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.849+0000 7f296be00640 1 --2- 172.21.15.28:0/946198545 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55842c881590 0x55842c883a20 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9aae1bf6cea5e49b server_cookie=c04c4de2a1ecc0b6 in_seq=0 out_seq=0 2024-04-24T08:12:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.849+0000 7f2952000640 10 client.15759 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.850+0000 7f2952000640 1 -- 172.21.15.28:0/946198545 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55842c7d3700 con 0x55842c881590 2024-04-24T08:12:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.850+0000 7f2952000640 10 client.15759 handle_command_reply: tid=0 2024-04-24T08:12:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.850+0000 7f29761e0740 1 client.15759 shutdown 2024-04-24T08:12:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.850+0000 7f29761e0740 1 -- 172.21.15.28:0/946198545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 msgr2=0x55842c86e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.850+0000 7f29761e0740 1 --2- 172.21.15.28:0/946198545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 0x55842c86e1b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f296000c6a0 tx=0x7f2960004ae0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.850+0000 7f29761e0740 1 -- 172.21.15.28:0/946198545 shutdown_connections 2024-04-24T08:12:42.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.850+0000 7f29761e0740 1 --2- 172.21.15.28:0/946198545 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55842c881590 0x55842c883a20 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.850+0000 7f29761e0740 1 --2- 172.21.15.28:0/946198545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c871260 0x55842c86ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.850+0000 7f29761e0740 1 --2- 172.21.15.28:0/946198545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c866b90 0x55842c86e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.850+0000 7f29761e0740 1 --2- 172.21.15.28:0/946198545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 0x55842c86e1b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.850+0000 7f2952000640 0 client.15759 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.850+0000 7f29761e0740 1 -- 172.21.15.28:0/946198545 >> 172.21.15.28:0/946198545 conn(0x55842c862590 msgr2=0x55842c868390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.851+0000 7f29761e0740 1 -- 172.21.15.28:0/946198545 shutdown_connections 2024-04-24T08:12:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.851+0000 7f29761e0740 1 -- 172.21.15.28:0/946198545 wait complete. 2024-04-24T08:12:42.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.851+0000 7f29761e0740 20 client.15759 trim_cache size 0 max 16384 2024-04-24T08:12:42.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.885+0000 7f29761e0740 1 Processor -- start 2024-04-24T08:12:42.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.885+0000 7f29761e0740 1 -- start start 2024-04-24T08:12:42.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.885+0000 7f29761e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 0x55842c87cf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.885+0000 7f29761e0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c866b90 0x55842c86d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.885+0000 7f29761e0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c871260 0x55842c8892f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.885+0000 7f29761e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f296c1b27a0 con 0x55842c866270 2024-04-24T08:12:42.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.885+0000 7f29761e0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f296c1b0070 con 0x55842c871260 2024-04-24T08:12:42.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.885+0000 7f29761e0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f296c1adec0 con 0x55842c866b90 2024-04-24T08:12:42.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.886+0000 7f2970a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 0x55842c87cf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.886+0000 7f296be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c866b90 0x55842c86d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.886+0000 7f296be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c866b90 0x55842c86d980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:45022/0 (socket says 172.21.15.28:45022) 2024-04-24T08:12:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.886+0000 7f296be00640 1 -- 172.21.15.28:0/2552028081 learned_addr learned my addr 172.21.15.28:0/2552028081 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.886+0000 7f2971400640 1 --2- 172.21.15.28:0/2552028081 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c871260 0x55842c8892f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.886+0000 7f296be00640 1 -- 172.21.15.28:0/2552028081 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c871260 msgr2=0x55842c8892f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.886+0000 7f296be00640 1 --2- 172.21.15.28:0/2552028081 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c871260 0x55842c8892f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.886+0000 7f296be00640 1 -- 172.21.15.28:0/2552028081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 msgr2=0x55842c87cf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.886+0000 7f296be00640 1 --2- 172.21.15.28:0/2552028081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 0x55842c87cf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.886+0000 7f296be00640 1 -- 172.21.15.28:0/2552028081 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55842c8619a0 con 0x55842c866b90 2024-04-24T08:12:42.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.887+0000 7f296be00640 1 --2- 172.21.15.28:0/2552028081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c866b90 0x55842c86d980 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2958005e00 tx=0x7f29580056d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2ed1d367e7d60763 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.887+0000 7f295d200640 1 -- 172.21.15.28:0/2552028081 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f295804c070 con 0x55842c866b90 2024-04-24T08:12:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.887+0000 7f295d200640 1 -- 172.21.15.28:0/2552028081 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2958030dc0 con 0x55842c866b90 2024-04-24T08:12:42.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.887+0000 7f295d200640 1 -- 172.21.15.28:0/2552028081 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2958037040 con 0x55842c866b90 2024-04-24T08:12:42.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.888+0000 7f29761e0740 1 -- 172.21.15.28:0/2552028081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c866b90 msgr2=0x55842c86d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:42.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.888+0000 7f29761e0740 1 --2- 172.21.15.28:0/2552028081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c866b90 0x55842c86d980 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2958005e00 tx=0x7f29580056d0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.888+0000 7f29761e0740 1 -- 172.21.15.28:0/2552028081 shutdown_connections 2024-04-24T08:12:42.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.888+0000 7f29761e0740 1 --2- 172.21.15.28:0/2552028081 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c871260 0x55842c8892f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.888+0000 7f29761e0740 1 --2- 172.21.15.28:0/2552028081 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c866b90 0x55842c86d980 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.888+0000 7f29761e0740 1 --2- 172.21.15.28:0/2552028081 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c866270 0x55842c87cf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.888+0000 7f29761e0740 1 -- 172.21.15.28:0/2552028081 >> 172.21.15.28:0/2552028081 conn(0x55842c862590 msgr2=0x55842c86ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:42.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.888+0000 7f29761e0740 1 -- 172.21.15.28:0/2552028081 shutdown_connections 2024-04-24T08:12:42.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.888+0000 7f29761e0740 1 -- 172.21.15.28:0/2552028081 wait complete. 2024-04-24T08:12:42.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.889+0000 7f29761e0740 1 -- start start 2024-04-24T08:12:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.889+0000 7f29761e0740 1 Processor -- start 2024-04-24T08:12:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.889+0000 7f29761e0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c866270 0x55842c879420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.889+0000 7f29761e0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c866b90 0x55842c879960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.889+0000 7f29761e0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c871260 0x55842c876430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.889+0000 7f29761e0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f296c1ae530 con 0x55842c871260 2024-04-24T08:12:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.889+0000 7f29761e0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f296c1af9a0 con 0x55842c866270 2024-04-24T08:12:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.889+0000 7f29761e0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f296c1aec00 con 0x55842c866b90 2024-04-24T08:12:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.889+0000 7f29761e0740 10 client.0 _ll_register_callbacks cb 0x7f29600d4430 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.889+0000 7f2970a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c866270 0x55842c879420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.889+0000 7f2971400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c871260 0x55842c876430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.889+0000 7f2970a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c866270 0x55842c879420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41664/0 (socket says 172.21.15.28:41664) 2024-04-24T08:12:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.889+0000 7f2970a00640 1 -- 172.21.15.28:0/1849295720 learned_addr learned my addr 172.21.15.28:0/1849295720 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:42.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.890+0000 7f296be00640 1 --2- 172.21.15.28:0/1849295720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c866b90 0x55842c879960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.890+0000 7f2970a00640 1 -- 172.21.15.28:0/1849295720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c866b90 msgr2=0x55842c879960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:42.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.890+0000 7f2970a00640 1 --2- 172.21.15.28:0/1849295720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c866b90 0x55842c879960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.890+0000 7f2970a00640 1 -- 172.21.15.28:0/1849295720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c871260 msgr2=0x55842c876430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:42.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.890+0000 7f2970a00640 1 --2- 172.21.15.28:0/1849295720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c871260 0x55842c876430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.890+0000 7f2970a00640 1 -- 172.21.15.28:0/1849295720 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55842c8669c0 con 0x55842c866270 2024-04-24T08:12:42.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.890+0000 7f2970a00640 1 --2- 172.21.15.28:0/1849295720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c866270 0x55842c879420 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2960099580 tx=0x7f296009c7b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d996d2a91368cf88 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:42.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.890+0000 7f2952000640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:12:42.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.890+0000 7f2952000640 1 -- 172.21.15.28:0/1849295720 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29600ac060 con 0x55842c866270 2024-04-24T08:12:42.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.890+0000 7f29761e0740 1 -- 172.21.15.28:0/1849295720 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2958030dc0 con 0x55842c866270 2024-04-24T08:12:42.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.890+0000 7f2952000640 1 -- 172.21.15.28:0/1849295720 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f29600ce5e0 con 0x55842c866270 2024-04-24T08:12:42.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.891+0000 7f2952000640 1 -- 172.21.15.28:0/1849295720 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2960098e50 con 0x55842c866270 2024-04-24T08:12:42.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.891+0000 7f29761e0740 10 client.25868 fetch_fsmap learned FSMap version 25 2024-04-24T08:12:42.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.891+0000 7f29761e0740 1 -- 172.21.15.28:0/1849295720 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55842c8e8550 con 0x55842c866270 2024-04-24T08:12:42.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.891+0000 7f2952000640 1 -- 172.21.15.28:0/1849295720 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2958030dc0 con 0x55842c866270 2024-04-24T08:12:42.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.891+0000 7f2952000640 1 -- 172.21.15.28:0/1849295720 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f29600cebd0 con 0x55842c866270 2024-04-24T08:12:42.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.891+0000 7f29761e0740 10 client.25868 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:12:42.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.891+0000 7f29761e0740 10 client.25868 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:12:42.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.891+0000 7f29761e0740 20 client.25868 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:42.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.891+0000 7f29761e0740 1 --2- 172.21.15.28:0/1849295720 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55842c8e8c70 0x55842c8eb0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:42.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.891+0000 7f29761e0740 4 client.25868 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:12:42.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.891+0000 7f29761e0740 1 -- 172.21.15.28:0/1849295720 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55842c74f980 con 0x55842c8e8c70 2024-04-24T08:12:42.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.892+0000 7f296be00640 1 --2- 172.21.15.28:0/1849295720 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55842c8e8c70 0x55842c8eb0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:42.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.892+0000 7f296be00640 1 --2- 172.21.15.28:0/1849295720 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55842c8e8c70 0x55842c8eb0d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46714b885667bc2a server_cookie=16cf3e6d24da6d34 in_seq=0 out_seq=0 2024-04-24T08:12:42.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.892+0000 7f2952000640 10 client.25868 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:42.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.893+0000 7f2952000640 1 -- 172.21.15.28:0/1849295720 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55842c74f980 con 0x55842c8e8c70 2024-04-24T08:12:42.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.893+0000 7f2952000640 10 client.25868 handle_command_reply: tid=0 2024-04-24T08:12:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.893+0000 7f29761e0740 1 client.25868 shutdown 2024-04-24T08:12:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.893+0000 7f29761e0740 1 -- 172.21.15.28:0/1849295720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c866270 msgr2=0x55842c879420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.893+0000 7f29761e0740 1 --2- 172.21.15.28:0/1849295720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c866270 0x55842c879420 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f2960099580 tx=0x7f296009c7b0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.894+0000 7f29761e0740 1 -- 172.21.15.28:0/1849295720 shutdown_connections 2024-04-24T08:12:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.894+0000 7f29761e0740 1 --2- 172.21.15.28:0/1849295720 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55842c8e8c70 0x55842c8eb0d0 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.894+0000 7f29761e0740 1 --2- 172.21.15.28:0/1849295720 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55842c871260 0x55842c876430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.894+0000 7f29761e0740 1 --2- 172.21.15.28:0/1849295720 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55842c866b90 0x55842c879960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.894+0000 7f29761e0740 1 --2- 172.21.15.28:0/1849295720 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55842c866270 0x55842c879420 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.894+0000 7f2952000640 0 client.25868 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.894+0000 7f29761e0740 1 -- 172.21.15.28:0/1849295720 >> 172.21.15.28:0/1849295720 conn(0x55842c862590 msgr2=0x55842c867a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.894+0000 7f29761e0740 1 -- 172.21.15.28:0/1849295720 shutdown_connections 2024-04-24T08:12:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.894+0000 7f29761e0740 1 -- 172.21.15.28:0/1849295720 wait complete. 2024-04-24T08:12:42.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.894+0000 7f29761e0740 20 client.25868 trim_cache size 0 max 16384 2024-04-24T08:12:42.898 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:12:42.898 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-24T08:12:42.898 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "eb6d4250-04da-4a87-8805-5c3b00715606", 2024-04-24T08:12:42.898 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-24T08:12:42.898 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:12:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.895+0000 7f2971e00640 1 -- 172.21.15.28:0/800108715 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29540a3d20 msgr2=0x7f29540a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.895+0000 7f2971e00640 1 --2- 172.21.15.28:0/800108715 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29540a3d20 0x7f29540a61e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f2964005c10 tx=0x7f2964009a20 comp rx=0 tx=0).stop 2024-04-24T08:12:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.895+0000 7f2971e00640 1 -- 172.21.15.28:0/800108715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f296c102110 msgr2=0x7f296c19ab10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.895+0000 7f2971e00640 1 --2- 172.21.15.28:0/800108715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f296c102110 0x7f296c19ab10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f296000e9f0 tx=0x7f296000edf0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.895+0000 7f2971e00640 1 -- 172.21.15.28:0/800108715 shutdown_connections 2024-04-24T08:12:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.895+0000 7f2971e00640 1 --2- 172.21.15.28:0/800108715 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29540a3d20 0x7f29540a61e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.895+0000 7f2971e00640 1 --2- 172.21.15.28:0/800108715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f296c19f400 0x7f296c19f8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.895+0000 7f2971e00640 1 --2- 172.21.15.28:0/800108715 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f296c1071e0 0x7f296c19b050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.895+0000 7f2971e00640 1 --2- 172.21.15.28:0/800108715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f296c102110 0x7f296c19ab10 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.895+0000 7f2971e00640 1 -- 172.21.15.28:0/800108715 >> 172.21.15.28:0/800108715 conn(0x7f296c0fd9d0 msgr2=0x7f296c0ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.896+0000 7f2971e00640 1 -- 172.21.15.28:0/800108715 shutdown_connections 2024-04-24T08:12:42.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:42.896+0000 7f2971e00640 1 -- 172.21.15.28:0/800108715 wait complete. 2024-04-24T08:12:42.923 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:12:42.923 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:12:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.107+0000 7f8900c00640 1 -- 172.21.15.28:0/1438693278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88fc102bf0 msgr2=0x7f88fc106c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.107+0000 7f8900c00640 1 --2- 172.21.15.28:0/1438693278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88fc102bf0 0x7f88fc106c30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f88e4009900 tx=0x7f88e402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.107+0000 7f8900c00640 1 -- 172.21.15.28:0/1438693278 shutdown_connections 2024-04-24T08:12:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.107+0000 7f8900c00640 1 --2- 172.21.15.28:0/1438693278 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88fc102bf0 0x7f88fc106c30 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.107+0000 7f8900c00640 1 --2- 172.21.15.28:0/1438693278 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88fc107310 0x7f88fc10eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.107+0000 7f8900c00640 1 --2- 172.21.15.28:0/1438693278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88fc102240 0x7f88fc102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.107+0000 7f8900c00640 1 -- 172.21.15.28:0/1438693278 >> 172.21.15.28:0/1438693278 conn(0x7f88fc079450 msgr2=0x7f88fc079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:43.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.107+0000 7f8900c00640 1 -- 172.21.15.28:0/1438693278 shutdown_connections 2024-04-24T08:12:43.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.107+0000 7f8900c00640 1 -- 172.21.15.28:0/1438693278 wait complete. 2024-04-24T08:12:43.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.108+0000 7f8900c00640 1 Processor -- start 2024-04-24T08:12:43.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.109+0000 7f8900c00640 1 -- start start 2024-04-24T08:12:43.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.109+0000 7f8900c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88fc102240 0x7f88fc1a0770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.109+0000 7f8900c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88fc107310 0x7f88fc1a0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.109+0000 7f8900c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88fc19a920 0x7f88fc19add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.109+0000 7f8900c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc1123d0 con 0x7f88fc102240 2024-04-24T08:12:43.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.109+0000 7f8900c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc112250 con 0x7f88fc107310 2024-04-24T08:12:43.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.109+0000 7f8900c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc112550 con 0x7f88fc19a920 2024-04-24T08:12:43.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.109+0000 7f88f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88fc102240 0x7f88fc1a0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.109+0000 7f88f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88fc102240 0x7f88fc1a0770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59112/0 (socket says 172.21.15.28:59112) 2024-04-24T08:12:43.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.109+0000 7f88f9e00640 1 -- 172.21.15.28:0/3746879325 learned_addr learned my addr 172.21.15.28:0/3746879325 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:43.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.109+0000 7f88fa800640 1 --2- 172.21.15.28:0/3746879325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88fc19a920 0x7f88fc19add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.109+0000 7f88f3e00640 1 --2- 172.21.15.28:0/3746879325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88fc107310 0x7f88fc1a0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.110+0000 7f88f9e00640 1 -- 172.21.15.28:0/3746879325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88fc19a920 msgr2=0x7f88fc19add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:43.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.110+0000 7f88f9e00640 1 --2- 172.21.15.28:0/3746879325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88fc19a920 0x7f88fc19add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.110+0000 7f88f9e00640 1 -- 172.21.15.28:0/3746879325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88fc107310 msgr2=0x7f88fc1a0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:43.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.110+0000 7f88f9e00640 1 --2- 172.21.15.28:0/3746879325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88fc107310 0x7f88fc1a0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.110+0000 7f88f9e00640 1 -- 172.21.15.28:0/3746879325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88fc19b440 con 0x7f88fc102240 2024-04-24T08:12:43.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.110+0000 7f88f9e00640 1 --2- 172.21.15.28:0/3746879325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88fc102240 0x7f88fc1a0770 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00e9f0 tx=0x7f88ec00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:43.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.110+0000 7f88f2a00640 1 -- 172.21.15.28:0/3746879325 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec00cd80 con 0x7f88fc102240 2024-04-24T08:12:43.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.110+0000 7f88f2a00640 1 -- 172.21.15.28:0/3746879325 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88ec010910 con 0x7f88fc102240 2024-04-24T08:12:43.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.110+0000 7f8900c00640 1 -- 172.21.15.28:0/3746879325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88fc19b730 con 0x7f88fc102240 2024-04-24T08:12:43.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.110+0000 7f88f2a00640 1 -- 172.21.15.28:0/3746879325 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec018990 con 0x7f88fc102240 2024-04-24T08:12:43.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.110+0000 7f8900c00640 1 -- 172.21.15.28:0/3746879325 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88fc1a79d0 con 0x7f88fc102240 2024-04-24T08:12:43.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.114+0000 7f89039e4740 1 Processor -- start 2024-04-24T08:12:43.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.114+0000 7f88f2a00640 1 -- 172.21.15.28:0/3746879325 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f88ec010ac0 con 0x7f88fc102240 2024-04-24T08:12:43.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.114+0000 7f89039e4740 1 -- start start 2024-04-24T08:12:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.114+0000 7f89039e4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfad9860 0x558bcfad9c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.114+0000 7f89039e4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 0x558bcfae4310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.114+0000 7f89039e4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfae4850 0x558bcfae6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.114+0000 7f89039e4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558bcfa3f1f0 con 0x558bcfada180 2024-04-24T08:12:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.114+0000 7f89039e4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558bcf9e5350 con 0x558bcfae4850 2024-04-24T08:12:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.114+0000 7f89039e4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558bcfa3b1b0 con 0x558bcfad9860 2024-04-24T08:12:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.115+0000 7f88f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 0x558bcfae4310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.115+0000 7f88f3e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfae4850 0x558bcfae6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.115+0000 7f88f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 0x558bcfae4310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59114/0 (socket says 172.21.15.28:59114) 2024-04-24T08:12:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.115+0000 7f88f9e00640 1 -- 172.21.15.28:0/114692344 learned_addr learned my addr 172.21.15.28:0/114692344 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.115+0000 7f88fa800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfad9860 0x558bcfad9c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.115+0000 7f88f3e00640 1 -- 172.21.15.28:0/114692344 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfad9860 msgr2=0x558bcfad9c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:43.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.115+0000 7f88f3e00640 1 --2- 172.21.15.28:0/114692344 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfad9860 0x558bcfad9c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.115+0000 7f88f3e00640 1 -- 172.21.15.28:0/114692344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 msgr2=0x558bcfae4310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.115+0000 7f88f3e00640 1 --2- 172.21.15.28:0/114692344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 0x558bcfae4310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.115+0000 7f88f3e00640 1 -- 172.21.15.28:0/114692344 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bcfa39f50 con 0x558bcfae4850 2024-04-24T08:12:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.115+0000 7f88f3e00640 1 --2- 172.21.15.28:0/114692344 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfae4850 0x558bcfae6c40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f88e402a910 tx=0x7f88e4005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f75939fda87b8d47 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.116+0000 7f88f2a00640 1 --2- 172.21.15.28:0/3746879325 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88e00a3da0 0x7f88e00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.116+0000 7f88eb000640 1 -- 172.21.15.28:0/114692344 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e4035040 con 0x558bcfae4850 2024-04-24T08:12:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.116+0000 7f88eb000640 1 -- 172.21.15.28:0/114692344 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88e4005520 con 0x558bcfae4850 2024-04-24T08:12:43.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.116+0000 7f88eb000640 1 -- 172.21.15.28:0/114692344 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e40035f0 con 0x558bcfae4850 2024-04-24T08:12:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.116+0000 7f88f2a00640 1 -- 172.21.15.28:0/3746879325 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f88ec0d0b00 con 0x7f88fc102240 2024-04-24T08:12:43.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.116+0000 7f88fa800640 1 --2- 172.21.15.28:0/3746879325 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88e00a3da0 0x7f88e00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.117+0000 7f89039e4740 1 -- 172.21.15.28:0/114692344 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfae4850 msgr2=0x558bcfae6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.117+0000 7f88fa800640 1 --2- 172.21.15.28:0/3746879325 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88e00a3da0 0x7f88e00a6260 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f88f4005fd0 tx=0x7f88f4009500 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.117+0000 7f89039e4740 1 --2- 172.21.15.28:0/114692344 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfae4850 0x558bcfae6c40 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f88e402a910 tx=0x7f88e4005de0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.117+0000 7f89039e4740 1 -- 172.21.15.28:0/114692344 shutdown_connections 2024-04-24T08:12:43.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.117+0000 7f89039e4740 1 --2- 172.21.15.28:0/114692344 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 0x558bcfae4310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.117+0000 7f89039e4740 1 --2- 172.21.15.28:0/114692344 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfae4850 0x558bcfae6c40 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.117+0000 7f89039e4740 1 --2- 172.21.15.28:0/114692344 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfad9860 0x558bcfad9c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.117+0000 7f89039e4740 1 -- 172.21.15.28:0/114692344 >> 172.21.15.28:0/114692344 conn(0x558bcfad5b80 msgr2=0x558bcfad5f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.117+0000 7f89039e4740 1 -- 172.21.15.28:0/114692344 shutdown_connections 2024-04-24T08:12:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.117+0000 7f89039e4740 1 -- 172.21.15.28:0/114692344 wait complete. 2024-04-24T08:12:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.117+0000 7f89039e4740 1 -- start start 2024-04-24T08:12:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.118+0000 7f89039e4740 1 Processor -- start 2024-04-24T08:12:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.118+0000 7f89039e4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfad9860 0x558bcfae16f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.118+0000 7f89039e4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 0x558bcfae1c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.118+0000 7f89039e4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfae4850 0x558bcfae2170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.118+0000 7f89039e4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558bcfae75e0 con 0x558bcfada180 2024-04-24T08:12:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.118+0000 7f89039e4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558bcf9f8690 con 0x558bcfad9860 2024-04-24T08:12:43.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.118+0000 7f89039e4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558bcfae7760 con 0x558bcfae4850 2024-04-24T08:12:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.118+0000 7f89039e4740 10 client.0 _ll_register_callbacks cb 0x558bcfa34650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.118+0000 7f88f9e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfad9860 0x558bcfae16f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.118+0000 7f88f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 0x558bcfae1c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.118+0000 7f88f3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 0x558bcfae1c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59124/0 (socket says 172.21.15.28:59124) 2024-04-24T08:12:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.118+0000 7f88f3e00640 1 -- 172.21.15.28:0/2294806962 learned_addr learned my addr 172.21.15.28:0/2294806962 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.118+0000 7f88f9e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfad9860 0x558bcfae16f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41692/0 (socket says 172.21.15.28:41692) 2024-04-24T08:12:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.119+0000 7f88fa800640 1 --2- 172.21.15.28:0/2294806962 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfae4850 0x558bcfae2170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.119+0000 7f88f3e00640 1 -- 172.21.15.28:0/2294806962 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfae4850 msgr2=0x558bcfae2170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.119+0000 7f88f3e00640 1 --2- 172.21.15.28:0/2294806962 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfae4850 0x558bcfae2170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.119+0000 7f88f3e00640 1 -- 172.21.15.28:0/2294806962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfad9860 msgr2=0x558bcfae16f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:43.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.119+0000 7f88f3e00640 1 --2- 172.21.15.28:0/2294806962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfad9860 0x558bcfae16f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.119+0000 7f88f3e00640 1 -- 172.21.15.28:0/2294806962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bcf9f7920 con 0x558bcfada180 2024-04-24T08:12:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.119+0000 7f88f9e00640 1 --2- 172.21.15.28:0/2294806962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfad9860 0x558bcfae16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:12:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.119+0000 7f88fa800640 1 --2- 172.21.15.28:0/2294806962 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfae4850 0x558bcfae2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:12:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.119+0000 7f88f3e00640 1 --2- 172.21.15.28:0/2294806962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 0x558bcfae1c30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f88e40048f0 tx=0x7f88e4004920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed5a115b32ee5ae7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.119+0000 7f88df400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:12:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.119+0000 7f88df400640 1 -- 172.21.15.28:0/2294806962 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88e4052060 con 0x558bcfada180 2024-04-24T08:12:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.120+0000 7f89039e4740 1 -- 172.21.15.28:0/2294806962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88e4005520 con 0x558bcfada180 2024-04-24T08:12:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.120+0000 7f88df400640 1 -- 172.21.15.28:0/2294806962 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f88e4004390 con 0x558bcfada180 2024-04-24T08:12:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.120+0000 7f88df400640 1 -- 172.21.15.28:0/2294806962 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f88e4035070 con 0x558bcfada180 2024-04-24T08:12:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.120+0000 7f89039e4740 10 client.15777 fetch_fsmap learned FSMap version 25 2024-04-24T08:12:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.120+0000 7f89039e4740 1 -- 172.21.15.28:0/2294806962 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558bcf81b7d0 con 0x558bcfada180 2024-04-24T08:12:43.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.120+0000 7f88df400640 1 -- 172.21.15.28:0/2294806962 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88e4005520 con 0x558bcfada180 2024-04-24T08:12:43.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.120+0000 7f88df400640 1 -- 172.21.15.28:0/2294806962 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f88e4003950 con 0x558bcfada180 2024-04-24T08:12:43.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.121+0000 7f89039e4740 10 client.15777 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:12:43.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.121+0000 7f89039e4740 10 client.15777 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:12:43.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.121+0000 7f89039e4740 20 client.15777 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:43.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.121+0000 7f89039e4740 1 --2- 172.21.15.28:0/2294806962 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x558bcfaf4980 0x558bcfaf6d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.121+0000 7f89039e4740 4 client.15777 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:12:43.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.121+0000 7f89039e4740 1 -- 172.21.15.28:0/2294806962 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558bcfa4cc10 con 0x558bcfaf4980 2024-04-24T08:12:43.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.121+0000 7f88f9e00640 1 --2- 172.21.15.28:0/2294806962 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x558bcfaf4980 0x558bcfaf6d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.122+0000 7f88df400640 10 client.15777 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:43.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.122+0000 7f88f9e00640 1 --2- 172.21.15.28:0/2294806962 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x558bcfaf4980 0x558bcfaf6d70 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cf0baf08d40c299 server_cookie=57bfd2888632e7c in_seq=0 out_seq=0 2024-04-24T08:12:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.122+0000 7f88df400640 1 -- 172.21.15.28:0/2294806962 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558bcfa4cc10 con 0x558bcfaf4980 2024-04-24T08:12:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.122+0000 7f88df400640 10 client.15777 handle_command_reply: tid=0 2024-04-24T08:12:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.122+0000 7f89039e4740 1 client.15777 shutdown 2024-04-24T08:12:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.123+0000 7f89039e4740 1 -- 172.21.15.28:0/2294806962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 msgr2=0x558bcfae1c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.123+0000 7f89039e4740 1 --2- 172.21.15.28:0/2294806962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 0x558bcfae1c30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f88e40048f0 tx=0x7f88e4004920 comp rx=0 tx=0).stop 2024-04-24T08:12:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.123+0000 7f89039e4740 1 -- 172.21.15.28:0/2294806962 shutdown_connections 2024-04-24T08:12:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.123+0000 7f89039e4740 1 --2- 172.21.15.28:0/2294806962 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x558bcfaf4980 0x558bcfaf6d70 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.123+0000 7f89039e4740 1 --2- 172.21.15.28:0/2294806962 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 0x558bcfae1c30 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.123+0000 7f89039e4740 1 --2- 172.21.15.28:0/2294806962 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfae4850 0x558bcfae2170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.123+0000 7f88df400640 0 client.15777 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:43.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.123+0000 7f89039e4740 1 --2- 172.21.15.28:0/2294806962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfad9860 0x558bcfae16f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.123+0000 7f89039e4740 1 -- 172.21.15.28:0/2294806962 >> 172.21.15.28:0/2294806962 conn(0x558bcfad5b80 msgr2=0x558bcfadb880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.123+0000 7f89039e4740 1 -- 172.21.15.28:0/2294806962 shutdown_connections 2024-04-24T08:12:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.123+0000 7f89039e4740 1 -- 172.21.15.28:0/2294806962 wait complete. 2024-04-24T08:12:43.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.123+0000 7f89039e4740 20 client.15777 trim_cache size 0 max 16384 2024-04-24T08:12:43.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.141+0000 7f89039e4740 1 Processor -- start 2024-04-24T08:12:43.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.142+0000 7f89039e4740 1 -- start start 2024-04-24T08:12:43.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.142+0000 7f89039e4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfad9860 0x558bcfaf0500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.142+0000 7f89039e4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfada180 0x558bcfae06d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.142+0000 7f89039e4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfae4850 0x558bcfafc3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.142+0000 7f89039e4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc1b28e0 con 0x558bcfad9860 2024-04-24T08:12:43.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.142+0000 7f89039e4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc1b01b0 con 0x558bcfae4850 2024-04-24T08:12:43.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.142+0000 7f89039e4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc1ae000 con 0x558bcfada180 2024-04-24T08:12:43.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.142+0000 7f88f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfad9860 0x558bcfaf0500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.142+0000 7f88f9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfad9860 0x558bcfaf0500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:59136/0 (socket says 172.21.15.28:59136) 2024-04-24T08:12:43.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.142+0000 7f88f9e00640 1 -- 172.21.15.28:0/1209753858 learned_addr learned my addr 172.21.15.28:0/1209753858 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:43.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.142+0000 7f88fa800640 1 --2- 172.21.15.28:0/1209753858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfae4850 0x558bcfafc3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.143+0000 7f88f3e00640 1 --2- 172.21.15.28:0/1209753858 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfada180 0x558bcfae06d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.143+0000 7f88f9e00640 1 -- 172.21.15.28:0/1209753858 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfada180 msgr2=0x558bcfae06d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:43.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.143+0000 7f88f9e00640 1 --2- 172.21.15.28:0/1209753858 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfada180 0x558bcfae06d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.143+0000 7f88f9e00640 1 -- 172.21.15.28:0/1209753858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfae4850 msgr2=0x558bcfafc3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:43.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.143+0000 7f88f9e00640 1 --2- 172.21.15.28:0/1209753858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfae4850 0x558bcfafc3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.143+0000 7f88f9e00640 1 -- 172.21.15.28:0/1209753858 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558bcf7513f0 con 0x558bcfad9860 2024-04-24T08:12:43.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.143+0000 7f88f9e00640 1 --2- 172.21.15.28:0/1209753858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfad9860 0x558bcfaf0500 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f88ec09ca60 tx=0x7f88ec0d2f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65509ede0a9b3657 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:43.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.143+0000 7f88ea600640 1 -- 172.21.15.28:0/1209753858 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec0d8040 con 0x558bcfad9860 2024-04-24T08:12:43.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.143+0000 7f88ea600640 1 -- 172.21.15.28:0/1209753858 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88ec09cad0 con 0x558bcfad9860 2024-04-24T08:12:43.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.143+0000 7f88ea600640 1 -- 172.21.15.28:0/1209753858 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec0a5200 con 0x558bcfad9860 2024-04-24T08:12:43.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.144+0000 7f89039e4740 1 -- 172.21.15.28:0/1209753858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfad9860 msgr2=0x558bcfaf0500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:43.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.144+0000 7f89039e4740 1 --2- 172.21.15.28:0/1209753858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfad9860 0x558bcfaf0500 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f88ec09ca60 tx=0x7f88ec0d2f50 comp rx=0 tx=0).stop 2024-04-24T08:12:43.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.145+0000 7f89039e4740 1 -- 172.21.15.28:0/1209753858 shutdown_connections 2024-04-24T08:12:43.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.145+0000 7f89039e4740 1 --2- 172.21.15.28:0/1209753858 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfae4850 0x558bcfafc3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.145+0000 7f89039e4740 1 --2- 172.21.15.28:0/1209753858 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfada180 0x558bcfae06d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.145+0000 7f89039e4740 1 --2- 172.21.15.28:0/1209753858 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfad9860 0x558bcfaf0500 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.145+0000 7f89039e4740 1 -- 172.21.15.28:0/1209753858 >> 172.21.15.28:0/1209753858 conn(0x558bcfad5b80 msgr2=0x558bcfadffc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:43.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.145+0000 7f89039e4740 1 -- 172.21.15.28:0/1209753858 shutdown_connections 2024-04-24T08:12:43.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.145+0000 7f89039e4740 1 -- 172.21.15.28:0/1209753858 wait complete. 2024-04-24T08:12:43.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.145+0000 7f89039e4740 1 -- start start 2024-04-24T08:12:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.145+0000 7f89039e4740 1 Processor -- start 2024-04-24T08:12:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.146+0000 7f89039e4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfad9860 0x558bcfaeca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.146+0000 7f89039e4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 0x558bcfaecf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.146+0000 7f89039e4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfae4850 0x558bcfaddb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.146+0000 7f89039e4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc1ae670 con 0x558bcfada180 2024-04-24T08:12:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.146+0000 7f89039e4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc1afae0 con 0x558bcfad9860 2024-04-24T08:12:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.146+0000 7f89039e4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f88fc1aed40 con 0x558bcfae4850 2024-04-24T08:12:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.146+0000 7f89039e4740 10 client.0 _ll_register_callbacks cb 0x7f88ec0d4480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:12:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.146+0000 7f88f9e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfad9860 0x558bcfaeca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.146+0000 7f88f9e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfad9860 0x558bcfaeca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41700/0 (socket says 172.21.15.28:41700) 2024-04-24T08:12:43.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.146+0000 7f88f9e00640 1 -- 172.21.15.28:0/735741679 learned_addr learned my addr 172.21.15.28:0/735741679 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:12:43.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.146+0000 7f88f3e00640 1 --2- 172.21.15.28:0/735741679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 0x558bcfaecf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.147+0000 7f88fa800640 1 --2- 172.21.15.28:0/735741679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfae4850 0x558bcfaddb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.147+0000 7f88f9e00640 1 -- 172.21.15.28:0/735741679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfae4850 msgr2=0x558bcfaddb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:43.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.147+0000 7f88f9e00640 1 --2- 172.21.15.28:0/735741679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfae4850 0x558bcfaddb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.147+0000 7f88f9e00640 1 -- 172.21.15.28:0/735741679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 msgr2=0x558bcfaecf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:12:43.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.147+0000 7f88f9e00640 1 --2- 172.21.15.28:0/735741679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 0x558bcfaecf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.147+0000 7f88f9e00640 1 -- 172.21.15.28:0/735741679 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558bcf7110f0 con 0x558bcfad9860 2024-04-24T08:12:43.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.147+0000 7f88f9e00640 1 --2- 172.21.15.28:0/735741679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfad9860 0x558bcfaeca10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0a5160 tx=0x7f88ec005bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bb95c14fdb513d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:12:43.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.147+0000 7f88df400640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:12:43.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.147+0000 7f88df400640 1 -- 172.21.15.28:0/735741679 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88ec09c610 con 0x558bcfad9860 2024-04-24T08:12:43.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.147+0000 7f88df400640 1 -- 172.21.15.28:0/735741679 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f88ec0a5200 con 0x558bcfad9860 2024-04-24T08:12:43.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.147+0000 7f89039e4740 1 -- 172.21.15.28:0/735741679 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88ec09cad0 con 0x558bcfad9860 2024-04-24T08:12:43.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.148+0000 7f88df400640 1 -- 172.21.15.28:0/735741679 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f88ec09c430 con 0x558bcfad9860 2024-04-24T08:12:43.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.148+0000 7f89039e4740 10 client.25886 fetch_fsmap learned FSMap version 25 2024-04-24T08:12:43.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.148+0000 7f89039e4740 1 -- 172.21.15.28:0/735741679 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558bcfa04810 con 0x558bcfad9860 2024-04-24T08:12:43.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.148+0000 7f88df400640 1 -- 172.21.15.28:0/735741679 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88ec09cad0 con 0x558bcfad9860 2024-04-24T08:12:43.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.148+0000 7f88df400640 1 -- 172.21.15.28:0/735741679 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f88ec09bf20 con 0x558bcfad9860 2024-04-24T08:12:43.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.148+0000 7f89039e4740 10 client.25886 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:12:43.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.148+0000 7f89039e4740 10 client.25886 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:12:43.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.148+0000 7f89039e4740 20 client.25886 populate_metadata read hostname 'smithi028' 2024-04-24T08:12:43.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.148+0000 7f89039e4740 1 --2- 172.21.15.28:0/735741679 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x558bcfb5a980 0x558bcfb5cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:12:43.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.148+0000 7f89039e4740 4 client.25886 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:12:43.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.148+0000 7f89039e4740 1 -- 172.21.15.28:0/735741679 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x558bcfa46a80 con 0x558bcfb5a980 2024-04-24T08:12:43.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.148+0000 7f88f3e00640 1 --2- 172.21.15.28:0/735741679 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x558bcfb5a980 0x558bcfb5cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:12:43.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.149+0000 7f88f3e00640 1 --2- 172.21.15.28:0/735741679 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x558bcfb5a980 0x558bcfb5cda0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3298380e84b056e3 server_cookie=d0a50044bed8c974 in_seq=0 out_seq=0 2024-04-24T08:12:43.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.149+0000 7f88df400640 10 client.25886 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:43.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.149+0000 7f88df400640 1 -- 172.21.15.28:0/735741679 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x558bcfa46a80 con 0x558bcfb5a980 2024-04-24T08:12:43.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.149+0000 7f88df400640 10 client.25886 handle_command_reply: tid=0 2024-04-24T08:12:43.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.149+0000 7f89039e4740 1 client.25886 shutdown 2024-04-24T08:12:43.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.150+0000 7f89039e4740 1 -- 172.21.15.28:0/735741679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfad9860 msgr2=0x558bcfaeca10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:43.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.150+0000 7f89039e4740 1 --2- 172.21.15.28:0/735741679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfad9860 0x558bcfaeca10 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f88ec0a5160 tx=0x7f88ec005bf0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.150+0000 7f89039e4740 1 -- 172.21.15.28:0/735741679 shutdown_connections 2024-04-24T08:12:43.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.150+0000 7f89039e4740 1 --2- 172.21.15.28:0/735741679 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x558bcfb5a980 0x558bcfb5cda0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.150+0000 7f89039e4740 1 --2- 172.21.15.28:0/735741679 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558bcfada180 0x558bcfaecf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.150+0000 7f89039e4740 1 --2- 172.21.15.28:0/735741679 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558bcfae4850 0x558bcfaddb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.150+0000 7f89039e4740 1 --2- 172.21.15.28:0/735741679 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558bcfad9860 0x558bcfaeca10 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.150+0000 7f89039e4740 1 -- 172.21.15.28:0/735741679 >> 172.21.15.28:0/735741679 conn(0x558bcfad5b80 msgr2=0x558bcfadafb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:43.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.150+0000 7f88df400640 0 client.25886 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:12:43.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.150+0000 7f89039e4740 1 -- 172.21.15.28:0/735741679 shutdown_connections 2024-04-24T08:12:43.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.150+0000 7f89039e4740 1 -- 172.21.15.28:0/735741679 wait complete. 2024-04-24T08:12:43.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.150+0000 7f89039e4740 20 client.25886 trim_cache size 0 max 16384 2024-04-24T08:12:43.156 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:12:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:12:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:12:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "eb6d4250-04da-4a87-8805-5c3b00715606": { 2024-04-24T08:12:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:12:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "eb6d4250-04da-4a87-8805-5c3b00715606", 2024-04-24T08:12:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:12:43.156 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:12:43.157 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:12:43.157 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:12:43.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.151+0000 7f8900c00640 1 -- 172.21.15.28:0/3746879325 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88e00a3da0 msgr2=0x7f88e00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:43.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.151+0000 7f8900c00640 1 --2- 172.21.15.28:0/3746879325 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88e00a3da0 0x7f88e00a6260 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f88f4005fd0 tx=0x7f88f4009500 comp rx=0 tx=0).stop 2024-04-24T08:12:43.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.151+0000 7f8900c00640 1 -- 172.21.15.28:0/3746879325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88fc102240 msgr2=0x7f88fc1a0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:12:43.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.151+0000 7f8900c00640 1 --2- 172.21.15.28:0/3746879325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88fc102240 0x7f88fc1a0770 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f88ec00e9f0 tx=0x7f88ec00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.151+0000 7f8900c00640 1 -- 172.21.15.28:0/3746879325 shutdown_connections 2024-04-24T08:12:43.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.151+0000 7f8900c00640 1 --2- 172.21.15.28:0/3746879325 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88e00a3da0 0x7f88e00a6260 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.151+0000 7f8900c00640 1 --2- 172.21.15.28:0/3746879325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88fc19a920 0x7f88fc19add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.151+0000 7f8900c00640 1 --2- 172.21.15.28:0/3746879325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88fc107310 0x7f88fc1a0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.151+0000 7f8900c00640 1 --2- 172.21.15.28:0/3746879325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88fc102240 0x7f88fc1a0770 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:12:43.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.151+0000 7f8900c00640 1 -- 172.21.15.28:0/3746879325 >> 172.21.15.28:0/3746879325 conn(0x7f88fc079450 msgr2=0x7f88fc1001a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:12:43.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.151+0000 7f8900c00640 1 -- 172.21.15.28:0/3746879325 shutdown_connections 2024-04-24T08:12:43.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:12:43.151+0000 7f8900c00640 1 -- 172.21.15.28:0/3746879325 wait complete. 2024-04-24T08:12:43.168 INFO:tasks.cephfs.filesystem:scrub status for tag:eb6d4250-04da-4a87-8805-5c3b00715606 - {'path': '/', 'tag': 'eb6d4250-04da-4a87-8805-5c3b00715606', 'options': 'recursive,force'} 2024-04-24T08:12:43.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:43 smithi028 ceph-mon[27786]: pgmap v74: 97 pgs: 97 active+clean; 22 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.4 MiB/s wr, 60 op/s 2024-04-24T08:12:43.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:43 smithi117 ceph-mon[32438]: pgmap v74: 97 pgs: 97 active+clean; 22 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.4 MiB/s wr, 60 op/s 2024-04-24T08:12:43.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:43 smithi194 ceph-mon[32836]: pgmap v74: 97 pgs: 97 active+clean; 22 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 224 KiB/s rd, 1.4 MiB/s wr, 60 op/s 2024-04-24T08:12:44.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:44 smithi028 ceph-mon[27786]: scrub queued for path: / 2024-04-24T08:12:44.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:44 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:44.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:44 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:12:44.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:44 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:12:44.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:44 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:44.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:44 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:12:44.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:44 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:44.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:44 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:12:44.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:44 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:44.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:44 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:44.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:44 smithi117 ceph-mon[32438]: scrub queued for path: / 2024-04-24T08:12:44.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:44 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:44.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:44 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:12:44.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:44 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:12:44.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:44 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:44.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:44 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:12:44.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:44 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:44.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:44 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:12:44.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:44 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:44.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:44 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:44.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:44 smithi194 ceph-mon[32836]: scrub queued for path: / 2024-04-24T08:12:44.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:44 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:44.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:44 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:12:44.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:44 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:12:44.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:44 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:44.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:44 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:12:44.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:44 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:44.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:44 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:12:44.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:44 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:44.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:44 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:12:45.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:45 smithi028 ceph-mon[27786]: pgmap v75: 97 pgs: 97 active+clean; 22 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.3 MiB/s wr, 41 op/s 2024-04-24T08:12:45.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:45 smithi117 ceph-mon[32438]: pgmap v75: 97 pgs: 97 active+clean; 22 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.3 MiB/s wr, 41 op/s 2024-04-24T08:12:45.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:45 smithi194 ceph-mon[32836]: pgmap v75: 97 pgs: 97 active+clean; 22 MiB data, 376 MiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 1.3 MiB/s wr, 41 op/s 2024-04-24T08:12:47.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:47 smithi028 ceph-mon[27786]: pgmap v76: 97 pgs: 97 active+clean; 22 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 1.4 MiB/s wr, 67 op/s 2024-04-24T08:12:47.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:47 smithi117 ceph-mon[32438]: pgmap v76: 97 pgs: 97 active+clean; 22 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 1.4 MiB/s wr, 67 op/s 2024-04-24T08:12:47.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:47 smithi194 ceph-mon[32836]: pgmap v76: 97 pgs: 97 active+clean; 22 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 1.4 MiB/s wr, 67 op/s 2024-04-24T08:12:49.798 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:49 smithi028 ceph-mon[27786]: pgmap v77: 97 pgs: 97 active+clean; 22 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.2 MiB/s wr, 45 op/s 2024-04-24T08:12:49.799 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:49 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:12:49.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:49 smithi117 ceph-mon[32438]: pgmap v77: 97 pgs: 97 active+clean; 22 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.2 MiB/s wr, 45 op/s 2024-04-24T08:12:49.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:49 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:12:49.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:49 smithi194 ceph-mon[32836]: pgmap v77: 97 pgs: 97 active+clean; 22 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.2 MiB/s wr, 45 op/s 2024-04-24T08:12:49.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:49 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:12:50.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:50 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:12:50.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:50 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:12:50.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:50 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:12:51.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:51 smithi028 ceph-mon[27786]: pgmap v78: 97 pgs: 97 active+clean; 22 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.2 MiB/s wr, 45 op/s 2024-04-24T08:12:51.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:51 smithi117 ceph-mon[32438]: pgmap v78: 97 pgs: 97 active+clean; 22 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.2 MiB/s wr, 45 op/s 2024-04-24T08:12:51.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:51 smithi194 ceph-mon[32836]: pgmap v78: 97 pgs: 97 active+clean; 22 MiB data, 385 MiB used, 1.0 TiB / 1.0 TiB avail; 110 KiB/s rd, 1.2 MiB/s wr, 45 op/s 2024-04-24T08:12:53.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:53 smithi028 ceph-mon[27786]: pgmap v79: 97 pgs: 97 active+clean; 694 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 57 MiB/s wr, 74 op/s 2024-04-24T08:12:53.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:53 smithi117 ceph-mon[32438]: pgmap v79: 97 pgs: 97 active+clean; 694 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 57 MiB/s wr, 74 op/s 2024-04-24T08:12:53.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:53 smithi194 ceph-mon[32836]: pgmap v79: 97 pgs: 97 active+clean; 694 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 57 MiB/s wr, 74 op/s 2024-04-24T08:12:55.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:55 smithi117 ceph-mon[32438]: pgmap v80: 97 pgs: 97 active+clean; 694 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 56 MiB/s wr, 55 op/s 2024-04-24T08:12:55.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:55 smithi194 ceph-mon[32836]: pgmap v80: 97 pgs: 97 active+clean; 694 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 56 MiB/s wr, 55 op/s 2024-04-24T08:12:56.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:55 smithi028 ceph-mon[27786]: pgmap v80: 97 pgs: 97 active+clean; 694 MiB data, 500 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 56 MiB/s wr, 55 op/s 2024-04-24T08:12:57.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:57 smithi117 ceph-mon[32438]: pgmap v81: 97 pgs: 97 active+clean; 2.0 GiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 166 MiB/s wr, 97 op/s 2024-04-24T08:12:57.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:57 smithi194 ceph-mon[32836]: pgmap v81: 97 pgs: 97 active+clean; 2.0 GiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 166 MiB/s wr, 97 op/s 2024-04-24T08:12:58.042 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:57 smithi028 ceph-mon[27786]: pgmap v81: 97 pgs: 97 active+clean; 2.0 GiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 132 KiB/s rd, 166 MiB/s wr, 97 op/s 2024-04-24T08:12:59.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:12:59 smithi117 ceph-mon[32438]: pgmap v82: 97 pgs: 97 active+clean; 2.0 GiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 166 MiB/s wr, 71 op/s 2024-04-24T08:12:59.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:12:59 smithi194 ceph-mon[32836]: pgmap v82: 97 pgs: 97 active+clean; 2.0 GiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 166 MiB/s wr, 71 op/s 2024-04-24T08:13:00.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:12:59 smithi028 ceph-mon[27786]: pgmap v82: 97 pgs: 97 active+clean; 2.0 GiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 166 MiB/s wr, 71 op/s 2024-04-24T08:13:01.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:01 smithi117 ceph-mon[32438]: pgmap v83: 97 pgs: 97 active+clean; 2.0 GiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 166 MiB/s wr, 71 op/s 2024-04-24T08:13:01.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:01 smithi194 ceph-mon[32836]: pgmap v83: 97 pgs: 97 active+clean; 2.0 GiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 166 MiB/s wr, 71 op/s 2024-04-24T08:13:02.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:01 smithi028 ceph-mon[27786]: pgmap v83: 97 pgs: 97 active+clean; 2.0 GiB data, 768 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 166 MiB/s wr, 71 op/s 2024-04-24T08:13:03.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:03 smithi117 ceph-mon[32438]: pgmap v84: 97 pgs: 97 active+clean; 3.3 GiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 281 MiB/s wr, 127 op/s 2024-04-24T08:13:03.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:03 smithi194 ceph-mon[32836]: pgmap v84: 97 pgs: 97 active+clean; 3.3 GiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 281 MiB/s wr, 127 op/s 2024-04-24T08:13:04.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:03 smithi028 ceph-mon[27786]: pgmap v84: 97 pgs: 97 active+clean; 3.3 GiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 281 MiB/s wr, 127 op/s 2024-04-24T08:13:05.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:05 smithi117 ceph-mon[32438]: pgmap v85: 97 pgs: 97 active+clean; 3.3 GiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 225 MiB/s wr, 98 op/s 2024-04-24T08:13:05.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:05 smithi194 ceph-mon[32836]: pgmap v85: 97 pgs: 97 active+clean; 3.3 GiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 225 MiB/s wr, 98 op/s 2024-04-24T08:13:06.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:05 smithi028 ceph-mon[27786]: pgmap v85: 97 pgs: 97 active+clean; 3.3 GiB data, 1010 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 225 MiB/s wr, 98 op/s 2024-04-24T08:13:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:07 smithi117 ceph-mon[32438]: pgmap v86: 97 pgs: 97 active+clean; 4.9 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 360 MiB/s wr, 143 op/s 2024-04-24T08:13:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:07 smithi194 ceph-mon[32836]: pgmap v86: 97 pgs: 97 active+clean; 4.9 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 360 MiB/s wr, 143 op/s 2024-04-24T08:13:08.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:07 smithi028 ceph-mon[27786]: pgmap v86: 97 pgs: 97 active+clean; 4.9 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 114 KiB/s rd, 360 MiB/s wr, 143 op/s 2024-04-24T08:13:09.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:09 smithi117 ceph-mon[32438]: pgmap v87: 97 pgs: 97 active+clean; 4.9 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 250 MiB/s wr, 101 op/s 2024-04-24T08:13:09.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:09 smithi194 ceph-mon[32836]: pgmap v87: 97 pgs: 97 active+clean; 4.9 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 250 MiB/s wr, 101 op/s 2024-04-24T08:13:10.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:09 smithi028 ceph-mon[27786]: pgmap v87: 97 pgs: 97 active+clean; 4.9 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 250 MiB/s wr, 101 op/s 2024-04-24T08:13:11.842 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:11 smithi028 ceph-mon[27786]: pgmap v88: 97 pgs: 97 active+clean; 4.9 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 250 MiB/s wr, 101 op/s 2024-04-24T08:13:11.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:11 smithi117 ceph-mon[32438]: pgmap v88: 97 pgs: 97 active+clean; 4.9 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 250 MiB/s wr, 101 op/s 2024-04-24T08:13:11.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:11 smithi194 ceph-mon[32836]: pgmap v88: 97 pgs: 97 active+clean; 4.9 GiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 250 MiB/s wr, 101 op/s 2024-04-24T08:13:13.168 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:13:13.169 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:13:13.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.406+0000 7f8f7c200640 1 -- 172.21.15.28:0/3605528462 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f74109210 msgr2=0x7f8f741095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:13.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.406+0000 7f8f7c200640 1 --2- 172.21.15.28:0/3605528462 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f74109210 0x7f8f741095f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f8f60009900 tx=0x7f8f6002a440 comp rx=0 tx=0).stop 2024-04-24T08:13:13.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.406+0000 7f8f7c200640 1 -- 172.21.15.28:0/3605528462 shutdown_connections 2024-04-24T08:13:13.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.406+0000 7f8f7c200640 1 --2- 172.21.15.28:0/3605528462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f74115150 0x7f8f74117540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.406+0000 7f8f7c200640 1 --2- 172.21.15.28:0/3605528462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f74109b30 0x7f8f74114c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.406+0000 7f8f7c200640 1 --2- 172.21.15.28:0/3605528462 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f74109210 0x7f8f741095f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.406+0000 7f8f7c200640 1 -- 172.21.15.28:0/3605528462 >> 172.21.15.28:0/3605528462 conn(0x7f8f7406d110 msgr2=0x7f8f7406d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.407+0000 7f8f7c200640 1 -- 172.21.15.28:0/3605528462 shutdown_connections 2024-04-24T08:13:13.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.407+0000 7f8f7c200640 1 -- 172.21.15.28:0/3605528462 wait complete. 2024-04-24T08:13:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.408+0000 7f8f7c200640 1 Processor -- start 2024-04-24T08:13:13.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.409+0000 7f8f7c200640 1 -- start start 2024-04-24T08:13:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.409+0000 7f8f7c200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f74109210 0x7f8f741ad420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.409+0000 7f8f7c200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f74109b30 0x7f8f741ad960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.409+0000 7f8f7c200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f74115150 0x7f8f741a75d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.409+0000 7f8f7c200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f74118a80 con 0x7f8f74109210 2024-04-24T08:13:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.409+0000 7f8f7c200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8f74118900 con 0x7f8f74109b30 2024-04-24T08:13:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.409+0000 7f8f7c200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8f74118c00 con 0x7f8f74115150 2024-04-24T08:13:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.410+0000 7f8f78e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f74109b30 0x7f8f741ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.410+0000 7f8f7a200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f74115150 0x7f8f741a75d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.410+0000 7f8f78e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f74109b30 0x7f8f741ad960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40794/0 (socket says 172.21.15.28:40794) 2024-04-24T08:13:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.410+0000 7f8f78e00640 1 -- 172.21.15.28:0/1362359570 learned_addr learned my addr 172.21.15.28:0/1362359570 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.410+0000 7f8f7a200640 1 -- 172.21.15.28:0/1362359570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f74109b30 msgr2=0x7f8f741ad960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.410+0000 7f8f7a200640 1 --2- 172.21.15.28:0/1362359570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f74109b30 0x7f8f741ad960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.410+0000 7f8f7a200640 1 -- 172.21.15.28:0/1362359570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f74109210 msgr2=0x7f8f741ad420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.410+0000 7f8f7a200640 1 --2- 172.21.15.28:0/1362359570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f74109210 0x7f8f741ad420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.410+0000 7f8f7a200640 1 -- 172.21.15.28:0/1362359570 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f741a7d50 con 0x7f8f74115150 2024-04-24T08:13:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.411+0000 7f8f7a200640 1 --2- 172.21.15.28:0/1362359570 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f74115150 0x7f8f741a75d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8f6800c710 tx=0x7f8f6800cbd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.411+0000 7f8f72000640 1 -- 172.21.15.28:0/1362359570 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f68004220 con 0x7f8f74115150 2024-04-24T08:13:13.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.411+0000 7f8f72000640 1 -- 172.21.15.28:0/1362359570 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f6800ce20 con 0x7f8f74115150 2024-04-24T08:13:13.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.411+0000 7f8f7c200640 1 -- 172.21.15.28:0/1362359570 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f741a8040 con 0x7f8f74115150 2024-04-24T08:13:13.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.411+0000 7f8f72000640 1 -- 172.21.15.28:0/1362359570 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f68003e20 con 0x7f8f74115150 2024-04-24T08:13:13.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.411+0000 7f8f7c200640 1 -- 172.21.15.28:0/1362359570 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f741b42e0 con 0x7f8f74115150 2024-04-24T08:13:13.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.418+0000 7f8f72000640 1 -- 172.21.15.28:0/1362359570 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f8f6800f450 con 0x7f8f74115150 2024-04-24T08:13:13.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.420+0000 7f8f7f19b740 1 Processor -- start 2024-04-24T08:13:13.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.420+0000 7f8f7f19b740 1 -- start start 2024-04-24T08:13:13.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.421+0000 7f8f7f19b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae54860 0x55a38ae54c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.421+0000 7f8f7f19b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae55180 0x55a38ae5f310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.421+0000 7f8f7f19b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5f850 0x55a38ae61c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.421+0000 7f8f7f19b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a38adba1f0 con 0x55a38ae54860 2024-04-24T08:13:13.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.421+0000 7f8f7f19b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a38ad60350 con 0x55a38ae5f850 2024-04-24T08:13:13.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.421+0000 7f8f7f19b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a38adb61b0 con 0x55a38ae55180 2024-04-24T08:13:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.421+0000 7f8f78e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5f850 0x55a38ae61c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.422+0000 7f8f78e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5f850 0x55a38ae61c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40808/0 (socket says 172.21.15.28:40808) 2024-04-24T08:13:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.422+0000 7f8f78e00640 1 -- 172.21.15.28:0/3327375299 learned_addr learned my addr 172.21.15.28:0/3327375299 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.422+0000 7f8f78e00640 1 --2- 172.21.15.28:0/3327375299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae54860 0x55a38ae54c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.422+0000 7f8f78e00640 1 -- 172.21.15.28:0/3327375299 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae55180 msgr2=0x55a38ae5f310 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:13:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.422+0000 7f8f78e00640 1 --2- 172.21.15.28:0/3327375299 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae55180 0x55a38ae5f310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.422+0000 7f8f78e00640 1 -- 172.21.15.28:0/3327375299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae54860 msgr2=0x55a38ae54c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.422+0000 7f8f78e00640 1 --2- 172.21.15.28:0/3327375299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae54860 0x55a38ae54c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.422+0000 7f8f78e00640 1 -- 172.21.15.28:0/3327375299 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a38adb4f50 con 0x55a38ae5f850 2024-04-24T08:13:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.422+0000 7f8f72000640 1 --2- 172.21.15.28:0/1362359570 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8f5c0a3e00 0x7f8f5c0a62c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.423+0000 7f8f78e00640 1 --2- 172.21.15.28:0/3327375299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5f850 0x55a38ae61c40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c013a00 tx=0x7f8f6c009b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40b92325f281a253 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:13.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.423+0000 7f8f72000640 1 -- 172.21.15.28:0/1362359570 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f8f680d03b0 con 0x7f8f74115150 2024-04-24T08:13:13.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.423+0000 7f8f66a00640 1 -- 172.21.15.28:0/3327375299 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c003b70 con 0x55a38ae5f850 2024-04-24T08:13:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.423+0000 7f8f66a00640 1 -- 172.21.15.28:0/3327375299 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f6c003d10 con 0x55a38ae5f850 2024-04-24T08:13:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.423+0000 7f8f66a00640 1 -- 172.21.15.28:0/3327375299 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c003b70 con 0x55a38ae5f850 2024-04-24T08:13:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.424+0000 7f8f7f19b740 1 -- 172.21.15.28:0/3327375299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5f850 msgr2=0x55a38ae61c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.424+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/3327375299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5f850 0x55a38ae61c40 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c013a00 tx=0x7f8f6c009b40 comp rx=0 tx=0).stop 2024-04-24T08:13:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.424+0000 7f8f7f19b740 1 -- 172.21.15.28:0/3327375299 shutdown_connections 2024-04-24T08:13:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.424+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/3327375299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5f850 0x55a38ae61c40 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.424+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/3327375299 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae55180 0x55a38ae5f310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.425+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/3327375299 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae54860 0x55a38ae54c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.425+0000 7f8f7f19b740 1 -- 172.21.15.28:0/3327375299 >> 172.21.15.28:0/3327375299 conn(0x55a38ae50b80 msgr2=0x55a38ae50f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.428+0000 7f8f7f19b740 1 -- 172.21.15.28:0/3327375299 shutdown_connections 2024-04-24T08:13:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.428+0000 7f8f7f19b740 1 -- 172.21.15.28:0/3327375299 wait complete. 2024-04-24T08:13:13.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.429+0000 7f8f7f19b740 1 -- start start 2024-04-24T08:13:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.429+0000 7f8f7f19b740 1 Processor -- start 2024-04-24T08:13:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.430+0000 7f8f79800640 1 --2- 172.21.15.28:0/1362359570 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8f5c0a3e00 0x7f8f5c0a62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.430+0000 7f8f7f19b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae54860 0x55a38ae5ca00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.430+0000 7f8f7f19b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5f850 0x55a38ae5cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.430+0000 7f8f7f19b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae5d480 0x55a38ae7cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.430+0000 7f8f7f19b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a38ae73650 con 0x55a38ae5d480 2024-04-24T08:13:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.430+0000 7f8f7f19b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a38ad73690 con 0x55a38ae5f850 2024-04-24T08:13:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.430+0000 7f8f7f19b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a38ae737d0 con 0x55a38ae54860 2024-04-24T08:13:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.430+0000 7f8f7f19b740 10 client.0 _ll_register_callbacks cb 0x55a38adaf650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.431+0000 7f8f78e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5f850 0x55a38ae5cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.431+0000 7f8f78e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5f850 0x55a38ae5cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40820/0 (socket says 172.21.15.28:40820) 2024-04-24T08:13:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.431+0000 7f8f78e00640 1 -- 172.21.15.28:0/1823056407 learned_addr learned my addr 172.21.15.28:0/1823056407 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:13.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.431+0000 7f8f79800640 1 --2- 172.21.15.28:0/1823056407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae54860 0x55a38ae5ca00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.432+0000 7f8f7a200640 1 --2- 172.21.15.28:0/1823056407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae5d480 0x55a38ae7cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.432+0000 7f8f78e00640 1 -- 172.21.15.28:0/1823056407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae54860 msgr2=0x55a38ae5ca00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:13.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.432+0000 7f8f78e00640 1 --2- 172.21.15.28:0/1823056407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae54860 0x55a38ae5ca00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.432+0000 7f8f78e00640 1 -- 172.21.15.28:0/1823056407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae5d480 msgr2=0x55a38ae7cac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:13.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.432+0000 7f8f78e00640 1 --2- 172.21.15.28:0/1823056407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae5d480 0x55a38ae7cac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.432+0000 7f8f78e00640 1 -- 172.21.15.28:0/1823056407 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a38ad72920 con 0x55a38ae5f850 2024-04-24T08:13:13.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.432+0000 7f8f7a200640 1 --2- 172.21.15.28:0/1823056407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae5d480 0x55a38ae7cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:13.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.433+0000 7f8f79800640 1 --2- 172.21.15.28:0/1362359570 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8f5c0a3e00 0x7f8f5c0a62c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8f6002aa30 tx=0x7f8f60009680 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:13.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.433+0000 7f8f78e00640 1 --2- 172.21.15.28:0/1823056407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5f850 0x55a38ae5cf40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c011750 tx=0x7f8f6c003ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6bac10d7ac5cdb7b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:13.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.434+0000 7f8f5b400640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:13:13.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.434+0000 7f8f5b400640 1 -- 172.21.15.28:0/1823056407 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6c02f060 con 0x55a38ae5f850 2024-04-24T08:13:13.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.434+0000 7f8f5b400640 1 -- 172.21.15.28:0/1823056407 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f8f6c027070 con 0x55a38ae5f850 2024-04-24T08:13:13.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.435+0000 7f8f7f19b740 1 -- 172.21.15.28:0/1823056407 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f6c003d10 con 0x55a38ae5f850 2024-04-24T08:13:13.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.435+0000 7f8f5b400640 1 -- 172.21.15.28:0/1823056407 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8f6c02f060 con 0x55a38ae5f850 2024-04-24T08:13:13.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.435+0000 7f8f7f19b740 10 client.25898 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:13.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.436+0000 7f8f7f19b740 1 -- 172.21.15.28:0/1823056407 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a38ab967d0 con 0x55a38ae5f850 2024-04-24T08:13:13.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.436+0000 7f8f5b400640 1 -- 172.21.15.28:0/1823056407 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f6c003d10 con 0x55a38ae5f850 2024-04-24T08:13:13.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.436+0000 7f8f5b400640 1 -- 172.21.15.28:0/1823056407 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f8f6c00f040 con 0x55a38ae5f850 2024-04-24T08:13:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.437+0000 7f8f7f19b740 10 client.25898 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.437+0000 7f8f7f19b740 10 client.25898 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:13:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.437+0000 7f8f7f19b740 20 client.25898 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.437+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/1823056407 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a38ae81110 0x55a38ae83500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.437+0000 7f8f7f19b740 4 client.25898 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:13:13.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.437+0000 7f8f7f19b740 1 -- 172.21.15.28:0/1823056407 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a38adc7c10 con 0x55a38ae81110 2024-04-24T08:13:13.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.438+0000 7f8f79800640 1 --2- 172.21.15.28:0/1823056407 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a38ae81110 0x55a38ae83500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.439+0000 7f8f79800640 1 --2- 172.21.15.28:0/1823056407 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a38ae81110 0x55a38ae83500 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aa04a7db074051e4 server_cookie=a9c1c97f3c25879 in_seq=0 out_seq=0 2024-04-24T08:13:13.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.440+0000 7f8f5b400640 10 client.25898 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:13.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.444+0000 7f8f5b400640 1 -- 172.21.15.28:0/1823056407 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a38adc7c10 con 0x55a38ae81110 2024-04-24T08:13:13.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.444+0000 7f8f5b400640 10 client.25898 handle_command_reply: tid=0 2024-04-24T08:13:13.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.444+0000 7f8f7f19b740 1 client.25898 shutdown 2024-04-24T08:13:13.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.445+0000 7f8f7f19b740 1 -- 172.21.15.28:0/1823056407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5f850 msgr2=0x55a38ae5cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:13.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.445+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/1823056407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5f850 0x55a38ae5cf40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f8f6c011750 tx=0x7f8f6c003ba0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.445+0000 7f8f7f19b740 1 -- 172.21.15.28:0/1823056407 shutdown_connections 2024-04-24T08:13:13.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.445+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/1823056407 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a38ae81110 0x55a38ae83500 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.445+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/1823056407 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae5d480 0x55a38ae7cac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.445+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/1823056407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5f850 0x55a38ae5cf40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.445+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/1823056407 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae54860 0x55a38ae5ca00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.445+0000 7f8f7f19b740 1 -- 172.21.15.28:0/1823056407 >> 172.21.15.28:0/1823056407 conn(0x55a38ae50b80 msgr2=0x55a38ae56980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:13.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.445+0000 7f8f5b400640 0 client.25898 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:13.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.445+0000 7f8f7f19b740 1 -- 172.21.15.28:0/1823056407 shutdown_connections 2024-04-24T08:13:13.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.446+0000 7f8f7f19b740 1 -- 172.21.15.28:0/1823056407 wait complete. 2024-04-24T08:13:13.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.449+0000 7f8f7f19b740 20 client.25898 trim_cache size 0 max 16384 2024-04-24T08:13:13.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.496+0000 7f8f7f19b740 1 Processor -- start 2024-04-24T08:13:13.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.496+0000 7f8f7f19b740 1 -- start start 2024-04-24T08:13:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.496+0000 7f8f7f19b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae50b80 0x55a38ae88bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.496+0000 7f8f7f19b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae54860 0x55a38ae89130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.496+0000 7f8f7f19b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae5d480 0x55a38ae7c510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.496+0000 7f8f7f19b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f741b6e40 con 0x55a38ae54860 2024-04-24T08:13:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.496+0000 7f8f7f19b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8f741b4710 con 0x55a38ae50b80 2024-04-24T08:13:13.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.496+0000 7f8f7f19b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8f741a4690 con 0x55a38ae5d480 2024-04-24T08:13:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.497+0000 7f8f7a200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae5d480 0x55a38ae7c510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.497+0000 7f8f7a200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae5d480 0x55a38ae7c510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:56846/0 (socket says 172.21.15.28:56846) 2024-04-24T08:13:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.497+0000 7f8f7a200640 1 -- 172.21.15.28:0/504446943 learned_addr learned my addr 172.21.15.28:0/504446943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.498+0000 7f8f7a200640 1 -- 172.21.15.28:0/504446943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae50b80 msgr2=0x55a38ae88bf0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:13:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.498+0000 7f8f7a200640 1 --2- 172.21.15.28:0/504446943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae50b80 0x55a38ae88bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.498+0000 7f8f7a200640 1 -- 172.21.15.28:0/504446943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae54860 msgr2=0x55a38ae89130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:13.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.498+0000 7f8f7a200640 1 --2- 172.21.15.28:0/504446943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae54860 0x55a38ae89130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.498+0000 7f8f7a200640 1 -- 172.21.15.28:0/504446943 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a38aacc3f0 con 0x55a38ae5d480 2024-04-24T08:13:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.498+0000 7f8f7a200640 1 --2- 172.21.15.28:0/504446943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae5d480 0x55a38ae7c510 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8f6809d290 tx=0x7f8f6809cc60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8909cb041fba1344 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.499+0000 7f8f66000640 1 -- 172.21.15.28:0/504446943 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f680b5070 con 0x55a38ae5d480 2024-04-24T08:13:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.499+0000 7f8f66000640 1 -- 172.21.15.28:0/504446943 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f680a16b0 con 0x55a38ae5d480 2024-04-24T08:13:13.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.499+0000 7f8f66000640 1 -- 172.21.15.28:0/504446943 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6801d040 con 0x55a38ae5d480 2024-04-24T08:13:13.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.501+0000 7f8f7f19b740 1 -- 172.21.15.28:0/504446943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae5d480 msgr2=0x55a38ae7c510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:13.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.501+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/504446943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae5d480 0x55a38ae7c510 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f8f6809d290 tx=0x7f8f6809cc60 comp rx=0 tx=0).stop 2024-04-24T08:13:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.501+0000 7f8f7f19b740 1 -- 172.21.15.28:0/504446943 shutdown_connections 2024-04-24T08:13:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.501+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/504446943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae54860 0x55a38ae89130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.501+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/504446943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae5d480 0x55a38ae7c510 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.501+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/504446943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae50b80 0x55a38ae88bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.501+0000 7f8f7f19b740 1 -- 172.21.15.28:0/504446943 >> 172.21.15.28:0/504446943 conn(0x55a38ae5d0a0 msgr2=0x55a38ae5b2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.501+0000 7f8f7f19b740 1 -- 172.21.15.28:0/504446943 shutdown_connections 2024-04-24T08:13:13.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.501+0000 7f8f7f19b740 1 -- 172.21.15.28:0/504446943 wait complete. 2024-04-24T08:13:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.502+0000 7f8f7f19b740 1 -- start start 2024-04-24T08:13:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.503+0000 7f8f7f19b740 1 Processor -- start 2024-04-24T08:13:13.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.503+0000 7f8f7f19b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae50b80 0x55a38ae78cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.503+0000 7f8f7f19b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae54860 0x55a38ae59270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.503+0000 7f8f7f19b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5d480 0x55a38ae597b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.503+0000 7f8f7f19b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8f741a4d00 con 0x55a38ae54860 2024-04-24T08:13:13.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.503+0000 7f8f7f19b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8f741a6170 con 0x55a38ae5d480 2024-04-24T08:13:13.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.503+0000 7f8f7f19b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8f741a53d0 con 0x55a38ae50b80 2024-04-24T08:13:13.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.503+0000 7f8f7f19b740 10 client.0 _ll_register_callbacks cb 0x7f8f6c0138d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.504+0000 7f8f7a200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5d480 0x55a38ae597b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.504+0000 7f8f7a200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5d480 0x55a38ae597b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40840/0 (socket says 172.21.15.28:40840) 2024-04-24T08:13:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.504+0000 7f8f7a200640 1 -- 172.21.15.28:0/681482574 learned_addr learned my addr 172.21.15.28:0/681482574 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.504+0000 7f8f79800640 1 --2- 172.21.15.28:0/681482574 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae50b80 0x55a38ae78cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.504+0000 7f8f78e00640 1 --2- 172.21.15.28:0/681482574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae54860 0x55a38ae59270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.505+0000 7f8f7a200640 1 -- 172.21.15.28:0/681482574 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae50b80 msgr2=0x55a38ae78cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.505+0000 7f8f7a200640 1 --2- 172.21.15.28:0/681482574 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae50b80 0x55a38ae78cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.505+0000 7f8f7a200640 1 -- 172.21.15.28:0/681482574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae54860 msgr2=0x55a38ae59270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.505+0000 7f8f7a200640 1 --2- 172.21.15.28:0/681482574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae54860 0x55a38ae59270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.505+0000 7f8f7a200640 1 -- 172.21.15.28:0/681482574 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a38aa8c0f0 con 0x55a38ae5d480 2024-04-24T08:13:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.505+0000 7f8f78e00640 1 --2- 172.21.15.28:0/681482574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae54860 0x55a38ae59270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:13:13.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.505+0000 7f8f7a200640 1 --2- 172.21.15.28:0/681482574 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5d480 0x55a38ae597b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8f68099cd0 tx=0x7f8f68005750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2c82e328d81455cf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:13.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.505+0000 7f8f5b400640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:13:13.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.505+0000 7f8f5b400640 1 -- 172.21.15.28:0/681482574 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f6801d040 con 0x55a38ae5d480 2024-04-24T08:13:13.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.505+0000 7f8f5b400640 1 -- 172.21.15.28:0/681482574 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f8f6809c750 con 0x55a38ae5d480 2024-04-24T08:13:13.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.505+0000 7f8f7f19b740 1 -- 172.21.15.28:0/681482574 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f680a16b0 con 0x55a38ae5d480 2024-04-24T08:13:13.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.506+0000 7f8f5b400640 1 -- 172.21.15.28:0/681482574 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f8f6807e050 con 0x55a38ae5d480 2024-04-24T08:13:13.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.506+0000 7f8f7f19b740 10 client.25901 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:13.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.506+0000 7f8f7f19b740 1 -- 172.21.15.28:0/681482574 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a38ad7f810 con 0x55a38ae5d480 2024-04-24T08:13:13.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.506+0000 7f8f5b400640 1 -- 172.21.15.28:0/681482574 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f6809c950 con 0x55a38ae5d480 2024-04-24T08:13:13.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.506+0000 7f8f5b400640 1 -- 172.21.15.28:0/681482574 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f8f6809c350 con 0x55a38ae5d480 2024-04-24T08:13:13.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.506+0000 7f8f7f19b740 10 client.25901 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:13.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.506+0000 7f8f7f19b740 10 client.25901 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:13:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.506+0000 7f8f7f19b740 20 client.25901 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.506+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/681482574 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a38ae5f850 0x55a38aee95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.506+0000 7f8f7f19b740 4 client.25901 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:13:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.506+0000 7f8f7f19b740 1 -- 172.21.15.28:0/681482574 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f8f680a16b0 con 0x55a38ae5f850 2024-04-24T08:13:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.506+0000 7f8f79800640 1 --2- 172.21.15.28:0/681482574 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a38ae5f850 0x55a38aee95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.507+0000 7f8f79800640 1 --2- 172.21.15.28:0/681482574 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a38ae5f850 0x55a38aee95e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=626bc0a396bb06f6 server_cookie=a92823b06fe228d in_seq=0 out_seq=0 2024-04-24T08:13:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.507+0000 7f8f5b400640 10 client.25901 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.508+0000 7f8f5b400640 1 -- 172.21.15.28:0/681482574 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f8f680a16b0 con 0x55a38ae5f850 2024-04-24T08:13:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.508+0000 7f8f5b400640 10 client.25901 handle_command_reply: tid=0 2024-04-24T08:13:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.508+0000 7f8f7f19b740 1 client.25901 shutdown 2024-04-24T08:13:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.508+0000 7f8f7f19b740 1 -- 172.21.15.28:0/681482574 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5d480 msgr2=0x55a38ae597b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.508+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/681482574 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5d480 0x55a38ae597b0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f8f68099cd0 tx=0x7f8f68005750 comp rx=0 tx=0).stop 2024-04-24T08:13:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.508+0000 7f8f7f19b740 1 -- 172.21.15.28:0/681482574 shutdown_connections 2024-04-24T08:13:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.508+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/681482574 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a38ae5f850 0x55a38aee95e0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.508+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/681482574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a38ae54860 0x55a38ae59270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.508+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/681482574 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a38ae5d480 0x55a38ae597b0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.508+0000 7f8f5b400640 0 client.25901 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:13.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.508+0000 7f8f7f19b740 1 --2- 172.21.15.28:0/681482574 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a38ae50b80 0x55a38ae78cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.508+0000 7f8f7f19b740 1 -- 172.21.15.28:0/681482574 >> 172.21.15.28:0/681482574 conn(0x55a38ae5d0a0 msgr2=0x55a38ae4f390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:13.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.508+0000 7f8f7f19b740 1 -- 172.21.15.28:0/681482574 shutdown_connections 2024-04-24T08:13:13.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.509+0000 7f8f7f19b740 1 -- 172.21.15.28:0/681482574 wait complete. 2024-04-24T08:13:13.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.509+0000 7f8f7f19b740 20 client.25901 trim_cache size 0 max 16384 2024-04-24T08:13:13.511 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:13:13.511 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-24T08:13:13.511 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-24T08:13:13.511 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:13:13.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.513+0000 7f8f7c200640 1 -- 172.21.15.28:0/1362359570 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8f5c0a3e00 msgr2=0x7f8f5c0a62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:13.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.513+0000 7f8f7c200640 1 --2- 172.21.15.28:0/1362359570 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8f5c0a3e00 0x7f8f5c0a62c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f8f6002aa30 tx=0x7f8f60009680 comp rx=0 tx=0).stop 2024-04-24T08:13:13.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.513+0000 7f8f7c200640 1 -- 172.21.15.28:0/1362359570 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f74115150 msgr2=0x7f8f741a75d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:13.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.513+0000 7f8f7c200640 1 --2- 172.21.15.28:0/1362359570 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f74115150 0x7f8f741a75d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f8f6800c710 tx=0x7f8f6800cbd0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.515+0000 7f8f7c200640 1 -- 172.21.15.28:0/1362359570 shutdown_connections 2024-04-24T08:13:13.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.515+0000 7f8f7c200640 1 --2- 172.21.15.28:0/1362359570 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8f5c0a3e00 0x7f8f5c0a62c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.515+0000 7f8f7c200640 1 --2- 172.21.15.28:0/1362359570 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8f74115150 0x7f8f741a75d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.515+0000 7f8f7c200640 1 --2- 172.21.15.28:0/1362359570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f74109b30 0x7f8f741ad960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.515+0000 7f8f7c200640 1 --2- 172.21.15.28:0/1362359570 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8f74109210 0x7f8f741ad420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.515+0000 7f8f7c200640 1 -- 172.21.15.28:0/1362359570 >> 172.21.15.28:0/1362359570 conn(0x7f8f7406d110 msgr2=0x7f8f74070d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:13.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.515+0000 7f8f7c200640 1 -- 172.21.15.28:0/1362359570 shutdown_connections 2024-04-24T08:13:13.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.515+0000 7f8f7c200640 1 -- 172.21.15.28:0/1362359570 wait complete. 2024-04-24T08:13:13.544 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:13:13.544 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:13:13.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.783+0000 7f0519200640 1 -- 172.21.15.28:0/2351904691 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f051410c9b0 msgr2=0x7f0514109540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:13.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.783+0000 7f0519200640 1 --2- 172.21.15.28:0/2351904691 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f051410c9b0 0x7f0514109540 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f04fc009900 tx=0x7f04fc02a410 comp rx=0 tx=0).stop 2024-04-24T08:13:13.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.783+0000 7f0519200640 1 -- 172.21.15.28:0/2351904691 shutdown_connections 2024-04-24T08:13:13.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.783+0000 7f0519200640 1 --2- 172.21.15.28:0/2351904691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051410ee40 0x7f051410f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.783+0000 7f0519200640 1 --2- 172.21.15.28:0/2351904691 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f051410c9b0 0x7f0514109540 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.783+0000 7f0519200640 1 --2- 172.21.15.28:0/2351904691 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f051410c540 0x7f0514109000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.783+0000 7f0519200640 1 -- 172.21.15.28:0/2351904691 >> 172.21.15.28:0/2351904691 conn(0x7f051406cf10 msgr2=0x7f051406d320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:13.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.785+0000 7f0519200640 1 -- 172.21.15.28:0/2351904691 shutdown_connections 2024-04-24T08:13:13.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.785+0000 7f0519200640 1 -- 172.21.15.28:0/2351904691 wait complete. 2024-04-24T08:13:13.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.788+0000 7f0519200640 1 Processor -- start 2024-04-24T08:13:13.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.788+0000 7f0519200640 1 -- start start 2024-04-24T08:13:13.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.789+0000 7f0519200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f051410c540 0x7f05141a7560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.789+0000 7f0519200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051410c9b0 0x7f05141a7aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.792+0000 7f050aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051410c9b0 0x7f05141a7aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.792+0000 7f050aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051410c9b0 0x7f05141a7aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38532/0 (socket says 172.21.15.28:38532) 2024-04-24T08:13:13.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.792+0000 7f0513400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f051410c540 0x7f05141a7560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.792+0000 7f0513400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f051410c540 0x7f05141a7560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40850/0 (socket says 172.21.15.28:40850) 2024-04-24T08:13:13.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.792+0000 7f050aa00640 1 -- 172.21.15.28:0/3180391575 learned_addr learned my addr 172.21.15.28:0/3180391575 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:13.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.792+0000 7f0519200640 1 --2- 172.21.15.28:0/3180391575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f051410ee40 0x7f05141abe20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.793+0000 7f0519200640 1 -- 172.21.15.28:0/3180391575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0514121000 con 0x7f051410c9b0 2024-04-24T08:13:13.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.793+0000 7f0519200640 1 -- 172.21.15.28:0/3180391575 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f0514120e80 con 0x7f051410c540 2024-04-24T08:13:13.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.793+0000 7f0519200640 1 -- 172.21.15.28:0/3180391575 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f0514121180 con 0x7f051410ee40 2024-04-24T08:13:13.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.793+0000 7f0513e00640 1 --2- 172.21.15.28:0/3180391575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f051410ee40 0x7f05141abe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.793+0000 7f050aa00640 1 -- 172.21.15.28:0/3180391575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f051410ee40 msgr2=0x7f05141abe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:13.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.793+0000 7f050aa00640 1 --2- 172.21.15.28:0/3180391575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f051410ee40 0x7f05141abe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.793+0000 7f050aa00640 1 -- 172.21.15.28:0/3180391575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f051410c540 msgr2=0x7f05141a7560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:13.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.793+0000 7f050aa00640 1 --2- 172.21.15.28:0/3180391575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f051410c540 0x7f05141a7560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:13.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.794+0000 7f050aa00640 1 -- 172.21.15.28:0/3180391575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05141ac550 con 0x7f051410c9b0 2024-04-24T08:13:13.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.794+0000 7f050aa00640 1 --2- 172.21.15.28:0/3180391575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051410c9b0 0x7f05141a7aa0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f04fc02a980 tx=0x7f04fc003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:13.795 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.795+0000 7f0510c00640 1 -- 172.21.15.28:0/3180391575 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc03f070 con 0x7f051410c9b0 2024-04-24T08:13:13.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.795+0000 7f0510c00640 1 -- 172.21.15.28:0/3180391575 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f04fc031c90 con 0x7f051410c9b0 2024-04-24T08:13:13.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.795+0000 7f0510c00640 1 -- 172.21.15.28:0/3180391575 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f04fc035030 con 0x7f051410c9b0 2024-04-24T08:13:13.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.795+0000 7f0519200640 1 -- 172.21.15.28:0/3180391575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05141ac7e0 con 0x7f051410c9b0 2024-04-24T08:13:13.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.796+0000 7f0519200640 1 -- 172.21.15.28:0/3180391575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05141b40d0 con 0x7f051410c9b0 2024-04-24T08:13:13.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.798+0000 7f0509600640 1 -- 172.21.15.28:0/3180391575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04ec005290 con 0x7f051410c9b0 2024-04-24T08:13:13.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.806+0000 7f0510c00640 1 -- 172.21.15.28:0/3180391575 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f04fc042070 con 0x7f051410c9b0 2024-04-24T08:13:13.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.809+0000 7f0510c00640 1 --2- 172.21.15.28:0/3180391575 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f04f80a3de0 0x7f04f80a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:13.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.809+0000 7f0510c00640 1 -- 172.21.15.28:0/3180391575 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f04fc0f1290 con 0x7f051410c9b0 2024-04-24T08:13:13.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.810+0000 7f0513400640 1 --2- 172.21.15.28:0/3180391575 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f04f80a3de0 0x7f04f80a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:13.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.810+0000 7f0510c00640 1 -- 172.21.15.28:0/3180391575 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f04fc0bd2c0 con 0x7f051410c9b0 2024-04-24T08:13:13.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:13.811+0000 7f0513400640 1 --2- 172.21.15.28:0/3180391575 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f04f80a3de0 0x7f04f80a62a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0504009bd0 tx=0x7f0504005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:13.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:13 smithi117 ceph-mon[32438]: pgmap v89: 97 pgs: 97 active+clean; 5.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 324 MiB/s wr, 134 op/s 2024-04-24T08:13:13.901 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:13 smithi028 ceph-mon[27786]: pgmap v89: 97 pgs: 97 active+clean; 5.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 324 MiB/s wr, 134 op/s 2024-04-24T08:13:13.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:13 smithi194 ceph-mon[32836]: pgmap v89: 97 pgs: 97 active+clean; 5.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 324 MiB/s wr, 134 op/s 2024-04-24T08:13:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.204+0000 7f0509600640 1 -- 172.21.15.28:0/3180391575 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f04ec005580 con 0x7f051410c9b0 2024-04-24T08:13:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.205+0000 7f0510c00640 1 -- 172.21.15.28:0/3180391575 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11872 (secure 0 0 0) 0x7f04fc0c15c0 con 0x7f051410c9b0 2024-04-24T08:13:14.205 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:13:14.206 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:12:18.621376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24572,24574,24556]},"id":1}]} 2024-04-24T08:13:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-04-24T08:13:14.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.207+0000 7f0509600640 1 -- 172.21.15.28:0/3180391575 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f04f80a3de0 msgr2=0x7f04f80a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:14.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.207+0000 7f0509600640 1 --2- 172.21.15.28:0/3180391575 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f04f80a3de0 0x7f04f80a62a0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f0504009bd0 tx=0x7f0504005f90 comp rx=0 tx=0).stop 2024-04-24T08:13:14.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.207+0000 7f0509600640 1 -- 172.21.15.28:0/3180391575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051410c9b0 msgr2=0x7f05141a7aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:14.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.207+0000 7f0509600640 1 --2- 172.21.15.28:0/3180391575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051410c9b0 0x7f05141a7aa0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f04fc02a980 tx=0x7f04fc003b90 comp rx=0 tx=0).stop 2024-04-24T08:13:14.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.208+0000 7f0509600640 1 -- 172.21.15.28:0/3180391575 shutdown_connections 2024-04-24T08:13:14.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.208+0000 7f0509600640 1 --2- 172.21.15.28:0/3180391575 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f04f80a3de0 0x7f04f80a62a0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.208+0000 7f0509600640 1 --2- 172.21.15.28:0/3180391575 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f051410c9b0 0x7f05141a7aa0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.208+0000 7f0509600640 1 --2- 172.21.15.28:0/3180391575 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f051410ee40 0x7f05141abe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.208+0000 7f0509600640 1 --2- 172.21.15.28:0/3180391575 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f051410c540 0x7f05141a7560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.208+0000 7f0509600640 1 -- 172.21.15.28:0/3180391575 >> 172.21.15.28:0/3180391575 conn(0x7f051406cf10 msgr2=0x7f051410ad50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.208+0000 7f0509600640 1 -- 172.21.15.28:0/3180391575 shutdown_connections 2024-04-24T08:13:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.208+0000 7f0509600640 1 -- 172.21.15.28:0/3180391575 wait complete. 2024-04-24T08:13:14.228 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:13:14.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.569+0000 7f422b200640 1 -- 172.21.15.28:0/3922384970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f422c074cd0 msgr2=0x7f422c10e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:14.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.569+0000 7f422b200640 1 --2- 172.21.15.28:0/3922384970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f422c074cd0 0x7f422c10e220 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f4218009900 tx=0x7f421802a480 comp rx=0 tx=0).stop 2024-04-24T08:13:14.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.569+0000 7f422b200640 1 -- 172.21.15.28:0/3922384970 shutdown_connections 2024-04-24T08:13:14.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.569+0000 7f422b200640 1 --2- 172.21.15.28:0/3922384970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f422c10e760 0x7f422c110b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.569+0000 7f422b200640 1 --2- 172.21.15.28:0/3922384970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f422c074cd0 0x7f422c10e220 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.569+0000 7f422b200640 1 --2- 172.21.15.28:0/3922384970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f422c0743b0 0x7f422c074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.569+0000 7f422b200640 1 -- 172.21.15.28:0/3922384970 >> 172.21.15.28:0/3922384970 conn(0x7f422c0fd9d0 msgr2=0x7f422c0ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:14.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.569+0000 7f422b200640 1 -- 172.21.15.28:0/3922384970 shutdown_connections 2024-04-24T08:13:14.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.569+0000 7f422b200640 1 -- 172.21.15.28:0/3922384970 wait complete. 2024-04-24T08:13:14.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.570+0000 7f422b200640 1 Processor -- start 2024-04-24T08:13:14.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.570+0000 7f422b200640 1 -- start start 2024-04-24T08:13:14.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.571+0000 7f422b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f422c0743b0 0x7f422c1a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.571+0000 7f422b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f422c074cd0 0x7f422c1a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.571+0000 7f422b200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f422c10e760 0x7f422c19a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.571+0000 7f422b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f422c1144c0 con 0x7f422c074cd0 2024-04-24T08:13:14.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.571+0000 7f422b200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f422c114340 con 0x7f422c10e760 2024-04-24T08:13:14.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.571+0000 7f422b200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f422c114640 con 0x7f422c0743b0 2024-04-24T08:13:14.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.571+0000 7f422a800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f422c10e760 0x7f422c19a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.571+0000 7f422a800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f422c10e760 0x7f422c19a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40868/0 (socket says 172.21.15.28:40868) 2024-04-24T08:13:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.571+0000 7f422a800640 1 -- 172.21.15.28:0/442901057 learned_addr learned my addr 172.21.15.28:0/442901057 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.571+0000 7f4229e00640 1 --2- 172.21.15.28:0/442901057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f422c0743b0 0x7f422c1a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.572+0000 7f4229400640 1 --2- 172.21.15.28:0/442901057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f422c074cd0 0x7f422c1a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.572+0000 7f422a800640 1 -- 172.21.15.28:0/442901057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f422c0743b0 msgr2=0x7f422c1a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.572+0000 7f422a800640 1 --2- 172.21.15.28:0/442901057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f422c0743b0 0x7f422c1a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.572+0000 7f422a800640 1 -- 172.21.15.28:0/442901057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f422c074cd0 msgr2=0x7f422c1a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.572+0000 7f422a800640 1 --2- 172.21.15.28:0/442901057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f422c074cd0 0x7f422c1a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.572+0000 7f422a800640 1 -- 172.21.15.28:0/442901057 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f422c19afb0 con 0x7f422c10e760 2024-04-24T08:13:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.573+0000 7f422a800640 1 --2- 172.21.15.28:0/442901057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f422c10e760 0x7f422c19a7e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f422400eb10 tx=0x7f422400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.573+0000 7f4222a00640 1 -- 172.21.15.28:0/442901057 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4224004b20 con 0x7f422c10e760 2024-04-24T08:13:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.573+0000 7f4222a00640 1 -- 172.21.15.28:0/442901057 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4224004cc0 con 0x7f422c10e760 2024-04-24T08:13:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.573+0000 7f4222a00640 1 -- 172.21.15.28:0/442901057 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4224010760 con 0x7f422c10e760 2024-04-24T08:13:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.573+0000 7f422b200640 1 -- 172.21.15.28:0/442901057 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f422c19b2a0 con 0x7f422c10e760 2024-04-24T08:13:14.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.574+0000 7f422b200640 1 -- 172.21.15.28:0/442901057 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f422c101db0 con 0x7f422c10e760 2024-04-24T08:13:14.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.579+0000 7f4233883740 1 Processor -- start 2024-04-24T08:13:14.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.583+0000 7f4222a00640 1 -- 172.21.15.28:0/442901057 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4224010900 con 0x7f422c10e760 2024-04-24T08:13:14.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.584+0000 7f4233883740 1 -- start start 2024-04-24T08:13:14.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.587+0000 7f4233883740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b19fdb40 0x5576b19fdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.587+0000 7f4233883740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b19fe460 0x5576b1a085f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.588+0000 7f4233883740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b1a08b30 0x5576b1a0af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.588+0000 7f4233883740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5576b19631d0 con 0x5576b19fdb40 2024-04-24T08:13:14.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.588+0000 7f4233883740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5576b1909450 con 0x5576b19fe460 2024-04-24T08:13:14.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.588+0000 7f4233883740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5576b195f190 con 0x5576b1a08b30 2024-04-24T08:13:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.588+0000 7f4222a00640 1 --2- 172.21.15.28:0/442901057 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f42100a3da0 0x7f42100a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.588+0000 7f4222a00640 1 -- 172.21.15.28:0/442901057 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f42240d0d80 con 0x7f422c10e760 2024-04-24T08:13:14.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.589+0000 7f422a800640 1 --2- 172.21.15.28:0/442901057 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f42100a3da0 0x7f42100a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.591+0000 7f4229400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b19fdb40 0x5576b19fdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.592+0000 7f422a800640 1 --2- 172.21.15.28:0/442901057 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f42100a3da0 0x7f42100a6260 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f42240ceab0 tx=0x7f4224005b40 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.594+0000 7f4229e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b19fe460 0x5576b1a085f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.594+0000 7f4229e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b19fe460 0x5576b1a085f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40874/0 (socket says 172.21.15.28:40874) 2024-04-24T08:13:14.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.594+0000 7f4229e00640 1 -- 172.21.15.28:0/1156492568 learned_addr learned my addr 172.21.15.28:0/1156492568 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.594+0000 7f4229e00640 1 -- 172.21.15.28:0/1156492568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b1a08b30 msgr2=0x5576b1a0af20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:13:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.594+0000 7f4229e00640 1 --2- 172.21.15.28:0/1156492568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b1a08b30 0x5576b1a0af20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.594+0000 7f4229e00640 1 -- 172.21.15.28:0/1156492568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b19fdb40 msgr2=0x5576b19fdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.594+0000 7f4229e00640 1 --2- 172.21.15.28:0/1156492568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b19fdb40 0x5576b19fdf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.595+0000 7f4229e00640 1 -- 172.21.15.28:0/1156492568 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576b19c6310 con 0x5576b19fe460 2024-04-24T08:13:14.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.595+0000 7f4229e00640 1 --2- 172.21.15.28:0/1156492568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b19fe460 0x5576b1a085f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f421c009cf0 tx=0x7f421c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49418203b278b140 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.596+0000 7f4217000640 1 -- 172.21.15.28:0/1156492568 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c004230 con 0x5576b19fe460 2024-04-24T08:13:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.596+0000 7f4217000640 1 -- 172.21.15.28:0/1156492568 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f421c0043d0 con 0x5576b19fe460 2024-04-24T08:13:14.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.596+0000 7f4217000640 1 -- 172.21.15.28:0/1156492568 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c004230 con 0x5576b19fe460 2024-04-24T08:13:14.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.597+0000 7f4233883740 1 -- 172.21.15.28:0/1156492568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b19fe460 msgr2=0x5576b1a085f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:14.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.597+0000 7f4233883740 1 --2- 172.21.15.28:0/1156492568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b19fe460 0x5576b1a085f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f421c009cf0 tx=0x7f421c011810 comp rx=0 tx=0).stop 2024-04-24T08:13:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.599+0000 7f4233883740 1 -- 172.21.15.28:0/1156492568 shutdown_connections 2024-04-24T08:13:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.599+0000 7f4233883740 1 --2- 172.21.15.28:0/1156492568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b1a08b30 0x5576b1a0af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.599+0000 7f4233883740 1 --2- 172.21.15.28:0/1156492568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b19fe460 0x5576b1a085f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.599+0000 7f4233883740 1 --2- 172.21.15.28:0/1156492568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b19fdb40 0x5576b19fdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.599+0000 7f4233883740 1 -- 172.21.15.28:0/1156492568 >> 172.21.15.28:0/1156492568 conn(0x5576b19f9e60 msgr2=0x5576b19fa240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.600+0000 7f4233883740 1 -- 172.21.15.28:0/1156492568 shutdown_connections 2024-04-24T08:13:14.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.601+0000 7f4233883740 1 -- 172.21.15.28:0/1156492568 wait complete. 2024-04-24T08:13:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.601+0000 7f4233883740 1 -- start start 2024-04-24T08:13:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.602+0000 7f4233883740 1 Processor -- start 2024-04-24T08:13:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.603+0000 7f4233883740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b19fdb40 0x5576b1a08580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.603+0000 7f4233883740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fe460 0x5576b1a05870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.603+0000 7f4233883740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b1a08b30 0x5576b1a05fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.603+0000 7f4233883740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5576b1a0ba10 con 0x5576b1a08b30 2024-04-24T08:13:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.603+0000 7f4233883740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5576b191c810 con 0x5576b19fdb40 2024-04-24T08:13:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.603+0000 7f4233883740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5576b1a0bb90 con 0x5576b19fe460 2024-04-24T08:13:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.604+0000 7f4229e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fe460 0x5576b1a05870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.604+0000 7f4229e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fe460 0x5576b1a05870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:56918/0 (socket says 172.21.15.28:56918) 2024-04-24T08:13:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.604+0000 7f4229e00640 1 -- 172.21.15.28:0/2561916667 learned_addr learned my addr 172.21.15.28:0/2561916667 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.606+0000 7f4229e00640 1 -- 172.21.15.28:0/2561916667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b19fdb40 msgr2=0x5576b1a08580 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:13:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.606+0000 7f4229e00640 1 --2- 172.21.15.28:0/2561916667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b19fdb40 0x5576b1a08580 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.606+0000 7f4229e00640 1 -- 172.21.15.28:0/2561916667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b1a08b30 msgr2=0x5576b1a05fb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:13:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.606+0000 7f4229e00640 1 --2- 172.21.15.28:0/2561916667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b1a08b30 0x5576b1a05fb0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.606+0000 7f4229e00640 1 -- 172.21.15.28:0/2561916667 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576b192d3f0 con 0x5576b19fe460 2024-04-24T08:13:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.607+0000 7f4229e00640 1 --2- 172.21.15.28:0/2561916667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fe460 0x5576b1a05870 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f421c0058c0 tx=0x7f421c003ec0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3a118403bf9ef405 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.607+0000 7f4233883740 10 client.0 _ll_register_callbacks cb 0x5576b18ea140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:14.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.607+0000 7f4233883740 1 -- 172.21.15.28:0/2561916667 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f421c0043d0 con 0x5576b19fe460 2024-04-24T08:13:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.607+0000 7f420b400640 10 client.25957 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:13:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.607+0000 7f420b400640 1 -- 172.21.15.28:0/2561916667 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c053050 con 0x5576b19fe460 2024-04-24T08:13:14.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.607+0000 7f420b400640 1 -- 172.21.15.28:0/2561916667 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f421c0035f0 con 0x5576b19fe460 2024-04-24T08:13:14.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.628+0000 7f420b400640 1 -- 172.21.15.28:0/2561916667 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f421c049070 con 0x5576b19fe460 2024-04-24T08:13:14.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.628+0000 7f4233883740 10 client.25957 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:14.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.628+0000 7f4233883740 1 -- 172.21.15.28:0/2561916667 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({fsmap=25}) -- 0x5576b16753f0 con 0x5576b19fe460 2024-04-24T08:13:14.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.628+0000 7f420b400640 1 -- 172.21.15.28:0/2561916667 <== mon.1 v2:172.21.15.194:3300/0 4 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f421c0037f0 con 0x5576b19fe460 2024-04-24T08:13:14.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.628+0000 7f4233883740 10 client.25957 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:14.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.628+0000 7f4233883740 10 client.25957 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:13:14.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.628+0000 7f4233883740 20 client.25957 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:14.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.628+0000 7f4233883740 1 --2- 172.21.15.28:0/2561916667 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5576b1a0faf0 0x5576b1a11f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.628+0000 7f4233883740 4 client.25957 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:13:14.629 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.628+0000 7f4233883740 1 -- 172.21.15.28:0/2561916667 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576b1970bf0 con 0x5576b1a0faf0 2024-04-24T08:13:14.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.629+0000 7f4229400640 1 --2- 172.21.15.28:0/2561916667 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5576b1a0faf0 0x5576b1a11f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.631+0000 7f4229400640 1 --2- 172.21.15.28:0/2561916667 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5576b1a0faf0 0x5576b1a11f00 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c03ce25939704e67 server_cookie=d3e3a872bf04f414 in_seq=0 out_seq=0 2024-04-24T08:13:14.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.631+0000 7f420b400640 10 client.25957 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:13:14.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.631+0000 7f420b400640 1 -- 172.21.15.28:0/2561916667 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5576b1970bf0 con 0x5576b1a0faf0 2024-04-24T08:13:14.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.631+0000 7f420b400640 10 client.25957 handle_command_reply: tid=0 2024-04-24T08:13:14.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.632+0000 7f4233883740 1 client.25957 shutdown 2024-04-24T08:13:14.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.632+0000 7f4233883740 1 -- 172.21.15.28:0/2561916667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fe460 msgr2=0x5576b1a05870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:14.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.632+0000 7f4233883740 1 --2- 172.21.15.28:0/2561916667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fe460 0x5576b1a05870 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f421c0058c0 tx=0x7f421c003ec0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.632+0000 7f4233883740 1 -- 172.21.15.28:0/2561916667 shutdown_connections 2024-04-24T08:13:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.632+0000 7f4233883740 1 --2- 172.21.15.28:0/2561916667 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5576b1a0faf0 0x5576b1a11f00 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.632+0000 7f4233883740 1 --2- 172.21.15.28:0/2561916667 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b1a08b30 0x5576b1a05fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.632+0000 7f4233883740 1 --2- 172.21.15.28:0/2561916667 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fe460 0x5576b1a05870 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.632+0000 7f4233883740 1 --2- 172.21.15.28:0/2561916667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b19fdb40 0x5576b1a08580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.632+0000 7f4233883740 1 -- 172.21.15.28:0/2561916667 >> 172.21.15.28:0/2561916667 conn(0x5576b19f9e60 msgr2=0x5576b19ffc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:14.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.633+0000 7f420b400640 0 client.25957 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:13:14.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.636+0000 7f4233883740 1 -- 172.21.15.28:0/2561916667 shutdown_connections 2024-04-24T08:13:14.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.636+0000 7f4233883740 1 -- 172.21.15.28:0/2561916667 wait complete. 2024-04-24T08:13:14.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.636+0000 7f4233883740 20 client.25957 trim_cache size 0 max 16384 2024-04-24T08:13:14.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.668+0000 7f4233883740 1 Processor -- start 2024-04-24T08:13:14.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.669+0000 7f4233883740 1 -- start start 2024-04-24T08:13:14.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.669+0000 7f4233883740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fdb40 0x5576b19f8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.669+0000 7f4233883740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b19fe460 0x5576b1a080b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.669+0000 7f4233883740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b1a08b30 0x5576b1a25160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.669+0000 7f4233883740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f422c0729e0 con 0x5576b1a08b30 2024-04-24T08:13:14.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.669+0000 7f4233883740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f422c0702b0 con 0x5576b19fe460 2024-04-24T08:13:14.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.669+0000 7f4233883740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f422c06e100 con 0x5576b19fdb40 2024-04-24T08:13:14.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.670+0000 7f4229400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fdb40 0x5576b19f8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.670+0000 7f4229400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fdb40 0x5576b19f8dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:56932/0 (socket says 172.21.15.28:56932) 2024-04-24T08:13:14.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.670+0000 7f4229400640 1 -- 172.21.15.28:0/57737544 learned_addr learned my addr 172.21.15.28:0/57737544 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:14.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.670+0000 7f4229400640 1 --2- 172.21.15.28:0/57737544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b1a08b30 0x5576b1a25160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.670+0000 7f4229e00640 1 --2- 172.21.15.28:0/57737544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b19fe460 0x5576b1a080b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.671+0000 7f4229400640 1 -- 172.21.15.28:0/57737544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b19fe460 msgr2=0x5576b1a080b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:14.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.671+0000 7f4229400640 1 --2- 172.21.15.28:0/57737544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b19fe460 0x5576b1a080b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.671+0000 7f4229400640 1 -- 172.21.15.28:0/57737544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b1a08b30 msgr2=0x5576b1a25160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:14.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.671+0000 7f4229400640 1 --2- 172.21.15.28:0/57737544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b1a08b30 0x5576b1a25160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.672+0000 7f4229400640 1 -- 172.21.15.28:0/57737544 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576b1a0c520 con 0x5576b19fdb40 2024-04-24T08:13:14.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.673+0000 7f4229400640 1 --2- 172.21.15.28:0/57737544 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fdb40 0x5576b19f8dd0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4218009850 tx=0x7f4218009680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4f4cb4592ca1965 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:14.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.674+0000 7f4216600640 1 -- 172.21.15.28:0/57737544 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4218035040 con 0x5576b19fdb40 2024-04-24T08:13:14.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.674+0000 7f4216600640 1 -- 172.21.15.28:0/57737544 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4218005050 con 0x5576b19fdb40 2024-04-24T08:13:14.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.674+0000 7f4216600640 1 -- 172.21.15.28:0/57737544 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4218004c20 con 0x5576b19fdb40 2024-04-24T08:13:14.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.676+0000 7f4233883740 1 -- 172.21.15.28:0/57737544 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fdb40 msgr2=0x5576b19f8dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:14.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.676+0000 7f4233883740 1 --2- 172.21.15.28:0/57737544 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fdb40 0x5576b19f8dd0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f4218009850 tx=0x7f4218009680 comp rx=0 tx=0).stop 2024-04-24T08:13:14.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.677+0000 7f4233883740 1 -- 172.21.15.28:0/57737544 shutdown_connections 2024-04-24T08:13:14.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.677+0000 7f4233883740 1 --2- 172.21.15.28:0/57737544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b1a08b30 0x5576b1a25160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.677+0000 7f4233883740 1 --2- 172.21.15.28:0/57737544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b19fe460 0x5576b1a080b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.677+0000 7f4233883740 1 --2- 172.21.15.28:0/57737544 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fdb40 0x5576b19f8dd0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.677+0000 7f4233883740 1 -- 172.21.15.28:0/57737544 >> 172.21.15.28:0/57737544 conn(0x5576b19f9e60 msgr2=0x5576b19f86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:14.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.678+0000 7f4233883740 1 -- 172.21.15.28:0/57737544 shutdown_connections 2024-04-24T08:13:14.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.678+0000 7f4233883740 1 -- 172.21.15.28:0/57737544 wait complete. 2024-04-24T08:13:14.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.679+0000 7f4233883740 1 -- start start 2024-04-24T08:13:14.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.679+0000 7f4233883740 1 Processor -- start 2024-04-24T08:13:14.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.680+0000 7f4233883740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b19fdb40 0x5576b1a17b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.681+0000 7f4233883740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fe460 0x5576b1a02bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.681+0000 7f4233883740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b1a08b30 0x5576b1a032c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.681+0000 7f4233883740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f422c06e770 con 0x5576b19fdb40 2024-04-24T08:13:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.681+0000 7f4233883740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f422c06fbe0 con 0x5576b1a08b30 2024-04-24T08:13:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.681+0000 7f4233883740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f422c06ee40 con 0x5576b19fe460 2024-04-24T08:13:14.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.681+0000 7f4233883740 10 client.0 _ll_register_callbacks cb 0x7f421c009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:14.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.682+0000 7f4229e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fe460 0x5576b1a02bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.682+0000 7f4229400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b19fdb40 0x5576b1a17b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.683+0000 7f4229e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fe460 0x5576b1a02bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:56936/0 (socket says 172.21.15.28:56936) 2024-04-24T08:13:14.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.683+0000 7f4229e00640 1 -- 172.21.15.28:0/1184557122 learned_addr learned my addr 172.21.15.28:0/1184557122 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:14.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.683+0000 7f4229e00640 1 -- 172.21.15.28:0/1184557122 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b1a08b30 msgr2=0x5576b1a032c0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:13:14.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.683+0000 7f4229e00640 1 --2- 172.21.15.28:0/1184557122 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b1a08b30 0x5576b1a032c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.683+0000 7f4229e00640 1 -- 172.21.15.28:0/1184557122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b19fdb40 msgr2=0x5576b1a17b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:14.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.683+0000 7f4229e00640 1 --2- 172.21.15.28:0/1184557122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b19fdb40 0x5576b1a17b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.683+0000 7f4229e00640 1 -- 172.21.15.28:0/1184557122 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576b173f7d0 con 0x5576b19fe460 2024-04-24T08:13:14.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.684+0000 7f4229400640 1 --2- 172.21.15.28:0/1184557122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b19fdb40 0x5576b1a17b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:14.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.685+0000 7f420b400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:13:14.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.685+0000 7f4229e00640 1 --2- 172.21.15.28:0/1184557122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fe460 0x5576b1a02bd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x5576b1a050d0 tx=0x7f421c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8e9d680fc5fc556 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:14.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.686+0000 7f420b400640 1 -- 172.21.15.28:0/1184557122 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f421c05c070 con 0x5576b19fe460 2024-04-24T08:13:14.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.687+0000 7f420b400640 1 -- 172.21.15.28:0/1184557122 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f421c004e10 con 0x5576b19fe460 2024-04-24T08:13:14.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.687+0000 7f4233883740 1 -- 172.21.15.28:0/1184557122 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4218005050 con 0x5576b19fe460 2024-04-24T08:13:14.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.687+0000 7f420b400640 1 -- 172.21.15.28:0/1184557122 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f421c066050 con 0x5576b19fe460 2024-04-24T08:13:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.688+0000 7f4233883740 10 client.25963 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.688+0000 7f4233883740 1 -- 172.21.15.28:0/1184557122 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5576b1a7ec90 con 0x5576b19fe460 2024-04-24T08:13:14.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.688+0000 7f420b400640 1 -- 172.21.15.28:0/1184557122 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4218005050 con 0x5576b19fe460 2024-04-24T08:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.688+0000 7f420b400640 1 -- 172.21.15.28:0/1184557122 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f421c045950 con 0x5576b19fe460 2024-04-24T08:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.689+0000 7f4233883740 10 client.25963 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.689+0000 7f4233883740 10 client.25963 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.689+0000 7f4233883740 20 client.25963 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.689+0000 7f4233883740 1 --2- 172.21.15.28:0/1184557122 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5576b1a7f3e0 0x5576b1a81820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.689+0000 7f4233883740 4 client.25963 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:13:14.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.689+0000 7f4233883740 1 -- 172.21.15.28:0/1184557122 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5576b196aa60 con 0x5576b1a7f3e0 2024-04-24T08:13:14.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.690+0000 7f4229400640 1 --2- 172.21.15.28:0/1184557122 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5576b1a7f3e0 0x5576b1a81820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:14.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.691+0000 7f4229400640 1 --2- 172.21.15.28:0/1184557122 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5576b1a7f3e0 0x5576b1a81820 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c65b7615a3e07eab server_cookie=a6e49358b57b9976 in_seq=0 out_seq=0 2024-04-24T08:13:14.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.691+0000 7f420b400640 10 client.25963 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:13:14.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.692+0000 7f420b400640 1 -- 172.21.15.28:0/1184557122 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5576b196aa60 con 0x5576b1a7f3e0 2024-04-24T08:13:14.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.692+0000 7f420b400640 10 client.25963 handle_command_reply: tid=0 2024-04-24T08:13:14.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.693+0000 7f4233883740 1 client.25963 shutdown 2024-04-24T08:13:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.694+0000 7f4233883740 1 -- 172.21.15.28:0/1184557122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fe460 msgr2=0x5576b1a02bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.694+0000 7f4233883740 1 --2- 172.21.15.28:0/1184557122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fe460 0x5576b1a02bd0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x5576b1a050d0 tx=0x7f421c011810 comp rx=0 tx=0).stop 2024-04-24T08:13:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.694+0000 7f4233883740 1 -- 172.21.15.28:0/1184557122 shutdown_connections 2024-04-24T08:13:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.694+0000 7f4233883740 1 --2- 172.21.15.28:0/1184557122 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5576b1a7f3e0 0x5576b1a81820 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.694+0000 7f4233883740 1 --2- 172.21.15.28:0/1184557122 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5576b1a08b30 0x5576b1a032c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.694+0000 7f420b400640 0 client.25963 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:13:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.694+0000 7f4233883740 1 --2- 172.21.15.28:0/1184557122 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5576b19fe460 0x5576b1a02bd0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.694+0000 7f4233883740 1 --2- 172.21.15.28:0/1184557122 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5576b19fdb40 0x5576b1a17b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.694+0000 7f4233883740 1 -- 172.21.15.28:0/1184557122 >> 172.21.15.28:0/1184557122 conn(0x5576b19f9e60 msgr2=0x5576b19f8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.695+0000 7f4233883740 1 -- 172.21.15.28:0/1184557122 shutdown_connections 2024-04-24T08:13:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.695+0000 7f4233883740 1 -- 172.21.15.28:0/1184557122 wait complete. 2024-04-24T08:13:14.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.695+0000 7f4233883740 20 client.25963 trim_cache size 0 max 16384 2024-04-24T08:13:14.696 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:13:14.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.700+0000 7f422b200640 1 -- 172.21.15.28:0/442901057 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f42100a3da0 msgr2=0x7f42100a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:14.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.700+0000 7f422b200640 1 --2- 172.21.15.28:0/442901057 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f42100a3da0 0x7f42100a6260 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f42240ceab0 tx=0x7f4224005b40 comp rx=0 tx=0).stop 2024-04-24T08:13:14.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.700+0000 7f422b200640 1 -- 172.21.15.28:0/442901057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f422c10e760 msgr2=0x7f422c19a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:14.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.700+0000 7f422b200640 1 --2- 172.21.15.28:0/442901057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f422c10e760 0x7f422c19a7e0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f422400eb10 tx=0x7f422400c2d0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.701+0000 7f422b200640 1 -- 172.21.15.28:0/442901057 shutdown_connections 2024-04-24T08:13:14.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.701+0000 7f422b200640 1 --2- 172.21.15.28:0/442901057 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f42100a3da0 0x7f42100a6260 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.701+0000 7f422b200640 1 --2- 172.21.15.28:0/442901057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f422c074cd0 0x7f422c1a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.701+0000 7f422b200640 1 --2- 172.21.15.28:0/442901057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f422c10e760 0x7f422c19a7e0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.701+0000 7f422b200640 1 --2- 172.21.15.28:0/442901057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f422c0743b0 0x7f422c1a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:14.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.701+0000 7f422b200640 1 -- 172.21.15.28:0/442901057 >> 172.21.15.28:0/442901057 conn(0x7f422c0fd9d0 msgr2=0x7f422c0fe740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:14.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.701+0000 7f422b200640 1 -- 172.21.15.28:0/442901057 shutdown_connections 2024-04-24T08:13:14.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:14.701+0000 7f422b200640 1 -- 172.21.15.28:0/442901057 wait complete. 2024-04-24T08:13:14.730 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:13:14.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:14 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3180391575' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:13:15.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:14 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3180391575' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:13:15.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:14 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3180391575' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:13:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.136+0000 7f36f8800640 1 -- 172.21.15.28:0/998405069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36f0076530 msgr2=0x7f36f0076910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.136+0000 7f36f8800640 1 --2- 172.21.15.28:0/998405069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36f0076530 0x7f36f0076910 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f36ec009cd0 tx=0x7f36ec02a930 comp rx=0 tx=0).stop 2024-04-24T08:13:15.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.139+0000 7f36f8800640 1 -- 172.21.15.28:0/998405069 shutdown_connections 2024-04-24T08:13:15.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.139+0000 7f36f8800640 1 --2- 172.21.15.28:0/998405069 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f0076e50 0x7f36f0112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.139+0000 7f36f8800640 1 --2- 172.21.15.28:0/998405069 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f36f0112d80 0x7f36f0115170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.139+0000 7f36f8800640 1 --2- 172.21.15.28:0/998405069 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36f0076530 0x7f36f0076910 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.139+0000 7f36f8800640 1 -- 172.21.15.28:0/998405069 >> 172.21.15.28:0/998405069 conn(0x7f36f00ffb70 msgr2=0x7f36f0101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:15.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.141+0000 7f36f8800640 1 -- 172.21.15.28:0/998405069 shutdown_connections 2024-04-24T08:13:15.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.141+0000 7f36f8800640 1 -- 172.21.15.28:0/998405069 wait complete. 2024-04-24T08:13:15.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.143+0000 7f36f8800640 1 Processor -- start 2024-04-24T08:13:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.143+0000 7f36f8800640 1 -- start start 2024-04-24T08:13:15.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.144+0000 7f36f8800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f36f0076530 0x7f36f019ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.144+0000 7f36f8800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f0076e50 0x7f36f019f3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.144+0000 7f36f8800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36f0112d80 0x7f36f01a36d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.144+0000 7f36f8800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f36f0118920 con 0x7f36f0076e50 2024-04-24T08:13:15.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.144+0000 7f36f8800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f36f01187a0 con 0x7f36f0112d80 2024-04-24T08:13:15.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.144+0000 7f36f8800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f36f0118aa0 con 0x7f36f0076530 2024-04-24T08:13:15.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.145+0000 7f36f5e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f36f0076530 0x7f36f019ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.145+0000 7f36f5e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f36f0076530 0x7f36f019ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:56954/0 (socket says 172.21.15.28:56954) 2024-04-24T08:13:15.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.145+0000 7f36f5e00640 1 -- 172.21.15.28:0/615997437 learned_addr learned my addr 172.21.15.28:0/615997437 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:15.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.145+0000 7f36f6800640 1 --2- 172.21.15.28:0/615997437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36f0112d80 0x7f36f01a36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.146+0000 7f36f5e00640 1 -- 172.21.15.28:0/615997437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36f0112d80 msgr2=0x7f36f01a36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.146+0000 7f36f5e00640 1 --2- 172.21.15.28:0/615997437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36f0112d80 0x7f36f01a36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.146+0000 7f36f5e00640 1 -- 172.21.15.28:0/615997437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f0076e50 msgr2=0x7f36f019f3a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:13:15.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.146+0000 7f36f5e00640 1 --2- 172.21.15.28:0/615997437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f0076e50 0x7f36f019f3a0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.146+0000 7f36f5e00640 1 -- 172.21.15.28:0/615997437 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f36f01a3e00 con 0x7f36f0076530 2024-04-24T08:13:15.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.147+0000 7f36f5e00640 1 --2- 172.21.15.28:0/615997437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f36f0076530 0x7f36f019ee60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f36ec005e30 tx=0x7f36ec031870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.148+0000 7f36eaa00640 1 -- 172.21.15.28:0/615997437 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec003b70 con 0x7f36f0076530 2024-04-24T08:13:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.148+0000 7f36eaa00640 1 -- 172.21.15.28:0/615997437 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36ec02d070 con 0x7f36f0076530 2024-04-24T08:13:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.148+0000 7f36eaa00640 1 -- 172.21.15.28:0/615997437 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36ec02f930 con 0x7f36f0076530 2024-04-24T08:13:15.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.149+0000 7f36f8800640 1 -- 172.21.15.28:0/615997437 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f36f01a4090 con 0x7f36f0076530 2024-04-24T08:13:15.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.150+0000 7f36f8800640 1 -- 172.21.15.28:0/615997437 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f36f01ab980 con 0x7f36f0076530 2024-04-24T08:13:15.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.155+0000 7f36eaa00640 1 -- 172.21.15.28:0/615997437 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f36ec02fad0 con 0x7f36f0076530 2024-04-24T08:13:15.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.160+0000 7f36fb61e740 1 Processor -- start 2024-04-24T08:13:15.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.161+0000 7f36fb61e740 1 -- start start 2024-04-24T08:13:15.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.161+0000 7f36fb61e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622c4b40 0x5585622c4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.161+0000 7f36fb61e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c5460 0x5585622cf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.161+0000 7f36fb61e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622cfb30 0x5585622d1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.161+0000 7f36fb61e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55856222a1d0 con 0x5585622cfb30 2024-04-24T08:13:15.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.161+0000 7f36fb61e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5585621d0450 con 0x5585622c5460 2024-04-24T08:13:15.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.161+0000 7f36fb61e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558562226190 con 0x5585622c4b40 2024-04-24T08:13:15.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.163+0000 7f36f5e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c5460 0x5585622cf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.163+0000 7f36f5e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c5460 0x5585622cf5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40920/0 (socket says 172.21.15.28:40920) 2024-04-24T08:13:15.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.163+0000 7f36f6800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622c4b40 0x5585622c4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.164+0000 7f36f6800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622c4b40 0x5585622c4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:56968/0 (socket says 172.21.15.28:56968) 2024-04-24T08:13:15.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.164+0000 7f36f6800640 1 -- 172.21.15.28:0/1171323339 learned_addr learned my addr 172.21.15.28:0/1171323339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:15.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.164+0000 7f36f6800640 1 -- 172.21.15.28:0/1171323339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c5460 msgr2=0x5585622cf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.164+0000 7f36f6800640 1 --2- 172.21.15.28:0/1171323339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c5460 0x5585622cf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.164+0000 7f36f6800640 1 -- 172.21.15.28:0/1171323339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622cfb30 msgr2=0x5585622d1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.165+0000 7f36eaa00640 1 --2- 172.21.15.28:0/615997437 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f36d80a3da0 0x7f36d80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.165+0000 7f36eaa00640 1 -- 172.21.15.28:0/615997437 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f36ec0f1b40 con 0x7f36f0076530 2024-04-24T08:13:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.166+0000 7f36f5400640 1 --2- 172.21.15.28:0/1171323339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622cfb30 0x5585622d1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.167+0000 7f36f6800640 1 --2- 172.21.15.28:0/1171323339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622cfb30 0x5585622d1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.167+0000 7f36f6800640 1 -- 172.21.15.28:0/1171323339 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55856228d310 con 0x5585622c4b40 2024-04-24T08:13:15.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.167+0000 7f36f5400640 1 --2- 172.21.15.28:0/1171323339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622cfb30 0x5585622d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:15.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.168+0000 7f36f6800640 1 --2- 172.21.15.28:0/1171323339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622c4b40 0x5585622c4f20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f36e4009ba0 tx=0x7f36e4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=96026dea1af4e90e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.168+0000 7f36f5e00640 1 --2- 172.21.15.28:0/615997437 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f36d80a3da0 0x7f36d80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.168+0000 7f36df000640 1 -- 172.21.15.28:0/1171323339 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e4004230 con 0x5585622c4b40 2024-04-24T08:13:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.168+0000 7f36df000640 1 -- 172.21.15.28:0/1171323339 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36e4009e10 con 0x5585622c4b40 2024-04-24T08:13:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.169+0000 7f36fb61e740 1 -- 172.21.15.28:0/1171323339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622c4b40 msgr2=0x5585622c4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.169+0000 7f36df000640 1 -- 172.21.15.28:0/1171323339 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e4004230 con 0x5585622c4b40 2024-04-24T08:13:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.169+0000 7f36fb61e740 1 --2- 172.21.15.28:0/1171323339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622c4b40 0x5585622c4f20 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f36e4009ba0 tx=0x7f36e4011810 comp rx=0 tx=0).stop 2024-04-24T08:13:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.169+0000 7f36fb61e740 1 -- 172.21.15.28:0/1171323339 shutdown_connections 2024-04-24T08:13:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.169+0000 7f36fb61e740 1 --2- 172.21.15.28:0/1171323339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622cfb30 0x5585622d1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.169+0000 7f36fb61e740 1 --2- 172.21.15.28:0/1171323339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c5460 0x5585622cf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.169+0000 7f36fb61e740 1 --2- 172.21.15.28:0/1171323339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622c4b40 0x5585622c4f20 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.169+0000 7f36fb61e740 1 -- 172.21.15.28:0/1171323339 >> 172.21.15.28:0/1171323339 conn(0x5585622c0e60 msgr2=0x5585622c1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.170+0000 7f36fb61e740 1 -- 172.21.15.28:0/1171323339 shutdown_connections 2024-04-24T08:13:15.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.170+0000 7f36fb61e740 1 -- 172.21.15.28:0/1171323339 wait complete. 2024-04-24T08:13:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.170+0000 7f36fb61e740 1 -- start start 2024-04-24T08:13:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.171+0000 7f36fb61e740 1 Processor -- start 2024-04-24T08:13:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.171+0000 7f36fb61e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622c4b40 0x5585622cf510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.172+0000 7f36f6800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622c4b40 0x5585622cf510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.172+0000 7f36f6800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622c4b40 0x5585622cf510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:56984/0 (socket says 172.21.15.28:56984) 2024-04-24T08:13:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.172+0000 7f36f6800640 1 -- 172.21.15.28:0/818583448 learned_addr learned my addr 172.21.15.28:0/818583448 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.172+0000 7f36fb61e740 1 --2- 172.21.15.28:0/818583448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c5460 0x5585622cc800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.173+0000 7f36f5400640 1 --2- 172.21.15.28:0/818583448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c5460 0x5585622cc800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.173+0000 7f36fb61e740 1 --2- 172.21.15.28:0/818583448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622cfb30 0x5585622cce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.173+0000 7f36fb61e740 1 -- 172.21.15.28:0/818583448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5585622e3a20 con 0x5585622c5460 2024-04-24T08:13:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.173+0000 7f36fb61e740 1 -- 172.21.15.28:0/818583448 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5585621e3810 con 0x5585622cfb30 2024-04-24T08:13:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.173+0000 7f36fb61e740 1 -- 172.21.15.28:0/818583448 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5585622e3ba0 con 0x5585622c4b40 2024-04-24T08:13:15.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.173+0000 7f36fb61e740 10 client.0 _ll_register_callbacks cb 0x5585621b1140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:15.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.175+0000 7f36f5400640 1 -- 172.21.15.28:0/818583448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622c4b40 msgr2=0x5585622cf510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:15.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.175+0000 7f36f5400640 1 --2- 172.21.15.28:0/818583448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622c4b40 0x5585622cf510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.175+0000 7f36f5400640 1 -- 172.21.15.28:0/818583448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622cfb30 msgr2=0x5585622cce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:15.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.175+0000 7f36f6800640 1 --2- 172.21.15.28:0/818583448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622cfb30 0x5585622cce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.176+0000 7f36f5400640 1 --2- 172.21.15.28:0/818583448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622cfb30 0x5585622cce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.176+0000 7f36f5400640 1 -- 172.21.15.28:0/818583448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585621f43f0 con 0x5585622c5460 2024-04-24T08:13:15.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.176+0000 7f36f5400640 1 --2- 172.21.15.28:0/818583448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c5460 0x5585622cc800 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f36e00058c0 tx=0x7f36e000d850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ae9f54060223996 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:15.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.177+0000 7f36d3400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:13:15.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.177+0000 7f36f5e00640 1 --2- 172.21.15.28:0/615997437 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f36d80a3da0 0x7f36d80a6260 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x5585622c6100 tx=0x7f36ec0f55f0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:15.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.179+0000 7f36d3400640 1 -- 172.21.15.28:0/818583448 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e0004220 con 0x5585622c5460 2024-04-24T08:13:15.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.179+0000 7f36d3400640 1 -- 172.21.15.28:0/818583448 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f36e00043c0 con 0x5585622c5460 2024-04-24T08:13:15.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.182+0000 7f36fb61e740 1 -- 172.21.15.28:0/818583448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36e4009e10 con 0x5585622c5460 2024-04-24T08:13:15.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.184+0000 7f36d3400640 1 -- 172.21.15.28:0/818583448 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f36e000f6f0 con 0x5585622c5460 2024-04-24T08:13:15.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.184+0000 7f36fb61e740 10 client.15807 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:15.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.184+0000 7f36fb61e740 1 -- 172.21.15.28:0/818583448 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558561f3c3f0 con 0x5585622c5460 2024-04-24T08:13:15.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.185+0000 7f36d3400640 1 -- 172.21.15.28:0/818583448 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36e4009e10 con 0x5585622c5460 2024-04-24T08:13:15.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.186+0000 7f36d3400640 1 -- 172.21.15.28:0/818583448 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f36e0003c40 con 0x5585622c5460 2024-04-24T08:13:15.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.186+0000 7f36fb61e740 10 client.15807 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:15.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.187+0000 7f36fb61e740 10 client.15807 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:13:15.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.187+0000 7f36fb61e740 20 client.15807 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:15.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.187+0000 7f36fb61e740 1 --2- 172.21.15.28:0/818583448 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5585622f0b00 0x5585622f2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.187+0000 7f36fb61e740 4 client.15807 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:13:15.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.187+0000 7f36fb61e740 1 -- 172.21.15.28:0/818583448 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558562237bf0 con 0x5585622f0b00 2024-04-24T08:13:15.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.188+0000 7f36f6800640 1 --2- 172.21.15.28:0/818583448 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5585622f0b00 0x5585622f2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.188+0000 7f36d3400640 10 client.15807 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:13:15.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.191+0000 7f36f6800640 1 --2- 172.21.15.28:0/818583448 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5585622f0b00 0x5585622f2f20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b1ebb11689c277b2 server_cookie=be951e49e59d7012 in_seq=0 out_seq=0 2024-04-24T08:13:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.192+0000 7f36d3400640 1 -- 172.21.15.28:0/818583448 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558562237bf0 con 0x5585622f0b00 2024-04-24T08:13:15.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.192+0000 7f36d3400640 10 client.15807 handle_command_reply: tid=0 2024-04-24T08:13:15.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.192+0000 7f36fb61e740 1 client.15807 shutdown 2024-04-24T08:13:15.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.194+0000 7f36fb61e740 1 -- 172.21.15.28:0/818583448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c5460 msgr2=0x5585622cc800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.194+0000 7f36fb61e740 1 --2- 172.21.15.28:0/818583448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c5460 0x5585622cc800 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f36e00058c0 tx=0x7f36e000d850 comp rx=0 tx=0).stop 2024-04-24T08:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.194+0000 7f36fb61e740 1 -- 172.21.15.28:0/818583448 shutdown_connections 2024-04-24T08:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.194+0000 7f36fb61e740 1 --2- 172.21.15.28:0/818583448 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5585622f0b00 0x5585622f2f20 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.194+0000 7f36fb61e740 1 --2- 172.21.15.28:0/818583448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c5460 0x5585622cc800 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.194+0000 7f36d3400640 0 client.15807 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.194+0000 7f36fb61e740 1 --2- 172.21.15.28:0/818583448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622cfb30 0x5585622cce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.194+0000 7f36fb61e740 1 --2- 172.21.15.28:0/818583448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622c4b40 0x5585622cf510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.195+0000 7f36fb61e740 1 -- 172.21.15.28:0/818583448 >> 172.21.15.28:0/818583448 conn(0x5585622c0e60 msgr2=0x5585622c6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.195+0000 7f36fb61e740 1 -- 172.21.15.28:0/818583448 shutdown_connections 2024-04-24T08:13:15.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.195+0000 7f36fb61e740 1 -- 172.21.15.28:0/818583448 wait complete. 2024-04-24T08:13:15.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.195+0000 7f36fb61e740 20 client.15807 trim_cache size 0 max 16384 2024-04-24T08:13:15.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.232+0000 7f36fb61e740 1 Processor -- start 2024-04-24T08:13:15.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.232+0000 7f36fb61e740 1 -- start start 2024-04-24T08:13:15.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.232+0000 7f36fb61e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c4b40 0x5585622ec930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.233+0000 7f36fb61e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c5460 0x5585622cb960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.233+0000 7f36f6800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c4b40 0x5585622ec930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.233+0000 7f36f6800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c4b40 0x5585622ec930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40936/0 (socket says 172.21.15.28:40936) 2024-04-24T08:13:15.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.233+0000 7f36f6800640 1 -- 172.21.15.28:0/2510651765 learned_addr learned my addr 172.21.15.28:0/2510651765 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:15.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.233+0000 7f36fb61e740 1 --2- 172.21.15.28:0/2510651765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622cfb30 0x5585622cf040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.233+0000 7f36fb61e740 1 -- 172.21.15.28:0/2510651765 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f36f01ae3d0 con 0x5585622c5460 2024-04-24T08:13:15.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.233+0000 7f36fb61e740 1 -- 172.21.15.28:0/2510651765 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f36f01abca0 con 0x5585622c4b40 2024-04-24T08:13:15.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.233+0000 7f36fb61e740 1 -- 172.21.15.28:0/2510651765 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f36f010c0e0 con 0x5585622cfb30 2024-04-24T08:13:15.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.234+0000 7f36f6800640 1 -- 172.21.15.28:0/2510651765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622cfb30 msgr2=0x5585622cf040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:15.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.234+0000 7f36f5400640 1 --2- 172.21.15.28:0/2510651765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c5460 0x5585622cb960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.234+0000 7f36f6800640 1 --2- 172.21.15.28:0/2510651765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622cfb30 0x5585622cf040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.234+0000 7f36f6800640 1 -- 172.21.15.28:0/2510651765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c5460 msgr2=0x5585622cb960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:15.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.234+0000 7f36f6800640 1 --2- 172.21.15.28:0/2510651765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c5460 0x5585622cb960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.234+0000 7f36f6800640 1 -- 172.21.15.28:0/2510651765 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5585620067d0 con 0x5585622c4b40 2024-04-24T08:13:15.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.234+0000 7f36f5400640 1 --2- 172.21.15.28:0/2510651765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c5460 0x5585622cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:15.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.235+0000 7f36f6800640 1 --2- 172.21.15.28:0/2510651765 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c4b40 0x5585622ec930 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f36e40058c0 tx=0x7f36e4004810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a8870394cddec07b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:15.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.237+0000 7f36de600640 1 -- 172.21.15.28:0/2510651765 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e401e040 con 0x5585622c4b40 2024-04-24T08:13:15.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.242+0000 7f36de600640 1 -- 172.21.15.28:0/2510651765 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36e4004530 con 0x5585622c4b40 2024-04-24T08:13:15.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.243+0000 7f36de600640 1 -- 172.21.15.28:0/2510651765 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e4003ab0 con 0x5585622c4b40 2024-04-24T08:13:15.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.244+0000 7f36fb61e740 1 -- 172.21.15.28:0/2510651765 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c4b40 msgr2=0x5585622ec930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.244+0000 7f36fb61e740 1 --2- 172.21.15.28:0/2510651765 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c4b40 0x5585622ec930 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f36e40058c0 tx=0x7f36e4004810 comp rx=0 tx=0).stop 2024-04-24T08:13:15.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.244+0000 7f36fb61e740 1 -- 172.21.15.28:0/2510651765 shutdown_connections 2024-04-24T08:13:15.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.244+0000 7f36fb61e740 1 --2- 172.21.15.28:0/2510651765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c5460 0x5585622cb960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.244+0000 7f36fb61e740 1 --2- 172.21.15.28:0/2510651765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622cfb30 0x5585622cf040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.244+0000 7f36fb61e740 1 --2- 172.21.15.28:0/2510651765 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c4b40 0x5585622ec930 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.244+0000 7f36fb61e740 1 -- 172.21.15.28:0/2510651765 >> 172.21.15.28:0/2510651765 conn(0x5585622c0e60 msgr2=0x5585622c62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:15.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.245+0000 7f36fb61e740 1 -- 172.21.15.28:0/2510651765 shutdown_connections 2024-04-24T08:13:15.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.245+0000 7f36fb61e740 1 -- 172.21.15.28:0/2510651765 wait complete. 2024-04-24T08:13:15.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.245+0000 7f36fb61e740 1 -- start start 2024-04-24T08:13:15.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.246+0000 7f36fb61e740 1 Processor -- start 2024-04-24T08:13:15.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.247+0000 7f36fb61e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c4b40 0x5585622e9100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.248+0000 7f36fb61e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c5460 0x5585622c9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.248+0000 7f36fb61e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622cfb30 0x5585622c9fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.248+0000 7f36f6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c4b40 0x5585622e9100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.248+0000 7f36f6800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c4b40 0x5585622e9100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38644/0 (socket says 172.21.15.28:38644) 2024-04-24T08:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.248+0000 7f36f6800640 1 -- 172.21.15.28:0/1600068557 learned_addr learned my addr 172.21.15.28:0/1600068557 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.249+0000 7f36fb61e740 1 -- 172.21.15.28:0/1600068557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f36f010c750 con 0x5585622c4b40 2024-04-24T08:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.249+0000 7f36fb61e740 1 -- 172.21.15.28:0/1600068557 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f36f010dbc0 con 0x5585622c5460 2024-04-24T08:13:15.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.249+0000 7f36fb61e740 1 -- 172.21.15.28:0/1600068557 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f36f010ce20 con 0x5585622cfb30 2024-04-24T08:13:15.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.250+0000 7f36fb61e740 10 client.0 _ll_register_callbacks cb 0x7f36e0005db0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.251+0000 7f36f5400640 1 --2- 172.21.15.28:0/1600068557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c5460 0x5585622c9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.251+0000 7f36f6800640 1 --2- 172.21.15.28:0/1600068557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622cfb30 0x5585622c9fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.251+0000 7f36f6800640 1 -- 172.21.15.28:0/1600068557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622cfb30 msgr2=0x5585622c9fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.251+0000 7f36f6800640 1 --2- 172.21.15.28:0/1600068557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622cfb30 0x5585622c9fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.251+0000 7f36f6800640 1 -- 172.21.15.28:0/1600068557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c5460 msgr2=0x5585622c9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:15.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.251+0000 7f36f6800640 1 --2- 172.21.15.28:0/1600068557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c5460 0x5585622c9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.251+0000 7f36f6800640 1 -- 172.21.15.28:0/1600068557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5585622ca6b0 con 0x5585622c4b40 2024-04-24T08:13:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.251+0000 7f36f5400640 1 --2- 172.21.15.28:0/1600068557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c5460 0x5585622c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:13:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.251+0000 7f36f6800640 1 --2- 172.21.15.28:0/1600068557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c4b40 0x5585622e9100 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x5585622ede60 tx=0x7f36e4003690 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b043b4a59929d2d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.251+0000 7f36d3400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:13:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.252+0000 7f36d3400640 1 -- 172.21.15.28:0/1600068557 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f36e4003d80 con 0x5585622c4b40 2024-04-24T08:13:15.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.252+0000 7f36d3400640 1 -- 172.21.15.28:0/1600068557 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f36e402b100 con 0x5585622c4b40 2024-04-24T08:13:15.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.252+0000 7f36fb61e740 1 -- 172.21.15.28:0/1600068557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f36e4004530 con 0x5585622c4b40 2024-04-24T08:13:15.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.253+0000 7f36d3400640 1 -- 172.21.15.28:0/1600068557 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f36e402b300 con 0x5585622c4b40 2024-04-24T08:13:15.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.253+0000 7f36fb61e740 10 client.15810 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:15.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.253+0000 7f36fb61e740 1 -- 172.21.15.28:0/1600068557 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5585622d1340 con 0x5585622c4b40 2024-04-24T08:13:15.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.254+0000 7f36d3400640 1 -- 172.21.15.28:0/1600068557 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f36e4004530 con 0x5585622c4b40 2024-04-24T08:13:15.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.254+0000 7f36d3400640 1 -- 172.21.15.28:0/1600068557 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f36e4018070 con 0x5585622c4b40 2024-04-24T08:13:15.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.255+0000 7f36fb61e740 10 client.15810 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:15.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.255+0000 7f36fb61e740 10 client.15810 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:13:15.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.255+0000 7f36fb61e740 20 client.15810 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:15.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.255+0000 7f36fb61e740 1 --2- 172.21.15.28:0/1600068557 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5585622f0b00 0x558562359260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.255+0000 7f36fb61e740 4 client.15810 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:13:15.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.255+0000 7f36fb61e740 1 -- 172.21.15.28:0/1600068557 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558562231a60 con 0x5585622f0b00 2024-04-24T08:13:15.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.256+0000 7f36f5400640 1 --2- 172.21.15.28:0/1600068557 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5585622f0b00 0x558562359260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.257+0000 7f36d3400640 10 client.15810 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:13:15.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.257+0000 7f36f5400640 1 --2- 172.21.15.28:0/1600068557 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5585622f0b00 0x558562359260 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7c186baa5e292652 server_cookie=368669650ac9e16c in_seq=0 out_seq=0 2024-04-24T08:13:15.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.257+0000 7f36d3400640 1 -- 172.21.15.28:0/1600068557 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558562231a60 con 0x5585622f0b00 2024-04-24T08:13:15.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.257+0000 7f36d3400640 10 client.15810 handle_command_reply: tid=0 2024-04-24T08:13:15.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.257+0000 7f36fb61e740 1 client.15810 shutdown 2024-04-24T08:13:15.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.258+0000 7f36fb61e740 1 -- 172.21.15.28:0/1600068557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c4b40 msgr2=0x5585622e9100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.258+0000 7f36fb61e740 1 --2- 172.21.15.28:0/1600068557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c4b40 0x5585622e9100 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x5585622ede60 tx=0x7f36e4003690 comp rx=0 tx=0).stop 2024-04-24T08:13:15.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.259+0000 7f36fb61e740 1 -- 172.21.15.28:0/1600068557 shutdown_connections 2024-04-24T08:13:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.259+0000 7f36fb61e740 1 --2- 172.21.15.28:0/1600068557 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5585622f0b00 0x558562359260 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.259+0000 7f36fb61e740 1 --2- 172.21.15.28:0/1600068557 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5585622cfb30 0x5585622c9fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.259+0000 7f36fb61e740 1 --2- 172.21.15.28:0/1600068557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5585622c5460 0x5585622c9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.259+0000 7f36fb61e740 1 --2- 172.21.15.28:0/1600068557 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5585622c4b40 0x5585622e9100 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.259+0000 7f36fb61e740 1 -- 172.21.15.28:0/1600068557 >> 172.21.15.28:0/1600068557 conn(0x5585622c0e60 msgr2=0x5585622cb7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:15.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.259+0000 7f36d3400640 0 client.15810 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:13:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.261+0000 7f36fb61e740 1 -- 172.21.15.28:0/1600068557 shutdown_connections 2024-04-24T08:13:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.262+0000 7f36fb61e740 1 -- 172.21.15.28:0/1600068557 wait complete. 2024-04-24T08:13:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.263+0000 7f36fb61e740 20 client.15810 trim_cache size 0 max 16384 2024-04-24T08:13:15.264 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:13:15.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.269+0000 7f36f8800640 1 -- 172.21.15.28:0/615997437 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f36d80a3da0 msgr2=0x7f36d80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.269+0000 7f36f8800640 1 --2- 172.21.15.28:0/615997437 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f36d80a3da0 0x7f36d80a6260 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x5585622c6100 tx=0x7f36ec0f55f0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.269+0000 7f36f8800640 1 -- 172.21.15.28:0/615997437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f36f0076530 msgr2=0x7f36f019ee60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.269+0000 7f36f8800640 1 --2- 172.21.15.28:0/615997437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f36f0076530 0x7f36f019ee60 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f36ec005e30 tx=0x7f36ec031870 comp rx=0 tx=0).stop 2024-04-24T08:13:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.273+0000 7f36f8800640 1 -- 172.21.15.28:0/615997437 shutdown_connections 2024-04-24T08:13:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.273+0000 7f36f8800640 1 --2- 172.21.15.28:0/615997437 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f36d80a3da0 0x7f36d80a6260 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.273+0000 7f36f8800640 1 --2- 172.21.15.28:0/615997437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f36f0076e50 0x7f36f019f3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.273+0000 7f36f8800640 1 --2- 172.21.15.28:0/615997437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36f0112d80 0x7f36f01a36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.273+0000 7f36f8800640 1 --2- 172.21.15.28:0/615997437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f36f0076530 0x7f36f019ee60 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.273+0000 7f36f8800640 1 -- 172.21.15.28:0/615997437 >> 172.21.15.28:0/615997437 conn(0x7f36f00ffb70 msgr2=0x7f36f01144a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.276+0000 7f36f8800640 1 -- 172.21.15.28:0/615997437 shutdown_connections 2024-04-24T08:13:15.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.276+0000 7f36f8800640 1 -- 172.21.15.28:0/615997437 wait complete. 2024-04-24T08:13:15.366 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:13:15.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.809+0000 7f2953200640 1 -- 172.21.15.28:0/2866944541 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2954106250 msgr2=0x7f29541066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.809+0000 7f2953200640 1 --2- 172.21.15.28:0/2866944541 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2954106250 0x7f29541066e0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f294c009ca0 tx=0x7f294c02a940 comp rx=0 tx=0).stop 2024-04-24T08:13:15.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.812+0000 7f2953200640 1 -- 172.21.15.28:0/2866944541 shutdown_connections 2024-04-24T08:13:15.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.812+0000 7f2953200640 1 --2- 172.21.15.28:0/2866944541 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2954101770 0x7f2954105c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.812+0000 7f2953200640 1 --2- 172.21.15.28:0/2866944541 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2954106250 0x7f29541066e0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.812+0000 7f2953200640 1 --2- 172.21.15.28:0/2866944541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2954100e50 0x7f2954101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.812+0000 7f2953200640 1 -- 172.21.15.28:0/2866944541 >> 172.21.15.28:0/2866944541 conn(0x7f29540fc880 msgr2=0x7f29540feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:15.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.812+0000 7f2953200640 1 -- 172.21.15.28:0/2866944541 shutdown_connections 2024-04-24T08:13:15.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.812+0000 7f2953200640 1 -- 172.21.15.28:0/2866944541 wait complete. 2024-04-24T08:13:15.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.813+0000 7f2953200640 1 Processor -- start 2024-04-24T08:13:15.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.813+0000 7f2953200640 1 -- start start 2024-04-24T08:13:15.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.813+0000 7f2953200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2954100e50 0x7f29541a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.813+0000 7f2953200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2954101770 0x7f29541a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.813+0000 7f2953200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2954106250 0x7f295419eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.813+0000 7f2953200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2954118850 con 0x7f2954106250 2024-04-24T08:13:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.813+0000 7f2953200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f29541186d0 con 0x7f2954101770 2024-04-24T08:13:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.813+0000 7f2953200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f29541189d0 con 0x7f2954100e50 2024-04-24T08:13:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.814+0000 7f2951400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2954101770 0x7f29541a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.814+0000 7f2951400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2954101770 0x7f29541a4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40968/0 (socket says 172.21.15.28:40968) 2024-04-24T08:13:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.814+0000 7f2951400640 1 -- 172.21.15.28:0/608847727 learned_addr learned my addr 172.21.15.28:0/608847727 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.814+0000 7f2952800640 1 --2- 172.21.15.28:0/608847727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2954106250 0x7f295419eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.814+0000 7f2951e00640 1 --2- 172.21.15.28:0/608847727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2954100e50 0x7f29541a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.814+0000 7f2951400640 1 -- 172.21.15.28:0/608847727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2954100e50 msgr2=0x7f29541a4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.814+0000 7f2951400640 1 --2- 172.21.15.28:0/608847727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2954100e50 0x7f29541a4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.814+0000 7f2951400640 1 -- 172.21.15.28:0/608847727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2954106250 msgr2=0x7f295419eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.814+0000 7f2951400640 1 --2- 172.21.15.28:0/608847727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2954106250 0x7f295419eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.814+0000 7f2951400640 1 -- 172.21.15.28:0/608847727 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f295419f3f0 con 0x7f2954101770 2024-04-24T08:13:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.815+0000 7f2951400640 1 --2- 172.21.15.28:0/608847727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2954101770 0x7f29541a4fb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f294000d740 tx=0x7f294000dc00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:15.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.815+0000 7f294aa00640 1 -- 172.21.15.28:0/608847727 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2940017070 con 0x7f2954101770 2024-04-24T08:13:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.815+0000 7f2953200640 1 -- 172.21.15.28:0/608847727 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f295419f6e0 con 0x7f2954101770 2024-04-24T08:13:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.815+0000 7f2953200640 1 -- 172.21.15.28:0/608847727 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f29541ab9f0 con 0x7f2954101770 2024-04-24T08:13:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.815+0000 7f294aa00640 1 -- 172.21.15.28:0/608847727 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f294000de50 con 0x7f2954101770 2024-04-24T08:13:15.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.815+0000 7f294aa00640 1 -- 172.21.15.28:0/608847727 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294000f620 con 0x7f2954101770 2024-04-24T08:13:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.819+0000 7f295be42740 1 Processor -- start 2024-04-24T08:13:15.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.819+0000 7f295be42740 1 -- start start 2024-04-24T08:13:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.820+0000 7f295be42740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e65fdb40 0x5617e65fdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.820+0000 7f295be42740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fe460 0x5617e66085f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.820+0000 7f295be42740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e6608b30 0x5617e660af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.820+0000 7f295be42740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5617e65631d0 con 0x5617e65fe460 2024-04-24T08:13:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.820+0000 7f295be42740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5617e6509450 con 0x5617e6608b30 2024-04-24T08:13:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.820+0000 7f295be42740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5617e655f190 con 0x5617e65fdb40 2024-04-24T08:13:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.820+0000 7f294aa00640 1 -- 172.21.15.28:0/608847727 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f2940009c30 con 0x7f2954101770 2024-04-24T08:13:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.820+0000 7f2952800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e65fdb40 0x5617e65fdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.820+0000 7f2952800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e65fdb40 0x5617e65fdf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:57024/0 (socket says 172.21.15.28:57024) 2024-04-24T08:13:15.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.820+0000 7f2952800640 1 -- 172.21.15.28:0/1680641553 learned_addr learned my addr 172.21.15.28:0/1680641553 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.820+0000 7f2951400640 1 --2- 172.21.15.28:0/1680641553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e6608b30 0x5617e660af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.820+0000 7f2951e00640 1 --2- 172.21.15.28:0/1680641553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fe460 0x5617e66085f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.821+0000 7f2952800640 1 -- 172.21.15.28:0/1680641553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e6608b30 msgr2=0x5617e660af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.821+0000 7f2952800640 1 --2- 172.21.15.28:0/1680641553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e6608b30 0x5617e660af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.821+0000 7f2952800640 1 -- 172.21.15.28:0/1680641553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fe460 msgr2=0x5617e66085f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.821+0000 7f2952800640 1 --2- 172.21.15.28:0/1680641553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fe460 0x5617e66085f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.821+0000 7f2952800640 1 -- 172.21.15.28:0/1680641553 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617e65c6310 con 0x5617e65fdb40 2024-04-24T08:13:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.821+0000 7f2951e00640 1 --2- 172.21.15.28:0/1680641553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fe460 0x5617e66085f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:15.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.821+0000 7f2952800640 1 --2- 172.21.15.28:0/1680641553 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e65fdb40 0x5617e65fdf20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f294c005fa0 tx=0x7f294c009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd60318bb72c6a09 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:15.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.821+0000 7f293f400640 1 -- 172.21.15.28:0/1680641553 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c034040 con 0x5617e65fdb40 2024-04-24T08:13:15.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.821+0000 7f293f400640 1 -- 172.21.15.28:0/1680641553 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f294c02f450 con 0x5617e65fdb40 2024-04-24T08:13:15.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.821+0000 7f293f400640 1 -- 172.21.15.28:0/1680641553 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c0308a0 con 0x5617e65fdb40 2024-04-24T08:13:15.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.822+0000 7f294aa00640 1 --2- 172.21.15.28:0/608847727 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29380a3da0 0x7f29380a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.822+0000 7f294aa00640 1 -- 172.21.15.28:0/608847727 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f29400d0270 con 0x7f2954101770 2024-04-24T08:13:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.822+0000 7f295be42740 1 -- 172.21.15.28:0/1680641553 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e65fdb40 msgr2=0x5617e65fdf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.822+0000 7f295be42740 1 --2- 172.21.15.28:0/1680641553 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e65fdb40 0x5617e65fdf20 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f294c005fa0 tx=0x7f294c009800 comp rx=0 tx=0).stop 2024-04-24T08:13:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.822+0000 7f295be42740 1 -- 172.21.15.28:0/1680641553 shutdown_connections 2024-04-24T08:13:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.822+0000 7f295be42740 1 --2- 172.21.15.28:0/1680641553 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fe460 0x5617e66085f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.822+0000 7f295be42740 1 --2- 172.21.15.28:0/1680641553 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e6608b30 0x5617e660af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.822+0000 7f295be42740 1 --2- 172.21.15.28:0/1680641553 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e65fdb40 0x5617e65fdf20 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.822+0000 7f295be42740 1 -- 172.21.15.28:0/1680641553 >> 172.21.15.28:0/1680641553 conn(0x5617e65f9e60 msgr2=0x5617e65fa240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.822+0000 7f295be42740 1 -- 172.21.15.28:0/1680641553 shutdown_connections 2024-04-24T08:13:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.823+0000 7f2951e00640 1 --2- 172.21.15.28:0/608847727 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29380a3da0 0x7f29380a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.823+0000 7f295be42740 1 -- 172.21.15.28:0/1680641553 wait complete. 2024-04-24T08:13:15.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.823+0000 7f295be42740 1 -- start start 2024-04-24T08:13:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.824+0000 7f295be42740 1 Processor -- start 2024-04-24T08:13:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.824+0000 7f2951e00640 1 --2- 172.21.15.28:0/608847727 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29380a3da0 0x7f29380a6260 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x5617e65ff100 tx=0x7f2944005f30 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.824+0000 7f295be42740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e65fdb40 0x5617e6608480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.824+0000 7f295be42740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fe460 0x5617e6605770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.824+0000 7f295be42740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e6608b30 0x5617e6605cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.824+0000 7f295be42740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5617e660b960 con 0x5617e65fe460 2024-04-24T08:13:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.824+0000 7f295be42740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5617e651c810 con 0x5617e6608b30 2024-04-24T08:13:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.824+0000 7f295be42740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5617e660bae0 con 0x5617e65fdb40 2024-04-24T08:13:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.824+0000 7f295be42740 10 client.0 _ll_register_callbacks cb 0x5617e64ea140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.825+0000 7f2952800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e6608b30 0x5617e6605cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.825+0000 7f2952800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e6608b30 0x5617e6605cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40978/0 (socket says 172.21.15.28:40978) 2024-04-24T08:13:15.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.825+0000 7f2952800640 1 -- 172.21.15.28:0/748351054 learned_addr learned my addr 172.21.15.28:0/748351054 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.825+0000 7f2951e00640 1 --2- 172.21.15.28:0/748351054 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e65fdb40 0x5617e6608480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.825+0000 7f2952800640 1 -- 172.21.15.28:0/748351054 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e65fdb40 msgr2=0x5617e6608480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.825+0000 7f2952800640 1 --2- 172.21.15.28:0/748351054 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e65fdb40 0x5617e6608480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.825+0000 7f2951400640 1 --2- 172.21.15.28:0/748351054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fe460 0x5617e6605770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.825+0000 7f2952800640 1 -- 172.21.15.28:0/748351054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fe460 msgr2=0x5617e6605770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.825+0000 7f2952800640 1 --2- 172.21.15.28:0/748351054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fe460 0x5617e6605770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.825+0000 7f2952800640 1 -- 172.21.15.28:0/748351054 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617e652d3f0 con 0x5617e6608b30 2024-04-24T08:13:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.825+0000 7f2951400640 1 --2- 172.21.15.28:0/748351054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fe460 0x5617e6605770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.825+0000 7f2952800640 1 --2- 172.21.15.28:0/748351054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e6608b30 0x5617e6605cd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f294c009bf0 tx=0x7f294c02fd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=933b18ff4ce25aec server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.825+0000 7f2933e00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:13:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.825+0000 7f2933e00640 1 -- 172.21.15.28:0/748351054 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c051060 con 0x5617e6608b30 2024-04-24T08:13:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.826+0000 7f295be42740 1 -- 172.21.15.28:0/748351054 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f294c02f450 con 0x5617e6608b30 2024-04-24T08:13:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.826+0000 7f2933e00640 1 -- 172.21.15.28:0/748351054 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f294c030640 con 0x5617e6608b30 2024-04-24T08:13:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.826+0000 7f2933e00640 1 -- 172.21.15.28:0/748351054 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f294c034070 con 0x5617e6608b30 2024-04-24T08:13:15.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.826+0000 7f295be42740 10 client.25946 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.826+0000 7f295be42740 1 -- 172.21.15.28:0/748351054 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5617e62753f0 con 0x5617e6608b30 2024-04-24T08:13:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.826+0000 7f2933e00640 1 -- 172.21.15.28:0/748351054 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f294c02f450 con 0x5617e6608b30 2024-04-24T08:13:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.826+0000 7f2933e00640 1 -- 172.21.15.28:0/748351054 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f294c02f070 con 0x5617e6608b30 2024-04-24T08:13:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.826+0000 7f295be42740 10 client.25946 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.826+0000 7f295be42740 10 client.25946 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:13:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.826+0000 7f295be42740 20 client.25946 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:15.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.826+0000 7f295be42740 1 --2- 172.21.15.28:0/748351054 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5617e6618a50 0x5617e661ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.826+0000 7f295be42740 4 client.25946 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:13:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.826+0000 7f295be42740 1 -- 172.21.15.28:0/748351054 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5617e6570bf0 con 0x5617e6618a50 2024-04-24T08:13:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.827+0000 7f2951e00640 1 --2- 172.21.15.28:0/748351054 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5617e6618a50 0x5617e661ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.828+0000 7f2951e00640 1 --2- 172.21.15.28:0/748351054 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5617e6618a50 0x5617e661ae40 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8c5e019dae026b8d server_cookie=da841b1dc413a14e in_seq=0 out_seq=0 2024-04-24T08:13:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.828+0000 7f2933e00640 10 client.25946 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.829+0000 7f2933e00640 1 -- 172.21.15.28:0/748351054 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5617e6570bf0 con 0x5617e6618a50 2024-04-24T08:13:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.829+0000 7f2933e00640 10 client.25946 handle_command_reply: tid=0 2024-04-24T08:13:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.829+0000 7f295be42740 1 client.25946 shutdown 2024-04-24T08:13:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.829+0000 7f295be42740 1 -- 172.21.15.28:0/748351054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e6608b30 msgr2=0x5617e6605cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.829+0000 7f295be42740 1 --2- 172.21.15.28:0/748351054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e6608b30 0x5617e6605cd0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f294c009bf0 tx=0x7f294c02fd30 comp rx=0 tx=0).stop 2024-04-24T08:13:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.829+0000 7f295be42740 1 -- 172.21.15.28:0/748351054 shutdown_connections 2024-04-24T08:13:15.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.829+0000 7f295be42740 1 --2- 172.21.15.28:0/748351054 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5617e6618a50 0x5617e661ae40 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.829+0000 7f295be42740 1 --2- 172.21.15.28:0/748351054 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fe460 0x5617e6605770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.829+0000 7f295be42740 1 --2- 172.21.15.28:0/748351054 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e6608b30 0x5617e6605cd0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.829+0000 7f295be42740 1 --2- 172.21.15.28:0/748351054 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e65fdb40 0x5617e6608480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.829+0000 7f295be42740 1 -- 172.21.15.28:0/748351054 >> 172.21.15.28:0/748351054 conn(0x5617e65f9e60 msgr2=0x5617e65f8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.829+0000 7f2933e00640 0 client.25946 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.829+0000 7f295be42740 1 -- 172.21.15.28:0/748351054 shutdown_connections 2024-04-24T08:13:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.829+0000 7f295be42740 1 -- 172.21.15.28:0/748351054 wait complete. 2024-04-24T08:13:15.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.830+0000 7f295be42740 20 client.25946 trim_cache size 0 max 16384 2024-04-24T08:13:15.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.845+0000 7f295be42740 1 Processor -- start 2024-04-24T08:13:15.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.845+0000 7f295be42740 1 -- start start 2024-04-24T08:13:15.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.845+0000 7f295be42740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fdb40 0x5617e66148b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.846+0000 7f295be42740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e65fe460 0x5617e660ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.846+0000 7f295be42740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e6608b30 0x5617e66047b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.846+0000 7f295be42740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2934011100 con 0x5617e65fdb40 2024-04-24T08:13:15.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.846+0000 7f295be42740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f293400e9d0 con 0x5617e65fe460 2024-04-24T08:13:15.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.846+0000 7f295be42740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f293400c820 con 0x5617e6608b30 2024-04-24T08:13:15.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.846+0000 7f2951e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fdb40 0x5617e66148b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.846+0000 7f2952800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e6608b30 0x5617e66047b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.846+0000 7f2952800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e6608b30 0x5617e66047b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:57054/0 (socket says 172.21.15.28:57054) 2024-04-24T08:13:15.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.846+0000 7f2952800640 1 -- 172.21.15.28:0/1985766132 learned_addr learned my addr 172.21.15.28:0/1985766132 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:15.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.846+0000 7f2952800640 1 -- 172.21.15.28:0/1985766132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e65fe460 msgr2=0x5617e660ae30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:13:15.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.846+0000 7f2952800640 1 --2- 172.21.15.28:0/1985766132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e65fe460 0x5617e660ae30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.846+0000 7f2952800640 1 -- 172.21.15.28:0/1985766132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fdb40 msgr2=0x5617e66148b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:15.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.846+0000 7f2952800640 1 --2- 172.21.15.28:0/1985766132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fdb40 0x5617e66148b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.846+0000 7f2952800640 1 -- 172.21.15.28:0/1985766132 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617e633f7d0 con 0x5617e6608b30 2024-04-24T08:13:15.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.847+0000 7f2952800640 1 --2- 172.21.15.28:0/1985766132 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e6608b30 0x5617e66047b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f294c000f80 tx=0x7f294c005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3c6e835a0563bbe server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:15.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.847+0000 7f293ea00640 1 -- 172.21.15.28:0/1985766132 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c051040 con 0x5617e6608b30 2024-04-24T08:13:15.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.847+0000 7f293ea00640 1 -- 172.21.15.28:0/1985766132 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f294c055910 con 0x5617e6608b30 2024-04-24T08:13:15.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.847+0000 7f293ea00640 1 -- 172.21.15.28:0/1985766132 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c030ba0 con 0x5617e6608b30 2024-04-24T08:13:15.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.848+0000 7f295be42740 1 -- 172.21.15.28:0/1985766132 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e6608b30 msgr2=0x5617e66047b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.848+0000 7f295be42740 1 --2- 172.21.15.28:0/1985766132 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e6608b30 0x5617e66047b0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f294c000f80 tx=0x7f294c005ef0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.848+0000 7f295be42740 1 -- 172.21.15.28:0/1985766132 shutdown_connections 2024-04-24T08:13:15.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.848+0000 7f295be42740 1 --2- 172.21.15.28:0/1985766132 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e6608b30 0x5617e66047b0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.848+0000 7f295be42740 1 --2- 172.21.15.28:0/1985766132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e65fe460 0x5617e660ae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.848+0000 7f295be42740 1 --2- 172.21.15.28:0/1985766132 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fdb40 0x5617e66148b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.848+0000 7f295be42740 1 -- 172.21.15.28:0/1985766132 >> 172.21.15.28:0/1985766132 conn(0x5617e65f9e60 msgr2=0x5617e66040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:15.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.849+0000 7f295be42740 1 -- 172.21.15.28:0/1985766132 shutdown_connections 2024-04-24T08:13:15.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.849+0000 7f295be42740 1 -- 172.21.15.28:0/1985766132 wait complete. 2024-04-24T08:13:15.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.850+0000 7f295be42740 1 -- start start 2024-04-24T08:13:15.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.850+0000 7f295be42740 1 Processor -- start 2024-04-24T08:13:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.850+0000 7f295be42740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fdb40 0x5617e6611030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.850+0000 7f295be42740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e65fe460 0x5617e6602040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.850+0000 7f295be42740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e6608b30 0x5617e6602580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.851+0000 7f295be42740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f293400ce90 con 0x5617e65fdb40 2024-04-24T08:13:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.851+0000 7f295be42740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f293400e300 con 0x5617e65fe460 2024-04-24T08:13:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.851+0000 7f295be42740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f293400d560 con 0x5617e6608b30 2024-04-24T08:13:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.851+0000 7f295be42740 10 client.0 _ll_register_callbacks cb 0x7f294c02fc60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.851+0000 7f2952800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e6608b30 0x5617e6602580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.851+0000 7f2952800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e6608b30 0x5617e6602580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:57068/0 (socket says 172.21.15.28:57068) 2024-04-24T08:13:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.851+0000 7f2952800640 1 -- 172.21.15.28:0/2793404644 learned_addr learned my addr 172.21.15.28:0/2793404644 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.851+0000 7f2951e00640 1 --2- 172.21.15.28:0/2793404644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fdb40 0x5617e6611030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.851+0000 7f2952800640 1 -- 172.21.15.28:0/2793404644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e65fe460 msgr2=0x5617e6602040 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:13:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.851+0000 7f2952800640 1 --2- 172.21.15.28:0/2793404644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e65fe460 0x5617e6602040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.851+0000 7f2952800640 1 -- 172.21.15.28:0/2793404644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fdb40 msgr2=0x5617e6611030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.851+0000 7f2952800640 1 --2- 172.21.15.28:0/2793404644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fdb40 0x5617e6611030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.851+0000 7f2952800640 1 -- 172.21.15.28:0/2793404644 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617e6602c20 con 0x5617e6608b30 2024-04-24T08:13:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.852+0000 7f2952800640 1 --2- 172.21.15.28:0/2793404644 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e6608b30 0x5617e6602580 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f294c002230 tx=0x7f294c030820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e85c935c8a2eaeb8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.852+0000 7f2933e00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:13:15.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.852+0000 7f2933e00640 1 -- 172.21.15.28:0/2793404644 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f294c051070 con 0x5617e6608b30 2024-04-24T08:13:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.852+0000 7f2933e00640 1 -- 172.21.15.28:0/2793404644 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f294c030920 con 0x5617e6608b30 2024-04-24T08:13:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.852+0000 7f295be42740 1 -- 172.21.15.28:0/2793404644 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f294c055910 con 0x5617e6608b30 2024-04-24T08:13:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.852+0000 7f2933e00640 1 -- 172.21.15.28:0/2793404644 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f294c030b20 con 0x5617e6608b30 2024-04-24T08:13:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.852+0000 7f295be42740 10 client.25996 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.852+0000 7f295be42740 1 -- 172.21.15.28:0/2793404644 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5617e667ed60 con 0x5617e6608b30 2024-04-24T08:13:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.853+0000 7f2933e00640 1 -- 172.21.15.28:0/2793404644 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f294c030cc0 con 0x5617e6608b30 2024-04-24T08:13:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.855+0000 7f2933e00640 1 -- 172.21.15.28:0/2793404644 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f294c03a4e0 con 0x5617e6608b30 2024-04-24T08:13:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.855+0000 7f295be42740 10 client.25996 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.855+0000 7f295be42740 10 client.25996 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:13:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.855+0000 7f295be42740 20 client.25996 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.855+0000 7f295be42740 1 --2- 172.21.15.28:0/2793404644 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5617e667f230 0x5617e6681620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.855+0000 7f295be42740 4 client.25996 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:13:15.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.855+0000 7f295be42740 1 -- 172.21.15.28:0/2793404644 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "damage ls"}) -- 0x7f294c055910 con 0x5617e667f230 2024-04-24T08:13:15.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.857+0000 7f2951e00640 1 --2- 172.21.15.28:0/2793404644 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5617e667f230 0x5617e6681620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:15.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.858+0000 7f2951e00640 1 --2- 172.21.15.28:0/2793404644 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5617e667f230 0x5617e6681620 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d8ee7a9e3db1fc53 server_cookie=a75828f3fc736b60 in_seq=0 out_seq=0 2024-04-24T08:13:15.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.858+0000 7f2933e00640 10 client.25996 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:15.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.859+0000 7f2933e00640 1 -- 172.21.15.28:0/2793404644 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x7f294c055910 con 0x5617e667f230 2024-04-24T08:13:15.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.859+0000 7f2933e00640 10 client.25996 handle_command_reply: tid=0 2024-04-24T08:13:15.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.859+0000 7f295be42740 1 client.25996 shutdown 2024-04-24T08:13:15.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.859+0000 7f295be42740 1 -- 172.21.15.28:0/2793404644 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e6608b30 msgr2=0x5617e6602580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.859+0000 7f295be42740 1 --2- 172.21.15.28:0/2793404644 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e6608b30 0x5617e6602580 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f294c002230 tx=0x7f294c030820 comp rx=0 tx=0).stop 2024-04-24T08:13:15.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.859+0000 7f295be42740 1 -- 172.21.15.28:0/2793404644 shutdown_connections 2024-04-24T08:13:15.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.859+0000 7f295be42740 1 --2- 172.21.15.28:0/2793404644 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5617e667f230 0x5617e6681620 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.859+0000 7f295be42740 1 --2- 172.21.15.28:0/2793404644 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5617e6608b30 0x5617e6602580 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.859+0000 7f295be42740 1 --2- 172.21.15.28:0/2793404644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5617e65fe460 0x5617e6602040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.859+0000 7f295be42740 1 --2- 172.21.15.28:0/2793404644 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5617e65fdb40 0x5617e6611030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.859+0000 7f295be42740 1 -- 172.21.15.28:0/2793404644 >> 172.21.15.28:0/2793404644 conn(0x5617e65f9e60 msgr2=0x5617e65f86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:15.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.859+0000 7f2933e00640 0 client.25996 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:15.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.861+0000 7f295be42740 1 -- 172.21.15.28:0/2793404644 shutdown_connections 2024-04-24T08:13:15.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.861+0000 7f295be42740 1 -- 172.21.15.28:0/2793404644 wait complete. 2024-04-24T08:13:15.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.861+0000 7f295be42740 20 client.25996 trim_cache size 0 max 16384 2024-04-24T08:13:15.861 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:13:15.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.862+0000 7f2953200640 1 -- 172.21.15.28:0/608847727 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29380a3da0 msgr2=0x7f29380a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.862+0000 7f2953200640 1 --2- 172.21.15.28:0/608847727 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29380a3da0 0x7f29380a6260 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x5617e65ff100 tx=0x7f2944005f30 comp rx=0 tx=0).stop 2024-04-24T08:13:15.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.862+0000 7f2953200640 1 -- 172.21.15.28:0/608847727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2954101770 msgr2=0x7f29541a4fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:15.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.862+0000 7f2953200640 1 --2- 172.21.15.28:0/608847727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2954101770 0x7f29541a4fb0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f294000d740 tx=0x7f294000dc00 comp rx=0 tx=0).stop 2024-04-24T08:13:15.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.862+0000 7f2953200640 1 -- 172.21.15.28:0/608847727 shutdown_connections 2024-04-24T08:13:15.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.862+0000 7f2953200640 1 --2- 172.21.15.28:0/608847727 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29380a3da0 0x7f29380a6260 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.862+0000 7f2953200640 1 --2- 172.21.15.28:0/608847727 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2954106250 0x7f295419eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.862+0000 7f2953200640 1 --2- 172.21.15.28:0/608847727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2954101770 0x7f29541a4fb0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.862+0000 7f2953200640 1 --2- 172.21.15.28:0/608847727 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2954100e50 0x7f29541a4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:15.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.862+0000 7f2953200640 1 -- 172.21.15.28:0/608847727 >> 172.21.15.28:0/608847727 conn(0x7f29540fc880 msgr2=0x7f29540fd860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:15.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.862+0000 7f2953200640 1 -- 172.21.15.28:0/608847727 shutdown_connections 2024-04-24T08:13:15.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:15.862+0000 7f2953200640 1 -- 172.21.15.28:0/608847727 wait complete. 2024-04-24T08:13:15.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:15 smithi194 ceph-mon[32836]: pgmap v90: 97 pgs: 97 active+clean; 5.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 208 MiB/s wr, 78 op/s 2024-04-24T08:13:16.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:15 smithi028 ceph-mon[27786]: pgmap v90: 97 pgs: 97 active+clean; 5.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 208 MiB/s wr, 78 op/s 2024-04-24T08:13:16.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:15 smithi117 ceph-mon[32438]: pgmap v90: 97 pgs: 97 active+clean; 5.8 GiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 208 MiB/s wr, 78 op/s 2024-04-24T08:13:16.887 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:13:16.887 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:16.887 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:13:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.141+0000 7f5d03800640 1 -- 172.21.15.28:0/3352592899 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cfc109580 msgr2=0x7f5cfc111150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.141+0000 7f5d03800640 1 --2- 172.21.15.28:0/3352592899 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cfc109580 0x7f5cfc111150 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4009ca0 tx=0x7f5cf402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.141+0000 7f5d03800640 1 -- 172.21.15.28:0/3352592899 shutdown_connections 2024-04-24T08:13:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.141+0000 7f5d03800640 1 --2- 172.21.15.28:0/3352592899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cfc104e60 0x7f5cfc108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.141+0000 7f5d03800640 1 --2- 172.21.15.28:0/3352592899 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cfc109580 0x7f5cfc111150 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.141+0000 7f5d03800640 1 --2- 172.21.15.28:0/3352592899 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5cfc1044b0 0x7f5cfc104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.141+0000 7f5d03800640 1 -- 172.21.15.28:0/3352592899 >> 172.21.15.28:0/3352592899 conn(0x7f5cfc0ffd90 msgr2=0x7f5cfc1021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:17.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.141+0000 7f5d03800640 1 -- 172.21.15.28:0/3352592899 shutdown_connections 2024-04-24T08:13:17.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.141+0000 7f5d03800640 1 -- 172.21.15.28:0/3352592899 wait complete. 2024-04-24T08:13:17.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.142+0000 7f5d03800640 1 Processor -- start 2024-04-24T08:13:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.142+0000 7f5d03800640 1 -- start start 2024-04-24T08:13:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.143+0000 7f5d03800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cfc1044b0 0x7f5cfc076510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.143+0000 7f5d03800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cfc109580 0x7f5cfc076a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.143+0000 7f5d03800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5cfc076f90 0x7f5cfc1ab0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.143+0000 7f5d03800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc114670 con 0x7f5cfc109580 2024-04-24T08:13:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.143+0000 7f5d03800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1144f0 con 0x7f5cfc1044b0 2024-04-24T08:13:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.143+0000 7f5d03800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1147f0 con 0x7f5cfc076f90 2024-04-24T08:13:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.143+0000 7f5cfbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cfc109580 0x7f5cfc076a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.143+0000 7f5cfbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cfc109580 0x7f5cfc076a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38714/0 (socket says 172.21.15.28:38714) 2024-04-24T08:13:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.143+0000 7f5cfbe00640 1 -- 172.21.15.28:0/2943898412 learned_addr learned my addr 172.21.15.28:0/2943898412 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.143+0000 7f5d00e00640 1 --2- 172.21.15.28:0/2943898412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cfc1044b0 0x7f5cfc076510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.144+0000 7f5d01800640 1 --2- 172.21.15.28:0/2943898412 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5cfc076f90 0x7f5cfc1ab0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.144+0000 7f5d00e00640 1 -- 172.21.15.28:0/2943898412 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5cfc076f90 msgr2=0x7f5cfc1ab0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.144+0000 7f5d00e00640 1 --2- 172.21.15.28:0/2943898412 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5cfc076f90 0x7f5cfc1ab0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.144+0000 7f5d00e00640 1 -- 172.21.15.28:0/2943898412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cfc109580 msgr2=0x7f5cfc076a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.144+0000 7f5d00e00640 1 --2- 172.21.15.28:0/2943898412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cfc109580 0x7f5cfc076a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.144+0000 7f5d00e00640 1 -- 172.21.15.28:0/2943898412 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cfc19a880 con 0x7f5cfc1044b0 2024-04-24T08:13:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.144+0000 7f5d01800640 1 --2- 172.21.15.28:0/2943898412 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5cfc076f90 0x7f5cfc1ab0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:13:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.144+0000 7f5d00e00640 1 --2- 172.21.15.28:0/2943898412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cfc1044b0 0x7f5cfc076510 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000ea20 tx=0x7f5cf000eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.144+0000 7f5cf9600640 1 -- 172.21.15.28:0/2943898412 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf000cd50 con 0x7f5cfc1044b0 2024-04-24T08:13:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.144+0000 7f5cf9600640 1 -- 172.21.15.28:0/2943898412 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cf00035f0 con 0x7f5cfc1044b0 2024-04-24T08:13:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.144+0000 7f5cf9600640 1 -- 172.21.15.28:0/2943898412 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf0010690 con 0x7f5cfc1044b0 2024-04-24T08:13:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.145+0000 7f5d03800640 1 -- 172.21.15.28:0/2943898412 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cfc19ab10 con 0x7f5cfc1044b0 2024-04-24T08:13:17.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.145+0000 7f5d03800640 1 -- 172.21.15.28:0/2943898412 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cfc19b010 con 0x7f5cfc1044b0 2024-04-24T08:13:17.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.150+0000 7f5cf9600640 1 -- 172.21.15.28:0/2943898412 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5cf00037a0 con 0x7f5cfc1044b0 2024-04-24T08:13:17.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.155+0000 7f5cf9600640 1 --2- 172.21.15.28:0/2943898412 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5ce40a3da0 0x7f5ce40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.155+0000 7f5cf9600640 1 -- 172.21.15.28:0/2943898412 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f5cf00d0bc0 con 0x7f5cfc1044b0 2024-04-24T08:13:17.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.156+0000 7f5d0662c740 1 Processor -- start 2024-04-24T08:13:17.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.156+0000 7f5cfbe00640 1 --2- 172.21.15.28:0/2943898412 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5ce40a3da0 0x7f5ce40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.157+0000 7f5d0662c740 1 -- start start 2024-04-24T08:13:17.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.158+0000 7f5d0662c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b2090 0x55c2197b2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.159+0000 7f5d0662c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197b29b0 0x55c2197bcb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.159+0000 7f5d0662c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197bd080 0x55c2197bf470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.159+0000 7f5d0662c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c219717e70 con 0x55c2197b2090 2024-04-24T08:13:17.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.159+0000 7f5d0662c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c2196be380 con 0x55c2197b29b0 2024-04-24T08:13:17.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.159+0000 7f5d0662c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c219713e30 con 0x55c2197bd080 2024-04-24T08:13:17.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.159+0000 7f5d00e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b2090 0x55c2197b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.159+0000 7f5d00e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b2090 0x55c2197b2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38730/0 (socket says 172.21.15.28:38730) 2024-04-24T08:13:17.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.159+0000 7f5d00e00640 1 -- 172.21.15.28:0/3643846880 learned_addr learned my addr 172.21.15.28:0/3643846880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:17.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.160+0000 7f5d01800640 1 --2- 172.21.15.28:0/3643846880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197bd080 0x55c2197bf470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.160+0000 7f5cfbe00640 1 --2- 172.21.15.28:0/2943898412 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5ce40a3da0 0x7f5ce40a6260 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5ce80095f0 tx=0x7f5ce8005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:17.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.160+0000 7f5d00e00640 1 -- 172.21.15.28:0/3643846880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197bd080 msgr2=0x55c2197bf470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:17.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.161+0000 7f5d00e00640 1 --2- 172.21.15.28:0/3643846880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197bd080 0x55c2197bf470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.161+0000 7f5d00e00640 1 -- 172.21.15.28:0/3643846880 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197b29b0 msgr2=0x55c2197bcb40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:13:17.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.161+0000 7f5d00e00640 1 --2- 172.21.15.28:0/3643846880 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197b29b0 0x55c2197bcb40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.161+0000 7f5d00e00640 1 -- 172.21.15.28:0/3643846880 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2194f27e0 con 0x55c2197b2090 2024-04-24T08:13:17.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.161+0000 7f5d01800640 1 --2- 172.21.15.28:0/3643846880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197bd080 0x55c2197bf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:13:17.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.161+0000 7f5d00e00640 1 --2- 172.21.15.28:0/3643846880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b2090 0x55c2197b2470 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00a5220 tx=0x7f5cf009c750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc96f3f7c27d3c53 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:17.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.162+0000 7f5cee000640 1 -- 172.21.15.28:0/3643846880 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf00a0e90 con 0x55c2197b2090 2024-04-24T08:13:17.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.162+0000 7f5cee000640 1 -- 172.21.15.28:0/3643846880 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cf00a5750 con 0x55c2197b2090 2024-04-24T08:13:17.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.162+0000 7f5cee000640 1 -- 172.21.15.28:0/3643846880 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf00ce510 con 0x55c2197b2090 2024-04-24T08:13:17.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.163+0000 7f5d0662c740 1 -- 172.21.15.28:0/3643846880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b2090 msgr2=0x55c2197b2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.163+0000 7f5d0662c740 1 --2- 172.21.15.28:0/3643846880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b2090 0x55c2197b2470 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f5cf00a5220 tx=0x7f5cf009c750 comp rx=0 tx=0).stop 2024-04-24T08:13:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.164+0000 7f5d0662c740 1 -- 172.21.15.28:0/3643846880 shutdown_connections 2024-04-24T08:13:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.164+0000 7f5d0662c740 1 --2- 172.21.15.28:0/3643846880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197bd080 0x55c2197bf470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.164+0000 7f5d0662c740 1 --2- 172.21.15.28:0/3643846880 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197b29b0 0x55c2197bcb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.164+0000 7f5d0662c740 1 --2- 172.21.15.28:0/3643846880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b2090 0x55c2197b2470 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.164+0000 7f5d0662c740 1 -- 172.21.15.28:0/3643846880 >> 172.21.15.28:0/3643846880 conn(0x55c2197ae3b0 msgr2=0x55c2197ae790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:17.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.164+0000 7f5d0662c740 1 -- 172.21.15.28:0/3643846880 shutdown_connections 2024-04-24T08:13:17.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.164+0000 7f5d0662c740 1 -- 172.21.15.28:0/3643846880 wait complete. 2024-04-24T08:13:17.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.164+0000 7f5d0662c740 1 -- start start 2024-04-24T08:13:17.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.165+0000 7f5d0662c740 1 Processor -- start 2024-04-24T08:13:17.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.165+0000 7f5d0662c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197b2090 0x55c2197ba080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.165+0000 7f5d0662c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b29b0 0x55c2197ba5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.165+0000 7f5d0662c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197bd080 0x55c2197bab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.165+0000 7f5d0662c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c2197bfec0 con 0x55c2197b29b0 2024-04-24T08:13:17.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.165+0000 7f5d0662c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c2196d1660 con 0x55c2197bd080 2024-04-24T08:13:17.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.165+0000 7f5d0662c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c2197c0040 con 0x55c2197b2090 2024-04-24T08:13:17.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.165+0000 7f5cfbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b29b0 0x55c2197ba5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.165+0000 7f5cfbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b29b0 0x55c2197ba5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38746/0 (socket says 172.21.15.28:38746) 2024-04-24T08:13:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.165+0000 7f5cfbe00640 1 -- 172.21.15.28:0/1127322853 learned_addr learned my addr 172.21.15.28:0/1127322853 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:17.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.165+0000 7f5d0662c740 10 client.0 _ll_register_callbacks cb 0x55c2196bfaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.166+0000 7f5d00e00640 1 --2- 172.21.15.28:0/1127322853 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197b2090 0x55c2197ba080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.166+0000 7f5d01800640 1 --2- 172.21.15.28:0/1127322853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197bd080 0x55c2197bab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.166+0000 7f5cfbe00640 1 -- 172.21.15.28:0/1127322853 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197b2090 msgr2=0x55c2197ba080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.166+0000 7f5cfbe00640 1 --2- 172.21.15.28:0/1127322853 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197b2090 0x55c2197ba080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.166+0000 7f5cfbe00640 1 -- 172.21.15.28:0/1127322853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197bd080 msgr2=0x55c2197bab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.166+0000 7f5cfbe00640 1 --2- 172.21.15.28:0/1127322853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197bd080 0x55c2197bab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.166+0000 7f5cfbe00640 1 -- 172.21.15.28:0/1127322853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c219552f70 con 0x55c2197b29b0 2024-04-24T08:13:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.167+0000 7f5cfbe00640 1 --2- 172.21.15.28:0/1127322853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b29b0 0x55c2197ba5c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8005bc0 tx=0x7f5ce8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1715a0693d56b454 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:17.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.167+0000 7f5ce2a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:13:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.167+0000 7f5ce2a00640 1 -- 172.21.15.28:0/1127322853 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce801ccb0 con 0x55c2197b29b0 2024-04-24T08:13:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.167+0000 7f5ce2a00640 1 -- 172.21.15.28:0/1127322853 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f5ce8028480 con 0x55c2197b29b0 2024-04-24T08:13:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.167+0000 7f5d0662c740 1 -- 172.21.15.28:0/1127322853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5cf00a5750 con 0x55c2197b29b0 2024-04-24T08:13:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.168+0000 7f5ce2a00640 1 -- 172.21.15.28:0/1127322853 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5ce801ce50 con 0x55c2197b29b0 2024-04-24T08:13:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.168+0000 7f5d0662c740 10 client.15834 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.168+0000 7f5d0662c740 1 -- 172.21.15.28:0/1127322853 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c2193ea170 con 0x55c2197b29b0 2024-04-24T08:13:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.168+0000 7f5ce2a00640 1 -- 172.21.15.28:0/1127322853 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5cf00a5750 con 0x55c2197b29b0 2024-04-24T08:13:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.168+0000 7f5ce2a00640 1 -- 172.21.15.28:0/1127322853 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f5ce8032860 con 0x55c2197b29b0 2024-04-24T08:13:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.168+0000 7f5d0662c740 10 client.15834 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.168+0000 7f5d0662c740 10 client.15834 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:13:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.168+0000 7f5d0662c740 20 client.15834 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:17.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.168+0000 7f5d0662c740 1 --2- 172.21.15.28:0/1127322853 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55c2197cd510 0x55c2197cf920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.168+0000 7f5d0662c740 4 client.15834 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:13:17.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.168+0000 7f5d0662c740 1 -- 172.21.15.28:0/1127322853 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c219725890 con 0x55c2197cd510 2024-04-24T08:13:17.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.169+0000 7f5d00e00640 1 --2- 172.21.15.28:0/1127322853 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55c2197cd510 0x55c2197cf920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.169+0000 7f5d00e00640 1 --2- 172.21.15.28:0/1127322853 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55c2197cd510 0x55c2197cf920 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f85f613e9a6223d server_cookie=5cfb9af4a1078fcc in_seq=0 out_seq=0 2024-04-24T08:13:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.169+0000 7f5ce2a00640 10 client.15834 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.171+0000 7f5ce2a00640 1 -- 172.21.15.28:0/1127322853 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c219725890 con 0x55c2197cd510 2024-04-24T08:13:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.171+0000 7f5ce2a00640 10 client.15834 handle_command_reply: tid=0 2024-04-24T08:13:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.171+0000 7f5d0662c740 1 client.15834 shutdown 2024-04-24T08:13:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.171+0000 7f5d0662c740 1 -- 172.21.15.28:0/1127322853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b29b0 msgr2=0x55c2197ba5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.171+0000 7f5d0662c740 1 --2- 172.21.15.28:0/1127322853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b29b0 0x55c2197ba5c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8005bc0 tx=0x7f5ce8004730 comp rx=0 tx=0).stop 2024-04-24T08:13:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.171+0000 7f5d0662c740 1 -- 172.21.15.28:0/1127322853 shutdown_connections 2024-04-24T08:13:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.171+0000 7f5d0662c740 1 --2- 172.21.15.28:0/1127322853 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55c2197cd510 0x55c2197cf920 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.171+0000 7f5d0662c740 1 --2- 172.21.15.28:0/1127322853 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b29b0 0x55c2197ba5c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.171+0000 7f5d0662c740 1 --2- 172.21.15.28:0/1127322853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197bd080 0x55c2197bab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.171+0000 7f5d0662c740 1 --2- 172.21.15.28:0/1127322853 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197b2090 0x55c2197ba080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.171+0000 7f5d0662c740 1 -- 172.21.15.28:0/1127322853 >> 172.21.15.28:0/1127322853 conn(0x55c2197ae3b0 msgr2=0x55c2197b41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.171+0000 7f5ce2a00640 0 client.15834 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.172+0000 7f5d0662c740 1 -- 172.21.15.28:0/1127322853 shutdown_connections 2024-04-24T08:13:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.172+0000 7f5d0662c740 1 -- 172.21.15.28:0/1127322853 wait complete. 2024-04-24T08:13:17.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.172+0000 7f5d0662c740 20 client.15834 trim_cache size 0 max 16384 2024-04-24T08:13:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.210+0000 7f5d0662c740 1 Processor -- start 2024-04-24T08:13:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.211+0000 7f5d0662c740 1 -- start start 2024-04-24T08:13:17.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.211+0000 7f5d0662c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197b2090 0x55c2197b8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.211+0000 7f5d0662c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197bd080 0x55c2197c8d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.211+0000 7f5d0662c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197adc90 0x55c2197b4900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.211+0000 7f5d0662c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1bf640 con 0x55c2197adc90 2024-04-24T08:13:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.211+0000 7f5d0662c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1bcf10 con 0x55c2197bd080 2024-04-24T08:13:17.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.211+0000 7f5d0662c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1a17c0 con 0x55c2197b2090 2024-04-24T08:13:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.212+0000 7f5cfbe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197bd080 0x55c2197c8d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.212+0000 7f5cfbe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197bd080 0x55c2197c8d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41022/0 (socket says 172.21.15.28:41022) 2024-04-24T08:13:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.212+0000 7f5cfbe00640 1 -- 172.21.15.28:0/627222244 learned_addr learned my addr 172.21.15.28:0/627222244 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.212+0000 7f5d00e00640 1 --2- 172.21.15.28:0/627222244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197b2090 0x55c2197b8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.212+0000 7f5cfbe00640 1 -- 172.21.15.28:0/627222244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197b2090 msgr2=0x55c2197b8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.212+0000 7f5cfbe00640 1 --2- 172.21.15.28:0/627222244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197b2090 0x55c2197b8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.212+0000 7f5cfbe00640 1 -- 172.21.15.28:0/627222244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197adc90 msgr2=0x55c2197b4900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.212+0000 7f5d01800640 1 --2- 172.21.15.28:0/627222244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197adc90 0x55c2197b4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.213+0000 7f5cfbe00640 1 --2- 172.21.15.28:0/627222244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197adc90 0x55c2197b4900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.213+0000 7f5cfbe00640 1 -- 172.21.15.28:0/627222244 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c2197b4e40 con 0x55c2197bd080 2024-04-24T08:13:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.213+0000 7f5d01800640 1 --2- 172.21.15.28:0/627222244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197adc90 0x55c2197b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:17.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.213+0000 7f5cfbe00640 1 --2- 172.21.15.28:0/627222244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197bd080 0x55c2197c8d80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8009880 tx=0x7f5ce801c790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53098e5ce49e19a8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.213+0000 7f5ced600640 1 -- 172.21.15.28:0/627222244 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce803f070 con 0x55c2197bd080 2024-04-24T08:13:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.214+0000 7f5ced600640 1 -- 172.21.15.28:0/627222244 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ce801ce50 con 0x55c2197bd080 2024-04-24T08:13:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.214+0000 7f5ced600640 1 -- 172.21.15.28:0/627222244 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ce801d940 con 0x55c2197bd080 2024-04-24T08:13:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.215+0000 7f5d0662c740 1 -- 172.21.15.28:0/627222244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197bd080 msgr2=0x55c2197c8d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.215+0000 7f5d0662c740 1 --2- 172.21.15.28:0/627222244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197bd080 0x55c2197c8d80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5ce8009880 tx=0x7f5ce801c790 comp rx=0 tx=0).stop 2024-04-24T08:13:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.215+0000 7f5d0662c740 1 -- 172.21.15.28:0/627222244 shutdown_connections 2024-04-24T08:13:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.215+0000 7f5d0662c740 1 --2- 172.21.15.28:0/627222244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197adc90 0x55c2197b4900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.215+0000 7f5d0662c740 1 --2- 172.21.15.28:0/627222244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197bd080 0x55c2197c8d80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.215+0000 7f5d0662c740 1 --2- 172.21.15.28:0/627222244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197b2090 0x55c2197b8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.216+0000 7f5d0662c740 1 -- 172.21.15.28:0/627222244 >> 172.21.15.28:0/627222244 conn(0x55c2197ae3b0 msgr2=0x55c2197ba8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:17.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.216+0000 7f5d0662c740 1 -- 172.21.15.28:0/627222244 shutdown_connections 2024-04-24T08:13:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.216+0000 7f5d0662c740 1 -- 172.21.15.28:0/627222244 wait complete. 2024-04-24T08:13:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.217+0000 7f5d0662c740 1 -- start start 2024-04-24T08:13:17.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.217+0000 7f5d0662c740 1 Processor -- start 2024-04-24T08:13:17.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.218+0000 7f5d0662c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197adc90 0x55c2197c5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.218+0000 7f5d0662c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b2090 0x55c2197c57d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.218+0000 7f5d0662c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197bd080 0x55c21982d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.219+0000 7f5d0662c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1a1e30 con 0x55c2197b2090 2024-04-24T08:13:17.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.219+0000 7f5d0662c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1bc840 con 0x55c2197adc90 2024-04-24T08:13:17.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.219+0000 7f5d0662c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5cfc1a2500 con 0x55c2197bd080 2024-04-24T08:13:17.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.219+0000 7f5d0662c740 10 client.0 _ll_register_callbacks cb 0x7f5ce80097b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.219+0000 7f5d01800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197bd080 0x55c21982d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.219+0000 7f5d00e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197adc90 0x55c2197c5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.219+0000 7f5d01800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197bd080 0x55c21982d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:57126/0 (socket says 172.21.15.28:57126) 2024-04-24T08:13:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.219+0000 7f5d01800640 1 -- 172.21.15.28:0/825570872 learned_addr learned my addr 172.21.15.28:0/825570872 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:17.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.220+0000 7f5cfbe00640 1 --2- 172.21.15.28:0/825570872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b2090 0x55c2197c57d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.220+0000 7f5d01800640 1 -- 172.21.15.28:0/825570872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197adc90 msgr2=0x55c2197c5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.220+0000 7f5d01800640 1 --2- 172.21.15.28:0/825570872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197adc90 0x55c2197c5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.220+0000 7f5d01800640 1 -- 172.21.15.28:0/825570872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b2090 msgr2=0x55c2197c57d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.220+0000 7f5d01800640 1 --2- 172.21.15.28:0/825570872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b2090 0x55c2197c57d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.220+0000 7f5d01800640 1 -- 172.21.15.28:0/825570872 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c21982d8b0 con 0x55c2197bd080 2024-04-24T08:13:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.220+0000 7f5d01800640 1 --2- 172.21.15.28:0/825570872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197bd080 0x55c21982d0d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4005cc0 tx=0x7f5cf402f460 comp rx=0 tx=0).ready entity=mon.1 client_cookie=87a725d4beaf039d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:17.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.221+0000 7f5ce2a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:13:17.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.221+0000 7f5ce2a00640 1 -- 172.21.15.28:0/825570872 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5cf4003b80 con 0x55c2197bd080 2024-04-24T08:13:17.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.221+0000 7f5ce2a00640 1 -- 172.21.15.28:0/825570872 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f5cf4003d20 con 0x55c2197bd080 2024-04-24T08:13:17.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.221+0000 7f5d0662c740 1 -- 172.21.15.28:0/825570872 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5ce801ce50 con 0x55c2197bd080 2024-04-24T08:13:17.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.222+0000 7f5ce2a00640 1 -- 172.21.15.28:0/825570872 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5cf4031450 con 0x55c2197bd080 2024-04-24T08:13:17.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.222+0000 7f5d0662c740 10 client.26011 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:17.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.222+0000 7f5d0662c740 1 -- 172.21.15.28:0/825570872 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c219845730 con 0x55c2197bd080 2024-04-24T08:13:17.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.222+0000 7f5ce2a00640 1 -- 172.21.15.28:0/825570872 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ce801ce50 con 0x55c2197bd080 2024-04-24T08:13:17.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.222+0000 7f5ce2a00640 1 -- 172.21.15.28:0/825570872 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f5cf402f610 con 0x55c2197bd080 2024-04-24T08:13:17.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.225+0000 7f5d0662c740 10 client.26011 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:17.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.225+0000 7f5d0662c740 10 client.26011 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:13:17.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.225+0000 7f5d0662c740 20 client.26011 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:17.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.225+0000 7f5d0662c740 1 --2- 172.21.15.28:0/825570872 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55c219845e90 0x55c219848280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.225+0000 7f5d0662c740 4 client.26011 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:13:17.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.225+0000 7f5d0662c740 1 -- 172.21.15.28:0/825570872 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c21971f700 con 0x55c219845e90 2024-04-24T08:13:17.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.227+0000 7f5d00e00640 1 --2- 172.21.15.28:0/825570872 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55c219845e90 0x55c219848280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.229+0000 7f5d00e00640 1 --2- 172.21.15.28:0/825570872 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55c219845e90 0x55c219848280 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=54ccd0bcd2066b37 server_cookie=4a3770ddd2c94b40 in_seq=0 out_seq=0 2024-04-24T08:13:17.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.229+0000 7f5ce2a00640 10 client.26011 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.230+0000 7f5ce2a00640 1 -- 172.21.15.28:0/825570872 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c21971f700 con 0x55c219845e90 2024-04-24T08:13:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.230+0000 7f5ce2a00640 10 client.26011 handle_command_reply: tid=0 2024-04-24T08:13:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.231+0000 7f5d0662c740 1 client.26011 shutdown 2024-04-24T08:13:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.231+0000 7f5d0662c740 1 -- 172.21.15.28:0/825570872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197bd080 msgr2=0x55c21982d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.231+0000 7f5d0662c740 1 --2- 172.21.15.28:0/825570872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197bd080 0x55c21982d0d0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f5cf4005cc0 tx=0x7f5cf402f460 comp rx=0 tx=0).stop 2024-04-24T08:13:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.231+0000 7f5d0662c740 1 -- 172.21.15.28:0/825570872 shutdown_connections 2024-04-24T08:13:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.231+0000 7f5d0662c740 1 --2- 172.21.15.28:0/825570872 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55c219845e90 0x55c219848280 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.232+0000 7f5d0662c740 1 --2- 172.21.15.28:0/825570872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c2197b2090 0x55c2197c57d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.232+0000 7f5d0662c740 1 --2- 172.21.15.28:0/825570872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c2197bd080 0x55c21982d0d0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.232+0000 7f5d0662c740 1 --2- 172.21.15.28:0/825570872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c2197adc90 0x55c2197c5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.232+0000 7f5d0662c740 1 -- 172.21.15.28:0/825570872 >> 172.21.15.28:0/825570872 conn(0x55c2197ae3b0 msgr2=0x55c2197acc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:17.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.232+0000 7f5ce2a00640 0 client.26011 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.232+0000 7f5d0662c740 1 -- 172.21.15.28:0/825570872 shutdown_connections 2024-04-24T08:13:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.233+0000 7f5d0662c740 1 -- 172.21.15.28:0/825570872 wait complete. 2024-04-24T08:13:17.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.235+0000 7f5d0662c740 20 client.26011 trim_cache size 0 max 16384 2024-04-24T08:13:17.235 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:13:17.235 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-24T08:13:17.235 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "4683be7f-73e1-4643-a6d4-9e63a3a2b18c", 2024-04-24T08:13:17.235 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-24T08:13:17.235 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:13:17.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.236+0000 7f5d03800640 1 -- 172.21.15.28:0/2943898412 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5ce40a3da0 msgr2=0x7f5ce40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.236+0000 7f5d03800640 1 --2- 172.21.15.28:0/2943898412 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5ce40a3da0 0x7f5ce40a6260 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f5ce80095f0 tx=0x7f5ce8005f90 comp rx=0 tx=0).stop 2024-04-24T08:13:17.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.236+0000 7f5d03800640 1 -- 172.21.15.28:0/2943898412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cfc1044b0 msgr2=0x7f5cfc076510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.236+0000 7f5d03800640 1 --2- 172.21.15.28:0/2943898412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cfc1044b0 0x7f5cfc076510 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f5cf000ea20 tx=0x7f5cf000eee0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.236+0000 7f5d03800640 1 -- 172.21.15.28:0/2943898412 shutdown_connections 2024-04-24T08:13:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.236+0000 7f5d03800640 1 --2- 172.21.15.28:0/2943898412 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5ce40a3da0 0x7f5ce40a6260 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.236+0000 7f5d03800640 1 --2- 172.21.15.28:0/2943898412 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5cfc109580 0x7f5cfc076a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.236+0000 7f5d03800640 1 --2- 172.21.15.28:0/2943898412 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5cfc076f90 0x7f5cfc1ab0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.236+0000 7f5d03800640 1 --2- 172.21.15.28:0/2943898412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5cfc1044b0 0x7f5cfc076510 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.236+0000 7f5d03800640 1 -- 172.21.15.28:0/2943898412 >> 172.21.15.28:0/2943898412 conn(0x7f5cfc0ffd90 msgr2=0x7f5cfc1008d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.240+0000 7f5d03800640 1 -- 172.21.15.28:0/2943898412 shutdown_connections 2024-04-24T08:13:17.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.240+0000 7f5d03800640 1 -- 172.21.15.28:0/2943898412 wait complete. 2024-04-24T08:13:17.299 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:13:17.299 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:13:17.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.574+0000 7fe194a00640 1 -- 172.21.15.28:0/3071543293 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe190101720 msgr2=0x7fe19010e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.574+0000 7fe194a00640 1 --2- 172.21.15.28:0/3071543293 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe190101720 0x7fe19010e470 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fe178009900 tx=0x7fe17802a3f0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.575+0000 7fe194a00640 1 -- 172.21.15.28:0/3071543293 shutdown_connections 2024-04-24T08:13:17.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.575+0000 7fe194a00640 1 --2- 172.21.15.28:0/3071543293 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe19010e9b0 0x7fe190110da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.575+0000 7fe194a00640 1 --2- 172.21.15.28:0/3071543293 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe190101720 0x7fe19010e470 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.575+0000 7fe194a00640 1 --2- 172.21.15.28:0/3071543293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe190100e00 0x7fe1901011e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.575+0000 7fe194a00640 1 -- 172.21.15.28:0/3071543293 >> 172.21.15.28:0/3071543293 conn(0x7fe190078000 msgr2=0x7fe190078410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:17.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.575+0000 7fe194a00640 1 -- 172.21.15.28:0/3071543293 shutdown_connections 2024-04-24T08:13:17.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.575+0000 7fe194a00640 1 -- 172.21.15.28:0/3071543293 wait complete. 2024-04-24T08:13:17.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.576+0000 7fe194a00640 1 Processor -- start 2024-04-24T08:13:17.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.577+0000 7fe194a00640 1 -- start start 2024-04-24T08:13:17.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.577+0000 7fe194a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe190100e00 0x7fe1901a0760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.577+0000 7fe194a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe190101720 0x7fe1901a0ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.577+0000 7fe194a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe19010e9b0 0x7fe19019a910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.577+0000 7fe194a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe190114630 con 0x7fe190101720 2024-04-24T08:13:17.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.577+0000 7fe194a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe1901144b0 con 0x7fe19010e9b0 2024-04-24T08:13:17.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.577+0000 7fe194a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe1901147b0 con 0x7fe190100e00 2024-04-24T08:13:17.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.577+0000 7fe18e800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe19010e9b0 0x7fe19019a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.577+0000 7fe18e800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe19010e9b0 0x7fe19019a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41036/0 (socket says 172.21.15.28:41036) 2024-04-24T08:13:17.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.577+0000 7fe18e800640 1 -- 172.21.15.28:0/3891194181 learned_addr learned my addr 172.21.15.28:0/3891194181 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:17.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.577+0000 7fe187e00640 1 --2- 172.21.15.28:0/3891194181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe190101720 0x7fe1901a0ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.578+0000 7fe18e800640 1 -- 172.21.15.28:0/3891194181 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe190100e00 msgr2=0x7fe1901a0760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.578+0000 7fe18e800640 1 --2- 172.21.15.28:0/3891194181 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe190100e00 0x7fe1901a0760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.578+0000 7fe18e800640 1 -- 172.21.15.28:0/3891194181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe190101720 msgr2=0x7fe1901a0ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.578+0000 7fe18e800640 1 --2- 172.21.15.28:0/3891194181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe190101720 0x7fe1901a0ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.578+0000 7fe18e800640 1 -- 172.21.15.28:0/3891194181 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe19019b060 con 0x7fe19010e9b0 2024-04-24T08:13:17.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.578+0000 7fe187e00640 1 --2- 172.21.15.28:0/3891194181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe190101720 0x7fe1901a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:13:17.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.578+0000 7fe18e800640 1 --2- 172.21.15.28:0/3891194181 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe19010e9b0 0x7fe19019a910 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe18800eb70 tx=0x7fe18800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:17.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.578+0000 7fe186a00640 1 -- 172.21.15.28:0/3891194181 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe188004d20 con 0x7fe19010e9b0 2024-04-24T08:13:17.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.578+0000 7fe194a00640 1 -- 172.21.15.28:0/3891194181 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe19019b350 con 0x7fe19010e9b0 2024-04-24T08:13:17.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.579+0000 7fe186a00640 1 -- 172.21.15.28:0/3891194181 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe18800ce70 con 0x7fe19010e9b0 2024-04-24T08:13:17.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.579+0000 7fe194a00640 1 -- 172.21.15.28:0/3891194181 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe19006ba20 con 0x7fe19010e9b0 2024-04-24T08:13:17.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.580+0000 7fe186a00640 1 -- 172.21.15.28:0/3891194181 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe188005220 con 0x7fe19010e9b0 2024-04-24T08:13:17.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.585+0000 7fe186a00640 1 -- 172.21.15.28:0/3891194181 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe1880053c0 con 0x7fe19010e9b0 2024-04-24T08:13:17.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.586+0000 7fe1979d9740 1 Processor -- start 2024-04-24T08:13:17.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.586+0000 7fe1979d9740 1 -- start start 2024-04-24T08:13:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.587+0000 7fe1979d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 0x55a84c08f990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.587+0000 7fe1979d9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 0x55a84c0a2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.587+0000 7fe1979d9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c0a3060 0x55a84c0a5450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.587+0000 7fe1979d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a84bff51f0 con 0x55a84c08f5b0 2024-04-24T08:13:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.587+0000 7fe1979d9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a84bf9b350 con 0x55a84c08ff60 2024-04-24T08:13:17.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.587+0000 7fe1979d9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a84bff11b0 con 0x55a84c0a3060 2024-04-24T08:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.588+0000 7fe187e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 0x55a84c08f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.588+0000 7fe18de00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 0x55a84c0a2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.588+0000 7fe187e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 0x55a84c08f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38772/0 (socket says 172.21.15.28:38772) 2024-04-24T08:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.588+0000 7fe187e00640 1 -- 172.21.15.28:0/1303926273 learned_addr learned my addr 172.21.15.28:0/1303926273 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.589+0000 7fe186a00640 1 --2- 172.21.15.28:0/3891194181 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe1740a3d60 0x7fe1740a6220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.589+0000 7fe187e00640 1 --2- 172.21.15.28:0/1303926273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c0a3060 0x55a84c0a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.589+0000 7fe18de00640 1 -- 172.21.15.28:0/1303926273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c0a3060 msgr2=0x55a84c0a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.589+0000 7fe18de00640 1 --2- 172.21.15.28:0/1303926273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c0a3060 0x55a84c0a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.589+0000 7fe18de00640 1 -- 172.21.15.28:0/1303926273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 msgr2=0x55a84c08f990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.589+0000 7fe18e800640 1 --2- 172.21.15.28:0/3891194181 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe1740a3d60 0x7fe1740a6220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.589+0000 7fe18de00640 1 --2- 172.21.15.28:0/1303926273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 0x55a84c08f990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.589+0000 7fe186a00640 1 -- 172.21.15.28:0/3891194181 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fe1880d0550 con 0x7fe19010e9b0 2024-04-24T08:13:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.590+0000 7fe18de00640 1 -- 172.21.15.28:0/1303926273 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a84bfeff50 con 0x55a84c08ff60 2024-04-24T08:13:17.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.590+0000 7fe187e00640 1 --2- 172.21.15.28:0/1303926273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 0x55a84c08f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:13:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.590+0000 7fe18e800640 1 --2- 172.21.15.28:0/3891194181 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe1740a3d60 0x7fe1740a6220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe1880cdd30 tx=0x7fe188066b60 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.591+0000 7fe18de00640 1 --2- 172.21.15.28:0/1303926273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 0x55a84c0a2b20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe180009b10 tx=0x7fe18000e350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=424c908b581a33c8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.591+0000 7fe17f000640 1 -- 172.21.15.28:0/1303926273 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18000ed50 con 0x55a84c08ff60 2024-04-24T08:13:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.591+0000 7fe17f000640 1 -- 172.21.15.28:0/1303926273 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe180005020 con 0x55a84c08ff60 2024-04-24T08:13:17.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.591+0000 7fe17f000640 1 -- 172.21.15.28:0/1303926273 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe180004b60 con 0x55a84c08ff60 2024-04-24T08:13:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.592+0000 7fe1979d9740 1 -- 172.21.15.28:0/1303926273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 msgr2=0x55a84c0a2b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.592+0000 7fe1979d9740 1 --2- 172.21.15.28:0/1303926273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 0x55a84c0a2b20 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fe180009b10 tx=0x7fe18000e350 comp rx=0 tx=0).stop 2024-04-24T08:13:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.592+0000 7fe1979d9740 1 -- 172.21.15.28:0/1303926273 shutdown_connections 2024-04-24T08:13:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.592+0000 7fe1979d9740 1 --2- 172.21.15.28:0/1303926273 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c0a3060 0x55a84c0a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.592+0000 7fe1979d9740 1 --2- 172.21.15.28:0/1303926273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 0x55a84c0a2b20 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.592+0000 7fe1979d9740 1 --2- 172.21.15.28:0/1303926273 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 0x55a84c08f990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.592+0000 7fe1979d9740 1 -- 172.21.15.28:0/1303926273 >> 172.21.15.28:0/1303926273 conn(0x55a84c08bb80 msgr2=0x55a84c08bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.593+0000 7fe1979d9740 1 -- 172.21.15.28:0/1303926273 shutdown_connections 2024-04-24T08:13:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.593+0000 7fe1979d9740 1 -- 172.21.15.28:0/1303926273 wait complete. 2024-04-24T08:13:17.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.593+0000 7fe1979d9740 1 -- start start 2024-04-24T08:13:17.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.593+0000 7fe1979d9740 1 Processor -- start 2024-04-24T08:13:17.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.594+0000 7fe1979d9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c08f5b0 0x55a84c09ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.594+0000 7fe1979d9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 0x55a84c0a29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.594+0000 7fe1979d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c0a3060 0x55a84c09d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.594+0000 7fe1979d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55a84c0926b0 con 0x55a84c0a3060 2024-04-24T08:13:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.594+0000 7fe1979d9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55a84bfae690 con 0x55a84c08ff60 2024-04-24T08:13:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.594+0000 7fe1979d9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55a84c092830 con 0x55a84c08f5b0 2024-04-24T08:13:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.594+0000 7fe1979d9740 10 client.0 _ll_register_callbacks cb 0x55a84bfea650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.594+0000 7fe18de00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 0x55a84c0a29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.594+0000 7fe18de00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 0x55a84c0a29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41052/0 (socket says 172.21.15.28:41052) 2024-04-24T08:13:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.594+0000 7fe18de00640 1 -- 172.21.15.28:0/3858200594 learned_addr learned my addr 172.21.15.28:0/3858200594 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.594+0000 7fe187e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c08f5b0 0x55a84c09ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.595+0000 7fe187e00640 1 --2- 172.21.15.28:0/3858200594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c0a3060 0x55a84c09d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.595+0000 7fe18de00640 1 -- 172.21.15.28:0/3858200594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c08f5b0 msgr2=0x55a84c09ccc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.595+0000 7fe18de00640 1 --2- 172.21.15.28:0/3858200594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c08f5b0 0x55a84c09ccc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.595+0000 7fe18de00640 1 -- 172.21.15.28:0/3858200594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c0a3060 msgr2=0x55a84c09d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:17.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.595+0000 7fe18de00640 1 --2- 172.21.15.28:0/3858200594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c0a3060 0x55a84c09d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.595+0000 7fe18de00640 1 -- 172.21.15.28:0/3858200594 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a84bfad920 con 0x55a84c08ff60 2024-04-24T08:13:17.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.595+0000 7fe18de00640 1 --2- 172.21.15.28:0/3858200594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 0x55a84c0a29e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe180005730 tx=0x7fe180005480 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dcbb4ca8a66216c2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:17.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.595+0000 7fe173400640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:13:17.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.595+0000 7fe173400640 1 -- 172.21.15.28:0/3858200594 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe180004cb0 con 0x55a84c08ff60 2024-04-24T08:13:17.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.596+0000 7fe1979d9740 1 -- 172.21.15.28:0/3858200594 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe180005020 con 0x55a84c08ff60 2024-04-24T08:13:17.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.596+0000 7fe173400640 1 -- 172.21.15.28:0/3858200594 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fe180018470 con 0x55a84c08ff60 2024-04-24T08:13:17.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.596+0000 7fe173400640 1 -- 172.21.15.28:0/3858200594 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe180004e50 con 0x55a84c08ff60 2024-04-24T08:13:17.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.596+0000 7fe1979d9740 10 client.25973 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:17.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.596+0000 7fe1979d9740 1 -- 172.21.15.28:0/3858200594 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a84bdd17d0 con 0x55a84c08ff60 2024-04-24T08:13:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.596+0000 7fe173400640 1 -- 172.21.15.28:0/3858200594 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe180005020 con 0x55a84c08ff60 2024-04-24T08:13:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.597+0000 7fe173400640 1 -- 172.21.15.28:0/3858200594 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fe180010700 con 0x55a84c08ff60 2024-04-24T08:13:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.597+0000 7fe1979d9740 10 client.25973 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.597+0000 7fe1979d9740 10 client.25973 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:13:17.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.597+0000 7fe1979d9740 20 client.25973 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.597+0000 7fe1979d9740 1 --2- 172.21.15.28:0/3858200594 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a84c0bbb00 0x55a84c0bdef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.597+0000 7fe1979d9740 4 client.25973 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:13:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.597+0000 7fe1979d9740 1 -- 172.21.15.28:0/3858200594 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a84c002c10 con 0x55a84c0bbb00 2024-04-24T08:13:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.597+0000 7fe187e00640 1 --2- 172.21.15.28:0/3858200594 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a84c0bbb00 0x55a84c0bdef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.598+0000 7fe187e00640 1 --2- 172.21.15.28:0/3858200594 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a84c0bbb00 0x55a84c0bdef0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f8ef8f1a8b8e283 server_cookie=20ad6b96302f802d in_seq=0 out_seq=0 2024-04-24T08:13:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.598+0000 7fe173400640 10 client.25973 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.598+0000 7fe173400640 1 -- 172.21.15.28:0/3858200594 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a84c002c10 con 0x55a84c0bbb00 2024-04-24T08:13:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.598+0000 7fe173400640 10 client.25973 handle_command_reply: tid=0 2024-04-24T08:13:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.598+0000 7fe1979d9740 1 client.25973 shutdown 2024-04-24T08:13:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.599+0000 7fe1979d9740 1 -- 172.21.15.28:0/3858200594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 msgr2=0x55a84c0a29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.599+0000 7fe1979d9740 1 --2- 172.21.15.28:0/3858200594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 0x55a84c0a29e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe180005730 tx=0x7fe180005480 comp rx=0 tx=0).stop 2024-04-24T08:13:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.599+0000 7fe1979d9740 1 -- 172.21.15.28:0/3858200594 shutdown_connections 2024-04-24T08:13:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.599+0000 7fe1979d9740 1 --2- 172.21.15.28:0/3858200594 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a84c0bbb00 0x55a84c0bdef0 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.599+0000 7fe1979d9740 1 --2- 172.21.15.28:0/3858200594 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c0a3060 0x55a84c09d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.599+0000 7fe1979d9740 1 --2- 172.21.15.28:0/3858200594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 0x55a84c0a29e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.599+0000 7fe1979d9740 1 --2- 172.21.15.28:0/3858200594 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c08f5b0 0x55a84c09ccc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.599+0000 7fe173400640 0 client.25973 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.599+0000 7fe1979d9740 1 -- 172.21.15.28:0/3858200594 >> 172.21.15.28:0/3858200594 conn(0x55a84c08bb80 msgr2=0x55a84c091980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.599+0000 7fe1979d9740 1 -- 172.21.15.28:0/3858200594 shutdown_connections 2024-04-24T08:13:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.599+0000 7fe1979d9740 1 -- 172.21.15.28:0/3858200594 wait complete. 2024-04-24T08:13:17.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.599+0000 7fe1979d9740 20 client.25973 trim_cache size 0 max 16384 2024-04-24T08:13:17.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.649+0000 7fe1979d9740 1 Processor -- start 2024-04-24T08:13:17.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.650+0000 7fe1979d9740 1 -- start start 2024-04-24T08:13:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.650+0000 7fe1979d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 0x55a84c09c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.650+0000 7fe1979d9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c08ff60 0x55a84c0c36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.650+0000 7fe1979d9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c0a3060 0x55a84c0c3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.650+0000 7fe1979d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe190072980 con 0x55a84c08f5b0 2024-04-24T08:13:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.650+0000 7fe1979d9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe190070250 con 0x55a84c0a3060 2024-04-24T08:13:17.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.650+0000 7fe1979d9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe19006e0a0 con 0x55a84c08ff60 2024-04-24T08:13:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.651+0000 7fe18de00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c08ff60 0x55a84c0c36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.651+0000 7fe18de00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c08ff60 0x55a84c0c36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:57170/0 (socket says 172.21.15.28:57170) 2024-04-24T08:13:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.651+0000 7fe187e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c0a3060 0x55a84c0c3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.651+0000 7fe187e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c0a3060 0x55a84c0c3c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41066/0 (socket says 172.21.15.28:41066) 2024-04-24T08:13:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.651+0000 7fe187e00640 1 -- 172.21.15.28:0/3507273147 learned_addr learned my addr 172.21.15.28:0/3507273147 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.652+0000 7fe187e00640 1 -- 172.21.15.28:0/3507273147 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c08ff60 msgr2=0x55a84c0c36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:17.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.652+0000 7fe187e00640 1 --2- 172.21.15.28:0/3507273147 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c08ff60 0x55a84c0c36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.652+0000 7fe187e00640 1 -- 172.21.15.28:0/3507273147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 msgr2=0x55a84c09c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:17.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.652+0000 7fe187e00640 1 --2- 172.21.15.28:0/3507273147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 0x55a84c09c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.652+0000 7fe187e00640 1 -- 172.21.15.28:0/3507273147 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a84bd073f0 con 0x55a84c0a3060 2024-04-24T08:13:17.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.652+0000 7fe187e00640 1 --2- 172.21.15.28:0/3507273147 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c0a3060 0x55a84c0c3c10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe178006010 tx=0x7fe178031420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=70d2625d45280380 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.656+0000 7fe17e600640 1 -- 172.21.15.28:0/3507273147 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe178035040 con 0x55a84c0a3060 2024-04-24T08:13:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.656+0000 7fe17e600640 1 -- 172.21.15.28:0/3507273147 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe178031690 con 0x55a84c0a3060 2024-04-24T08:13:17.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.656+0000 7fe17e600640 1 -- 172.21.15.28:0/3507273147 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe17804d940 con 0x55a84c0a3060 2024-04-24T08:13:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.657+0000 7fe1979d9740 1 -- 172.21.15.28:0/3507273147 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c0a3060 msgr2=0x55a84c0c3c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.657+0000 7fe1979d9740 1 --2- 172.21.15.28:0/3507273147 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c0a3060 0x55a84c0c3c10 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe178006010 tx=0x7fe178031420 comp rx=0 tx=0).stop 2024-04-24T08:13:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.657+0000 7fe1979d9740 1 -- 172.21.15.28:0/3507273147 shutdown_connections 2024-04-24T08:13:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.657+0000 7fe1979d9740 1 --2- 172.21.15.28:0/3507273147 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c0a3060 0x55a84c0c3c10 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.657+0000 7fe1979d9740 1 --2- 172.21.15.28:0/3507273147 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c08ff60 0x55a84c0c36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.657+0000 7fe1979d9740 1 --2- 172.21.15.28:0/3507273147 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 0x55a84c09c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.657+0000 7fe1979d9740 1 -- 172.21.15.28:0/3507273147 >> 172.21.15.28:0/3507273147 conn(0x55a84c08bb80 msgr2=0x55a84c0910f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.657+0000 7fe1979d9740 1 -- 172.21.15.28:0/3507273147 shutdown_connections 2024-04-24T08:13:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.658+0000 7fe1979d9740 1 -- 172.21.15.28:0/3507273147 wait complete. 2024-04-24T08:13:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.658+0000 7fe1979d9740 1 -- start start 2024-04-24T08:13:17.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.659+0000 7fe1979d9740 1 Processor -- start 2024-04-24T08:13:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.659+0000 7fe1979d9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 0x55a84c094dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.659+0000 7fe1979d9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 0x55a84c095300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.659+0000 7fe1979d9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c0a3060 0x55a84c095840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.659+0000 7fe1979d9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe19006e710 con 0x55a84c08f5b0 2024-04-24T08:13:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.659+0000 7fe1979d9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe19006fb80 con 0x55a84c08ff60 2024-04-24T08:13:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.659+0000 7fe1979d9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe19006ede0 con 0x55a84c0a3060 2024-04-24T08:13:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.659+0000 7fe1979d9740 10 client.0 _ll_register_callbacks cb 0x7fe1880093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:17.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.660+0000 7fe187e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 0x55a84c094dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.660+0000 7fe187e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 0x55a84c094dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:38794/0 (socket says 172.21.15.28:38794) 2024-04-24T08:13:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.660+0000 7fe187e00640 1 -- 172.21.15.28:0/472619403 learned_addr learned my addr 172.21.15.28:0/472619403 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.660+0000 7fe187e00640 1 --2- 172.21.15.28:0/472619403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c0a3060 0x55a84c095840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.660+0000 7fe18de00640 1 --2- 172.21.15.28:0/472619403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 0x55a84c095300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.660+0000 7fe18de00640 1 -- 172.21.15.28:0/472619403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c0a3060 msgr2=0x55a84c095840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.660+0000 7fe18de00640 1 --2- 172.21.15.28:0/472619403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c0a3060 0x55a84c095840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.660+0000 7fe18de00640 1 -- 172.21.15.28:0/472619403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 msgr2=0x55a84c094dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.660+0000 7fe18de00640 1 --2- 172.21.15.28:0/472619403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 0x55a84c094dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.660+0000 7fe18de00640 1 -- 172.21.15.28:0/472619403 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a84bcc70f0 con 0x55a84c08ff60 2024-04-24T08:13:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.660+0000 7fe187e00640 1 --2- 172.21.15.28:0/472619403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 0x55a84c094dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:13:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.661+0000 7fe18de00640 1 --2- 172.21.15.28:0/472619403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 0x55a84c095300 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe18000a010 tx=0x7fe180024470 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be5781fb4cd215a2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:17.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.661+0000 7fe173400640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:13:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.661+0000 7fe173400640 1 -- 172.21.15.28:0/472619403 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe18000e850 con 0x55a84c08ff60 2024-04-24T08:13:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.661+0000 7fe173400640 1 -- 172.21.15.28:0/472619403 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fe180004ab0 con 0x55a84c08ff60 2024-04-24T08:13:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.661+0000 7fe1979d9740 1 -- 172.21.15.28:0/472619403 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe178031690 con 0x55a84c08ff60 2024-04-24T08:13:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.662+0000 7fe173400640 1 -- 172.21.15.28:0/472619403 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe180010430 con 0x55a84c08ff60 2024-04-24T08:13:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.662+0000 7fe1979d9740 10 client.25979 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.662+0000 7fe1979d9740 1 -- 172.21.15.28:0/472619403 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a84bfba810 con 0x55a84c08ff60 2024-04-24T08:13:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.663+0000 7fe173400640 1 -- 172.21.15.28:0/472619403 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe180004510 con 0x55a84c08ff60 2024-04-24T08:13:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.663+0000 7fe173400640 1 -- 172.21.15.28:0/472619403 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fe180003960 con 0x55a84c08ff60 2024-04-24T08:13:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.663+0000 7fe1979d9740 10 client.25979 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.663+0000 7fe1979d9740 10 client.25979 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:13:17.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.663+0000 7fe1979d9740 20 client.25979 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.663+0000 7fe1979d9740 1 --2- 172.21.15.28:0/472619403 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a84c121fc0 0x55a84c1243e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.663+0000 7fe1979d9740 4 client.25979 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:13:17.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.663+0000 7fe1979d9740 1 -- 172.21.15.28:0/472619403 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fe178031690 con 0x55a84c121fc0 2024-04-24T08:13:17.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.664+0000 7fe187e00640 1 --2- 172.21.15.28:0/472619403 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a84c121fc0 0x55a84c1243e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:17.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.665+0000 7fe187e00640 1 --2- 172.21.15.28:0/472619403 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a84c121fc0 0x55a84c1243e0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26d7dd50b0a6d5e9 server_cookie=d8eb9bb9c4cea0c3 in_seq=0 out_seq=0 2024-04-24T08:13:17.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.665+0000 7fe173400640 10 client.25979 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:17.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.665+0000 7fe173400640 1 -- 172.21.15.28:0/472619403 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7fe178031690 con 0x55a84c121fc0 2024-04-24T08:13:17.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.665+0000 7fe173400640 10 client.25979 handle_command_reply: tid=0 2024-04-24T08:13:17.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.665+0000 7fe1979d9740 1 client.25979 shutdown 2024-04-24T08:13:17.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.665+0000 7fe1979d9740 1 -- 172.21.15.28:0/472619403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 msgr2=0x55a84c095300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.665+0000 7fe1979d9740 1 --2- 172.21.15.28:0/472619403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 0x55a84c095300 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fe18000a010 tx=0x7fe180024470 comp rx=0 tx=0).stop 2024-04-24T08:13:17.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.665+0000 7fe1979d9740 1 -- 172.21.15.28:0/472619403 shutdown_connections 2024-04-24T08:13:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.665+0000 7fe1979d9740 1 --2- 172.21.15.28:0/472619403 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55a84c121fc0 0x55a84c1243e0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.665+0000 7fe1979d9740 1 --2- 172.21.15.28:0/472619403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55a84c0a3060 0x55a84c095840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.665+0000 7fe1979d9740 1 --2- 172.21.15.28:0/472619403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55a84c08ff60 0x55a84c095300 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.665+0000 7fe1979d9740 1 --2- 172.21.15.28:0/472619403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55a84c08f5b0 0x55a84c094dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.665+0000 7fe1979d9740 1 -- 172.21.15.28:0/472619403 >> 172.21.15.28:0/472619403 conn(0x55a84c08bb80 msgr2=0x55a84c091980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.665+0000 7fe173400640 0 client.25979 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.666+0000 7fe1979d9740 1 -- 172.21.15.28:0/472619403 shutdown_connections 2024-04-24T08:13:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.666+0000 7fe1979d9740 1 -- 172.21.15.28:0/472619403 wait complete. 2024-04-24T08:13:17.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.666+0000 7fe1979d9740 20 client.25979 trim_cache size 0 max 16384 2024-04-24T08:13:17.668 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:13:17.668 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:13:17.668 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:13:17.669 INFO:teuthology.orchestra.run.smithi028.stdout: "4683be7f-73e1-4643-a6d4-9e63a3a2b18c": { 2024-04-24T08:13:17.669 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:13:17.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "4683be7f-73e1-4643-a6d4-9e63a3a2b18c", 2024-04-24T08:13:17.669 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:13:17.669 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:13:17.669 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:13:17.669 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:13:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.666+0000 7fe194a00640 1 -- 172.21.15.28:0/3891194181 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe1740a3d60 msgr2=0x7fe1740a6220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.666+0000 7fe194a00640 1 --2- 172.21.15.28:0/3891194181 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe1740a3d60 0x7fe1740a6220 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fe1880cdd30 tx=0x7fe188066b60 comp rx=0 tx=0).stop 2024-04-24T08:13:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.666+0000 7fe194a00640 1 -- 172.21.15.28:0/3891194181 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe19010e9b0 msgr2=0x7fe19019a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.666+0000 7fe194a00640 1 --2- 172.21.15.28:0/3891194181 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe19010e9b0 0x7fe19019a910 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fe18800eb70 tx=0x7fe18800c6a0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.667+0000 7fe194a00640 1 -- 172.21.15.28:0/3891194181 shutdown_connections 2024-04-24T08:13:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.667+0000 7fe194a00640 1 --2- 172.21.15.28:0/3891194181 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe1740a3d60 0x7fe1740a6220 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.667+0000 7fe194a00640 1 --2- 172.21.15.28:0/3891194181 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe190101720 0x7fe1901a0ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.667+0000 7fe194a00640 1 --2- 172.21.15.28:0/3891194181 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe19010e9b0 0x7fe19019a910 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.667+0000 7fe194a00640 1 --2- 172.21.15.28:0/3891194181 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe190100e00 0x7fe1901a0760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.667+0000 7fe194a00640 1 -- 172.21.15.28:0/3891194181 >> 172.21.15.28:0/3891194181 conn(0x7fe190078000 msgr2=0x7fe19010fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.667+0000 7fe194a00640 1 -- 172.21.15.28:0/3891194181 shutdown_connections 2024-04-24T08:13:17.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:17.667+0000 7fe194a00640 1 -- 172.21.15.28:0/3891194181 wait complete. 2024-04-24T08:13:17.685 INFO:tasks.cephfs.filesystem:scrub status for tag:4683be7f-73e1-4643-a6d4-9e63a3a2b18c - {'path': '/', 'tag': '4683be7f-73e1-4643-a6d4-9e63a3a2b18c', 'options': 'recursive,force'} 2024-04-24T08:13:17.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:17 smithi194 ceph-mon[32836]: pgmap v91: 97 pgs: 97 active+clean; 7.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 318 MiB/s wr, 120 op/s 2024-04-24T08:13:17.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:17 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:13:17.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:17 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:17.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:17 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:13:17.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:17 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:17.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:17 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:13:17.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:17 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:18.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:17 smithi028 ceph-mon[27786]: pgmap v91: 97 pgs: 97 active+clean; 7.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 318 MiB/s wr, 120 op/s 2024-04-24T08:13:18.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:17 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:13:18.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:17 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:18.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:17 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:13:18.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:17 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:18.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:17 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:13:18.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:17 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:18.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:17 smithi117 ceph-mon[32438]: pgmap v91: 97 pgs: 97 active+clean; 7.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 117 KiB/s rd, 318 MiB/s wr, 120 op/s 2024-04-24T08:13:18.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:17 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:13:18.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:17 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:18.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:17 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:13:18.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:17 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:18.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:17 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:13:18.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:17 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:18.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:18 smithi194 ceph-mon[32836]: scrub queued for path: / 2024-04-24T08:13:18.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:18 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:18.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:18 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:13:18.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:18 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:19.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:18 smithi028 ceph-mon[27786]: scrub queued for path: / 2024-04-24T08:13:19.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:18 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:19.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:18 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:13:19.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:18 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:19.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:18 smithi117 ceph-mon[32438]: scrub queued for path: / 2024-04-24T08:13:19.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:18 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:19.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:18 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:13:19.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:18 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:19.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:19 smithi194 ceph-mon[32836]: pgmap v92: 97 pgs: 97 active+clean; 7.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 184 MiB/s wr, 74 op/s 2024-04-24T08:13:20.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:19 smithi028 ceph-mon[27786]: pgmap v92: 97 pgs: 97 active+clean; 7.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 184 MiB/s wr, 74 op/s 2024-04-24T08:13:20.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:19 smithi117 ceph-mon[32438]: pgmap v92: 97 pgs: 97 active+clean; 7.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 184 MiB/s wr, 74 op/s 2024-04-24T08:13:21.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:21 smithi194 ceph-mon[32836]: pgmap v93: 97 pgs: 97 active+clean; 7.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 184 MiB/s wr, 74 op/s 2024-04-24T08:13:22.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:21 smithi028 ceph-mon[27786]: pgmap v93: 97 pgs: 97 active+clean; 7.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 184 MiB/s wr, 74 op/s 2024-04-24T08:13:22.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:21 smithi117 ceph-mon[32438]: pgmap v93: 97 pgs: 97 active+clean; 7.0 GiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 184 MiB/s wr, 74 op/s 2024-04-24T08:13:23.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:23 smithi194 ceph-mon[32836]: pgmap v94: 97 pgs: 97 active+clean; 8.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 285 MiB/s wr, 113 op/s 2024-04-24T08:13:23.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:23 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:13:24.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:23 smithi028 ceph-mon[27786]: pgmap v94: 97 pgs: 97 active+clean; 8.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 285 MiB/s wr, 113 op/s 2024-04-24T08:13:24.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:23 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:13:24.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:23 smithi117 ceph-mon[32438]: pgmap v94: 97 pgs: 97 active+clean; 8.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 113 KiB/s rd, 285 MiB/s wr, 113 op/s 2024-04-24T08:13:24.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:23 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:13:24.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:24 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:13:25.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:24 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:13:25.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:24 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:13:25.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:25 smithi028 ceph-mon[27786]: pgmap v95: 97 pgs: 97 active+clean; 8.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 211 MiB/s wr, 79 op/s 2024-04-24T08:13:26.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:25 smithi117 ceph-mon[32438]: pgmap v95: 97 pgs: 97 active+clean; 8.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 211 MiB/s wr, 79 op/s 2024-04-24T08:13:26.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:25 smithi194 ceph-mon[32836]: pgmap v95: 97 pgs: 97 active+clean; 8.2 GiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 71 KiB/s rd, 211 MiB/s wr, 79 op/s 2024-04-24T08:13:27.033 INFO:tasks.workunit.client.0.smithi028.stdout:fs setup took 46 secs 2024-04-24T08:13:28.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:27 smithi028 ceph-mon[27786]: pgmap v96: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 318 MiB/s wr, 119 op/s 2024-04-24T08:13:28.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:27 smithi117 ceph-mon[32438]: pgmap v96: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 318 MiB/s wr, 119 op/s 2024-04-24T08:13:28.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:27 smithi194 ceph-mon[32836]: pgmap v96: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 318 MiB/s wr, 119 op/s 2024-04-24T08:13:30.048 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:29 smithi028 ceph-mon[27786]: pgmap v97: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 208 MiB/s wr, 78 op/s 2024-04-24T08:13:30.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:29 smithi117 ceph-mon[32438]: pgmap v97: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 208 MiB/s wr, 78 op/s 2024-04-24T08:13:30.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:29 smithi194 ceph-mon[32836]: pgmap v97: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 208 MiB/s wr, 78 op/s 2024-04-24T08:13:31.991 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:31 smithi194 ceph-mon[32836]: pgmap v98: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 208 MiB/s wr, 78 op/s 2024-04-24T08:13:32.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:31 smithi028 ceph-mon[27786]: pgmap v98: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 208 MiB/s wr, 78 op/s 2024-04-24T08:13:32.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:31 smithi117 ceph-mon[32438]: pgmap v98: 97 pgs: 97 active+clean; 9.5 GiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 208 MiB/s wr, 78 op/s 2024-04-24T08:13:33.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:32 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:13:33.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:32 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:13:33.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:32 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:13:34.046 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:33 smithi028 ceph-mon[27786]: pgmap v99: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 348 MiB/s wr, 118 op/s 2024-04-24T08:13:34.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:33 smithi117 ceph-mon[32438]: pgmap v99: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 348 MiB/s wr, 118 op/s 2024-04-24T08:13:34.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:33 smithi194 ceph-mon[32836]: pgmap v99: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 87 KiB/s rd, 348 MiB/s wr, 118 op/s 2024-04-24T08:13:35.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:34 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:35.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:34 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:13:35.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:34 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:13:35.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:34 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:35.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:34 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:13:35.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:34 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:13:35.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:34 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:13:35.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:34 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:13:35.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:34 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:13:35.454 INFO:tasks.workunit.client.0.smithi028.stdout:Syncing()...8 sec 2024-04-24T08:13:35.456 INFO:tasks.workunit.client.0.smithi028.stdout:Starting Actual Benchmark At: Wed Apr 24 08:13:35 2024 2024-04-24T08:13:35.456 INFO:tasks.workunit.client.0.smithi028.stdout: 2024-04-24T08:13:36.055 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:35 smithi028 ceph-mon[27786]: pgmap v100: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 246 MiB/s wr, 79 op/s 2024-04-24T08:13:36.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:35 smithi117 ceph-mon[32438]: pgmap v100: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 246 MiB/s wr, 79 op/s 2024-04-24T08:13:36.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:35 smithi194 ceph-mon[32836]: pgmap v100: 97 pgs: 97 active+clean; 11 GiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 246 MiB/s wr, 79 op/s 2024-04-24T08:13:37.045 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:36 smithi028 ceph-mon[27786]: pgmap v101: 97 pgs: 97 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 365 MiB/s wr, 123 op/s 2024-04-24T08:13:37.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:36 smithi117 ceph-mon[32438]: pgmap v101: 97 pgs: 97 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 365 MiB/s wr, 123 op/s 2024-04-24T08:13:37.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:36 smithi194 ceph-mon[32836]: pgmap v101: 97 pgs: 97 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 365 MiB/s wr, 123 op/s 2024-04-24T08:13:39.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:39 smithi117 ceph-mon[32438]: pgmap v102: 97 pgs: 97 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 258 MiB/s wr, 83 op/s 2024-04-24T08:13:39.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:39 smithi194 ceph-mon[32836]: pgmap v102: 97 pgs: 97 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 258 MiB/s wr, 83 op/s 2024-04-24T08:13:39.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:39 smithi028 ceph-mon[27786]: pgmap v102: 97 pgs: 97 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 258 MiB/s wr, 83 op/s 2024-04-24T08:13:41.594 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:41 smithi028 ceph-mon[27786]: pgmap v103: 97 pgs: 97 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 258 MiB/s wr, 83 op/s 2024-04-24T08:13:41.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:41 smithi117 ceph-mon[32438]: pgmap v103: 97 pgs: 97 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 258 MiB/s wr, 83 op/s 2024-04-24T08:13:41.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:41 smithi194 ceph-mon[32836]: pgmap v103: 97 pgs: 97 active+clean; 13 GiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 258 MiB/s wr, 83 op/s 2024-04-24T08:13:43.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:43 smithi117 ceph-mon[32438]: pgmap v104: 97 pgs: 97 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 297 MiB/s wr, 6.42k op/s 2024-04-24T08:13:43.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:43 smithi194 ceph-mon[32836]: pgmap v104: 97 pgs: 97 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 297 MiB/s wr, 6.42k op/s 2024-04-24T08:13:43.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:43 smithi028 ceph-mon[27786]: pgmap v104: 97 pgs: 97 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 204 KiB/s rd, 297 MiB/s wr, 6.42k op/s 2024-04-24T08:13:45.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:45 smithi117 ceph-mon[32438]: pgmap v105: 97 pgs: 97 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 157 MiB/s wr, 6.38k op/s 2024-04-24T08:13:45.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:45 smithi194 ceph-mon[32836]: pgmap v105: 97 pgs: 97 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 157 MiB/s wr, 6.38k op/s 2024-04-24T08:13:45.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:45 smithi028 ceph-mon[27786]: pgmap v105: 97 pgs: 97 active+clean; 13 GiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 193 KiB/s rd, 157 MiB/s wr, 6.38k op/s 2024-04-24T08:13:47.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:47 smithi117 ceph-mon[32438]: pgmap v106: 97 pgs: 97 active+clean; 13 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 188 MiB/s wr, 10.57k op/s 2024-04-24T08:13:47.686 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:13:47.686 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:13:47.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:47 smithi194 ceph-mon[32836]: pgmap v106: 97 pgs: 97 active+clean; 13 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 188 MiB/s wr, 10.57k op/s 2024-04-24T08:13:47.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:47 smithi028 ceph-mon[27786]: pgmap v106: 97 pgs: 97 active+clean; 13 GiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 188 MiB/s wr, 10.57k op/s 2024-04-24T08:13:48.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.092+0000 7f7e1e000640 1 -- 172.21.15.28:0/3089191447 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e10035020 con 0x7f7e20109b30 2024-04-24T08:13:48.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.093+0000 7f7e26600640 1 -- 172.21.15.28:0/3089191447 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e20109b30 msgr2=0x7f7e20114c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.093+0000 7f7e26600640 1 --2- 172.21.15.28:0/3089191447 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e20109b30 0x7f7e20114c10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7e10009900 tx=0x7f7e1002a440 comp rx=0 tx=0).stop 2024-04-24T08:13:48.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.093+0000 7f7e26600640 1 -- 172.21.15.28:0/3089191447 shutdown_connections 2024-04-24T08:13:48.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.093+0000 7f7e26600640 1 --2- 172.21.15.28:0/3089191447 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e20115150 0x7f7e20117540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.093+0000 7f7e26600640 1 --2- 172.21.15.28:0/3089191447 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e20109b30 0x7f7e20114c10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.093+0000 7f7e26600640 1 --2- 172.21.15.28:0/3089191447 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e20109210 0x7f7e201095f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.093+0000 7f7e26600640 1 -- 172.21.15.28:0/3089191447 >> 172.21.15.28:0/3089191447 conn(0x7f7e2006d110 msgr2=0x7f7e2006d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:48.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.093+0000 7f7e26600640 1 -- 172.21.15.28:0/3089191447 shutdown_connections 2024-04-24T08:13:48.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.093+0000 7f7e26600640 1 -- 172.21.15.28:0/3089191447 wait complete. 2024-04-24T08:13:48.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.095+0000 7f7e26600640 1 Processor -- start 2024-04-24T08:13:48.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.095+0000 7f7e26600640 1 -- start start 2024-04-24T08:13:48.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.095+0000 7f7e26600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e20109210 0x7f7e201a35c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.095+0000 7f7e26600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e20109b30 0x7f7e201a3b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.095+0000 7f7e26600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e20115150 0x7f7e201a7e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.096+0000 7f7e1f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e20109210 0x7f7e201a35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.096+0000 7f7e1f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e20109210 0x7f7e201a35c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:57046/0 (socket says 172.21.15.28:57046) 2024-04-24T08:13:48.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.096+0000 7f7e1ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e20109b30 0x7f7e201a3b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.096+0000 7f7e1fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e20115150 0x7f7e201a7e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.096+0000 7f7e1fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e20115150 0x7f7e201a7e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:38568/0 (socket says 172.21.15.28:38568) 2024-04-24T08:13:48.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.096+0000 7f7e26600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2011ad30 con 0x7f7e20109b30 2024-04-24T08:13:48.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.096+0000 7f7e1f400640 1 -- 172.21.15.28:0/2408113580 learned_addr learned my addr 172.21.15.28:0/2408113580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:48.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.096+0000 7f7e26600640 1 -- 172.21.15.28:0/2408113580 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2011abb0 con 0x7f7e20109210 2024-04-24T08:13:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.096+0000 7f7e26600640 1 -- 172.21.15.28:0/2408113580 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7e2011aeb0 con 0x7f7e20115150 2024-04-24T08:13:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.097+0000 7f7e1ea00640 1 -- 172.21.15.28:0/2408113580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e20115150 msgr2=0x7f7e201a7e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.097+0000 7f7e1ea00640 1 --2- 172.21.15.28:0/2408113580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e20115150 0x7f7e201a7e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.097+0000 7f7e1ea00640 1 -- 172.21.15.28:0/2408113580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e20109210 msgr2=0x7f7e201a35c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.097+0000 7f7e1ea00640 1 --2- 172.21.15.28:0/2408113580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e20109210 0x7f7e201a35c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.097+0000 7f7e1ea00640 1 -- 172.21.15.28:0/2408113580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e201a85b0 con 0x7f7e20109b30 2024-04-24T08:13:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.097+0000 7f7e1fe00640 1 --2- 172.21.15.28:0/2408113580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e20115150 0x7f7e201a7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:13:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.097+0000 7f7e1f400640 1 --2- 172.21.15.28:0/2408113580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e20109210 0x7f7e201a35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:13:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.098+0000 7f7e1ea00640 1 --2- 172.21.15.28:0/2408113580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e20109b30 0x7f7e201a3b00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7e10006010 tx=0x7f7e10032b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.098+0000 7f7e17e00640 1 -- 172.21.15.28:0/2408113580 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1003f030 con 0x7f7e20109b30 2024-04-24T08:13:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.098+0000 7f7e26600640 1 -- 172.21.15.28:0/2408113580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e201a8840 con 0x7f7e20109b30 2024-04-24T08:13:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.098+0000 7f7e26600640 1 -- 172.21.15.28:0/2408113580 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e201b0130 con 0x7f7e20109b30 2024-04-24T08:13:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.098+0000 7f7e17e00640 1 -- 172.21.15.28:0/2408113580 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e10032e20 con 0x7f7e20109b30 2024-04-24T08:13:48.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.098+0000 7f7e17e00640 1 -- 172.21.15.28:0/2408113580 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e10049060 con 0x7f7e20109b30 2024-04-24T08:13:48.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.104+0000 7f7e295b2740 1 Processor -- start 2024-04-24T08:13:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.104+0000 7f7e295b2740 1 -- start start 2024-04-24T08:13:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.105+0000 7f7e17e00640 1 -- 172.21.15.28:0/2408113580 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7e10042030 con 0x7f7e20109b30 2024-04-24T08:13:48.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.106+0000 7f7e295b2740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf0d860 0x55d44bf0dc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.106+0000 7f7e295b2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0e180 0x55d44bf18310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.106+0000 7f7e295b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf18850 0x55d44bf1ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.106+0000 7f7e1fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf0d860 0x55d44bf0dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.106+0000 7f7e1fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf0d860 0x55d44bf0dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:38584/0 (socket says 172.21.15.28:38584) 2024-04-24T08:13:48.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.106+0000 7f7e1f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0e180 0x55d44bf18310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.106+0000 7f7e1f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0e180 0x55d44bf18310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:57050/0 (socket says 172.21.15.28:57050) 2024-04-24T08:13:48.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.106+0000 7f7e295b2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d44be731f0 con 0x55d44bf18850 2024-04-24T08:13:48.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.106+0000 7f7e295b2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d44be19350 con 0x55d44bf0e180 2024-04-24T08:13:48.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.106+0000 7f7e295b2740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d44be6f1b0 con 0x55d44bf0d860 2024-04-24T08:13:48.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.106+0000 7f7e1fe00640 1 -- 172.21.15.28:0/4261025400 learned_addr learned my addr 172.21.15.28:0/4261025400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.107+0000 7f7e1ea00640 1 --2- 172.21.15.28:0/4261025400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf18850 0x55d44bf1ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.107+0000 7f7e1fe00640 1 -- 172.21.15.28:0/4261025400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0e180 msgr2=0x55d44bf18310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.107+0000 7f7e1fe00640 1 --2- 172.21.15.28:0/4261025400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0e180 0x55d44bf18310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.107+0000 7f7e1fe00640 1 -- 172.21.15.28:0/4261025400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf18850 msgr2=0x55d44bf1ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.107+0000 7f7e1fe00640 1 --2- 172.21.15.28:0/4261025400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf18850 0x55d44bf1ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.107+0000 7f7e1fe00640 1 -- 172.21.15.28:0/4261025400 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d44be6df50 con 0x55d44bf0d860 2024-04-24T08:13:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.108+0000 7f7e1fe00640 1 --2- 172.21.15.28:0/4261025400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf0d860 0x55d44bf0dc40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7e1800c450 tx=0x7f7e1800c910 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebdcda1b3176a0e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.108+0000 7f7e14c00640 1 -- 172.21.15.28:0/4261025400 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18007430 con 0x55d44bf0d860 2024-04-24T08:13:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.108+0000 7f7e14c00640 1 -- 172.21.15.28:0/4261025400 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e18007ab0 con 0x55d44bf0d860 2024-04-24T08:13:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.108+0000 7f7e14c00640 1 -- 172.21.15.28:0/4261025400 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1800ac10 con 0x55d44bf0d860 2024-04-24T08:13:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.109+0000 7f7e295b2740 1 -- 172.21.15.28:0/4261025400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf0d860 msgr2=0x55d44bf0dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.109+0000 7f7e295b2740 1 --2- 172.21.15.28:0/4261025400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf0d860 0x55d44bf0dc40 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7e1800c450 tx=0x7f7e1800c910 comp rx=0 tx=0).stop 2024-04-24T08:13:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.109+0000 7f7e295b2740 1 -- 172.21.15.28:0/4261025400 shutdown_connections 2024-04-24T08:13:48.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.109+0000 7f7e295b2740 1 --2- 172.21.15.28:0/4261025400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf18850 0x55d44bf1ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.109+0000 7f7e295b2740 1 --2- 172.21.15.28:0/4261025400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0e180 0x55d44bf18310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.109+0000 7f7e295b2740 1 --2- 172.21.15.28:0/4261025400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf0d860 0x55d44bf0dc40 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.109+0000 7f7e295b2740 1 -- 172.21.15.28:0/4261025400 >> 172.21.15.28:0/4261025400 conn(0x55d44bf09b80 msgr2=0x55d44bf09f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:48.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.110+0000 7f7e295b2740 1 -- 172.21.15.28:0/4261025400 shutdown_connections 2024-04-24T08:13:48.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.110+0000 7f7e295b2740 1 -- 172.21.15.28:0/4261025400 wait complete. 2024-04-24T08:13:48.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.110+0000 7f7e295b2740 1 -- start start 2024-04-24T08:13:48.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.111+0000 7f7e295b2740 1 Processor -- start 2024-04-24T08:13:48.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.111+0000 7f7e17e00640 1 --2- 172.21.15.28:0/2408113580 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7e040a3c30 0x7f7e040a60f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.112+0000 7f7e295b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf0d860 0x55d44bf157d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.112+0000 7f7e17e00640 1 -- 172.21.15.28:0/2408113580 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f7e100f0b30 con 0x7f7e20109b30 2024-04-24T08:13:48.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.112+0000 7f7e295b2740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf0e180 0x55d44bf15d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.112+0000 7f7e295b2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf18850 0x55d44bf16250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.112+0000 7f7e295b2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d44bf1b690 con 0x55d44bf0d860 2024-04-24T08:13:48.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.112+0000 7f7e295b2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d44be2c690 con 0x55d44bf18850 2024-04-24T08:13:48.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.112+0000 7f7e295b2740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d44bf1b810 con 0x55d44bf0e180 2024-04-24T08:13:48.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.112+0000 7f7e295b2740 10 client.0 _ll_register_callbacks cb 0x55d44be68650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:48.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.112+0000 7f7e1ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf0e180 0x55d44bf15d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.112+0000 7f7e1ea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf0e180 0x55d44bf15d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:38598/0 (socket says 172.21.15.28:38598) 2024-04-24T08:13:48.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.113+0000 7f7e1ea00640 1 -- 172.21.15.28:0/2011251136 learned_addr learned my addr 172.21.15.28:0/2011251136 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:48.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.113+0000 7f7e1ea00640 1 -- 172.21.15.28:0/2011251136 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf18850 msgr2=0x55d44bf16250 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:13:48.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.113+0000 7f7e1ea00640 1 --2- 172.21.15.28:0/2011251136 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf18850 0x55d44bf16250 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.113+0000 7f7e1ea00640 1 -- 172.21.15.28:0/2011251136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf0d860 msgr2=0x55d44bf157d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:48.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.113+0000 7f7e1ea00640 1 --2- 172.21.15.28:0/2011251136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf0d860 0x55d44bf157d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.113+0000 7f7e1ea00640 1 -- 172.21.15.28:0/2011251136 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d44be2b920 con 0x55d44bf0e180 2024-04-24T08:13:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.114+0000 7f7e0d600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:13:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.114+0000 7f7e1ea00640 1 --2- 172.21.15.28:0/2011251136 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf0e180 0x55d44bf15d10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7e10031440 tx=0x7f7e100bc940 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cdd194a7bdace4b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.114+0000 7f7e0d600640 1 -- 172.21.15.28:0/2011251136 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e10035040 con 0x55d44bf0e180 2024-04-24T08:13:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.114+0000 7f7e295b2740 1 -- 172.21.15.28:0/2011251136 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e18007ab0 con 0x55d44bf0e180 2024-04-24T08:13:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.114+0000 7f7e0d600640 1 -- 172.21.15.28:0/2011251136 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f7e1003b460 con 0x55d44bf0e180 2024-04-24T08:13:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.115+0000 7f7e0d600640 1 -- 172.21.15.28:0/2011251136 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7e10035040 con 0x55d44bf0e180 2024-04-24T08:13:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.115+0000 7f7e295b2740 10 client.26035 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.115+0000 7f7e295b2740 1 -- 172.21.15.28:0/2011251136 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55d44bc4f7d0 con 0x55d44bf0e180 2024-04-24T08:13:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.115+0000 7f7e0d600640 1 -- 172.21.15.28:0/2011251136 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e18007ab0 con 0x55d44bf0e180 2024-04-24T08:13:48.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.115+0000 7f7e0d600640 1 -- 172.21.15.28:0/2011251136 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f7e1004c060 con 0x55d44bf0e180 2024-04-24T08:13:48.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.115+0000 7f7e295b2740 10 client.26035 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:48.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.115+0000 7f7e295b2740 10 client.26035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:13:48.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.115+0000 7f7e295b2740 20 client.26035 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:48.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.115+0000 7f7e295b2740 1 --2- 172.21.15.28:0/2011251136 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55d44bf31390 0x55d44bf33850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.115+0000 7f7e295b2740 4 client.26035 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:13:48.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.115+0000 7f7e295b2740 1 -- 172.21.15.28:0/2011251136 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d44be80c10 con 0x55d44bf31390 2024-04-24T08:13:48.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.116+0000 7f7e1f400640 1 --2- 172.21.15.28:0/2011251136 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55d44bf31390 0x55d44bf33850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.116+0000 7f7e1fe00640 1 --2- 172.21.15.28:0/2408113580 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7e040a3c30 0x7f7e040a60f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.117+0000 7f7e0d600640 10 client.26035 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:48.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.117+0000 7f7e1f400640 1 --2- 172.21.15.28:0/2011251136 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55d44bf31390 0x55d44bf33850 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe696cedfd3b1de5 server_cookie=45cc537a3bc8dd75 in_seq=0 out_seq=0 2024-04-24T08:13:48.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.118+0000 7f7e1fe00640 1 --2- 172.21.15.28:0/2408113580 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7e040a3c30 0x7f7e040a60f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7e1800c450 tx=0x7f7e180094f0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:48.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.121+0000 7f7e0d600640 1 -- 172.21.15.28:0/2011251136 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d44be80c10 con 0x55d44bf31390 2024-04-24T08:13:48.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.121+0000 7f7e0d600640 10 client.26035 handle_command_reply: tid=0 2024-04-24T08:13:48.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.122+0000 7f7e295b2740 1 client.26035 shutdown 2024-04-24T08:13:48.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.124+0000 7f7e295b2740 1 -- 172.21.15.28:0/2011251136 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf0e180 msgr2=0x55d44bf15d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.124+0000 7f7e295b2740 1 --2- 172.21.15.28:0/2011251136 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf0e180 0x55d44bf15d10 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7e10031440 tx=0x7f7e100bc940 comp rx=0 tx=0).stop 2024-04-24T08:13:48.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.126+0000 7f7e295b2740 1 -- 172.21.15.28:0/2011251136 shutdown_connections 2024-04-24T08:13:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.126+0000 7f7e295b2740 1 --2- 172.21.15.28:0/2011251136 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55d44bf31390 0x55d44bf33850 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.126+0000 7f7e295b2740 1 --2- 172.21.15.28:0/2011251136 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf18850 0x55d44bf16250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.126+0000 7f7e295b2740 1 --2- 172.21.15.28:0/2011251136 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf0e180 0x55d44bf15d10 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.126+0000 7f7e295b2740 1 --2- 172.21.15.28:0/2011251136 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf0d860 0x55d44bf157d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.126+0000 7f7e295b2740 1 -- 172.21.15.28:0/2011251136 >> 172.21.15.28:0/2011251136 conn(0x55d44bf09b80 msgr2=0x55d44bf0f980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.129+0000 7f7e0d600640 0 client.26035 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.129+0000 7f7e295b2740 1 -- 172.21.15.28:0/2011251136 shutdown_connections 2024-04-24T08:13:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.129+0000 7f7e295b2740 1 -- 172.21.15.28:0/2011251136 wait complete. 2024-04-24T08:13:48.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.131+0000 7f7e295b2740 20 client.26035 trim_cache size 0 max 16384 2024-04-24T08:13:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.171+0000 7f7e295b2740 1 Processor -- start 2024-04-24T08:13:48.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.171+0000 7f7e295b2740 1 -- start start 2024-04-24T08:13:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.171+0000 7f7e295b2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0d860 0x55d44bf10000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.172+0000 7f7e295b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf0e180 0x55d44bf08aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.172+0000 7f7e295b2740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf18850 0x55d44bf39160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.172+0000 7f7e295b2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e201b6d60 con 0x55d44bf0e180 2024-04-24T08:13:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.172+0000 7f7e295b2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7e201b4630 con 0x55d44bf0d860 2024-04-24T08:13:48.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.172+0000 7f7e295b2740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7e201b2420 con 0x55d44bf18850 2024-04-24T08:13:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.172+0000 7f7e1f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0d860 0x55d44bf10000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.172+0000 7f7e1ea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf0e180 0x55d44bf08aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.172+0000 7f7e1f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0d860 0x55d44bf10000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:57054/0 (socket says 172.21.15.28:57054) 2024-04-24T08:13:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.172+0000 7f7e1f400640 1 -- 172.21.15.28:0/2359600625 learned_addr learned my addr 172.21.15.28:0/2359600625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.173+0000 7f7e1fe00640 1 --2- 172.21.15.28:0/2359600625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf18850 0x55d44bf39160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.173+0000 7f7e1f400640 1 -- 172.21.15.28:0/2359600625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf18850 msgr2=0x55d44bf39160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.173+0000 7f7e1f400640 1 --2- 172.21.15.28:0/2359600625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf18850 0x55d44bf39160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.173+0000 7f7e1f400640 1 -- 172.21.15.28:0/2359600625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf0e180 msgr2=0x55d44bf08aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.173+0000 7f7e1f400640 1 --2- 172.21.15.28:0/2359600625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf0e180 0x55d44bf08aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.173+0000 7f7e1f400640 1 -- 172.21.15.28:0/2359600625 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d44bb853f0 con 0x55d44bf0d860 2024-04-24T08:13:48.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.174+0000 7f7e1f400640 1 --2- 172.21.15.28:0/2359600625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0d860 0x55d44bf10000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7e08006fd0 tx=0x7f7e080079e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b08e28d6480053cc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:48.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.175+0000 7f7e0fe00640 1 -- 172.21.15.28:0/2359600625 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e08015620 con 0x55d44bf0d860 2024-04-24T08:13:48.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.176+0000 7f7e0fe00640 1 -- 172.21.15.28:0/2359600625 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e08015ca0 con 0x55d44bf0d860 2024-04-24T08:13:48.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.176+0000 7f7e0fe00640 1 -- 172.21.15.28:0/2359600625 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e08023e40 con 0x55d44bf0d860 2024-04-24T08:13:48.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.177+0000 7f7e295b2740 1 -- 172.21.15.28:0/2359600625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0d860 msgr2=0x55d44bf10000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.177+0000 7f7e295b2740 1 --2- 172.21.15.28:0/2359600625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0d860 0x55d44bf10000 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7e08006fd0 tx=0x7f7e080079e0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.178+0000 7f7e295b2740 1 -- 172.21.15.28:0/2359600625 shutdown_connections 2024-04-24T08:13:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.178+0000 7f7e295b2740 1 --2- 172.21.15.28:0/2359600625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf0e180 0x55d44bf08aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.178+0000 7f7e295b2740 1 --2- 172.21.15.28:0/2359600625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf18850 0x55d44bf39160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.178+0000 7f7e295b2740 1 --2- 172.21.15.28:0/2359600625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0d860 0x55d44bf10000 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.178+0000 7f7e295b2740 1 -- 172.21.15.28:0/2359600625 >> 172.21.15.28:0/2359600625 conn(0x55d44bf09b80 msgr2=0x55d44bf08390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:48.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.178+0000 7f7e295b2740 1 -- 172.21.15.28:0/2359600625 shutdown_connections 2024-04-24T08:13:48.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.179+0000 7f7e295b2740 1 -- 172.21.15.28:0/2359600625 wait complete. 2024-04-24T08:13:48.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.179+0000 7f7e295b2740 1 -- start start 2024-04-24T08:13:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.179+0000 7f7e295b2740 1 Processor -- start 2024-04-24T08:13:48.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.180+0000 7f7e295b2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf0d860 0x55d44bf29580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.181+0000 7f7e295b2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0e180 0x55d44bf120a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.181+0000 7f7e295b2740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf18850 0x55d44bf125e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.181+0000 7f7e1fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf18850 0x55d44bf125e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.181+0000 7f7e1fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf18850 0x55d44bf125e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:38624/0 (socket says 172.21.15.28:38624) 2024-04-24T08:13:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.181+0000 7f7e1fe00640 1 -- 172.21.15.28:0/1989111605 learned_addr learned my addr 172.21.15.28:0/1989111605 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.181+0000 7f7e295b2740 1 -- 172.21.15.28:0/1989111605 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e201b2af0 con 0x55d44bf0d860 2024-04-24T08:13:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.181+0000 7f7e1ea00640 1 --2- 172.21.15.28:0/1989111605 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0e180 0x55d44bf120a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.181+0000 7f7e295b2740 1 -- 172.21.15.28:0/1989111605 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7e201b3f60 con 0x55d44bf0e180 2024-04-24T08:13:48.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.181+0000 7f7e1f400640 1 --2- 172.21.15.28:0/1989111605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf0d860 0x55d44bf29580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.181+0000 7f7e295b2740 1 -- 172.21.15.28:0/1989111605 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7e201b31c0 con 0x55d44bf18850 2024-04-24T08:13:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.182+0000 7f7e295b2740 10 client.0 _ll_register_callbacks cb 0x7f7e10003f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.182+0000 7f7e1fe00640 1 -- 172.21.15.28:0/1989111605 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0e180 msgr2=0x55d44bf120a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.182+0000 7f7e1fe00640 1 --2- 172.21.15.28:0/1989111605 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0e180 0x55d44bf120a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.182+0000 7f7e1fe00640 1 -- 172.21.15.28:0/1989111605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf0d860 msgr2=0x55d44bf29580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.182+0000 7f7e1fe00640 1 --2- 172.21.15.28:0/1989111605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf0d860 0x55d44bf29580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.182+0000 7f7e1fe00640 1 -- 172.21.15.28:0/1989111605 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d44bb450f0 con 0x55d44bf18850 2024-04-24T08:13:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.182+0000 7f7e295b2740 1 -- 172.21.15.28:0/1989111605 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e08015ca0 con 0x55d44bf18850 2024-04-24T08:13:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.182+0000 7f7e1ea00640 1 --2- 172.21.15.28:0/1989111605 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0e180 0x55d44bf120a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:13:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.183+0000 7f7e1fe00640 1 --2- 172.21.15.28:0/1989111605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf18850 0x55d44bf125e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7e1800afa0 tx=0x7f7e18004b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=34b20798d55a3694 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.184+0000 7f7e0d600640 10 client.26041 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:13:48.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.184+0000 7f7e0d600640 1 -- 172.21.15.28:0/1989111605 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18031060 con 0x55d44bf18850 2024-04-24T08:13:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.184+0000 7f7e0d600640 1 -- 172.21.15.28:0/1989111605 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f7e1801b070 con 0x55d44bf18850 2024-04-24T08:13:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.184+0000 7f7e0d600640 1 -- 172.21.15.28:0/1989111605 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7e18009c40 con 0x55d44bf18850 2024-04-24T08:13:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.184+0000 7f7e295b2740 10 client.26041 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:48.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.184+0000 7f7e295b2740 1 -- 172.21.15.28:0/1989111605 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({fsmap=25}) -- 0x55d44be38810 con 0x55d44bf18850 2024-04-24T08:13:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.185+0000 7f7e0d600640 1 -- 172.21.15.28:0/1989111605 <== mon.1 v2:172.21.15.194:3300/0 4 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f7e18020430 con 0x55d44bf18850 2024-04-24T08:13:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.185+0000 7f7e295b2740 10 client.26041 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.185+0000 7f7e295b2740 10 client.26041 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:13:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.185+0000 7f7e295b2740 20 client.26041 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.185+0000 7f7e295b2740 1 --2- 172.21.15.28:0/1989111605 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55d44bf97730 0x55d44bf99b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.185+0000 7f7e295b2740 4 client.26041 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:13:48.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.185+0000 7f7e295b2740 1 -- 172.21.15.28:0/1989111605 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d44be7aa80 con 0x55d44bf97730 2024-04-24T08:13:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.186+0000 7f7e1f400640 1 --2- 172.21.15.28:0/1989111605 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55d44bf97730 0x55d44bf99b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.187+0000 7f7e1f400640 1 --2- 172.21.15.28:0/1989111605 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55d44bf97730 0x55d44bf99b50 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fa189120dd6db6b server_cookie=54ab7b16b98fb719 in_seq=0 out_seq=0 2024-04-24T08:13:48.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.187+0000 7f7e0d600640 10 client.26041 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.187+0000 7f7e0d600640 1 -- 172.21.15.28:0/1989111605 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55d44be7aa80 con 0x55d44bf97730 2024-04-24T08:13:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.187+0000 7f7e0d600640 10 client.26041 handle_command_reply: tid=0 2024-04-24T08:13:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.187+0000 7f7e295b2740 1 client.26041 shutdown 2024-04-24T08:13:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.187+0000 7f7e295b2740 1 -- 172.21.15.28:0/1989111605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf18850 msgr2=0x55d44bf125e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.187+0000 7f7e295b2740 1 --2- 172.21.15.28:0/1989111605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf18850 0x55d44bf125e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f7e1800afa0 tx=0x7f7e18004b50 comp rx=0 tx=0).stop 2024-04-24T08:13:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.187+0000 7f7e295b2740 1 -- 172.21.15.28:0/1989111605 shutdown_connections 2024-04-24T08:13:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.187+0000 7f7e295b2740 1 --2- 172.21.15.28:0/1989111605 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55d44bf97730 0x55d44bf99b50 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.187+0000 7f7e295b2740 1 --2- 172.21.15.28:0/1989111605 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d44bf18850 0x55d44bf125e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.187+0000 7f7e295b2740 1 --2- 172.21.15.28:0/1989111605 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d44bf0e180 0x55d44bf120a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.187+0000 7f7e295b2740 1 --2- 172.21.15.28:0/1989111605 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d44bf0d860 0x55d44bf29580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.187+0000 7f7e0d600640 0 client.26041 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:48.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.187+0000 7f7e295b2740 1 -- 172.21.15.28:0/1989111605 >> 172.21.15.28:0/1989111605 conn(0x55d44bf09b80 msgr2=0x55d44bf16020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.188+0000 7f7e295b2740 1 -- 172.21.15.28:0/1989111605 shutdown_connections 2024-04-24T08:13:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.188+0000 7f7e295b2740 1 -- 172.21.15.28:0/1989111605 wait complete. 2024-04-24T08:13:48.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.188+0000 7f7e295b2740 20 client.26041 trim_cache size 0 max 16384 2024-04-24T08:13:48.189 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:13:48.190 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-24T08:13:48.190 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-24T08:13:48.190 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:13:48.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.189+0000 7f7e26600640 1 -- 172.21.15.28:0/2408113580 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7e040a3c30 msgr2=0x7f7e040a60f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.189+0000 7f7e26600640 1 --2- 172.21.15.28:0/2408113580 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7e040a3c30 0x7f7e040a60f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f7e1800c450 tx=0x7f7e180094f0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.189+0000 7f7e26600640 1 -- 172.21.15.28:0/2408113580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e20109b30 msgr2=0x7f7e201a3b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.189+0000 7f7e26600640 1 --2- 172.21.15.28:0/2408113580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e20109b30 0x7f7e201a3b00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f7e10006010 tx=0x7f7e10032b70 comp rx=0 tx=0).stop 2024-04-24T08:13:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.190+0000 7f7e26600640 1 -- 172.21.15.28:0/2408113580 shutdown_connections 2024-04-24T08:13:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.190+0000 7f7e26600640 1 --2- 172.21.15.28:0/2408113580 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7e040a3c30 0x7f7e040a60f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.190+0000 7f7e26600640 1 --2- 172.21.15.28:0/2408113580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e20109b30 0x7f7e201a3b00 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.190+0000 7f7e26600640 1 --2- 172.21.15.28:0/2408113580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e20115150 0x7f7e201a7e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.190+0000 7f7e26600640 1 --2- 172.21.15.28:0/2408113580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e20109210 0x7f7e201a35c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.190+0000 7f7e26600640 1 -- 172.21.15.28:0/2408113580 >> 172.21.15.28:0/2408113580 conn(0x7f7e2006d110 msgr2=0x7f7e2010ab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.190+0000 7f7e26600640 1 -- 172.21.15.28:0/2408113580 shutdown_connections 2024-04-24T08:13:48.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.190+0000 7f7e26600640 1 -- 172.21.15.28:0/2408113580 wait complete. 2024-04-24T08:13:48.224 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:13:48.225 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:13:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.630+0000 7f7173200640 1 -- 172.21.15.28:0/3651868313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7174076530 msgr2=0x7f7174076910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.630+0000 7f7173200640 1 --2- 172.21.15.28:0/3651868313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7174076530 0x7f7174076910 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7164009930 tx=0x7f716402a440 comp rx=0 tx=0).stop 2024-04-24T08:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.630+0000 7f7173200640 1 -- 172.21.15.28:0/3651868313 shutdown_connections 2024-04-24T08:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.630+0000 7f7173200640 1 --2- 172.21.15.28:0/3651868313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7174076e50 0x7f7174112780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.630+0000 7f7173200640 1 --2- 172.21.15.28:0/3651868313 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7174112cc0 0x7f71741150b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.630+0000 7f7173200640 1 --2- 172.21.15.28:0/3651868313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7174076530 0x7f7174076910 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.630+0000 7f7173200640 1 -- 172.21.15.28:0/3651868313 >> 172.21.15.28:0/3651868313 conn(0x7f71740ffad0 msgr2=0x7f7174101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.630+0000 7f7173200640 1 -- 172.21.15.28:0/3651868313 shutdown_connections 2024-04-24T08:13:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.630+0000 7f7173200640 1 -- 172.21.15.28:0/3651868313 wait complete. 2024-04-24T08:13:48.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.635+0000 7f7173200640 1 Processor -- start 2024-04-24T08:13:48.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.635+0000 7f7173200640 1 -- start start 2024-04-24T08:13:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.636+0000 7f7173200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7174076530 0x7f717419edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.636+0000 7f7173200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7174076e50 0x7f717419f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.636+0000 7f7173200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7174112cc0 0x7f71741a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.636+0000 7f7173200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7174118860 con 0x7f7174112cc0 2024-04-24T08:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.636+0000 7f7173200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f71741186e0 con 0x7f7174076530 2024-04-24T08:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.636+0000 7f7173200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f71741189e0 con 0x7f7174076e50 2024-04-24T08:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.636+0000 7f7171e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7174076530 0x7f717419edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.636+0000 7f7171e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7174076530 0x7f717419edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:57070/0 (socket says 172.21.15.28:57070) 2024-04-24T08:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.636+0000 7f7171e00640 1 -- 172.21.15.28:0/2257665825 learned_addr learned my addr 172.21.15.28:0/2257665825 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.637+0000 7f7171e00640 1 -- 172.21.15.28:0/2257665825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7174076e50 msgr2=0x7f717419f300 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-04-24T08:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.637+0000 7f7171e00640 1 --2- 172.21.15.28:0/2257665825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7174076e50 0x7f717419f300 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.637+0000 7f7171e00640 1 -- 172.21.15.28:0/2257665825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7174112cc0 msgr2=0x7f71741a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.637+0000 7f7171e00640 1 --2- 172.21.15.28:0/2257665825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7174112cc0 0x7f71741a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.637+0000 7f7171e00640 1 -- 172.21.15.28:0/2257665825 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f71741a3db0 con 0x7f7174076530 2024-04-24T08:13:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.637+0000 7f7171e00640 1 --2- 172.21.15.28:0/2257665825 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7174076530 0x7f717419edc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f716402a9b0 tx=0x7f7164003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.638+0000 7f716aa00640 1 -- 172.21.15.28:0/2257665825 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f716403f070 con 0x7f7174076530 2024-04-24T08:13:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.638+0000 7f716aa00640 1 -- 172.21.15.28:0/2257665825 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7164031c90 con 0x7f7174076530 2024-04-24T08:13:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.638+0000 7f7173200640 1 -- 172.21.15.28:0/2257665825 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f71741a4040 con 0x7f7174076530 2024-04-24T08:13:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.639+0000 7f716aa00640 1 -- 172.21.15.28:0/2257665825 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7164035030 con 0x7f7174076530 2024-04-24T08:13:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.639+0000 7f7173200640 1 -- 172.21.15.28:0/2257665825 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f717406da30 con 0x7f7174076530 2024-04-24T08:13:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.642+0000 7f7173200640 1 -- 172.21.15.28:0/2257665825 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7174068d40 con 0x7f7174076530 2024-04-24T08:13:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.648+0000 7f716aa00640 1 -- 172.21.15.28:0/2257665825 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f716403bd10 con 0x7f7174076530 2024-04-24T08:13:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.652+0000 7f716aa00640 1 --2- 172.21.15.28:0/2257665825 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f71580a3de0 0x7f71580a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.652+0000 7f716aa00640 1 -- 172.21.15.28:0/2257665825 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f71640f13a0 con 0x7f7174076530 2024-04-24T08:13:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.653+0000 7f7171400640 1 --2- 172.21.15.28:0/2257665825 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f71580a3de0 0x7f71580a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:48.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.653+0000 7f716aa00640 1 -- 172.21.15.28:0/2257665825 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f716406f020 con 0x7f7174076530 2024-04-24T08:13:48.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:48.654+0000 7f7171400640 1 --2- 172.21.15.28:0/2257665825 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f71580a3de0 0x7f71580a62a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f71741035e0 tx=0x7f7160005b80 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:49.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.183+0000 7f7173200640 1 -- 172.21.15.28:0/2257665825 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f71741a0110 con 0x7f7174076530 2024-04-24T08:13:49.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.185+0000 7f716aa00640 1 -- 172.21.15.28:0/2257665825 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11872 (secure 0 0 0) 0x7f71640bda20 con 0x7f7174076530 2024-04-24T08:13:49.185 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:13:49.186 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":25,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:12:18.621376+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24572,24574,24556]},"id":1}]} 2024-04-24T08:13:49.186 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 25 2024-04-24T08:13:49.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.190+0000 7f7173200640 1 -- 172.21.15.28:0/2257665825 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f71580a3de0 msgr2=0x7f71580a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.190+0000 7f7173200640 1 --2- 172.21.15.28:0/2257665825 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f71580a3de0 0x7f71580a62a0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f71741035e0 tx=0x7f7160005b80 comp rx=0 tx=0).stop 2024-04-24T08:13:49.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.190+0000 7f7173200640 1 -- 172.21.15.28:0/2257665825 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7174076530 msgr2=0x7f717419edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.190+0000 7f7173200640 1 --2- 172.21.15.28:0/2257665825 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7174076530 0x7f717419edc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f716402a9b0 tx=0x7f7164003b90 comp rx=0 tx=0).stop 2024-04-24T08:13:49.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.191+0000 7f7173200640 1 -- 172.21.15.28:0/2257665825 shutdown_connections 2024-04-24T08:13:49.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.191+0000 7f7173200640 1 --2- 172.21.15.28:0/2257665825 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f71580a3de0 0x7f71580a62a0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.191+0000 7f7173200640 1 --2- 172.21.15.28:0/2257665825 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7174112cc0 0x7f71741a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.191+0000 7f7173200640 1 --2- 172.21.15.28:0/2257665825 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7174076e50 0x7f717419f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.191+0000 7f7173200640 1 --2- 172.21.15.28:0/2257665825 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7174076530 0x7f717419edc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.191+0000 7f7173200640 1 -- 172.21.15.28:0/2257665825 >> 172.21.15.28:0/2257665825 conn(0x7f71740ffad0 msgr2=0x7f7174101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.192+0000 7f7173200640 1 -- 172.21.15.28:0/2257665825 shutdown_connections 2024-04-24T08:13:49.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.193+0000 7f7173200640 1 -- 172.21.15.28:0/2257665825 wait complete. 2024-04-24T08:13:49.230 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:13:49.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:49 smithi028 ceph-mon[27786]: pgmap v107: 97 pgs: 97 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 101 MiB/s wr, 14.94k op/s 2024-04-24T08:13:49.540 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:49 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2257665825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:13:49.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.567+0000 7f4494800640 1 -- 172.21.15.28:0/824688339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f448c11b7e0 msgr2=0x7f448c11dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.568+0000 7f4494800640 1 --2- 172.21.15.28:0/824688339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f448c11b7e0 0x7f448c11dbd0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f447c01a910 tx=0x7f447c03b450 comp rx=0 tx=0).stop 2024-04-24T08:13:49.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.569+0000 7f4494800640 1 -- 172.21.15.28:0/824688339 shutdown_connections 2024-04-24T08:13:49.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.569+0000 7f4494800640 1 --2- 172.21.15.28:0/824688339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f448c11b7e0 0x7f448c11dbd0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.569+0000 7f4494800640 1 --2- 172.21.15.28:0/824688339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f448c10e880 0x7f448c10ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.569+0000 7f4494800640 1 --2- 172.21.15.28:0/824688339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f448c110850 0x7f448c10e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.569+0000 7f4494800640 1 -- 172.21.15.28:0/824688339 >> 172.21.15.28:0/824688339 conn(0x7f448c06d110 msgr2=0x7f448c06d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:49.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.570+0000 7f4494800640 1 -- 172.21.15.28:0/824688339 shutdown_connections 2024-04-24T08:13:49.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.571+0000 7f4494800640 1 -- 172.21.15.28:0/824688339 wait complete. 2024-04-24T08:13:49.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.572+0000 7f4494800640 1 Processor -- start 2024-04-24T08:13:49.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.572+0000 7f4494800640 1 -- start start 2024-04-24T08:13:49.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.573+0000 7f4494800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f448c10e880 0x7f448c11ae50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.573+0000 7f4494800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f448c110850 0x7f448c116f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.573+0000 7f4491e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f448c10e880 0x7f448c11ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.574+0000 7f4491400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f448c110850 0x7f448c116f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.573+0000 7f4494800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f448c11b7e0 0x7f448c1174c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.574+0000 7f4494800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f448c113970 con 0x7f448c10e880 2024-04-24T08:13:49.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.574+0000 7f4494800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f448c1137f0 con 0x7f448c110850 2024-04-24T08:13:49.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.574+0000 7f4494800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f448c113af0 con 0x7f448c11b7e0 2024-04-24T08:13:49.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.574+0000 7f4491400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f448c110850 0x7f448c116f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:57082/0 (socket says 172.21.15.28:57082) 2024-04-24T08:13:49.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.574+0000 7f4491400640 1 -- 172.21.15.28:0/1169366193 learned_addr learned my addr 172.21.15.28:0/1169366193 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:49.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.574+0000 7f4492800640 1 --2- 172.21.15.28:0/1169366193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f448c11b7e0 0x7f448c1174c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.575+0000 7f4491400640 1 -- 172.21.15.28:0/1169366193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f448c11b7e0 msgr2=0x7f448c1174c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.575+0000 7f4491400640 1 --2- 172.21.15.28:0/1169366193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f448c11b7e0 0x7f448c1174c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.575+0000 7f4491400640 1 -- 172.21.15.28:0/1169366193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f448c10e880 msgr2=0x7f448c11ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.575+0000 7f4491400640 1 --2- 172.21.15.28:0/1169366193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f448c10e880 0x7f448c11ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.575+0000 7f4491400640 1 -- 172.21.15.28:0/1169366193 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f448c117ab0 con 0x7f448c110850 2024-04-24T08:13:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.576+0000 7f4491400640 1 --2- 172.21.15.28:0/1169366193 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f448c110850 0x7f448c116f80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f448000ea20 tx=0x7f448000eee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.577+0000 7f4486a00640 1 -- 172.21.15.28:0/1169366193 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448000cd80 con 0x7f448c110850 2024-04-24T08:13:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.577+0000 7f4486a00640 1 -- 172.21.15.28:0/1169366193 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4480010910 con 0x7f448c110850 2024-04-24T08:13:49.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.577+0000 7f4486a00640 1 -- 172.21.15.28:0/1169366193 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4480017900 con 0x7f448c110850 2024-04-24T08:13:49.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.578+0000 7f4494800640 1 -- 172.21.15.28:0/1169366193 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f448c117da0 con 0x7f448c110850 2024-04-24T08:13:49.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.578+0000 7f4494800640 1 -- 172.21.15.28:0/1169366193 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f448c1ab8f0 con 0x7f448c110850 2024-04-24T08:13:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.583+0000 7f44977b1740 1 Processor -- start 2024-04-24T08:13:49.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.583+0000 7f44977b1740 1 -- start start 2024-04-24T08:13:49.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.587+0000 7f44977b1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f9240e950 0x558f9240ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.587+0000 7f44977b1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240f270 0x558f92419400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.587+0000 7f44977b1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f92419940 0x558f9241bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.587+0000 7f44977b1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558f923741d0 con 0x558f9240e950 2024-04-24T08:13:49.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.587+0000 7f44977b1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558f9231a450 con 0x558f92419940 2024-04-24T08:13:49.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.587+0000 7f44977b1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558f92370190 con 0x558f9240f270 2024-04-24T08:13:49.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.588+0000 7f4491e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240f270 0x558f92419400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.588+0000 7f4492800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f9240e950 0x558f9240ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.589+0000 7f4491e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240f270 0x558f92419400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:38688/0 (socket says 172.21.15.28:38688) 2024-04-24T08:13:49.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.589+0000 7f4491e00640 1 -- 172.21.15.28:0/1439597243 learned_addr learned my addr 172.21.15.28:0/1439597243 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:49.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.589+0000 7f4486a00640 1 -- 172.21.15.28:0/1169366193 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4480010ac0 con 0x7f448c110850 2024-04-24T08:13:49.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.590+0000 7f4491400640 1 --2- 172.21.15.28:0/1439597243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f92419940 0x558f9241bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.592+0000 7f4491e00640 1 -- 172.21.15.28:0/1439597243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f92419940 msgr2=0x558f9241bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:49.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.592+0000 7f4491e00640 1 --2- 172.21.15.28:0/1439597243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f92419940 0x558f9241bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.592+0000 7f4491e00640 1 -- 172.21.15.28:0/1439597243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f9240e950 msgr2=0x558f9240ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:49.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.592+0000 7f4491e00640 1 --2- 172.21.15.28:0/1439597243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f9240e950 0x558f9240ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.592+0000 7f4491e00640 1 -- 172.21.15.28:0/1439597243 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f923d7310 con 0x558f9240f270 2024-04-24T08:13:49.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.592+0000 7f4491400640 1 --2- 172.21.15.28:0/1439597243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f92419940 0x558f9241bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:49.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.593+0000 7f4486a00640 1 --2- 172.21.15.28:0/1169366193 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f44740a3f20 0x7f44740a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.593+0000 7f4486a00640 1 -- 172.21.15.28:0/1169366193 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f44800d0e30 con 0x7f448c110850 2024-04-24T08:13:49.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.594+0000 7f4492800640 1 --2- 172.21.15.28:0/1169366193 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f44740a3f20 0x7f44740a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.594+0000 7f4491e00640 1 --2- 172.21.15.28:0/1439597243 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240f270 0x558f92419400 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f448800a4b0 tx=0x7f44880096d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f642207c05c0a694 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:49.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.595+0000 7f447b000640 1 -- 172.21.15.28:0/1439597243 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4488018020 con 0x558f9240f270 2024-04-24T08:13:49.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.595+0000 7f447b000640 1 -- 172.21.15.28:0/1439597243 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4488014cf0 con 0x558f9240f270 2024-04-24T08:13:49.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.595+0000 7f447b000640 1 -- 172.21.15.28:0/1439597243 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4488013db0 con 0x558f9240f270 2024-04-24T08:13:49.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.596+0000 7f44977b1740 1 -- 172.21.15.28:0/1439597243 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240f270 msgr2=0x558f92419400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.596+0000 7f44977b1740 1 --2- 172.21.15.28:0/1439597243 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240f270 0x558f92419400 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f448800a4b0 tx=0x7f44880096d0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.596+0000 7f4492800640 1 --2- 172.21.15.28:0/1169366193 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f44740a3f20 0x7f44740a63e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f447c0058c0 tx=0x7f447c005850 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:49.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.596+0000 7f44977b1740 1 -- 172.21.15.28:0/1439597243 shutdown_connections 2024-04-24T08:13:49.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.596+0000 7f44977b1740 1 --2- 172.21.15.28:0/1439597243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f92419940 0x558f9241bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.596+0000 7f44977b1740 1 --2- 172.21.15.28:0/1439597243 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240f270 0x558f92419400 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.596+0000 7f44977b1740 1 --2- 172.21.15.28:0/1439597243 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f9240e950 0x558f9240ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.596+0000 7f44977b1740 1 -- 172.21.15.28:0/1439597243 >> 172.21.15.28:0/1439597243 conn(0x558f9240ac70 msgr2=0x558f9240b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:49.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.597+0000 7f44977b1740 1 -- 172.21.15.28:0/1439597243 shutdown_connections 2024-04-24T08:13:49.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.600+0000 7f44977b1740 1 -- 172.21.15.28:0/1439597243 wait complete. 2024-04-24T08:13:49.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.600+0000 7f44977b1740 1 -- start start 2024-04-24T08:13:49.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.601+0000 7f44977b1740 1 Processor -- start 2024-04-24T08:13:49.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.604+0000 7f44977b1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240e950 0x558f92416700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.605+0000 7f44977b1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f9240f270 0x558f92416c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.605+0000 7f44977b1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f92419940 0x558f92417180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.605+0000 7f44977b1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558f9241c7d0 con 0x558f9240f270 2024-04-24T08:13:49.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.605+0000 7f44977b1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558f9232d810 con 0x558f92419940 2024-04-24T08:13:49.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.605+0000 7f44977b1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558f9241c950 con 0x558f9240e950 2024-04-24T08:13:49.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.606+0000 7f4491e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240e950 0x558f92416700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.609+0000 7f4491e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240e950 0x558f92416700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:38692/0 (socket says 172.21.15.28:38692) 2024-04-24T08:13:49.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.609+0000 7f4491e00640 1 -- 172.21.15.28:0/3924845584 learned_addr learned my addr 172.21.15.28:0/3924845584 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:49.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.609+0000 7f44977b1740 10 client.0 _ll_register_callbacks cb 0x558f922fb140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:49.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.609+0000 7f4492800640 1 --2- 172.21.15.28:0/3924845584 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f92419940 0x558f92417180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.607+0000 7f4491400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f9240f270 0x558f92416c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.611+0000 7f4492800640 1 -- 172.21.15.28:0/3924845584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240e950 msgr2=0x558f92416700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:49.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.611+0000 7f4492800640 1 --2- 172.21.15.28:0/3924845584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240e950 0x558f92416700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.611+0000 7f4492800640 1 -- 172.21.15.28:0/3924845584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f9240f270 msgr2=0x558f92416c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.611+0000 7f4492800640 1 --2- 172.21.15.28:0/3924845584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f9240f270 0x558f92416c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.611+0000 7f4492800640 1 -- 172.21.15.28:0/3924845584 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f9233e3f0 con 0x558f92419940 2024-04-24T08:13:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.613+0000 7f4492800640 1 --2- 172.21.15.28:0/3924845584 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f92419940 0x558f92417180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f447c01a860 tx=0x7f447c01a270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1039f29657e3754d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:49.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.613+0000 7f446f400640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:13:49.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.615+0000 7f446f400640 1 -- 172.21.15.28:0/3924845584 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f447c0054c0 con 0x558f92419940 2024-04-24T08:13:49.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.615+0000 7f44977b1740 1 -- 172.21.15.28:0/3924845584 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4488014cf0 con 0x558f92419940 2024-04-24T08:13:49.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.615+0000 7f446f400640 1 -- 172.21.15.28:0/3924845584 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f447c003800 con 0x558f92419940 2024-04-24T08:13:49.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.616+0000 7f446f400640 1 -- 172.21.15.28:0/3924845584 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f447c003a00 con 0x558f92419940 2024-04-24T08:13:49.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.616+0000 7f44977b1740 10 client.26009 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:49.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.616+0000 7f44977b1740 1 -- 172.21.15.28:0/3924845584 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558f920863f0 con 0x558f92419940 2024-04-24T08:13:49.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.616+0000 7f446f400640 1 -- 172.21.15.28:0/3924845584 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4488014cf0 con 0x558f92419940 2024-04-24T08:13:49.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.617+0000 7f446f400640 1 -- 172.21.15.28:0/3924845584 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f447c042440 con 0x558f92419940 2024-04-24T08:13:49.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.617+0000 7f44977b1740 10 client.26009 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:49.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.617+0000 7f44977b1740 10 client.26009 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:13:49.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.617+0000 7f44977b1740 20 client.26009 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:49.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.618+0000 7f44977b1740 1 --2- 172.21.15.28:0/3924845584 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558f92429b00 0x558f9242bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.618+0000 7f44977b1740 4 client.26009 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:13:49.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.618+0000 7f44977b1740 1 -- 172.21.15.28:0/3924845584 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558f92381bf0 con 0x558f92429b00 2024-04-24T08:13:49.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.620+0000 7f4491e00640 1 --2- 172.21.15.28:0/3924845584 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558f92429b00 0x558f9242bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.622+0000 7f4491e00640 1 --2- 172.21.15.28:0/3924845584 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558f92429b00 0x558f9242bef0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e4c56155a1e0a003 server_cookie=508b5876968d4d6 in_seq=0 out_seq=0 2024-04-24T08:13:49.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.622+0000 7f446f400640 10 client.26009 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:13:49.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.624+0000 7f446f400640 1 -- 172.21.15.28:0/3924845584 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558f92381bf0 con 0x558f92429b00 2024-04-24T08:13:49.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.624+0000 7f446f400640 10 client.26009 handle_command_reply: tid=0 2024-04-24T08:13:49.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:49 smithi117 ceph-mon[32438]: pgmap v107: 97 pgs: 97 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 101 MiB/s wr, 14.94k op/s 2024-04-24T08:13:49.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:49 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2257665825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:13:49.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.630+0000 7f44977b1740 1 client.26009 shutdown 2024-04-24T08:13:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.633+0000 7f44977b1740 1 -- 172.21.15.28:0/3924845584 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f92419940 msgr2=0x558f92417180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.634+0000 7f44977b1740 1 --2- 172.21.15.28:0/3924845584 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f92419940 0x558f92417180 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f447c01a860 tx=0x7f447c01a270 comp rx=0 tx=0).stop 2024-04-24T08:13:49.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.637+0000 7f44977b1740 1 -- 172.21.15.28:0/3924845584 shutdown_connections 2024-04-24T08:13:49.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.637+0000 7f44977b1740 1 --2- 172.21.15.28:0/3924845584 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558f92429b00 0x558f9242bef0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.637+0000 7f44977b1740 1 --2- 172.21.15.28:0/3924845584 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f9240f270 0x558f92416c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.637+0000 7f44977b1740 1 --2- 172.21.15.28:0/3924845584 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f92419940 0x558f92417180 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.637+0000 7f44977b1740 1 --2- 172.21.15.28:0/3924845584 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240e950 0x558f92416700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.637+0000 7f44977b1740 1 -- 172.21.15.28:0/3924845584 >> 172.21.15.28:0/3924845584 conn(0x558f9240ac70 msgr2=0x558f92410a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:49.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.637+0000 7f446f400640 0 client.26009 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:13:49.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.638+0000 7f44977b1740 1 -- 172.21.15.28:0/3924845584 shutdown_connections 2024-04-24T08:13:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.641+0000 7f44977b1740 1 -- 172.21.15.28:0/3924845584 wait complete. 2024-04-24T08:13:49.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.641+0000 7f44977b1740 20 client.26009 trim_cache size 0 max 16384 2024-04-24T08:13:49.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.674+0000 7f44977b1740 1 Processor -- start 2024-04-24T08:13:49.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.677+0000 7f44977b1740 1 -- start start 2024-04-24T08:13:49.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.677+0000 7f44977b1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f9240e950 0x558f92425690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.678+0000 7f44977b1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f9240f270 0x558f92415860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.678+0000 7f44977b1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f92419940 0x558f92416fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.678+0000 7f44977b1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f448c1b6db0 con 0x558f9240f270 2024-04-24T08:13:49.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.678+0000 7f44977b1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f448c1b4680 con 0x558f9240e950 2024-04-24T08:13:49.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.678+0000 7f44977b1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f448c1b24d0 con 0x558f92419940 2024-04-24T08:13:49.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.678+0000 7f4491e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f9240e950 0x558f92425690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.678+0000 7f4491400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f9240f270 0x558f92415860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.678+0000 7f4491e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f9240e950 0x558f92425690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:36562/0 (socket says 172.21.15.28:36562) 2024-04-24T08:13:49.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.678+0000 7f4491e00640 1 -- 172.21.15.28:0/4288115712 learned_addr learned my addr 172.21.15.28:0/4288115712 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:49.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.678+0000 7f4492800640 1 --2- 172.21.15.28:0/4288115712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f92419940 0x558f92416fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.682+0000 7f4491e00640 1 -- 172.21.15.28:0/4288115712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f92419940 msgr2=0x558f92416fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:49.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.682+0000 7f4491e00640 1 --2- 172.21.15.28:0/4288115712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f92419940 0x558f92416fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.682+0000 7f4491e00640 1 -- 172.21.15.28:0/4288115712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f9240f270 msgr2=0x558f92415860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.682+0000 7f4491e00640 1 --2- 172.21.15.28:0/4288115712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f9240f270 0x558f92415860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.682+0000 7f4491e00640 1 -- 172.21.15.28:0/4288115712 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558f921507d0 con 0x558f9240e950 2024-04-24T08:13:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.682+0000 7f4491400640 1 --2- 172.21.15.28:0/4288115712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f9240f270 0x558f92415860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:13:49.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.683+0000 7f4491e00640 1 --2- 172.21.15.28:0/4288115712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f9240e950 0x558f92425690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4488007170 tx=0x7f448800c040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d54faa6b6d5b214 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.683+0000 7f447a600640 1 -- 172.21.15.28:0/4288115712 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4488018040 con 0x558f9240e950 2024-04-24T08:13:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.684+0000 7f447a600640 1 -- 172.21.15.28:0/4288115712 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4488013e60 con 0x558f9240e950 2024-04-24T08:13:49.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.684+0000 7f447a600640 1 -- 172.21.15.28:0/4288115712 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f448802a2d0 con 0x558f9240e950 2024-04-24T08:13:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.685+0000 7f44977b1740 1 -- 172.21.15.28:0/4288115712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f9240e950 msgr2=0x558f92425690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.685+0000 7f44977b1740 1 --2- 172.21.15.28:0/4288115712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f9240e950 0x558f92425690 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f4488007170 tx=0x7f448800c040 comp rx=0 tx=0).stop 2024-04-24T08:13:49.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.686+0000 7f44977b1740 1 -- 172.21.15.28:0/4288115712 shutdown_connections 2024-04-24T08:13:49.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.686+0000 7f44977b1740 1 --2- 172.21.15.28:0/4288115712 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f9240f270 0x558f92415860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.686+0000 7f44977b1740 1 --2- 172.21.15.28:0/4288115712 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f92419940 0x558f92416fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.686+0000 7f44977b1740 1 --2- 172.21.15.28:0/4288115712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f9240e950 0x558f92425690 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.686+0000 7f44977b1740 1 -- 172.21.15.28:0/4288115712 >> 172.21.15.28:0/4288115712 conn(0x558f9240ac70 msgr2=0x558f92409560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.687+0000 7f44977b1740 1 -- 172.21.15.28:0/4288115712 shutdown_connections 2024-04-24T08:13:49.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.687+0000 7f44977b1740 1 -- 172.21.15.28:0/4288115712 wait complete. 2024-04-24T08:13:49.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.687+0000 7f44977b1740 1 -- start start 2024-04-24T08:13:49.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.688+0000 7f44977b1740 1 Processor -- start 2024-04-24T08:13:49.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.689+0000 7f44977b1740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f9240e950 0x558f924171f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.689+0000 7f44977b1740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240f270 0x558f924155d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.690+0000 7f4491e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f9240e950 0x558f924171f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:49 smithi194 ceph-mon[32836]: pgmap v107: 97 pgs: 97 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 101 MiB/s wr, 14.94k op/s 2024-04-24T08:13:49.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:49 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2257665825' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:13:49.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.690+0000 7f44977b1740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f92419940 0x558f92415c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.690+0000 7f44977b1740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f448c1b2b40 con 0x558f92419940 2024-04-24T08:13:49.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.690+0000 7f44977b1740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f448c1b3fb0 con 0x558f9240e950 2024-04-24T08:13:49.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.690+0000 7f44977b1740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f448c1b3210 con 0x558f9240f270 2024-04-24T08:13:49.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.690+0000 7f44977b1740 10 client.0 _ll_register_callbacks cb 0x7f447c01a790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:49.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.690+0000 7f4491e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f9240e950 0x558f924171f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:36572/0 (socket says 172.21.15.28:36572) 2024-04-24T08:13:49.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.690+0000 7f4491e00640 1 -- 172.21.15.28:0/4098533414 learned_addr learned my addr 172.21.15.28:0/4098533414 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:49.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.690+0000 7f4491400640 1 --2- 172.21.15.28:0/4098533414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240f270 0x558f924155d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.691+0000 7f4492800640 1 --2- 172.21.15.28:0/4098533414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f92419940 0x558f92415c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.692+0000 7f4491400640 1 -- 172.21.15.28:0/4098533414 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f9240e950 msgr2=0x558f924171f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:49.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.692+0000 7f4491400640 1 --2- 172.21.15.28:0/4098533414 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f9240e950 0x558f924171f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.692+0000 7f4491400640 1 -- 172.21.15.28:0/4098533414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f92419940 msgr2=0x558f92415c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:49.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.692+0000 7f4491400640 1 --2- 172.21.15.28:0/4098533414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f92419940 0x558f92415c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.692+0000 7f4491400640 1 -- 172.21.15.28:0/4098533414 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558f92420ba0 con 0x558f9240f270 2024-04-24T08:13:49.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.693+0000 7f4491e00640 1 --2- 172.21.15.28:0/4098533414 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f9240e950 0x558f924171f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:13:49.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.693+0000 7f4491400640 1 --2- 172.21.15.28:0/4098533414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240f270 0x558f924155d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f44800ce500 tx=0x7f448009c950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d5b82014ffcc5e3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:49.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.693+0000 7f4492800640 1 --2- 172.21.15.28:0/4098533414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f92419940 0x558f92415c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:49.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.694+0000 7f446f400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:13:49.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.694+0000 7f446f400640 1 -- 172.21.15.28:0/4098533414 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44800a1100 con 0x558f9240f270 2024-04-24T08:13:49.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.694+0000 7f446f400640 1 -- 172.21.15.28:0/4098533414 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f44800993c0 con 0x558f9240f270 2024-04-24T08:13:49.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.694+0000 7f44977b1740 1 -- 172.21.15.28:0/4098533414 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4488013e60 con 0x558f9240f270 2024-04-24T08:13:49.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.695+0000 7f446f400640 1 -- 172.21.15.28:0/4098533414 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f44800a59c0 con 0x558f9240f270 2024-04-24T08:13:49.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.697+0000 7f44977b1740 10 client.26062 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:49.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.697+0000 7f44977b1740 1 -- 172.21.15.28:0/4098533414 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558f92421f00 con 0x558f9240f270 2024-04-24T08:13:49.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.699+0000 7f446f400640 1 -- 172.21.15.28:0/4098533414 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4488013e60 con 0x558f9240f270 2024-04-24T08:13:49.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.700+0000 7f446f400640 1 -- 172.21.15.28:0/4098533414 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f44800d4020 con 0x558f9240f270 2024-04-24T08:13:49.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.700+0000 7f44977b1740 10 client.26062 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:49.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.700+0000 7f44977b1740 10 client.26062 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:13:49.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.700+0000 7f44977b1740 20 client.26062 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:49.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.700+0000 7f44977b1740 1 --2- 172.21.15.28:0/4098533414 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558f9248ffe0 0x558f924923d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:49.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.700+0000 7f44977b1740 4 client.26062 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:13:49.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.700+0000 7f44977b1740 1 -- 172.21.15.28:0/4098533414 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558f9237ba60 con 0x558f9248ffe0 2024-04-24T08:13:49.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.701+0000 7f4491e00640 1 --2- 172.21.15.28:0/4098533414 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558f9248ffe0 0x558f924923d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:49.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.703+0000 7f4491e00640 1 --2- 172.21.15.28:0/4098533414 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558f9248ffe0 0x558f924923d0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cd42786ed8db001f server_cookie=f6032b393c0cb99b in_seq=0 out_seq=0 2024-04-24T08:13:49.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.709+0000 7f446f400640 10 client.26062 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:13:49.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.709+0000 7f446f400640 1 -- 172.21.15.28:0/4098533414 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558f9237ba60 con 0x558f9248ffe0 2024-04-24T08:13:49.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.709+0000 7f446f400640 10 client.26062 handle_command_reply: tid=0 2024-04-24T08:13:49.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.709+0000 7f44977b1740 1 client.26062 shutdown 2024-04-24T08:13:49.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.711+0000 7f44977b1740 1 -- 172.21.15.28:0/4098533414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240f270 msgr2=0x558f924155d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.711+0000 7f44977b1740 1 --2- 172.21.15.28:0/4098533414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240f270 0x558f924155d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f44800ce500 tx=0x7f448009c950 comp rx=0 tx=0).stop 2024-04-24T08:13:49.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.711+0000 7f44977b1740 1 -- 172.21.15.28:0/4098533414 shutdown_connections 2024-04-24T08:13:49.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.711+0000 7f44977b1740 1 --2- 172.21.15.28:0/4098533414 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558f9248ffe0 0x558f924923d0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.711+0000 7f44977b1740 1 --2- 172.21.15.28:0/4098533414 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558f92419940 0x558f92415c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.711+0000 7f44977b1740 1 --2- 172.21.15.28:0/4098533414 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558f9240f270 0x558f924155d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.711+0000 7f44977b1740 1 --2- 172.21.15.28:0/4098533414 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558f9240e950 0x558f924171f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.711+0000 7f446f400640 0 client.26062 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:13:49.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.711+0000 7f44977b1740 1 -- 172.21.15.28:0/4098533414 >> 172.21.15.28:0/4098533414 conn(0x558f9240ac70 msgr2=0x558f92417f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:49.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.713+0000 7f44977b1740 1 -- 172.21.15.28:0/4098533414 shutdown_connections 2024-04-24T08:13:49.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.716+0000 7f44977b1740 1 -- 172.21.15.28:0/4098533414 wait complete. 2024-04-24T08:13:49.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.716+0000 7f44977b1740 20 client.26062 trim_cache size 0 max 16384 2024-04-24T08:13:49.720 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:13:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.720+0000 7f4494800640 1 -- 172.21.15.28:0/1169366193 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f44740a3f20 msgr2=0x7f44740a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.720+0000 7f4494800640 1 --2- 172.21.15.28:0/1169366193 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f44740a3f20 0x7f44740a63e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f447c0058c0 tx=0x7f447c005850 comp rx=0 tx=0).stop 2024-04-24T08:13:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.720+0000 7f4494800640 1 -- 172.21.15.28:0/1169366193 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f448c110850 msgr2=0x7f448c116f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.720+0000 7f4494800640 1 --2- 172.21.15.28:0/1169366193 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f448c110850 0x7f448c116f80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f448000ea20 tx=0x7f448000eee0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.721+0000 7f4494800640 1 -- 172.21.15.28:0/1169366193 shutdown_connections 2024-04-24T08:13:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.721+0000 7f4494800640 1 --2- 172.21.15.28:0/1169366193 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f44740a3f20 0x7f44740a63e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.721+0000 7f4494800640 1 --2- 172.21.15.28:0/1169366193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f448c11b7e0 0x7f448c1174c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.721+0000 7f4494800640 1 --2- 172.21.15.28:0/1169366193 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f448c110850 0x7f448c116f80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.721+0000 7f4494800640 1 --2- 172.21.15.28:0/1169366193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f448c10e880 0x7f448c11ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:49.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.721+0000 7f4494800640 1 -- 172.21.15.28:0/1169366193 >> 172.21.15.28:0/1169366193 conn(0x7f448c06d110 msgr2=0x7f448c11ce50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:49.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.721+0000 7f4494800640 1 -- 172.21.15.28:0/1169366193 shutdown_connections 2024-04-24T08:13:49.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:49.721+0000 7f4494800640 1 -- 172.21.15.28:0/1169366193 wait complete. 2024-04-24T08:13:49.779 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:13:50.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.156+0000 7fade0c00640 1 -- 172.21.15.28:0/4128361242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadd4098160 msgr2=0x7fadd4098540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.156+0000 7fade0c00640 1 --2- 172.21.15.28:0/4128361242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadd4098160 0x7fadd4098540 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fadd0009900 tx=0x7fadd002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.157+0000 7fade0c00640 1 -- 172.21.15.28:0/4128361242 shutdown_connections 2024-04-24T08:13:50.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.157+0000 7fade0c00640 1 --2- 172.21.15.28:0/4128361242 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd409d230 0x7fadd40a4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.157+0000 7fade0c00640 1 --2- 172.21.15.28:0/4128361242 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd4098b10 0x7fadd409cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.157+0000 7fade0c00640 1 --2- 172.21.15.28:0/4128361242 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadd4098160 0x7fadd4098540 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.157+0000 7fade0c00640 1 -- 172.21.15.28:0/4128361242 >> 172.21.15.28:0/4128361242 conn(0x7fadd40939c0 msgr2=0x7fadd4095de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:50.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.158+0000 7fade0c00640 1 -- 172.21.15.28:0/4128361242 shutdown_connections 2024-04-24T08:13:50.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.158+0000 7fade0c00640 1 -- 172.21.15.28:0/4128361242 wait complete. 2024-04-24T08:13:50.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.159+0000 7fade0c00640 1 Processor -- start 2024-04-24T08:13:50.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.160+0000 7fade0c00640 1 -- start start 2024-04-24T08:13:50.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.160+0000 7fade0c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd4098160 0x7fadd41343f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.160+0000 7fade0c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadd409d230 0x7fadd4134930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.160+0000 7fade0c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd412e5f0 0x7fadd412eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.160+0000 7fade0c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadd40a8310 con 0x7fadd4098160 2024-04-24T08:13:50.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.160+0000 7fade0c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fadd40a8190 con 0x7fadd409d230 2024-04-24T08:13:50.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.160+0000 7fade0c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fadd40a8490 con 0x7fadd412e5f0 2024-04-24T08:13:50.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.161+0000 7faddbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd412e5f0 0x7fadd412eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.161+0000 7faddbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd412e5f0 0x7fadd412eaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:46926/0 (socket says 172.21.15.28:46926) 2024-04-24T08:13:50.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.161+0000 7faddbe00640 1 -- 172.21.15.28:0/808476599 learned_addr learned my addr 172.21.15.28:0/808476599 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:50.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.161+0000 7faddaa00640 1 --2- 172.21.15.28:0/808476599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadd409d230 0x7fadd4134930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.162+0000 7faddaa00640 1 -- 172.21.15.28:0/808476599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd412e5f0 msgr2=0x7fadd412eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.162+0000 7faddaa00640 1 --2- 172.21.15.28:0/808476599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd412e5f0 0x7fadd412eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.162+0000 7faddaa00640 1 -- 172.21.15.28:0/808476599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd4098160 msgr2=0x7fadd41343f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.162+0000 7faddaa00640 1 --2- 172.21.15.28:0/808476599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd4098160 0x7fadd41343f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.162+0000 7faddaa00640 1 -- 172.21.15.28:0/808476599 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadd412f110 con 0x7fadd409d230 2024-04-24T08:13:50.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.162+0000 7faddaa00640 1 --2- 172.21.15.28:0/808476599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadd409d230 0x7fadd4134930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fadc800c940 tx=0x7fadc800ce00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:50.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.163+0000 7fadcfe00640 1 -- 172.21.15.28:0/808476599 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc8007bb0 con 0x7fadd409d230 2024-04-24T08:13:50.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.163+0000 7fadcfe00640 1 -- 172.21.15.28:0/808476599 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadc8007d50 con 0x7fadd409d230 2024-04-24T08:13:50.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.163+0000 7fadcfe00640 1 -- 172.21.15.28:0/808476599 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc800e7d0 con 0x7fadd409d230 2024-04-24T08:13:50.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.163+0000 7fade0c00640 1 -- 172.21.15.28:0/808476599 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadd412f400 con 0x7fadd409d230 2024-04-24T08:13:50.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.163+0000 7fade0c00640 1 -- 172.21.15.28:0/808476599 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadd413b650 con 0x7fadd409d230 2024-04-24T08:13:50.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.167+0000 7fade504e740 1 Processor -- start 2024-04-24T08:13:50.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.169+0000 7fade504e740 1 -- start start 2024-04-24T08:13:50.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.169+0000 7fadcfe00640 1 -- 172.21.15.28:0/808476599 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fadc800f450 con 0x7fadd409d230 2024-04-24T08:13:50.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.169+0000 7fade504e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6ea350 0x564e7c6ea730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.170+0000 7fade504e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6eac70 0x564e7c6f4e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.170+0000 7fade504e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6f5340 0x564e7c6f7730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.170+0000 7fade504e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564e7c6471d0 con 0x564e7c6eac70 2024-04-24T08:13:50.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.170+0000 7fade504e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564e7c5ed450 con 0x564e7c6f5340 2024-04-24T08:13:50.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.170+0000 7fade504e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564e7c643190 con 0x564e7c6ea350 2024-04-24T08:13:50.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.170+0000 7faddb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6eac70 0x564e7c6f4e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.171+0000 7faddbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6ea350 0x564e7c6ea730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.171+0000 7faddbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6ea350 0x564e7c6ea730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:46934/0 (socket says 172.21.15.28:46934) 2024-04-24T08:13:50.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.171+0000 7faddbe00640 1 -- 172.21.15.28:0/579311676 learned_addr learned my addr 172.21.15.28:0/579311676 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:50.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.171+0000 7fadcfe00640 1 --2- 172.21.15.28:0/808476599 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fadc40a3e20 0x7fadc40a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.172+0000 7faddaa00640 1 --2- 172.21.15.28:0/579311676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6f5340 0x564e7c6f7730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.172+0000 7faddbe00640 1 -- 172.21.15.28:0/579311676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6f5340 msgr2=0x564e7c6f7730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:50.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.172+0000 7faddbe00640 1 --2- 172.21.15.28:0/579311676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6f5340 0x564e7c6f7730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.172+0000 7faddbe00640 1 -- 172.21.15.28:0/579311676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6eac70 msgr2=0x564e7c6f4e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:50.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.172+0000 7faddbe00640 1 --2- 172.21.15.28:0/579311676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6eac70 0x564e7c6f4e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.172+0000 7faddbe00640 1 -- 172.21.15.28:0/579311676 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e7c6aa310 con 0x564e7c6ea350 2024-04-24T08:13:50.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.172+0000 7faddbe00640 1 --2- 172.21.15.28:0/808476599 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fadc40a3e20 0x7fadc40a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.173+0000 7fadcfe00640 1 -- 172.21.15.28:0/808476599 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fadc80d0cd0 con 0x7fadd409d230 2024-04-24T08:13:50.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.173+0000 7faddbe00640 1 --2- 172.21.15.28:0/579311676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6ea350 0x564e7c6ea730 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faddc050e50 tx=0x7faddc050330 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f1eed14d242a8a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:50.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.176+0000 7fadc3e00640 1 -- 172.21.15.28:0/579311676 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faddc083670 con 0x564e7c6ea350 2024-04-24T08:13:50.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.176+0000 7fadc3e00640 1 -- 172.21.15.28:0/579311676 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faddc068e50 con 0x564e7c6ea350 2024-04-24T08:13:50.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.176+0000 7fadc3e00640 1 -- 172.21.15.28:0/579311676 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faddc084a10 con 0x564e7c6ea350 2024-04-24T08:13:50.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.176+0000 7faddbe00640 1 --2- 172.21.15.28:0/808476599 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fadc40a3e20 0x7fadc40a62e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7faddc050810 tx=0x7faddc086000 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:50.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.177+0000 7fade504e740 1 -- 172.21.15.28:0/579311676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6ea350 msgr2=0x564e7c6ea730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.177+0000 7fade504e740 1 --2- 172.21.15.28:0/579311676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6ea350 0x564e7c6ea730 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7faddc050e50 tx=0x7faddc050330 comp rx=0 tx=0).stop 2024-04-24T08:13:50.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.178+0000 7fade504e740 1 -- 172.21.15.28:0/579311676 shutdown_connections 2024-04-24T08:13:50.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.178+0000 7fade504e740 1 --2- 172.21.15.28:0/579311676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6eac70 0x564e7c6f4e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.178+0000 7fade504e740 1 --2- 172.21.15.28:0/579311676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6f5340 0x564e7c6f7730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.178+0000 7fade504e740 1 --2- 172.21.15.28:0/579311676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6ea350 0x564e7c6ea730 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.178+0000 7fade504e740 1 -- 172.21.15.28:0/579311676 >> 172.21.15.28:0/579311676 conn(0x564e7c6e6670 msgr2=0x564e7c6e6a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:50.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.178+0000 7fade504e740 1 -- 172.21.15.28:0/579311676 shutdown_connections 2024-04-24T08:13:50.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.180+0000 7fade504e740 1 -- 172.21.15.28:0/579311676 wait complete. 2024-04-24T08:13:50.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.181+0000 7fade504e740 1 -- start start 2024-04-24T08:13:50.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.181+0000 7fade504e740 1 Processor -- start 2024-04-24T08:13:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.182+0000 7fade504e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6ea350 0x564e7c6f2140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.182+0000 7fade504e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6eac70 0x564e7c6f2680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.182+0000 7fade504e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6f5340 0x564e7c6f2bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.182+0000 7fade504e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564e7c7091e0 con 0x564e7c6eac70 2024-04-24T08:13:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.182+0000 7fade504e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564e7c600810 con 0x564e7c6f5340 2024-04-24T08:13:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.182+0000 7fade504e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564e7c709360 con 0x564e7c6ea350 2024-04-24T08:13:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.182+0000 7fade504e740 10 client.0 _ll_register_callbacks cb 0x564e7c5ce140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.183+0000 7faddb400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6ea350 0x564e7c6f2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.183+0000 7faddb400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6ea350 0x564e7c6f2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:46940/0 (socket says 172.21.15.28:46940) 2024-04-24T08:13:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.183+0000 7faddb400640 1 -- 172.21.15.28:0/3806319233 learned_addr learned my addr 172.21.15.28:0/3806319233 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.183+0000 7faddb400640 1 -- 172.21.15.28:0/3806319233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6f5340 msgr2=0x564e7c6f2bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.183+0000 7faddb400640 1 --2- 172.21.15.28:0/3806319233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6f5340 0x564e7c6f2bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.183+0000 7faddb400640 1 -- 172.21.15.28:0/3806319233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6eac70 msgr2=0x564e7c6f2680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.183+0000 7faddb400640 1 --2- 172.21.15.28:0/3806319233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6eac70 0x564e7c6f2680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.183+0000 7faddb400640 1 -- 172.21.15.28:0/3806319233 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e7c6113f0 con 0x564e7c6ea350 2024-04-24T08:13:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.184+0000 7faddb400640 1 --2- 172.21.15.28:0/3806319233 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6ea350 0x564e7c6f2140 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fadd0033000 tx=0x7fadd0005250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ffc5731bf91e80b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:50.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.186+0000 7fadc0c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:13:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.186+0000 7fadc0c00640 1 -- 172.21.15.28:0/3806319233 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadd0040070 con 0x564e7c6ea350 2024-04-24T08:13:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.187+0000 7fadc0c00640 1 -- 172.21.15.28:0/3806319233 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fadd00048c0 con 0x564e7c6ea350 2024-04-24T08:13:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.187+0000 7fade504e740 1 -- 172.21.15.28:0/3806319233 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faddc068e50 con 0x564e7c6ea350 2024-04-24T08:13:50.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.189+0000 7fadc0c00640 1 -- 172.21.15.28:0/3806319233 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fadd004a050 con 0x564e7c6ea350 2024-04-24T08:13:50.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.189+0000 7fade504e740 10 client.26071 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:50.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.189+0000 7fade504e740 1 -- 172.21.15.28:0/3806319233 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564e7c3593f0 con 0x564e7c6ea350 2024-04-24T08:13:50.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.190+0000 7fadc0c00640 1 -- 172.21.15.28:0/3806319233 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faddc068e50 con 0x564e7c6ea350 2024-04-24T08:13:50.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.191+0000 7fadc0c00640 1 -- 172.21.15.28:0/3806319233 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fadd003a040 con 0x564e7c6ea350 2024-04-24T08:13:50.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.191+0000 7fade504e740 10 client.26071 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:50.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.191+0000 7fade504e740 10 client.26071 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:13:50.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.191+0000 7fade504e740 20 client.26071 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:50.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.191+0000 7fade504e740 1 --2- 172.21.15.28:0/3806319233 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x564e7c6e17b0 0x564e7c6e3c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.192+0000 7fade504e740 4 client.26071 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:13:50.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.192+0000 7fade504e740 1 -- 172.21.15.28:0/3806319233 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564e7c654bf0 con 0x564e7c6e17b0 2024-04-24T08:13:50.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.192+0000 7faddaa00640 1 --2- 172.21.15.28:0/3806319233 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x564e7c6e17b0 0x564e7c6e3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.192+0000 7faddaa00640 1 --2- 172.21.15.28:0/3806319233 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x564e7c6e17b0 0x564e7c6e3c40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a02d8d781c91fe2e server_cookie=7dcec4e52c88243 in_seq=0 out_seq=0 2024-04-24T08:13:50.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.193+0000 7fadc0c00640 10 client.26071 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:13:50.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.195+0000 7fadc0c00640 1 -- 172.21.15.28:0/3806319233 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564e7c654bf0 con 0x564e7c6e17b0 2024-04-24T08:13:50.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.195+0000 7fadc0c00640 10 client.26071 handle_command_reply: tid=0 2024-04-24T08:13:50.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.195+0000 7fade504e740 1 client.26071 shutdown 2024-04-24T08:13:50.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.196+0000 7fade504e740 1 -- 172.21.15.28:0/3806319233 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6ea350 msgr2=0x564e7c6f2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.196+0000 7fade504e740 1 --2- 172.21.15.28:0/3806319233 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6ea350 0x564e7c6f2140 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fadd0033000 tx=0x7fadd0005250 comp rx=0 tx=0).stop 2024-04-24T08:13:50.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.196+0000 7fade504e740 1 -- 172.21.15.28:0/3806319233 shutdown_connections 2024-04-24T08:13:50.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.196+0000 7fade504e740 1 --2- 172.21.15.28:0/3806319233 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x564e7c6e17b0 0x564e7c6e3c40 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.196+0000 7fade504e740 1 --2- 172.21.15.28:0/3806319233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6eac70 0x564e7c6f2680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.197+0000 7fade504e740 1 --2- 172.21.15.28:0/3806319233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6f5340 0x564e7c6f2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.197+0000 7fadc0c00640 0 client.26071 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:13:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.197+0000 7fade504e740 1 --2- 172.21.15.28:0/3806319233 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6ea350 0x564e7c6f2140 secure :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fadd0033000 tx=0x7fadd0005250 comp rx=0 tx=0).stop 2024-04-24T08:13:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.197+0000 7fade504e740 1 -- 172.21.15.28:0/3806319233 >> 172.21.15.28:0/3806319233 conn(0x564e7c6e6670 msgr2=0x564e7c6ec470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.197+0000 7fade504e740 1 -- 172.21.15.28:0/3806319233 shutdown_connections 2024-04-24T08:13:50.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.198+0000 7fade504e740 1 -- 172.21.15.28:0/3806319233 wait complete. 2024-04-24T08:13:50.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.198+0000 7fade504e740 20 client.26071 trim_cache size 0 max 16384 2024-04-24T08:13:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.235+0000 7fade504e740 1 Processor -- start 2024-04-24T08:13:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.235+0000 7fade504e740 1 -- start start 2024-04-24T08:13:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.236+0000 7fade504e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6e17b0 0x564e7c7120a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.236+0000 7fade504e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6eac70 0x564e7c6f2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.237+0000 7fade504e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6f5340 0x564e7c728690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.237+0000 7faddb400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6e17b0 0x564e7c7120a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.237+0000 7faddb400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6e17b0 0x564e7c7120a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:46948/0 (socket says 172.21.15.28:46948) 2024-04-24T08:13:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.237+0000 7faddaa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6eac70 0x564e7c6f2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.238+0000 7fade504e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadd4142230 con 0x564e7c6f5340 2024-04-24T08:13:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.238+0000 7fade504e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fadd413fb00 con 0x564e7c6eac70 2024-04-24T08:13:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.238+0000 7faddb400640 1 -- 172.21.15.28:0/774925668 learned_addr learned my addr 172.21.15.28:0/774925668 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.238+0000 7faddaa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6eac70 0x564e7c6f2a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:36632/0 (socket says 172.21.15.28:36632) 2024-04-24T08:13:50.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.238+0000 7fade504e740 1 -- 172.21.15.28:0/774925668 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fadd413d9b0 con 0x564e7c6e17b0 2024-04-24T08:13:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.238+0000 7faddaa00640 1 -- 172.21.15.28:0/774925668 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6e17b0 msgr2=0x564e7c7120a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.238+0000 7faddaa00640 1 --2- 172.21.15.28:0/774925668 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6e17b0 0x564e7c7120a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.238+0000 7faddaa00640 1 -- 172.21.15.28:0/774925668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6f5340 msgr2=0x564e7c728690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.238+0000 7faddaa00640 1 --2- 172.21.15.28:0/774925668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6f5340 0x564e7c728690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.238+0000 7faddaa00640 1 -- 172.21.15.28:0/774925668 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564e7c4237d0 con 0x564e7c6eac70 2024-04-24T08:13:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.239+0000 7faddaa00640 1 --2- 172.21.15.28:0/774925668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6eac70 0x564e7c6f2a10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fadc80ce3a0 tx=0x7fadc80bd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=abd4f613257d97f0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:50.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.239+0000 7fadc3400640 1 -- 172.21.15.28:0/774925668 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc80afa80 con 0x564e7c6eac70 2024-04-24T08:13:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.240+0000 7fadc3400640 1 -- 172.21.15.28:0/774925668 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadc809cbf0 con 0x564e7c6eac70 2024-04-24T08:13:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.241+0000 7fadc3400640 1 -- 172.21.15.28:0/774925668 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc800fac0 con 0x564e7c6eac70 2024-04-24T08:13:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.241+0000 7fade504e740 1 -- 172.21.15.28:0/774925668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6eac70 msgr2=0x564e7c6f2a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.241+0000 7fade504e740 1 --2- 172.21.15.28:0/774925668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6eac70 0x564e7c6f2a10 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fadc80ce3a0 tx=0x7fadc80bd040 comp rx=0 tx=0).stop 2024-04-24T08:13:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.242+0000 7fade504e740 1 -- 172.21.15.28:0/774925668 shutdown_connections 2024-04-24T08:13:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.242+0000 7fade504e740 1 --2- 172.21.15.28:0/774925668 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6f5340 0x564e7c728690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.242+0000 7fade504e740 1 --2- 172.21.15.28:0/774925668 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6eac70 0x564e7c6f2a10 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.242+0000 7fade504e740 1 --2- 172.21.15.28:0/774925668 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6e17b0 0x564e7c7120a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.242+0000 7fade504e740 1 -- 172.21.15.28:0/774925668 >> 172.21.15.28:0/774925668 conn(0x564e7c6e6670 msgr2=0x564e7c6dcf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:50.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.243+0000 7fade504e740 1 -- 172.21.15.28:0/774925668 shutdown_connections 2024-04-24T08:13:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.243+0000 7fade504e740 1 -- 172.21.15.28:0/774925668 wait complete. 2024-04-24T08:13:50.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.244+0000 7fade504e740 1 -- start start 2024-04-24T08:13:50.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.244+0000 7fade504e740 1 Processor -- start 2024-04-24T08:13:50.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.247+0000 7fade504e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6e17b0 0x564e7c6f67b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.247+0000 7fade504e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6eac70 0x564e7c6f6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.247+0000 7faddb400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6e17b0 0x564e7c6f67b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.247+0000 7faddb400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6e17b0 0x564e7c6f67b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:36638/0 (socket says 172.21.15.28:36638) 2024-04-24T08:13:50.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.248+0000 7faddaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6eac70 0x564e7c6f6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.248+0000 7faddaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6eac70 0x564e7c6f6cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47926/0 (socket says 172.21.15.28:47926) 2024-04-24T08:13:50.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.248+0000 7faddb400640 1 -- 172.21.15.28:0/1149184441 learned_addr learned my addr 172.21.15.28:0/1149184441 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:50.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.248+0000 7fade504e740 1 --2- 172.21.15.28:0/1149184441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6f5340 0x564e7c70cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.248+0000 7fade504e740 1 -- 172.21.15.28:0/1149184441 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fadd413e000 con 0x564e7c6eac70 2024-04-24T08:13:50.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.248+0000 7fade504e740 1 -- 172.21.15.28:0/1149184441 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fadd413f430 con 0x564e7c6e17b0 2024-04-24T08:13:50.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.248+0000 7fade504e740 1 -- 172.21.15.28:0/1149184441 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fadd413e690 con 0x564e7c6f5340 2024-04-24T08:13:50.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.248+0000 7fade504e740 10 client.0 _ll_register_callbacks cb 0x7fadd0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:50.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.249+0000 7faddbe00640 1 --2- 172.21.15.28:0/1149184441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6f5340 0x564e7c70cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.249+0000 7faddaa00640 1 -- 172.21.15.28:0/1149184441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6f5340 msgr2=0x564e7c70cbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:50.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.249+0000 7faddaa00640 1 --2- 172.21.15.28:0/1149184441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6f5340 0x564e7c70cbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.249+0000 7faddaa00640 1 -- 172.21.15.28:0/1149184441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6e17b0 msgr2=0x564e7c6f67b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:50.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.249+0000 7faddaa00640 1 --2- 172.21.15.28:0/1149184441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6e17b0 0x564e7c6f67b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.249+0000 7faddaa00640 1 -- 172.21.15.28:0/1149184441 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564e7c70d130 con 0x564e7c6eac70 2024-04-24T08:13:50.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.250+0000 7faddaa00640 1 --2- 172.21.15.28:0/1149184441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6eac70 0x564e7c6f6cf0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x564e7c6f0920 tx=0x7fadc80076c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4619a179e10b281f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:50.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.250+0000 7fadc0c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:13:50.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.250+0000 7fadc0c00640 1 -- 172.21.15.28:0/1149184441 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadc80bb020 con 0x564e7c6eac70 2024-04-24T08:13:50.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.251+0000 7fade504e740 1 -- 172.21.15.28:0/1149184441 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fadc809cbf0 con 0x564e7c6eac70 2024-04-24T08:13:50.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.251+0000 7fadc0c00640 1 -- 172.21.15.28:0/1149184441 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fadc80b3040 con 0x564e7c6eac70 2024-04-24T08:13:50.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.252+0000 7fadc0c00640 1 -- 172.21.15.28:0/1149184441 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fadc800fac0 con 0x564e7c6eac70 2024-04-24T08:13:50.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.252+0000 7fade504e740 10 client.15885 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:50.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.252+0000 7fade504e740 1 -- 172.21.15.28:0/1149184441 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x564e7c785630 con 0x564e7c6eac70 2024-04-24T08:13:50.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.253+0000 7fadc0c00640 1 -- 172.21.15.28:0/1149184441 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadc809cbf0 con 0x564e7c6eac70 2024-04-24T08:13:50.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.253+0000 7fadc0c00640 1 -- 172.21.15.28:0/1149184441 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fadc8020070 con 0x564e7c6eac70 2024-04-24T08:13:50.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.253+0000 7fade504e740 10 client.15885 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:50.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.253+0000 7fade504e740 10 client.15885 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:13:50.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.253+0000 7fade504e740 20 client.15885 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:50.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.253+0000 7fade504e740 1 --2- 172.21.15.28:0/1149184441 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x564e7c785d10 0x564e7c788130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.253+0000 7fade504e740 4 client.15885 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:13:50.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.253+0000 7fade504e740 1 -- 172.21.15.28:0/1149184441 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564e7c64ea60 con 0x564e7c785d10 2024-04-24T08:13:50.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.254+0000 7faddb400640 1 --2- 172.21.15.28:0/1149184441 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x564e7c785d10 0x564e7c788130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.255+0000 7faddb400640 1 --2- 172.21.15.28:0/1149184441 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x564e7c785d10 0x564e7c788130 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=fa2ba6a726e9b5ab server_cookie=fb1786359e7e414e in_seq=0 out_seq=0 2024-04-24T08:13:50.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.255+0000 7fadc0c00640 10 client.15885 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:13:50.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.255+0000 7fadc0c00640 1 -- 172.21.15.28:0/1149184441 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564e7c64ea60 con 0x564e7c785d10 2024-04-24T08:13:50.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.255+0000 7fadc0c00640 10 client.15885 handle_command_reply: tid=0 2024-04-24T08:13:50.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.255+0000 7fade504e740 1 client.15885 shutdown 2024-04-24T08:13:50.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.256+0000 7fade504e740 1 -- 172.21.15.28:0/1149184441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6eac70 msgr2=0x564e7c6f6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.256+0000 7fade504e740 1 --2- 172.21.15.28:0/1149184441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6eac70 0x564e7c6f6cf0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x564e7c6f0920 tx=0x7fadc80076c0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.256+0000 7fade504e740 1 -- 172.21.15.28:0/1149184441 shutdown_connections 2024-04-24T08:13:50.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.256+0000 7fade504e740 1 --2- 172.21.15.28:0/1149184441 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x564e7c785d10 0x564e7c788130 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.256+0000 7fade504e740 1 --2- 172.21.15.28:0/1149184441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564e7c6eac70 0x564e7c6f6cf0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.256+0000 7fadc0c00640 0 client.15885 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:13:50.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.256+0000 7fade504e740 1 --2- 172.21.15.28:0/1149184441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564e7c6f5340 0x564e7c70cbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.256+0000 7fade504e740 1 --2- 172.21.15.28:0/1149184441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564e7c6e17b0 0x564e7c6f67b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.256+0000 7fade504e740 1 -- 172.21.15.28:0/1149184441 >> 172.21.15.28:0/1149184441 conn(0x564e7c6e6670 msgr2=0x564e7c6ebaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:50.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.259+0000 7fade504e740 1 -- 172.21.15.28:0/1149184441 shutdown_connections 2024-04-24T08:13:50.262 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:13:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.261+0000 7fade504e740 1 -- 172.21.15.28:0/1149184441 wait complete. 2024-04-24T08:13:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.261+0000 7fade504e740 20 client.15885 trim_cache size 0 max 16384 2024-04-24T08:13:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.261+0000 7fade0c00640 1 -- 172.21.15.28:0/808476599 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fadc40a3e20 msgr2=0x7fadc40a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.262+0000 7fade0c00640 1 --2- 172.21.15.28:0/808476599 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fadc40a3e20 0x7fadc40a62e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7faddc050810 tx=0x7faddc086000 comp rx=0 tx=0).stop 2024-04-24T08:13:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.262+0000 7fade0c00640 1 -- 172.21.15.28:0/808476599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadd409d230 msgr2=0x7fadd4134930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.262+0000 7fade0c00640 1 --2- 172.21.15.28:0/808476599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadd409d230 0x7fadd4134930 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fadc800c940 tx=0x7fadc800ce00 comp rx=0 tx=0).stop 2024-04-24T08:13:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.262+0000 7fade0c00640 1 -- 172.21.15.28:0/808476599 shutdown_connections 2024-04-24T08:13:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.262+0000 7fade0c00640 1 --2- 172.21.15.28:0/808476599 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fadc40a3e20 0x7fadc40a62e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.262+0000 7fade0c00640 1 --2- 172.21.15.28:0/808476599 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fadd412e5f0 0x7fadd412eaa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.262+0000 7fade0c00640 1 --2- 172.21.15.28:0/808476599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fadd409d230 0x7fadd4134930 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.262+0000 7fade0c00640 1 --2- 172.21.15.28:0/808476599 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fadd4098160 0x7fadd41343f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.262+0000 7fade0c00640 1 -- 172.21.15.28:0/808476599 >> 172.21.15.28:0/808476599 conn(0x7fadd40939c0 msgr2=0x7fadd4095d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:50.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.263+0000 7fade0c00640 1 -- 172.21.15.28:0/808476599 shutdown_connections 2024-04-24T08:13:50.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.263+0000 7fade0c00640 1 -- 172.21.15.28:0/808476599 wait complete. 2024-04-24T08:13:50.302 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:13:50.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.688+0000 7f2693c00640 1 -- 172.21.15.28:0/2022292269 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268c100d60 msgr2=0x7f268c106d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.688+0000 7f2693c00640 1 --2- 172.21.15.28:0/2022292269 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268c100d60 0x7f268c106d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f2674009c10 tx=0x7f267402a6e0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.688+0000 7f2693c00640 1 -- 172.21.15.28:0/2022292269 shutdown_connections 2024-04-24T08:13:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.688+0000 7f2693c00640 1 --2- 172.21.15.28:0/2022292269 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f268c107470 0x7f268c10ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.688+0000 7f2693c00640 1 --2- 172.21.15.28:0/2022292269 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268c100d60 0x7f268c106d40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.688+0000 7f2693c00640 1 --2- 172.21.15.28:0/2022292269 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1003b0 0x7f268c100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.688+0000 7f2693c00640 1 -- 172.21.15.28:0/2022292269 >> 172.21.15.28:0/2022292269 conn(0x7f268c077060 msgr2=0x7f268c0fe450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.688+0000 7f2693c00640 1 -- 172.21.15.28:0/2022292269 shutdown_connections 2024-04-24T08:13:50.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.689+0000 7f2693c00640 1 -- 172.21.15.28:0/2022292269 wait complete. 2024-04-24T08:13:50.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.690+0000 7f2693c00640 1 Processor -- start 2024-04-24T08:13:50.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.691+0000 7f2693c00640 1 -- start start 2024-04-24T08:13:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.691+0000 7f2693c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1003b0 0x7f268c10c830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.691+0000 7f2693c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f268c107470 0x7f268c10cd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.691+0000 7f2693c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268c1a8d40 0x7f268c1ab130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.691+0000 7f2693c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f268c1103c0 con 0x7f268c1003b0 2024-04-24T08:13:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.693+0000 7f2693c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f268c103a20 con 0x7f268c1a8d40 2024-04-24T08:13:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.693+0000 7f2693c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f268c110540 con 0x7f268c107470 2024-04-24T08:13:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.692+0000 7f2691c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268c1a8d40 0x7f268c1ab130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.693+0000 7f268be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f268c107470 0x7f268c10cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.693+0000 7f2691c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268c1a8d40 0x7f268c1ab130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:36668/0 (socket says 172.21.15.28:36668) 2024-04-24T08:13:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.693+0000 7f2691c00640 1 -- 172.21.15.28:0/335415418 learned_addr learned my addr 172.21.15.28:0/335415418 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:50.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.693+0000 7f2691200640 1 --2- 172.21.15.28:0/335415418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1003b0 0x7f268c10c830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.694+0000 7f268be00640 1 -- 172.21.15.28:0/335415418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268c1a8d40 msgr2=0x7f268c1ab130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.694+0000 7f268be00640 1 --2- 172.21.15.28:0/335415418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268c1a8d40 0x7f268c1ab130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.694+0000 7f268be00640 1 -- 172.21.15.28:0/335415418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1003b0 msgr2=0x7f268c10c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.694+0000 7f268be00640 1 --2- 172.21.15.28:0/335415418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1003b0 0x7f268c10c830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.694+0000 7f268be00640 1 -- 172.21.15.28:0/335415418 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f268c114850 con 0x7f268c107470 2024-04-24T08:13:50.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.694+0000 7f268be00640 1 --2- 172.21.15.28:0/335415418 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f268c107470 0x7f268c10cd70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2674034040 tx=0x7f26740305d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:50.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.695+0000 7f2689600640 1 -- 172.21.15.28:0/335415418 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2674041020 con 0x7f268c107470 2024-04-24T08:13:50.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.695+0000 7f2693c00640 1 -- 172.21.15.28:0/335415418 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f268c114ae0 con 0x7f268c107470 2024-04-24T08:13:50.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.695+0000 7f2689600640 1 -- 172.21.15.28:0/335415418 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2674003e50 con 0x7f268c107470 2024-04-24T08:13:50.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.695+0000 7f2689600640 1 -- 172.21.15.28:0/335415418 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2674036040 con 0x7f268c107470 2024-04-24T08:13:50.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.695+0000 7f2693c00640 1 -- 172.21.15.28:0/335415418 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f268c114ff0 con 0x7f268c107470 2024-04-24T08:13:50.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.707+0000 7f2689600640 1 -- 172.21.15.28:0/335415418 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f2674044020 con 0x7f268c107470 2024-04-24T08:13:50.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.710+0000 7f26969c3740 1 Processor -- start 2024-04-24T08:13:50.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.712+0000 7f26969c3740 1 -- start start 2024-04-24T08:13:50.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.713+0000 7f2689600640 1 --2- 172.21.15.28:0/335415418 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f26700a3e20 0x7f26700a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.715+0000 7f2691c00640 1 --2- 172.21.15.28:0/335415418 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f26700a3e20 0x7f26700a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.715+0000 7f26969c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d6311950 0x5652d6311d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.715+0000 7f26969c3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d6312270 0x5652d631c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.716+0000 7f268be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d6312270 0x5652d631c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.716+0000 7f268be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d6312270 0x5652d631c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:46986/0 (socket says 172.21.15.28:46986) 2024-04-24T08:13:50.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.716+0000 7f268be00640 1 -- 172.21.15.28:0/380768713 learned_addr learned my addr 172.21.15.28:0/380768713 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:50.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.715+0000 7f2689600640 1 -- 172.21.15.28:0/335415418 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f26740f15d0 con 0x7f268c107470 2024-04-24T08:13:50.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.718+0000 7f26969c3740 1 --2- 172.21.15.28:0/380768713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d631c940 0x5652d631ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.718+0000 7f26969c3740 1 -- 172.21.15.28:0/380768713 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5652d62771d0 con 0x5652d6311950 2024-04-24T08:13:50.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.718+0000 7f26969c3740 1 -- 172.21.15.28:0/380768713 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5652d621d450 con 0x5652d631c940 2024-04-24T08:13:50.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.718+0000 7f26969c3740 1 -- 172.21.15.28:0/380768713 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5652d6273190 con 0x5652d6312270 2024-04-24T08:13:50.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.718+0000 7f268be00640 1 -- 172.21.15.28:0/380768713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d631c940 msgr2=0x5652d631ed30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:13:50.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.718+0000 7f268be00640 1 --2- 172.21.15.28:0/380768713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d631c940 0x5652d631ed30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.718+0000 7f268be00640 1 -- 172.21.15.28:0/380768713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d6311950 msgr2=0x5652d6311d30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:13:50.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.718+0000 7f268be00640 1 --2- 172.21.15.28:0/380768713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d6311950 0x5652d6311d30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.718+0000 7f268be00640 1 -- 172.21.15.28:0/380768713 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652d62da310 con 0x5652d6312270 2024-04-24T08:13:50.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.718+0000 7f268be00640 1 --2- 172.21.15.28:0/380768713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d6312270 0x5652d631c400 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f26740b9950 tx=0x7f2674031a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aa28c6918d68964a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:50.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.719+0000 7f2682000640 1 -- 172.21.15.28:0/380768713 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26740c18a0 con 0x5652d6312270 2024-04-24T08:13:50.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.719+0000 7f2682000640 1 -- 172.21.15.28:0/380768713 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26740c6160 con 0x5652d6312270 2024-04-24T08:13:50.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.719+0000 7f2682000640 1 -- 172.21.15.28:0/380768713 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26740c5cd0 con 0x5652d6312270 2024-04-24T08:13:50.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.720+0000 7f26969c3740 1 -- 172.21.15.28:0/380768713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d6312270 msgr2=0x5652d631c400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.720+0000 7f26969c3740 1 --2- 172.21.15.28:0/380768713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d6312270 0x5652d631c400 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f26740b9950 tx=0x7f2674031a00 comp rx=0 tx=0).stop 2024-04-24T08:13:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.721+0000 7f26969c3740 1 -- 172.21.15.28:0/380768713 shutdown_connections 2024-04-24T08:13:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.721+0000 7f2691c00640 1 --2- 172.21.15.28:0/335415418 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f26700a3e20 0x7f26700a62e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x5652d630ee00 tx=0x7f2684009260 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.721+0000 7f26969c3740 1 --2- 172.21.15.28:0/380768713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d631c940 0x5652d631ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.721+0000 7f26969c3740 1 --2- 172.21.15.28:0/380768713 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d6312270 0x5652d631c400 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.721+0000 7f26969c3740 1 --2- 172.21.15.28:0/380768713 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d6311950 0x5652d6311d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.721+0000 7f26969c3740 1 -- 172.21.15.28:0/380768713 >> 172.21.15.28:0/380768713 conn(0x5652d630dc70 msgr2=0x5652d630e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.721+0000 7f26969c3740 1 -- 172.21.15.28:0/380768713 shutdown_connections 2024-04-24T08:13:50.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.722+0000 7f26969c3740 1 -- 172.21.15.28:0/380768713 wait complete. 2024-04-24T08:13:50.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.722+0000 7f26969c3740 1 -- start start 2024-04-24T08:13:50.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.723+0000 7f26969c3740 1 Processor -- start 2024-04-24T08:13:50.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.724+0000 7f26969c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d6311950 0x5652d6319700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.725+0000 7f2691200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d6311950 0x5652d6319700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.725+0000 7f2691200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d6311950 0x5652d6319700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47958/0 (socket says 172.21.15.28:47958) 2024-04-24T08:13:50.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.725+0000 7f2691200640 1 -- 172.21.15.28:0/2770525633 learned_addr learned my addr 172.21.15.28:0/2770525633 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:50.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.725+0000 7f26969c3740 1 --2- 172.21.15.28:0/2770525633 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d6312270 0x5652d6319c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.726+0000 7f26969c3740 1 --2- 172.21.15.28:0/2770525633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d631c940 0x5652d631a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.726+0000 7f268be00640 1 --2- 172.21.15.28:0/2770525633 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d6312270 0x5652d6319c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.726+0000 7f2691c00640 1 --2- 172.21.15.28:0/2770525633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d631c940 0x5652d631a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.727+0000 7f26969c3740 1 -- 172.21.15.28:0/2770525633 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5652d631f4f0 con 0x5652d6311950 2024-04-24T08:13:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.727+0000 7f26969c3740 1 -- 172.21.15.28:0/2770525633 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5652d6230810 con 0x5652d631c940 2024-04-24T08:13:50.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.727+0000 7f26969c3740 1 -- 172.21.15.28:0/2770525633 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5652d631f670 con 0x5652d6312270 2024-04-24T08:13:50.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.727+0000 7f268be00640 1 -- 172.21.15.28:0/2770525633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d631c940 msgr2=0x5652d631a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:50.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.727+0000 7f268be00640 1 --2- 172.21.15.28:0/2770525633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d631c940 0x5652d631a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.727+0000 7f268be00640 1 -- 172.21.15.28:0/2770525633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d6311950 msgr2=0x5652d6319700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:50.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.728+0000 7f268be00640 1 --2- 172.21.15.28:0/2770525633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d6311950 0x5652d6319700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.728+0000 7f268be00640 1 -- 172.21.15.28:0/2770525633 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652d62413f0 con 0x5652d6312270 2024-04-24T08:13:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.728+0000 7f2691200640 1 --2- 172.21.15.28:0/2770525633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d6311950 0x5652d6319700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:13:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.728+0000 7f268be00640 1 --2- 172.21.15.28:0/2770525633 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d6312270 0x5652d6319c40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f26740b9950 tx=0x7f26740b9120 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8fb97bf8f6109ff2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.729+0000 7f267aa00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:13:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.729+0000 7f26969c3740 10 client.0 _ll_register_callbacks cb 0x5652d61fe140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.729+0000 7f26969c3740 1 -- 172.21.15.28:0/2770525633 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26740c6160 con 0x5652d6312270 2024-04-24T08:13:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.729+0000 7f267aa00640 1 -- 172.21.15.28:0/2770525633 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26740ef120 con 0x5652d6312270 2024-04-24T08:13:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.729+0000 7f267aa00640 1 -- 172.21.15.28:0/2770525633 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f26740c5cd0 con 0x5652d6312270 2024-04-24T08:13:50.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.729+0000 7f267aa00640 1 -- 172.21.15.28:0/2770525633 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f26740c5ed0 con 0x5652d6312270 2024-04-24T08:13:50.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.729+0000 7f26969c3740 10 client.26089 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:50.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.729+0000 7f26969c3740 1 -- 172.21.15.28:0/2770525633 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({fsmap=25}) -- 0x5652d5f893f0 con 0x5652d6312270 2024-04-24T08:13:50.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.730+0000 7f267aa00640 1 -- 172.21.15.28:0/2770525633 <== mon.1 v2:172.21.15.194:3300/0 4 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f26740dc2e0 con 0x5652d6312270 2024-04-24T08:13:50.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.731+0000 7f26969c3740 10 client.26089 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:50.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.731+0000 7f26969c3740 10 client.26089 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:13:50.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.731+0000 7f26969c3740 20 client.26089 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:50.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.731+0000 7f26969c3740 1 --2- 172.21.15.28:0/2770525633 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5652d632c360 0x5652d632e7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.731+0000 7f26969c3740 4 client.26089 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:13:50.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.731+0000 7f26969c3740 1 -- 172.21.15.28:0/2770525633 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5652d6284bf0 con 0x5652d632c360 2024-04-24T08:13:50.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.734+0000 7f2691200640 1 --2- 172.21.15.28:0/2770525633 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5652d632c360 0x5652d632e7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.741+0000 7f267aa00640 10 client.26089 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:50.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.741+0000 7f2691200640 1 --2- 172.21.15.28:0/2770525633 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5652d632c360 0x5652d632e7f0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bebc029757af1721 server_cookie=5ae94001232ea8c9 in_seq=0 out_seq=0 2024-04-24T08:13:50.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.751+0000 7f267aa00640 1 -- 172.21.15.28:0/2770525633 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5652d6284bf0 con 0x5652d632c360 2024-04-24T08:13:50.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.751+0000 7f267aa00640 10 client.26089 handle_command_reply: tid=0 2024-04-24T08:13:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.751+0000 7f26969c3740 1 client.26089 shutdown 2024-04-24T08:13:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.752+0000 7f26969c3740 1 -- 172.21.15.28:0/2770525633 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d6312270 msgr2=0x5652d6319c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.752+0000 7f26969c3740 1 --2- 172.21.15.28:0/2770525633 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d6312270 0x5652d6319c40 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f26740b9950 tx=0x7f26740b9120 comp rx=0 tx=0).stop 2024-04-24T08:13:50.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.754+0000 7f26969c3740 1 -- 172.21.15.28:0/2770525633 shutdown_connections 2024-04-24T08:13:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.754+0000 7f26969c3740 1 --2- 172.21.15.28:0/2770525633 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5652d632c360 0x5652d632e7f0 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.754+0000 7f26969c3740 1 --2- 172.21.15.28:0/2770525633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d631c940 0x5652d631a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.754+0000 7f26969c3740 1 --2- 172.21.15.28:0/2770525633 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d6312270 0x5652d6319c40 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.754+0000 7f26969c3740 1 --2- 172.21.15.28:0/2770525633 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d6311950 0x5652d6319700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.754+0000 7f26969c3740 1 -- 172.21.15.28:0/2770525633 >> 172.21.15.28:0/2770525633 conn(0x5652d630dc70 msgr2=0x5652d6313a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.754+0000 7f267aa00640 0 client.26089 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.754+0000 7f26969c3740 1 -- 172.21.15.28:0/2770525633 shutdown_connections 2024-04-24T08:13:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.754+0000 7f26969c3740 1 -- 172.21.15.28:0/2770525633 wait complete. 2024-04-24T08:13:50.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.754+0000 7f26969c3740 20 client.26089 trim_cache size 0 max 16384 2024-04-24T08:13:50.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.814+0000 7f26969c3740 1 Processor -- start 2024-04-24T08:13:50.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.814+0000 7f26969c3740 1 -- start start 2024-04-24T08:13:50.817 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.816+0000 7f26969c3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d6311950 0x5652d63140b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.816+0000 7f26969c3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d6312270 0x5652d6319fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.816+0000 7f26969c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d631c940 0x5652d630cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.816+0000 7f26969c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f268c1bf4e0 con 0x5652d631c940 2024-04-24T08:13:50.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.816+0000 7f26969c3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f268c1bcdb0 con 0x5652d6312270 2024-04-24T08:13:50.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.816+0000 7f26969c3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f268c11b560 con 0x5652d6311950 2024-04-24T08:13:50.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.817+0000 7f268be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d6312270 0x5652d6319fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.817+0000 7f268be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d6312270 0x5652d6319fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:36702/0 (socket says 172.21.15.28:36702) 2024-04-24T08:13:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.817+0000 7f268be00640 1 -- 172.21.15.28:0/3184456752 learned_addr learned my addr 172.21.15.28:0/3184456752 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.818+0000 7f268be00640 1 -- 172.21.15.28:0/3184456752 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d6311950 msgr2=0x5652d63140b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.818+0000 7f268be00640 1 --2- 172.21.15.28:0/3184456752 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d6311950 0x5652d63140b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.818+0000 7f2691c00640 1 --2- 172.21.15.28:0/3184456752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d631c940 0x5652d630cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.818+0000 7f268be00640 1 -- 172.21.15.28:0/3184456752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d631c940 msgr2=0x5652d630cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.818+0000 7f268be00640 1 --2- 172.21.15.28:0/3184456752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d631c940 0x5652d630cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.818+0000 7f268be00640 1 -- 172.21.15.28:0/3184456752 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5652d63120a0 con 0x5652d6312270 2024-04-24T08:13:50.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.819+0000 7f2691c00640 1 --2- 172.21.15.28:0/3184456752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d631c940 0x5652d630cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:50.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.819+0000 7f268be00640 1 --2- 172.21.15.28:0/3184456752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d6312270 0x5652d6319fa0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f26740bd480 tx=0x7f26740bf000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1e127b6826b9816 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:50.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.819+0000 7f2681600640 1 -- 172.21.15.28:0/3184456752 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26740c18a0 con 0x5652d6312270 2024-04-24T08:13:50.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.822+0000 7f2681600640 1 -- 172.21.15.28:0/3184456752 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26740c6160 con 0x5652d6312270 2024-04-24T08:13:50.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.822+0000 7f2681600640 1 -- 172.21.15.28:0/3184456752 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26740f1140 con 0x5652d6312270 2024-04-24T08:13:50.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.824+0000 7f26969c3740 1 -- 172.21.15.28:0/3184456752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d6312270 msgr2=0x5652d6319fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.824+0000 7f26969c3740 1 --2- 172.21.15.28:0/3184456752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d6312270 0x5652d6319fa0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f26740bd480 tx=0x7f26740bf000 comp rx=0 tx=0).stop 2024-04-24T08:13:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.825+0000 7f26969c3740 1 -- 172.21.15.28:0/3184456752 shutdown_connections 2024-04-24T08:13:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.825+0000 7f26969c3740 1 --2- 172.21.15.28:0/3184456752 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d631c940 0x5652d630cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.825+0000 7f26969c3740 1 --2- 172.21.15.28:0/3184456752 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d6312270 0x5652d6319fa0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.825+0000 7f26969c3740 1 --2- 172.21.15.28:0/3184456752 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d6311950 0x5652d63140b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.825+0000 7f26969c3740 1 -- 172.21.15.28:0/3184456752 >> 172.21.15.28:0/3184456752 conn(0x5652d630dc70 msgr2=0x5652d630c560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:50.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.825+0000 7f26969c3740 1 -- 172.21.15.28:0/3184456752 shutdown_connections 2024-04-24T08:13:50.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.825+0000 7f26969c3740 1 -- 172.21.15.28:0/3184456752 wait complete. 2024-04-24T08:13:50.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.826+0000 7f26969c3740 1 -- start start 2024-04-24T08:13:50.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.827+0000 7f26969c3740 1 Processor -- start 2024-04-24T08:13:50.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.829+0000 7f26969c3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d6311950 0x5652d6318c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.829+0000 7f26969c3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d6312270 0x5652d6319180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.829+0000 7f26969c3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d631c940 0x5652d6323890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.830+0000 7f268be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d6312270 0x5652d6319180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.830+0000 7f26969c3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f268c11bbd0 con 0x5652d6311950 2024-04-24T08:13:50.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.830+0000 7f26969c3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f268c1bc700 con 0x5652d6312270 2024-04-24T08:13:50.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.830+0000 7f26969c3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f268c11c2a0 con 0x5652d631c940 2024-04-24T08:13:50.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.830+0000 7f268be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d6312270 0x5652d6319180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:36704/0 (socket says 172.21.15.28:36704) 2024-04-24T08:13:50.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.830+0000 7f268be00640 1 -- 172.21.15.28:0/1716806836 learned_addr learned my addr 172.21.15.28:0/1716806836 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:50.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.830+0000 7f26969c3740 10 client.0 _ll_register_callbacks cb 0x7f26740bd030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:50.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.830+0000 7f2691c00640 1 --2- 172.21.15.28:0/1716806836 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d631c940 0x5652d6323890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.830+0000 7f268be00640 1 -- 172.21.15.28:0/1716806836 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d631c940 msgr2=0x5652d6323890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:50.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.830+0000 7f268be00640 1 --2- 172.21.15.28:0/1716806836 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d631c940 0x5652d6323890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.830+0000 7f268be00640 1 -- 172.21.15.28:0/1716806836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d6311950 msgr2=0x5652d6318c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:50.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.830+0000 7f268be00640 1 --2- 172.21.15.28:0/1716806836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d6311950 0x5652d6318c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.830+0000 7f268be00640 1 -- 172.21.15.28:0/1716806836 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5652d60537d0 con 0x5652d6312270 2024-04-24T08:13:50.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.830+0000 7f268be00640 1 --2- 172.21.15.28:0/1716806836 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d6312270 0x5652d6319180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f26740bb7a0 tx=0x7f2674031a00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0e8a689107bf770 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:50.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.830+0000 7f267aa00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:13:50.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.831+0000 7f2691c00640 1 --2- 172.21.15.28:0/1716806836 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d631c940 0x5652d6323890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:50.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.831+0000 7f267aa00640 1 -- 172.21.15.28:0/1716806836 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26740b9040 con 0x5652d6312270 2024-04-24T08:13:50.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.831+0000 7f267aa00640 1 -- 172.21.15.28:0/1716806836 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f26740f1350 con 0x5652d6312270 2024-04-24T08:13:50.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.831+0000 7f26969c3740 1 -- 172.21.15.28:0/1716806836 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26740c6160 con 0x5652d6312270 2024-04-24T08:13:50.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.832+0000 7f267aa00640 1 -- 172.21.15.28:0/1716806836 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f26740c18a0 con 0x5652d6312270 2024-04-24T08:13:50.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.832+0000 7f26969c3740 10 client.26045 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:50.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.832+0000 7f26969c3740 1 -- 172.21.15.28:0/1716806836 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5652d60537d0 con 0x5652d6312270 2024-04-24T08:13:50.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.833+0000 7f267aa00640 1 -- 172.21.15.28:0/1716806836 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26740c6160 con 0x5652d6312270 2024-04-24T08:13:50.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.834+0000 7f267aa00640 1 -- 172.21.15.28:0/1716806836 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f2674004460 con 0x5652d6312270 2024-04-24T08:13:50.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.834+0000 7f26969c3740 10 client.26045 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:50.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.834+0000 7f26969c3740 10 client.26045 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:13:50.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.834+0000 7f26969c3740 20 client.26045 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:50.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.834+0000 7f26969c3740 1 --2- 172.21.15.28:0/1716806836 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5652d6324f90 0x5652d6394950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:50.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.834+0000 7f26969c3740 4 client.26045 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:13:50.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.834+0000 7f26969c3740 1 -- 172.21.15.28:0/1716806836 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5652d627ea60 con 0x5652d6324f90 2024-04-24T08:13:50.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.837+0000 7f2691200640 1 --2- 172.21.15.28:0/1716806836 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5652d6324f90 0x5652d6394950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:50.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.838+0000 7f2691200640 1 --2- 172.21.15.28:0/1716806836 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5652d6324f90 0x5652d6394950 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc65e4191930d0e8 server_cookie=464fbbe72034d47e in_seq=0 out_seq=0 2024-04-24T08:13:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.838+0000 7f267aa00640 10 client.26045 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.838+0000 7f267aa00640 1 -- 172.21.15.28:0/1716806836 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5652d627ea60 con 0x5652d6324f90 2024-04-24T08:13:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.838+0000 7f267aa00640 10 client.26045 handle_command_reply: tid=0 2024-04-24T08:13:50.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.839+0000 7f26969c3740 1 client.26045 shutdown 2024-04-24T08:13:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.839+0000 7f26969c3740 1 -- 172.21.15.28:0/1716806836 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d6312270 msgr2=0x5652d6319180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.839+0000 7f26969c3740 1 --2- 172.21.15.28:0/1716806836 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d6312270 0x5652d6319180 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f26740bb7a0 tx=0x7f2674031a00 comp rx=0 tx=0).stop 2024-04-24T08:13:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.839+0000 7f26969c3740 1 -- 172.21.15.28:0/1716806836 shutdown_connections 2024-04-24T08:13:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.839+0000 7f26969c3740 1 --2- 172.21.15.28:0/1716806836 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5652d6324f90 0x5652d6394950 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.839+0000 7f267aa00640 0 client.26045 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.839+0000 7f26969c3740 1 --2- 172.21.15.28:0/1716806836 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5652d631c940 0x5652d6323890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.839+0000 7f26969c3740 1 --2- 172.21.15.28:0/1716806836 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5652d6312270 0x5652d6319180 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.839+0000 7f26969c3740 1 --2- 172.21.15.28:0/1716806836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5652d6311950 0x5652d6318c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.839+0000 7f26969c3740 1 -- 172.21.15.28:0/1716806836 >> 172.21.15.28:0/1716806836 conn(0x5652d630dc70 msgr2=0x5652d631af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.839+0000 7f26969c3740 1 -- 172.21.15.28:0/1716806836 shutdown_connections 2024-04-24T08:13:50.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.839+0000 7f26969c3740 1 -- 172.21.15.28:0/1716806836 wait complete. 2024-04-24T08:13:50.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.840+0000 7f26969c3740 20 client.26045 trim_cache size 0 max 16384 2024-04-24T08:13:50.841 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:13:50.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.841+0000 7f2693c00640 1 -- 172.21.15.28:0/335415418 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f26700a3e20 msgr2=0x7f26700a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.841+0000 7f2693c00640 1 --2- 172.21.15.28:0/335415418 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f26700a3e20 0x7f26700a62e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x5652d630ee00 tx=0x7f2684009260 comp rx=0 tx=0).stop 2024-04-24T08:13:50.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.841+0000 7f2693c00640 1 -- 172.21.15.28:0/335415418 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f268c107470 msgr2=0x7f268c10cd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:50.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.841+0000 7f2693c00640 1 --2- 172.21.15.28:0/335415418 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f268c107470 0x7f268c10cd70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f2674034040 tx=0x7f26740305d0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.843+0000 7f2693c00640 1 -- 172.21.15.28:0/335415418 shutdown_connections 2024-04-24T08:13:50.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.843+0000 7f2693c00640 1 --2- 172.21.15.28:0/335415418 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f26700a3e20 0x7f26700a62e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.843+0000 7f2693c00640 1 --2- 172.21.15.28:0/335415418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268c1a8d40 0x7f268c1ab130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.843+0000 7f2693c00640 1 --2- 172.21.15.28:0/335415418 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f268c107470 0x7f268c10cd70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.843+0000 7f2693c00640 1 --2- 172.21.15.28:0/335415418 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f268c1003b0 0x7f268c10c830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:50.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.843+0000 7f2693c00640 1 -- 172.21.15.28:0/335415418 >> 172.21.15.28:0/335415418 conn(0x7f268c077060 msgr2=0x7f268c105520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:50.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.843+0000 7f2693c00640 1 -- 172.21.15.28:0/335415418 shutdown_connections 2024-04-24T08:13:50.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:50.843+0000 7f2693c00640 1 -- 172.21.15.28:0/335415418 wait complete. 2024-04-24T08:13:51.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:51 smithi117 ceph-mon[32438]: pgmap v108: 97 pgs: 97 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 101 MiB/s wr, 14.94k op/s 2024-04-24T08:13:51.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:51 smithi194 ceph-mon[32836]: pgmap v108: 97 pgs: 97 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 101 MiB/s wr, 14.94k op/s 2024-04-24T08:13:51.792 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:51 smithi028 ceph-mon[27786]: pgmap v108: 97 pgs: 97 active+clean; 13 GiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 436 KiB/s rd, 101 MiB/s wr, 14.94k op/s 2024-04-24T08:13:51.886 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:13:51.886 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:13:51.886 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:13:52.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.290+0000 7fe447c00640 1 -- 172.21.15.28:0/111155241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe44011b7e0 msgr2=0x7fe44011dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.290+0000 7fe447c00640 1 --2- 172.21.15.28:0/111155241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe44011b7e0 0x7fe44011dbd0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fe438009be0 tx=0x7fe43802a6d0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.291+0000 7fe447c00640 1 -- 172.21.15.28:0/111155241 shutdown_connections 2024-04-24T08:13:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.291+0000 7fe447c00640 1 --2- 172.21.15.28:0/111155241 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe44011b7e0 0x7fe44011dbd0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.291+0000 7fe447c00640 1 --2- 172.21.15.28:0/111155241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe44010e880 0x7fe44010ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.291+0000 7fe447c00640 1 --2- 172.21.15.28:0/111155241 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe440110850 0x7fe44010e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.291+0000 7fe447c00640 1 -- 172.21.15.28:0/111155241 >> 172.21.15.28:0/111155241 conn(0x7fe44006d110 msgr2=0x7fe44006d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.291+0000 7fe447c00640 1 -- 172.21.15.28:0/111155241 shutdown_connections 2024-04-24T08:13:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.291+0000 7fe447c00640 1 -- 172.21.15.28:0/111155241 wait complete. 2024-04-24T08:13:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.293+0000 7fe447c00640 1 Processor -- start 2024-04-24T08:13:52.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.293+0000 7fe447c00640 1 -- start start 2024-04-24T08:13:52.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.293+0000 7fe447c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe44010e880 0x7fe44011add0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.293+0000 7fe447c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe440110850 0x7fe440116e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.293+0000 7fe447c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe44011b7e0 0x7fe4401173a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.294+0000 7fe447c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe440113950 con 0x7fe440110850 2024-04-24T08:13:52.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.294+0000 7fe447c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe4401137d0 con 0x7fe44010e880 2024-04-24T08:13:52.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.294+0000 7fe447c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe440113ad0 con 0x7fe44011b7e0 2024-04-24T08:13:52.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.294+0000 7fe445200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe44010e880 0x7fe44011add0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.294+0000 7fe445c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe44011b7e0 0x7fe4401173a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.294+0000 7fe445c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe44011b7e0 0x7fe4401173a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:47034/0 (socket says 172.21.15.28:47034) 2024-04-24T08:13:52.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.294+0000 7fe445c00640 1 -- 172.21.15.28:0/238364374 learned_addr learned my addr 172.21.15.28:0/238364374 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:52.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.294+0000 7fe43fe00640 1 --2- 172.21.15.28:0/238364374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe440110850 0x7fe440116e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.295+0000 7fe445c00640 1 -- 172.21.15.28:0/238364374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe44010e880 msgr2=0x7fe44011add0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.295+0000 7fe445c00640 1 --2- 172.21.15.28:0/238364374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe44010e880 0x7fe44011add0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.295+0000 7fe445c00640 1 -- 172.21.15.28:0/238364374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe440110850 msgr2=0x7fe440116e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.295+0000 7fe445c00640 1 --2- 172.21.15.28:0/238364374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe440110850 0x7fe440116e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.295+0000 7fe445c00640 1 -- 172.21.15.28:0/238364374 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe440117960 con 0x7fe44011b7e0 2024-04-24T08:13:52.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.295+0000 7fe445c00640 1 --2- 172.21.15.28:0/238364374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe44011b7e0 0x7fe4401173a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe438009b30 tx=0x7fe438030710 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:52.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.295+0000 7fe43d600640 1 -- 172.21.15.28:0/238364374 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe438041020 con 0x7fe44011b7e0 2024-04-24T08:13:52.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.296+0000 7fe43d600640 1 -- 172.21.15.28:0/238364374 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe438003e50 con 0x7fe44011b7e0 2024-04-24T08:13:52.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.296+0000 7fe447c00640 1 -- 172.21.15.28:0/238364374 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe440117bf0 con 0x7fe44011b7e0 2024-04-24T08:13:52.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.296+0000 7fe43d600640 1 -- 172.21.15.28:0/238364374 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe438036040 con 0x7fe44011b7e0 2024-04-24T08:13:52.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.296+0000 7fe447c00640 1 -- 172.21.15.28:0/238364374 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4401ab860 con 0x7fe44011b7e0 2024-04-24T08:13:52.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.300+0000 7fe44aa3c740 1 Processor -- start 2024-04-24T08:13:52.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.301+0000 7fe44aa3c740 1 -- start start 2024-04-24T08:13:52.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.301+0000 7fe44aa3c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e73d090 0x55916e73d470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.301+0000 7fe43d600640 1 -- 172.21.15.28:0/238364374 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe438044020 con 0x7fe44011b7e0 2024-04-24T08:13:52.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.303+0000 7fe44aa3c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d9b0 0x55916e747b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.303+0000 7fe44aa3c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e748080 0x55916e74a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.303+0000 7fe44aa3c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55916e6a2e70 con 0x55916e748080 2024-04-24T08:13:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.303+0000 7fe44aa3c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55916e649380 con 0x55916e73d090 2024-04-24T08:13:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.303+0000 7fe44aa3c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55916e69ee30 con 0x55916e73d9b0 2024-04-24T08:13:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.303+0000 7fe445200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d9b0 0x55916e747b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.303+0000 7fe445200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d9b0 0x55916e747b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:47042/0 (socket says 172.21.15.28:47042) 2024-04-24T08:13:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.303+0000 7fe445200640 1 -- 172.21.15.28:0/2738716561 learned_addr learned my addr 172.21.15.28:0/2738716561 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.303+0000 7fe43d600640 1 --2- 172.21.15.28:0/238364374 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4280a3e20 0x7fe4280a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.304+0000 7fe43d600640 1 -- 172.21.15.28:0/238364374 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fe4380f1fa0 con 0x7fe44011b7e0 2024-04-24T08:13:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.304+0000 7fe445200640 1 -- 172.21.15.28:0/2738716561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e73d090 msgr2=0x55916e73d470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:13:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.304+0000 7fe445200640 1 --2- 172.21.15.28:0/2738716561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e73d090 0x55916e73d470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.304+0000 7fe445200640 1 -- 172.21.15.28:0/2738716561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e748080 msgr2=0x55916e74a470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:13:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.304+0000 7fe445200640 1 --2- 172.21.15.28:0/2738716561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e748080 0x55916e74a470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.304+0000 7fe445200640 1 -- 172.21.15.28:0/2738716561 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55916e47d7e0 con 0x55916e73d9b0 2024-04-24T08:13:52.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.304+0000 7fe445c00640 1 --2- 172.21.15.28:0/238364374 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4280a3e20 0x7fe4280a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.304+0000 7fe445200640 1 --2- 172.21.15.28:0/2738716561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d9b0 0x55916e747b40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe43400bcb0 tx=0x7fe43400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cdfeab62dd5fdff server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:52.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.305+0000 7fe432000640 1 -- 172.21.15.28:0/2738716561 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe434015070 con 0x55916e73d9b0 2024-04-24T08:13:52.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.305+0000 7fe432000640 1 -- 172.21.15.28:0/2738716561 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe43401a940 con 0x55916e73d9b0 2024-04-24T08:13:52.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.305+0000 7fe432000640 1 -- 172.21.15.28:0/2738716561 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe434011b20 con 0x55916e73d9b0 2024-04-24T08:13:52.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.306+0000 7fe44aa3c740 1 -- 172.21.15.28:0/2738716561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d9b0 msgr2=0x55916e747b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.306+0000 7fe445c00640 1 --2- 172.21.15.28:0/238364374 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4280a3e20 0x7fe4280a62e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe438009370 tx=0x7fe43802a6d0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:52.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.306+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/2738716561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d9b0 0x55916e747b40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fe43400bcb0 tx=0x7fe43400f730 comp rx=0 tx=0).stop 2024-04-24T08:13:52.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.307+0000 7fe44aa3c740 1 -- 172.21.15.28:0/2738716561 shutdown_connections 2024-04-24T08:13:52.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.307+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/2738716561 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e748080 0x55916e74a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.307+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/2738716561 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d9b0 0x55916e747b40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.307+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/2738716561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e73d090 0x55916e73d470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.307+0000 7fe44aa3c740 1 -- 172.21.15.28:0/2738716561 >> 172.21.15.28:0/2738716561 conn(0x55916e7393b0 msgr2=0x55916e739790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:52.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.308+0000 7fe44aa3c740 1 -- 172.21.15.28:0/2738716561 shutdown_connections 2024-04-24T08:13:52.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.308+0000 7fe44aa3c740 1 -- 172.21.15.28:0/2738716561 wait complete. 2024-04-24T08:13:52.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.308+0000 7fe44aa3c740 1 -- start start 2024-04-24T08:13:52.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.309+0000 7fe44aa3c740 1 Processor -- start 2024-04-24T08:13:52.310 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.310+0000 7fe44aa3c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d090 0x55916e744f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.310+0000 7fe44aa3c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e73d9b0 0x55916e7454c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.310+0000 7fe44aa3c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e748080 0x55916e745a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.310+0000 7fe44aa3c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55916e74ae70 con 0x55916e73d9b0 2024-04-24T08:13:52.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.310+0000 7fe44aa3c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55916e65c660 con 0x55916e748080 2024-04-24T08:13:52.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.310+0000 7fe44aa3c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55916e74aff0 con 0x55916e73d090 2024-04-24T08:13:52.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.310+0000 7fe44aa3c740 10 client.0 _ll_register_callbacks cb 0x55916e64aaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:52.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.311+0000 7fe43fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d090 0x55916e744f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.311+0000 7fe43fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d090 0x55916e744f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:47054/0 (socket says 172.21.15.28:47054) 2024-04-24T08:13:52.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.311+0000 7fe43fe00640 1 -- 172.21.15.28:0/4147336943 learned_addr learned my addr 172.21.15.28:0/4147336943 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:52.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.311+0000 7fe43fe00640 1 --2- 172.21.15.28:0/4147336943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e748080 0x55916e745a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.311+0000 7fe445200640 1 --2- 172.21.15.28:0/4147336943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e73d9b0 0x55916e7454c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.311+0000 7fe43fe00640 1 -- 172.21.15.28:0/4147336943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e748080 msgr2=0x55916e745a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:52.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.311+0000 7fe43fe00640 1 --2- 172.21.15.28:0/4147336943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e748080 0x55916e745a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.311+0000 7fe43fe00640 1 -- 172.21.15.28:0/4147336943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e73d9b0 msgr2=0x55916e7454c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:52.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.311+0000 7fe43fe00640 1 --2- 172.21.15.28:0/4147336943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e73d9b0 0x55916e7454c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.311+0000 7fe43fe00640 1 -- 172.21.15.28:0/4147336943 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55916e4ddf70 con 0x55916e73d090 2024-04-24T08:13:52.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.311+0000 7fe43fe00640 1 --2- 172.21.15.28:0/4147336943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d090 0x55916e744f80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe42c01a6a0 tx=0x7fe42c01a630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed2fa1426f4d6a27 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:52.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.312+0000 7fe426a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:13:52.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.312+0000 7fe426a00640 1 -- 172.21.15.28:0/4147336943 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c022280 con 0x55916e73d090 2024-04-24T08:13:52.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.312+0000 7fe44aa3c740 1 -- 172.21.15.28:0/4147336943 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe43401a940 con 0x55916e73d090 2024-04-24T08:13:52.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.312+0000 7fe426a00640 1 -- 172.21.15.28:0/4147336943 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fe42c02b070 con 0x55916e73d090 2024-04-24T08:13:52.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.313+0000 7fe426a00640 1 -- 172.21.15.28:0/4147336943 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe42c035050 con 0x55916e73d090 2024-04-24T08:13:52.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.314+0000 7fe44aa3c740 10 client.26101 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:52.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.314+0000 7fe44aa3c740 1 -- 172.21.15.28:0/4147336943 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55916e375170 con 0x55916e73d090 2024-04-24T08:13:52.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.314+0000 7fe426a00640 1 -- 172.21.15.28:0/4147336943 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe43401a940 con 0x55916e73d090 2024-04-24T08:13:52.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.314+0000 7fe426a00640 1 -- 172.21.15.28:0/4147336943 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fe42c0035f0 con 0x55916e73d090 2024-04-24T08:13:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.317+0000 7fe44aa3c740 10 client.26101 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.317+0000 7fe44aa3c740 10 client.26101 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:13:52.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.317+0000 7fe44aa3c740 20 client.26101 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:52.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.317+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/4147336943 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55916e758430 0x55916e75a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.317+0000 7fe44aa3c740 4 client.26101 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:13:52.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.317+0000 7fe44aa3c740 1 -- 172.21.15.28:0/4147336943 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55916e6b0890 con 0x55916e758430 2024-04-24T08:13:52.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.319+0000 7fe445200640 1 --2- 172.21.15.28:0/4147336943 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55916e758430 0x55916e75a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.320+0000 7fe445200640 1 --2- 172.21.15.28:0/4147336943 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55916e758430 0x55916e75a820 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=860c775a20afe30f server_cookie=d486865a45a69026 in_seq=0 out_seq=0 2024-04-24T08:13:52.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.320+0000 7fe426a00640 10 client.26101 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:52.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.321+0000 7fe426a00640 1 -- 172.21.15.28:0/4147336943 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55916e6b0890 con 0x55916e758430 2024-04-24T08:13:52.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.321+0000 7fe426a00640 10 client.26101 handle_command_reply: tid=0 2024-04-24T08:13:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.321+0000 7fe44aa3c740 1 client.26101 shutdown 2024-04-24T08:13:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.322+0000 7fe44aa3c740 1 -- 172.21.15.28:0/4147336943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d090 msgr2=0x55916e744f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.322+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/4147336943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d090 0x55916e744f80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fe42c01a6a0 tx=0x7fe42c01a630 comp rx=0 tx=0).stop 2024-04-24T08:13:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.322+0000 7fe44aa3c740 1 -- 172.21.15.28:0/4147336943 shutdown_connections 2024-04-24T08:13:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.322+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/4147336943 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55916e758430 0x55916e75a820 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.322+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/4147336943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e73d9b0 0x55916e7454c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.322+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/4147336943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e748080 0x55916e745a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.322+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/4147336943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d090 0x55916e744f80 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.322+0000 7fe44aa3c740 1 -- 172.21.15.28:0/4147336943 >> 172.21.15.28:0/4147336943 conn(0x55916e7393b0 msgr2=0x55916e73f1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.322+0000 7fe426a00640 0 client.26101 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.322+0000 7fe44aa3c740 1 -- 172.21.15.28:0/4147336943 shutdown_connections 2024-04-24T08:13:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.322+0000 7fe44aa3c740 1 -- 172.21.15.28:0/4147336943 wait complete. 2024-04-24T08:13:52.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.322+0000 7fe44aa3c740 20 client.26101 trim_cache size 0 max 16384 2024-04-24T08:13:52.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.371+0000 7fe44aa3c740 1 Processor -- start 2024-04-24T08:13:52.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.371+0000 7fe44aa3c740 1 -- start start 2024-04-24T08:13:52.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.371+0000 7fe44aa3c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d090 0x55916e73f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.371+0000 7fe44aa3c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e73d9b0 0x55916e744860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.371+0000 7fe44aa3c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e748080 0x55916e738390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.371+0000 7fe44aa3c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4401b6d30 con 0x55916e748080 2024-04-24T08:13:52.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.371+0000 7fe44aa3c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe4401b4600 con 0x55916e73d9b0 2024-04-24T08:13:52.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.371+0000 7fe44aa3c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe4401b23f0 con 0x55916e73d090 2024-04-24T08:13:52.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.372+0000 7fe43fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d090 0x55916e73f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.372+0000 7fe43fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d090 0x55916e73f900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:47066/0 (socket says 172.21.15.28:47066) 2024-04-24T08:13:52.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.372+0000 7fe43fe00640 1 -- 172.21.15.28:0/2098508140 learned_addr learned my addr 172.21.15.28:0/2098508140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:52.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.372+0000 7fe445200640 1 --2- 172.21.15.28:0/2098508140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e73d9b0 0x55916e744860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.372+0000 7fe43fe00640 1 -- 172.21.15.28:0/2098508140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e73d9b0 msgr2=0x55916e744860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.372+0000 7fe43fe00640 1 --2- 172.21.15.28:0/2098508140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e73d9b0 0x55916e744860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.372+0000 7fe43fe00640 1 -- 172.21.15.28:0/2098508140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e748080 msgr2=0x55916e738390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.372+0000 7fe43fe00640 1 --2- 172.21.15.28:0/2098508140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e748080 0x55916e738390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.372+0000 7fe43fe00640 1 -- 172.21.15.28:0/2098508140 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55916e7391e0 con 0x55916e73d090 2024-04-24T08:13:52.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.373+0000 7fe43fe00640 1 --2- 172.21.15.28:0/2098508140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d090 0x55916e73f900 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe42c038f30 tx=0x7fe42c01a270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d859f78193ac3e58 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:52.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.374+0000 7fe431600640 1 -- 172.21.15.28:0/2098508140 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c035050 con 0x55916e73d090 2024-04-24T08:13:52.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.374+0000 7fe431600640 1 -- 172.21.15.28:0/2098508140 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe42c02fbc0 con 0x55916e73d090 2024-04-24T08:13:52.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.374+0000 7fe431600640 1 -- 172.21.15.28:0/2098508140 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe42c02fec0 con 0x55916e73d090 2024-04-24T08:13:52.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.375+0000 7fe44aa3c740 1 -- 172.21.15.28:0/2098508140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d090 msgr2=0x55916e73f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.375+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/2098508140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d090 0x55916e73f900 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fe42c038f30 tx=0x7fe42c01a270 comp rx=0 tx=0).stop 2024-04-24T08:13:52.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.375+0000 7fe44aa3c740 1 -- 172.21.15.28:0/2098508140 shutdown_connections 2024-04-24T08:13:52.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.375+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/2098508140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e748080 0x55916e738390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.375+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/2098508140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e73d9b0 0x55916e744860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.375+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/2098508140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d090 0x55916e73f900 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.375+0000 7fe44aa3c740 1 -- 172.21.15.28:0/2098508140 >> 172.21.15.28:0/2098508140 conn(0x55916e7393b0 msgr2=0x55916e737c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:52.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.376+0000 7fe44aa3c740 1 -- 172.21.15.28:0/2098508140 shutdown_connections 2024-04-24T08:13:52.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.376+0000 7fe44aa3c740 1 -- 172.21.15.28:0/2098508140 wait complete. 2024-04-24T08:13:52.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.376+0000 7fe44aa3c740 1 -- start start 2024-04-24T08:13:52.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.376+0000 7fe44aa3c740 1 Processor -- start 2024-04-24T08:13:52.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.377+0000 7fe44aa3c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e73d090 0x55916e7501d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.377+0000 7fe44aa3c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d9b0 0x55916e750710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.377+0000 7fe44aa3c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e748080 0x55916e74b690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.377+0000 7fe44aa3c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4401b2ac0 con 0x55916e73d090 2024-04-24T08:13:52.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.377+0000 7fe44aa3c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe4401b3f30 con 0x55916e748080 2024-04-24T08:13:52.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.377+0000 7fe44aa3c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe4401b3190 con 0x55916e73d9b0 2024-04-24T08:13:52.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.377+0000 7fe44aa3c740 10 client.0 _ll_register_callbacks cb 0x7fe42c01aa60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:52.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.378+0000 7fe445200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d9b0 0x55916e750710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.378+0000 7fe445200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d9b0 0x55916e750710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:47074/0 (socket says 172.21.15.28:47074) 2024-04-24T08:13:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.378+0000 7fe445200640 1 -- 172.21.15.28:0/1514789832 learned_addr learned my addr 172.21.15.28:0/1514789832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.378+0000 7fe445200640 1 -- 172.21.15.28:0/1514789832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e748080 msgr2=0x55916e74b690 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:13:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.378+0000 7fe445200640 1 --2- 172.21.15.28:0/1514789832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e748080 0x55916e74b690 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.378+0000 7fe445200640 1 -- 172.21.15.28:0/1514789832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e73d090 msgr2=0x55916e7501d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.379+0000 7fe43fe00640 1 --2- 172.21.15.28:0/1514789832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e73d090 0x55916e7501d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.379+0000 7fe445200640 1 --2- 172.21.15.28:0/1514789832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e73d090 0x55916e7501d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.379+0000 7fe445200640 1 -- 172.21.15.28:0/1514789832 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55916e74bec0 con 0x55916e73d9b0 2024-04-24T08:13:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.379+0000 7fe43fe00640 1 --2- 172.21.15.28:0/1514789832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e73d090 0x55916e7501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.380+0000 7fe445200640 1 --2- 172.21.15.28:0/1514789832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d9b0 0x55916e750710 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe43400fc30 tx=0x7fe43400fd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d937d321db30dd21 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.380+0000 7fe426a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:13:52.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.381+0000 7fe426a00640 1 -- 172.21.15.28:0/1514789832 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe434015040 con 0x55916e73d9b0 2024-04-24T08:13:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.381+0000 7fe426a00640 1 -- 172.21.15.28:0/1514789832 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fe434005240 con 0x55916e73d9b0 2024-04-24T08:13:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.381+0000 7fe44aa3c740 1 -- 172.21.15.28:0/1514789832 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe42c02fbc0 con 0x55916e73d9b0 2024-04-24T08:13:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.381+0000 7fe426a00640 1 -- 172.21.15.28:0/1514789832 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe434035050 con 0x55916e73d9b0 2024-04-24T08:13:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.381+0000 7fe44aa3c740 10 client.26107 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:52.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.381+0000 7fe44aa3c740 1 -- 172.21.15.28:0/1514789832 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55916e7be3d0 con 0x55916e73d9b0 2024-04-24T08:13:52.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.382+0000 7fe426a00640 1 -- 172.21.15.28:0/1514789832 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe42c02fbc0 con 0x55916e73d9b0 2024-04-24T08:13:52.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.383+0000 7fe426a00640 1 -- 172.21.15.28:0/1514789832 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7fe434004930 con 0x55916e73d9b0 2024-04-24T08:13:52.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.383+0000 7fe44aa3c740 10 client.26107 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.383+0000 7fe44aa3c740 10 client.26107 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:13:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.383+0000 7fe44aa3c740 20 client.26107 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.384+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/1514789832 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55916e7be870 0x55916e7c0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.384+0000 7fe44aa3c740 4 client.26107 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:13:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.384+0000 7fe44aa3c740 1 -- 172.21.15.28:0/1514789832 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55916e6aa700 con 0x55916e7be870 2024-04-24T08:13:52.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.384+0000 7fe43fe00640 1 --2- 172.21.15.28:0/1514789832 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55916e7be870 0x55916e7c0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.385+0000 7fe43fe00640 1 --2- 172.21.15.28:0/1514789832 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55916e7be870 0x55916e7c0cb0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7ba32be6bca3acd7 server_cookie=dde0048b451826e7 in_seq=0 out_seq=0 2024-04-24T08:13:52.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.385+0000 7fe426a00640 10 client.26107 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:52.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.388+0000 7fe426a00640 1 -- 172.21.15.28:0/1514789832 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55916e6aa700 con 0x55916e7be870 2024-04-24T08:13:52.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.388+0000 7fe426a00640 10 client.26107 handle_command_reply: tid=0 2024-04-24T08:13:52.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.388+0000 7fe44aa3c740 1 client.26107 shutdown 2024-04-24T08:13:52.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.389+0000 7fe44aa3c740 1 -- 172.21.15.28:0/1514789832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d9b0 msgr2=0x55916e750710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.389+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/1514789832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d9b0 0x55916e750710 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fe43400fc30 tx=0x7fe43400fd60 comp rx=0 tx=0).stop 2024-04-24T08:13:52.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.389+0000 7fe44aa3c740 1 -- 172.21.15.28:0/1514789832 shutdown_connections 2024-04-24T08:13:52.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.389+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/1514789832 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55916e7be870 0x55916e7c0cb0 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.389+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/1514789832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55916e748080 0x55916e74b690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.389+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/1514789832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55916e73d9b0 0x55916e750710 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.389+0000 7fe44aa3c740 1 --2- 172.21.15.28:0/1514789832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55916e73d090 0x55916e7501d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.389+0000 7fe44aa3c740 1 -- 172.21.15.28:0/1514789832 >> 172.21.15.28:0/1514789832 conn(0x55916e7393b0 msgr2=0x55916e738510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:52.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.390+0000 7fe426a00640 0 client.26107 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:52.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.390+0000 7fe44aa3c740 1 -- 172.21.15.28:0/1514789832 shutdown_connections 2024-04-24T08:13:52.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.390+0000 7fe44aa3c740 1 -- 172.21.15.28:0/1514789832 wait complete. 2024-04-24T08:13:52.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.390+0000 7fe44aa3c740 20 client.26107 trim_cache size 0 max 16384 2024-04-24T08:13:52.391 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:13:52.392 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-24T08:13:52.392 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "ae51db93-2230-4cd0-a1aa-d9c0bee08435", 2024-04-24T08:13:52.392 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-24T08:13:52.392 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:13:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.392+0000 7fe447c00640 1 -- 172.21.15.28:0/238364374 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4280a3e20 msgr2=0x7fe4280a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.392+0000 7fe447c00640 1 --2- 172.21.15.28:0/238364374 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4280a3e20 0x7fe4280a62e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe438009370 tx=0x7fe43802a6d0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.392+0000 7fe447c00640 1 -- 172.21.15.28:0/238364374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe44011b7e0 msgr2=0x7fe4401173a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.392+0000 7fe447c00640 1 --2- 172.21.15.28:0/238364374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe44011b7e0 0x7fe4401173a0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fe438009b30 tx=0x7fe438030710 comp rx=0 tx=0).stop 2024-04-24T08:13:52.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.393+0000 7fe447c00640 1 -- 172.21.15.28:0/238364374 shutdown_connections 2024-04-24T08:13:52.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.393+0000 7fe447c00640 1 --2- 172.21.15.28:0/238364374 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4280a3e20 0x7fe4280a62e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.393+0000 7fe447c00640 1 --2- 172.21.15.28:0/238364374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe440110850 0x7fe440116e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.393+0000 7fe447c00640 1 --2- 172.21.15.28:0/238364374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe44011b7e0 0x7fe4401173a0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.393+0000 7fe447c00640 1 --2- 172.21.15.28:0/238364374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe44010e880 0x7fe44011add0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.393+0000 7fe447c00640 1 -- 172.21.15.28:0/238364374 >> 172.21.15.28:0/238364374 conn(0x7fe44006d110 msgr2=0x7fe44011cd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:52.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.395+0000 7fe447c00640 1 -- 172.21.15.28:0/238364374 shutdown_connections 2024-04-24T08:13:52.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.395+0000 7fe447c00640 1 -- 172.21.15.28:0/238364374 wait complete. 2024-04-24T08:13:52.433 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:13:52.433 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:13:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.748+0000 7f291b200640 1 -- 172.21.15.28:0/1818280545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f291c10de20 msgr2=0x7f291c1159f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.748+0000 7f291b200640 1 --2- 172.21.15.28:0/1818280545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f291c10de20 0x7f291c1159f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f2914009ca0 tx=0x7f291402b9d0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.748+0000 7f291b200640 1 -- 172.21.15.28:0/1818280545 shutdown_connections 2024-04-24T08:13:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.748+0000 7f291b200640 1 --2- 172.21.15.28:0/1818280545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f291c10de20 0x7f291c1159f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.748+0000 7f291b200640 1 --2- 172.21.15.28:0/1818280545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f291c109c20 0x7f291c10d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.748+0000 7f291b200640 1 --2- 172.21.15.28:0/1818280545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f291c109270 0x7f291c109650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.748+0000 7f291b200640 1 -- 172.21.15.28:0/1818280545 >> 172.21.15.28:0/1818280545 conn(0x7f291c06d3e0 msgr2=0x7f291c06d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.749+0000 7f291b200640 1 -- 172.21.15.28:0/1818280545 shutdown_connections 2024-04-24T08:13:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.749+0000 7f291b200640 1 -- 172.21.15.28:0/1818280545 wait complete. 2024-04-24T08:13:52.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.750+0000 7f291b200640 1 Processor -- start 2024-04-24T08:13:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.751+0000 7f291b200640 1 -- start start 2024-04-24T08:13:52.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.751+0000 7f291b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f291c109270 0x7f291c1ad400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.751+0000 7f291b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f291c10de20 0x7f291c1ad940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.751+0000 7f291b200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f291c1a75b0 0x7f291c1a7a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.751+0000 7f291b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f291c118a80 con 0x7f291c10de20 2024-04-24T08:13:52.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.751+0000 7f291b200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f291c118900 con 0x7f291c1a75b0 2024-04-24T08:13:52.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.751+0000 7f291b200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f291c118c00 con 0x7f291c109270 2024-04-24T08:13:52.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.752+0000 7f2919e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f291c109270 0x7f291c1ad400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.752+0000 7f2919e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f291c109270 0x7f291c1ad400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:47096/0 (socket says 172.21.15.28:47096) 2024-04-24T08:13:52.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.752+0000 7f2919e00640 1 -- 172.21.15.28:0/1147776267 learned_addr learned my addr 172.21.15.28:0/1147776267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:52.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.752+0000 7f291a800640 1 --2- 172.21.15.28:0/1147776267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f291c1a75b0 0x7f291c1a7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.753+0000 7f2919e00640 1 -- 172.21.15.28:0/1147776267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f291c1a75b0 msgr2=0x7f291c1a7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.753+0000 7f2919e00640 1 --2- 172.21.15.28:0/1147776267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f291c1a75b0 0x7f291c1a7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.753+0000 7f2919e00640 1 -- 172.21.15.28:0/1147776267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f291c10de20 msgr2=0x7f291c1ad940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.753+0000 7f2919e00640 1 --2- 172.21.15.28:0/1147776267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f291c10de20 0x7f291c1ad940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.753+0000 7f2919e00640 1 -- 172.21.15.28:0/1147776267 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f291c1a80d0 con 0x7f291c109270 2024-04-24T08:13:52.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.754+0000 7f2919e00640 1 --2- 172.21.15.28:0/1147776267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f291c109270 0x7f291c1ad400 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f290c00d950 tx=0x7f290c00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.754+0000 7f2912a00640 1 -- 172.21.15.28:0/1147776267 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290c017070 con 0x7f291c109270 2024-04-24T08:13:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.754+0000 7f2912a00640 1 -- 172.21.15.28:0/1147776267 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f290c009c30 con 0x7f291c109270 2024-04-24T08:13:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.754+0000 7f2912a00640 1 -- 172.21.15.28:0/1147776267 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290c017070 con 0x7f291c109270 2024-04-24T08:13:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.754+0000 7f291b200640 1 -- 172.21.15.28:0/1147776267 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f291c1a83c0 con 0x7f291c109270 2024-04-24T08:13:52.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.755+0000 7f291b200640 1 -- 172.21.15.28:0/1147776267 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f291c1b45c0 con 0x7f291c109270 2024-04-24T08:13:52.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.761+0000 7f2924176740 1 Processor -- start 2024-04-24T08:13:52.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.762+0000 7f2912a00640 1 -- 172.21.15.28:0/1147776267 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f290c010460 con 0x7f291c109270 2024-04-24T08:13:52.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.764+0000 7f2924176740 1 -- start start 2024-04-24T08:13:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.766+0000 7f2924176740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a7a30 0x5639e89a7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.766+0000 7f2924176740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a8350 0x5639e89b24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.767+0000 7f291a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a7a30 0x5639e89a7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.767+0000 7f291a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a7a30 0x5639e89a7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:47110/0 (socket says 172.21.15.28:47110) 2024-04-24T08:13:52.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.767+0000 7f2912a00640 1 --2- 172.21.15.28:0/1147776267 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29000a3fe0 0x7f29000a64a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.767+0000 7f2919e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a8350 0x5639e89b24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.767+0000 7f2919e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a8350 0x5639e89b24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:36754/0 (socket says 172.21.15.28:36754) 2024-04-24T08:13:52.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.767+0000 7f2924176740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89b2a20 0x5639e89b4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.767+0000 7f291a800640 1 -- 172.21.15.28:0/892773884 learned_addr learned my addr 172.21.15.28:0/892773884 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:52.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.767+0000 7f2912a00640 1 -- 172.21.15.28:0/1147776267 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f290c0cffd0 con 0x7f291c109270 2024-04-24T08:13:52.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.768+0000 7f2924176740 1 -- 172.21.15.28:0/892773884 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5639e890d1f0 con 0x5639e89b2a20 2024-04-24T08:13:52.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.768+0000 7f2924176740 1 -- 172.21.15.28:0/892773884 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5639e88b3350 con 0x5639e89a8350 2024-04-24T08:13:52.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.768+0000 7f2924176740 1 -- 172.21.15.28:0/892773884 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5639e89091b0 con 0x5639e89a7a30 2024-04-24T08:13:52.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.769+0000 7f291a800640 1 --2- 172.21.15.28:0/1147776267 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29000a3fe0 0x7f29000a64a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.770+0000 7f2919e00640 1 -- 172.21.15.28:0/892773884 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a7a30 msgr2=0x5639e89a7e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.770+0000 7f2919e00640 1 --2- 172.21.15.28:0/892773884 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a7a30 0x5639e89a7e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.770+0000 7f2919e00640 1 -- 172.21.15.28:0/892773884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89b2a20 msgr2=0x5639e89b4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.770+0000 7f2919e00640 1 --2- 172.21.15.28:0/892773884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89b2a20 0x5639e89b4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.770+0000 7f2919e00640 1 -- 172.21.15.28:0/892773884 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639e8907f50 con 0x5639e89a8350 2024-04-24T08:13:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.770+0000 7f2919e00640 1 --2- 172.21.15.28:0/892773884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a8350 0x5639e89b24e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f290c0cd700 tx=0x7f290c010c00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=329742c306e3cc8f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:52.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.771+0000 7f2907400640 1 -- 172.21.15.28:0/892773884 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290c020070 con 0x5639e89a8350 2024-04-24T08:13:52.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.771+0000 7f2907400640 1 -- 172.21.15.28:0/892773884 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f290c0cde50 con 0x5639e89a8350 2024-04-24T08:13:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.771+0000 7f2907400640 1 -- 172.21.15.28:0/892773884 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290c0d2070 con 0x5639e89a8350 2024-04-24T08:13:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.772+0000 7f2924176740 1 -- 172.21.15.28:0/892773884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a8350 msgr2=0x5639e89b24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.772+0000 7f2924176740 1 --2- 172.21.15.28:0/892773884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a8350 0x5639e89b24e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f290c0cd700 tx=0x7f290c010c00 comp rx=0 tx=0).stop 2024-04-24T08:13:52.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.773+0000 7f2924176740 1 -- 172.21.15.28:0/892773884 shutdown_connections 2024-04-24T08:13:52.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.773+0000 7f2924176740 1 --2- 172.21.15.28:0/892773884 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89b2a20 0x5639e89b4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.773+0000 7f291a800640 1 --2- 172.21.15.28:0/1147776267 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29000a3fe0 0x7f29000a64a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2914030040 tx=0x7f29140098f0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:52.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.773+0000 7f2924176740 1 --2- 172.21.15.28:0/892773884 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a8350 0x5639e89b24e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.773+0000 7f2924176740 1 --2- 172.21.15.28:0/892773884 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a7a30 0x5639e89a7e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.773+0000 7f2924176740 1 -- 172.21.15.28:0/892773884 >> 172.21.15.28:0/892773884 conn(0x5639e89a3d50 msgr2=0x5639e89a4130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.774+0000 7f2924176740 1 -- 172.21.15.28:0/892773884 shutdown_connections 2024-04-24T08:13:52.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.774+0000 7f2924176740 1 -- 172.21.15.28:0/892773884 wait complete. 2024-04-24T08:13:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.775+0000 7f2924176740 1 -- start start 2024-04-24T08:13:52.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.776+0000 7f2924176740 1 Processor -- start 2024-04-24T08:13:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.777+0000 7f2924176740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a7a30 0x5639e89afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.777+0000 7f2924176740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89a8350 0x5639e89aff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.777+0000 7f2924176740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89b2a20 0x5639e89b04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.778+0000 7f291a800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89b2a20 0x5639e89b04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.778+0000 7f291a800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89b2a20 0x5639e89b04c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:36764/0 (socket says 172.21.15.28:36764) 2024-04-24T08:13:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.778+0000 7f291a800640 1 -- 172.21.15.28:0/1050344164 learned_addr learned my addr 172.21.15.28:0/1050344164 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.778+0000 7f2924176740 1 -- 172.21.15.28:0/1050344164 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5639e89b57b0 con 0x5639e89a8350 2024-04-24T08:13:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.778+0000 7f2924176740 1 -- 172.21.15.28:0/1050344164 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5639e88c6690 con 0x5639e89b2a20 2024-04-24T08:13:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.778+0000 7f2924176740 1 -- 172.21.15.28:0/1050344164 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5639e89b5930 con 0x5639e89a7a30 2024-04-24T08:13:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.779+0000 7f291a800640 1 -- 172.21.15.28:0/1050344164 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a7a30 msgr2=0x5639e89afa40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:13:52.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.779+0000 7f291a800640 1 --2- 172.21.15.28:0/1050344164 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a7a30 0x5639e89afa40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.779+0000 7f291a800640 1 -- 172.21.15.28:0/1050344164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89a8350 msgr2=0x5639e89aff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.779+0000 7f291a800640 1 --2- 172.21.15.28:0/1050344164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89a8350 0x5639e89aff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.779+0000 7f291a800640 1 -- 172.21.15.28:0/1050344164 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639e88c5920 con 0x5639e89b2a20 2024-04-24T08:13:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.779+0000 7f2924176740 10 client.0 _ll_register_callbacks cb 0x5639e8902650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.779+0000 7f2924176740 1 -- 172.21.15.28:0/1050344164 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f290c0cde50 con 0x5639e89b2a20 2024-04-24T08:13:52.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.780+0000 7f28fbe00640 10 client.26069 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:13:52.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.780+0000 7f291a800640 1 --2- 172.21.15.28:0/1050344164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89b2a20 0x5639e89b04c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f291402bfc0 tx=0x7f2914005fa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=629a5375f9532f7f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:52.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.781+0000 7f28fbe00640 1 -- 172.21.15.28:0/1050344164 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2914057060 con 0x5639e89b2a20 2024-04-24T08:13:52.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.781+0000 7f28fbe00640 1 -- 172.21.15.28:0/1050344164 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f2914039450 con 0x5639e89b2a20 2024-04-24T08:13:52.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.781+0000 7f28fbe00640 1 -- 172.21.15.28:0/1050344164 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2914034040 con 0x5639e89b2a20 2024-04-24T08:13:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.782+0000 7f2924176740 10 client.26069 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:52.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.782+0000 7f2924176740 1 -- 172.21.15.28:0/1050344164 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({fsmap=25}) -- 0x5639e86e97d0 con 0x5639e89b2a20 2024-04-24T08:13:52.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.783+0000 7f28fbe00640 1 -- 172.21.15.28:0/1050344164 <== mon.2 v2:172.21.15.117:3300/0 4 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f2914051030 con 0x5639e89b2a20 2024-04-24T08:13:52.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.783+0000 7f2924176740 10 client.26069 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:52.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.783+0000 7f2924176740 10 client.26069 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:13:52.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.783+0000 7f2924176740 20 client.26069 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:52.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.783+0000 7f2924176740 1 --2- 172.21.15.28:0/1050344164 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5639e89c2bd0 0x5639e89c5090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.783+0000 7f2924176740 4 client.26069 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:13:52.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.783+0000 7f2924176740 1 -- 172.21.15.28:0/1050344164 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639e891ac10 con 0x5639e89c2bd0 2024-04-24T08:13:52.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.785+0000 7f2919e00640 1 --2- 172.21.15.28:0/1050344164 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5639e89c2bd0 0x5639e89c5090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.786+0000 7f2919e00640 1 --2- 172.21.15.28:0/1050344164 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5639e89c2bd0 0x5639e89c5090 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4402056d315da4dd server_cookie=70e6d9ca442f5017 in_seq=0 out_seq=0 2024-04-24T08:13:52.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.786+0000 7f28fbe00640 10 client.26069 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:52.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.789+0000 7f28fbe00640 1 -- 172.21.15.28:0/1050344164 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5639e891ac10 con 0x5639e89c2bd0 2024-04-24T08:13:52.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.790+0000 7f28fbe00640 10 client.26069 handle_command_reply: tid=0 2024-04-24T08:13:52.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.790+0000 7f2924176740 1 client.26069 shutdown 2024-04-24T08:13:52.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.791+0000 7f2924176740 1 -- 172.21.15.28:0/1050344164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89b2a20 msgr2=0x5639e89b04c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.791+0000 7f2924176740 1 --2- 172.21.15.28:0/1050344164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89b2a20 0x5639e89b04c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f291402bfc0 tx=0x7f2914005fa0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.791+0000 7f2924176740 1 -- 172.21.15.28:0/1050344164 shutdown_connections 2024-04-24T08:13:52.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.791+0000 7f2924176740 1 --2- 172.21.15.28:0/1050344164 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5639e89c2bd0 0x5639e89c5090 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.791+0000 7f2924176740 1 --2- 172.21.15.28:0/1050344164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89a8350 0x5639e89aff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.791+0000 7f2924176740 1 --2- 172.21.15.28:0/1050344164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89b2a20 0x5639e89b04c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.791+0000 7f28fbe00640 0 client.26069 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:52.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.791+0000 7f2924176740 1 --2- 172.21.15.28:0/1050344164 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a7a30 0x5639e89afa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.791+0000 7f2924176740 1 -- 172.21.15.28:0/1050344164 >> 172.21.15.28:0/1050344164 conn(0x5639e89a3d50 msgr2=0x5639e89a2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.792+0000 7f2924176740 1 -- 172.21.15.28:0/1050344164 shutdown_connections 2024-04-24T08:13:52.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.794+0000 7f2924176740 1 -- 172.21.15.28:0/1050344164 wait complete. 2024-04-24T08:13:52.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.795+0000 7f2924176740 20 client.26069 trim_cache size 0 max 16384 2024-04-24T08:13:52.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.848+0000 7f2924176740 1 Processor -- start 2024-04-24T08:13:52.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.848+0000 7f2924176740 1 -- start start 2024-04-24T08:13:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.850+0000 7f2924176740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a7a30 0x5639e89ae310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.850+0000 7f2924176740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a8350 0x5639e89b02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.850+0000 7f2924176740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89b2a20 0x5639e89ca9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.850+0000 7f2924176740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f291c1bf420 con 0x5639e89b2a20 2024-04-24T08:13:52.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.850+0000 7f2924176740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f291c1bccf0 con 0x5639e89a8350 2024-04-24T08:13:52.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.850+0000 7f2924176740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f291c1bab40 con 0x5639e89a7a30 2024-04-24T08:13:52.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.851+0000 7f2919400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a8350 0x5639e89b02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.851+0000 7f2919400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a8350 0x5639e89b02e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:36778/0 (socket says 172.21.15.28:36778) 2024-04-24T08:13:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.851+0000 7f2919400640 1 -- 172.21.15.28:0/2406927564 learned_addr learned my addr 172.21.15.28:0/2406927564 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.851+0000 7f2919e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a7a30 0x5639e89ae310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.851+0000 7f291a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89b2a20 0x5639e89ca9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.852+0000 7f2919e00640 1 -- 172.21.15.28:0/2406927564 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a8350 msgr2=0x5639e89b02e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.852+0000 7f2919e00640 1 --2- 172.21.15.28:0/2406927564 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a8350 0x5639e89b02e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.852+0000 7f2919e00640 1 -- 172.21.15.28:0/2406927564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89b2a20 msgr2=0x5639e89ca9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.852+0000 7f2919e00640 1 --2- 172.21.15.28:0/2406927564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89b2a20 0x5639e89ca9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.852+0000 7f2919e00640 1 -- 172.21.15.28:0/2406927564 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639e861f3f0 con 0x5639e89a7a30 2024-04-24T08:13:52.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.853+0000 7f2919e00640 1 --2- 172.21.15.28:0/2406927564 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a7a30 0x5639e89ae310 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x5639e89b5460 tx=0x7f290c000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=105e6450089e4191 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.853+0000 7f2906a00640 1 -- 172.21.15.28:0/2406927564 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290c014070 con 0x5639e89a7a30 2024-04-24T08:13:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.853+0000 7f2906a00640 1 -- 172.21.15.28:0/2406927564 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f290c0cd980 con 0x5639e89a7a30 2024-04-24T08:13:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.853+0000 7f2906a00640 1 -- 172.21.15.28:0/2406927564 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f290c020040 con 0x5639e89a7a30 2024-04-24T08:13:52.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.854+0000 7f2924176740 1 -- 172.21.15.28:0/2406927564 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a7a30 msgr2=0x5639e89ae310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.854+0000 7f2924176740 1 --2- 172.21.15.28:0/2406927564 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a7a30 0x5639e89ae310 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x5639e89b5460 tx=0x7f290c000bc0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.855+0000 7f2924176740 1 -- 172.21.15.28:0/2406927564 shutdown_connections 2024-04-24T08:13:52.855 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.855+0000 7f2924176740 1 --2- 172.21.15.28:0/2406927564 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89b2a20 0x5639e89ca9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.855+0000 7f2924176740 1 --2- 172.21.15.28:0/2406927564 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a8350 0x5639e89b02e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.855+0000 7f2924176740 1 --2- 172.21.15.28:0/2406927564 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a7a30 0x5639e89ae310 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.855+0000 7f2924176740 1 -- 172.21.15.28:0/2406927564 >> 172.21.15.28:0/2406927564 conn(0x5639e89a3d50 msgr2=0x5639e89a2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:52.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.856+0000 7f2924176740 1 -- 172.21.15.28:0/2406927564 shutdown_connections 2024-04-24T08:13:52.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.856+0000 7f2924176740 1 -- 172.21.15.28:0/2406927564 wait complete. 2024-04-24T08:13:52.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.857+0000 7f2924176740 1 -- start start 2024-04-24T08:13:52.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.858+0000 7f2924176740 1 Processor -- start 2024-04-24T08:13:52.861 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.861+0000 7f2924176740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a7a30 0x5639e89ba820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.861+0000 7f2924176740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a8350 0x5639e89bad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.862+0000 7f2924176740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89b2a20 0x5639e89b6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.862+0000 7f2924176740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f291c1bb1b0 con 0x5639e89b2a20 2024-04-24T08:13:52.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.862+0000 7f2924176740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f291c1bc620 con 0x5639e89a7a30 2024-04-24T08:13:52.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.862+0000 7f2924176740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f291c1bb880 con 0x5639e89a8350 2024-04-24T08:13:52.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.862+0000 7f2924176740 10 client.0 _ll_register_callbacks cb 0x7f291402fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:13:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.863+0000 7f2919e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a7a30 0x5639e89ba820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.863+0000 7f2919e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a7a30 0x5639e89ba820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:36780/0 (socket says 172.21.15.28:36780) 2024-04-24T08:13:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.864+0000 7f291a800640 1 --2- 172.21.15.28:0/2640353690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89b2a20 0x5639e89b6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.863+0000 7f2919e00640 1 -- 172.21.15.28:0/2640353690 learned_addr learned my addr 172.21.15.28:0/2640353690 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:13:52.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.865+0000 7f2919400640 1 --2- 172.21.15.28:0/2640353690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a8350 0x5639e89bad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.865+0000 7f291a800640 1 -- 172.21.15.28:0/2640353690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a8350 msgr2=0x5639e89bad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.865+0000 7f291a800640 1 --2- 172.21.15.28:0/2640353690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a8350 0x5639e89bad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.865+0000 7f291a800640 1 -- 172.21.15.28:0/2640353690 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a7a30 msgr2=0x5639e89ba820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:13:52.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.865+0000 7f291a800640 1 --2- 172.21.15.28:0/2640353690 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a7a30 0x5639e89ba820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.865+0000 7f291a800640 1 -- 172.21.15.28:0/2640353690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639e85df0f0 con 0x5639e89b2a20 2024-04-24T08:13:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.866+0000 7f2919400640 1 --2- 172.21.15.28:0/2640353690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a8350 0x5639e89bad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:13:52.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.866+0000 7f2919e00640 1 --2- 172.21.15.28:0/2640353690 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a7a30 0x5639e89ba820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:13:52.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.867+0000 7f291a800640 1 --2- 172.21.15.28:0/2640353690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89b2a20 0x5639e89b6cc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2914009a60 tx=0x7f291403a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd137d7ae0dba9d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:13:52.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.867+0000 7f28fbe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:13:52.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.870+0000 7f28fbe00640 1 -- 172.21.15.28:0/2640353690 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2914039760 con 0x5639e89b2a20 2024-04-24T08:13:52.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.870+0000 7f28fbe00640 1 -- 172.21.15.28:0/2640353690 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f2914039900 con 0x5639e89b2a20 2024-04-24T08:13:52.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.871+0000 7f2924176740 1 -- 172.21.15.28:0/2640353690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f290c0cd980 con 0x5639e89b2a20 2024-04-24T08:13:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.871+0000 7f28fbe00640 1 -- 172.21.15.28:0/2640353690 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f2914039b00 con 0x5639e89b2a20 2024-04-24T08:13:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.872+0000 7f2924176740 10 client.15912 fetch_fsmap learned FSMap version 25 2024-04-24T08:13:52.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.872+0000 7f2924176740 1 -- 172.21.15.28:0/2640353690 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5639e88d2810 con 0x5639e89b2a20 2024-04-24T08:13:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.874+0000 7f28fbe00640 1 -- 172.21.15.28:0/2640353690 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f290c0cd980 con 0x5639e89b2a20 2024-04-24T08:13:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.874+0000 7f28fbe00640 1 -- 172.21.15.28:0/2640353690 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f2914063070 con 0x5639e89b2a20 2024-04-24T08:13:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.875+0000 7f2924176740 10 client.15912 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:13:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.875+0000 7f2924176740 10 client.15912 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:13:52.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.875+0000 7f2924176740 20 client.15912 populate_metadata read hostname 'smithi028' 2024-04-24T08:13:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.875+0000 7f2924176740 1 --2- 172.21.15.28:0/2640353690 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5639e8a29f90 0x5639e8a2c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:13:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.875+0000 7f2924176740 4 client.15912 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:13:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.875+0000 7f2924176740 1 -- 172.21.15.28:0/2640353690 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5639e8914a80 con 0x5639e8a29f90 2024-04-24T08:13:52.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.876+0000 7f2919e00640 1 --2- 172.21.15.28:0/2640353690 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5639e8a29f90 0x5639e8a2c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:13:52.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.877+0000 7f2919e00640 1 --2- 172.21.15.28:0/2640353690 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5639e8a29f90 0x5639e8a2c380 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6a37235aeb41244 server_cookie=34de809f2423447d in_seq=0 out_seq=0 2024-04-24T08:13:52.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.877+0000 7f28fbe00640 10 client.15912 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.880+0000 7f28fbe00640 1 -- 172.21.15.28:0/2640353690 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5639e8914a80 con 0x5639e8a29f90 2024-04-24T08:13:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.880+0000 7f28fbe00640 10 client.15912 handle_command_reply: tid=0 2024-04-24T08:13:52.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.880+0000 7f2924176740 1 client.15912 shutdown 2024-04-24T08:13:52.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.881+0000 7f2924176740 1 -- 172.21.15.28:0/2640353690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89b2a20 msgr2=0x5639e89b6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.881+0000 7f2924176740 1 --2- 172.21.15.28:0/2640353690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89b2a20 0x5639e89b6cc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f2914009a60 tx=0x7f291403a000 comp rx=0 tx=0).stop 2024-04-24T08:13:52.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.881+0000 7f2924176740 1 -- 172.21.15.28:0/2640353690 shutdown_connections 2024-04-24T08:13:52.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.881+0000 7f2924176740 1 --2- 172.21.15.28:0/2640353690 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5639e8a29f90 0x5639e8a2c380 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.882+0000 7f2924176740 1 --2- 172.21.15.28:0/2640353690 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639e89b2a20 0x5639e89b6cc0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.882+0000 7f28fbe00640 0 client.15912 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:13:52.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.882+0000 7f2924176740 1 --2- 172.21.15.28:0/2640353690 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639e89a8350 0x5639e89bad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.882+0000 7f2924176740 1 --2- 172.21.15.28:0/2640353690 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639e89a7a30 0x5639e89ba820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.882+0000 7f2924176740 1 -- 172.21.15.28:0/2640353690 >> 172.21.15.28:0/2640353690 conn(0x5639e89a3d50 msgr2=0x5639e89a2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:52.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.882+0000 7f2924176740 1 -- 172.21.15.28:0/2640353690 shutdown_connections 2024-04-24T08:13:52.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.882+0000 7f2924176740 1 -- 172.21.15.28:0/2640353690 wait complete. 2024-04-24T08:13:52.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.882+0000 7f2924176740 20 client.15912 trim_cache size 0 max 16384 2024-04-24T08:13:52.883 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:13:52.883 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (211 inodes in the stack)", 2024-04-24T08:13:52.884 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:13:52.884 INFO:teuthology.orchestra.run.smithi028.stdout: "ae51db93-2230-4cd0-a1aa-d9c0bee08435": { 2024-04-24T08:13:52.884 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:13:52.884 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "ae51db93-2230-4cd0-a1aa-d9c0bee08435", 2024-04-24T08:13:52.884 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:13:52.884 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:13:52.884 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:13:52.884 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:13:52.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.884+0000 7f291b200640 1 -- 172.21.15.28:0/1147776267 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29000a3fe0 msgr2=0x7f29000a64a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.884+0000 7f291b200640 1 --2- 172.21.15.28:0/1147776267 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29000a3fe0 0x7f29000a64a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f2914030040 tx=0x7f29140098f0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.884+0000 7f291b200640 1 -- 172.21.15.28:0/1147776267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f291c109270 msgr2=0x7f291c1ad400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:13:52.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.884+0000 7f291b200640 1 --2- 172.21.15.28:0/1147776267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f291c109270 0x7f291c1ad400 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f290c00d950 tx=0x7f290c00de10 comp rx=0 tx=0).stop 2024-04-24T08:13:52.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.884+0000 7f291b200640 1 -- 172.21.15.28:0/1147776267 shutdown_connections 2024-04-24T08:13:52.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.884+0000 7f291b200640 1 --2- 172.21.15.28:0/1147776267 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f29000a3fe0 0x7f29000a64a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.884+0000 7f291b200640 1 --2- 172.21.15.28:0/1147776267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f291c10de20 0x7f291c1ad940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.884+0000 7f291b200640 1 --2- 172.21.15.28:0/1147776267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f291c1a75b0 0x7f291c1a7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.884+0000 7f291b200640 1 --2- 172.21.15.28:0/1147776267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f291c109270 0x7f291c1ad400 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:13:52.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.884+0000 7f291b200640 1 -- 172.21.15.28:0/1147776267 >> 172.21.15.28:0/1147776267 conn(0x7f291c06d3e0 msgr2=0x7f291c0719d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:13:52.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.884+0000 7f291b200640 1 -- 172.21.15.28:0/1147776267 shutdown_connections 2024-04-24T08:13:52.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:13:52.885+0000 7f291b200640 1 -- 172.21.15.28:0/1147776267 wait complete. 2024-04-24T08:13:52.916 INFO:tasks.cephfs.filesystem:scrub status for tag:ae51db93-2230-4cd0-a1aa-d9c0bee08435 - {'path': '/', 'tag': 'ae51db93-2230-4cd0-a1aa-d9c0bee08435', 'options': 'recursive,force'} 2024-04-24T08:13:53.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:53 smithi117 ceph-mon[32438]: pgmap v109: 97 pgs: 97 active+clean; 13 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 141 MiB/s wr, 20.07k op/s 2024-04-24T08:13:53.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:53 smithi117 ceph-mon[32438]: scrub queued for path: / 2024-04-24T08:13:53.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:53 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:53.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:53 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:13:53.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:53 smithi194 ceph-mon[32836]: pgmap v109: 97 pgs: 97 active+clean; 13 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 141 MiB/s wr, 20.07k op/s 2024-04-24T08:13:53.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:53 smithi194 ceph-mon[32836]: scrub queued for path: / 2024-04-24T08:13:53.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:53 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:53.691 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:53 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:13:53.791 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:53 smithi028 ceph-mon[27786]: pgmap v109: 97 pgs: 97 active+clean; 13 GiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 582 KiB/s rd, 141 MiB/s wr, 20.07k op/s 2024-04-24T08:13:53.792 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:53 smithi028 ceph-mon[27786]: scrub queued for path: / 2024-04-24T08:13:53.792 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:53 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:13:53.792 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:53 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:13:55.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:55 smithi194 ceph-mon[32836]: pgmap v110: 97 pgs: 97 active+clean; 14 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 118 MiB/s wr, 15.77k op/s 2024-04-24T08:13:55.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:55 smithi028 ceph-mon[27786]: pgmap v110: 97 pgs: 97 active+clean; 14 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 118 MiB/s wr, 15.77k op/s 2024-04-24T08:13:55.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:55 smithi117 ceph-mon[32438]: pgmap v110: 97 pgs: 97 active+clean; 14 GiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 400 KiB/s rd, 118 MiB/s wr, 15.77k op/s 2024-04-24T08:13:57.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:57 smithi194 ceph-mon[32836]: pgmap v111: 97 pgs: 97 active+clean; 14 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 163 MiB/s wr, 21.09k op/s 2024-04-24T08:13:57.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:57 smithi028 ceph-mon[27786]: pgmap v111: 97 pgs: 97 active+clean; 14 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 163 MiB/s wr, 21.09k op/s 2024-04-24T08:13:57.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:57 smithi117 ceph-mon[32438]: pgmap v111: 97 pgs: 97 active+clean; 14 GiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 163 MiB/s wr, 21.09k op/s 2024-04-24T08:13:59.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:13:59 smithi194 ceph-mon[32836]: pgmap v112: 97 pgs: 97 active+clean; 14 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 152 MiB/s wr, 19.44k op/s 2024-04-24T08:13:59.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:13:59 smithi028 ceph-mon[27786]: pgmap v112: 97 pgs: 97 active+clean; 14 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 152 MiB/s wr, 19.44k op/s 2024-04-24T08:13:59.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:13:59 smithi117 ceph-mon[32438]: pgmap v112: 97 pgs: 97 active+clean; 14 GiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 509 KiB/s rd, 152 MiB/s wr, 19.44k op/s 2024-04-24T08:14:01.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:01 smithi117 ceph-mon[32438]: pgmap v113: 97 pgs: 97 active+clean; 14 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 140 MiB/s wr, 17.19k op/s 2024-04-24T08:14:01.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:01 smithi194 ceph-mon[32836]: pgmap v113: 97 pgs: 97 active+clean; 14 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 140 MiB/s wr, 17.19k op/s 2024-04-24T08:14:01.793 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:01 smithi028 ceph-mon[27786]: pgmap v113: 97 pgs: 97 active+clean; 14 GiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 140 MiB/s wr, 17.19k op/s 2024-04-24T08:14:03.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:03 smithi194 ceph-mon[32836]: pgmap v114: 97 pgs: 97 active+clean; 15 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 165 MiB/s wr, 20.50k op/s 2024-04-24T08:14:03.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:03 smithi028 ceph-mon[27786]: pgmap v114: 97 pgs: 97 active+clean; 15 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 165 MiB/s wr, 20.50k op/s 2024-04-24T08:14:03.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:03 smithi117 ceph-mon[32438]: pgmap v114: 97 pgs: 97 active+clean; 15 GiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 165 MiB/s wr, 20.50k op/s 2024-04-24T08:14:05.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:05 smithi194 ceph-mon[32836]: pgmap v115: 97 pgs: 97 active+clean; 15 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 143 MiB/s wr, 17.47k op/s 2024-04-24T08:14:05.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:05 smithi028 ceph-mon[27786]: pgmap v115: 97 pgs: 97 active+clean; 15 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 143 MiB/s wr, 17.47k op/s 2024-04-24T08:14:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:05 smithi117 ceph-mon[32438]: pgmap v115: 97 pgs: 97 active+clean; 15 GiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 143 MiB/s wr, 17.47k op/s 2024-04-24T08:14:07.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:07 smithi028 ceph-mon[27786]: pgmap v116: 97 pgs: 97 active+clean; 15 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 159 MiB/s wr, 19.23k op/s 2024-04-24T08:14:07.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:07 smithi117 ceph-mon[32438]: pgmap v116: 97 pgs: 97 active+clean; 15 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 159 MiB/s wr, 19.23k op/s 2024-04-24T08:14:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:07 smithi194 ceph-mon[32836]: pgmap v116: 97 pgs: 97 active+clean; 15 GiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 311 KiB/s rd, 159 MiB/s wr, 19.23k op/s 2024-04-24T08:14:09.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:09 smithi028 ceph-mon[27786]: pgmap v117: 97 pgs: 97 active+clean; 15 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 138 MiB/s wr, 16.69k op/s 2024-04-24T08:14:09.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:09 smithi117 ceph-mon[32438]: pgmap v117: 97 pgs: 97 active+clean; 15 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 138 MiB/s wr, 16.69k op/s 2024-04-24T08:14:09.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:09 smithi194 ceph-mon[32836]: pgmap v117: 97 pgs: 97 active+clean; 15 GiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 209 KiB/s rd, 138 MiB/s wr, 16.69k op/s 2024-04-24T08:14:11.794 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:11 smithi028 ceph-mon[27786]: pgmap v118: 97 pgs: 97 active+clean; 15 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 138 MiB/s wr, 16.44k op/s 2024-04-24T08:14:11.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:11 smithi117 ceph-mon[32438]: pgmap v118: 97 pgs: 97 active+clean; 15 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 138 MiB/s wr, 16.44k op/s 2024-04-24T08:14:11.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:11 smithi194 ceph-mon[32836]: pgmap v118: 97 pgs: 97 active+clean; 15 GiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 259 KiB/s rd, 138 MiB/s wr, 16.44k op/s 2024-04-24T08:14:13.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:13 smithi028 ceph-mon[27786]: pgmap v119: 97 pgs: 97 active+clean; 15 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 165 MiB/s wr, 19.06k op/s 2024-04-24T08:14:13.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:13 smithi117 ceph-mon[32438]: pgmap v119: 97 pgs: 97 active+clean; 15 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 165 MiB/s wr, 19.06k op/s 2024-04-24T08:14:13.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:13 smithi194 ceph-mon[32836]: pgmap v119: 97 pgs: 97 active+clean; 15 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 260 KiB/s rd, 165 MiB/s wr, 19.06k op/s 2024-04-24T08:14:15.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:15 smithi028 ceph-mon[27786]: pgmap v120: 97 pgs: 97 active+clean; 15 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 148 MiB/s wr, 16.63k op/s 2024-04-24T08:14:15.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:15 smithi117 ceph-mon[32438]: pgmap v120: 97 pgs: 97 active+clean; 15 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 148 MiB/s wr, 16.63k op/s 2024-04-24T08:14:15.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:15 smithi194 ceph-mon[32836]: pgmap v120: 97 pgs: 97 active+clean; 15 GiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 214 KiB/s rd, 148 MiB/s wr, 16.63k op/s 2024-04-24T08:14:17.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:17 smithi028 ceph-mon[27786]: pgmap v121: 97 pgs: 97 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 169 MiB/s wr, 19.51k op/s 2024-04-24T08:14:17.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:17 smithi117 ceph-mon[32438]: pgmap v121: 97 pgs: 97 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 169 MiB/s wr, 19.51k op/s 2024-04-24T08:14:17.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:17 smithi194 ceph-mon[32836]: pgmap v121: 97 pgs: 97 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 218 KiB/s rd, 169 MiB/s wr, 19.51k op/s 2024-04-24T08:14:19.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:19 smithi028 ceph-mon[27786]: pgmap v122: 97 pgs: 97 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 160 MiB/s wr, 18.81k op/s 2024-04-24T08:14:19.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:19 smithi117 ceph-mon[32438]: pgmap v122: 97 pgs: 97 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 160 MiB/s wr, 18.81k op/s 2024-04-24T08:14:19.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:19 smithi194 ceph-mon[32836]: pgmap v122: 97 pgs: 97 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 160 MiB/s wr, 18.81k op/s 2024-04-24T08:14:21.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:21 smithi028 ceph-mon[27786]: pgmap v123: 97 pgs: 97 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 136 MiB/s wr, 16.01k op/s 2024-04-24T08:14:21.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:21 smithi117 ceph-mon[32438]: pgmap v123: 97 pgs: 97 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 136 MiB/s wr, 16.01k op/s 2024-04-24T08:14:21.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:21 smithi194 ceph-mon[32836]: pgmap v123: 97 pgs: 97 active+clean; 15 GiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 136 MiB/s wr, 16.01k op/s 2024-04-24T08:14:22.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:14:22.918 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:14:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.251+0000 7f7e2ca00640 1 -- 172.21.15.28:0/1128227137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e28104cd0 msgr2=0x7f7e28108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.252+0000 7f7e2ca00640 1 --2- 172.21.15.28:0/1128227137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e28104cd0 0x7f7e28108d10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f7e14009900 tx=0x7f7e1402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.252+0000 7f7e2ca00640 1 -- 172.21.15.28:0/1128227137 shutdown_connections 2024-04-24T08:14:23.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.252+0000 7f7e2ca00640 1 --2- 172.21.15.28:0/1128227137 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e281093f0 0x7f7e28110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.252+0000 7f7e2ca00640 1 --2- 172.21.15.28:0/1128227137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e28104cd0 0x7f7e28108d10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.253+0000 7f7e2ca00640 1 --2- 172.21.15.28:0/1128227137 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e28104320 0x7f7e28104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.253+0000 7f7e2ca00640 1 -- 172.21.15.28:0/1128227137 >> 172.21.15.28:0/1128227137 conn(0x7f7e280ffb40 msgr2=0x7f7e28101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.253+0000 7f7e2ca00640 1 -- 172.21.15.28:0/1128227137 shutdown_connections 2024-04-24T08:14:23.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.253+0000 7f7e2ca00640 1 -- 172.21.15.28:0/1128227137 wait complete. 2024-04-24T08:14:23.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.257+0000 7f7e2ca00640 1 Processor -- start 2024-04-24T08:14:23.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.258+0000 7f7e2ca00640 1 -- start start 2024-04-24T08:14:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.259+0000 7f7e2ca00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e28104320 0x7f7e28110ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.259+0000 7f7e2ca00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e281093f0 0x7f7e2810ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.259+0000 7f7e2ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e2810f370 0x7f7e2810f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.259+0000 7f7e27400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e28104320 0x7f7e28110ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:23.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.260+0000 7f7e27400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e28104320 0x7f7e28110ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44490/0 (socket says 172.21.15.28:44490) 2024-04-24T08:14:23.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.260+0000 7f7e27400640 1 -- 172.21.15.28:0/1157913961 learned_addr learned my addr 172.21.15.28:0/1157913961 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:14:23.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.260+0000 7f7e2ca00640 1 -- 172.21.15.28:0/1157913961 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e28076ba0 con 0x7f7e2810f370 2024-04-24T08:14:23.260 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.260+0000 7f7e2ca00640 1 -- 172.21.15.28:0/1157913961 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7e28076a20 con 0x7f7e281093f0 2024-04-24T08:14:23.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.260+0000 7f7e2ca00640 1 -- 172.21.15.28:0/1157913961 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7e28076d20 con 0x7f7e28104320 2024-04-24T08:14:23.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.260+0000 7f7e26a00640 1 --2- 172.21.15.28:0/1157913961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e281093f0 0x7f7e2810ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:23.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.261+0000 7f7e27400640 1 -- 172.21.15.28:0/1157913961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e281093f0 msgr2=0x7f7e2810ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:23.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.261+0000 7f7e27400640 1 --2- 172.21.15.28:0/1157913961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e281093f0 0x7f7e2810ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.261+0000 7f7e27400640 1 -- 172.21.15.28:0/1157913961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e2810f370 msgr2=0x7f7e2810f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:23.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.261+0000 7f7e27400640 1 --2- 172.21.15.28:0/1157913961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e2810f370 0x7f7e2810f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.261+0000 7f7e27400640 1 -- 172.21.15.28:0/1157913961 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e281ab490 con 0x7f7e28104320 2024-04-24T08:14:23.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.263+0000 7f7e27400640 1 --2- 172.21.15.28:0/1157913961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e28104320 0x7f7e28110ce0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7e1800c920 tx=0x7f7e1800cde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:23.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.263+0000 7f7e1fe00640 1 -- 172.21.15.28:0/1157913961 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18007c70 con 0x7f7e28104320 2024-04-24T08:14:23.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.263+0000 7f7e1fe00640 1 -- 172.21.15.28:0/1157913961 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e18007e10 con 0x7f7e28104320 2024-04-24T08:14:23.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.263+0000 7f7e1fe00640 1 -- 172.21.15.28:0/1157913961 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e18018890 con 0x7f7e28104320 2024-04-24T08:14:23.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.264+0000 7f7e2ca00640 1 -- 172.21.15.28:0/1157913961 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e281ab780 con 0x7f7e28104320 2024-04-24T08:14:23.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.264+0000 7f7e2ca00640 1 -- 172.21.15.28:0/1157913961 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e281abc50 con 0x7f7e28104320 2024-04-24T08:14:23.280 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.280+0000 7f7e1fe00640 1 -- 172.21.15.28:0/1157913961 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7e18018a30 con 0x7f7e28104320 2024-04-24T08:14:23.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.280+0000 7f7e30f98740 1 Processor -- start 2024-04-24T08:14:23.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.281+0000 7f7e30f98740 1 -- start start 2024-04-24T08:14:23.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.285+0000 7f7e30f98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56502cc62a30 0x56502cc62e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.285+0000 7f7e30f98740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc63350 0x56502cc6d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.285+0000 7f7e30f98740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc6da20 0x56502cc6fe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.285+0000 7f7e30f98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56502cbc81f0 con 0x56502cc62a30 2024-04-24T08:14:23.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.285+0000 7f7e30f98740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56502cb6e350 con 0x56502cc6da20 2024-04-24T08:14:23.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.285+0000 7f7e30f98740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56502cbc41b0 con 0x56502cc63350 2024-04-24T08:14:23.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.286+0000 7f7e26a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc6da20 0x56502cc6fe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:23.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.286+0000 7f7e26a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc6da20 0x56502cc6fe10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51826/0 (socket says 172.21.15.28:51826) 2024-04-24T08:14:23.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.286+0000 7f7e26a00640 1 -- 172.21.15.28:0/3697063216 learned_addr learned my addr 172.21.15.28:0/3697063216 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:14:23.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.286+0000 7f7e27400640 1 --2- 172.21.15.28:0/3697063216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc63350 0x56502cc6d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:23.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.287+0000 7f7e1fe00640 1 --2- 172.21.15.28:0/1157913961 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7e0c0a3d20 0x7f7e0c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.287+0000 7f7e26a00640 1 -- 172.21.15.28:0/3697063216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc63350 msgr2=0x56502cc6d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:23.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.287+0000 7f7e26a00640 1 --2- 172.21.15.28:0/3697063216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc63350 0x56502cc6d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.287+0000 7f7e26a00640 1 -- 172.21.15.28:0/3697063216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56502cc62a30 msgr2=0x56502cc62e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:23.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.287+0000 7f7e26a00640 1 --2- 172.21.15.28:0/3697063216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56502cc62a30 0x56502cc62e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.287+0000 7f7e26a00640 1 -- 172.21.15.28:0/3697063216 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56502cbc2f50 con 0x56502cc6da20 2024-04-24T08:14:23.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.289+0000 7f7e1fe00640 1 -- 172.21.15.28:0/1157913961 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f7e180d08a0 con 0x7f7e28104320 2024-04-24T08:14:23.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.290+0000 7f7e26a00640 1 --2- 172.21.15.28:0/3697063216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc6da20 0x56502cc6fe10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7e2806b560 tx=0x7f7e140040f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5a099566ee7021f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:23.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.290+0000 7f7e13e00640 1 -- 172.21.15.28:0/3697063216 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e1403f070 con 0x56502cc6da20 2024-04-24T08:14:23.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.290+0000 7f7e13e00640 1 -- 172.21.15.28:0/3697063216 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e14005500 con 0x56502cc6da20 2024-04-24T08:14:23.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.290+0000 7f7e13e00640 1 -- 172.21.15.28:0/3697063216 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e14035070 con 0x56502cc6da20 2024-04-24T08:14:23.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.295+0000 7f7e27e00640 1 --2- 172.21.15.28:0/1157913961 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7e0c0a3d20 0x7f7e0c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:23.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.296+0000 7f7e30f98740 1 -- 172.21.15.28:0/3697063216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc6da20 msgr2=0x56502cc6fe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:23.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.296+0000 7f7e30f98740 1 --2- 172.21.15.28:0/3697063216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc6da20 0x56502cc6fe10 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7e2806b560 tx=0x7f7e140040f0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.299 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.299+0000 7f7e30f98740 1 -- 172.21.15.28:0/3697063216 shutdown_connections 2024-04-24T08:14:23.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.299+0000 7f7e30f98740 1 --2- 172.21.15.28:0/3697063216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc6da20 0x56502cc6fe10 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.299+0000 7f7e30f98740 1 --2- 172.21.15.28:0/3697063216 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc63350 0x56502cc6d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.299+0000 7f7e30f98740 1 --2- 172.21.15.28:0/3697063216 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56502cc62a30 0x56502cc62e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.299+0000 7f7e30f98740 1 -- 172.21.15.28:0/3697063216 >> 172.21.15.28:0/3697063216 conn(0x56502cc5ed50 msgr2=0x56502cc5f130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:23.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.300+0000 7f7e27e00640 1 --2- 172.21.15.28:0/1157913961 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7e0c0a3d20 0x7f7e0c0a61e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7e20001d90 tx=0x7f7e20001500 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:23.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.301+0000 7f7e30f98740 1 -- 172.21.15.28:0/3697063216 shutdown_connections 2024-04-24T08:14:23.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.302+0000 7f7e30f98740 1 -- 172.21.15.28:0/3697063216 wait complete. 2024-04-24T08:14:23.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.302+0000 7f7e30f98740 1 -- start start 2024-04-24T08:14:23.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.302+0000 7f7e30f98740 1 Processor -- start 2024-04-24T08:14:23.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.305+0000 7f7e30f98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56502cc62a30 0x56502cc6a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.306+0000 7f7e30f98740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc63350 0x56502cc6af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.306+0000 7f7e30f98740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc6da20 0x56502cc6b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.306+0000 7f7e27e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc6da20 0x56502cc6b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:23.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.307+0000 7f7e27e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc6da20 0x56502cc6b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44500/0 (socket says 172.21.15.28:44500) 2024-04-24T08:14:23.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.307+0000 7f7e27e00640 1 -- 172.21.15.28:0/931625343 learned_addr learned my addr 172.21.15.28:0/931625343 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:14:23.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.307+0000 7f7e30f98740 1 -- 172.21.15.28:0/931625343 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x56502cc817c0 con 0x56502cc62a30 2024-04-24T08:14:23.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.307+0000 7f7e30f98740 1 -- 172.21.15.28:0/931625343 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56502cb81690 con 0x56502cc63350 2024-04-24T08:14:23.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.307+0000 7f7e30f98740 1 -- 172.21.15.28:0/931625343 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x56502cc81940 con 0x56502cc6da20 2024-04-24T08:14:23.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.307+0000 7f7e27e00640 1 -- 172.21.15.28:0/931625343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc63350 msgr2=0x56502cc6af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:23.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.306+0000 7f7e26a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc63350 0x56502cc6af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:23.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.308+0000 7f7e27e00640 1 --2- 172.21.15.28:0/931625343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc63350 0x56502cc6af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.308+0000 7f7e27e00640 1 -- 172.21.15.28:0/931625343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56502cc62a30 msgr2=0x56502cc6a9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:23.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.308+0000 7f7e27e00640 1 --2- 172.21.15.28:0/931625343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56502cc62a30 0x56502cc6a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.309 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.308+0000 7f7e27e00640 1 -- 172.21.15.28:0/931625343 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56502cb80920 con 0x56502cc6da20 2024-04-24T08:14:23.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.308+0000 7f7e26a00640 1 --2- 172.21.15.28:0/931625343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc63350 0x56502cc6af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:14:23.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.309+0000 7f7e30f98740 10 client.0 _ll_register_callbacks cb 0x56502cbbd650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:14:23.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.309+0000 7f7e30f98740 1 -- 172.21.15.28:0/931625343 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e14005500 con 0x56502cc6da20 2024-04-24T08:14:23.311 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.309+0000 7f7e27e00640 1 --2- 172.21.15.28:0/931625343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc6da20 0x56502cc6b470 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7e20006450 tx=0x7f7e20006c70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e1a1984ae9ae063 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:23.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.311+0000 7f7e10c00640 10 client.26128 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:14:23.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.312+0000 7f7e10c00640 1 -- 172.21.15.28:0/931625343 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e20007e60 con 0x56502cc6da20 2024-04-24T08:14:23.312 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.312+0000 7f7e10c00640 1 -- 172.21.15.28:0/931625343 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f7e20016110 con 0x56502cc6da20 2024-04-24T08:14:23.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.313+0000 7f7e10c00640 1 -- 172.21.15.28:0/931625343 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7e20016950 con 0x56502cc6da20 2024-04-24T08:14:23.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.313+0000 7f7e30f98740 10 client.26128 fetch_fsmap learned FSMap version 25 2024-04-24T08:14:23.313 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.313+0000 7f7e30f98740 1 -- 172.21.15.28:0/931625343 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({fsmap=25}) -- 0x56502c9a47d0 con 0x56502cc6da20 2024-04-24T08:14:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.314+0000 7f7e10c00640 1 -- 172.21.15.28:0/931625343 <== mon.1 v2:172.21.15.194:3300/0 4 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f7e20016af0 con 0x56502cc6da20 2024-04-24T08:14:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.316+0000 7f7e30f98740 10 client.26128 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:14:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.316+0000 7f7e30f98740 10 client.26128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:14:23.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.316+0000 7f7e30f98740 20 client.26128 populate_metadata read hostname 'smithi028' 2024-04-24T08:14:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.316+0000 7f7e30f98740 1 --2- 172.21.15.28:0/931625343 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56502cc8ec70 0x56502cc91100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.316+0000 7f7e30f98740 4 client.26128 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:14:23.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.316+0000 7f7e30f98740 1 -- 172.21.15.28:0/931625343 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56502cbd5c10 con 0x56502cc8ec70 2024-04-24T08:14:23.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.317+0000 7f7e27400640 1 --2- 172.21.15.28:0/931625343 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56502cc8ec70 0x56502cc91100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:23.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.320+0000 7f7e27400640 1 --2- 172.21.15.28:0/931625343 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56502cc8ec70 0x56502cc91100 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99b3b4a163634463 server_cookie=a1ba6e892676c1af in_seq=0 out_seq=0 2024-04-24T08:14:23.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.320+0000 7f7e10c00640 10 client.26128 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:14:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.323+0000 7f7e10c00640 1 -- 172.21.15.28:0/931625343 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56502cbd5c10 con 0x56502cc8ec70 2024-04-24T08:14:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.323+0000 7f7e10c00640 10 client.26128 handle_command_reply: tid=0 2024-04-24T08:14:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.323+0000 7f7e30f98740 1 client.26128 shutdown 2024-04-24T08:14:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.324+0000 7f7e30f98740 1 -- 172.21.15.28:0/931625343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc6da20 msgr2=0x56502cc6b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.324+0000 7f7e30f98740 1 --2- 172.21.15.28:0/931625343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc6da20 0x56502cc6b470 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f7e20006450 tx=0x7f7e20006c70 comp rx=0 tx=0).stop 2024-04-24T08:14:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.324+0000 7f7e30f98740 1 -- 172.21.15.28:0/931625343 shutdown_connections 2024-04-24T08:14:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.324+0000 7f7e30f98740 1 --2- 172.21.15.28:0/931625343 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56502cc8ec70 0x56502cc91100 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.324+0000 7f7e30f98740 1 --2- 172.21.15.28:0/931625343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc6da20 0x56502cc6b470 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.324+0000 7f7e10c00640 0 client.26128 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:14:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.324+0000 7f7e30f98740 1 --2- 172.21.15.28:0/931625343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc63350 0x56502cc6af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.324+0000 7f7e30f98740 1 --2- 172.21.15.28:0/931625343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56502cc62a30 0x56502cc6a9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.324+0000 7f7e30f98740 1 -- 172.21.15.28:0/931625343 >> 172.21.15.28:0/931625343 conn(0x56502cc5ed50 msgr2=0x56502cc5dfd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:23.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.326+0000 7f7e30f98740 1 -- 172.21.15.28:0/931625343 shutdown_connections 2024-04-24T08:14:23.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.326+0000 7f7e30f98740 1 -- 172.21.15.28:0/931625343 wait complete. 2024-04-24T08:14:23.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.326+0000 7f7e30f98740 20 client.26128 trim_cache size 0 max 16384 2024-04-24T08:14:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.379+0000 7f7e30f98740 1 Processor -- start 2024-04-24T08:14:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.379+0000 7f7e30f98740 1 -- start start 2024-04-24T08:14:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.381+0000 7f7e30f98740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc62a30 0x56502cc8a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.381+0000 7f7e30f98740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc63350 0x56502cc969d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.381+0000 7f7e30f98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56502cc6da20 0x56502cc96f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.381+0000 7f7e30f98740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e281ae2d0 con 0x56502cc6da20 2024-04-24T08:14:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.381+0000 7f7e30f98740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7e28073e50 con 0x56502cc63350 2024-04-24T08:14:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.381+0000 7f7e30f98740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7e28071ca0 con 0x56502cc62a30 2024-04-24T08:14:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.382+0000 7f7e27400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc62a30 0x56502cc8a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.382+0000 7f7e27400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc62a30 0x56502cc8a680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44506/0 (socket says 172.21.15.28:44506) 2024-04-24T08:14:23.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.382+0000 7f7e26a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc63350 0x56502cc969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:23.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.382+0000 7f7e26a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc63350 0x56502cc969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51854/0 (socket says 172.21.15.28:51854) 2024-04-24T08:14:23.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.382+0000 7f7e26a00640 1 -- 172.21.15.28:0/489068170 learned_addr learned my addr 172.21.15.28:0/489068170 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:14:23.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.383+0000 7f7e26a00640 1 -- 172.21.15.28:0/489068170 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc62a30 msgr2=0x56502cc8a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:23.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.383+0000 7f7e26a00640 1 --2- 172.21.15.28:0/489068170 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc62a30 0x56502cc8a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.383+0000 7f7e26a00640 1 -- 172.21.15.28:0/489068170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56502cc6da20 msgr2=0x56502cc96f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:23.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.383+0000 7f7e26a00640 1 --2- 172.21.15.28:0/489068170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56502cc6da20 0x56502cc96f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.383+0000 7f7e26a00640 1 -- 172.21.15.28:0/489068170 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56502c8da3f0 con 0x56502cc63350 2024-04-24T08:14:23.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.383+0000 7f7e26a00640 1 --2- 172.21.15.28:0/489068170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc63350 0x56502cc969d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7e1402a910 tx=0x7f7e140045e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0fb4cad40e3b562 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:23.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.384+0000 7f7e13400640 1 -- 172.21.15.28:0/489068170 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e14035070 con 0x56502cc63350 2024-04-24T08:14:23.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.384+0000 7f7e13400640 1 -- 172.21.15.28:0/489068170 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e14031a90 con 0x56502cc63350 2024-04-24T08:14:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.384+0000 7f7e13400640 1 -- 172.21.15.28:0/489068170 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e14035070 con 0x56502cc63350 2024-04-24T08:14:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.385+0000 7f7e30f98740 1 -- 172.21.15.28:0/489068170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc63350 msgr2=0x56502cc969d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.385+0000 7f7e30f98740 1 --2- 172.21.15.28:0/489068170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc63350 0x56502cc969d0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f7e1402a910 tx=0x7f7e140045e0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.386+0000 7f7e30f98740 1 -- 172.21.15.28:0/489068170 shutdown_connections 2024-04-24T08:14:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.386+0000 7f7e30f98740 1 --2- 172.21.15.28:0/489068170 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56502cc6da20 0x56502cc96f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.386+0000 7f7e30f98740 1 --2- 172.21.15.28:0/489068170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc63350 0x56502cc969d0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.386+0000 7f7e30f98740 1 --2- 172.21.15.28:0/489068170 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc62a30 0x56502cc8a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.386+0000 7f7e30f98740 1 -- 172.21.15.28:0/489068170 >> 172.21.15.28:0/489068170 conn(0x56502cc5ed50 msgr2=0x56502cc641c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.386+0000 7f7e30f98740 1 -- 172.21.15.28:0/489068170 shutdown_connections 2024-04-24T08:14:23.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.386+0000 7f7e30f98740 1 -- 172.21.15.28:0/489068170 wait complete. 2024-04-24T08:14:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.387+0000 7f7e30f98740 1 -- start start 2024-04-24T08:14:23.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.387+0000 7f7e30f98740 1 Processor -- start 2024-04-24T08:14:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.389+0000 7f7e30f98740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc62a30 0x56502cc827d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.389+0000 7f7e30f98740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56502cc63350 0x56502cc82d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.389+0000 7f7e27400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc62a30 0x56502cc827d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:23.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.389+0000 7f7e27400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc62a30 0x56502cc827d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44514/0 (socket says 172.21.15.28:44514) 2024-04-24T08:14:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.390+0000 7f7e27400640 1 -- 172.21.15.28:0/741560672 learned_addr learned my addr 172.21.15.28:0/741560672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:14:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.390+0000 7f7e30f98740 1 --2- 172.21.15.28:0/741560672 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc6da20 0x56502cc83250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.390+0000 7f7e30f98740 1 -- 172.21.15.28:0/741560672 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7e28072310 con 0x56502cc63350 2024-04-24T08:14:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.391+0000 7f7e30f98740 1 -- 172.21.15.28:0/741560672 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7e28073780 con 0x56502cc6da20 2024-04-24T08:14:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.391+0000 7f7e26a00640 1 --2- 172.21.15.28:0/741560672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56502cc63350 0x56502cc82d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:23.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.391+0000 7f7e27e00640 1 --2- 172.21.15.28:0/741560672 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc6da20 0x56502cc83250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:23.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.391+0000 7f7e30f98740 1 -- 172.21.15.28:0/741560672 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7e280729e0 con 0x56502cc62a30 2024-04-24T08:14:23.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.391+0000 7f7e30f98740 10 client.0 _ll_register_callbacks cb 0x7f7e20001af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:14:23.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.392+0000 7f7e27400640 1 -- 172.21.15.28:0/741560672 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc6da20 msgr2=0x56502cc83250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:23.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.392+0000 7f7e27400640 1 --2- 172.21.15.28:0/741560672 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc6da20 0x56502cc83250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.392+0000 7f7e27400640 1 -- 172.21.15.28:0/741560672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56502cc63350 msgr2=0x56502cc82d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:14:23.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.392+0000 7f7e27400640 1 --2- 172.21.15.28:0/741560672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56502cc63350 0x56502cc82d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.392+0000 7f7e27400640 1 -- 172.21.15.28:0/741560672 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56502c89a0f0 con 0x56502cc62a30 2024-04-24T08:14:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.393+0000 7f7e27e00640 1 --2- 172.21.15.28:0/741560672 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc6da20 0x56502cc83250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:14:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.393+0000 7f7e10c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:14:23.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.393+0000 7f7e27400640 1 --2- 172.21.15.28:0/741560672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc62a30 0x56502cc827d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7e180ce080 tx=0x7f7e18000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c15273dd1325ff8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:14:23.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.394+0000 7f7e10c00640 1 -- 172.21.15.28:0/741560672 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e180a5490 con 0x56502cc62a30 2024-04-24T08:14:23.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.394+0000 7f7e10c00640 1 -- 172.21.15.28:0/741560672 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f7e1800fb30 con 0x56502cc62a30 2024-04-24T08:14:23.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.394+0000 7f7e30f98740 1 -- 172.21.15.28:0/741560672 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e14031a90 con 0x56502cc62a30 2024-04-24T08:14:23.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.396+0000 7f7e10c00640 1 -- 172.21.15.28:0/741560672 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f7e18018d20 con 0x56502cc62a30 2024-04-24T08:14:23.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.396+0000 7f7e30f98740 10 client.26134 fetch_fsmap learned FSMap version 25 2024-04-24T08:14:23.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.396+0000 7f7e30f98740 1 -- 172.21.15.28:0/741560672 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56502cb8d810 con 0x56502cc62a30 2024-04-24T08:14:23.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.397+0000 7f7e10c00640 1 -- 172.21.15.28:0/741560672 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e14031a90 con 0x56502cc62a30 2024-04-24T08:14:23.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.397+0000 7f7e10c00640 1 -- 172.21.15.28:0/741560672 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 25) ==== 8342+0+0 (secure 0 0 0) 0x7f7e18020020 con 0x56502cc62a30 2024-04-24T08:14:23.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.397+0000 7f7e30f98740 10 client.26134 fetch_fsmap finished waiting for FSMap version 25 2024-04-24T08:14:23.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.397+0000 7f7e30f98740 10 client.26134 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:14:23.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.397+0000 7f7e30f98740 20 client.26134 populate_metadata read hostname 'smithi028' 2024-04-24T08:14:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.397+0000 7f7e30f98740 1 --2- 172.21.15.28:0/741560672 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56502ccf5a30 0x56502ccf7e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:14:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.397+0000 7f7e30f98740 4 client.26134 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:14:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.397+0000 7f7e30f98740 1 -- 172.21.15.28:0/741560672 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x56502cbcfa80 con 0x56502ccf5a30 2024-04-24T08:14:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.398+0000 7f7e26a00640 1 --2- 172.21.15.28:0/741560672 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56502ccf5a30 0x56502ccf7e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:14:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.399+0000 7f7e26a00640 1 --2- 172.21.15.28:0/741560672 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56502ccf5a30 0x56502ccf7e20 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2f55c8629490bac server_cookie=3a4d575cee1096fc in_seq=0 out_seq=0 2024-04-24T08:14:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.399+0000 7f7e10c00640 10 client.26134 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:14:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.400+0000 7f7e10c00640 1 -- 172.21.15.28:0/741560672 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x56502cbcfa80 con 0x56502ccf5a30 2024-04-24T08:14:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.400+0000 7f7e10c00640 10 client.26134 handle_command_reply: tid=0 2024-04-24T08:14:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.400+0000 7f7e30f98740 1 client.26134 shutdown 2024-04-24T08:14:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.400+0000 7f7e30f98740 1 -- 172.21.15.28:0/741560672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc62a30 msgr2=0x56502cc827d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.400+0000 7f7e30f98740 1 --2- 172.21.15.28:0/741560672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc62a30 0x56502cc827d0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7e180ce080 tx=0x7f7e18000bc0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.401+0000 7f7e30f98740 1 -- 172.21.15.28:0/741560672 shutdown_connections 2024-04-24T08:14:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.401+0000 7f7e30f98740 1 --2- 172.21.15.28:0/741560672 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x56502ccf5a30 0x56502ccf7e20 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.401+0000 7f7e30f98740 1 --2- 172.21.15.28:0/741560672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56502cc63350 0x56502cc82d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.401+0000 7f7e30f98740 1 --2- 172.21.15.28:0/741560672 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x56502cc6da20 0x56502cc83250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.401+0000 7f7e30f98740 1 --2- 172.21.15.28:0/741560672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56502cc62a30 0x56502cc827d0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.401+0000 7f7e30f98740 1 -- 172.21.15.28:0/741560672 >> 172.21.15.28:0/741560672 conn(0x56502cc5ed50 msgr2=0x56502cc64a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.401+0000 7f7e10c00640 0 client.26134 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:14:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.401+0000 7f7e30f98740 1 -- 172.21.15.28:0/741560672 shutdown_connections 2024-04-24T08:14:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.402+0000 7f7e30f98740 1 -- 172.21.15.28:0/741560672 wait complete. 2024-04-24T08:14:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.402+0000 7f7e30f98740 20 client.26134 trim_cache size 0 max 16384 2024-04-24T08:14:23.405 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:14:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (171 inodes in the stack)", 2024-04-24T08:14:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:14:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "ae51db93-2230-4cd0-a1aa-d9c0bee08435": { 2024-04-24T08:14:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:14:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "ae51db93-2230-4cd0-a1aa-d9c0bee08435", 2024-04-24T08:14:23.405 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:14:23.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:14:23.406 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:14:23.406 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:14:23.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.404+0000 7f7e2ca00640 1 -- 172.21.15.28:0/1157913961 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7e0c0a3d20 msgr2=0x7f7e0c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:23.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.404+0000 7f7e2ca00640 1 --2- 172.21.15.28:0/1157913961 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7e0c0a3d20 0x7f7e0c0a61e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7e20001d90 tx=0x7f7e20001500 comp rx=0 tx=0).stop 2024-04-24T08:14:23.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.404+0000 7f7e2ca00640 1 -- 172.21.15.28:0/1157913961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e28104320 msgr2=0x7f7e28110ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:14:23.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.404+0000 7f7e2ca00640 1 --2- 172.21.15.28:0/1157913961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e28104320 0x7f7e28110ce0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f7e1800c920 tx=0x7f7e1800cde0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.405+0000 7f7e2ca00640 1 -- 172.21.15.28:0/1157913961 shutdown_connections 2024-04-24T08:14:23.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.405+0000 7f7e2ca00640 1 --2- 172.21.15.28:0/1157913961 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7e0c0a3d20 0x7f7e0c0a61e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.405+0000 7f7e2ca00640 1 --2- 172.21.15.28:0/1157913961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7e2810f370 0x7f7e2810f7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.405+0000 7f7e2ca00640 1 --2- 172.21.15.28:0/1157913961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e281093f0 0x7f7e2810ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.408+0000 7f7e2ca00640 1 --2- 172.21.15.28:0/1157913961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7e28104320 0x7f7e28110ce0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:14:23.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.408+0000 7f7e2ca00640 1 -- 172.21.15.28:0/1157913961 >> 172.21.15.28:0/1157913961 conn(0x7f7e280ffb40 msgr2=0x7f7e28100760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:14:23.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.408+0000 7f7e2ca00640 1 -- 172.21.15.28:0/1157913961 shutdown_connections 2024-04-24T08:14:23.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:14:23.408+0000 7f7e2ca00640 1 -- 172.21.15.28:0/1157913961 wait complete. 2024-04-24T08:14:23.447 INFO:tasks.cephfs.filesystem:scrub status for tag:ae51db93-2230-4cd0-a1aa-d9c0bee08435 - {'path': '/', 'tag': 'ae51db93-2230-4cd0-a1aa-d9c0bee08435', 'options': 'recursive,force'} 2024-04-24T08:14:23.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:23 smithi028 ceph-mon[27786]: pgmap v124: 97 pgs: 97 active+clean; 15 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 156 MiB/s wr, 19.27k op/s 2024-04-24T08:14:23.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:23 smithi117 ceph-mon[32438]: pgmap v124: 97 pgs: 97 active+clean; 15 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 156 MiB/s wr, 19.27k op/s 2024-04-24T08:14:23.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:23 smithi194 ceph-mon[32836]: pgmap v124: 97 pgs: 97 active+clean; 15 GiB data, 13 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 156 MiB/s wr, 19.27k op/s 2024-04-24T08:14:25.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:25 smithi028 ceph-mon[27786]: pgmap v125: 97 pgs: 97 active+clean; 15 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 116 MiB/s wr, 15.18k op/s 2024-04-24T08:14:25.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:25 smithi117 ceph-mon[32438]: pgmap v125: 97 pgs: 97 active+clean; 15 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 116 MiB/s wr, 15.18k op/s 2024-04-24T08:14:25.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:25 smithi194 ceph-mon[32836]: pgmap v125: 97 pgs: 97 active+clean; 15 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 116 MiB/s wr, 15.18k op/s 2024-04-24T08:14:27.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:27 smithi117 ceph-mon[32438]: pgmap v126: 97 pgs: 97 active+clean; 15 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 139 MiB/s wr, 18.47k op/s 2024-04-24T08:14:27.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:27 smithi194 ceph-mon[32836]: pgmap v126: 97 pgs: 97 active+clean; 15 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 139 MiB/s wr, 18.47k op/s 2024-04-24T08:14:28.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:27 smithi028 ceph-mon[27786]: pgmap v126: 97 pgs: 97 active+clean; 15 GiB data, 14 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 139 MiB/s wr, 18.47k op/s 2024-04-24T08:14:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:29 smithi117 ceph-mon[32438]: pgmap v127: 97 pgs: 97 active+clean; 16 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 134 MiB/s wr, 18.42k op/s 2024-04-24T08:14:29.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:29 smithi194 ceph-mon[32836]: pgmap v127: 97 pgs: 97 active+clean; 16 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 134 MiB/s wr, 18.42k op/s 2024-04-24T08:14:30.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:29 smithi028 ceph-mon[27786]: pgmap v127: 97 pgs: 97 active+clean; 16 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 134 MiB/s wr, 18.42k op/s 2024-04-24T08:14:31.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:31 smithi117 ceph-mon[32438]: pgmap v128: 97 pgs: 97 active+clean; 16 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 116 MiB/s wr, 15.99k op/s 2024-04-24T08:14:31.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:31 smithi194 ceph-mon[32836]: pgmap v128: 97 pgs: 97 active+clean; 16 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 116 MiB/s wr, 15.99k op/s 2024-04-24T08:14:32.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:31 smithi028 ceph-mon[27786]: pgmap v128: 97 pgs: 97 active+clean; 16 GiB data, 15 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 116 MiB/s wr, 15.99k op/s 2024-04-24T08:14:33.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:33 smithi117 ceph-mon[32438]: pgmap v129: 97 pgs: 97 active+clean; 16 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 137 MiB/s wr, 19.33k op/s 2024-04-24T08:14:33.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:33 smithi194 ceph-mon[32836]: pgmap v129: 97 pgs: 97 active+clean; 16 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 137 MiB/s wr, 19.33k op/s 2024-04-24T08:14:34.042 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:33 smithi028 ceph-mon[27786]: pgmap v129: 97 pgs: 97 active+clean; 16 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 96 KiB/s rd, 137 MiB/s wr, 19.33k op/s 2024-04-24T08:14:34.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:34 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:14:34.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:34 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:14:34.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:34 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:14:34.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:34 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:14:34.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:34 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:14:34.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:34 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:14:34.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:34 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:14:34.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:34 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:14:35.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:34 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:14:35.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:34 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:14:35.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:34 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:14:35.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:34 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:14:35.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:35 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:14:35.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:35 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:14:35.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:35 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:14:35.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:35 smithi117 ceph-mon[32438]: pgmap v130: 97 pgs: 97 active+clean; 16 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 98 MiB/s wr, 14.19k op/s 2024-04-24T08:14:35.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:35 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:35.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:35 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:14:35.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:35 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:14:35.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:35 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:14:35.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:35 smithi194 ceph-mon[32836]: pgmap v130: 97 pgs: 97 active+clean; 16 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 98 MiB/s wr, 14.19k op/s 2024-04-24T08:14:35.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:35 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:36.043 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:35 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:14:36.043 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:35 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:14:36.043 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:35 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:14:36.043 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:35 smithi028 ceph-mon[27786]: pgmap v130: 97 pgs: 97 active+clean; 16 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 39 KiB/s rd, 98 MiB/s wr, 14.19k op/s 2024-04-24T08:14:36.043 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:35 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:14:37.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:37 smithi117 ceph-mon[32438]: pgmap v131: 97 pgs: 97 active+clean; 16 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 93 MiB/s wr, 13.51k op/s 2024-04-24T08:14:37.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:37 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:37.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:37 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:37.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:37 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:14:37.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:37 smithi194 ceph-mon[32836]: pgmap v131: 97 pgs: 97 active+clean; 16 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 93 MiB/s wr, 13.51k op/s 2024-04-24T08:14:37.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:37 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:37.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:37 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:37.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:37 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:14:38.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:37 smithi028 ceph-mon[27786]: pgmap v131: 97 pgs: 97 active+clean; 16 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 93 MiB/s wr, 13.51k op/s 2024-04-24T08:14:38.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:37 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:14:38.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:37 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:14:38.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:37 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:14:39.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:39 smithi117 ceph-mon[32438]: pgmap v132: 97 pgs: 97 active+clean; 16 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 MiB/s wr, 11.98k op/s 2024-04-24T08:14:39.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:39 smithi194 ceph-mon[32836]: pgmap v132: 97 pgs: 97 active+clean; 16 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 MiB/s wr, 11.98k op/s 2024-04-24T08:14:40.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:39 smithi028 ceph-mon[27786]: pgmap v132: 97 pgs: 97 active+clean; 16 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 76 MiB/s wr, 11.98k op/s 2024-04-24T08:14:40.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:40 smithi117 ceph-mon[32438]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:14:40.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:40 smithi194 ceph-mon[32836]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:14:41.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:40 smithi028 ceph-mon[27786]: fsmap cephfs:3 {0=g=up:active,1=b=up:active,2=i=up:active} 3 up:standby-replay 6 up:standby 2024-04-24T08:14:41.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:41 smithi117 ceph-mon[32438]: pgmap v133: 97 pgs: 97 active+clean; 16 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 44 MiB/s wr, 7.52k op/s 2024-04-24T08:14:41.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:41 smithi194 ceph-mon[32836]: pgmap v133: 97 pgs: 97 active+clean; 16 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 44 MiB/s wr, 7.52k op/s 2024-04-24T08:14:41.949 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:41 smithi028 ceph-mon[27786]: pgmap v133: 97 pgs: 97 active+clean; 16 GiB data, 16 GiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 44 MiB/s wr, 7.52k op/s 2024-04-24T08:14:43.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:43 smithi117 ceph-mon[32438]: pgmap v134: 97 pgs: 97 active+clean; 16 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 62 MiB/s wr, 10.75k op/s 2024-04-24T08:14:43.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:43 smithi194 ceph-mon[32836]: pgmap v134: 97 pgs: 97 active+clean; 16 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 62 MiB/s wr, 10.75k op/s 2024-04-24T08:14:44.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:43 smithi028 ceph-mon[27786]: pgmap v134: 97 pgs: 97 active+clean; 16 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 62 KiB/s rd, 62 MiB/s wr, 10.75k op/s 2024-04-24T08:14:44.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:44 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:14:44.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:44 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:14:45.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:44 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:14:45.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:44 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:14:45.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:44 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:14:45.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:44 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:14:45.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:45 smithi194 ceph-mon[32836]: pgmap v135: 97 pgs: 97 active+clean; 16 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 53 MiB/s wr, 9.40k op/s 2024-04-24T08:14:46.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:45 smithi028 ceph-mon[27786]: pgmap v135: 97 pgs: 97 active+clean; 16 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 53 MiB/s wr, 9.40k op/s 2024-04-24T08:14:46.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:45 smithi117 ceph-mon[32438]: pgmap v135: 97 pgs: 97 active+clean; 16 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 53 MiB/s wr, 9.40k op/s 2024-04-24T08:14:47.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:47 smithi194 ceph-mon[32836]: pgmap v136: 97 pgs: 97 active+clean; 16 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 54 MiB/s wr, 9.51k op/s 2024-04-24T08:14:48.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:47 smithi028 ceph-mon[27786]: pgmap v136: 97 pgs: 97 active+clean; 16 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 54 MiB/s wr, 9.51k op/s 2024-04-24T08:14:48.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:47 smithi117 ceph-mon[32438]: pgmap v136: 97 pgs: 97 active+clean; 16 GiB data, 17 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 54 MiB/s wr, 9.51k op/s 2024-04-24T08:14:49.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:49 smithi194 ceph-mon[32836]: pgmap v137: 97 pgs: 97 active+clean; 16 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 79 MiB/s wr, 14.10k op/s 2024-04-24T08:14:50.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:49 smithi028 ceph-mon[27786]: pgmap v137: 97 pgs: 97 active+clean; 16 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 79 MiB/s wr, 14.10k op/s 2024-04-24T08:14:50.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:49 smithi117 ceph-mon[32438]: pgmap v137: 97 pgs: 97 active+clean; 16 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 79 MiB/s wr, 14.10k op/s 2024-04-24T08:14:51.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:51 smithi194 ceph-mon[32836]: pgmap v138: 97 pgs: 97 active+clean; 16 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 73 MiB/s wr, 12.71k op/s 2024-04-24T08:14:52.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:51 smithi028 ceph-mon[27786]: pgmap v138: 97 pgs: 97 active+clean; 16 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 73 MiB/s wr, 12.71k op/s 2024-04-24T08:14:52.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:51 smithi117 ceph-mon[32438]: pgmap v138: 97 pgs: 97 active+clean; 16 GiB data, 19 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 73 MiB/s wr, 12.71k op/s 2024-04-24T08:14:53.449 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:14:53.449 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:14:53.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:53 smithi194 ceph-mon[32836]: pgmap v139: 97 pgs: 97 active+clean; 17 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 104 MiB/s wr, 18.61k op/s 2024-04-24T08:14:54.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:53 smithi028 ceph-mon[27786]: pgmap v139: 97 pgs: 97 active+clean; 17 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 104 MiB/s wr, 18.61k op/s 2024-04-24T08:14:54.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:53 smithi117 ceph-mon[32438]: pgmap v139: 97 pgs: 97 active+clean; 17 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 104 MiB/s wr, 18.61k op/s 2024-04-24T08:14:55.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:55 smithi194 ceph-mon[32836]: pgmap v140: 97 pgs: 97 active+clean; 17 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 MiB/s wr, 14.96k op/s 2024-04-24T08:14:56.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:55 smithi028 ceph-mon[27786]: pgmap v140: 97 pgs: 97 active+clean; 17 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 MiB/s wr, 14.96k op/s 2024-04-24T08:14:56.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:55 smithi117 ceph-mon[32438]: pgmap v140: 97 pgs: 97 active+clean; 17 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 83 MiB/s wr, 14.96k op/s 2024-04-24T08:14:57.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:57 smithi194 ceph-mon[32836]: pgmap v141: 97 pgs: 97 active+clean; 17 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 76 MiB/s wr, 14.10k op/s 2024-04-24T08:14:58.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:57 smithi117 ceph-mon[32438]: pgmap v141: 97 pgs: 97 active+clean; 17 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 76 MiB/s wr, 14.10k op/s 2024-04-24T08:14:58.230 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:57 smithi028 ceph-mon[27786]: pgmap v141: 97 pgs: 97 active+clean; 17 GiB data, 20 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 76 MiB/s wr, 14.10k op/s 2024-04-24T08:15:00.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:14:59 smithi117 ceph-mon[32438]: pgmap v142: 97 pgs: 97 active+clean; 17 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 86 MiB/s wr, 15.97k op/s 2024-04-24T08:15:00.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:14:59 smithi194 ceph-mon[32836]: pgmap v142: 97 pgs: 97 active+clean; 17 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 86 MiB/s wr, 15.97k op/s 2024-04-24T08:15:00.979 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:14:59 smithi028 ceph-mon[27786]: pgmap v142: 97 pgs: 97 active+clean; 17 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 86 MiB/s wr, 15.97k op/s 2024-04-24T08:15:01.991 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:01 smithi194 ceph-mon[32836]: pgmap v143: 97 pgs: 97 active+clean; 17 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 61 MiB/s wr, 11.51k op/s 2024-04-24T08:15:02.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:01 smithi028 ceph-mon[27786]: pgmap v143: 97 pgs: 97 active+clean; 17 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 61 MiB/s wr, 11.51k op/s 2024-04-24T08:15:02.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:01 smithi117 ceph-mon[32438]: pgmap v143: 97 pgs: 97 active+clean; 17 GiB data, 21 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 61 MiB/s wr, 11.51k op/s 2024-04-24T08:15:04.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:03 smithi028 ceph-mon[27786]: pgmap v144: 97 pgs: 97 active+clean; 17 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 81 MiB/s wr, 14.97k op/s 2024-04-24T08:15:04.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:03 smithi117 ceph-mon[32438]: pgmap v144: 97 pgs: 97 active+clean; 17 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 81 MiB/s wr, 14.97k op/s 2024-04-24T08:15:04.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:03 smithi194 ceph-mon[32836]: pgmap v144: 97 pgs: 97 active+clean; 17 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 104 KiB/s rd, 81 MiB/s wr, 14.97k op/s 2024-04-24T08:15:06.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:05 smithi028 ceph-mon[27786]: pgmap v145: 97 pgs: 97 active+clean; 17 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 48 MiB/s wr, 8.76k op/s 2024-04-24T08:15:06.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:05 smithi117 ceph-mon[32438]: pgmap v145: 97 pgs: 97 active+clean; 17 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 48 MiB/s wr, 8.76k op/s 2024-04-24T08:15:06.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:05 smithi194 ceph-mon[32836]: pgmap v145: 97 pgs: 97 active+clean; 17 GiB data, 22 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 48 MiB/s wr, 8.76k op/s 2024-04-24T08:15:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.097+0000 7ff933600640 1 -- 172.21.15.28:0/1954757998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff92c076e50 msgr2=0x7ff92c1129a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.097+0000 7ff933600640 1 --2- 172.21.15.28:0/1954757998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff92c076e50 0x7ff92c1129a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7ff918009890 tx=0x7ff91802a390 comp rx=0 tx=0).stop 2024-04-24T08:15:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.097+0000 7ff933600640 1 -- 172.21.15.28:0/1954757998 shutdown_connections 2024-04-24T08:15:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.097+0000 7ff933600640 1 --2- 172.21.15.28:0/1954757998 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff92c112ee0 0x7ff92c1152d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.097+0000 7ff933600640 1 --2- 172.21.15.28:0/1954757998 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff92c076e50 0x7ff92c1129a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.097+0000 7ff933600640 1 --2- 172.21.15.28:0/1954757998 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff92c076530 0x7ff92c076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.097+0000 7ff933600640 1 -- 172.21.15.28:0/1954757998 >> 172.21.15.28:0/1954757998 conn(0x7ff92c0ffd90 msgr2=0x7ff92c1021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.098+0000 7ff933600640 1 -- 172.21.15.28:0/1954757998 shutdown_connections 2024-04-24T08:15:07.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.098+0000 7ff933600640 1 -- 172.21.15.28:0/1954757998 wait complete. 2024-04-24T08:15:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.099+0000 7ff933600640 1 Processor -- start 2024-04-24T08:15:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.099+0000 7ff933600640 1 -- start start 2024-04-24T08:15:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.100+0000 7ff933600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff92c076530 0x7ff92c073a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.100+0000 7ff933600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff92c076e50 0x7ff92c06db80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.100+0000 7ff933600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff92c112ee0 0x7ff92c06e0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.100+0000 7ff933600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c06c430 con 0x7ff92c112ee0 2024-04-24T08:15:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.100+0000 7ff933600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c06c2b0 con 0x7ff92c076530 2024-04-24T08:15:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.100+0000 7ff933600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff92c06c5b0 con 0x7ff92c076e50 2024-04-24T08:15:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.100+0000 7ff92be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff92c076e50 0x7ff92c06db80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.100+0000 7ff930c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff92c076530 0x7ff92c073a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.100+0000 7ff930c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff92c076530 0x7ff92c073a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:50778/0 (socket says 172.21.15.28:50778) 2024-04-24T08:15:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.100+0000 7ff92be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff92c076e50 0x7ff92c06db80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:58008/0 (socket says 172.21.15.28:58008) 2024-04-24T08:15:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.100+0000 7ff930c00640 1 -- 172.21.15.28:0/817971094 learned_addr learned my addr 172.21.15.28:0/817971094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.101+0000 7ff930c00640 1 -- 172.21.15.28:0/817971094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff92c076e50 msgr2=0x7ff92c06db80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.101+0000 7ff930c00640 1 --2- 172.21.15.28:0/817971094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff92c076e50 0x7ff92c06db80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.101+0000 7ff930c00640 1 -- 172.21.15.28:0/817971094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff92c112ee0 msgr2=0x7ff92c06e0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.101+0000 7ff930c00640 1 --2- 172.21.15.28:0/817971094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff92c112ee0 0x7ff92c06e0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.101+0000 7ff930c00640 1 -- 172.21.15.28:0/817971094 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff92c06e930 con 0x7ff92c076530 2024-04-24T08:15:07.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.101+0000 7ff92be00640 1 --2- 172.21.15.28:0/817971094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff92c076e50 0x7ff92c06db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:15:07.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.101+0000 7ff930c00640 1 --2- 172.21.15.28:0/817971094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff92c076530 0x7ff92c073a70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff92000efb0 tx=0x7ff92000c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:07.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.101+0000 7ff929600640 1 -- 172.21.15.28:0/817971094 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92001b070 con 0x7ff92c076530 2024-04-24T08:15:07.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.101+0000 7ff933600640 1 -- 172.21.15.28:0/817971094 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff92c1b3de0 con 0x7ff92c076530 2024-04-24T08:15:07.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.102+0000 7ff933600640 1 -- 172.21.15.28:0/817971094 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff92c1b4250 con 0x7ff92c076530 2024-04-24T08:15:07.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.102+0000 7ff929600640 1 -- 172.21.15.28:0/817971094 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff92000ba50 con 0x7ff92c076530 2024-04-24T08:15:07.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.102+0000 7ff929600640 1 -- 172.21.15.28:0/817971094 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff920016c70 con 0x7ff92c076530 2024-04-24T08:15:07.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.106+0000 7ff929600640 1 -- 172.21.15.28:0/817971094 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff92001e070 con 0x7ff92c076530 2024-04-24T08:15:07.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.108+0000 7ff929600640 1 --2- 172.21.15.28:0/817971094 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff9140a3da0 0x7ff9140a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.109+0000 7ff929600640 1 -- 172.21.15.28:0/817971094 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7ff92009b790 con 0x7ff92c076530 2024-04-24T08:15:07.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.109+0000 7ff92be00640 1 --2- 172.21.15.28:0/817971094 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff9140a3da0 0x7ff9140a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.149+0000 7ff92be00640 1 --2- 172.21.15.28:0/817971094 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff9140a3da0 0x7ff9140a6260 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff91802a980 tx=0x7ff918005d70 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:07.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.149+0000 7ff936419740 1 Processor -- start 2024-04-24T08:15:07.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.150+0000 7ff936419740 1 -- start start 2024-04-24T08:15:07.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.151+0000 7ff936419740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 0x5614c462fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.151+0000 7ff936419740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c4630180 0x5614c463a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.151+0000 7ff930c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 0x5614c462fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.151+0000 7ff930c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 0x5614c462fc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:46122/0 (socket says 172.21.15.28:46122) 2024-04-24T08:15:07.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.151+0000 7ff930c00640 1 -- 172.21.15.28:0/3431024500 learned_addr learned my addr 172.21.15.28:0/3431024500 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:07.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.151+0000 7ff92be00640 1 --2- 172.21.15.28:0/3431024500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c4630180 0x5614c463a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.152+0000 7ff936419740 1 --2- 172.21.15.28:0/3431024500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c463a850 0x5614c463cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.152+0000 7ff936419740 1 -- 172.21.15.28:0/3431024500 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5614c45951f0 con 0x5614c462f860 2024-04-24T08:15:07.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.152+0000 7ff936419740 1 -- 172.21.15.28:0/3431024500 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5614c453b350 con 0x5614c4630180 2024-04-24T08:15:07.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.152+0000 7ff936419740 1 -- 172.21.15.28:0/3431024500 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5614c45911b0 con 0x5614c463a850 2024-04-24T08:15:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.153+0000 7ff92be00640 1 -- 172.21.15.28:0/3431024500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c463a850 msgr2=0x5614c463cc40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:15:07.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.153+0000 7ff931600640 1 --2- 172.21.15.28:0/3431024500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c463a850 0x5614c463cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.153+0000 7ff92be00640 1 --2- 172.21.15.28:0/3431024500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c463a850 0x5614c463cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.153+0000 7ff92be00640 1 -- 172.21.15.28:0/3431024500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 msgr2=0x5614c462fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.153+0000 7ff92be00640 1 --2- 172.21.15.28:0/3431024500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 0x5614c462fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.153+0000 7ff92be00640 1 -- 172.21.15.28:0/3431024500 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614c458ff50 con 0x5614c4630180 2024-04-24T08:15:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.153+0000 7ff930c00640 1 --2- 172.21.15.28:0/3431024500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 0x5614c462fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:15:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.154+0000 7ff92be00640 1 --2- 172.21.15.28:0/3431024500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c4630180 0x5614c463a310 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff918000c00 tx=0x7ff91803c000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2676711cba1ef27d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:07.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.154+0000 7ff91dc00640 1 -- 172.21.15.28:0/3431024500 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff918042070 con 0x5614c4630180 2024-04-24T08:15:07.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.155+0000 7ff91dc00640 1 -- 172.21.15.28:0/3431024500 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff91804e070 con 0x5614c4630180 2024-04-24T08:15:07.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.156+0000 7ff936419740 1 -- 172.21.15.28:0/3431024500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c4630180 msgr2=0x5614c463a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.156+0000 7ff936419740 1 --2- 172.21.15.28:0/3431024500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c4630180 0x5614c463a310 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7ff918000c00 tx=0x7ff91803c000 comp rx=0 tx=0).stop 2024-04-24T08:15:07.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.156+0000 7ff936419740 1 -- 172.21.15.28:0/3431024500 shutdown_connections 2024-04-24T08:15:07.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.156+0000 7ff936419740 1 --2- 172.21.15.28:0/3431024500 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c463a850 0x5614c463cc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.156+0000 7ff936419740 1 --2- 172.21.15.28:0/3431024500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c4630180 0x5614c463a310 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.156+0000 7ff936419740 1 --2- 172.21.15.28:0/3431024500 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 0x5614c462fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.157+0000 7ff936419740 1 -- 172.21.15.28:0/3431024500 >> 172.21.15.28:0/3431024500 conn(0x5614c462bb80 msgr2=0x5614c462bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.157+0000 7ff936419740 1 -- 172.21.15.28:0/3431024500 shutdown_connections 2024-04-24T08:15:07.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.157+0000 7ff936419740 1 -- 172.21.15.28:0/3431024500 wait complete. 2024-04-24T08:15:07.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.158+0000 7ff936419740 1 -- start start 2024-04-24T08:15:07.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.159+0000 7ff936419740 1 Processor -- start 2024-04-24T08:15:07.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.160+0000 7ff936419740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 0x5614c46376a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.160+0000 7ff936419740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c4630180 0x5614c4637be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.161+0000 7ff936419740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c463a850 0x5614c4638120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.161+0000 7ff936419740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5614c463d680 con 0x5614c462f860 2024-04-24T08:15:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.161+0000 7ff936419740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5614c454e690 con 0x5614c4630180 2024-04-24T08:15:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.161+0000 7ff936419740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5614c463d800 con 0x5614c463a850 2024-04-24T08:15:07.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.161+0000 7ff936419740 10 client.0 _ll_register_callbacks cb 0x5614c458a650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:07.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.162+0000 7ff92be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c4630180 0x5614c4637be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.162+0000 7ff930c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 0x5614c46376a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.162+0000 7ff931600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c463a850 0x5614c4638120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.162+0000 7ff931600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c463a850 0x5614c4638120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:58032/0 (socket says 172.21.15.28:58032) 2024-04-24T08:15:07.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.162+0000 7ff931600640 1 -- 172.21.15.28:0/2679889620 learned_addr learned my addr 172.21.15.28:0/2679889620 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.163+0000 7ff930c00640 1 -- 172.21.15.28:0/2679889620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c463a850 msgr2=0x5614c4638120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.163+0000 7ff930c00640 1 --2- 172.21.15.28:0/2679889620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c463a850 0x5614c4638120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.163+0000 7ff930c00640 1 -- 172.21.15.28:0/2679889620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c4630180 msgr2=0x5614c4637be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.163+0000 7ff930c00640 1 --2- 172.21.15.28:0/2679889620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c4630180 0x5614c4637be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.164+0000 7ff930c00640 1 -- 172.21.15.28:0/2679889620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614c454d920 con 0x5614c462f860 2024-04-24T08:15:07.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.165+0000 7ff930c00640 1 --2- 172.21.15.28:0/2679889620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 0x5614c46376a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff92000ef80 tx=0x7ff9200a4d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3617323c0169f420 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:07.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.165+0000 7ff912000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:15:07.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.166+0000 7ff912000640 1 -- 172.21.15.28:0/2679889620 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92009a120 con 0x5614c462f860 2024-04-24T08:15:07.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.166+0000 7ff912000640 1 -- 172.21.15.28:0/2679889620 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7ff92009d050 con 0x5614c462f860 2024-04-24T08:15:07.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.166+0000 7ff936419740 1 -- 172.21.15.28:0/2679889620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff91804e070 con 0x5614c462f860 2024-04-24T08:15:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.167+0000 7ff912000640 1 -- 172.21.15.28:0/2679889620 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff92009a2c0 con 0x5614c462f860 2024-04-24T08:15:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.167+0000 7ff936419740 10 client.15924 fetch_fsmap learned FSMap version 26 2024-04-24T08:15:07.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.168+0000 7ff936419740 1 -- 172.21.15.28:0/2679889620 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5614c43717d0 con 0x5614c462f860 2024-04-24T08:15:07.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.168+0000 7ff912000640 1 -- 172.21.15.28:0/2679889620 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff91804e070 con 0x5614c462f860 2024-04-24T08:15:07.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.169+0000 7ff912000640 1 -- 172.21.15.28:0/2679889620 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff92000b040 con 0x5614c462f860 2024-04-24T08:15:07.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.169+0000 7ff936419740 10 client.15924 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:15:07.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.169+0000 7ff936419740 10 client.15924 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:15:07.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.169+0000 7ff936419740 20 client.15924 populate_metadata read hostname 'smithi028' 2024-04-24T08:15:07.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.169+0000 7ff936419740 1 --2- 172.21.15.28:0/2679889620 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5614c464abd0 0x5614c464cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.170+0000 7ff936419740 4 client.15924 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:15:07.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.170+0000 7ff936419740 1 -- 172.21.15.28:0/2679889620 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5614c45a2c10 con 0x5614c464abd0 2024-04-24T08:15:07.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.170+0000 7ff92be00640 1 --2- 172.21.15.28:0/2679889620 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5614c464abd0 0x5614c464cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.172+0000 7ff92be00640 1 --2- 172.21.15.28:0/2679889620 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5614c464abd0 0x5614c464cfc0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45fe2ba9cbd1a9e3 server_cookie=4281e1802ff0f9d1 in_seq=0 out_seq=0 2024-04-24T08:15:07.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.172+0000 7ff912000640 10 client.15924 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.174+0000 7ff912000640 1 -- 172.21.15.28:0/2679889620 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5614c45a2c10 con 0x5614c464abd0 2024-04-24T08:15:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.174+0000 7ff912000640 10 client.15924 handle_command_reply: tid=0 2024-04-24T08:15:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.174+0000 7ff936419740 1 client.15924 shutdown 2024-04-24T08:15:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.175+0000 7ff936419740 1 -- 172.21.15.28:0/2679889620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 msgr2=0x5614c46376a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.175+0000 7ff936419740 1 --2- 172.21.15.28:0/2679889620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 0x5614c46376a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ff92000ef80 tx=0x7ff9200a4d00 comp rx=0 tx=0).stop 2024-04-24T08:15:07.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.175+0000 7ff936419740 1 -- 172.21.15.28:0/2679889620 shutdown_connections 2024-04-24T08:15:07.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.175+0000 7ff936419740 1 --2- 172.21.15.28:0/2679889620 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5614c464abd0 0x5614c464cfc0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.175+0000 7ff936419740 1 --2- 172.21.15.28:0/2679889620 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c463a850 0x5614c4638120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.175+0000 7ff936419740 1 --2- 172.21.15.28:0/2679889620 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c4630180 0x5614c4637be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.175+0000 7ff936419740 1 --2- 172.21.15.28:0/2679889620 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 0x5614c46376a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.175+0000 7ff936419740 1 -- 172.21.15.28:0/2679889620 >> 172.21.15.28:0/2679889620 conn(0x5614c462bb80 msgr2=0x5614c4631880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:07.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.175+0000 7ff912000640 0 client.15924 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:07.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.176+0000 7ff936419740 1 -- 172.21.15.28:0/2679889620 shutdown_connections 2024-04-24T08:15:07.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.177+0000 7ff936419740 1 -- 172.21.15.28:0/2679889620 wait complete. 2024-04-24T08:15:07.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.177+0000 7ff936419740 20 client.15924 trim_cache size 0 max 16384 2024-04-24T08:15:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.216+0000 7ff936419740 1 Processor -- start 2024-04-24T08:15:07.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.216+0000 7ff936419740 1 -- start start 2024-04-24T08:15:07.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.216+0000 7ff936419740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 0x5614c46465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.216+0000 7ff936419740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c4630180 0x5614c4652820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.216+0000 7ff936419740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c463a850 0x5614c4652d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.216+0000 7ff936419740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c011190 con 0x5614c462f860 2024-04-24T08:15:07.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.216+0000 7ff936419740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c00ea60 con 0x5614c463a850 2024-04-24T08:15:07.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.216+0000 7ff936419740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c00c8b0 con 0x5614c4630180 2024-04-24T08:15:07.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.217+0000 7ff92be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c4630180 0x5614c4652820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.217+0000 7ff92be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c4630180 0x5614c4652820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:58048/0 (socket says 172.21.15.28:58048) 2024-04-24T08:15:07.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.217+0000 7ff92be00640 1 -- 172.21.15.28:0/1026471212 learned_addr learned my addr 172.21.15.28:0/1026471212 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:07.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.217+0000 7ff930c00640 1 --2- 172.21.15.28:0/1026471212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 0x5614c46465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.217+0000 7ff931600640 1 --2- 172.21.15.28:0/1026471212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c463a850 0x5614c4652d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.217+0000 7ff931600640 1 -- 172.21.15.28:0/1026471212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c4630180 msgr2=0x5614c4652820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:07.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.217+0000 7ff931600640 1 --2- 172.21.15.28:0/1026471212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c4630180 0x5614c4652820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.217+0000 7ff931600640 1 -- 172.21.15.28:0/1026471212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 msgr2=0x5614c46465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:07.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.217+0000 7ff931600640 1 --2- 172.21.15.28:0/1026471212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 0x5614c46465f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.217+0000 7ff931600640 1 -- 172.21.15.28:0/1026471212 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5614c42a73f0 con 0x5614c463a850 2024-04-24T08:15:07.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.218+0000 7ff930c00640 1 --2- 172.21.15.28:0/1026471212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 0x5614c46465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:15:07.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.218+0000 7ff931600640 1 --2- 172.21.15.28:0/1026471212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c463a850 0x5614c4652d60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff924004650 tx=0x7ff924009da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b6bf7a77cf079755 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:07.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.219+0000 7ff91d200640 1 -- 172.21.15.28:0/1026471212 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff924011c80 con 0x5614c463a850 2024-04-24T08:15:07.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.219+0000 7ff91d200640 1 -- 172.21.15.28:0/1026471212 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff924011e20 con 0x5614c463a850 2024-04-24T08:15:07.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.219+0000 7ff91d200640 1 -- 172.21.15.28:0/1026471212 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92401a8f0 con 0x5614c463a850 2024-04-24T08:15:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.222+0000 7ff936419740 1 -- 172.21.15.28:0/1026471212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c463a850 msgr2=0x5614c4652d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.222+0000 7ff936419740 1 --2- 172.21.15.28:0/1026471212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c463a850 0x5614c4652d60 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7ff924004650 tx=0x7ff924009da0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.222+0000 7ff936419740 1 -- 172.21.15.28:0/1026471212 shutdown_connections 2024-04-24T08:15:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.222+0000 7ff936419740 1 --2- 172.21.15.28:0/1026471212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c463a850 0x5614c4652d60 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.222+0000 7ff936419740 1 --2- 172.21.15.28:0/1026471212 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c4630180 0x5614c4652820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.222+0000 7ff936419740 1 --2- 172.21.15.28:0/1026471212 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c462f860 0x5614c46465f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.222+0000 7ff936419740 1 -- 172.21.15.28:0/1026471212 >> 172.21.15.28:0/1026471212 conn(0x5614c462bb80 msgr2=0x5614c4632990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.222+0000 7ff936419740 1 -- 172.21.15.28:0/1026471212 shutdown_connections 2024-04-24T08:15:07.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.222+0000 7ff936419740 1 -- 172.21.15.28:0/1026471212 wait complete. 2024-04-24T08:15:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.223+0000 7ff936419740 1 -- start start 2024-04-24T08:15:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.223+0000 7ff936419740 1 Processor -- start 2024-04-24T08:15:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.224+0000 7ff936419740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c462f860 0x5614c46338f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.224+0000 7ff936419740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c4630180 0x5614c4633e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.224+0000 7ff936419740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c463a850 0x5614c4634370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.224+0000 7ff936419740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c00cf20 con 0x5614c4630180 2024-04-24T08:15:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.224+0000 7ff936419740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c00e390 con 0x5614c463a850 2024-04-24T08:15:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.224+0000 7ff936419740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff90c00d5f0 con 0x5614c462f860 2024-04-24T08:15:07.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.224+0000 7ff936419740 10 client.0 _ll_register_callbacks cb 0x7ff920019070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:07.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.226+0000 7ff930c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c462f860 0x5614c46338f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.226+0000 7ff931600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c463a850 0x5614c4634370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.226+0000 7ff931600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c463a850 0x5614c4634370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:50816/0 (socket says 172.21.15.28:50816) 2024-04-24T08:15:07.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.226+0000 7ff931600640 1 -- 172.21.15.28:0/2387101886 learned_addr learned my addr 172.21.15.28:0/2387101886 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:07.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.227+0000 7ff92be00640 1 --2- 172.21.15.28:0/2387101886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c4630180 0x5614c4633e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.228+0000 7ff930c00640 1 -- 172.21.15.28:0/2387101886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c463a850 msgr2=0x5614c4634370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:07.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.228+0000 7ff930c00640 1 --2- 172.21.15.28:0/2387101886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c463a850 0x5614c4634370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.228+0000 7ff930c00640 1 -- 172.21.15.28:0/2387101886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c4630180 msgr2=0x5614c4633e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:07.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.228+0000 7ff930c00640 1 --2- 172.21.15.28:0/2387101886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c4630180 0x5614c4633e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.228+0000 7ff930c00640 1 -- 172.21.15.28:0/2387101886 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5614c42670f0 con 0x5614c462f860 2024-04-24T08:15:07.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.228+0000 7ff931600640 1 --2- 172.21.15.28:0/2387101886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c463a850 0x5614c4634370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:15:07.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.228+0000 7ff92be00640 1 --2- 172.21.15.28:0/2387101886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c4630180 0x5614c4633e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:15:07.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.229+0000 7ff930c00640 1 --2- 172.21.15.28:0/2387101886 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c462f860 0x5614c46338f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff92000bfd0 tx=0x7ff920016640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cf115cf9bd9bf182 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:07.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.229+0000 7ff912000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:15:07.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.230+0000 7ff912000640 1 -- 172.21.15.28:0/2387101886 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92009a460 con 0x5614c462f860 2024-04-24T08:15:07.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.230+0000 7ff912000640 1 -- 172.21.15.28:0/2387101886 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7ff92009d660 con 0x5614c462f860 2024-04-24T08:15:07.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.230+0000 7ff936419740 1 -- 172.21.15.28:0/2387101886 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff924011e20 con 0x5614c462f860 2024-04-24T08:15:07.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.231+0000 7ff912000640 1 -- 172.21.15.28:0/2387101886 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff9200bb030 con 0x5614c462f860 2024-04-24T08:15:07.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.231+0000 7ff936419740 10 client.26149 fetch_fsmap learned FSMap version 26 2024-04-24T08:15:07.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.231+0000 7ff936419740 1 -- 172.21.15.28:0/2387101886 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5614c455a810 con 0x5614c462f860 2024-04-24T08:15:07.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.232+0000 7ff912000640 1 -- 172.21.15.28:0/2387101886 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff924011e20 con 0x5614c462f860 2024-04-24T08:15:07.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.233+0000 7ff912000640 1 -- 172.21.15.28:0/2387101886 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff9200997c0 con 0x5614c462f860 2024-04-24T08:15:07.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.233+0000 7ff936419740 10 client.26149 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:15:07.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.233+0000 7ff936419740 10 client.26149 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:15:07.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.233+0000 7ff936419740 20 client.26149 populate_metadata read hostname 'smithi028' 2024-04-24T08:15:07.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.233+0000 7ff936419740 1 --2- 172.21.15.28:0/2387101886 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5614c4643090 0x5614c46b30a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.233+0000 7ff936419740 4 client.26149 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:15:07.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.233+0000 7ff936419740 1 -- 172.21.15.28:0/2387101886 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5614c459ca80 con 0x5614c4643090 2024-04-24T08:15:07.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.234+0000 7ff92be00640 1 --2- 172.21.15.28:0/2387101886 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5614c4643090 0x5614c46b30a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.236+0000 7ff92be00640 1 --2- 172.21.15.28:0/2387101886 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5614c4643090 0x5614c46b30a0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b677c6f4097126f server_cookie=337946e1705499c8 in_seq=0 out_seq=0 2024-04-24T08:15:07.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.236+0000 7ff912000640 10 client.26149 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:07.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.239+0000 7ff912000640 1 -- 172.21.15.28:0/2387101886 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5614c459ca80 con 0x5614c4643090 2024-04-24T08:15:07.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.239+0000 7ff912000640 10 client.26149 handle_command_reply: tid=0 2024-04-24T08:15:07.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.239+0000 7ff936419740 1 client.26149 shutdown 2024-04-24T08:15:07.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.240+0000 7ff936419740 1 -- 172.21.15.28:0/2387101886 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c462f860 msgr2=0x5614c46338f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.240+0000 7ff936419740 1 --2- 172.21.15.28:0/2387101886 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c462f860 0x5614c46338f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff92000bfd0 tx=0x7ff920016640 comp rx=0 tx=0).stop 2024-04-24T08:15:07.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.241+0000 7ff936419740 1 -- 172.21.15.28:0/2387101886 shutdown_connections 2024-04-24T08:15:07.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.241+0000 7ff936419740 1 --2- 172.21.15.28:0/2387101886 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5614c4643090 0x5614c46b30a0 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.241+0000 7ff936419740 1 --2- 172.21.15.28:0/2387101886 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5614c4630180 0x5614c4633e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.241+0000 7ff936419740 1 --2- 172.21.15.28:0/2387101886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5614c463a850 0x5614c4634370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.241+0000 7ff936419740 1 --2- 172.21.15.28:0/2387101886 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5614c462f860 0x5614c46338f0 secure :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7ff92000bfd0 tx=0x7ff920016640 comp rx=0 tx=0).stop 2024-04-24T08:15:07.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.241+0000 7ff936419740 1 -- 172.21.15.28:0/2387101886 >> 172.21.15.28:0/2387101886 conn(0x5614c462bb80 msgr2=0x5614c462a390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:07.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.241+0000 7ff912000640 0 client.26149 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:07.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.241+0000 7ff936419740 1 -- 172.21.15.28:0/2387101886 shutdown_connections 2024-04-24T08:15:07.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.241+0000 7ff936419740 1 -- 172.21.15.28:0/2387101886 wait complete. 2024-04-24T08:15:07.242 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:15:07.243 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-24T08:15:07.243 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-24T08:15:07.243 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:15:07.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.242+0000 7ff936419740 20 client.26149 trim_cache size 0 max 16384 2024-04-24T08:15:07.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.243+0000 7ff933600640 1 -- 172.21.15.28:0/817971094 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff9140a3da0 msgr2=0x7ff9140a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.243+0000 7ff933600640 1 --2- 172.21.15.28:0/817971094 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff9140a3da0 0x7ff9140a6260 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7ff91802a980 tx=0x7ff918005d70 comp rx=0 tx=0).stop 2024-04-24T08:15:07.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.243+0000 7ff933600640 1 -- 172.21.15.28:0/817971094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff92c076530 msgr2=0x7ff92c073a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.243+0000 7ff933600640 1 --2- 172.21.15.28:0/817971094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff92c076530 0x7ff92c073a70 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff92000efb0 tx=0x7ff92000c800 comp rx=0 tx=0).stop 2024-04-24T08:15:07.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.244+0000 7ff933600640 1 -- 172.21.15.28:0/817971094 shutdown_connections 2024-04-24T08:15:07.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.244+0000 7ff933600640 1 --2- 172.21.15.28:0/817971094 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff9140a3da0 0x7ff9140a6260 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.244+0000 7ff933600640 1 --2- 172.21.15.28:0/817971094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff92c112ee0 0x7ff92c06e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.244+0000 7ff933600640 1 --2- 172.21.15.28:0/817971094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff92c076e50 0x7ff92c06db80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.244+0000 7ff933600640 1 --2- 172.21.15.28:0/817971094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff92c076530 0x7ff92c073a70 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.244+0000 7ff933600640 1 -- 172.21.15.28:0/817971094 >> 172.21.15.28:0/817971094 conn(0x7ff92c0ffd90 msgr2=0x7ff92c114600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:07.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.244+0000 7ff933600640 1 -- 172.21.15.28:0/817971094 shutdown_connections 2024-04-24T08:15:07.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.244+0000 7ff933600640 1 -- 172.21.15.28:0/817971094 wait complete. 2024-04-24T08:15:07.292 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:15:07.292 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:15:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.699+0000 7f9cabe00640 1 -- 172.21.15.28:0/702472753 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cac069400 msgr2=0x7f9cac0697e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.699+0000 7f9cabe00640 1 --2- 172.21.15.28:0/702472753 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cac069400 0x7f9cac0697e0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c009900 tx=0x7f9c9c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.699+0000 7f9cabe00640 1 -- 172.21.15.28:0/702472753 shutdown_connections 2024-04-24T08:15:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.699+0000 7f9cabe00640 1 --2- 172.21.15.28:0/702472753 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cac069db0 0x7f9cac106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.699+0000 7f9cabe00640 1 --2- 172.21.15.28:0/702472753 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cac1071a0 0x7f9cac10ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.699+0000 7f9cabe00640 1 --2- 172.21.15.28:0/702472753 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cac069400 0x7f9cac0697e0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.699+0000 7f9cabe00640 1 -- 172.21.15.28:0/702472753 >> 172.21.15.28:0/702472753 conn(0x7f9cac076840 msgr2=0x7f9cac076c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.699+0000 7f9cabe00640 1 -- 172.21.15.28:0/702472753 shutdown_connections 2024-04-24T08:15:07.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.699+0000 7f9cabe00640 1 -- 172.21.15.28:0/702472753 wait complete. 2024-04-24T08:15:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.700+0000 7f9cabe00640 1 Processor -- start 2024-04-24T08:15:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.700+0000 7f9cabe00640 1 -- start start 2024-04-24T08:15:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.700+0000 7f9cabe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cac069400 0x7f9cac19e440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.700+0000 7f9cabe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cac1071a0 0x7f9cac19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.700+0000 7f9cabe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cac1985f0 0x7f9cac198aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.701+0000 7f9cabe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9cac112230 con 0x7f9cac1071a0 2024-04-24T08:15:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.701+0000 7f9cabe00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9cac1120b0 con 0x7f9cac069400 2024-04-24T08:15:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.701+0000 7f9cabe00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9cac1123b0 con 0x7f9cac1985f0 2024-04-24T08:15:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.701+0000 7f9ca3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cac1071a0 0x7f9cac19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.701+0000 7f9caaa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cac069400 0x7f9cac19e440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.701+0000 7f9cab400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cac1985f0 0x7f9cac198aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.701+0000 7f9cab400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cac1985f0 0x7f9cac198aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:58062/0 (socket says 172.21.15.28:58062) 2024-04-24T08:15:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.701+0000 7f9caaa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cac069400 0x7f9cac19e440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:50836/0 (socket says 172.21.15.28:50836) 2024-04-24T08:15:07.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.701+0000 7f9cab400640 1 -- 172.21.15.28:0/2235375455 learned_addr learned my addr 172.21.15.28:0/2235375455 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.701+0000 7f9cab400640 1 -- 172.21.15.28:0/2235375455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cac069400 msgr2=0x7f9cac19e440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.701+0000 7f9cab400640 1 --2- 172.21.15.28:0/2235375455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cac069400 0x7f9cac19e440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.701+0000 7f9cab400640 1 -- 172.21.15.28:0/2235375455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cac1071a0 msgr2=0x7f9cac19e980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.701+0000 7f9cab400640 1 --2- 172.21.15.28:0/2235375455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cac1071a0 0x7f9cac19e980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.701+0000 7f9cab400640 1 -- 172.21.15.28:0/2235375455 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cac199110 con 0x7f9cac1985f0 2024-04-24T08:15:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.701+0000 7f9ca3e00640 1 --2- 172.21.15.28:0/2235375455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cac1071a0 0x7f9cac19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:15:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.702+0000 7f9cab400640 1 --2- 172.21.15.28:0/2235375455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cac1985f0 0x7f9cac198aa0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9ca400eb10 tx=0x7f9ca400c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.702+0000 7f9ca3400640 1 -- 172.21.15.28:0/2235375455 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4004b20 con 0x7f9cac1985f0 2024-04-24T08:15:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.702+0000 7f9cabe00640 1 -- 172.21.15.28:0/2235375455 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cac199400 con 0x7f9cac1985f0 2024-04-24T08:15:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.702+0000 7f9cabe00640 1 -- 172.21.15.28:0/2235375455 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cac1a56a0 con 0x7f9cac1985f0 2024-04-24T08:15:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.702+0000 7f9ca3400640 1 -- 172.21.15.28:0/2235375455 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ca4004cc0 con 0x7f9cac1985f0 2024-04-24T08:15:07.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.702+0000 7f9ca3400640 1 -- 172.21.15.28:0/2235375455 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ca4010760 con 0x7f9cac1985f0 2024-04-24T08:15:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.703+0000 7f9cabe00640 1 -- 172.21.15.28:0/2235375455 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c88005290 con 0x7f9cac1985f0 2024-04-24T08:15:07.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.705+0000 7f9ca3400640 1 -- 172.21.15.28:0/2235375455 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9ca4005020 con 0x7f9cac1985f0 2024-04-24T08:15:07.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.706+0000 7f9ca3400640 1 --2- 172.21.15.28:0/2235375455 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9c900a3da0 0x7f9c900a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.706+0000 7f9ca3400640 1 -- 172.21.15.28:0/2235375455 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f9ca40d0dd0 con 0x7f9cac1985f0 2024-04-24T08:15:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.706+0000 7f9caaa00640 1 --2- 172.21.15.28:0/2235375455 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9c900a3da0 0x7f9c900a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:07.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.707+0000 7f9caaa00640 1 --2- 172.21.15.28:0/2235375455 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9c900a3da0 0x7f9c900a6260 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c009230 tx=0x7f9c9c0058d0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:07.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:07.710+0000 7f9ca3400640 1 -- 172.21.15.28:0/2235375455 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f9ca409d3f0 con 0x7f9cac1985f0 2024-04-24T08:15:08.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:07 smithi028 ceph-mon[27786]: pgmap v146: 97 pgs: 97 active+clean; 17 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 54 MiB/s wr, 9.63k op/s 2024-04-24T08:15:08.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:07 smithi117 ceph-mon[32438]: pgmap v146: 97 pgs: 97 active+clean; 17 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 54 MiB/s wr, 9.63k op/s 2024-04-24T08:15:08.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:08.182+0000 7f9cabe00640 1 -- 172.21.15.28:0/2235375455 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f9c88005850 con 0x7f9cac1985f0 2024-04-24T08:15:08.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:08.184+0000 7f9ca3400640 1 -- 172.21.15.28:0/2235375455 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11871 (secure 0 0 0) 0x7f9ca409ccf0 con 0x7f9cac1985f0 2024-04-24T08:15:08.187 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:15:08.187 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:14:38.663141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:15:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-24T08:15:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:08.187+0000 7f9cabe00640 1 -- 172.21.15.28:0/2235375455 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9c900a3da0 msgr2=0x7f9c900a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:08.187+0000 7f9cabe00640 1 --2- 172.21.15.28:0/2235375455 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9c900a3da0 0x7f9c900a6260 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9c9c009230 tx=0x7f9c9c0058d0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:08.187+0000 7f9cabe00640 1 -- 172.21.15.28:0/2235375455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cac1985f0 msgr2=0x7f9cac198aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:08.187+0000 7f9cabe00640 1 --2- 172.21.15.28:0/2235375455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cac1985f0 0x7f9cac198aa0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f9ca400eb10 tx=0x7f9ca400c2d0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:08.187+0000 7f9cabe00640 1 -- 172.21.15.28:0/2235375455 shutdown_connections 2024-04-24T08:15:08.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:08.187+0000 7f9cabe00640 1 --2- 172.21.15.28:0/2235375455 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9c900a3da0 0x7f9c900a6260 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:08.187+0000 7f9cabe00640 1 --2- 172.21.15.28:0/2235375455 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9cac1071a0 0x7f9cac19e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:08.187+0000 7f9cabe00640 1 --2- 172.21.15.28:0/2235375455 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9cac1985f0 0x7f9cac198aa0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:08.187+0000 7f9cabe00640 1 --2- 172.21.15.28:0/2235375455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cac069400 0x7f9cac19e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:08.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:08.187+0000 7f9cabe00640 1 -- 172.21.15.28:0/2235375455 >> 172.21.15.28:0/2235375455 conn(0x7f9cac076840 msgr2=0x7f9cac0ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:08.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:08.188+0000 7f9cabe00640 1 -- 172.21.15.28:0/2235375455 shutdown_connections 2024-04-24T08:15:08.190 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:08.188+0000 7f9cabe00640 1 -- 172.21.15.28:0/2235375455 wait complete. 2024-04-24T08:15:08.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:07 smithi194 ceph-mon[32836]: pgmap v146: 97 pgs: 97 active+clean; 17 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 112 KiB/s rd, 54 MiB/s wr, 9.63k op/s 2024-04-24T08:15:08.220 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:15:09.029 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:08 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2235375455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:15:09.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:08 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2235375455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:15:09.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:08 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2235375455' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:15:10.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:09 smithi028 ceph-mon[27786]: pgmap v147: 97 pgs: 97 active+clean; 18 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 62 MiB/s wr, 10.98k op/s 2024-04-24T08:15:10.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:09 smithi117 ceph-mon[32438]: pgmap v147: 97 pgs: 97 active+clean; 18 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 62 MiB/s wr, 10.98k op/s 2024-04-24T08:15:10.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:09 smithi194 ceph-mon[32836]: pgmap v147: 97 pgs: 97 active+clean; 18 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 120 KiB/s rd, 62 MiB/s wr, 10.98k op/s 2024-04-24T08:15:12.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:11 smithi028 ceph-mon[27786]: pgmap v148: 97 pgs: 97 active+clean; 18 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 50 MiB/s wr, 8.80k op/s 2024-04-24T08:15:12.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:11 smithi117 ceph-mon[32438]: pgmap v148: 97 pgs: 97 active+clean; 18 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 50 MiB/s wr, 8.80k op/s 2024-04-24T08:15:12.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:11 smithi194 ceph-mon[32836]: pgmap v148: 97 pgs: 97 active+clean; 18 GiB data, 23 GiB used, 1.0 TiB / 1.0 TiB avail; 119 KiB/s rd, 50 MiB/s wr, 8.80k op/s 2024-04-24T08:15:13.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.048+0000 7effc1600640 1 -- 172.21.15.28:0/2802693654 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effbc1044b0 msgr2=0x7effbc104890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.048+0000 7effc1600640 1 --2- 172.21.15.28:0/2802693654 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effbc1044b0 0x7effbc104890 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7effa4009900 tx=0x7effa402a400 comp rx=0 tx=0).stop 2024-04-24T08:15:13.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.049+0000 7effc1600640 1 -- 172.21.15.28:0/2802693654 shutdown_connections 2024-04-24T08:15:13.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.049+0000 7effc1600640 1 --2- 172.21.15.28:0/2802693654 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effbc104e60 0x7effbc108ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.049+0000 7effc1600640 1 --2- 172.21.15.28:0/2802693654 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effbc109580 0x7effbc111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.050+0000 7effc1600640 1 --2- 172.21.15.28:0/2802693654 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effbc1044b0 0x7effbc104890 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.050+0000 7effc1600640 1 -- 172.21.15.28:0/2802693654 >> 172.21.15.28:0/2802693654 conn(0x7effbc0ffd90 msgr2=0x7effbc1021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:13.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.051+0000 7effc1600640 1 -- 172.21.15.28:0/2802693654 shutdown_connections 2024-04-24T08:15:13.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.051+0000 7effc1600640 1 -- 172.21.15.28:0/2802693654 wait complete. 2024-04-24T08:15:13.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.053+0000 7effc1600640 1 Processor -- start 2024-04-24T08:15:13.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.053+0000 7effc1600640 1 -- start start 2024-04-24T08:15:13.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.053+0000 7effc1600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effbc1044b0 0x7effbc073800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.053+0000 7effc1600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effbc109580 0x7effbc06c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.053+0000 7effc1600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effbc073e50 0x7effbc06cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.053+0000 7effc1600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7effbc114660 con 0x7effbc109580 2024-04-24T08:15:13.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.053+0000 7effc1600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7effbc1144e0 con 0x7effbc073e50 2024-04-24T08:15:13.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.053+0000 7effc1600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7effbc1147e0 con 0x7effbc1044b0 2024-04-24T08:15:13.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.054+0000 7effba000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effbc109580 0x7effbc06c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.054+0000 7effbaa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effbc1044b0 0x7effbc073800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.054+0000 7effbaa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effbc1044b0 0x7effbc073800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:38816/0 (socket says 172.21.15.28:38816) 2024-04-24T08:15:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.054+0000 7effbaa00640 1 -- 172.21.15.28:0/4211196554 learned_addr learned my addr 172.21.15.28:0/4211196554 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.054+0000 7effbb400640 1 --2- 172.21.15.28:0/4211196554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effbc073e50 0x7effbc06cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.054+0000 7effbaa00640 1 -- 172.21.15.28:0/4211196554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effbc073e50 msgr2=0x7effbc06cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.054+0000 7effbaa00640 1 --2- 172.21.15.28:0/4211196554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effbc073e50 0x7effbc06cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.055+0000 7effbaa00640 1 -- 172.21.15.28:0/4211196554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effbc109580 msgr2=0x7effbc06c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.055+0000 7effbaa00640 1 --2- 172.21.15.28:0/4211196554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effbc109580 0x7effbc06c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.055+0000 7effbaa00640 1 -- 172.21.15.28:0/4211196554 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effbc071320 con 0x7effbc1044b0 2024-04-24T08:15:13.057 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.055+0000 7effbaa00640 1 --2- 172.21.15.28:0/4211196554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effbc1044b0 0x7effbc073800 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7effa402a970 tx=0x7effa4003b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.057+0000 7effb3400640 1 -- 172.21.15.28:0/4211196554 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa403f070 con 0x7effbc1044b0 2024-04-24T08:15:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.057+0000 7effc1600640 1 -- 172.21.15.28:0/4211196554 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effbc0715b0 con 0x7effbc1044b0 2024-04-24T08:15:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.058+0000 7effc1600640 1 -- 172.21.15.28:0/4211196554 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effbc071af0 con 0x7effbc1044b0 2024-04-24T08:15:13.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.058+0000 7effb3400640 1 -- 172.21.15.28:0/4211196554 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effa4031c90 con 0x7effbc1044b0 2024-04-24T08:15:13.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.058+0000 7effb3400640 1 -- 172.21.15.28:0/4211196554 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa4035030 con 0x7effbc1044b0 2024-04-24T08:15:13.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.071+0000 7effc4438740 1 Processor -- start 2024-04-24T08:15:13.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.073+0000 7effb3400640 1 -- 172.21.15.28:0/4211196554 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7effa4042070 con 0x7effbc1044b0 2024-04-24T08:15:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.074+0000 7effc4438740 1 -- start start 2024-04-24T08:15:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.074+0000 7effc4438740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 0x558ae4d0ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.074+0000 7effc4438740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0f050 0x558ae4d21c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.075+0000 7effbb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 0x558ae4d0ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.075+0000 7effb3400640 1 --2- 172.21.15.28:0/4211196554 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7effa00a3de0 0x7effa00a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.075+0000 7effbb400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 0x558ae4d0ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56710/0 (socket says 172.21.15.28:56710) 2024-04-24T08:15:13.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.075+0000 7effb3400640 1 -- 172.21.15.28:0/4211196554 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7effa40f0df0 con 0x7effbc1044b0 2024-04-24T08:15:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.076+0000 7effba000640 1 --2- 172.21.15.28:0/4211196554 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7effa00a3de0 0x7effa00a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.076+0000 7effc4438740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 0x558ae4d24540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.076+0000 7effc4438740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558ae4c741d0 con 0x558ae4d0e6a0 2024-04-24T08:15:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.076+0000 7effc4438740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558ae4c1a450 con 0x558ae4d0f050 2024-04-24T08:15:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.076+0000 7effc4438740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558ae4c70190 con 0x558ae4d22150 2024-04-24T08:15:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.076+0000 7effbaa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0f050 0x558ae4d21c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.076+0000 7effbb400640 1 -- 172.21.15.28:0/4215442656 learned_addr learned my addr 172.21.15.28:0/4215442656 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.076+0000 7effbaa00640 1 -- 172.21.15.28:0/4215442656 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 msgr2=0x558ae4d24540 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:15:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.076+0000 7effbaa00640 1 --2- 172.21.15.28:0/4215442656 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 0x558ae4d24540 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.076+0000 7effbaa00640 1 -- 172.21.15.28:0/4215442656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 msgr2=0x558ae4d0ea80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.076+0000 7effbaa00640 1 --2- 172.21.15.28:0/4215442656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 0x558ae4d0ea80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.077+0000 7effbaa00640 1 -- 172.21.15.28:0/4215442656 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ae4cd7310 con 0x558ae4d0f050 2024-04-24T08:15:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.077+0000 7effba000640 1 --2- 172.21.15.28:0/4211196554 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7effa00a3de0 0x7effa00a62a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7effac005fd0 tx=0x7effac005850 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.077+0000 7effbaa00640 1 --2- 172.21.15.28:0/4215442656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0f050 0x558ae4d21c10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7effa40ee520 tx=0x7effa4003f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2e0b6f9929944e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:13.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.077+0000 7effab400640 1 -- 172.21.15.28:0/4215442656 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa40bde00 con 0x558ae4d0f050 2024-04-24T08:15:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.077+0000 7effab400640 1 -- 172.21.15.28:0/4215442656 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effa40c21b0 con 0x558ae4d0f050 2024-04-24T08:15:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.077+0000 7effab400640 1 -- 172.21.15.28:0/4215442656 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa40c6840 con 0x558ae4d0f050 2024-04-24T08:15:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.079+0000 7effc4438740 1 -- 172.21.15.28:0/4215442656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0f050 msgr2=0x558ae4d21c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.079+0000 7effc4438740 1 --2- 172.21.15.28:0/4215442656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0f050 0x558ae4d21c10 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7effa40ee520 tx=0x7effa4003f90 comp rx=0 tx=0).stop 2024-04-24T08:15:13.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.080+0000 7effc4438740 1 -- 172.21.15.28:0/4215442656 shutdown_connections 2024-04-24T08:15:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.080+0000 7effc4438740 1 --2- 172.21.15.28:0/4215442656 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 0x558ae4d24540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.080+0000 7effc4438740 1 --2- 172.21.15.28:0/4215442656 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0f050 0x558ae4d21c10 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.081+0000 7effc4438740 1 --2- 172.21.15.28:0/4215442656 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 0x558ae4d0ea80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.081+0000 7effc4438740 1 -- 172.21.15.28:0/4215442656 >> 172.21.15.28:0/4215442656 conn(0x558ae4d0ac70 msgr2=0x558ae4d0b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:13.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.083+0000 7effc4438740 1 -- 172.21.15.28:0/4215442656 shutdown_connections 2024-04-24T08:15:13.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.083+0000 7effc4438740 1 -- 172.21.15.28:0/4215442656 wait complete. 2024-04-24T08:15:13.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.084+0000 7effc4438740 1 -- start start 2024-04-24T08:15:13.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.084+0000 7effc4438740 1 Processor -- start 2024-04-24T08:15:13.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.085+0000 7effc4438740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 0x558ae4d21be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.086+0000 7effc4438740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0f050 0x558ae4d1eed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.086+0000 7effbaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 0x558ae4d21be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.086+0000 7effbaa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 0x558ae4d21be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56720/0 (socket says 172.21.15.28:56720) 2024-04-24T08:15:13.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.086+0000 7effc4438740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 0x558ae4d1f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.087+0000 7effbaa00640 1 -- 172.21.15.28:0/2524298826 learned_addr learned my addr 172.21.15.28:0/2524298826 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:13.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.087+0000 7effc4438740 1 -- 172.21.15.28:0/2524298826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558ae4d24cd0 con 0x558ae4d0e6a0 2024-04-24T08:15:13.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.087+0000 7effc4438740 1 -- 172.21.15.28:0/2524298826 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558ae4c2d810 con 0x558ae4d0f050 2024-04-24T08:15:13.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.087+0000 7effc4438740 1 -- 172.21.15.28:0/2524298826 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558ae4d24e50 con 0x558ae4d22150 2024-04-24T08:15:13.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.087+0000 7effc4438740 10 client.0 _ll_register_callbacks cb 0x558ae4bfb140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:13.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.087+0000 7effbb400640 1 --2- 172.21.15.28:0/2524298826 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 0x558ae4d1f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.088+0000 7effbaa00640 1 -- 172.21.15.28:0/2524298826 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 msgr2=0x558ae4d1f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:13.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.088+0000 7effbaa00640 1 --2- 172.21.15.28:0/2524298826 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 0x558ae4d1f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.088+0000 7effbaa00640 1 -- 172.21.15.28:0/2524298826 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0f050 msgr2=0x558ae4d1eed0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:15:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.088+0000 7effbaa00640 1 --2- 172.21.15.28:0/2524298826 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0f050 0x558ae4d1eed0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.088+0000 7effbaa00640 1 -- 172.21.15.28:0/2524298826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ae4c3e3f0 con 0x558ae4d0e6a0 2024-04-24T08:15:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.088+0000 7effbb400640 1 --2- 172.21.15.28:0/2524298826 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 0x558ae4d1f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:13.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.088+0000 7effbaa00640 1 --2- 172.21.15.28:0/2524298826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 0x558ae4d21be0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7effa4000c00 tx=0x7effa40ba6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fdb60131796402 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:13.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.088+0000 7eff9fe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:15:13.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.089+0000 7eff9fe00640 1 -- 172.21.15.28:0/2524298826 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa40d71b0 con 0x558ae4d0e6a0 2024-04-24T08:15:13.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.089+0000 7eff9fe00640 1 -- 172.21.15.28:0/2524298826 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7effa40c65e0 con 0x558ae4d0e6a0 2024-04-24T08:15:13.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.090+0000 7effc4438740 1 -- 172.21.15.28:0/2524298826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effa40c21b0 con 0x558ae4d0e6a0 2024-04-24T08:15:13.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.090+0000 7eff9fe00640 1 -- 172.21.15.28:0/2524298826 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7effa40d7350 con 0x558ae4d0e6a0 2024-04-24T08:15:13.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.091+0000 7effc4438740 10 client.15948 fetch_fsmap learned FSMap version 26 2024-04-24T08:15:13.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.091+0000 7effc4438740 1 -- 172.21.15.28:0/2524298826 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558ae49863f0 con 0x558ae4d0e6a0 2024-04-24T08:15:13.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.091+0000 7eff9fe00640 1 -- 172.21.15.28:0/2524298826 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effa40c21b0 con 0x558ae4d0e6a0 2024-04-24T08:15:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.092+0000 7eff9fe00640 1 -- 172.21.15.28:0/2524298826 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7effa403baf0 con 0x558ae4d0e6a0 2024-04-24T08:15:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.092+0000 7effc4438740 10 client.15948 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:15:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.092+0000 7effc4438740 10 client.15948 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:15:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.092+0000 7effc4438740 20 client.15948 populate_metadata read hostname 'smithi028' 2024-04-24T08:15:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.092+0000 7effc4438740 1 --2- 172.21.15.28:0/2524298826 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558ae4d31f50 0x558ae4d34340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.092+0000 7effc4438740 4 client.15948 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:15:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.092+0000 7effc4438740 1 -- 172.21.15.28:0/2524298826 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558ae4c81bf0 con 0x558ae4d31f50 2024-04-24T08:15:13.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.092+0000 7effba000640 1 --2- 172.21.15.28:0/2524298826 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558ae4d31f50 0x558ae4d34340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.093+0000 7effba000640 1 --2- 172.21.15.28:0/2524298826 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558ae4d31f50 0x558ae4d34340 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4e265b2737bca508 server_cookie=2daa6193632f6147 in_seq=0 out_seq=0 2024-04-24T08:15:13.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.093+0000 7eff9fe00640 10 client.15948 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:15:13.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.094+0000 7eff9fe00640 1 -- 172.21.15.28:0/2524298826 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558ae4c81bf0 con 0x558ae4d31f50 2024-04-24T08:15:13.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.094+0000 7eff9fe00640 10 client.15948 handle_command_reply: tid=0 2024-04-24T08:15:13.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.094+0000 7effc4438740 1 client.15948 shutdown 2024-04-24T08:15:13.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.094+0000 7effc4438740 1 -- 172.21.15.28:0/2524298826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 msgr2=0x558ae4d21be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.094+0000 7effc4438740 1 --2- 172.21.15.28:0/2524298826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 0x558ae4d21be0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7effa4000c00 tx=0x7effa40ba6a0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.097+0000 7effc4438740 1 -- 172.21.15.28:0/2524298826 shutdown_connections 2024-04-24T08:15:13.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.098+0000 7effc4438740 1 --2- 172.21.15.28:0/2524298826 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558ae4d31f50 0x558ae4d34340 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.098+0000 7effc4438740 1 --2- 172.21.15.28:0/2524298826 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 0x558ae4d1f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.098+0000 7effc4438740 1 --2- 172.21.15.28:0/2524298826 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0f050 0x558ae4d1eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.098+0000 7effc4438740 1 --2- 172.21.15.28:0/2524298826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 0x558ae4d21be0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.098+0000 7eff9fe00640 0 client.15948 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:15:13.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.098+0000 7effc4438740 1 -- 172.21.15.28:0/2524298826 >> 172.21.15.28:0/2524298826 conn(0x558ae4d0ac70 msgr2=0x558ae4d10a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:13.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.099+0000 7effc4438740 1 -- 172.21.15.28:0/2524298826 shutdown_connections 2024-04-24T08:15:13.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.099+0000 7effc4438740 1 -- 172.21.15.28:0/2524298826 wait complete. 2024-04-24T08:15:13.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.100+0000 7effc4438740 20 client.15948 trim_cache size 0 max 16384 2024-04-24T08:15:13.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.130+0000 7effc4438740 1 Processor -- start 2024-04-24T08:15:13.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.131+0000 7effc4438740 1 -- start start 2024-04-24T08:15:13.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.131+0000 7effc4438740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 0x558ae4d2dc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.131+0000 7effc4438740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0f050 0x558ae4d21710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.132+0000 7effc4438740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 0x558ae4d39c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.132+0000 7effc4438740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff98011100 con 0x558ae4d0e6a0 2024-04-24T08:15:13.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.132+0000 7effc4438740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7eff9800e9d0 con 0x558ae4d0f050 2024-04-24T08:15:13.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.132+0000 7effc4438740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7eff9800c820 con 0x558ae4d22150 2024-04-24T08:15:13.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.132+0000 7effba000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0f050 0x558ae4d21710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.132+0000 7effba000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0f050 0x558ae4d21710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51048/0 (socket says 172.21.15.28:51048) 2024-04-24T08:15:13.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.132+0000 7effba000640 1 -- 172.21.15.28:0/3059338655 learned_addr learned my addr 172.21.15.28:0/3059338655 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:13.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.133+0000 7effbaa00640 1 --2- 172.21.15.28:0/3059338655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 0x558ae4d2dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.133+0000 7effbb400640 1 --2- 172.21.15.28:0/3059338655 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 0x558ae4d39c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.133+0000 7effba000640 1 -- 172.21.15.28:0/3059338655 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 msgr2=0x558ae4d39c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:13.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.133+0000 7effba000640 1 --2- 172.21.15.28:0/3059338655 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 0x558ae4d39c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.133+0000 7effba000640 1 -- 172.21.15.28:0/3059338655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 msgr2=0x558ae4d2dc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:13.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.133+0000 7effba000640 1 --2- 172.21.15.28:0/3059338655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 0x558ae4d2dc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.133+0000 7effba000640 1 -- 172.21.15.28:0/3059338655 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558ae4d0edf0 con 0x558ae4d0f050 2024-04-24T08:15:13.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.133+0000 7effbaa00640 1 --2- 172.21.15.28:0/3059338655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 0x558ae4d2dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:15:13.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.134+0000 7effba000640 1 --2- 172.21.15.28:0/3059338655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0f050 0x558ae4d21710 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7effbc06e040 tx=0x7effac015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7aab553d224ddcf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:13.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.134+0000 7effaaa00640 1 -- 172.21.15.28:0/3059338655 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac02b070 con 0x558ae4d0f050 2024-04-24T08:15:13.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.135+0000 7effaaa00640 1 -- 172.21.15.28:0/3059338655 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effac011cf0 con 0x558ae4d0f050 2024-04-24T08:15:13.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.135+0000 7effaaa00640 1 -- 172.21.15.28:0/3059338655 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac01e680 con 0x558ae4d0f050 2024-04-24T08:15:13.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.136+0000 7effc4438740 1 -- 172.21.15.28:0/3059338655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0f050 msgr2=0x558ae4d21710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.136+0000 7effc4438740 1 --2- 172.21.15.28:0/3059338655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0f050 0x558ae4d21710 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7effbc06e040 tx=0x7effac015000 comp rx=0 tx=0).stop 2024-04-24T08:15:13.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.136+0000 7effc4438740 1 -- 172.21.15.28:0/3059338655 shutdown_connections 2024-04-24T08:15:13.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.136+0000 7effc4438740 1 --2- 172.21.15.28:0/3059338655 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 0x558ae4d39c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.136+0000 7effc4438740 1 --2- 172.21.15.28:0/3059338655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0f050 0x558ae4d21710 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.137+0000 7effc4438740 1 --2- 172.21.15.28:0/3059338655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0e6a0 0x558ae4d2dc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.137+0000 7effc4438740 1 -- 172.21.15.28:0/3059338655 >> 172.21.15.28:0/3059338655 conn(0x558ae4d0ac70 msgr2=0x558ae4d1d7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.137+0000 7effc4438740 1 -- 172.21.15.28:0/3059338655 shutdown_connections 2024-04-24T08:15:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.137+0000 7effc4438740 1 -- 172.21.15.28:0/3059338655 wait complete. 2024-04-24T08:15:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.138+0000 7effc4438740 1 -- start start 2024-04-24T08:15:13.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.138+0000 7effc4438740 1 Processor -- start 2024-04-24T08:15:13.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.139+0000 7effc4438740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0e6a0 0x558ae4d23680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.140+0000 7effc4438740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0f050 0x558ae4d23bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.140+0000 7effbaa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0e6a0 0x558ae4d23680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.140+0000 7effbaa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0e6a0 0x558ae4d23680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51060/0 (socket says 172.21.15.28:51060) 2024-04-24T08:15:13.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.140+0000 7effbaa00640 1 -- 172.21.15.28:0/3395950477 learned_addr learned my addr 172.21.15.28:0/3395950477 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:13.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.141+0000 7effc4438740 1 --2- 172.21.15.28:0/3395950477 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 0x558ae4d28640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.141+0000 7effc4438740 1 -- 172.21.15.28:0/3395950477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7eff9800ce90 con 0x558ae4d0f050 2024-04-24T08:15:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.141+0000 7effc4438740 1 -- 172.21.15.28:0/3395950477 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7eff9800e300 con 0x558ae4d0e6a0 2024-04-24T08:15:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.141+0000 7effc4438740 1 -- 172.21.15.28:0/3395950477 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7eff9800d560 con 0x558ae4d22150 2024-04-24T08:15:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.141+0000 7effc4438740 10 client.0 _ll_register_callbacks cb 0x7effa40bdbd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.141+0000 7effbb400640 1 --2- 172.21.15.28:0/3395950477 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 0x558ae4d28640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.141+0000 7effba000640 1 --2- 172.21.15.28:0/3395950477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0f050 0x558ae4d23bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.142+0000 7effba000640 1 -- 172.21.15.28:0/3395950477 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 msgr2=0x558ae4d28640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:13.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.142+0000 7effba000640 1 --2- 172.21.15.28:0/3395950477 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 0x558ae4d28640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.142+0000 7effba000640 1 -- 172.21.15.28:0/3395950477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0e6a0 msgr2=0x558ae4d23680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:13.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.142+0000 7effba000640 1 --2- 172.21.15.28:0/3395950477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0e6a0 0x558ae4d23680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.142+0000 7effba000640 1 -- 172.21.15.28:0/3395950477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558ae4a507d0 con 0x558ae4d0f050 2024-04-24T08:15:13.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.143+0000 7effbaa00640 1 --2- 172.21.15.28:0/3395950477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0e6a0 0x558ae4d23680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:15:13.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.143+0000 7eff9fe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:15:13.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.143+0000 7effba000640 1 --2- 172.21.15.28:0/3395950477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0f050 0x558ae4d23bc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x558ae4a6dda0 tx=0x7effac005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29d836a6bc77add9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:13.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.144+0000 7eff9fe00640 1 -- 172.21.15.28:0/3395950477 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effac02b070 con 0x558ae4d0f050 2024-04-24T08:15:13.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.144+0000 7eff9fe00640 1 -- 172.21.15.28:0/3395950477 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7effac025050 con 0x558ae4d0f050 2024-04-24T08:15:13.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.144+0000 7effc4438740 1 -- 172.21.15.28:0/3395950477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effac011cf0 con 0x558ae4d0f050 2024-04-24T08:15:13.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.144+0000 7eff9fe00640 1 -- 172.21.15.28:0/3395950477 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7effac025990 con 0x558ae4d0f050 2024-04-24T08:15:13.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.144+0000 7effc4438740 10 client.15954 fetch_fsmap learned FSMap version 26 2024-04-24T08:15:13.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.144+0000 7effc4438740 1 -- 172.21.15.28:0/3395950477 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558ae4d98930 con 0x558ae4d0f050 2024-04-24T08:15:13.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.146+0000 7eff9fe00640 1 -- 172.21.15.28:0/3395950477 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effac011cf0 con 0x558ae4d0f050 2024-04-24T08:15:13.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.146+0000 7eff9fe00640 1 -- 172.21.15.28:0/3395950477 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7effac004860 con 0x558ae4d0f050 2024-04-24T08:15:13.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.146+0000 7effc4438740 10 client.15954 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:15:13.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.146+0000 7effc4438740 10 client.15954 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:15:13.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.146+0000 7effc4438740 20 client.15954 populate_metadata read hostname 'smithi028' 2024-04-24T08:15:13.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.146+0000 7effc4438740 1 --2- 172.21.15.28:0/3395950477 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558ae4d98e50 0x558ae4d9b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.146+0000 7effc4438740 4 client.15954 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:15:13.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.146+0000 7effc4438740 1 -- 172.21.15.28:0/3395950477 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558ae4c7ba60 con 0x558ae4d98e50 2024-04-24T08:15:13.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.147+0000 7effbaa00640 1 --2- 172.21.15.28:0/3395950477 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558ae4d98e50 0x558ae4d9b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.148+0000 7effbaa00640 1 --2- 172.21.15.28:0/3395950477 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558ae4d98e50 0x558ae4d9b270 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=91e420f8e8d3542b server_cookie=f4077569db40fb17 in_seq=0 out_seq=0 2024-04-24T08:15:13.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.148+0000 7eff9fe00640 10 client.15954 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:15:13.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.148+0000 7eff9fe00640 1 -- 172.21.15.28:0/3395950477 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558ae4c7ba60 con 0x558ae4d98e50 2024-04-24T08:15:13.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.148+0000 7eff9fe00640 10 client.15954 handle_command_reply: tid=0 2024-04-24T08:15:13.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.148+0000 7effc4438740 1 client.15954 shutdown 2024-04-24T08:15:13.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.150+0000 7effc4438740 1 -- 172.21.15.28:0/3395950477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0f050 msgr2=0x558ae4d23bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.150+0000 7effc4438740 1 --2- 172.21.15.28:0/3395950477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0f050 0x558ae4d23bc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x558ae4a6dda0 tx=0x7effac005c80 comp rx=0 tx=0).stop 2024-04-24T08:15:13.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.150+0000 7effc4438740 1 -- 172.21.15.28:0/3395950477 shutdown_connections 2024-04-24T08:15:13.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.150+0000 7effc4438740 1 --2- 172.21.15.28:0/3395950477 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558ae4d98e50 0x558ae4d9b270 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.150+0000 7effc4438740 1 --2- 172.21.15.28:0/3395950477 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558ae4d0f050 0x558ae4d23bc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.150+0000 7effc4438740 1 --2- 172.21.15.28:0/3395950477 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558ae4d22150 0x558ae4d28640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.150+0000 7effc4438740 1 --2- 172.21.15.28:0/3395950477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558ae4d0e6a0 0x558ae4d23680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.150+0000 7effc4438740 1 -- 172.21.15.28:0/3395950477 >> 172.21.15.28:0/3395950477 conn(0x558ae4d0ac70 msgr2=0x558ae4d101e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:13.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.150+0000 7eff9fe00640 0 client.15954 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:15:13.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.151+0000 7effc4438740 1 -- 172.21.15.28:0/3395950477 shutdown_connections 2024-04-24T08:15:13.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.151+0000 7effc4438740 1 -- 172.21.15.28:0/3395950477 wait complete. 2024-04-24T08:15:13.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.151+0000 7effc4438740 20 client.15954 trim_cache size 0 max 16384 2024-04-24T08:15:13.153 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:15:13.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.152+0000 7effc1600640 1 -- 172.21.15.28:0/4211196554 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7effa00a3de0 msgr2=0x7effa00a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.152+0000 7effc1600640 1 --2- 172.21.15.28:0/4211196554 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7effa00a3de0 0x7effa00a62a0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7effac005fd0 tx=0x7effac005850 comp rx=0 tx=0).stop 2024-04-24T08:15:13.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.152+0000 7effc1600640 1 -- 172.21.15.28:0/4211196554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effbc1044b0 msgr2=0x7effbc073800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.152+0000 7effc1600640 1 --2- 172.21.15.28:0/4211196554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effbc1044b0 0x7effbc073800 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7effa402a970 tx=0x7effa4003b90 comp rx=0 tx=0).stop 2024-04-24T08:15:13.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.153+0000 7effc1600640 1 -- 172.21.15.28:0/4211196554 shutdown_connections 2024-04-24T08:15:13.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.153+0000 7effc1600640 1 --2- 172.21.15.28:0/4211196554 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7effa00a3de0 0x7effa00a62a0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.153+0000 7effc1600640 1 --2- 172.21.15.28:0/4211196554 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7effbc109580 0x7effbc06c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.153+0000 7effc1600640 1 --2- 172.21.15.28:0/4211196554 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effbc073e50 0x7effbc06cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.153+0000 7effc1600640 1 --2- 172.21.15.28:0/4211196554 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7effbc1044b0 0x7effbc073800 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.153+0000 7effc1600640 1 -- 172.21.15.28:0/4211196554 >> 172.21.15.28:0/4211196554 conn(0x7effbc0ffd90 msgr2=0x7effbc102140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:13.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.154+0000 7effc1600640 1 -- 172.21.15.28:0/4211196554 shutdown_connections 2024-04-24T08:15:13.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.154+0000 7effc1600640 1 -- 172.21.15.28:0/4211196554 wait complete. 2024-04-24T08:15:13.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:15:13.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.502+0000 7f5e5d800640 1 -- 172.21.15.28:0/1620331294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e58101770 msgr2=0x7f5e58105c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.502+0000 7f5e5d800640 1 --2- 172.21.15.28:0/1620331294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e58101770 0x7f5e58105c30 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009860 tx=0x7f5e4402a3e0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.502+0000 7f5e5d800640 1 -- 172.21.15.28:0/1620331294 shutdown_connections 2024-04-24T08:15:13.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.503+0000 7f5e5d800640 1 --2- 172.21.15.28:0/1620331294 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58106250 0x7f5e581066e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.503+0000 7f5e5d800640 1 --2- 172.21.15.28:0/1620331294 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e58101770 0x7f5e58105c30 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.503+0000 7f5e5d800640 1 --2- 172.21.15.28:0/1620331294 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e58100e50 0x7f5e58101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.503+0000 7f5e5d800640 1 -- 172.21.15.28:0/1620331294 >> 172.21.15.28:0/1620331294 conn(0x7f5e580fc880 msgr2=0x7f5e580feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:13.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.504+0000 7f5e5d800640 1 -- 172.21.15.28:0/1620331294 shutdown_connections 2024-04-24T08:15:13.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.504+0000 7f5e5d800640 1 -- 172.21.15.28:0/1620331294 wait complete. 2024-04-24T08:15:13.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.505+0000 7f5e5d800640 1 Processor -- start 2024-04-24T08:15:13.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.506+0000 7f5e5d800640 1 -- start start 2024-04-24T08:15:13.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.507+0000 7f5e5d800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e58100e50 0x7f5e58073810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.507+0000 7f5e5d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58101770 0x7f5e5806c890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.507+0000 7f5e5d800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e58106250 0x7f5e5806cdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.507+0000 7f5e5d800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e58118850 con 0x7f5e58101770 2024-04-24T08:15:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.507+0000 7f5e5d800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5e581186d0 con 0x7f5e58106250 2024-04-24T08:15:13.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.507+0000 7f5e5d800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5e581189d0 con 0x7f5e58100e50 2024-04-24T08:15:13.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.508+0000 7f5e57e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e58100e50 0x7f5e58073810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.508+0000 7f5e57e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e58100e50 0x7f5e58073810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:38876/0 (socket says 172.21.15.28:38876) 2024-04-24T08:15:13.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.508+0000 7f5e57e00640 1 -- 172.21.15.28:0/282362491 learned_addr learned my addr 172.21.15.28:0/282362491 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:13.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.508+0000 7f5e57e00640 1 -- 172.21.15.28:0/282362491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e58106250 msgr2=0x7f5e5806cdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.508+0000 7f5e5ce00640 1 --2- 172.21.15.28:0/282362491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e58106250 0x7f5e5806cdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.508+0000 7f5e57e00640 1 --2- 172.21.15.28:0/282362491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e58106250 0x7f5e5806cdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.508+0000 7f5e57e00640 1 -- 172.21.15.28:0/282362491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58101770 msgr2=0x7f5e5806c890 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:15:13.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.509+0000 7f5e57e00640 1 --2- 172.21.15.28:0/282362491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58101770 0x7f5e5806c890 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.509+0000 7f5e57e00640 1 -- 172.21.15.28:0/282362491 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5e5806d390 con 0x7f5e58100e50 2024-04-24T08:15:13.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.509+0000 7f5e57e00640 1 --2- 172.21.15.28:0/282362491 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e58100e50 0x7f5e58073810 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800d950 tx=0x7f5e4800de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:13.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.509+0000 7f5e54c00640 1 -- 172.21.15.28:0/282362491 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e48017070 con 0x7f5e58100e50 2024-04-24T08:15:13.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.510+0000 7f5e54c00640 1 -- 172.21.15.28:0/282362491 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e48009c30 con 0x7f5e58100e50 2024-04-24T08:15:13.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.510+0000 7f5e5d800640 1 -- 172.21.15.28:0/282362491 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5e5806d680 con 0x7f5e58100e50 2024-04-24T08:15:13.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.510+0000 7f5e54c00640 1 -- 172.21.15.28:0/282362491 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e4800f6b0 con 0x7f5e58100e50 2024-04-24T08:15:13.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.510+0000 7f5e5d800640 1 -- 172.21.15.28:0/282362491 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5e580716b0 con 0x7f5e58100e50 2024-04-24T08:15:13.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.511+0000 7f5e5ce00640 1 --2- 172.21.15.28:0/282362491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e58106250 0x7f5e5806cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:13.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.516+0000 7f5e54c00640 1 -- 172.21.15.28:0/282362491 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5e480109d0 con 0x7f5e58100e50 2024-04-24T08:15:13.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.517+0000 7f5e61b99740 1 Processor -- start 2024-04-24T08:15:13.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.517+0000 7f5e61b99740 1 -- start start 2024-04-24T08:15:13.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.519+0000 7f5e61b99740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569b4b40 0x561f569b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.519+0000 7f5e61b99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569bf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.519+0000 7f5e61b99740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569bfb30 0x561f569c1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.519+0000 7f5e61b99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561f5691a1d0 con 0x561f569b5460 2024-04-24T08:15:13.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.519+0000 7f5e61b99740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561f568c0450 con 0x561f569b4b40 2024-04-24T08:15:13.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.519+0000 7f5e61b99740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561f56916190 con 0x561f569bfb30 2024-04-24T08:15:13.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.519+0000 7f5e5ce00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569b4b40 0x561f569b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.519+0000 7f5e57400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569bfb30 0x561f569c1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.519+0000 7f5e57400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569bfb30 0x561f569c1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:38882/0 (socket says 172.21.15.28:38882) 2024-04-24T08:15:13.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.519+0000 7f5e57400640 1 -- 172.21.15.28:0/1420633879 learned_addr learned my addr 172.21.15.28:0/1420633879 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:13.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.520+0000 7f5e57e00640 1 --2- 172.21.15.28:0/1420633879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569bf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.520+0000 7f5e54c00640 1 --2- 172.21.15.28:0/282362491 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5e3c0a3e20 0x7f5e3c0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.520+0000 7f5e5ce00640 1 -- 172.21.15.28:0/1420633879 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569bfb30 msgr2=0x561f569c1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.520+0000 7f5e5ce00640 1 --2- 172.21.15.28:0/1420633879 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569bfb30 0x561f569c1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.520+0000 7f5e5ce00640 1 -- 172.21.15.28:0/1420633879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 msgr2=0x561f569bf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.520+0000 7f5e5ce00640 1 --2- 172.21.15.28:0/1420633879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569bf5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.520+0000 7f5e5ce00640 1 -- 172.21.15.28:0/1420633879 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f5697d310 con 0x561f569b4b40 2024-04-24T08:15:13.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.521+0000 7f5e54c00640 1 -- 172.21.15.28:0/282362491 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f5e480d04e0 con 0x7f5e58100e50 2024-04-24T08:15:13.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.523+0000 7f5e5ce00640 1 --2- 172.21.15.28:0/1420633879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569b4b40 0x561f569b4f20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5e50009e40 tx=0x7f5e5000f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f7ecd9e0ddf8ee7c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:13.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.523+0000 7f5e5ce00640 1 --2- 172.21.15.28:0/282362491 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5e3c0a3e20 0x7f5e3c0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.524+0000 7f5e4d600640 1 -- 172.21.15.28:0/1420633879 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e500116e0 con 0x561f569b4b40 2024-04-24T08:15:13.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.524+0000 7f5e4d600640 1 -- 172.21.15.28:0/1420633879 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e50011d60 con 0x561f569b4b40 2024-04-24T08:15:13.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.525+0000 7f5e4d600640 1 -- 172.21.15.28:0/1420633879 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50010430 con 0x561f569b4b40 2024-04-24T08:15:13.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.526+0000 7f5e61b99740 1 -- 172.21.15.28:0/1420633879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569b4b40 msgr2=0x561f569b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.526+0000 7f5e5ce00640 1 --2- 172.21.15.28:0/282362491 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5e3c0a3e20 0x7f5e3c0a62e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5e5001dfd0 tx=0x7f5e50010690 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:13.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.526+0000 7f5e61b99740 1 --2- 172.21.15.28:0/1420633879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569b4b40 0x561f569b4f20 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f5e50009e40 tx=0x7f5e5000f730 comp rx=0 tx=0).stop 2024-04-24T08:15:13.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.526+0000 7f5e61b99740 1 -- 172.21.15.28:0/1420633879 shutdown_connections 2024-04-24T08:15:13.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.526+0000 7f5e61b99740 1 --2- 172.21.15.28:0/1420633879 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569bf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.526+0000 7f5e61b99740 1 --2- 172.21.15.28:0/1420633879 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569bfb30 0x561f569c1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.526+0000 7f5e61b99740 1 --2- 172.21.15.28:0/1420633879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569b4b40 0x561f569b4f20 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.526+0000 7f5e61b99740 1 -- 172.21.15.28:0/1420633879 >> 172.21.15.28:0/1420633879 conn(0x561f569b0e60 msgr2=0x561f569b1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:13.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.527+0000 7f5e61b99740 1 -- 172.21.15.28:0/1420633879 shutdown_connections 2024-04-24T08:15:13.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.527+0000 7f5e61b99740 1 -- 172.21.15.28:0/1420633879 wait complete. 2024-04-24T08:15:13.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.529+0000 7f5e61b99740 1 -- start start 2024-04-24T08:15:13.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.529+0000 7f5e61b99740 1 Processor -- start 2024-04-24T08:15:13.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.530+0000 7f5e61b99740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569b4b40 0x561f569bf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.531+0000 7f5e61b99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569bc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.531+0000 7f5e61b99740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569bfb30 0x561f569bcfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.531+0000 7f5e61b99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561f569c2a10 con 0x561f569b5460 2024-04-24T08:15:13.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.531+0000 7f5e61b99740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561f568d3810 con 0x561f569bfb30 2024-04-24T08:15:13.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.531+0000 7f5e61b99740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561f569c2b90 con 0x561f569b4b40 2024-04-24T08:15:13.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.531+0000 7f5e61b99740 10 client.0 _ll_register_callbacks cb 0x561f568a1140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:13.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.532+0000 7f5e57e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569b4b40 0x561f569bf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.532+0000 7f5e57e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569b4b40 0x561f569bf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:38892/0 (socket says 172.21.15.28:38892) 2024-04-24T08:15:13.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.532+0000 7f5e57e00640 1 -- 172.21.15.28:0/1280241459 learned_addr learned my addr 172.21.15.28:0/1280241459 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:13.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.532+0000 7f5e57400640 1 --2- 172.21.15.28:0/1280241459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569bc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.532+0000 7f5e5ce00640 1 --2- 172.21.15.28:0/1280241459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569bfb30 0x561f569bcfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.533+0000 7f5e5ce00640 1 -- 172.21.15.28:0/1280241459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569b4b40 msgr2=0x561f569bf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:13.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.533+0000 7f5e5ce00640 1 --2- 172.21.15.28:0/1280241459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569b4b40 0x561f569bf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.533+0000 7f5e5ce00640 1 -- 172.21.15.28:0/1280241459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 msgr2=0x561f569bc8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:13.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.533+0000 7f5e5ce00640 1 --2- 172.21.15.28:0/1280241459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569bc8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.533+0000 7f5e5ce00640 1 -- 172.21.15.28:0/1280241459 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f568e43f0 con 0x561f569bfb30 2024-04-24T08:15:13.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.534+0000 7f5e5ce00640 1 --2- 172.21.15.28:0/1280241459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569bfb30 0x561f569bcfe0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5e5001f040 tx=0x7f5e50016000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=132890b2cbc5c20c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:13.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.534+0000 7f5e42000640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:15:13.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.535+0000 7f5e42000640 1 -- 172.21.15.28:0/1280241459 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e50035030 con 0x561f569bfb30 2024-04-24T08:15:13.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.535+0000 7f5e42000640 1 -- 172.21.15.28:0/1280241459 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f5e500103d0 con 0x561f569bfb30 2024-04-24T08:15:13.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.535+0000 7f5e61b99740 1 -- 172.21.15.28:0/1280241459 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e50011d60 con 0x561f569bfb30 2024-04-24T08:15:13.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.535+0000 7f5e42000640 1 -- 172.21.15.28:0/1280241459 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5e5001d410 con 0x561f569bfb30 2024-04-24T08:15:13.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.536+0000 7f5e61b99740 10 client.26138 fetch_fsmap learned FSMap version 26 2024-04-24T08:15:13.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.536+0000 7f5e61b99740 1 -- 172.21.15.28:0/1280241459 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561f5662c3f0 con 0x561f569bfb30 2024-04-24T08:15:13.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.536+0000 7f5e42000640 1 -- 172.21.15.28:0/1280241459 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e50011d60 con 0x561f569bfb30 2024-04-24T08:15:13.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.537+0000 7f5e42000640 1 -- 172.21.15.28:0/1280241459 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f5e500114e0 con 0x561f569bfb30 2024-04-24T08:15:13.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.537+0000 7f5e61b99740 10 client.26138 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:15:13.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.537+0000 7f5e61b99740 10 client.26138 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:15:13.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.537+0000 7f5e61b99740 20 client.26138 populate_metadata read hostname 'smithi028' 2024-04-24T08:15:13.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.537+0000 7f5e61b99740 1 --2- 172.21.15.28:0/1280241459 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x561f569cfd30 0x561f569d2120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.537+0000 7f5e61b99740 4 client.26138 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:15:13.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.537+0000 7f5e61b99740 1 -- 172.21.15.28:0/1280241459 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561f56927bf0 con 0x561f569cfd30 2024-04-24T08:15:13.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.537+0000 7f5e57e00640 1 --2- 172.21.15.28:0/1280241459 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x561f569cfd30 0x561f569d2120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.538+0000 7f5e57e00640 1 --2- 172.21.15.28:0/1280241459 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x561f569cfd30 0x561f569d2120 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=53113121cf2002e server_cookie=276335cfa737682e in_seq=0 out_seq=0 2024-04-24T08:15:13.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.538+0000 7f5e42000640 10 client.26138 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:15:13.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.539+0000 7f5e42000640 1 -- 172.21.15.28:0/1280241459 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561f56927bf0 con 0x561f569cfd30 2024-04-24T08:15:13.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.540+0000 7f5e42000640 10 client.26138 handle_command_reply: tid=0 2024-04-24T08:15:13.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.540+0000 7f5e61b99740 1 client.26138 shutdown 2024-04-24T08:15:13.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.540+0000 7f5e61b99740 1 -- 172.21.15.28:0/1280241459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569bfb30 msgr2=0x561f569bcfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.540+0000 7f5e61b99740 1 --2- 172.21.15.28:0/1280241459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569bfb30 0x561f569bcfe0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f5e5001f040 tx=0x7f5e50016000 comp rx=0 tx=0).stop 2024-04-24T08:15:13.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.541+0000 7f5e61b99740 1 -- 172.21.15.28:0/1280241459 shutdown_connections 2024-04-24T08:15:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.541+0000 7f5e61b99740 1 --2- 172.21.15.28:0/1280241459 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x561f569cfd30 0x561f569d2120 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.541+0000 7f5e61b99740 1 --2- 172.21.15.28:0/1280241459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569bc8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.541+0000 7f5e61b99740 1 --2- 172.21.15.28:0/1280241459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569bfb30 0x561f569bcfe0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.541+0000 7f5e61b99740 1 --2- 172.21.15.28:0/1280241459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569b4b40 0x561f569bf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.541+0000 7f5e42000640 0 client.26138 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:15:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.541+0000 7f5e61b99740 1 -- 172.21.15.28:0/1280241459 >> 172.21.15.28:0/1280241459 conn(0x561f569b0e60 msgr2=0x561f569b6b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:13.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.542+0000 7f5e61b99740 1 -- 172.21.15.28:0/1280241459 shutdown_connections 2024-04-24T08:15:13.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.542+0000 7f5e61b99740 1 -- 172.21.15.28:0/1280241459 wait complete. 2024-04-24T08:15:13.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.542+0000 7f5e61b99740 20 client.26138 trim_cache size 0 max 16384 2024-04-24T08:15:13.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.571+0000 7f5e61b99740 1 Processor -- start 2024-04-24T08:15:13.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.571+0000 7f5e61b99740 1 -- start start 2024-04-24T08:15:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.572+0000 7f5e61b99740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569b4b40 0x561f569cb8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.572+0000 7f5e61b99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569bc7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.572+0000 7f5e61b99740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569bfb30 0x561f569bf0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.572+0000 7f5e61b99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e581bf4c0 con 0x561f569b5460 2024-04-24T08:15:13.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.572+0000 7f5e61b99740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5e581bcd90 con 0x561f569bfb30 2024-04-24T08:15:13.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.572+0000 7f5e61b99740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5e581babe0 con 0x561f569b4b40 2024-04-24T08:15:13.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.572+0000 7f5e57400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569bc7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.573+0000 7f5e57400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569bc7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:56774/0 (socket says 172.21.15.28:56774) 2024-04-24T08:15:13.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.573+0000 7f5e57400640 1 -- 172.21.15.28:0/1200617743 learned_addr learned my addr 172.21.15.28:0/1200617743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:13.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.574+0000 7f5e5ce00640 1 --2- 172.21.15.28:0/1200617743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569bfb30 0x561f569bf0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.575+0000 7f5e57e00640 1 --2- 172.21.15.28:0/1200617743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569b4b40 0x561f569cb8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.575+0000 7f5e57400640 1 -- 172.21.15.28:0/1200617743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569b4b40 msgr2=0x561f569cb8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.575+0000 7f5e57400640 1 --2- 172.21.15.28:0/1200617743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569b4b40 0x561f569cb8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.575+0000 7f5e57400640 1 -- 172.21.15.28:0/1200617743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569bfb30 msgr2=0x561f569bf0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.575+0000 7f5e57400640 1 --2- 172.21.15.28:0/1200617743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569bfb30 0x561f569bf0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.575+0000 7f5e57400640 1 -- 172.21.15.28:0/1200617743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561f566f67d0 con 0x561f569b5460 2024-04-24T08:15:13.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.576+0000 7f5e57400640 1 --2- 172.21.15.28:0/1200617743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569bc7a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009830 tx=0x7f5e44005da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee113f5ba4a2ca1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:13.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.576+0000 7f5e4cc00640 1 -- 172.21.15.28:0/1200617743 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e44037040 con 0x561f569b5460 2024-04-24T08:15:13.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.576+0000 7f5e4cc00640 1 -- 172.21.15.28:0/1200617743 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e440305d0 con 0x561f569b5460 2024-04-24T08:15:13.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.576+0000 7f5e4cc00640 1 -- 172.21.15.28:0/1200617743 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e44003830 con 0x561f569b5460 2024-04-24T08:15:13.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.578+0000 7f5e61b99740 1 -- 172.21.15.28:0/1200617743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 msgr2=0x561f569bc7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.578+0000 7f5e61b99740 1 --2- 172.21.15.28:0/1200617743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569bc7a0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5e44009830 tx=0x7f5e44005da0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.579+0000 7f5e61b99740 1 -- 172.21.15.28:0/1200617743 shutdown_connections 2024-04-24T08:15:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.579+0000 7f5e61b99740 1 --2- 172.21.15.28:0/1200617743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569bc7a0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.579+0000 7f5e61b99740 1 --2- 172.21.15.28:0/1200617743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569bfb30 0x561f569bf0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.579+0000 7f5e61b99740 1 --2- 172.21.15.28:0/1200617743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569b4b40 0x561f569cb8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.579+0000 7f5e61b99740 1 -- 172.21.15.28:0/1200617743 >> 172.21.15.28:0/1200617743 conn(0x561f569b0e60 msgr2=0x561f569bb8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:13.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.582+0000 7f5e61b99740 1 -- 172.21.15.28:0/1200617743 shutdown_connections 2024-04-24T08:15:13.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.583+0000 7f5e61b99740 1 -- 172.21.15.28:0/1200617743 wait complete. 2024-04-24T08:15:13.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.583+0000 7f5e61b99740 1 -- start start 2024-04-24T08:15:13.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.584+0000 7f5e61b99740 1 Processor -- start 2024-04-24T08:15:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.585+0000 7f5e61b99740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569b4b40 0x561f569c7de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.585+0000 7f5e61b99740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569c8320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.586+0000 7f5e61b99740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569bfb30 0x561f569b9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.586+0000 7f5e57e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569b4b40 0x561f569c7de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.586+0000 7f5e61b99740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5e581bb250 con 0x561f569b5460 2024-04-24T08:15:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.586+0000 7f5e61b99740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5e581bc6c0 con 0x561f569b4b40 2024-04-24T08:15:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.586+0000 7f5e61b99740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5e581bb920 con 0x561f569bfb30 2024-04-24T08:15:13.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.586+0000 7f5e61b99740 10 client.0 _ll_register_callbacks cb 0x7f5e50015070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.586+0000 7f5e57e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569b4b40 0x561f569c7de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51114/0 (socket says 172.21.15.28:51114) 2024-04-24T08:15:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.587+0000 7f5e57e00640 1 -- 172.21.15.28:0/2097093768 learned_addr learned my addr 172.21.15.28:0/2097093768 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.587+0000 7f5e5ce00640 1 --2- 172.21.15.28:0/2097093768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569bfb30 0x561f569b9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.587+0000 7f5e57400640 1 --2- 172.21.15.28:0/2097093768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569c8320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.587+0000 7f5e57e00640 1 -- 172.21.15.28:0/2097093768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569bfb30 msgr2=0x561f569b9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.587+0000 7f5e57e00640 1 --2- 172.21.15.28:0/2097093768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569bfb30 0x561f569b9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.587+0000 7f5e57e00640 1 -- 172.21.15.28:0/2097093768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 msgr2=0x561f569c8320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:13.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.588+0000 7f5e57e00640 1 --2- 172.21.15.28:0/2097093768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569c8320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.588+0000 7f5e57e00640 1 -- 172.21.15.28:0/2097093768 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561f569ba090 con 0x561f569b4b40 2024-04-24T08:15:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.588+0000 7f5e57400640 1 --2- 172.21.15.28:0/2097093768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.589+0000 7f5e57e00640 1 --2- 172.21.15.28:0/2097093768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569b4b40 0x561f569c7de0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5e480058c0 tx=0x7f5e480d07d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1226aacc0894173d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.589+0000 7f5e42000640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:15:13.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.589+0000 7f5e42000640 1 -- 172.21.15.28:0/2097093768 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5e48020040 con 0x561f569b4b40 2024-04-24T08:15:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.589+0000 7f5e61b99740 1 -- 172.21.15.28:0/2097093768 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5e440305d0 con 0x561f569b4b40 2024-04-24T08:15:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.589+0000 7f5e42000640 1 -- 172.21.15.28:0/2097093768 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f5e48098a70 con 0x561f569b4b40 2024-04-24T08:15:13.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.590+0000 7f5e42000640 1 -- 172.21.15.28:0/2097093768 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5e480a07b0 con 0x561f569b4b40 2024-04-24T08:15:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.590+0000 7f5e61b99740 10 client.26144 fetch_fsmap learned FSMap version 26 2024-04-24T08:15:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.590+0000 7f5e61b99740 1 -- 172.21.15.28:0/2097093768 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561f569c0770 con 0x561f569b4b40 2024-04-24T08:15:13.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.590+0000 7f5e42000640 1 -- 172.21.15.28:0/2097093768 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5e440305d0 con 0x561f569b4b40 2024-04-24T08:15:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.591+0000 7f5e42000640 1 -- 172.21.15.28:0/2097093768 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f5e48010460 con 0x561f569b4b40 2024-04-24T08:15:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.591+0000 7f5e61b99740 10 client.26144 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:15:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.591+0000 7f5e61b99740 10 client.26144 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:15:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.591+0000 7f5e61b99740 20 client.26144 populate_metadata read hostname 'smithi028' 2024-04-24T08:15:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.591+0000 7f5e61b99740 1 --2- 172.21.15.28:0/2097093768 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x561f56a36150 0x561f56a38540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.591+0000 7f5e61b99740 4 client.26144 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:15:13.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.591+0000 7f5e61b99740 1 -- 172.21.15.28:0/2097093768 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561f56921a60 con 0x561f56a36150 2024-04-24T08:15:13.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.593+0000 7f5e57400640 1 --2- 172.21.15.28:0/2097093768 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x561f56a36150 0x561f56a38540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.594+0000 7f5e57400640 1 --2- 172.21.15.28:0/2097093768 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x561f56a36150 0x561f56a38540 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=93232b1a073b25fc server_cookie=510f211dac5304c5 in_seq=0 out_seq=0 2024-04-24T08:15:13.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.594+0000 7f5e42000640 10 client.26144 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:15:13.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.594+0000 7f5e42000640 1 -- 172.21.15.28:0/2097093768 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x561f56921a60 con 0x561f56a36150 2024-04-24T08:15:13.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.594+0000 7f5e42000640 10 client.26144 handle_command_reply: tid=0 2024-04-24T08:15:13.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.594+0000 7f5e61b99740 1 client.26144 shutdown 2024-04-24T08:15:13.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.595+0000 7f5e61b99740 1 -- 172.21.15.28:0/2097093768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569b4b40 msgr2=0x561f569c7de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.595+0000 7f5e61b99740 1 --2- 172.21.15.28:0/2097093768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569b4b40 0x561f569c7de0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5e480058c0 tx=0x7f5e480d07d0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.597+0000 7f5e61b99740 1 -- 172.21.15.28:0/2097093768 shutdown_connections 2024-04-24T08:15:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.597+0000 7f5e61b99740 1 --2- 172.21.15.28:0/2097093768 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x561f56a36150 0x561f56a38540 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.597+0000 7f5e42000640 0 client.26144 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:15:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.597+0000 7f5e61b99740 1 --2- 172.21.15.28:0/2097093768 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561f569b5460 0x561f569c8320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.597+0000 7f5e61b99740 1 --2- 172.21.15.28:0/2097093768 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561f569bfb30 0x561f569b9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.597+0000 7f5e61b99740 1 --2- 172.21.15.28:0/2097093768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561f569b4b40 0x561f569c7de0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.598+0000 7f5e61b99740 1 -- 172.21.15.28:0/2097093768 >> 172.21.15.28:0/2097093768 conn(0x561f569b0e60 msgr2=0x561f569af6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.599+0000 7f5e61b99740 1 -- 172.21.15.28:0/2097093768 shutdown_connections 2024-04-24T08:15:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.599+0000 7f5e61b99740 1 -- 172.21.15.28:0/2097093768 wait complete. 2024-04-24T08:15:13.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.599+0000 7f5e61b99740 20 client.26144 trim_cache size 0 max 16384 2024-04-24T08:15:13.601 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:15:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.603+0000 7f5e5d800640 1 -- 172.21.15.28:0/282362491 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5e3c0a3e20 msgr2=0x7f5e3c0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.603+0000 7f5e5d800640 1 --2- 172.21.15.28:0/282362491 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5e3c0a3e20 0x7f5e3c0a62e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5e5001dfd0 tx=0x7f5e50010690 comp rx=0 tx=0).stop 2024-04-24T08:15:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.603+0000 7f5e5d800640 1 -- 172.21.15.28:0/282362491 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e58100e50 msgr2=0x7f5e58073810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:13.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.603+0000 7f5e5d800640 1 --2- 172.21.15.28:0/282362491 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e58100e50 0x7f5e58073810 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f5e4800d950 tx=0x7f5e4800de10 comp rx=0 tx=0).stop 2024-04-24T08:15:13.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.605+0000 7f5e5d800640 1 -- 172.21.15.28:0/282362491 shutdown_connections 2024-04-24T08:15:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.605+0000 7f5e5d800640 1 --2- 172.21.15.28:0/282362491 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5e3c0a3e20 0x7f5e3c0a62e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.605+0000 7f5e5d800640 1 --2- 172.21.15.28:0/282362491 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5e58101770 0x7f5e5806c890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.605+0000 7f5e5d800640 1 --2- 172.21.15.28:0/282362491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e58106250 0x7f5e5806cdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.605+0000 7f5e5d800640 1 --2- 172.21.15.28:0/282362491 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5e58100e50 0x7f5e58073810 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:13.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.605+0000 7f5e5d800640 1 -- 172.21.15.28:0/282362491 >> 172.21.15.28:0/282362491 conn(0x7f5e580fc880 msgr2=0x7f5e580fd920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.606+0000 7f5e5d800640 1 -- 172.21.15.28:0/282362491 shutdown_connections 2024-04-24T08:15:13.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:13.606+0000 7f5e5d800640 1 -- 172.21.15.28:0/282362491 wait complete. 2024-04-24T08:15:13.644 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:15:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.015+0000 7f117f400640 1 -- 172.21.15.28:0/2050972604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1178101780 msgr2=0x7f117810e4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.015+0000 7f117f400640 1 --2- 172.21.15.28:0/2050972604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1178101780 0x7f117810e4d0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f1164009900 tx=0x7f116402a400 comp rx=0 tx=0).stop 2024-04-24T08:15:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.022+0000 7f117f400640 1 -- 172.21.15.28:0/2050972604 shutdown_connections 2024-04-24T08:15:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.022+0000 7f117f400640 1 --2- 172.21.15.28:0/2050972604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1178101780 0x7f117810e4d0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.022+0000 7f117f400640 1 --2- 172.21.15.28:0/2050972604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f117810ea10 0x7f1178110e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.022+0000 7f117f400640 1 --2- 172.21.15.28:0/2050972604 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1178100e60 0x7f1178101240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.022+0000 7f117f400640 1 -- 172.21.15.28:0/2050972604 >> 172.21.15.28:0/2050972604 conn(0x7f1178078060 msgr2=0x7f1178078470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:14.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.024+0000 7f117f400640 1 -- 172.21.15.28:0/2050972604 shutdown_connections 2024-04-24T08:15:14.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.024+0000 7f117f400640 1 -- 172.21.15.28:0/2050972604 wait complete. 2024-04-24T08:15:14.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.025+0000 7f117f400640 1 Processor -- start 2024-04-24T08:15:14.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.026+0000 7f117f400640 1 -- start start 2024-04-24T08:15:14.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.027+0000 7f117f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1178100e60 0x7f1178073710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.028+0000 7f117f400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1178101780 0x7f1178073c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.028+0000 7f117f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f117810ea10 0x7f117806c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.028+0000 7f117f400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1178114650 con 0x7f1178101780 2024-04-24T08:15:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.028+0000 7f117f400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f11781144d0 con 0x7f117810ea10 2024-04-24T08:15:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.028+0000 7f117f400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f11781147d0 con 0x7f1178100e60 2024-04-24T08:15:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.029+0000 7f117d400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f117810ea10 0x7f117806c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.029+0000 7f1177e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1178101780 0x7f1178073c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.029+0000 7f117d400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f117810ea10 0x7f117806c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51132/0 (socket says 172.21.15.28:51132) 2024-04-24T08:15:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.029+0000 7f117d400640 1 -- 172.21.15.28:0/4277337167 learned_addr learned my addr 172.21.15.28:0/4277337167 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.029+0000 7f117d400640 1 -- 172.21.15.28:0/4277337167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1178100e60 msgr2=0x7f1178073710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.029+0000 7f117d400640 1 --2- 172.21.15.28:0/4277337167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1178100e60 0x7f1178073710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.029+0000 7f117d400640 1 -- 172.21.15.28:0/4277337167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1178101780 msgr2=0x7f1178073c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.030+0000 7f117d400640 1 --2- 172.21.15.28:0/4277337167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1178101780 0x7f1178073c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.030+0000 7f117d400640 1 -- 172.21.15.28:0/4277337167 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f117806d190 con 0x7f117810ea10 2024-04-24T08:15:14.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.030+0000 7f1177e00640 1 --2- 172.21.15.28:0/4277337167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1178101780 0x7f1178073c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:15:14.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.031+0000 7f117d400640 1 --2- 172.21.15.28:0/4277337167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f117810ea10 0x7f117806c8d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f117000cab0 tx=0x7f117000cf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.032+0000 7f1175600640 1 -- 172.21.15.28:0/4277337167 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117000adc0 con 0x7f117810ea10 2024-04-24T08:15:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.032+0000 7f1175600640 1 -- 172.21.15.28:0/4277337167 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f117000f930 con 0x7f117810ea10 2024-04-24T08:15:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.032+0000 7f1175600640 1 -- 172.21.15.28:0/4277337167 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1170018b10 con 0x7f117810ea10 2024-04-24T08:15:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.032+0000 7f117f400640 1 -- 172.21.15.28:0/4277337167 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f117806d480 con 0x7f117810ea10 2024-04-24T08:15:14.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.033+0000 7f117f400640 1 -- 172.21.15.28:0/4277337167 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1178071410 con 0x7f117810ea10 2024-04-24T08:15:14.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:13 smithi028 ceph-mon[27786]: pgmap v149: 97 pgs: 97 active+clean; 18 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 68 MiB/s wr, 11.99k op/s 2024-04-24T08:15:14.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.041+0000 7f118216c740 1 Processor -- start 2024-04-24T08:15:14.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.048+0000 7f1175600640 1 -- 172.21.15.28:0/4277337167 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f117000fae0 con 0x7f117810ea10 2024-04-24T08:15:14.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.049+0000 7f118216c740 1 -- start start 2024-04-24T08:15:14.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.051+0000 7f118216c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783c4160 0x5577783c4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.051+0000 7f118216c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4a80 0x5577783cec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.051+0000 7f118216c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 0x5577783d1540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.051+0000 7f118216c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5577783211d0 con 0x5577783c4a80 2024-04-24T08:15:14.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.051+0000 7f118216c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5577782c7450 con 0x5577783c4160 2024-04-24T08:15:14.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.051+0000 7f118216c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55777831d190 con 0x5577783cf150 2024-04-24T08:15:14.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.051+0000 7f117ca00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4a80 0x5577783cec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.051+0000 7f1177e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 0x5577783d1540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.051+0000 7f1177e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 0x5577783d1540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:38940/0 (socket says 172.21.15.28:38940) 2024-04-24T08:15:14.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.051+0000 7f1177e00640 1 -- 172.21.15.28:0/48339687 learned_addr learned my addr 172.21.15.28:0/48339687 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:14.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.052+0000 7f1177e00640 1 -- 172.21.15.28:0/48339687 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783c4160 msgr2=0x5577783c4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:14.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.052+0000 7f1177e00640 1 --2- 172.21.15.28:0/48339687 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783c4160 0x5577783c4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.052+0000 7f1177e00640 1 -- 172.21.15.28:0/48339687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4a80 msgr2=0x5577783cec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:14.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.052+0000 7f1177e00640 1 --2- 172.21.15.28:0/48339687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4a80 0x5577783cec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.052+0000 7f1177e00640 1 -- 172.21.15.28:0/48339687 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557778384310 con 0x5577783cf150 2024-04-24T08:15:14.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.053+0000 7f1175600640 1 --2- 172.21.15.28:0/4277337167 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f11600a3e20 0x7f11600a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.054+0000 7f1175600640 1 -- 172.21.15.28:0/4277337167 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f11700d0ef0 con 0x7f117810ea10 2024-04-24T08:15:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.054+0000 7f1177e00640 1 --2- 172.21.15.28:0/48339687 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 0x5577783d1540 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1164009690 tx=0x7f11640040f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c6003691580ba3a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.054+0000 7f1169c00640 1 -- 172.21.15.28:0/48339687 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1164041020 con 0x5577783cf150 2024-04-24T08:15:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.055+0000 7f117ca00640 1 --2- 172.21.15.28:0/4277337167 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f11600a3e20 0x7f11600a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.055+0000 7f1169c00640 1 -- 172.21.15.28:0/48339687 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1164003ad0 con 0x5577783cf150 2024-04-24T08:15:14.056 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.055+0000 7f1169c00640 1 -- 172.21.15.28:0/48339687 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1164035040 con 0x5577783cf150 2024-04-24T08:15:14.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.058+0000 7f118216c740 1 -- 172.21.15.28:0/48339687 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 msgr2=0x5577783d1540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.058+0000 7f118216c740 1 --2- 172.21.15.28:0/48339687 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 0x5577783d1540 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f1164009690 tx=0x7f11640040f0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.058+0000 7f118216c740 1 -- 172.21.15.28:0/48339687 shutdown_connections 2024-04-24T08:15:14.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.058+0000 7f118216c740 1 --2- 172.21.15.28:0/48339687 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4a80 0x5577783cec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.058+0000 7f118216c740 1 --2- 172.21.15.28:0/48339687 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 0x5577783d1540 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.059+0000 7f117ca00640 1 --2- 172.21.15.28:0/4277337167 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f11600a3e20 0x7f11600a62e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f116c00a500 tx=0x7f116c00a490 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:14.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.060+0000 7f118216c740 1 --2- 172.21.15.28:0/48339687 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783c4160 0x5577783c4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.060+0000 7f118216c740 1 -- 172.21.15.28:0/48339687 >> 172.21.15.28:0/48339687 conn(0x5577783c0480 msgr2=0x5577783c0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.060+0000 7f118216c740 1 -- 172.21.15.28:0/48339687 shutdown_connections 2024-04-24T08:15:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.063+0000 7f118216c740 1 -- 172.21.15.28:0/48339687 wait complete. 2024-04-24T08:15:14.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.063+0000 7f118216c740 1 -- start start 2024-04-24T08:15:14.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.064+0000 7f118216c740 1 Processor -- start 2024-04-24T08:15:14.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.065+0000 7f118216c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4160 0x5577783cec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.065+0000 7f118216c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783c4a80 0x5577783cbef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.065+0000 7f118216c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 0x5577783cc630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.065+0000 7f118216c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5577783d1f80 con 0x5577783c4160 2024-04-24T08:15:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.065+0000 7f118216c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5577782da810 con 0x5577783c4a80 2024-04-24T08:15:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.065+0000 7f118216c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5577783d2100 con 0x5577783cf150 2024-04-24T08:15:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.065+0000 7f118216c740 10 client.0 _ll_register_callbacks cb 0x5577782a8140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.066+0000 7f117d400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 0x5577783cc630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.066+0000 7f117d400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 0x5577783cc630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:38942/0 (socket says 172.21.15.28:38942) 2024-04-24T08:15:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.066+0000 7f117d400640 1 -- 172.21.15.28:0/2131195960 learned_addr learned my addr 172.21.15.28:0/2131195960 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.066+0000 7f117ca00640 1 --2- 172.21.15.28:0/2131195960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4160 0x5577783cec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.066+0000 7f1177e00640 1 --2- 172.21.15.28:0/2131195960 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783c4a80 0x5577783cbef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.067+0000 7f117d400640 1 -- 172.21.15.28:0/2131195960 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783c4a80 msgr2=0x5577783cbef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:14.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.067+0000 7f117d400640 1 --2- 172.21.15.28:0/2131195960 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783c4a80 0x5577783cbef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.067+0000 7f117d400640 1 -- 172.21.15.28:0/2131195960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4160 msgr2=0x5577783cec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:14.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.067+0000 7f117d400640 1 --2- 172.21.15.28:0/2131195960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4160 0x5577783cec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.067+0000 7f117d400640 1 -- 172.21.15.28:0/2131195960 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577782eb3f0 con 0x5577783cf150 2024-04-24T08:15:14.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.068+0000 7f117d400640 1 --2- 172.21.15.28:0/2131195960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 0x5577783cc630 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f11700a5550 tx=0x7f117009ca10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=85558785cf14d166 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:14.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.068+0000 7f115e000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:15:14.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.069+0000 7f115e000640 1 -- 172.21.15.28:0/2131195960 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11700b6060 con 0x5577783cf150 2024-04-24T08:15:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.069+0000 7f115e000640 1 -- 172.21.15.28:0/2131195960 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f11700a55f0 con 0x5577783cf150 2024-04-24T08:15:14.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.071+0000 7f118216c740 1 -- 172.21.15.28:0/2131195960 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1164003ad0 con 0x5577783cf150 2024-04-24T08:15:14.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.072+0000 7f115e000640 1 -- 172.21.15.28:0/2131195960 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f11700bb050 con 0x5577783cf150 2024-04-24T08:15:14.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.072+0000 7f118216c740 10 client.26191 fetch_fsmap learned FSMap version 26 2024-04-24T08:15:14.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.072+0000 7f118216c740 1 -- 172.21.15.28:0/2131195960 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5577780333f0 con 0x5577783cf150 2024-04-24T08:15:14.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.074+0000 7f115e000640 1 -- 172.21.15.28:0/2131195960 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1164003ad0 con 0x5577783cf150 2024-04-24T08:15:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.075+0000 7f115e000640 1 -- 172.21.15.28:0/2131195960 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1170004b40 con 0x5577783cf150 2024-04-24T08:15:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.075+0000 7f118216c740 10 client.26191 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:15:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.075+0000 7f118216c740 10 client.26191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:15:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.075+0000 7f118216c740 20 client.26191 populate_metadata read hostname 'smithi028' 2024-04-24T08:15:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.075+0000 7f118216c740 1 --2- 172.21.15.28:0/2131195960 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5577783df740 0x5577783e1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.076+0000 7f118216c740 4 client.26191 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:15:14.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.076+0000 7f118216c740 1 -- 172.21.15.28:0/2131195960 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55777832ebf0 con 0x5577783df740 2024-04-24T08:15:14.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.078+0000 7f117ca00640 1 --2- 172.21.15.28:0/2131195960 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5577783df740 0x5577783e1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.079+0000 7f115e000640 10 client.26191 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:14.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.079+0000 7f117ca00640 1 --2- 172.21.15.28:0/2131195960 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5577783df740 0x5577783e1bb0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53d952bbf2297af3 server_cookie=a03b96094f02ab26 in_seq=0 out_seq=0 2024-04-24T08:15:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.081+0000 7f115e000640 1 -- 172.21.15.28:0/2131195960 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55777832ebf0 con 0x5577783df740 2024-04-24T08:15:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.081+0000 7f115e000640 10 client.26191 handle_command_reply: tid=0 2024-04-24T08:15:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.081+0000 7f118216c740 1 client.26191 shutdown 2024-04-24T08:15:14.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.082+0000 7f118216c740 1 -- 172.21.15.28:0/2131195960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 msgr2=0x5577783cc630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.082+0000 7f118216c740 1 --2- 172.21.15.28:0/2131195960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 0x5577783cc630 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f11700a5550 tx=0x7f117009ca10 comp rx=0 tx=0).stop 2024-04-24T08:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.082+0000 7f118216c740 1 -- 172.21.15.28:0/2131195960 shutdown_connections 2024-04-24T08:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.082+0000 7f118216c740 1 --2- 172.21.15.28:0/2131195960 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5577783df740 0x5577783e1bb0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.082+0000 7f115e000640 0 client.26191 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.082+0000 7f118216c740 1 --2- 172.21.15.28:0/2131195960 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 0x5577783cc630 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.082+0000 7f118216c740 1 --2- 172.21.15.28:0/2131195960 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783c4a80 0x5577783cbef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.082+0000 7f118216c740 1 --2- 172.21.15.28:0/2131195960 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4160 0x5577783cec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.082+0000 7f118216c740 1 -- 172.21.15.28:0/2131195960 >> 172.21.15.28:0/2131195960 conn(0x5577783c0480 msgr2=0x5577783b6df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.083+0000 7f118216c740 1 -- 172.21.15.28:0/2131195960 shutdown_connections 2024-04-24T08:15:14.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.083+0000 7f118216c740 1 -- 172.21.15.28:0/2131195960 wait complete. 2024-04-24T08:15:14.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.084+0000 7f118216c740 20 client.26191 trim_cache size 0 max 16384 2024-04-24T08:15:14.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:13 smithi117 ceph-mon[32438]: pgmap v149: 97 pgs: 97 active+clean; 18 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 68 MiB/s wr, 11.99k op/s 2024-04-24T08:15:14.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.134+0000 7f118216c740 1 Processor -- start 2024-04-24T08:15:14.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.136+0000 7f118216c740 1 -- start start 2024-04-24T08:15:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.137+0000 7f118216c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4160 0x5577783dae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.137+0000 7f118216c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783c4a80 0x5577783ce730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.137+0000 7f118216c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 0x5577783e7480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.137+0000 7f118216c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11781b6bd0 con 0x5577783c4160 2024-04-24T08:15:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.137+0000 7f118216c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f11781b44a0 con 0x5577783c4a80 2024-04-24T08:15:14.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.137+0000 7f118216c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f11781070f0 con 0x5577783cf150 2024-04-24T08:15:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.138+0000 7f117d400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 0x5577783e7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.138+0000 7f1177e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783c4a80 0x5577783ce730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.138+0000 7f1177e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783c4a80 0x5577783ce730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51172/0 (socket says 172.21.15.28:51172) 2024-04-24T08:15:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.138+0000 7f1177e00640 1 -- 172.21.15.28:0/1147689511 learned_addr learned my addr 172.21.15.28:0/1147689511 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:14.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.139+0000 7f117ca00640 1 --2- 172.21.15.28:0/1147689511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4160 0x5577783dae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.140+0000 7f1177e00640 1 -- 172.21.15.28:0/1147689511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 msgr2=0x5577783e7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:14.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.140+0000 7f1177e00640 1 --2- 172.21.15.28:0/1147689511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 0x5577783e7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.140+0000 7f1177e00640 1 -- 172.21.15.28:0/1147689511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4160 msgr2=0x5577783dae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:14.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.140+0000 7f1177e00640 1 --2- 172.21.15.28:0/1147689511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4160 0x5577783dae40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.140+0000 7f1177e00640 1 -- 172.21.15.28:0/1147689511 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5577783c48b0 con 0x5577783c4a80 2024-04-24T08:15:14.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.141+0000 7f1177e00640 1 --2- 172.21.15.28:0/1147689511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783c4a80 0x5577783ce730 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f116402a9c0 tx=0x7f116403d700 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bdc293bb6bdfd93 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:14.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.142+0000 7f1169200640 1 -- 172.21.15.28:0/1147689511 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1164041030 con 0x5577783c4a80 2024-04-24T08:15:14.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.142+0000 7f1169200640 1 -- 172.21.15.28:0/1147689511 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f116404f070 con 0x5577783c4a80 2024-04-24T08:15:14.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.142+0000 7f1169200640 1 -- 172.21.15.28:0/1147689511 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1164035040 con 0x5577783c4a80 2024-04-24T08:15:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.144+0000 7f118216c740 1 -- 172.21.15.28:0/1147689511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783c4a80 msgr2=0x5577783ce730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.144+0000 7f118216c740 1 --2- 172.21.15.28:0/1147689511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783c4a80 0x5577783ce730 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f116402a9c0 tx=0x7f116403d700 comp rx=0 tx=0).stop 2024-04-24T08:15:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.145+0000 7f118216c740 1 -- 172.21.15.28:0/1147689511 shutdown_connections 2024-04-24T08:15:14.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.145+0000 7f118216c740 1 --2- 172.21.15.28:0/1147689511 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783cf150 0x5577783e7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.145+0000 7f118216c740 1 --2- 172.21.15.28:0/1147689511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783c4a80 0x5577783ce730 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.146+0000 7f118216c740 1 --2- 172.21.15.28:0/1147689511 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4160 0x5577783dae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.146+0000 7f118216c740 1 -- 172.21.15.28:0/1147689511 >> 172.21.15.28:0/1147689511 conn(0x5577783c0480 msgr2=0x5577783c0860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:14.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.146+0000 7f118216c740 1 -- 172.21.15.28:0/1147689511 shutdown_connections 2024-04-24T08:15:14.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.147+0000 7f118216c740 1 -- 172.21.15.28:0/1147689511 wait complete. 2024-04-24T08:15:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.148+0000 7f118216c740 1 -- start start 2024-04-24T08:15:14.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.148+0000 7f118216c740 1 Processor -- start 2024-04-24T08:15:14.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.150+0000 7f118216c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4160 0x5577783c9b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.150+0000 7f118216c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783c4a80 0x5577783d7350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.151+0000 7f118216c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783cf150 0x5577783d78d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.151+0000 7f117d400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783cf150 0x5577783d78d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.151+0000 7f117d400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783cf150 0x5577783d78d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51188/0 (socket says 172.21.15.28:51188) 2024-04-24T08:15:14.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.151+0000 7f117d400640 1 -- 172.21.15.28:0/1014039994 learned_addr learned my addr 172.21.15.28:0/1014039994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:14.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.151+0000 7f118216c740 1 -- 172.21.15.28:0/1014039994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f11781b2960 con 0x5577783c4160 2024-04-24T08:15:14.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.151+0000 7f118216c740 1 -- 172.21.15.28:0/1014039994 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f11781b3dd0 con 0x5577783cf150 2024-04-24T08:15:14.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.151+0000 7f118216c740 1 -- 172.21.15.28:0/1014039994 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f11781b3030 con 0x5577783c4a80 2024-04-24T08:15:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.153+0000 7f117d400640 1 -- 172.21.15.28:0/1014039994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783c4a80 msgr2=0x5577783d7350 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:15:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.153+0000 7f117d400640 1 --2- 172.21.15.28:0/1014039994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783c4a80 0x5577783d7350 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.153+0000 7f117d400640 1 -- 172.21.15.28:0/1014039994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4160 msgr2=0x5577783c9b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.153+0000 7f118216c740 10 client.0 _ll_register_callbacks cb 0x7f11700072a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.152+0000 7f117ca00640 1 --2- 172.21.15.28:0/1014039994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4160 0x5577783c9b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.154+0000 7f117d400640 1 --2- 172.21.15.28:0/1014039994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4160 0x5577783c9b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.154+0000 7f117d400640 1 -- 172.21.15.28:0/1014039994 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5577780fd7d0 con 0x5577783cf150 2024-04-24T08:15:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.154+0000 7f118216c740 1 -- 172.21.15.28:0/1014039994 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f116404f070 con 0x5577783cf150 2024-04-24T08:15:14.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.154+0000 7f117d400640 1 --2- 172.21.15.28:0/1014039994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783cf150 0x5577783d78d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f117009cda0 tx=0x7f117009ca80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c52a18db88e6dcae server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.157+0000 7f115e000640 10 client.26156 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:15:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.157+0000 7f115e000640 1 -- 172.21.15.28:0/1014039994 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11700b6040 con 0x5577783cf150 2024-04-24T08:15:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.157+0000 7f115e000640 1 -- 172.21.15.28:0/1014039994 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f11700860a0 con 0x5577783cf150 2024-04-24T08:15:14.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.157+0000 7f115e000640 1 -- 172.21.15.28:0/1014039994 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1170080080 con 0x5577783cf150 2024-04-24T08:15:14.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.157+0000 7f118216c740 10 client.26156 fetch_fsmap learned FSMap version 26 2024-04-24T08:15:14.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.157+0000 7f118216c740 1 -- 172.21.15.28:0/1014039994 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({fsmap=26}) -- 0x557778445ff0 con 0x5577783cf150 2024-04-24T08:15:14.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.159+0000 7f115e000640 1 -- 172.21.15.28:0/1014039994 <== mon.2 v2:172.21.15.117:3300/0 4 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f117000f450 con 0x5577783cf150 2024-04-24T08:15:14.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.159+0000 7f118216c740 10 client.26156 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:15:14.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.159+0000 7f118216c740 10 client.26156 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:15:14.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.159+0000 7f118216c740 20 client.26156 populate_metadata read hostname 'smithi028' 2024-04-24T08:15:14.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.159+0000 7f118216c740 1 --2- 172.21.15.28:0/1014039994 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557778446420 0x557778448890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:14.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.159+0000 7f118216c740 4 client.26156 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:15:14.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.159+0000 7f118216c740 1 -- 172.21.15.28:0/1014039994 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "damage ls"}) -- 0x557778328a60 con 0x557778446420 2024-04-24T08:15:14.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.160+0000 7f117ca00640 1 --2- 172.21.15.28:0/1014039994 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557778446420 0x557778448890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:14.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.161+0000 7f117ca00640 1 --2- 172.21.15.28:0/1014039994 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557778446420 0x557778448890 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3bfc1b09d8b6afb8 server_cookie=f7b08b1870398ec9 in_seq=0 out_seq=0 2024-04-24T08:15:14.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.161+0000 7f115e000640 10 client.26156 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:14.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.162+0000 7f115e000640 1 -- 172.21.15.28:0/1014039994 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x557778328a60 con 0x557778446420 2024-04-24T08:15:14.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.162+0000 7f115e000640 10 client.26156 handle_command_reply: tid=0 2024-04-24T08:15:14.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.162+0000 7f118216c740 1 client.26156 shutdown 2024-04-24T08:15:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.163+0000 7f118216c740 1 -- 172.21.15.28:0/1014039994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783cf150 msgr2=0x5577783d78d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.163+0000 7f118216c740 1 --2- 172.21.15.28:0/1014039994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783cf150 0x5577783d78d0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f117009cda0 tx=0x7f117009ca80 comp rx=0 tx=0).stop 2024-04-24T08:15:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.163+0000 7f118216c740 1 -- 172.21.15.28:0/1014039994 shutdown_connections 2024-04-24T08:15:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.163+0000 7f118216c740 1 --2- 172.21.15.28:0/1014039994 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x557778446420 0x557778448890 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.163+0000 7f118216c740 1 --2- 172.21.15.28:0/1014039994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5577783cf150 0x5577783d78d0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.163+0000 7f118216c740 1 --2- 172.21.15.28:0/1014039994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5577783c4a80 0x5577783d7350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.163+0000 7f118216c740 1 --2- 172.21.15.28:0/1014039994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5577783c4160 0x5577783c9b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.163+0000 7f118216c740 1 -- 172.21.15.28:0/1014039994 >> 172.21.15.28:0/1014039994 conn(0x5577783c0480 msgr2=0x5577783b6560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.163+0000 7f115e000640 0 client.26156 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.163+0000 7f118216c740 1 -- 172.21.15.28:0/1014039994 shutdown_connections 2024-04-24T08:15:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.164+0000 7f118216c740 1 -- 172.21.15.28:0/1014039994 wait complete. 2024-04-24T08:15:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.164+0000 7f118216c740 20 client.26156 trim_cache size 0 max 16384 2024-04-24T08:15:14.166 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:15:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.166+0000 7f117f400640 1 -- 172.21.15.28:0/4277337167 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f11600a3e20 msgr2=0x7f11600a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.166+0000 7f117f400640 1 --2- 172.21.15.28:0/4277337167 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f11600a3e20 0x7f11600a62e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f116c00a500 tx=0x7f116c00a490 comp rx=0 tx=0).stop 2024-04-24T08:15:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.166+0000 7f117f400640 1 -- 172.21.15.28:0/4277337167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f117810ea10 msgr2=0x7f117806c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.166+0000 7f117f400640 1 --2- 172.21.15.28:0/4277337167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f117810ea10 0x7f117806c8d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f117000cab0 tx=0x7f117000cf70 comp rx=0 tx=0).stop 2024-04-24T08:15:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.166+0000 7f117f400640 1 -- 172.21.15.28:0/4277337167 shutdown_connections 2024-04-24T08:15:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.166+0000 7f117f400640 1 --2- 172.21.15.28:0/4277337167 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f11600a3e20 0x7f11600a62e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.166+0000 7f117f400640 1 --2- 172.21.15.28:0/4277337167 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1178101780 0x7f1178073c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.166+0000 7f117f400640 1 --2- 172.21.15.28:0/4277337167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f117810ea10 0x7f117806c8d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.166+0000 7f117f400640 1 --2- 172.21.15.28:0/4277337167 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1178100e60 0x7f1178073710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.166+0000 7f117f400640 1 -- 172.21.15.28:0/4277337167 >> 172.21.15.28:0/4277337167 conn(0x7f1178078060 msgr2=0x7f117810fd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.167+0000 7f117f400640 1 -- 172.21.15.28:0/4277337167 shutdown_connections 2024-04-24T08:15:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:14.167+0000 7f117f400640 1 -- 172.21.15.28:0/4277337167 wait complete. 2024-04-24T08:15:14.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:13 smithi194 ceph-mon[32836]: pgmap v149: 97 pgs: 97 active+clean; 18 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 144 KiB/s rd, 68 MiB/s wr, 11.99k op/s 2024-04-24T08:15:15.210 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:15:15.210 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:15:15.211 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:15:15.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.556+0000 7f4dac400640 1 -- 172.21.15.28:0/4006034826 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4da4076e50 msgr2=0x7f4da4112940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.556+0000 7f4dac400640 1 --2- 172.21.15.28:0/4006034826 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4da4076e50 0x7f4da4112940 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4d90009930 tx=0x7f4d9002a460 comp rx=0 tx=0).stop 2024-04-24T08:15:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.557+0000 7f4dac400640 1 -- 172.21.15.28:0/4006034826 shutdown_connections 2024-04-24T08:15:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.557+0000 7f4dac400640 1 --2- 172.21.15.28:0/4006034826 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4da4112e80 0x7f4da4115270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.557+0000 7f4dac400640 1 --2- 172.21.15.28:0/4006034826 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4da4076e50 0x7f4da4112940 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.557+0000 7f4dac400640 1 --2- 172.21.15.28:0/4006034826 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da4076530 0x7f4da4076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.557+0000 7f4dac400640 1 -- 172.21.15.28:0/4006034826 >> 172.21.15.28:0/4006034826 conn(0x7f4da40ffc70 msgr2=0x7f4da4102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.557+0000 7f4dac400640 1 -- 172.21.15.28:0/4006034826 shutdown_connections 2024-04-24T08:15:15.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.557+0000 7f4dac400640 1 -- 172.21.15.28:0/4006034826 wait complete. 2024-04-24T08:15:15.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.559+0000 7f4dac400640 1 Processor -- start 2024-04-24T08:15:15.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.559+0000 7f4dac400640 1 -- start start 2024-04-24T08:15:15.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.560+0000 7f4dac400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da4076530 0x7f4da419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.560+0000 7f4dac400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4da4076e50 0x7f4da419f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.560+0000 7f4dac400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4da4112e80 0x7f4da41a38d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.560+0000 7f4dac400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4da4118a90 con 0x7f4da4076530 2024-04-24T08:15:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.560+0000 7f4dac400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4da4118910 con 0x7f4da4112e80 2024-04-24T08:15:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.560+0000 7f4dac400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4da4118c10 con 0x7f4da4076e50 2024-04-24T08:15:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.560+0000 7f4da9000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4da4076e50 0x7f4da419f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.560+0000 7f4daa400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4da4112e80 0x7f4da41a38d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.560+0000 7f4da9a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da4076530 0x7f4da419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.560+0000 7f4daa400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4da4112e80 0x7f4da41a38d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51206/0 (socket says 172.21.15.28:51206) 2024-04-24T08:15:15.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.560+0000 7f4daa400640 1 -- 172.21.15.28:0/4215128405 learned_addr learned my addr 172.21.15.28:0/4215128405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:15.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.561+0000 7f4daa400640 1 -- 172.21.15.28:0/4215128405 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4da4076e50 msgr2=0x7f4da419f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.561+0000 7f4daa400640 1 --2- 172.21.15.28:0/4215128405 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4da4076e50 0x7f4da419f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.561+0000 7f4daa400640 1 -- 172.21.15.28:0/4215128405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da4076530 msgr2=0x7f4da419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.561+0000 7f4daa400640 1 --2- 172.21.15.28:0/4215128405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da4076530 0x7f4da419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.561+0000 7f4daa400640 1 -- 172.21.15.28:0/4215128405 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4da41a4000 con 0x7f4da4112e80 2024-04-24T08:15:15.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.562+0000 7f4daa400640 1 --2- 172.21.15.28:0/4215128405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4da4112e80 0x7f4da41a38d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c00b3f0 tx=0x7f4d9c00b8b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:15.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.562+0000 7f4da2000640 1 -- 172.21.15.28:0/4215128405 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c015020 con 0x7f4da4112e80 2024-04-24T08:15:15.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.562+0000 7f4dac400640 1 -- 172.21.15.28:0/4215128405 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4da41a42f0 con 0x7f4da4112e80 2024-04-24T08:15:15.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.562+0000 7f4dac400640 1 -- 172.21.15.28:0/4215128405 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4da41abb80 con 0x7f4da4112e80 2024-04-24T08:15:15.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.562+0000 7f4da2000640 1 -- 172.21.15.28:0/4215128405 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d9c011d10 con 0x7f4da4112e80 2024-04-24T08:15:15.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.563+0000 7f4da2000640 1 -- 172.21.15.28:0/4215128405 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c00f2e0 con 0x7f4da4112e80 2024-04-24T08:15:15.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.569+0000 7f4daf20b740 1 Processor -- start 2024-04-24T08:15:15.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.569+0000 7f4da2000640 1 -- 172.21.15.28:0/4215128405 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4d9c018020 con 0x7f4da4112e80 2024-04-24T08:15:15.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.569+0000 7f4daf20b740 1 -- start start 2024-04-24T08:15:15.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.570+0000 7f4daf20b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb61090 0x564badb61470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.570+0000 7f4daf20b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 0x564badb6bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.570+0000 7f4daf20b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb6c080 0x564badb6e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.570+0000 7f4daf20b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564badac6e70 con 0x564badb61090 2024-04-24T08:15:15.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.570+0000 7f4daf20b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564bada6d380 con 0x564badb619b0 2024-04-24T08:15:15.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.570+0000 7f4daf20b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564badac2e30 con 0x564badb6c080 2024-04-24T08:15:15.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.570+0000 7f4da9a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 0x564badb6bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.571+0000 7f4da9a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 0x564badb6bb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51220/0 (socket says 172.21.15.28:51220) 2024-04-24T08:15:15.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.571+0000 7f4da9a00640 1 -- 172.21.15.28:0/3303970271 learned_addr learned my addr 172.21.15.28:0/3303970271 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:15.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.571+0000 7f4da9a00640 1 -- 172.21.15.28:0/3303970271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb6c080 msgr2=0x564badb6e470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:15:15.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.571+0000 7f4da9a00640 1 --2- 172.21.15.28:0/3303970271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb6c080 0x564badb6e470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.571+0000 7f4da9a00640 1 -- 172.21.15.28:0/3303970271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb61090 msgr2=0x564badb61470 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:15:15.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.571+0000 7f4da9a00640 1 --2- 172.21.15.28:0/3303970271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb61090 0x564badb61470 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.571+0000 7f4da9a00640 1 -- 172.21.15.28:0/3303970271 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564bad8a17e0 con 0x564badb619b0 2024-04-24T08:15:15.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.572+0000 7f4da9a00640 1 --2- 172.21.15.28:0/3303970271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 0x564badb6bb40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4d98003a20 tx=0x7f4d98003ee0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ccc4bf4ddecb303 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:15.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.572+0000 7f4d96a00640 1 -- 172.21.15.28:0/3303970271 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d98004860 con 0x564badb619b0 2024-04-24T08:15:15.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.572+0000 7f4d96a00640 1 -- 172.21.15.28:0/3303970271 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d98015070 con 0x564badb619b0 2024-04-24T08:15:15.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.572+0000 7f4d96a00640 1 -- 172.21.15.28:0/3303970271 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d98010900 con 0x564badb619b0 2024-04-24T08:15:15.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.573+0000 7f4daf20b740 1 -- 172.21.15.28:0/3303970271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 msgr2=0x564badb6bb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.573+0000 7f4daf20b740 1 --2- 172.21.15.28:0/3303970271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 0x564badb6bb40 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f4d98003a20 tx=0x7f4d98003ee0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.574+0000 7f4da2000640 1 --2- 172.21.15.28:0/4215128405 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4d8c0a3d20 0x7f4d8c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.574+0000 7f4da2000640 1 -- 172.21.15.28:0/4215128405 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f4d9c0cfd20 con 0x7f4da4112e80 2024-04-24T08:15:15.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.575+0000 7f4da9000640 1 --2- 172.21.15.28:0/4215128405 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4d8c0a3d20 0x7f4d8c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.576+0000 7f4da9000640 1 --2- 172.21.15.28:0/4215128405 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4d8c0a3d20 0x7f4d8c0a61e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4d9002a980 tx=0x7f4d90005fd0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:15.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.576+0000 7f4daf20b740 1 -- 172.21.15.28:0/3303970271 shutdown_connections 2024-04-24T08:15:15.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.576+0000 7f4daf20b740 1 --2- 172.21.15.28:0/3303970271 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb6c080 0x564badb6e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.576+0000 7f4daf20b740 1 --2- 172.21.15.28:0/3303970271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 0x564badb6bb40 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.576+0000 7f4daf20b740 1 --2- 172.21.15.28:0/3303970271 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb61090 0x564badb61470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.576+0000 7f4daf20b740 1 -- 172.21.15.28:0/3303970271 >> 172.21.15.28:0/3303970271 conn(0x564badb5d3b0 msgr2=0x564badb5d790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:15.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.576+0000 7f4daf20b740 1 -- 172.21.15.28:0/3303970271 shutdown_connections 2024-04-24T08:15:15.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.576+0000 7f4daf20b740 1 -- 172.21.15.28:0/3303970271 wait complete. 2024-04-24T08:15:15.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.576+0000 7f4daf20b740 1 -- start start 2024-04-24T08:15:15.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.577+0000 7f4daf20b740 1 Processor -- start 2024-04-24T08:15:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.579+0000 7f4daf20b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 0x564badb68e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.579+0000 7f4daf20b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 0x564badb69380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.579+0000 7f4daf20b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb6c080 0x564badb698c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.579+0000 7f4daf20b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x564badb7fe80 con 0x564badb6c080 2024-04-24T08:15:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.579+0000 7f4daf20b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564bada80660 con 0x564badb619b0 2024-04-24T08:15:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.579+0000 7f4daf20b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564badb80000 con 0x564badb61090 2024-04-24T08:15:15.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.579+0000 7f4daf20b740 10 client.0 _ll_register_callbacks cb 0x564bada6eaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:15.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.580+0000 7f4da9a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 0x564badb68e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.580+0000 7f4daa400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb6c080 0x564badb698c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.580+0000 7f4da9000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 0x564badb69380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.580+0000 7f4da9000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 0x564badb69380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51234/0 (socket says 172.21.15.28:51234) 2024-04-24T08:15:15.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.580+0000 7f4da9000640 1 -- 172.21.15.28:0/3525174154 learned_addr learned my addr 172.21.15.28:0/3525174154 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:15.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.581+0000 7f4daa400640 1 -- 172.21.15.28:0/3525174154 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 msgr2=0x564badb68e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:15.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.581+0000 7f4daa400640 1 --2- 172.21.15.28:0/3525174154 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 0x564badb68e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.581+0000 7f4daa400640 1 -- 172.21.15.28:0/3525174154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 msgr2=0x564badb69380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:15.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.581+0000 7f4daa400640 1 --2- 172.21.15.28:0/3525174154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 0x564badb69380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.581+0000 7f4daa400640 1 -- 172.21.15.28:0/3525174154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564bad901f70 con 0x564badb6c080 2024-04-24T08:15:15.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.582+0000 7f4daa400640 1 --2- 172.21.15.28:0/3525174154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb6c080 0x564badb698c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c0cd560 tx=0x7f4d9c010f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ceb3c4b7bdad21c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:15.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.582+0000 7f4d8b400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:15:15.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.582+0000 7f4d8b400640 1 -- 172.21.15.28:0/3525174154 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9c0a0160 con 0x564badb6c080 2024-04-24T08:15:15.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.582+0000 7f4d8b400640 1 -- 172.21.15.28:0/3525174154 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f4d9c0b52d0 con 0x564badb6c080 2024-04-24T08:15:15.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.582+0000 7f4daf20b740 1 -- 172.21.15.28:0/3525174154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d98015070 con 0x564badb6c080 2024-04-24T08:15:15.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.583+0000 7f4d8b400640 1 -- 172.21.15.28:0/3525174154 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4d9c0a4a20 con 0x564badb6c080 2024-04-24T08:15:15.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.583+0000 7f4daf20b740 10 client.15981 fetch_fsmap learned FSMap version 26 2024-04-24T08:15:15.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.583+0000 7f4daf20b740 1 -- 172.21.15.28:0/3525174154 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564bad799170 con 0x564badb6c080 2024-04-24T08:15:15.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.583+0000 7f4d8b400640 1 -- 172.21.15.28:0/3525174154 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d98015070 con 0x564badb6c080 2024-04-24T08:15:15.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.584+0000 7f4d8b400640 1 -- 172.21.15.28:0/3525174154 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4d9c086070 con 0x564badb6c080 2024-04-24T08:15:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.584+0000 7f4daf20b740 10 client.15981 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:15:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.584+0000 7f4daf20b740 10 client.15981 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:15:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.584+0000 7f4daf20b740 20 client.15981 populate_metadata read hostname 'smithi028' 2024-04-24T08:15:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.584+0000 7f4daf20b740 1 --2- 172.21.15.28:0/3525174154 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x564badb8d3b0 0x564badb8f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.585+0000 7f4daf20b740 4 client.15981 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:15:15.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.585+0000 7f4daf20b740 1 -- 172.21.15.28:0/3525174154 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564badad4890 con 0x564badb8d3b0 2024-04-24T08:15:15.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.585+0000 7f4da9a00640 1 --2- 172.21.15.28:0/3525174154 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x564badb8d3b0 0x564badb8f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.586+0000 7f4da9a00640 1 --2- 172.21.15.28:0/3525174154 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x564badb8d3b0 0x564badb8f7f0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=808749d961e7b31b server_cookie=17cd15d35373e3b4 in_seq=0 out_seq=0 2024-04-24T08:15:15.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.586+0000 7f4d8b400640 10 client.15981 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.591+0000 7f4d8b400640 1 -- 172.21.15.28:0/3525174154 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564badad4890 con 0x564badb8d3b0 2024-04-24T08:15:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.591+0000 7f4d8b400640 10 client.15981 handle_command_reply: tid=0 2024-04-24T08:15:15.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.593+0000 7f4daf20b740 1 client.15981 shutdown 2024-04-24T08:15:15.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.594+0000 7f4daf20b740 1 -- 172.21.15.28:0/3525174154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb6c080 msgr2=0x564badb698c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.594+0000 7f4daf20b740 1 --2- 172.21.15.28:0/3525174154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb6c080 0x564badb698c0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c0cd560 tx=0x7f4d9c010f70 comp rx=0 tx=0).stop 2024-04-24T08:15:15.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.595+0000 7f4daf20b740 1 -- 172.21.15.28:0/3525174154 shutdown_connections 2024-04-24T08:15:15.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.595+0000 7f4daf20b740 1 --2- 172.21.15.28:0/3525174154 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x564badb8d3b0 0x564badb8f7f0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.595+0000 7f4daf20b740 1 --2- 172.21.15.28:0/3525174154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb6c080 0x564badb698c0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.595+0000 7f4d8b400640 0 client.15981 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:15.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.595+0000 7f4daf20b740 1 --2- 172.21.15.28:0/3525174154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 0x564badb69380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.595+0000 7f4daf20b740 1 --2- 172.21.15.28:0/3525174154 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 0x564badb68e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.595+0000 7f4daf20b740 1 -- 172.21.15.28:0/3525174154 >> 172.21.15.28:0/3525174154 conn(0x564badb5d3b0 msgr2=0x564badb63070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:15.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.596+0000 7f4daf20b740 1 -- 172.21.15.28:0/3525174154 shutdown_connections 2024-04-24T08:15:15.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.597+0000 7f4daf20b740 1 -- 172.21.15.28:0/3525174154 wait complete. 2024-04-24T08:15:15.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.597+0000 7f4daf20b740 20 client.15981 trim_cache size 0 max 16384 2024-04-24T08:15:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.635+0000 7f4daf20b740 1 Processor -- start 2024-04-24T08:15:15.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.636+0000 7f4daf20b740 1 -- start start 2024-04-24T08:15:15.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.637+0000 7f4daf20b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 0x564badb88e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.637+0000 7f4daf20b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 0x564badb5c390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.637+0000 7f4daf20b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb6c080 0x564badb950c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.637+0000 7f4daf20b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d84011020 con 0x564badb6c080 2024-04-24T08:15:15.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.637+0000 7f4daf20b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8400e8f0 con 0x564badb619b0 2024-04-24T08:15:15.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.637+0000 7f4daf20b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8400c740 con 0x564badb61090 2024-04-24T08:15:15.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.638+0000 7f4da9a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 0x564badb88e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.638+0000 7f4da9a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 0x564badb88e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:38986/0 (socket says 172.21.15.28:38986) 2024-04-24T08:15:15.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.638+0000 7f4da9a00640 1 -- 172.21.15.28:0/3233103602 learned_addr learned my addr 172.21.15.28:0/3233103602 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.638+0000 7f4da9000640 1 --2- 172.21.15.28:0/3233103602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 0x564badb5c390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.639+0000 7f4daa400640 1 --2- 172.21.15.28:0/3233103602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb6c080 0x564badb950c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.639+0000 7f4da9a00640 1 -- 172.21.15.28:0/3233103602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 msgr2=0x564badb5c390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:15.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.639+0000 7f4da9a00640 1 --2- 172.21.15.28:0/3233103602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 0x564badb5c390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.639+0000 7f4da9a00640 1 -- 172.21.15.28:0/3233103602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb6c080 msgr2=0x564badb950c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:15.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.640+0000 7f4da9a00640 1 --2- 172.21.15.28:0/3233103602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb6c080 0x564badb950c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.640+0000 7f4da9a00640 1 -- 172.21.15.28:0/3233103602 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564badb617e0 con 0x564badb61090 2024-04-24T08:15:15.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.640+0000 7f4da9000640 1 --2- 172.21.15.28:0/3233103602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 0x564badb5c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:15:15.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.641+0000 7f4da9a00640 1 --2- 172.21.15.28:0/3233103602 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 0x564badb88e10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4d98010550 tx=0x7f4d98010580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=84536af5efdb2063 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:15.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.641+0000 7f4d96000640 1 -- 172.21.15.28:0/3233103602 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d9802b460 con 0x564badb61090 2024-04-24T08:15:15.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.641+0000 7f4d96000640 1 -- 172.21.15.28:0/3233103602 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d98015040 con 0x564badb61090 2024-04-24T08:15:15.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.641+0000 7f4d96000640 1 -- 172.21.15.28:0/3233103602 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d98010da0 con 0x564badb61090 2024-04-24T08:15:15.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.643+0000 7f4daf20b740 1 -- 172.21.15.28:0/3233103602 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 msgr2=0x564badb88e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.643+0000 7f4daf20b740 1 --2- 172.21.15.28:0/3233103602 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 0x564badb88e10 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4d98010550 tx=0x7f4d98010580 comp rx=0 tx=0).stop 2024-04-24T08:15:15.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.643+0000 7f4daf20b740 1 -- 172.21.15.28:0/3233103602 shutdown_connections 2024-04-24T08:15:15.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.643+0000 7f4daf20b740 1 --2- 172.21.15.28:0/3233103602 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb6c080 0x564badb950c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.643+0000 7f4daf20b740 1 --2- 172.21.15.28:0/3233103602 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb619b0 0x564badb5c390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.643+0000 7f4daf20b740 1 --2- 172.21.15.28:0/3233103602 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 0x564badb88e10 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.643+0000 7f4daf20b740 1 -- 172.21.15.28:0/3233103602 >> 172.21.15.28:0/3233103602 conn(0x564badb5d3b0 msgr2=0x564badb5bc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:15.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.644+0000 7f4daf20b740 1 -- 172.21.15.28:0/3233103602 shutdown_connections 2024-04-24T08:15:15.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.644+0000 7f4daf20b740 1 -- 172.21.15.28:0/3233103602 wait complete. 2024-04-24T08:15:15.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.644+0000 7f4daf20b740 1 -- start start 2024-04-24T08:15:15.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.645+0000 7f4daf20b740 1 Processor -- start 2024-04-24T08:15:15.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.646+0000 7f4daf20b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 0x564badb85330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.646+0000 7f4daf20b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb619b0 0x564badb85870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.646+0000 7f4daf20b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb6c080 0x564badb802e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.647+0000 7f4da9a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 0x564badb85330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.647+0000 7f4da9a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 0x564badb85330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:39002/0 (socket says 172.21.15.28:39002) 2024-04-24T08:15:15.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.647+0000 7f4da9000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb619b0 0x564badb85870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.647+0000 7f4da9a00640 1 -- 172.21.15.28:0/1314939692 learned_addr learned my addr 172.21.15.28:0/1314939692 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:15.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.647+0000 7f4daf20b740 1 -- 172.21.15.28:0/1314939692 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8400cdb0 con 0x564badb619b0 2024-04-24T08:15:15.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.647+0000 7f4daf20b740 1 -- 172.21.15.28:0/1314939692 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8400e220 con 0x564badb6c080 2024-04-24T08:15:15.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.647+0000 7f4daf20b740 1 -- 172.21.15.28:0/1314939692 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4d8400d480 con 0x564badb61090 2024-04-24T08:15:15.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.647+0000 7f4daf20b740 10 client.0 _ll_register_callbacks cb 0x7f4d9c098050 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:15.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.647+0000 7f4daa400640 1 --2- 172.21.15.28:0/1314939692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb6c080 0x564badb802e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.648+0000 7f4da9a00640 1 -- 172.21.15.28:0/1314939692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb6c080 msgr2=0x564badb802e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:15.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.648+0000 7f4da9a00640 1 --2- 172.21.15.28:0/1314939692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb6c080 0x564badb802e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.649+0000 7f4da9a00640 1 -- 172.21.15.28:0/1314939692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb619b0 msgr2=0x564badb85870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.649+0000 7f4da9a00640 1 --2- 172.21.15.28:0/1314939692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb619b0 0x564badb85870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.649+0000 7f4da9a00640 1 -- 172.21.15.28:0/1314939692 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564badb80ae0 con 0x564badb61090 2024-04-24T08:15:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.649+0000 7f4daa400640 1 --2- 172.21.15.28:0/1314939692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb6c080 0x564badb802e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:15:15.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.650+0000 7f4da9000640 1 --2- 172.21.15.28:0/1314939692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb619b0 0x564badb85870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:15:15.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.650+0000 7f4d8b400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:15:15.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.650+0000 7f4da9a00640 1 --2- 172.21.15.28:0/1314939692 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 0x564badb85330 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4d98006010 tx=0x7f4d980104b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93165f63f74a2153 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:15.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.651+0000 7f4d8b400640 1 -- 172.21.15.28:0/1314939692 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d98032070 con 0x564badb61090 2024-04-24T08:15:15.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.651+0000 7f4daf20b740 1 -- 172.21.15.28:0/1314939692 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d98015040 con 0x564badb61090 2024-04-24T08:15:15.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.651+0000 7f4d8b400640 1 -- 172.21.15.28:0/1314939692 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f4d98010d20 con 0x564badb61090 2024-04-24T08:15:15.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.652+0000 7f4d8b400640 1 -- 172.21.15.28:0/1314939692 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4d98004e50 con 0x564badb61090 2024-04-24T08:15:15.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.653+0000 7f4daf20b740 10 client.26209 fetch_fsmap learned FSMap version 26 2024-04-24T08:15:15.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.653+0000 7f4daf20b740 1 -- 172.21.15.28:0/1314939692 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564badbf43d0 con 0x564badb61090 2024-04-24T08:15:15.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.653+0000 7f4d8b400640 1 -- 172.21.15.28:0/1314939692 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d98015040 con 0x564badb61090 2024-04-24T08:15:15.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.655+0000 7f4d8b400640 1 -- 172.21.15.28:0/1314939692 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4d980051b0 con 0x564badb61090 2024-04-24T08:15:15.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.655+0000 7f4daf20b740 10 client.26209 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:15:15.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.656+0000 7f4daf20b740 10 client.26209 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:15:15.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.656+0000 7f4daf20b740 20 client.26209 populate_metadata read hostname 'smithi028' 2024-04-24T08:15:15.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.656+0000 7f4daf20b740 1 --2- 172.21.15.28:0/1314939692 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x564badb8d3b0 0x564badbf68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:15.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.656+0000 7f4daf20b740 4 client.26209 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:15:15.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.656+0000 7f4daf20b740 1 -- 172.21.15.28:0/1314939692 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x564badace700 con 0x564badb8d3b0 2024-04-24T08:15:15.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.657+0000 7f4da9000640 1 --2- 172.21.15.28:0/1314939692 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x564badb8d3b0 0x564badbf68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.659+0000 7f4da9000640 1 --2- 172.21.15.28:0/1314939692 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x564badb8d3b0 0x564badbf68e0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e5c76631751913f server_cookie=a409e866fccc983a in_seq=0 out_seq=0 2024-04-24T08:15:15.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.659+0000 7f4d8b400640 10 client.26209 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.662+0000 7f4d8b400640 1 -- 172.21.15.28:0/1314939692 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x564badace700 con 0x564badb8d3b0 2024-04-24T08:15:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.662+0000 7f4d8b400640 10 client.26209 handle_command_reply: tid=0 2024-04-24T08:15:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.662+0000 7f4daf20b740 1 client.26209 shutdown 2024-04-24T08:15:15.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.662+0000 7f4daf20b740 1 -- 172.21.15.28:0/1314939692 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 msgr2=0x564badb85330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.662+0000 7f4daf20b740 1 --2- 172.21.15.28:0/1314939692 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 0x564badb85330 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4d98006010 tx=0x7f4d980104b0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.663+0000 7f4daf20b740 1 -- 172.21.15.28:0/1314939692 shutdown_connections 2024-04-24T08:15:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.663+0000 7f4daf20b740 1 --2- 172.21.15.28:0/1314939692 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x564badb8d3b0 0x564badbf68e0 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.663+0000 7f4daf20b740 1 --2- 172.21.15.28:0/1314939692 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564badb619b0 0x564badb85870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.663+0000 7f4daf20b740 1 --2- 172.21.15.28:0/1314939692 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564badb6c080 0x564badb802e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.663+0000 7f4d8b400640 0 client.26209 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.663+0000 7f4daf20b740 1 --2- 172.21.15.28:0/1314939692 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564badb61090 0x564badb85330 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.663+0000 7f4daf20b740 1 -- 172.21.15.28:0/1314939692 >> 172.21.15.28:0/1314939692 conn(0x564badb5d3b0 msgr2=0x564badb63070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:15.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.663+0000 7f4daf20b740 1 -- 172.21.15.28:0/1314939692 shutdown_connections 2024-04-24T08:15:15.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.663+0000 7f4daf20b740 1 -- 172.21.15.28:0/1314939692 wait complete. 2024-04-24T08:15:15.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.664+0000 7f4daf20b740 20 client.26209 trim_cache size 0 max 16384 2024-04-24T08:15:15.665 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:15:15.665 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-24T08:15:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "e0e3930c-be83-4abb-8ef5-4d8186b7b565", 2024-04-24T08:15:15.666 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-24T08:15:15.666 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:15:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.666+0000 7f4dac400640 1 -- 172.21.15.28:0/4215128405 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4d8c0a3d20 msgr2=0x7f4d8c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.666+0000 7f4dac400640 1 --2- 172.21.15.28:0/4215128405 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4d8c0a3d20 0x7f4d8c0a61e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f4d9002a980 tx=0x7f4d90005fd0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.666+0000 7f4dac400640 1 -- 172.21.15.28:0/4215128405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4da4112e80 msgr2=0x7f4da41a38d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.666+0000 7f4dac400640 1 --2- 172.21.15.28:0/4215128405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4da4112e80 0x7f4da41a38d0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f4d9c00b3f0 tx=0x7f4d9c00b8b0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.666+0000 7f4dac400640 1 -- 172.21.15.28:0/4215128405 shutdown_connections 2024-04-24T08:15:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.666+0000 7f4dac400640 1 --2- 172.21.15.28:0/4215128405 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4d8c0a3d20 0x7f4d8c0a61e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.666+0000 7f4dac400640 1 --2- 172.21.15.28:0/4215128405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4da4112e80 0x7f4da41a38d0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.666+0000 7f4dac400640 1 --2- 172.21.15.28:0/4215128405 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4da4076e50 0x7f4da419f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.666+0000 7f4dac400640 1 --2- 172.21.15.28:0/4215128405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4da4076530 0x7f4da419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.666+0000 7f4dac400640 1 -- 172.21.15.28:0/4215128405 >> 172.21.15.28:0/4215128405 conn(0x7f4da40ffc70 msgr2=0x7f4da4102060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.666+0000 7f4dac400640 1 -- 172.21.15.28:0/4215128405 shutdown_connections 2024-04-24T08:15:15.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:15.667+0000 7f4dac400640 1 -- 172.21.15.28:0/4215128405 wait complete. 2024-04-24T08:15:15.700 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:15:15.700 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:15:16.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:15 smithi028 ceph-mon[27786]: pgmap v150: 97 pgs: 97 active+clean; 18 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 41 MiB/s wr, 7.27k op/s 2024-04-24T08:15:16.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:15 smithi117 ceph-mon[32438]: pgmap v150: 97 pgs: 97 active+clean; 18 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 41 MiB/s wr, 7.27k op/s 2024-04-24T08:15:16.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:15 smithi194 ceph-mon[32836]: pgmap v150: 97 pgs: 97 active+clean; 18 GiB data, 24 GiB used, 1.0 TiB / 1.0 TiB avail; 64 KiB/s rd, 41 MiB/s wr, 7.27k op/s 2024-04-24T08:15:17.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:16 smithi028 ceph-mon[27786]: scrub queued for path: / 2024-04-24T08:15:17.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:16 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:15:17.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:16 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:15:17.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:16 smithi117 ceph-mon[32438]: scrub queued for path: / 2024-04-24T08:15:17.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:16 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:15:17.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:16 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:15:17.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:16 smithi194 ceph-mon[32836]: scrub queued for path: / 2024-04-24T08:15:17.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:16 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:15:17.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:16 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:15:18.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:17 smithi028 ceph-mon[27786]: pgmap v151: 97 pgs: 97 active+clean; 18 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 43 MiB/s wr, 7.55k op/s 2024-04-24T08:15:18.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:17 smithi117 ceph-mon[32438]: pgmap v151: 97 pgs: 97 active+clean; 18 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 43 MiB/s wr, 7.55k op/s 2024-04-24T08:15:18.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:17 smithi194 ceph-mon[32836]: pgmap v151: 97 pgs: 97 active+clean; 18 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 43 MiB/s wr, 7.55k op/s 2024-04-24T08:15:20.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:19 smithi117 ceph-mon[32438]: pgmap v152: 97 pgs: 97 active+clean; 18 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 58 MiB/s wr, 10.23k op/s 2024-04-24T08:15:20.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:19 smithi194 ceph-mon[32836]: pgmap v152: 97 pgs: 97 active+clean; 18 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 58 MiB/s wr, 10.23k op/s 2024-04-24T08:15:20.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:19 smithi028 ceph-mon[27786]: pgmap v152: 97 pgs: 97 active+clean; 18 GiB data, 25 GiB used, 1.0 TiB / 1.0 TiB avail; 78 KiB/s rd, 58 MiB/s wr, 10.23k op/s 2024-04-24T08:15:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.616+0000 7fdfde800640 1 -- 172.21.15.28:0/967150810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd80742d0 msgr2=0x7fdfd80746b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.616+0000 7fdfde800640 1 --2- 172.21.15.28:0/967150810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd80742d0 0x7fdfd80746b0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8009860 tx=0x7fdfc802a3d0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.617+0000 7fdfde800640 1 -- 172.21.15.28:0/967150810 shutdown_connections 2024-04-24T08:15:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.617+0000 7fdfde800640 1 --2- 172.21.15.28:0/967150810 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd8105f50 0x7fdfd81150e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.617+0000 7fdfde800640 1 --2- 172.21.15.28:0/967150810 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfd8074bf0 0x7fdfd8112b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.617+0000 7fdfde800640 1 --2- 172.21.15.28:0/967150810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd80742d0 0x7fdfd80746b0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.617+0000 7fdfde800640 1 -- 172.21.15.28:0/967150810 >> 172.21.15.28:0/967150810 conn(0x7fdfd8079450 msgr2=0x7fdfd8079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.617+0000 7fdfde800640 1 -- 172.21.15.28:0/967150810 shutdown_connections 2024-04-24T08:15:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.617+0000 7fdfde800640 1 -- 172.21.15.28:0/967150810 wait complete. 2024-04-24T08:15:20.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.618+0000 7fdfde800640 1 Processor -- start 2024-04-24T08:15:20.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.620+0000 7fdfde800640 1 -- start start 2024-04-24T08:15:20.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.620+0000 7fdfde800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd80742d0 0x7fdfd8114da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.620+0000 7fdfde800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd8074bf0 0x7fdfd8110e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.620+0000 7fdfde800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfd8111370 0x7fdfd8111820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.620+0000 7fdfde800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd810cd60 con 0x7fdfd8074bf0 2024-04-24T08:15:20.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.620+0000 7fdfde800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd810cbe0 con 0x7fdfd80742d0 2024-04-24T08:15:20.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.620+0000 7fdfde800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd810cee0 con 0x7fdfd8111370 2024-04-24T08:15:20.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.621+0000 7fdfd7400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd80742d0 0x7fdfd8114da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.621+0000 7fdfd7400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd80742d0 0x7fdfd8114da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:46632/0 (socket says 172.21.15.28:46632) 2024-04-24T08:15:20.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.621+0000 7fdfd7400640 1 -- 172.21.15.28:0/2388002062 learned_addr learned my addr 172.21.15.28:0/2388002062 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:20.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.621+0000 7fdfd7e00640 1 --2- 172.21.15.28:0/2388002062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfd8111370 0x7fdfd8111820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.621+0000 7fdfd7e00640 1 -- 172.21.15.28:0/2388002062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd80742d0 msgr2=0x7fdfd8114da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.621+0000 7fdfd7e00640 1 --2- 172.21.15.28:0/2388002062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd80742d0 0x7fdfd8114da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.621+0000 7fdfd7e00640 1 -- 172.21.15.28:0/2388002062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd8074bf0 msgr2=0x7fdfd8110e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.622+0000 7fdfd6a00640 1 --2- 172.21.15.28:0/2388002062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd8074bf0 0x7fdfd8110e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.622+0000 7fdfd7e00640 1 --2- 172.21.15.28:0/2388002062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd8074bf0 0x7fdfd8110e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.622+0000 7fdfd7e00640 1 -- 172.21.15.28:0/2388002062 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfd81a2df0 con 0x7fdfd8111370 2024-04-24T08:15:20.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.622+0000 7fdfd7400640 1 --2- 172.21.15.28:0/2388002062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd80742d0 0x7fdfd8114da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:15:20.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.622+0000 7fdfd7e00640 1 --2- 172.21.15.28:0/2388002062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfd8111370 0x7fdfd8111820 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdfd000e940 tx=0x7fdfd000ee00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:20.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.623+0000 7fdfcfe00640 1 -- 172.21.15.28:0/2388002062 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd0004d20 con 0x7fdfd8111370 2024-04-24T08:15:20.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.623+0000 7fdfcfe00640 1 -- 172.21.15.28:0/2388002062 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfd0005020 con 0x7fdfd8111370 2024-04-24T08:15:20.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.623+0000 7fdfde800640 1 -- 172.21.15.28:0/2388002062 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfd81a30e0 con 0x7fdfd8111370 2024-04-24T08:15:20.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.623+0000 7fdfcfe00640 1 -- 172.21.15.28:0/2388002062 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd00116b0 con 0x7fdfd8111370 2024-04-24T08:15:20.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.623+0000 7fdfde800640 1 -- 172.21.15.28:0/2388002062 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfd81a3630 con 0x7fdfd8111370 2024-04-24T08:15:20.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.632+0000 7fdfe1662740 1 Processor -- start 2024-04-24T08:15:20.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.634+0000 7fdfcfe00640 1 -- 172.21.15.28:0/2388002062 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fdfd0005540 con 0x7fdfd8111370 2024-04-24T08:15:20.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.637+0000 7fdfcfe00640 1 --2- 172.21.15.28:0/2388002062 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdfbc0a3e20 0x7fdfbc0a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.637+0000 7fdfcfe00640 1 -- 172.21.15.28:0/2388002062 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fdfd00d08a0 con 0x7fdfd8111370 2024-04-24T08:15:20.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.638+0000 7fdfd6a00640 1 --2- 172.21.15.28:0/2388002062 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdfbc0a3e20 0x7fdfbc0a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.638+0000 7fdfe1662740 1 -- start start 2024-04-24T08:15:20.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.638+0000 7fdfe1662740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235bf6860 0x556235bf6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.638+0000 7fdfe1662740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235bf7180 0x556235c01310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.638+0000 7fdfe1662740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235c01850 0x556235c03c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.639+0000 7fdfd7400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235bf6860 0x556235bf6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.639+0000 7fdfe1662740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556235b5c1f0 con 0x556235c01850 2024-04-24T08:15:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.639+0000 7fdfe1662740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556235b02350 con 0x556235bf6860 2024-04-24T08:15:20.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.639+0000 7fdfe1662740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556235b581b0 con 0x556235bf7180 2024-04-24T08:15:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.639+0000 7fdfd7400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235bf6860 0x556235bf6c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:46640/0 (socket says 172.21.15.28:46640) 2024-04-24T08:15:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.639+0000 7fdfd7400640 1 -- 172.21.15.28:0/2896215991 learned_addr learned my addr 172.21.15.28:0/2896215991 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.640+0000 7fdfd7400640 1 -- 172.21.15.28:0/2896215991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235bf7180 msgr2=0x556235c01310 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:15:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.640+0000 7fdfd7400640 1 --2- 172.21.15.28:0/2896215991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235bf7180 0x556235c01310 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.640+0000 7fdfd7400640 1 -- 172.21.15.28:0/2896215991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235c01850 msgr2=0x556235c03c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:20.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.640+0000 7fdfd7400640 1 --2- 172.21.15.28:0/2896215991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235c01850 0x556235c03c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.640+0000 7fdfd7400640 1 -- 172.21.15.28:0/2896215991 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556235b56f50 con 0x556235bf6860 2024-04-24T08:15:20.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.640+0000 7fdfd7400640 1 --2- 172.21.15.28:0/2896215991 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235bf6860 0x556235bf6c40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8030f90 tx=0x7fdfc8030fc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65c84807e8c25950 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:20.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.641+0000 7fdfccc00640 1 -- 172.21.15.28:0/2896215991 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8030810 con 0x556235bf6860 2024-04-24T08:15:20.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.641+0000 7fdfccc00640 1 -- 172.21.15.28:0/2896215991 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc8041030 con 0x556235bf6860 2024-04-24T08:15:20.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.641+0000 7fdfccc00640 1 -- 172.21.15.28:0/2896215991 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfc8004920 con 0x556235bf6860 2024-04-24T08:15:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.643+0000 7fdfe1662740 1 -- 172.21.15.28:0/2896215991 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235bf6860 msgr2=0x556235bf6c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.643+0000 7fdfd6a00640 1 --2- 172.21.15.28:0/2388002062 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdfbc0a3e20 0x7fdfbc0a62e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4005fd0 tx=0x7fdfc4005850 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.643+0000 7fdfe1662740 1 --2- 172.21.15.28:0/2896215991 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235bf6860 0x556235bf6c40 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fdfc8030f90 tx=0x7fdfc8030fc0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.644+0000 7fdfe1662740 1 -- 172.21.15.28:0/2896215991 shutdown_connections 2024-04-24T08:15:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.644+0000 7fdfe1662740 1 --2- 172.21.15.28:0/2896215991 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235c01850 0x556235c03c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.644+0000 7fdfe1662740 1 --2- 172.21.15.28:0/2896215991 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235bf7180 0x556235c01310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.644+0000 7fdfe1662740 1 --2- 172.21.15.28:0/2896215991 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235bf6860 0x556235bf6c40 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.644+0000 7fdfe1662740 1 -- 172.21.15.28:0/2896215991 >> 172.21.15.28:0/2896215991 conn(0x556235bf2b80 msgr2=0x556235bf2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.645+0000 7fdfe1662740 1 -- 172.21.15.28:0/2896215991 shutdown_connections 2024-04-24T08:15:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.645+0000 7fdfe1662740 1 -- 172.21.15.28:0/2896215991 wait complete. 2024-04-24T08:15:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.646+0000 7fdfe1662740 1 -- start start 2024-04-24T08:15:20.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.646+0000 7fdfe1662740 1 Processor -- start 2024-04-24T08:15:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.648+0000 7fdfe1662740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235bf6860 0x556235bfe880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.648+0000 7fdfe1662740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235bf7180 0x556235bfedc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.648+0000 7fdfe1662740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235c01850 0x556235bff300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.649+0000 7fdfd7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235bf6860 0x556235bfe880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.649+0000 7fdfe1662740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x556235c04630 con 0x556235bf7180 2024-04-24T08:15:20.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.649+0000 7fdfe1662740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x556235b15690 con 0x556235c01850 2024-04-24T08:15:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.649+0000 7fdfe1662740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x556235c047b0 con 0x556235bf6860 2024-04-24T08:15:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.649+0000 7fdfe1662740 10 client.0 _ll_register_callbacks cb 0x556235b51650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.649+0000 7fdfd7e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235c01850 0x556235bff300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.649+0000 7fdfd7e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235c01850 0x556235bff300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:46646/0 (socket says 172.21.15.28:46646) 2024-04-24T08:15:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.649+0000 7fdfd7e00640 1 -- 172.21.15.28:0/3729588976 learned_addr learned my addr 172.21.15.28:0/3729588976 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:20.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.650+0000 7fdfd6a00640 1 --2- 172.21.15.28:0/3729588976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235bf7180 0x556235bfedc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.651+0000 7fdfd7e00640 1 -- 172.21.15.28:0/3729588976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235bf6860 msgr2=0x556235bfe880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.651+0000 7fdfd7e00640 1 --2- 172.21.15.28:0/3729588976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235bf6860 0x556235bfe880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.651+0000 7fdfd7e00640 1 -- 172.21.15.28:0/3729588976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235bf7180 msgr2=0x556235bfedc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:20.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.651+0000 7fdfd7e00640 1 --2- 172.21.15.28:0/3729588976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235bf7180 0x556235bfedc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.651+0000 7fdfd7e00640 1 -- 172.21.15.28:0/3729588976 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556235b14920 con 0x556235c01850 2024-04-24T08:15:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.651+0000 7fdfd6a00640 1 --2- 172.21.15.28:0/3729588976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235bf7180 0x556235bfedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:15:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.654+0000 7fdfd7e00640 1 --2- 172.21.15.28:0/3729588976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235c01850 0x556235bff300 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdfd00cdfd0 tx=0x7fdfd009c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6383d26b38c94240 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.655+0000 7fdfc1600640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:15:20.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.655+0000 7fdfc1600640 1 -- 172.21.15.28:0/3729588976 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd001f040 con 0x556235c01850 2024-04-24T08:15:20.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.656+0000 7fdfc1600640 1 -- 172.21.15.28:0/3729588976 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fdfd00ce1f0 con 0x556235c01850 2024-04-24T08:15:20.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.656+0000 7fdfe1662740 1 -- 172.21.15.28:0/3729588976 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfc8041030 con 0x556235c01850 2024-04-24T08:15:20.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.656+0000 7fdfc1600640 1 -- 172.21.15.28:0/3729588976 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdfd009c7c0 con 0x556235c01850 2024-04-24T08:15:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.656+0000 7fdfe1662740 10 client.26183 fetch_fsmap learned FSMap version 26 2024-04-24T08:15:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.656+0000 7fdfe1662740 1 -- 172.21.15.28:0/3729588976 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5562359387d0 con 0x556235c01850 2024-04-24T08:15:20.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.657+0000 7fdfc1600640 1 -- 172.21.15.28:0/3729588976 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfc8041030 con 0x556235c01850 2024-04-24T08:15:20.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.657+0000 7fdfc1600640 1 -- 172.21.15.28:0/3729588976 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fdfd000f040 con 0x556235c01850 2024-04-24T08:15:20.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.660+0000 7fdfe1662740 10 client.26183 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:15:20.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.660+0000 7fdfe1662740 10 client.26183 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:15:20.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.660+0000 7fdfe1662740 20 client.26183 populate_metadata read hostname 'smithi028' 2024-04-24T08:15:20.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.660+0000 7fdfe1662740 1 --2- 172.21.15.28:0/3729588976 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x556235c11ba0 0x556235c13fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.660+0000 7fdfe1662740 4 client.26183 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:15:20.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.660+0000 7fdfe1662740 1 -- 172.21.15.28:0/3729588976 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556235b69c10 con 0x556235c11ba0 2024-04-24T08:15:20.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.661+0000 7fdfd7400640 1 --2- 172.21.15.28:0/3729588976 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x556235c11ba0 0x556235c13fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.665+0000 7fdfc1600640 10 client.26183 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:20.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.665+0000 7fdfd7400640 1 --2- 172.21.15.28:0/3729588976 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x556235c11ba0 0x556235c13fc0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30560eec82840abe server_cookie=bb160a9a5bceed41 in_seq=0 out_seq=0 2024-04-24T08:15:20.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.666+0000 7fdfc1600640 1 -- 172.21.15.28:0/3729588976 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556235b69c10 con 0x556235c11ba0 2024-04-24T08:15:20.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.666+0000 7fdfc1600640 10 client.26183 handle_command_reply: tid=0 2024-04-24T08:15:20.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.667+0000 7fdfe1662740 1 client.26183 shutdown 2024-04-24T08:15:20.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.667+0000 7fdfe1662740 1 -- 172.21.15.28:0/3729588976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235c01850 msgr2=0x556235bff300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.667+0000 7fdfe1662740 1 --2- 172.21.15.28:0/3729588976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235c01850 0x556235bff300 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fdfd00cdfd0 tx=0x7fdfd009c6c0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.668+0000 7fdfe1662740 1 -- 172.21.15.28:0/3729588976 shutdown_connections 2024-04-24T08:15:20.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.668+0000 7fdfe1662740 1 --2- 172.21.15.28:0/3729588976 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x556235c11ba0 0x556235c13fc0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.668+0000 7fdfe1662740 1 --2- 172.21.15.28:0/3729588976 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235bf7180 0x556235bfedc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.668+0000 7fdfe1662740 1 --2- 172.21.15.28:0/3729588976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235c01850 0x556235bff300 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.668+0000 7fdfe1662740 1 --2- 172.21.15.28:0/3729588976 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235bf6860 0x556235bfe880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.668+0000 7fdfe1662740 1 -- 172.21.15.28:0/3729588976 >> 172.21.15.28:0/3729588976 conn(0x556235bf2b80 msgr2=0x556235bf8980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:20.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.670+0000 7fdfc1600640 0 client.26183 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:20.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.670+0000 7fdfe1662740 1 -- 172.21.15.28:0/3729588976 shutdown_connections 2024-04-24T08:15:20.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.670+0000 7fdfe1662740 1 -- 172.21.15.28:0/3729588976 wait complete. 2024-04-24T08:15:20.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.670+0000 7fdfe1662740 20 client.26183 trim_cache size 0 max 16384 2024-04-24T08:15:20.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.717+0000 7fdfe1662740 1 Processor -- start 2024-04-24T08:15:20.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.718+0000 7fdfe1662740 1 -- start start 2024-04-24T08:15:20.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.718+0000 7fdfe1662740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235bf6860 0x556235c0d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.718+0000 7fdfe1662740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235bf7180 0x556235bf2f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.718+0000 7fdfe1662740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235c01850 0x556235c19890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.718+0000 7fdfe1662740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd81ae560 con 0x556235bf6860 2024-04-24T08:15:20.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.718+0000 7fdfe1662740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd81abe30 con 0x556235c01850 2024-04-24T08:15:20.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.718+0000 7fdfe1662740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd81a9c80 con 0x556235bf7180 2024-04-24T08:15:20.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.719+0000 7fdfd7e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235c01850 0x556235c19890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.719+0000 7fdfd7e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235c01850 0x556235c19890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:46656/0 (socket says 172.21.15.28:46656) 2024-04-24T08:15:20.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.719+0000 7fdfd7e00640 1 -- 172.21.15.28:0/2763917844 learned_addr learned my addr 172.21.15.28:0/2763917844 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:20.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.719+0000 7fdfd6a00640 1 --2- 172.21.15.28:0/2763917844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235bf7180 0x556235bf2f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.719+0000 7fdfd7400640 1 --2- 172.21.15.28:0/2763917844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235bf6860 0x556235c0d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.719+0000 7fdfd7e00640 1 -- 172.21.15.28:0/2763917844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235bf7180 msgr2=0x556235bf2f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:20.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.719+0000 7fdfd7e00640 1 --2- 172.21.15.28:0/2763917844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235bf7180 0x556235bf2f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.719+0000 7fdfd7e00640 1 -- 172.21.15.28:0/2763917844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235bf6860 msgr2=0x556235c0d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:20.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.719+0000 7fdfd7e00640 1 --2- 172.21.15.28:0/2763917844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235bf6860 0x556235c0d4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.719+0000 7fdfd7e00640 1 -- 172.21.15.28:0/2763917844 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55623586e3f0 con 0x556235c01850 2024-04-24T08:15:20.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.719+0000 7fdfd7e00640 1 --2- 172.21.15.28:0/2763917844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235c01850 0x556235c19890 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdfd009c750 tx=0x7fdfd0005940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7e88ced61565578b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:20.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.720+0000 7fdfc3e00640 1 -- 172.21.15.28:0/2763917844 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd001f040 con 0x556235c01850 2024-04-24T08:15:20.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.720+0000 7fdfc3e00640 1 -- 172.21.15.28:0/2763917844 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfd00a0b70 con 0x556235c01850 2024-04-24T08:15:20.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.720+0000 7fdfc3e00640 1 -- 172.21.15.28:0/2763917844 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd001f040 con 0x556235c01850 2024-04-24T08:15:20.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.721+0000 7fdfe1662740 1 -- 172.21.15.28:0/2763917844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235c01850 msgr2=0x556235c19890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.721+0000 7fdfe1662740 1 --2- 172.21.15.28:0/2763917844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235c01850 0x556235c19890 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fdfd009c750 tx=0x7fdfd0005940 comp rx=0 tx=0).stop 2024-04-24T08:15:20.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.721+0000 7fdfe1662740 1 -- 172.21.15.28:0/2763917844 shutdown_connections 2024-04-24T08:15:20.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.721+0000 7fdfe1662740 1 --2- 172.21.15.28:0/2763917844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235c01850 0x556235c19890 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.721+0000 7fdfe1662740 1 --2- 172.21.15.28:0/2763917844 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235bf7180 0x556235bf2f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.721+0000 7fdfe1662740 1 --2- 172.21.15.28:0/2763917844 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235bf6860 0x556235c0d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.721+0000 7fdfe1662740 1 -- 172.21.15.28:0/2763917844 >> 172.21.15.28:0/2763917844 conn(0x556235bf2b80 msgr2=0x556235bfd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:20.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.721+0000 7fdfe1662740 1 -- 172.21.15.28:0/2763917844 shutdown_connections 2024-04-24T08:15:20.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.721+0000 7fdfe1662740 1 -- 172.21.15.28:0/2763917844 wait complete. 2024-04-24T08:15:20.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.722+0000 7fdfe1662740 1 -- start start 2024-04-24T08:15:20.723 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.722+0000 7fdfe1662740 1 Processor -- start 2024-04-24T08:15:20.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.723+0000 7fdfe1662740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235bf6860 0x556235c04bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.723+0000 7fdfe1662740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235bf7180 0x556235c050f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.723+0000 7fdfe1662740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235c01850 0x556235c760f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.723+0000 7fdfd7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235c01850 0x556235c760f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.723+0000 7fdfd7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235c01850 0x556235c760f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:58112/0 (socket says 172.21.15.28:58112) 2024-04-24T08:15:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.723+0000 7fdfd7e00640 1 -- 172.21.15.28:0/983004362 learned_addr learned my addr 172.21.15.28:0/983004362 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.723+0000 7fdfe1662740 1 -- 172.21.15.28:0/983004362 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd81aa2f0 con 0x556235bf6860 2024-04-24T08:15:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.723+0000 7fdfe1662740 1 -- 172.21.15.28:0/983004362 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd81ab760 con 0x556235bf7180 2024-04-24T08:15:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.723+0000 7fdfe1662740 1 -- 172.21.15.28:0/983004362 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdfd81aa9c0 con 0x556235c01850 2024-04-24T08:15:20.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.723+0000 7fdfe1662740 10 client.0 _ll_register_callbacks cb 0x7fdfd0005f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.723+0000 7fdfd6a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235bf7180 0x556235c050f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.724+0000 7fdfd7e00640 1 -- 172.21.15.28:0/983004362 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235bf7180 msgr2=0x556235c050f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.724+0000 7fdfd7e00640 1 --2- 172.21.15.28:0/983004362 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235bf7180 0x556235c050f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.724+0000 7fdfd7e00640 1 -- 172.21.15.28:0/983004362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235bf6860 msgr2=0x556235c04bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.724+0000 7fdfd7e00640 1 --2- 172.21.15.28:0/983004362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235bf6860 0x556235c04bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.724+0000 7fdfd7e00640 1 -- 172.21.15.28:0/983004362 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55623582e0f0 con 0x556235c01850 2024-04-24T08:15:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.724+0000 7fdfd7e00640 1 --2- 172.21.15.28:0/983004362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235c01850 0x556235c760f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdfd000c640 tx=0x7fdfd00054b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd1177769733cd1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.724+0000 7fdfc1600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:15:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.725+0000 7fdfc1600640 1 -- 172.21.15.28:0/983004362 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfd00aa070 con 0x556235c01850 2024-04-24T08:15:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.725+0000 7fdfc1600640 1 -- 172.21.15.28:0/983004362 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fdfd0098720 con 0x556235c01850 2024-04-24T08:15:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.725+0000 7fdfe1662740 1 -- 172.21.15.28:0/983004362 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fdfd00a0b70 con 0x556235c01850 2024-04-24T08:15:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.725+0000 7fdfc1600640 1 -- 172.21.15.28:0/983004362 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fdfd001f040 con 0x556235c01850 2024-04-24T08:15:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.726+0000 7fdfe1662740 10 client.26224 fetch_fsmap learned FSMap version 26 2024-04-24T08:15:20.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.726+0000 7fdfe1662740 1 -- 172.21.15.28:0/983004362 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x556235b21810 con 0x556235c01850 2024-04-24T08:15:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.726+0000 7fdfc1600640 1 -- 172.21.15.28:0/983004362 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfd00a0b70 con 0x556235c01850 2024-04-24T08:15:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.726+0000 7fdfc1600640 1 -- 172.21.15.28:0/983004362 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fdfd00a1d00 con 0x556235c01850 2024-04-24T08:15:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.727+0000 7fdfe1662740 10 client.26224 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:15:20.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.727+0000 7fdfe1662740 10 client.26224 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:15:20.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.727+0000 7fdfe1662740 20 client.26224 populate_metadata read hostname 'smithi028' 2024-04-24T08:15:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.727+0000 7fdfe1662740 1 --2- 172.21.15.28:0/983004362 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x556235c77960 0x556235c79d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.727+0000 7fdfe1662740 4 client.26224 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:15:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.727+0000 7fdfe1662740 1 -- 172.21.15.28:0/983004362 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556235b63a80 con 0x556235c77960 2024-04-24T08:15:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.728+0000 7fdfd7400640 1 --2- 172.21.15.28:0/983004362 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x556235c77960 0x556235c79d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.731+0000 7fdfd7400640 1 --2- 172.21.15.28:0/983004362 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x556235c77960 0x556235c79d50 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6b46a0cfa6b660c3 server_cookie=c96288d9a2863a6e in_seq=0 out_seq=0 2024-04-24T08:15:20.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.731+0000 7fdfc1600640 10 client.26224 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.734+0000 7fdfc1600640 1 -- 172.21.15.28:0/983004362 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x556235b63a80 con 0x556235c77960 2024-04-24T08:15:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.734+0000 7fdfc1600640 10 client.26224 handle_command_reply: tid=0 2024-04-24T08:15:20.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.734+0000 7fdfe1662740 1 client.26224 shutdown 2024-04-24T08:15:20.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.734+0000 7fdfe1662740 1 -- 172.21.15.28:0/983004362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235c01850 msgr2=0x556235c760f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.734+0000 7fdfe1662740 1 --2- 172.21.15.28:0/983004362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235c01850 0x556235c760f0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fdfd000c640 tx=0x7fdfd00054b0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.735+0000 7fdfe1662740 1 -- 172.21.15.28:0/983004362 shutdown_connections 2024-04-24T08:15:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.735+0000 7fdfe1662740 1 --2- 172.21.15.28:0/983004362 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x556235c77960 0x556235c79d50 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.736+0000 7fdfe1662740 1 --2- 172.21.15.28:0/983004362 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x556235c01850 0x556235c760f0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.736+0000 7fdfc1600640 0 client.26224 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.736+0000 7fdfe1662740 1 --2- 172.21.15.28:0/983004362 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x556235bf7180 0x556235c050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.736+0000 7fdfe1662740 1 --2- 172.21.15.28:0/983004362 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x556235bf6860 0x556235c04bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.736+0000 7fdfe1662740 1 -- 172.21.15.28:0/983004362 >> 172.21.15.28:0/983004362 conn(0x556235bf2b80 msgr2=0x556235bfe840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.736+0000 7fdfe1662740 1 -- 172.21.15.28:0/983004362 shutdown_connections 2024-04-24T08:15:20.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.737+0000 7fdfe1662740 1 -- 172.21.15.28:0/983004362 wait complete. 2024-04-24T08:15:20.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.737+0000 7fdfe1662740 20 client.26224 trim_cache size 0 max 16384 2024-04-24T08:15:20.740 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:15:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (217 inodes in the stack)", 2024-04-24T08:15:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:15:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: "e0e3930c-be83-4abb-8ef5-4d8186b7b565": { 2024-04-24T08:15:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:15:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "e0e3930c-be83-4abb-8ef5-4d8186b7b565", 2024-04-24T08:15:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:15:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:15:20.740 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:15:20.740 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:15:20.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.741+0000 7fdfde800640 1 -- 172.21.15.28:0/2388002062 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdfbc0a3e20 msgr2=0x7fdfbc0a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.741+0000 7fdfde800640 1 --2- 172.21.15.28:0/2388002062 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdfbc0a3e20 0x7fdfbc0a62e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdfc4005fd0 tx=0x7fdfc4005850 comp rx=0 tx=0).stop 2024-04-24T08:15:20.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.741+0000 7fdfde800640 1 -- 172.21.15.28:0/2388002062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfd8111370 msgr2=0x7fdfd8111820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.741+0000 7fdfde800640 1 --2- 172.21.15.28:0/2388002062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfd8111370 0x7fdfd8111820 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdfd000e940 tx=0x7fdfd000ee00 comp rx=0 tx=0).stop 2024-04-24T08:15:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.741+0000 7fdfde800640 1 -- 172.21.15.28:0/2388002062 shutdown_connections 2024-04-24T08:15:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.741+0000 7fdfde800640 1 --2- 172.21.15.28:0/2388002062 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdfbc0a3e20 0x7fdfbc0a62e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.741+0000 7fdfde800640 1 --2- 172.21.15.28:0/2388002062 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdfd8074bf0 0x7fdfd8110e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.741+0000 7fdfde800640 1 --2- 172.21.15.28:0/2388002062 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdfd8111370 0x7fdfd8111820 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.741+0000 7fdfde800640 1 --2- 172.21.15.28:0/2388002062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdfd80742d0 0x7fdfd8114da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.741+0000 7fdfde800640 1 -- 172.21.15.28:0/2388002062 >> 172.21.15.28:0/2388002062 conn(0x7fdfd8079450 msgr2=0x7fdfd81011b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.741+0000 7fdfde800640 1 -- 172.21.15.28:0/2388002062 shutdown_connections 2024-04-24T08:15:20.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:20.741+0000 7fdfde800640 1 -- 172.21.15.28:0/2388002062 wait complete. 2024-04-24T08:15:20.782 INFO:tasks.cephfs.filesystem:scrub status for tag:e0e3930c-be83-4abb-8ef5-4d8186b7b565 - {'path': '/', 'tag': 'e0e3930c-be83-4abb-8ef5-4d8186b7b565', 'options': 'recursive,force'} 2024-04-24T08:15:22.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:21 smithi117 ceph-mon[32438]: pgmap v153: 97 pgs: 97 active+clean; 18 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 48 MiB/s wr, 8.54k op/s 2024-04-24T08:15:22.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:21 smithi194 ceph-mon[32836]: pgmap v153: 97 pgs: 97 active+clean; 18 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 48 MiB/s wr, 8.54k op/s 2024-04-24T08:15:22.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:21 smithi028 ceph-mon[27786]: pgmap v153: 97 pgs: 97 active+clean; 18 GiB data, 26 GiB used, 1.0 TiB / 1.0 TiB avail; 47 KiB/s rd, 48 MiB/s wr, 8.54k op/s 2024-04-24T08:15:24.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:23 smithi117 ceph-mon[32438]: pgmap v154: 97 pgs: 97 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 64 MiB/s wr, 11.51k op/s 2024-04-24T08:15:24.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:23 smithi194 ceph-mon[32836]: pgmap v154: 97 pgs: 97 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 64 MiB/s wr, 11.51k op/s 2024-04-24T08:15:24.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:23 smithi028 ceph-mon[27786]: pgmap v154: 97 pgs: 97 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 64 MiB/s wr, 11.51k op/s 2024-04-24T08:15:25.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:24 smithi117 ceph-mon[32438]: pgmap v155: 97 pgs: 97 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 54 MiB/s wr, 9.70k op/s 2024-04-24T08:15:25.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:24 smithi194 ceph-mon[32836]: pgmap v155: 97 pgs: 97 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 54 MiB/s wr, 9.70k op/s 2024-04-24T08:15:25.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:24 smithi028 ceph-mon[27786]: pgmap v155: 97 pgs: 97 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 54 MiB/s wr, 9.70k op/s 2024-04-24T08:15:27.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:27 smithi194 ceph-mon[32836]: pgmap v156: 97 pgs: 97 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 55 MiB/s wr, 9.92k op/s 2024-04-24T08:15:27.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:27 smithi028 ceph-mon[27786]: pgmap v156: 97 pgs: 97 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 55 MiB/s wr, 9.92k op/s 2024-04-24T08:15:27.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:27 smithi117 ceph-mon[32438]: pgmap v156: 97 pgs: 97 active+clean; 18 GiB data, 27 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 55 MiB/s wr, 9.92k op/s 2024-04-24T08:15:29.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:29 smithi194 ceph-mon[32836]: pgmap v157: 97 pgs: 97 active+clean; 19 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 73 MiB/s wr, 13.60k op/s 2024-04-24T08:15:29.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:29 smithi028 ceph-mon[27786]: pgmap v157: 97 pgs: 97 active+clean; 19 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 73 MiB/s wr, 13.60k op/s 2024-04-24T08:15:29.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:29 smithi117 ceph-mon[32438]: pgmap v157: 97 pgs: 97 active+clean; 19 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 73 MiB/s wr, 13.60k op/s 2024-04-24T08:15:31.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:31 smithi028 ceph-mon[27786]: pgmap v158: 97 pgs: 97 active+clean; 19 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 55 MiB/s wr, 10.43k op/s 2024-04-24T08:15:31.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:31 smithi117 ceph-mon[32438]: pgmap v158: 97 pgs: 97 active+clean; 19 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 55 MiB/s wr, 10.43k op/s 2024-04-24T08:15:31.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:31 smithi194 ceph-mon[32836]: pgmap v158: 97 pgs: 97 active+clean; 19 GiB data, 28 GiB used, 1.0 TiB / 1.0 TiB avail; 48 KiB/s rd, 55 MiB/s wr, 10.43k op/s 2024-04-24T08:15:33.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:33 smithi028 ceph-mon[27786]: pgmap v159: 97 pgs: 97 active+clean; 19 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 MiB/s wr, 13.15k op/s 2024-04-24T08:15:33.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:33 smithi117 ceph-mon[32438]: pgmap v159: 97 pgs: 97 active+clean; 19 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 MiB/s wr, 13.15k op/s 2024-04-24T08:15:33.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:33 smithi194 ceph-mon[32836]: pgmap v159: 97 pgs: 97 active+clean; 19 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 69 MiB/s wr, 13.15k op/s 2024-04-24T08:15:35.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:35 smithi028 ceph-mon[27786]: pgmap v160: 97 pgs: 97 active+clean; 19 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 56 MiB/s wr, 10.68k op/s 2024-04-24T08:15:35.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:35 smithi117 ceph-mon[32438]: pgmap v160: 97 pgs: 97 active+clean; 19 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 56 MiB/s wr, 10.68k op/s 2024-04-24T08:15:35.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:35 smithi194 ceph-mon[32836]: pgmap v160: 97 pgs: 97 active+clean; 19 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 56 MiB/s wr, 10.68k op/s 2024-04-24T08:15:37.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:37 smithi028 ceph-mon[27786]: pgmap v161: 97 pgs: 97 active+clean; 19 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 52 MiB/s wr, 9.95k op/s 2024-04-24T08:15:37.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:37 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:15:37.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:37 smithi117 ceph-mon[32438]: pgmap v161: 97 pgs: 97 active+clean; 19 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 52 MiB/s wr, 9.95k op/s 2024-04-24T08:15:37.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:37 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:15:37.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:37 smithi194 ceph-mon[32836]: pgmap v161: 97 pgs: 97 active+clean; 19 GiB data, 29 GiB used, 1.0 TiB / 1.0 TiB avail; 46 KiB/s rd, 52 MiB/s wr, 9.95k op/s 2024-04-24T08:15:37.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:37 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:15:39.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:39 smithi028 ceph-mon[27786]: pgmap v162: 97 pgs: 97 active+clean; 19 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 64 MiB/s wr, 12.39k op/s 2024-04-24T08:15:39.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:39 smithi117 ceph-mon[32438]: pgmap v162: 97 pgs: 97 active+clean; 19 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 64 MiB/s wr, 12.39k op/s 2024-04-24T08:15:39.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:39 smithi194 ceph-mon[32836]: pgmap v162: 97 pgs: 97 active+clean; 19 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 64 MiB/s wr, 12.39k op/s 2024-04-24T08:15:41.793 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:41 smithi028 ceph-mon[27786]: pgmap v163: 97 pgs: 97 active+clean; 19 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 44 MiB/s wr, 8.28k op/s 2024-04-24T08:15:41.877 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:41 smithi117 ceph-mon[32438]: pgmap v163: 97 pgs: 97 active+clean; 19 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 44 MiB/s wr, 8.28k op/s 2024-04-24T08:15:41.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:41 smithi194 ceph-mon[32836]: pgmap v163: 97 pgs: 97 active+clean; 19 GiB data, 30 GiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 44 MiB/s wr, 8.28k op/s 2024-04-24T08:15:43.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:43 smithi028 ceph-mon[27786]: pgmap v164: 97 pgs: 97 active+clean; 19 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 60 MiB/s wr, 11.52k op/s 2024-04-24T08:15:43.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:43 smithi117 ceph-mon[32438]: pgmap v164: 97 pgs: 97 active+clean; 19 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 60 MiB/s wr, 11.52k op/s 2024-04-24T08:15:43.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:43 smithi194 ceph-mon[32836]: pgmap v164: 97 pgs: 97 active+clean; 19 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 65 KiB/s rd, 60 MiB/s wr, 11.52k op/s 2024-04-24T08:15:45.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:45 smithi028 ceph-mon[27786]: pgmap v165: 97 pgs: 97 active+clean; 19 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 43 MiB/s wr, 8.40k op/s 2024-04-24T08:15:45.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:45 smithi117 ceph-mon[32438]: pgmap v165: 97 pgs: 97 active+clean; 19 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 43 MiB/s wr, 8.40k op/s 2024-04-24T08:15:45.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:45 smithi194 ceph-mon[32836]: pgmap v165: 97 pgs: 97 active+clean; 19 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 43 MiB/s wr, 8.40k op/s 2024-04-24T08:15:47.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:47 smithi028 ceph-mon[27786]: pgmap v166: 97 pgs: 97 active+clean; 19 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 46 MiB/s wr, 9.00k op/s 2024-04-24T08:15:47.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:47 smithi117 ceph-mon[32438]: pgmap v166: 97 pgs: 97 active+clean; 19 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 46 MiB/s wr, 9.00k op/s 2024-04-24T08:15:47.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:47 smithi194 ceph-mon[32836]: pgmap v166: 97 pgs: 97 active+clean; 19 GiB data, 31 GiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 46 MiB/s wr, 9.00k op/s 2024-04-24T08:15:49.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:49 smithi117 ceph-mon[32438]: pgmap v167: 97 pgs: 97 active+clean; 20 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 56 MiB/s wr, 11.26k op/s 2024-04-24T08:15:49.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:49 smithi194 ceph-mon[32836]: pgmap v167: 97 pgs: 97 active+clean; 20 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 56 MiB/s wr, 11.26k op/s 2024-04-24T08:15:50.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:49 smithi028 ceph-mon[27786]: pgmap v167: 97 pgs: 97 active+clean; 20 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 56 MiB/s wr, 11.26k op/s 2024-04-24T08:15:50.784 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:15:50.784 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:15:50.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:50 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:50.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:50 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:15:50.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:50 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:15:50.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:50 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:50.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:50 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:15:50.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:50 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:15:51.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:50 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:15:51.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:50 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:15:51.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:50 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:15:51.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:51 smithi117 ceph-mon[32438]: pgmap v168: 97 pgs: 97 active+clean; 20 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 43 MiB/s wr, 8.60k op/s 2024-04-24T08:15:51.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:51 smithi194 ceph-mon[32836]: pgmap v168: 97 pgs: 97 active+clean; 20 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 43 MiB/s wr, 8.60k op/s 2024-04-24T08:15:52.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:51 smithi028 ceph-mon[27786]: pgmap v168: 97 pgs: 97 active+clean; 20 GiB data, 32 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 43 MiB/s wr, 8.60k op/s 2024-04-24T08:15:53.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:53 smithi117 ceph-mon[32438]: pgmap v169: 97 pgs: 97 active+clean; 20 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 62 MiB/s wr, 12.59k op/s 2024-04-24T08:15:53.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:53 smithi194 ceph-mon[32836]: pgmap v169: 97 pgs: 97 active+clean; 20 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 62 MiB/s wr, 12.59k op/s 2024-04-24T08:15:54.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:53 smithi028 ceph-mon[27786]: pgmap v169: 97 pgs: 97 active+clean; 20 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 62 MiB/s wr, 12.59k op/s 2024-04-24T08:15:55.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:55 smithi117 ceph-mon[32438]: pgmap v170: 97 pgs: 97 active+clean; 20 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 48 MiB/s wr, 9.93k op/s 2024-04-24T08:15:55.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:55 smithi194 ceph-mon[32836]: pgmap v170: 97 pgs: 97 active+clean; 20 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 48 MiB/s wr, 9.93k op/s 2024-04-24T08:15:56.044 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:55 smithi028 ceph-mon[27786]: pgmap v170: 97 pgs: 97 active+clean; 20 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 48 MiB/s wr, 9.93k op/s 2024-04-24T08:15:57.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:57 smithi117 ceph-mon[32438]: pgmap v171: 97 pgs: 97 active+clean; 20 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 46 MiB/s wr, 9.54k op/s 2024-04-24T08:15:57.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:57 smithi194 ceph-mon[32836]: pgmap v171: 97 pgs: 97 active+clean; 20 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 46 MiB/s wr, 9.54k op/s 2024-04-24T08:15:58.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:57 smithi028 ceph-mon[27786]: pgmap v171: 97 pgs: 97 active+clean; 20 GiB data, 33 GiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 46 MiB/s wr, 9.54k op/s 2024-04-24T08:15:58.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.337+0000 7f2a4c000640 1 -- 172.21.15.28:0/279858298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a3c0a1680 msgr2=0x7f2a3c0a9250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:58.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.337+0000 7f2a43e00640 1 -- 172.21.15.28:0/279858298 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a30037070 con 0x7f2a3c0a1680 2024-04-24T08:15:58.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.337+0000 7f2a4c000640 1 --2- 172.21.15.28:0/279858298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a3c0a1680 0x7f2a3c0a9250 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f2a30009860 tx=0x7f2a3002a340 comp rx=0 tx=0).stop 2024-04-24T08:15:58.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.338+0000 7f2a4c000640 1 -- 172.21.15.28:0/279858298 shutdown_connections 2024-04-24T08:15:58.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.338+0000 7f2a4c000640 1 --2- 172.21.15.28:0/279858298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a3c09d570 0x7f2a3c0a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.338+0000 7f2a4c000640 1 --2- 172.21.15.28:0/279858298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a3c0a1680 0x7f2a3c0a9250 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.338+0000 7f2a4c000640 1 --2- 172.21.15.28:0/279858298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a3c09cbc0 0x7f2a3c09cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.338+0000 7f2a4c000640 1 -- 172.21.15.28:0/279858298 >> 172.21.15.28:0/279858298 conn(0x7f2a3c0092d0 msgr2=0x7f2a3c0096e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:58.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.339+0000 7f2a4c000640 1 -- 172.21.15.28:0/279858298 shutdown_connections 2024-04-24T08:15:58.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.342+0000 7f2a4c000640 1 -- 172.21.15.28:0/279858298 wait complete. 2024-04-24T08:15:58.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.343+0000 7f2a4c000640 1 Processor -- start 2024-04-24T08:15:58.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.343+0000 7f2a4c000640 1 -- start start 2024-04-24T08:15:58.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.344+0000 7f2a4c000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a3c09cbc0 0x7f2a3c138640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.344+0000 7f2a4c000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a3c0a1680 0x7f2a3c138b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.344+0000 7f2a4c000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a3c1327b0 0x7f2a3c132c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.344+0000 7f2a4c000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c0ac780 con 0x7f2a3c09cbc0 2024-04-24T08:15:58.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.344+0000 7f2a4c000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c0ac600 con 0x7f2a3c1327b0 2024-04-24T08:15:58.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.344+0000 7f2a4c000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c0ac900 con 0x7f2a3c0a1680 2024-04-24T08:15:58.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.345+0000 7f2a48c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a3c0a1680 0x7f2a3c138b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:58.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.345+0000 7f2a4a000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a3c1327b0 0x7f2a3c132c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:58.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.345+0000 7f2a4a000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a3c1327b0 0x7f2a3c132c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:50860/0 (socket says 172.21.15.28:50860) 2024-04-24T08:15:58.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.345+0000 7f2a4a000640 1 -- 172.21.15.28:0/2634526785 learned_addr learned my addr 172.21.15.28:0/2634526785 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:58.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.345+0000 7f2a49600640 1 --2- 172.21.15.28:0/2634526785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a3c09cbc0 0x7f2a3c138640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:58.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.346+0000 7f2a49600640 1 -- 172.21.15.28:0/2634526785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a3c0a1680 msgr2=0x7f2a3c138b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:58.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.346+0000 7f2a49600640 1 --2- 172.21.15.28:0/2634526785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a3c0a1680 0x7f2a3c138b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.346+0000 7f2a49600640 1 -- 172.21.15.28:0/2634526785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a3c1327b0 msgr2=0x7f2a3c132c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:58.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.346+0000 7f2a49600640 1 --2- 172.21.15.28:0/2634526785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a3c1327b0 0x7f2a3c132c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.346+0000 7f2a49600640 1 -- 172.21.15.28:0/2634526785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a3c133340 con 0x7f2a3c09cbc0 2024-04-24T08:15:58.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.346+0000 7f2a4a000640 1 --2- 172.21.15.28:0/2634526785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a3c1327b0 0x7f2a3c132c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:15:58.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.346+0000 7f2a49600640 1 --2- 172.21.15.28:0/2634526785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a3c09cbc0 0x7f2a3c138640 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2a44000b60 tx=0x7f2a44069f70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:58.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.347+0000 7f2a42000640 1 -- 172.21.15.28:0/2634526785 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a44063020 con 0x7f2a3c09cbc0 2024-04-24T08:15:58.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.347+0000 7f2a42000640 1 -- 172.21.15.28:0/2634526785 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a440631c0 con 0x7f2a3c09cbc0 2024-04-24T08:15:58.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.347+0000 7f2a4c000640 1 -- 172.21.15.28:0/2634526785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a3c133630 con 0x7f2a3c09cbc0 2024-04-24T08:15:58.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.347+0000 7f2a42000640 1 -- 172.21.15.28:0/2634526785 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a44076670 con 0x7f2a3c09cbc0 2024-04-24T08:15:58.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.347+0000 7f2a4c000640 1 -- 172.21.15.28:0/2634526785 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a3c13f790 con 0x7f2a3c09cbc0 2024-04-24T08:15:58.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.358+0000 7f2a4ee49740 1 Processor -- start 2024-04-24T08:15:58.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.358+0000 7f2a42000640 1 -- 172.21.15.28:0/2634526785 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f2a4408a060 con 0x7f2a3c09cbc0 2024-04-24T08:15:58.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.358+0000 7f2a4ee49740 1 -- start start 2024-04-24T08:15:58.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.360+0000 7f2a4ee49740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a8ba30 0x5572f7a8be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.360+0000 7f2a4ee49740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a8c350 0x5572f7a964e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.360+0000 7f2a4ee49740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a96a20 0x5572f7a98e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.360+0000 7f2a4ee49740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5572f79f11f0 con 0x5572f7a8ba30 2024-04-24T08:15:58.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.360+0000 7f2a4ee49740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5572f7997350 con 0x5572f7a8c350 2024-04-24T08:15:58.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.360+0000 7f2a4ee49740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5572f79ed1b0 con 0x5572f7a96a20 2024-04-24T08:15:58.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.360+0000 7f2a49600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a8c350 0x5572f7a964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:58.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.360+0000 7f2a49600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a8c350 0x5572f7a964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:50866/0 (socket says 172.21.15.28:50866) 2024-04-24T08:15:58.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.360+0000 7f2a49600640 1 -- 172.21.15.28:0/2755346487 learned_addr learned my addr 172.21.15.28:0/2755346487 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:58.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.361+0000 7f2a49600640 1 -- 172.21.15.28:0/2755346487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a96a20 msgr2=0x5572f7a98e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:58.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.361+0000 7f2a49600640 1 --2- 172.21.15.28:0/2755346487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a96a20 0x5572f7a98e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.361+0000 7f2a49600640 1 -- 172.21.15.28:0/2755346487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a8ba30 msgr2=0x5572f7a8be10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:15:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.362+0000 7f2a4a000640 1 --2- 172.21.15.28:0/2755346487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a8ba30 0x5572f7a8be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.362+0000 7f2a49600640 1 --2- 172.21.15.28:0/2755346487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a8ba30 0x5572f7a8be10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.362+0000 7f2a49600640 1 -- 172.21.15.28:0/2755346487 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572f79ebf50 con 0x5572f7a8c350 2024-04-24T08:15:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.363+0000 7f2a49600640 1 --2- 172.21.15.28:0/2755346487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a8c350 0x5572f7a964e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2a44139d10 tx=0x7f2a44144690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d40169720182914 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:58.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.364+0000 7f2a36600640 1 -- 172.21.15.28:0/2755346487 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a4414b070 con 0x5572f7a8c350 2024-04-24T08:15:58.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.364+0000 7f2a36600640 1 -- 172.21.15.28:0/2755346487 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a4413ccf0 con 0x5572f7a8c350 2024-04-24T08:15:58.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.365+0000 7f2a42000640 1 --2- 172.21.15.28:0/2634526785 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2a2c0a3de0 0x7f2a2c0a62a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.365+0000 7f2a36600640 1 -- 172.21.15.28:0/2755346487 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a44147800 con 0x5572f7a8c350 2024-04-24T08:15:58.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.365+0000 7f2a42000640 1 -- 172.21.15.28:0/2634526785 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f2a4413c670 con 0x7f2a3c09cbc0 2024-04-24T08:15:58.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.366+0000 7f2a4ee49740 1 -- 172.21.15.28:0/2755346487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a8c350 msgr2=0x5572f7a964e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:58.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.366+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/2755346487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a8c350 0x5572f7a964e0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2a44139d10 tx=0x7f2a44144690 comp rx=0 tx=0).stop 2024-04-24T08:15:58.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.366+0000 7f2a4a000640 1 --2- 172.21.15.28:0/2634526785 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2a2c0a3de0 0x7f2a2c0a62a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:58.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.367+0000 7f2a4ee49740 1 -- 172.21.15.28:0/2755346487 shutdown_connections 2024-04-24T08:15:58.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.367+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/2755346487 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a96a20 0x5572f7a98e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.367+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/2755346487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a8c350 0x5572f7a964e0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.367+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/2755346487 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a8ba30 0x5572f7a8be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.367+0000 7f2a4ee49740 1 -- 172.21.15.28:0/2755346487 >> 172.21.15.28:0/2755346487 conn(0x5572f7a87d50 msgr2=0x5572f7a88130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:58.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.368+0000 7f2a4a000640 1 --2- 172.21.15.28:0/2634526785 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2a2c0a3de0 0x7f2a2c0a62a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2a30005940 tx=0x7f2a30005850 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:58.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.376+0000 7f2a4ee49740 1 -- 172.21.15.28:0/2755346487 shutdown_connections 2024-04-24T08:15:58.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.376+0000 7f2a4ee49740 1 -- 172.21.15.28:0/2755346487 wait complete. 2024-04-24T08:15:58.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.377+0000 7f2a4ee49740 1 -- start start 2024-04-24T08:15:58.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.377+0000 7f2a4ee49740 1 Processor -- start 2024-04-24T08:15:58.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.378+0000 7f2a4ee49740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a8ba30 0x5572f7a938e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.378+0000 7f2a4ee49740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a8c350 0x5572f7a93e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.378+0000 7f2a4ee49740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a96a20 0x5572f7a94360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.378+0000 7f2a48c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a8c350 0x5572f7a93e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:58.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.378+0000 7f2a49600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a8ba30 0x5572f7a938e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:58.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.378+0000 7f2a48c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a8c350 0x5572f7a93e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:35430/0 (socket says 172.21.15.28:35430) 2024-04-24T08:15:58.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.379+0000 7f2a49600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a8ba30 0x5572f7a938e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53906/0 (socket says 172.21.15.28:53906) 2024-04-24T08:15:58.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.379+0000 7f2a4a000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a96a20 0x5572f7a94360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:58.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.379+0000 7f2a4a000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a96a20 0x5572f7a94360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:50882/0 (socket says 172.21.15.28:50882) 2024-04-24T08:15:58.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.379+0000 7f2a48c00640 1 -- 172.21.15.28:0/3344442263 learned_addr learned my addr 172.21.15.28:0/3344442263 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:58.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.380+0000 7f2a4ee49740 1 -- 172.21.15.28:0/3344442263 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5572f7a998b0 con 0x5572f7a8ba30 2024-04-24T08:15:58.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.380+0000 7f2a4ee49740 1 -- 172.21.15.28:0/3344442263 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5572f79aa690 con 0x5572f7a96a20 2024-04-24T08:15:58.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.380+0000 7f2a4ee49740 1 -- 172.21.15.28:0/3344442263 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5572f7a99a30 con 0x5572f7a8c350 2024-04-24T08:15:58.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.380+0000 7f2a4ee49740 10 client.0 _ll_register_callbacks cb 0x5572f79e6650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:58.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.381+0000 7f2a48c00640 1 -- 172.21.15.28:0/3344442263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a96a20 msgr2=0x5572f7a94360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.381+0000 7f2a48c00640 1 --2- 172.21.15.28:0/3344442263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a96a20 0x5572f7a94360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.381+0000 7f2a48c00640 1 -- 172.21.15.28:0/3344442263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a8ba30 msgr2=0x5572f7a938e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.381+0000 7f2a48c00640 1 --2- 172.21.15.28:0/3344442263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a8ba30 0x5572f7a938e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.381+0000 7f2a48c00640 1 -- 172.21.15.28:0/3344442263 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572f79a9920 con 0x5572f7a8c350 2024-04-24T08:15:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.381+0000 7f2a2aa00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:15:58.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.381+0000 7f2a48c00640 1 --2- 172.21.15.28:0/3344442263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a8c350 0x5572f7a93e20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2a3800e650 tx=0x7f2a3800eb10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb681fbff45c4184 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:58.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.382+0000 7f2a2aa00640 1 -- 172.21.15.28:0/3344442263 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a38004250 con 0x5572f7a8c350 2024-04-24T08:15:58.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.382+0000 7f2a4ee49740 1 -- 172.21.15.28:0/3344442263 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a4413ccf0 con 0x5572f7a8c350 2024-04-24T08:15:58.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.382+0000 7f2a2aa00640 1 -- 172.21.15.28:0/3344442263 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f2a38004050 con 0x5572f7a8c350 2024-04-24T08:15:58.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.383+0000 7f2a2aa00640 1 -- 172.21.15.28:0/3344442263 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2a38004250 con 0x5572f7a8c350 2024-04-24T08:15:58.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.383+0000 7f2a4ee49740 10 client.26233 fetch_fsmap learned FSMap version 26 2024-04-24T08:15:58.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.383+0000 7f2a4ee49740 1 -- 172.21.15.28:0/3344442263 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5572f77cd7d0 con 0x5572f7a8c350 2024-04-24T08:15:58.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.385+0000 7f2a2aa00640 1 -- 172.21.15.28:0/3344442263 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a4413ccf0 con 0x5572f7a8c350 2024-04-24T08:15:58.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.385+0000 7f2a2aa00640 1 -- 172.21.15.28:0/3344442263 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2a38005340 con 0x5572f7a8c350 2024-04-24T08:15:58.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.385+0000 7f2a4ee49740 10 client.26233 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:15:58.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.385+0000 7f2a4ee49740 10 client.26233 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:15:58.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.385+0000 7f2a4ee49740 20 client.26233 populate_metadata read hostname 'smithi028' 2024-04-24T08:15:58.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.385+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/3344442263 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5572f7aa6e30 0x5572f7aa92d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.385+0000 7f2a4ee49740 4 client.26233 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:15:58.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.385+0000 7f2a4ee49740 1 -- 172.21.15.28:0/3344442263 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5572f79fec10 con 0x5572f7aa6e30 2024-04-24T08:15:58.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.386+0000 7f2a49600640 1 --2- 172.21.15.28:0/3344442263 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5572f7aa6e30 0x5572f7aa92d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:58.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.387+0000 7f2a49600640 1 --2- 172.21.15.28:0/3344442263 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5572f7aa6e30 0x5572f7aa92d0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8daf38449e73ae2d server_cookie=47b3e411c7b631b4 in_seq=0 out_seq=0 2024-04-24T08:15:58.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.387+0000 7f2a2aa00640 10 client.26233 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:58.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.391+0000 7f2a2aa00640 1 -- 172.21.15.28:0/3344442263 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5572f79fec10 con 0x5572f7aa6e30 2024-04-24T08:15:58.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.391+0000 7f2a2aa00640 10 client.26233 handle_command_reply: tid=0 2024-04-24T08:15:58.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.391+0000 7f2a4ee49740 1 client.26233 shutdown 2024-04-24T08:15:58.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.392+0000 7f2a4ee49740 1 -- 172.21.15.28:0/3344442263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a8c350 msgr2=0x5572f7a93e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:58.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.392+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/3344442263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a8c350 0x5572f7a93e20 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f2a3800e650 tx=0x7f2a3800eb10 comp rx=0 tx=0).stop 2024-04-24T08:15:58.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.392+0000 7f2a4ee49740 1 -- 172.21.15.28:0/3344442263 shutdown_connections 2024-04-24T08:15:58.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.392+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/3344442263 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5572f7aa6e30 0x5572f7aa92d0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.392+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/3344442263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a96a20 0x5572f7a94360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.392+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/3344442263 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a8c350 0x5572f7a93e20 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.393+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/3344442263 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a8ba30 0x5572f7a938e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.393+0000 7f2a4ee49740 1 -- 172.21.15.28:0/3344442263 >> 172.21.15.28:0/3344442263 conn(0x5572f7a87d50 msgr2=0x5572f7a86fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:58.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.393+0000 7f2a2aa00640 0 client.26233 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:58.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.395+0000 7f2a4ee49740 1 -- 172.21.15.28:0/3344442263 shutdown_connections 2024-04-24T08:15:58.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.395+0000 7f2a4ee49740 1 -- 172.21.15.28:0/3344442263 wait complete. 2024-04-24T08:15:58.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.395+0000 7f2a4ee49740 20 client.26233 trim_cache size 0 max 16384 2024-04-24T08:15:58.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.445+0000 7f2a4ee49740 1 Processor -- start 2024-04-24T08:15:58.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.445+0000 7f2a4ee49740 1 -- start start 2024-04-24T08:15:58.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.446+0000 7f2a4ee49740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a8ba30 0x5572f7aa2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.446+0000 7f2a4ee49740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a8c350 0x5572f7a94130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.446+0000 7f2a4ee49740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a96a20 0x5572f7a8e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.446+0000 7f2a4ee49740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c14a980 con 0x5572f7a96a20 2024-04-24T08:15:58.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.446+0000 7f2a4ee49740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c148250 con 0x5572f7a8ba30 2024-04-24T08:15:58.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.446+0000 7f2a4ee49740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c146040 con 0x5572f7a8c350 2024-04-24T08:15:58.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.447+0000 7f2a48c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a8c350 0x5572f7a94130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:58.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.447+0000 7f2a48c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a8c350 0x5572f7a94130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:35440/0 (socket says 172.21.15.28:35440) 2024-04-24T08:15:58.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.447+0000 7f2a48c00640 1 -- 172.21.15.28:0/3035774872 learned_addr learned my addr 172.21.15.28:0/3035774872 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:58.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.447+0000 7f2a49600640 1 --2- 172.21.15.28:0/3035774872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a8ba30 0x5572f7aa2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:58.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.447+0000 7f2a48c00640 1 -- 172.21.15.28:0/3035774872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a8ba30 msgr2=0x5572f7aa2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:58.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.447+0000 7f2a48c00640 1 --2- 172.21.15.28:0/3035774872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a8ba30 0x5572f7aa2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.447+0000 7f2a48c00640 1 -- 172.21.15.28:0/3035774872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a96a20 msgr2=0x5572f7a8e1d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:15:58.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.448+0000 7f2a4a000640 1 --2- 172.21.15.28:0/3035774872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a96a20 0x5572f7a8e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:58.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.448+0000 7f2a48c00640 1 --2- 172.21.15.28:0/3035774872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a96a20 0x5572f7a8e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.448+0000 7f2a48c00640 1 -- 172.21.15.28:0/3035774872 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572f77033f0 con 0x5572f7a8c350 2024-04-24T08:15:58.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.449+0000 7f2a48c00640 1 --2- 172.21.15.28:0/3035774872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a8c350 0x5572f7a94130 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2a38001eb0 tx=0x7f2a38004890 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e4143ed04a025436 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:58.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.449+0000 7f2a35c00640 1 -- 172.21.15.28:0/3035774872 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a38029040 con 0x5572f7a8c350 2024-04-24T08:15:58.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.450+0000 7f2a35c00640 1 -- 172.21.15.28:0/3035774872 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a38032070 con 0x5572f7a8c350 2024-04-24T08:15:58.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.450+0000 7f2a35c00640 1 -- 172.21.15.28:0/3035774872 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a38004d30 con 0x5572f7a8c350 2024-04-24T08:15:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.451+0000 7f2a4ee49740 1 -- 172.21.15.28:0/3035774872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a8c350 msgr2=0x5572f7a94130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.451+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/3035774872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a8c350 0x5572f7a94130 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2a38001eb0 tx=0x7f2a38004890 comp rx=0 tx=0).stop 2024-04-24T08:15:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.451+0000 7f2a4ee49740 1 -- 172.21.15.28:0/3035774872 shutdown_connections 2024-04-24T08:15:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.451+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/3035774872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a96a20 0x5572f7a8e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.451+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/3035774872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a8c350 0x5572f7a94130 secure :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f2a38001eb0 tx=0x7f2a38004890 comp rx=0 tx=0).stop 2024-04-24T08:15:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.451+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/3035774872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a8ba30 0x5572f7aa2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.451+0000 7f2a4ee49740 1 -- 172.21.15.28:0/3035774872 >> 172.21.15.28:0/3035774872 conn(0x5572f7a87d50 msgr2=0x5572f7a95110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:58.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.452+0000 7f2a4ee49740 1 -- 172.21.15.28:0/3035774872 shutdown_connections 2024-04-24T08:15:58.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.452+0000 7f2a4ee49740 1 -- 172.21.15.28:0/3035774872 wait complete. 2024-04-24T08:15:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.453+0000 7f2a4ee49740 1 -- start start 2024-04-24T08:15:58.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.453+0000 7f2a4ee49740 1 Processor -- start 2024-04-24T08:15:58.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.455+0000 7f2a4ee49740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a8ba30 0x5572f7a9ed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.455+0000 7f2a4ee49740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a96a20 0x5572f7a9f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.455+0000 7f2a4ee49740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a9bd30 0x5572f7a9c1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.455+0000 7f2a4ee49740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c146710 con 0x5572f7a8ba30 2024-04-24T08:15:58.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.455+0000 7f2a4ee49740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c147b80 con 0x5572f7a96a20 2024-04-24T08:15:58.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.455+0000 7f2a4ee49740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2a3c146de0 con 0x5572f7a9bd30 2024-04-24T08:15:58.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.455+0000 7f2a4ee49740 10 client.0 _ll_register_callbacks cb 0x7f2a3800cd10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:15:58.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.456+0000 7f2a4a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a9bd30 0x5572f7a9c1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:58.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.456+0000 7f2a4a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a9bd30 0x5572f7a9c1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:35448/0 (socket says 172.21.15.28:35448) 2024-04-24T08:15:58.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.456+0000 7f2a4a000640 1 -- 172.21.15.28:0/2581021715 learned_addr learned my addr 172.21.15.28:0/2581021715 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:15:58.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.456+0000 7f2a49600640 1 --2- 172.21.15.28:0/2581021715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a8ba30 0x5572f7a9ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:58.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.456+0000 7f2a48c00640 1 --2- 172.21.15.28:0/2581021715 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a96a20 0x5572f7a9f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:58.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.457+0000 7f2a48c00640 1 -- 172.21.15.28:0/2581021715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a9bd30 msgr2=0x5572f7a9c1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:58.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.457+0000 7f2a48c00640 1 --2- 172.21.15.28:0/2581021715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a9bd30 0x5572f7a9c1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.457+0000 7f2a48c00640 1 -- 172.21.15.28:0/2581021715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a8ba30 msgr2=0x5572f7a9ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:15:58.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.457+0000 7f2a48c00640 1 --2- 172.21.15.28:0/2581021715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a8ba30 0x5572f7a9ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.459 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.457+0000 7f2a48c00640 1 -- 172.21.15.28:0/2581021715 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572f76c30f0 con 0x5572f7a96a20 2024-04-24T08:15:58.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.458+0000 7f2a48c00640 1 --2- 172.21.15.28:0/2581021715 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a96a20 0x5572f7a9f2a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x5572f7a93460 tx=0x7f2a3800ad90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32448a539a996d7f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:15:58.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.458+0000 7f2a2aa00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:15:58.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.459+0000 7f2a2aa00640 1 -- 172.21.15.28:0/2581021715 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a3803d060 con 0x5572f7a96a20 2024-04-24T08:15:58.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.460+0000 7f2a2aa00640 1 -- 172.21.15.28:0/2581021715 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f2a3800a680 con 0x5572f7a96a20 2024-04-24T08:15:58.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.460+0000 7f2a4ee49740 1 -- 172.21.15.28:0/2581021715 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a38032070 con 0x5572f7a96a20 2024-04-24T08:15:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.462+0000 7f2a2aa00640 1 -- 172.21.15.28:0/2581021715 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2a38029040 con 0x5572f7a96a20 2024-04-24T08:15:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.462+0000 7f2a4ee49740 10 client.26207 fetch_fsmap learned FSMap version 26 2024-04-24T08:15:58.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.463+0000 7f2a4ee49740 1 -- 172.21.15.28:0/2581021715 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5572f79b6810 con 0x5572f7a96a20 2024-04-24T08:15:58.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.465+0000 7f2a2aa00640 1 -- 172.21.15.28:0/2581021715 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a38032070 con 0x5572f7a96a20 2024-04-24T08:15:58.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.465+0000 7f2a2aa00640 1 -- 172.21.15.28:0/2581021715 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2a38036450 con 0x5572f7a96a20 2024-04-24T08:15:58.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.465+0000 7f2a4ee49740 10 client.26207 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:15:58.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.465+0000 7f2a4ee49740 10 client.26207 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:15:58.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.465+0000 7f2a4ee49740 20 client.26207 populate_metadata read hostname 'smithi028' 2024-04-24T08:15:58.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.466+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/2581021715 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5572f7b1f4b0 0x5572f7b218d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:15:58.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.466+0000 7f2a4ee49740 4 client.26207 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:15:58.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.466+0000 7f2a4ee49740 1 -- 172.21.15.28:0/2581021715 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5572f79f8a80 con 0x5572f7b1f4b0 2024-04-24T08:15:58.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.468+0000 7f2a49600640 1 --2- 172.21.15.28:0/2581021715 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5572f7b1f4b0 0x5572f7b218d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:15:58.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.470+0000 7f2a49600640 1 --2- 172.21.15.28:0/2581021715 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5572f7b1f4b0 0x5572f7b218d0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=656b9fc863e13816 server_cookie=f4d7bd92ac629121 in_seq=0 out_seq=0 2024-04-24T08:15:58.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.470+0000 7f2a2aa00640 10 client.26207 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:58.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.470+0000 7f2a2aa00640 1 -- 172.21.15.28:0/2581021715 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x5572f79f8a80 con 0x5572f7b1f4b0 2024-04-24T08:15:58.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.470+0000 7f2a2aa00640 10 client.26207 handle_command_reply: tid=0 2024-04-24T08:15:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.471+0000 7f2a4ee49740 1 client.26207 shutdown 2024-04-24T08:15:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.471+0000 7f2a4ee49740 1 -- 172.21.15.28:0/2581021715 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a96a20 msgr2=0x5572f7a9f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.471+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/2581021715 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a96a20 0x5572f7a9f2a0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x5572f7a93460 tx=0x7f2a3800ad90 comp rx=0 tx=0).stop 2024-04-24T08:15:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.472+0000 7f2a4ee49740 1 -- 172.21.15.28:0/2581021715 shutdown_connections 2024-04-24T08:15:58.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.472+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/2581021715 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5572f7b1f4b0 0x5572f7b218d0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.472+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/2581021715 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5572f7a9bd30 0x5572f7a9c1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.472+0000 7f2a2aa00640 0 client.26207 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:15:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.473+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/2581021715 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5572f7a96a20 0x5572f7a9f2a0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.473+0000 7f2a4ee49740 1 --2- 172.21.15.28:0/2581021715 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5572f7a8ba30 0x5572f7a9ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.473+0000 7f2a4ee49740 1 -- 172.21.15.28:0/2581021715 >> 172.21.15.28:0/2581021715 conn(0x5572f7a87d50 msgr2=0x5572f7a93f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.473+0000 7f2a4ee49740 1 -- 172.21.15.28:0/2581021715 shutdown_connections 2024-04-24T08:15:58.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.473+0000 7f2a4ee49740 1 -- 172.21.15.28:0/2581021715 wait complete. 2024-04-24T08:15:58.476 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.475+0000 7f2a4ee49740 20 client.26207 trim_cache size 0 max 16384 2024-04-24T08:15:58.477 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:15:58.477 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (171 inodes in the stack)", 2024-04-24T08:15:58.477 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:15:58.477 INFO:teuthology.orchestra.run.smithi028.stdout: "e0e3930c-be83-4abb-8ef5-4d8186b7b565": { 2024-04-24T08:15:58.477 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:15:58.477 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "e0e3930c-be83-4abb-8ef5-4d8186b7b565", 2024-04-24T08:15:58.477 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:15:58.477 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:15:58.477 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:15:58.477 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:15:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.476+0000 7f2a4c000640 1 -- 172.21.15.28:0/2634526785 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2a2c0a3de0 msgr2=0x7f2a2c0a62a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.476+0000 7f2a4c000640 1 --2- 172.21.15.28:0/2634526785 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2a2c0a3de0 0x7f2a2c0a62a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f2a30005940 tx=0x7f2a30005850 comp rx=0 tx=0).stop 2024-04-24T08:15:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.476+0000 7f2a4c000640 1 -- 172.21.15.28:0/2634526785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a3c09cbc0 msgr2=0x7f2a3c138640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:15:58.478 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.476+0000 7f2a4c000640 1 --2- 172.21.15.28:0/2634526785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a3c09cbc0 0x7f2a3c138640 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f2a44000b60 tx=0x7f2a44069f70 comp rx=0 tx=0).stop 2024-04-24T08:15:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.476+0000 7f2a4c000640 1 -- 172.21.15.28:0/2634526785 shutdown_connections 2024-04-24T08:15:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.476+0000 7f2a4c000640 1 --2- 172.21.15.28:0/2634526785 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2a2c0a3de0 0x7f2a2c0a62a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.476+0000 7f2a4c000640 1 --2- 172.21.15.28:0/2634526785 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a3c1327b0 0x7f2a3c132c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.476+0000 7f2a4c000640 1 --2- 172.21.15.28:0/2634526785 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a3c0a1680 0x7f2a3c138b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.476+0000 7f2a4c000640 1 --2- 172.21.15.28:0/2634526785 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a3c09cbc0 0x7f2a3c138640 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:15:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.476+0000 7f2a4c000640 1 -- 172.21.15.28:0/2634526785 >> 172.21.15.28:0/2634526785 conn(0x7f2a3c0092d0 msgr2=0x7f2a3c09a7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:15:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.477+0000 7f2a4c000640 1 -- 172.21.15.28:0/2634526785 shutdown_connections 2024-04-24T08:15:58.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:15:58.477+0000 7f2a4c000640 1 -- 172.21.15.28:0/2634526785 wait complete. 2024-04-24T08:15:58.516 INFO:tasks.cephfs.filesystem:scrub status for tag:e0e3930c-be83-4abb-8ef5-4d8186b7b565 - {'path': '/', 'tag': 'e0e3930c-be83-4abb-8ef5-4d8186b7b565', 'options': 'recursive,force'} 2024-04-24T08:15:59.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:15:59 smithi117 ceph-mon[32438]: pgmap v172: 97 pgs: 97 active+clean; 20 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 53 MiB/s wr, 11.16k op/s 2024-04-24T08:15:59.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:15:59 smithi194 ceph-mon[32836]: pgmap v172: 97 pgs: 97 active+clean; 20 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 53 MiB/s wr, 11.16k op/s 2024-04-24T08:16:00.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:15:59 smithi028 ceph-mon[27786]: pgmap v172: 97 pgs: 97 active+clean; 20 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 108 KiB/s rd, 53 MiB/s wr, 11.16k op/s 2024-04-24T08:16:01.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:01 smithi117 ceph-mon[32438]: pgmap v173: 97 pgs: 97 active+clean; 20 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 39 MiB/s wr, 8.14k op/s 2024-04-24T08:16:01.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:01 smithi194 ceph-mon[32836]: pgmap v173: 97 pgs: 97 active+clean; 20 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 39 MiB/s wr, 8.14k op/s 2024-04-24T08:16:02.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:01 smithi028 ceph-mon[27786]: pgmap v173: 97 pgs: 97 active+clean; 20 GiB data, 34 GiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 39 MiB/s wr, 8.14k op/s 2024-04-24T08:16:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:03 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:16:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:03 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:03 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:16:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:03 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:03 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:03 smithi117 ceph-mon[32438]: pgmap v174: 97 pgs: 97 active+clean; 20 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 45 MiB/s wr, 9.38k op/s 2024-04-24T08:16:03.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:03 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:16:03.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:03 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:03.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:03 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:16:03.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:03 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:03.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:03 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:03.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:03 smithi194 ceph-mon[32836]: pgmap v174: 97 pgs: 97 active+clean; 20 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 45 MiB/s wr, 9.38k op/s 2024-04-24T08:16:04.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:03 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:16:04.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:03 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:04.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:03 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:16:04.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:03 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:04.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:03 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:04.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:03 smithi028 ceph-mon[27786]: pgmap v174: 97 pgs: 97 active+clean; 20 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 75 KiB/s rd, 45 MiB/s wr, 9.38k op/s 2024-04-24T08:16:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:05 smithi117 ceph-mon[32438]: pgmap v175: 97 pgs: 97 active+clean; 20 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 34 MiB/s wr, 6.98k op/s 2024-04-24T08:16:05.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:05 smithi194 ceph-mon[32836]: pgmap v175: 97 pgs: 97 active+clean; 20 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 34 MiB/s wr, 6.98k op/s 2024-04-24T08:16:06.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:05 smithi028 ceph-mon[27786]: pgmap v175: 97 pgs: 97 active+clean; 20 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 34 MiB/s wr, 6.98k op/s 2024-04-24T08:16:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:07 smithi194 ceph-mon[32836]: pgmap v176: 97 pgs: 97 active+clean; 20 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 31 MiB/s wr, 6.48k op/s 2024-04-24T08:16:08.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:07 smithi028 ceph-mon[27786]: pgmap v176: 97 pgs: 97 active+clean; 20 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 31 MiB/s wr, 6.48k op/s 2024-04-24T08:16:08.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:07 smithi117 ceph-mon[32438]: pgmap v176: 97 pgs: 97 active+clean; 20 GiB data, 35 GiB used, 1.0 TiB / 1.0 TiB avail; 92 KiB/s rd, 31 MiB/s wr, 6.48k op/s 2024-04-24T08:16:09.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:08 smithi028 ceph-mon[27786]: pgmap v177: 97 pgs: 97 active+clean; 20 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 42 MiB/s wr, 8.47k op/s 2024-04-24T08:16:09.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:08 smithi117 ceph-mon[32438]: pgmap v177: 97 pgs: 97 active+clean; 20 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 42 MiB/s wr, 8.47k op/s 2024-04-24T08:16:09.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:08 smithi194 ceph-mon[32836]: pgmap v177: 97 pgs: 97 active+clean; 20 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 42 MiB/s wr, 8.47k op/s 2024-04-24T08:16:10.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:09 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:16:10.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:09 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:16:10.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:09 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:16:10.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:09 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:16:10.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:09 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:16:11.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:11 smithi194 ceph-mon[32836]: pgmap v178: 97 pgs: 97 active+clean; 20 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 35 MiB/s wr, 7.08k op/s 2024-04-24T08:16:11.538 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:11 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:16:11.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:11 smithi028 ceph-mon[27786]: pgmap v178: 97 pgs: 97 active+clean; 20 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 35 MiB/s wr, 7.08k op/s 2024-04-24T08:16:11.627 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:11 smithi117 ceph-mon[32438]: pgmap v178: 97 pgs: 97 active+clean; 20 GiB data, 36 GiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 35 MiB/s wr, 7.08k op/s 2024-04-24T08:16:13.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:13 smithi194 ceph-mon[32836]: pgmap v179: 97 pgs: 97 active+clean; 20 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 41 MiB/s wr, 8.29k op/s 2024-04-24T08:16:13.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:13 smithi028 ceph-mon[27786]: pgmap v179: 97 pgs: 97 active+clean; 20 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 41 MiB/s wr, 8.29k op/s 2024-04-24T08:16:13.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:13 smithi117 ceph-mon[32438]: pgmap v179: 97 pgs: 97 active+clean; 20 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 101 KiB/s rd, 41 MiB/s wr, 8.29k op/s 2024-04-24T08:16:15.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:15 smithi194 ceph-mon[32836]: pgmap v180: 97 pgs: 97 active+clean; 21 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 39 MiB/s wr, 7.91k op/s 2024-04-24T08:16:15.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:15 smithi028 ceph-mon[27786]: pgmap v180: 97 pgs: 97 active+clean; 21 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 39 MiB/s wr, 7.91k op/s 2024-04-24T08:16:15.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:15 smithi117 ceph-mon[32438]: pgmap v180: 97 pgs: 97 active+clean; 21 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 39 MiB/s wr, 7.91k op/s 2024-04-24T08:16:17.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:17 smithi194 ceph-mon[32836]: pgmap v181: 97 pgs: 97 active+clean; 21 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 39 MiB/s wr, 7.80k op/s 2024-04-24T08:16:17.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:17 smithi028 ceph-mon[27786]: pgmap v181: 97 pgs: 97 active+clean; 21 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 39 MiB/s wr, 7.80k op/s 2024-04-24T08:16:17.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:17 smithi117 ceph-mon[32438]: pgmap v181: 97 pgs: 97 active+clean; 21 GiB data, 37 GiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 39 MiB/s wr, 7.80k op/s 2024-04-24T08:16:19.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:19 smithi194 ceph-mon[32836]: pgmap v182: 97 pgs: 97 active+clean; 21 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 45 MiB/s wr, 9.24k op/s 2024-04-24T08:16:19.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:19 smithi028 ceph-mon[27786]: pgmap v182: 97 pgs: 97 active+clean; 21 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 45 MiB/s wr, 9.24k op/s 2024-04-24T08:16:19.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:19 smithi117 ceph-mon[32438]: pgmap v182: 97 pgs: 97 active+clean; 21 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 90 KiB/s rd, 45 MiB/s wr, 9.24k op/s 2024-04-24T08:16:21.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:21 smithi194 ceph-mon[32836]: pgmap v183: 97 pgs: 97 active+clean; 21 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 7.82k op/s 2024-04-24T08:16:21.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:21 smithi028 ceph-mon[27786]: pgmap v183: 97 pgs: 97 active+clean; 21 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 7.82k op/s 2024-04-24T08:16:21.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:21 smithi117 ceph-mon[32438]: pgmap v183: 97 pgs: 97 active+clean; 21 GiB data, 38 GiB used, 1.0 TiB / 1.0 TiB avail; 55 KiB/s rd, 37 MiB/s wr, 7.82k op/s 2024-04-24T08:16:23.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:23 smithi194 ceph-mon[32836]: pgmap v184: 97 pgs: 97 active+clean; 21 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 39 MiB/s wr, 8.04k op/s 2024-04-24T08:16:23.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:23 smithi028 ceph-mon[27786]: pgmap v184: 97 pgs: 97 active+clean; 21 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 39 MiB/s wr, 8.04k op/s 2024-04-24T08:16:23.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:23 smithi117 ceph-mon[32438]: pgmap v184: 97 pgs: 97 active+clean; 21 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 56 KiB/s rd, 39 MiB/s wr, 8.04k op/s 2024-04-24T08:16:25.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:25 smithi028 ceph-mon[27786]: pgmap v185: 97 pgs: 97 active+clean; 21 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 39 MiB/s wr, 8.16k op/s 2024-04-24T08:16:25.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:25 smithi117 ceph-mon[32438]: pgmap v185: 97 pgs: 97 active+clean; 21 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 39 MiB/s wr, 8.16k op/s 2024-04-24T08:16:25.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:25 smithi194 ceph-mon[32836]: pgmap v185: 97 pgs: 97 active+clean; 21 GiB data, 39 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 39 MiB/s wr, 8.16k op/s 2024-04-24T08:16:27.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:27 smithi028 ceph-mon[27786]: pgmap v186: 97 pgs: 97 active+clean; 21 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 48 MiB/s wr, 10.12k op/s 2024-04-24T08:16:27.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:27 smithi117 ceph-mon[32438]: pgmap v186: 97 pgs: 97 active+clean; 21 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 48 MiB/s wr, 10.12k op/s 2024-04-24T08:16:27.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:27 smithi194 ceph-mon[32836]: pgmap v186: 97 pgs: 97 active+clean; 21 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 48 MiB/s wr, 10.12k op/s 2024-04-24T08:16:28.518 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:16:28.518 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:16:29.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:29 smithi028 ceph-mon[27786]: pgmap v187: 97 pgs: 97 active+clean; 21 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 49 MiB/s wr, 10.33k op/s 2024-04-24T08:16:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:29 smithi117 ceph-mon[32438]: pgmap v187: 97 pgs: 97 active+clean; 21 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 49 MiB/s wr, 10.33k op/s 2024-04-24T08:16:29.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:29 smithi194 ceph-mon[32836]: pgmap v187: 97 pgs: 97 active+clean; 21 GiB data, 40 GiB used, 1.0 TiB / 1.0 TiB avail; 84 KiB/s rd, 49 MiB/s wr, 10.33k op/s 2024-04-24T08:16:30.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.201+0000 7f4d59800640 1 -- 172.21.15.28:0/1032736948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d54109210 msgr2=0x7f4d541095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:30.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.201+0000 7f4d59800640 1 --2- 172.21.15.28:0/1032736948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d54109210 0x7f4d541095f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f4d44009900 tx=0x7f4d4402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.202+0000 7f4d59800640 1 -- 172.21.15.28:0/1032736948 shutdown_connections 2024-04-24T08:16:30.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.202+0000 7f4d59800640 1 --2- 172.21.15.28:0/1032736948 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54109b30 0x7f4d54114c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.202+0000 7f4d59800640 1 --2- 172.21.15.28:0/1032736948 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d54115150 0x7f4d54117540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.202+0000 7f4d59800640 1 --2- 172.21.15.28:0/1032736948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d54109210 0x7f4d541095f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.202+0000 7f4d59800640 1 -- 172.21.15.28:0/1032736948 >> 172.21.15.28:0/1032736948 conn(0x7f4d5406d110 msgr2=0x7f4d5406d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:30.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.202+0000 7f4d59800640 1 -- 172.21.15.28:0/1032736948 shutdown_connections 2024-04-24T08:16:30.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.202+0000 7f4d59800640 1 -- 172.21.15.28:0/1032736948 wait complete. 2024-04-24T08:16:30.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.204+0000 7f4d59800640 1 Processor -- start 2024-04-24T08:16:30.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.204+0000 7f4d59800640 1 -- start start 2024-04-24T08:16:30.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.205+0000 7f4d59800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d54109210 0x7f4d541a3410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.205+0000 7f4d59800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d54109b30 0x7f4d541a3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.205+0000 7f4d59800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54115150 0x7f4d541a7c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.205+0000 7f4d59800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5411ced0 con 0x7f4d54115150 2024-04-24T08:16:30.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.205+0000 7f4d59800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5411cd50 con 0x7f4d54109210 2024-04-24T08:16:30.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.205+0000 7f4d59800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4d5411d050 con 0x7f4d54109b30 2024-04-24T08:16:30.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.205+0000 7f4d52a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d54109210 0x7f4d541a3410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.205+0000 7f4d52a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d54109210 0x7f4d541a3410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:53112/0 (socket says 172.21.15.28:53112) 2024-04-24T08:16:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.205+0000 7f4d52a00640 1 -- 172.21.15.28:0/3822954806 learned_addr learned my addr 172.21.15.28:0/3822954806 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.206+0000 7f4d52000640 1 --2- 172.21.15.28:0/3822954806 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d54109b30 0x7f4d541a3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.206+0000 7f4d52a00640 1 -- 172.21.15.28:0/3822954806 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d54109b30 msgr2=0x7f4d541a3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.206+0000 7f4d52a00640 1 --2- 172.21.15.28:0/3822954806 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d54109b30 0x7f4d541a3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.206+0000 7f4d52a00640 1 -- 172.21.15.28:0/3822954806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54115150 msgr2=0x7f4d541a7c80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:16:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.206+0000 7f4d53400640 1 --2- 172.21.15.28:0/3822954806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54115150 0x7f4d541a7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.206+0000 7f4d52a00640 1 --2- 172.21.15.28:0/3822954806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54115150 0x7f4d541a7c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.206+0000 7f4d52a00640 1 -- 172.21.15.28:0/3822954806 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4d541a83b0 con 0x7f4d54109210 2024-04-24T08:16:30.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.207+0000 7f4d52a00640 1 --2- 172.21.15.28:0/3822954806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d54109210 0x7f4d541a3410 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4d44004770 tx=0x7f4d44003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:30.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.213+0000 7f4d4b400640 1 -- 172.21.15.28:0/3822954806 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4403f070 con 0x7f4d54109210 2024-04-24T08:16:30.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.213+0000 7f4d4b400640 1 -- 172.21.15.28:0/3822954806 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d44031ca0 con 0x7f4d54109210 2024-04-24T08:16:30.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.213+0000 7f4d4b400640 1 -- 172.21.15.28:0/3822954806 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d44035030 con 0x7f4d54109210 2024-04-24T08:16:30.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.214+0000 7f4d59800640 1 -- 172.21.15.28:0/3822954806 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4d541a8640 con 0x7f4d54109210 2024-04-24T08:16:30.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.214+0000 7f4d59800640 1 -- 172.21.15.28:0/3822954806 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4d541aff30 con 0x7f4d54109210 2024-04-24T08:16:30.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.225+0000 7f4d5c5c7740 1 Processor -- start 2024-04-24T08:16:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.225+0000 7f4d5c5c7740 1 -- start start 2024-04-24T08:16:30.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.225+0000 7f4d5c5c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639d00a45b0 0x5639d00a4990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.228+0000 7f4d4b400640 1 -- 172.21.15.28:0/3822954806 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4d44042070 con 0x7f4d54109210 2024-04-24T08:16:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.230+0000 7f4d4b400640 1 --2- 172.21.15.28:0/3822954806 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4d380a3e20 0x7f4d380a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.230+0000 7f4d4b400640 1 -- 172.21.15.28:0/3822954806 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f4d440f0d20 con 0x7f4d54109210 2024-04-24T08:16:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.231+0000 7f4d5c5c7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a4f60 0x5639d00b7b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.231+0000 7f4d5c5c7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00b8060 0x5639d00ba450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.231+0000 7f4d5c5c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5639d000a1f0 con 0x5639d00a45b0 2024-04-24T08:16:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.231+0000 7f4d5c5c7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5639cffb0350 con 0x5639d00b8060 2024-04-24T08:16:30.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.231+0000 7f4d5c5c7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5639d00061b0 con 0x5639d00a4f60 2024-04-24T08:16:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.233+0000 7f4d53400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00b8060 0x5639d00ba450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.233+0000 7f4d53400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00b8060 0x5639d00ba450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:53122/0 (socket says 172.21.15.28:53122) 2024-04-24T08:16:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.233+0000 7f4d53400640 1 -- 172.21.15.28:0/2495296351 learned_addr learned my addr 172.21.15.28:0/2495296351 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.233+0000 7f4d53400640 1 --2- 172.21.15.28:0/2495296351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639d00a45b0 0x5639d00a4990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.234+0000 7f4d52000640 1 --2- 172.21.15.28:0/2495296351 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a4f60 0x5639d00b7b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.234+0000 7f4d53400640 1 -- 172.21.15.28:0/2495296351 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a4f60 msgr2=0x5639d00b7b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.235+0000 7f4d53400640 1 --2- 172.21.15.28:0/2495296351 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a4f60 0x5639d00b7b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.235+0000 7f4d53400640 1 -- 172.21.15.28:0/2495296351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639d00a45b0 msgr2=0x5639d00a4990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.235+0000 7f4d53400640 1 --2- 172.21.15.28:0/2495296351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639d00a45b0 0x5639d00a4990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.235+0000 7f4d53400640 1 -- 172.21.15.28:0/2495296351 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639d0004f50 con 0x5639d00b8060 2024-04-24T08:16:30.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.236+0000 7f4d52a00640 1 --2- 172.21.15.28:0/3822954806 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4d380a3e20 0x7f4d380a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.246 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.238+0000 7f4d52a00640 1 --2- 172.21.15.28:0/3822954806 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4d380a3e20 0x7f4d380a62e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4d440b9610 tx=0x7f4d4402a3f0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:30.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.238+0000 7f4d53400640 1 --2- 172.21.15.28:0/2495296351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00b8060 0x5639d00ba450 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c0147a0 tx=0x7f4d4c009680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a406571da86bfc77 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:30.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.246+0000 7f4d3fe00640 1 -- 172.21.15.28:0/2495296351 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c0140f0 con 0x5639d00b8060 2024-04-24T08:16:30.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.247+0000 7f4d3fe00640 1 -- 172.21.15.28:0/2495296351 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d4c014290 con 0x5639d00b8060 2024-04-24T08:16:30.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.248+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2495296351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00b8060 msgr2=0x5639d00ba450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:30.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.248+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/2495296351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00b8060 0x5639d00ba450 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c0147a0 tx=0x7f4d4c009680 comp rx=0 tx=0).stop 2024-04-24T08:16:30.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.248+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2495296351 shutdown_connections 2024-04-24T08:16:30.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.248+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/2495296351 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00b8060 0x5639d00ba450 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.249+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/2495296351 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a4f60 0x5639d00b7b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.249+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/2495296351 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639d00a45b0 0x5639d00a4990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.249+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2495296351 >> 172.21.15.28:0/2495296351 conn(0x5639d00a0b80 msgr2=0x5639d00a0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:30.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.249+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2495296351 shutdown_connections 2024-04-24T08:16:30.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.249+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2495296351 wait complete. 2024-04-24T08:16:30.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.250+0000 7f4d5c5c7740 1 -- start start 2024-04-24T08:16:30.259 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.250+0000 7f4d5c5c7740 1 Processor -- start 2024-04-24T08:16:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.260+0000 7f4d5c5c7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00b4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.261 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.260+0000 7f4d5c5c7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00a4f60 0x5639d00b5460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.261+0000 7f4d53400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00b4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.261+0000 7f4d53400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00b4f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53208/0 (socket says 172.21.15.28:53208) 2024-04-24T08:16:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.262+0000 7f4d52000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00a4f60 0x5639d00b5460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.262+0000 7f4d5c5c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639d00b8060 0x5639d00b59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.262+0000 7f4d5c5c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5639d00baec0 con 0x5639d00b8060 2024-04-24T08:16:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.262+0000 7f4d5c5c7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5639cffc3690 con 0x5639d00a4f60 2024-04-24T08:16:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.262+0000 7f4d5c5c7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5639d00bb040 con 0x5639d00a45b0 2024-04-24T08:16:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.262+0000 7f4d53400640 1 -- 172.21.15.28:0/2153131019 learned_addr learned my addr 172.21.15.28:0/2153131019 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:30.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.262+0000 7f4d5c5c7740 10 client.0 _ll_register_callbacks cb 0x5639cffff650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:30.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.263+0000 7f4d53400640 1 -- 172.21.15.28:0/2153131019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00a4f60 msgr2=0x5639d00b5460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:30.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.264+0000 7f4d53400640 1 --2- 172.21.15.28:0/2153131019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00a4f60 0x5639d00b5460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.264+0000 7f4d53400640 1 -- 172.21.15.28:0/2153131019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639d00b8060 msgr2=0x5639d00b59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:30.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.264+0000 7f4d53400640 1 --2- 172.21.15.28:0/2153131019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639d00b8060 0x5639d00b59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.264+0000 7f4d53400640 1 -- 172.21.15.28:0/2153131019 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639cffc2920 con 0x5639d00a45b0 2024-04-24T08:16:30.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.264+0000 7f4d52000640 1 --2- 172.21.15.28:0/2153131019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00a4f60 0x5639d00b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:16:30.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.266+0000 7f4d53400640 1 --2- 172.21.15.28:0/2153131019 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00b4f20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c009a40 tx=0x7f4d4c0093c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d8abe36797112128 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:30.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.266+0000 7f4d3cc00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:16:30.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.267+0000 7f4d3cc00640 1 -- 172.21.15.28:0/2153131019 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c00f6d0 con 0x5639d00a45b0 2024-04-24T08:16:30.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.267+0000 7f4d3cc00640 1 -- 172.21.15.28:0/2153131019 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f4d4c00f870 con 0x5639d00a45b0 2024-04-24T08:16:30.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.267+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2153131019 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5639d0017c10 con 0x5639d00a45b0 2024-04-24T08:16:30.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.268+0000 7f4d3cc00640 1 -- 172.21.15.28:0/2153131019 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4d4c031050 con 0x5639d00a45b0 2024-04-24T08:16:30.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.269+0000 7f4d5c5c7740 10 client.26251 fetch_fsmap learned FSMap version 26 2024-04-24T08:16:30.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.269+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2153131019 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5639cfde67d0 con 0x5639d00a45b0 2024-04-24T08:16:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.270+0000 7f4d3cc00640 1 -- 172.21.15.28:0/2153131019 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5639d0017c10 con 0x5639d00a45b0 2024-04-24T08:16:30.271 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.270+0000 7f4d3cc00640 1 -- 172.21.15.28:0/2153131019 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4d4c00c030 con 0x5639d00a45b0 2024-04-24T08:16:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.271+0000 7f4d5c5c7740 10 client.26251 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:16:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.271+0000 7f4d5c5c7740 10 client.26251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:16:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.271+0000 7f4d5c5c7740 20 client.26251 populate_metadata read hostname 'smithi028' 2024-04-24T08:16:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.271+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/2153131019 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5639d00c82d0 0x5639d00ca6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.272+0000 7f4d5c5c7740 4 client.26251 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:16:30.273 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.272+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2153131019 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5639d0011a80 con 0x5639d00c82d0 2024-04-24T08:16:30.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.273+0000 7f4d52000640 1 --2- 172.21.15.28:0/2153131019 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5639d00c82d0 0x5639d00ca6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.275+0000 7f4d52000640 1 --2- 172.21.15.28:0/2153131019 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5639d00c82d0 0x5639d00ca6c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2f22d28071f39e5 server_cookie=10af06cc6312ef61 in_seq=0 out_seq=0 2024-04-24T08:16:30.276 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.275+0000 7f4d3cc00640 10 client.26251 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:16:30.281 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.280+0000 7f4d3cc00640 1 -- 172.21.15.28:0/2153131019 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5639d0011a80 con 0x5639d00c82d0 2024-04-24T08:16:30.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.280+0000 7f4d3cc00640 10 client.26251 handle_command_reply: tid=0 2024-04-24T08:16:30.282 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.280+0000 7f4d5c5c7740 1 client.26251 shutdown 2024-04-24T08:16:30.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.283+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2153131019 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 msgr2=0x5639d00b4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:30.284 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.283+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/2153131019 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00b4f20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c009a40 tx=0x7f4d4c0093c0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.283+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2153131019 shutdown_connections 2024-04-24T08:16:30.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.283+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/2153131019 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5639d00c82d0 0x5639d00ca6c0 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.283+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/2153131019 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639d00b8060 0x5639d00b59a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.283+0000 7f4d3cc00640 0 client.26251 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:16:30.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.283+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/2153131019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00a4f60 0x5639d00b5460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.283+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/2153131019 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00b4f20 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.286 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.283+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2153131019 >> 172.21.15.28:0/2153131019 conn(0x5639d00a0b80 msgr2=0x5639d00a69f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:30.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.286+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2153131019 shutdown_connections 2024-04-24T08:16:30.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.286+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2153131019 wait complete. 2024-04-24T08:16:30.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.287+0000 7f4d5c5c7740 20 client.26251 trim_cache size 0 max 16384 2024-04-24T08:16:30.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.338+0000 7f4d5c5c7740 1 Processor -- start 2024-04-24T08:16:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.339+0000 7f4d5c5c7740 1 -- start start 2024-04-24T08:16:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.339+0000 7f4d5c5c7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00c3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.339+0000 7f4d5c5c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639d00a4f60 0x5639d00b3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.340+0000 7f4d53400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00c3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.340+0000 7f4d5c5c7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00b8060 0x5639d00cfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.340+0000 7f4d5c5c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d541b6cd0 con 0x5639d00a4f60 2024-04-24T08:16:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.340+0000 7f4d5c5c7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4d541b45a0 con 0x5639d00b8060 2024-04-24T08:16:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.340+0000 7f4d5c5c7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4d541b2410 con 0x5639d00a45b0 2024-04-24T08:16:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.340+0000 7f4d53400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00c3e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53210/0 (socket says 172.21.15.28:53210) 2024-04-24T08:16:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.340+0000 7f4d53400640 1 -- 172.21.15.28:0/2159370780 learned_addr learned my addr 172.21.15.28:0/2159370780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.341+0000 7f4d53400640 1 -- 172.21.15.28:0/2159370780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00b8060 msgr2=0x5639d00cfc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:30.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.341+0000 7f4d53400640 1 --2- 172.21.15.28:0/2159370780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00b8060 0x5639d00cfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.341+0000 7f4d53400640 1 -- 172.21.15.28:0/2159370780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639d00a4f60 msgr2=0x5639d00b3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.341+0000 7f4d53400640 1 --2- 172.21.15.28:0/2159370780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639d00a4f60 0x5639d00b3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.341+0000 7f4d53400640 1 -- 172.21.15.28:0/2159370780 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5639cfd1c3f0 con 0x5639d00a45b0 2024-04-24T08:16:30.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.342+0000 7f4d53400640 1 --2- 172.21.15.28:0/2159370780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00c3e40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c01dbd0 tx=0x7f4d4c014760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbb0275c65138a8b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:30.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.344+0000 7f4d3f400640 1 -- 172.21.15.28:0/2159370780 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c01de10 con 0x5639d00a45b0 2024-04-24T08:16:30.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.345+0000 7f4d3f400640 1 -- 172.21.15.28:0/2159370780 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d4c00fbb0 con 0x5639d00a45b0 2024-04-24T08:16:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.345+0000 7f4d3f400640 1 -- 172.21.15.28:0/2159370780 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c01de10 con 0x5639d00a45b0 2024-04-24T08:16:30.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.346+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2159370780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 msgr2=0x5639d00c3e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:30.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.346+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/2159370780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00c3e40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c01dbd0 tx=0x7f4d4c014760 comp rx=0 tx=0).stop 2024-04-24T08:16:30.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.347+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2159370780 shutdown_connections 2024-04-24T08:16:30.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.347+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/2159370780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639d00a4f60 0x5639d00b3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.348+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/2159370780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00b8060 0x5639d00cfc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.350 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.348+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/2159370780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00c3e40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.348+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2159370780 >> 172.21.15.28:0/2159370780 conn(0x5639d00a0b80 msgr2=0x5639d00b5770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:30.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.348+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2159370780 shutdown_connections 2024-04-24T08:16:30.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.348+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/2159370780 wait complete. 2024-04-24T08:16:30.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.349+0000 7f4d5c5c7740 1 -- start start 2024-04-24T08:16:30.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.349+0000 7f4d5c5c7740 1 Processor -- start 2024-04-24T08:16:30.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.352+0000 7f4d5c5c7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00bb5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.353+0000 7f4d5c5c7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00a4f60 0x5639d00bbb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.353+0000 7f4d5c5c7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639d00b8060 0x5639d0126530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.353+0000 7f4d5c5c7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4d541b2a60 con 0x5639d00b8060 2024-04-24T08:16:30.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.353+0000 7f4d5c5c7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4d541b3ed0 con 0x5639d00a4f60 2024-04-24T08:16:30.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.353+0000 7f4d5c5c7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4d541b3130 con 0x5639d00a45b0 2024-04-24T08:16:30.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.353+0000 7f4d5c5c7740 10 client.0 _ll_register_callbacks cb 0x7f4d4000b180 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:30.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.355+0000 7f4d53400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00bb5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.355+0000 7f4d53400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00bb5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53212/0 (socket says 172.21.15.28:53212) 2024-04-24T08:16:30.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.355+0000 7f4d53400640 1 -- 172.21.15.28:0/882313672 learned_addr learned my addr 172.21.15.28:0/882313672 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:30.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.356+0000 7f4d52000640 1 --2- 172.21.15.28:0/882313672 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00a4f60 0x5639d00bbb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.356+0000 7f4d53400640 1 -- 172.21.15.28:0/882313672 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00a4f60 msgr2=0x5639d00bbb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:30.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.356+0000 7f4d53400640 1 --2- 172.21.15.28:0/882313672 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00a4f60 0x5639d00bbb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.356+0000 7f4d53400640 1 -- 172.21.15.28:0/882313672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639d00b8060 msgr2=0x5639d0126530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:30.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.356+0000 7f4d53400640 1 --2- 172.21.15.28:0/882313672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639d00b8060 0x5639d0126530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.356+0000 7f4d53400640 1 -- 172.21.15.28:0/882313672 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5639cfcdc0f0 con 0x5639d00a45b0 2024-04-24T08:16:30.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.358+0000 7f4d53400640 1 --2- 172.21.15.28:0/882313672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00bb5c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c005c10 tx=0x7f4d4c004fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e903b9d64349e18 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:30.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.358+0000 7f4d3cc00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:16:30.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.359+0000 7f4d3cc00640 1 -- 172.21.15.28:0/882313672 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4d4c03b060 con 0x5639d00a45b0 2024-04-24T08:16:30.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.359+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/882313672 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4d4c00fbb0 con 0x5639d00a45b0 2024-04-24T08:16:30.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.359+0000 7f4d3cc00640 1 -- 172.21.15.28:0/882313672 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f4d4c0097e0 con 0x5639d00a45b0 2024-04-24T08:16:30.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.360+0000 7f4d3cc00640 1 -- 172.21.15.28:0/882313672 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4d4c009c80 con 0x5639d00a45b0 2024-04-24T08:16:30.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.363+0000 7f4d5c5c7740 10 client.26257 fetch_fsmap learned FSMap version 26 2024-04-24T08:16:30.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.363+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/882313672 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5639cffcf810 con 0x5639d00a45b0 2024-04-24T08:16:30.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.363+0000 7f4d3cc00640 1 -- 172.21.15.28:0/882313672 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4d4c00fbb0 con 0x5639d00a45b0 2024-04-24T08:16:30.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.364+0000 7f4d3cc00640 1 -- 172.21.15.28:0/882313672 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4d4c00c070 con 0x5639d00a45b0 2024-04-24T08:16:30.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.364+0000 7f4d5c5c7740 10 client.26257 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:16:30.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.364+0000 7f4d5c5c7740 10 client.26257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:16:30.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.364+0000 7f4d5c5c7740 20 client.26257 populate_metadata read hostname 'smithi028' 2024-04-24T08:16:30.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.364+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/882313672 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5639d012e120 0x5639d0130540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.364+0000 7f4d5c5c7740 4 client.26257 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:16:30.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.364+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/882313672 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5639cff8d360 con 0x5639d012e120 2024-04-24T08:16:30.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.365+0000 7f4d52000640 1 --2- 172.21.15.28:0/882313672 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5639d012e120 0x5639d0130540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.366+0000 7f4d52000640 1 --2- 172.21.15.28:0/882313672 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5639d012e120 0x5639d0130540 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b354e6f248e257b server_cookie=8614b59ddaee6d1a in_seq=0 out_seq=0 2024-04-24T08:16:30.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.366+0000 7f4d3cc00640 10 client.26257 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:16:30.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.367+0000 7f4d3cc00640 1 -- 172.21.15.28:0/882313672 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5639cff8d360 con 0x5639d012e120 2024-04-24T08:16:30.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.367+0000 7f4d3cc00640 10 client.26257 handle_command_reply: tid=0 2024-04-24T08:16:30.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.367+0000 7f4d5c5c7740 1 client.26257 shutdown 2024-04-24T08:16:30.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.367+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/882313672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 msgr2=0x5639d00bb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:30.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.367+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/882313672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00bb5c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f4d4c005c10 tx=0x7f4d4c004fb0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.368+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/882313672 shutdown_connections 2024-04-24T08:16:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.368+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/882313672 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5639d012e120 0x5639d0130540 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.368+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/882313672 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5639d00b8060 0x5639d0126530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.368+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/882313672 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5639d00a4f60 0x5639d00bbb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.368+0000 7f4d5c5c7740 1 --2- 172.21.15.28:0/882313672 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5639d00a45b0 0x5639d00bb5c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.368+0000 7f4d3cc00640 0 client.26257 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:16:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.368+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/882313672 >> 172.21.15.28:0/882313672 conn(0x5639d00a0b80 msgr2=0x5639d00a60f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:30.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.368+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/882313672 shutdown_connections 2024-04-24T08:16:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.369+0000 7f4d5c5c7740 1 -- 172.21.15.28:0/882313672 wait complete. 2024-04-24T08:16:30.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.369+0000 7f4d5c5c7740 20 client.26257 trim_cache size 0 max 16384 2024-04-24T08:16:30.372 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:16:30.373 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-24T08:16:30.373 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-24T08:16:30.373 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:16:30.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.371+0000 7f4d59800640 1 -- 172.21.15.28:0/3822954806 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4d380a3e20 msgr2=0x7f4d380a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:30.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.371+0000 7f4d59800640 1 --2- 172.21.15.28:0/3822954806 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4d380a3e20 0x7f4d380a62e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f4d440b9610 tx=0x7f4d4402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.371+0000 7f4d59800640 1 -- 172.21.15.28:0/3822954806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d54109210 msgr2=0x7f4d541a3410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:30.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.371+0000 7f4d59800640 1 --2- 172.21.15.28:0/3822954806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d54109210 0x7f4d541a3410 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f4d44004770 tx=0x7f4d44003c10 comp rx=0 tx=0).stop 2024-04-24T08:16:30.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.371+0000 7f4d59800640 1 -- 172.21.15.28:0/3822954806 shutdown_connections 2024-04-24T08:16:30.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.371+0000 7f4d59800640 1 --2- 172.21.15.28:0/3822954806 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4d380a3e20 0x7f4d380a62e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.371+0000 7f4d59800640 1 --2- 172.21.15.28:0/3822954806 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4d54115150 0x7f4d541a7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.371+0000 7f4d59800640 1 --2- 172.21.15.28:0/3822954806 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4d54109b30 0x7f4d541a3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.371+0000 7f4d59800640 1 --2- 172.21.15.28:0/3822954806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d54109210 0x7f4d541a3410 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.371+0000 7f4d59800640 1 -- 172.21.15.28:0/3822954806 >> 172.21.15.28:0/3822954806 conn(0x7f4d5406d110 msgr2=0x7f4d54070d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:30.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.372+0000 7f4d59800640 1 -- 172.21.15.28:0/3822954806 shutdown_connections 2024-04-24T08:16:30.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.372+0000 7f4d59800640 1 -- 172.21.15.28:0/3822954806 wait complete. 2024-04-24T08:16:30.406 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:16:30.406 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:16:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.748+0000 7fdc94a00640 1 -- 172.21.15.28:0/3184850990 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc90109270 msgr2=0x7fdc90109650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.748+0000 7fdc94a00640 1 --2- 172.21.15.28:0/3184850990 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc90109270 0x7fdc90109650 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdc80007380 tx=0x7fdc8002afc0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.749+0000 7fdc94a00640 1 -- 172.21.15.28:0/3184850990 shutdown_connections 2024-04-24T08:16:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.749+0000 7fdc94a00640 1 --2- 172.21.15.28:0/3184850990 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc9010dd90 0x7fdc901159a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.749+0000 7fdc94a00640 1 --2- 172.21.15.28:0/3184850990 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc90109c20 0x7fdc9010d6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.749+0000 7fdc94a00640 1 --2- 172.21.15.28:0/3184850990 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc90109270 0x7fdc90109650 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.749+0000 7fdc94a00640 1 -- 172.21.15.28:0/3184850990 >> 172.21.15.28:0/3184850990 conn(0x7fdc9006d120 msgr2=0x7fdc9006d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.749+0000 7fdc94a00640 1 -- 172.21.15.28:0/3184850990 shutdown_connections 2024-04-24T08:16:30.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.749+0000 7fdc94a00640 1 -- 172.21.15.28:0/3184850990 wait complete. 2024-04-24T08:16:30.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.752+0000 7fdc94a00640 1 Processor -- start 2024-04-24T08:16:30.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.752+0000 7fdc94a00640 1 -- start start 2024-04-24T08:16:30.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.753+0000 7fdc94a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc90109270 0x7fdc901a7cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.753+0000 7fdc94a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc9010dd90 0x7fdc901a8200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.753+0000 7fdc94a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc90114b60 0x7fdc90114fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.753+0000 7fdc94a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9011ac30 con 0x7fdc90114b60 2024-04-24T08:16:30.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.753+0000 7fdc94a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9011aab0 con 0x7fdc90109270 2024-04-24T08:16:30.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.753+0000 7fdc94a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fdc9011adb0 con 0x7fdc9010dd90 2024-04-24T08:16:30.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.754+0000 7fdc8de00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc90109270 0x7fdc901a7cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.754+0000 7fdc8d400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc9010dd90 0x7fdc901a8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.754+0000 7fdc8de00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc90109270 0x7fdc901a7cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:53152/0 (socket says 172.21.15.28:53152) 2024-04-24T08:16:30.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.754+0000 7fdc8de00640 1 -- 172.21.15.28:0/4109017706 learned_addr learned my addr 172.21.15.28:0/4109017706 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.754+0000 7fdc8e800640 1 --2- 172.21.15.28:0/4109017706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc90114b60 0x7fdc90114fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.754+0000 7fdc8de00640 1 -- 172.21.15.28:0/4109017706 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc9010dd90 msgr2=0x7fdc901a8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.754+0000 7fdc8de00640 1 --2- 172.21.15.28:0/4109017706 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc9010dd90 0x7fdc901a8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.754+0000 7fdc8de00640 1 -- 172.21.15.28:0/4109017706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc90114b60 msgr2=0x7fdc90114fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.755+0000 7fdc8de00640 1 --2- 172.21.15.28:0/4109017706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc90114b60 0x7fdc90114fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.755+0000 7fdc8de00640 1 -- 172.21.15.28:0/4109017706 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdc90115850 con 0x7fdc90109270 2024-04-24T08:16:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.755+0000 7fdc8e800640 1 --2- 172.21.15.28:0/4109017706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc90114b60 0x7fdc90114fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:16:30.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.755+0000 7fdc8de00640 1 --2- 172.21.15.28:0/4109017706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc90109270 0x7fdc901a7cc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdc8002b5b0 tx=0x7fdc8002b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.755+0000 7fdc86a00640 1 -- 172.21.15.28:0/4109017706 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc80033930 con 0x7fdc90109270 2024-04-24T08:16:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.756+0000 7fdc94a00640 1 -- 172.21.15.28:0/4109017706 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdc901a37f0 con 0x7fdc90109270 2024-04-24T08:16:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.756+0000 7fdc94a00640 1 -- 172.21.15.28:0/4109017706 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdc901a3d30 con 0x7fdc90109270 2024-04-24T08:16:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.756+0000 7fdc86a00640 1 -- 172.21.15.28:0/4109017706 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdc80042070 con 0x7fdc90109270 2024-04-24T08:16:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.756+0000 7fdc86a00640 1 -- 172.21.15.28:0/4109017706 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdc8000a0d0 con 0x7fdc90109270 2024-04-24T08:16:30.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.758+0000 7fdc94a00640 1 -- 172.21.15.28:0/4109017706 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdc70005290 con 0x7fdc90109270 2024-04-24T08:16:30.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.763+0000 7fdc86a00640 1 -- 172.21.15.28:0/4109017706 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fdc8003a070 con 0x7fdc90109270 2024-04-24T08:16:30.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.766+0000 7fdc86a00640 1 --2- 172.21.15.28:0/4109017706 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdc740a3e20 0x7fdc740a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:30.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.767+0000 7fdc86a00640 1 -- 172.21.15.28:0/4109017706 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fdc800f0270 con 0x7fdc90109270 2024-04-24T08:16:30.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.768+0000 7fdc8d400640 1 --2- 172.21.15.28:0/4109017706 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdc740a3e20 0x7fdc740a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:30.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.769+0000 7fdc8d400640 1 --2- 172.21.15.28:0/4109017706 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdc740a3e20 0x7fdc740a62e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdc90115da0 tx=0x7fdc7c009260 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:30.776 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:30.773+0000 7fdc86a00640 1 -- 172.21.15.28:0/4109017706 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fdc800bc9a0 con 0x7fdc90109270 2024-04-24T08:16:31.283 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.282+0000 7fdc94a00640 1 -- 172.21.15.28:0/4109017706 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fdc70005580 con 0x7fdc90109270 2024-04-24T08:16:31.285 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.284+0000 7fdc86a00640 1 -- 172.21.15.28:0/4109017706 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11871 (secure 0 0 0) 0x7fdc800bc9a0 con 0x7fdc90109270 2024-04-24T08:16:31.287 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:16:31.287 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:14:38.663141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:16:31.287 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-24T08:16:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.288+0000 7fdc94a00640 1 -- 172.21.15.28:0/4109017706 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdc740a3e20 msgr2=0x7fdc740a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.288+0000 7fdc94a00640 1 --2- 172.21.15.28:0/4109017706 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdc740a3e20 0x7fdc740a62e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fdc90115da0 tx=0x7fdc7c009260 comp rx=0 tx=0).stop 2024-04-24T08:16:31.290 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.289+0000 7fdc94a00640 1 -- 172.21.15.28:0/4109017706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc90109270 msgr2=0x7fdc901a7cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.289+0000 7fdc94a00640 1 --2- 172.21.15.28:0/4109017706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc90109270 0x7fdc901a7cc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdc8002b5b0 tx=0x7fdc8002b900 comp rx=0 tx=0).stop 2024-04-24T08:16:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.289+0000 7fdc94a00640 1 -- 172.21.15.28:0/4109017706 shutdown_connections 2024-04-24T08:16:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.289+0000 7fdc94a00640 1 --2- 172.21.15.28:0/4109017706 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fdc740a3e20 0x7fdc740a62e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.289+0000 7fdc94a00640 1 --2- 172.21.15.28:0/4109017706 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fdc90114b60 0x7fdc90114fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.289+0000 7fdc94a00640 1 --2- 172.21.15.28:0/4109017706 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fdc9010dd90 0x7fdc901a8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.289+0000 7fdc94a00640 1 --2- 172.21.15.28:0/4109017706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc90109270 0x7fdc901a7cc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.289+0000 7fdc94a00640 1 -- 172.21.15.28:0/4109017706 >> 172.21.15.28:0/4109017706 conn(0x7fdc9006d120 msgr2=0x7fdc90072030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.289+0000 7fdc94a00640 1 -- 172.21.15.28:0/4109017706 shutdown_connections 2024-04-24T08:16:31.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.290+0000 7fdc94a00640 1 -- 172.21.15.28:0/4109017706 wait complete. 2024-04-24T08:16:31.321 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:16:31.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.643+0000 7fd853200640 1 -- 172.21.15.28:0/2103750402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd84c109000 msgr2=0x7fd84c109460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:31.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.643+0000 7fd853200640 1 --2- 172.21.15.28:0/2103750402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd84c109000 0x7fd84c109460 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fd83c009860 tx=0x7fd83c02a3d0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.645+0000 7fd853200640 1 -- 172.21.15.28:0/2103750402 shutdown_connections 2024-04-24T08:16:31.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.645+0000 7fd853200640 1 --2- 172.21.15.28:0/2103750402 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84c109a80 0x7fd84c11d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.645+0000 7fd853200640 1 --2- 172.21.15.28:0/2103750402 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd84c109000 0x7fd84c109460 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.645+0000 7fd853200640 1 --2- 172.21.15.28:0/2103750402 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd84c1106a0 0x7fd84c110a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.645+0000 7fd853200640 1 -- 172.21.15.28:0/2103750402 >> 172.21.15.28:0/2103750402 conn(0x7fd84c06cf80 msgr2=0x7fd84c06d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.645+0000 7fd853200640 1 -- 172.21.15.28:0/2103750402 shutdown_connections 2024-04-24T08:16:31.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.645+0000 7fd853200640 1 -- 172.21.15.28:0/2103750402 wait complete. 2024-04-24T08:16:31.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.647+0000 7fd853200640 1 Processor -- start 2024-04-24T08:16:31.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.647+0000 7fd853200640 1 -- start start 2024-04-24T08:16:31.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.648+0000 7fd853200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd84c109000 0x7fd84c1a73e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.648+0000 7fd853200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd84c109a80 0x7fd84c1a7920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.648+0000 7fd853200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84c1106a0 0x7fd84c1abc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.649+0000 7fd851400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd84c109a80 0x7fd84c1a7920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:31.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.649+0000 7fd851400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd84c109a80 0x7fd84c1a7920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53248/0 (socket says 172.21.15.28:53248) 2024-04-24T08:16:31.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.649+0000 7fd851e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd84c109000 0x7fd84c1a73e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:31.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.649+0000 7fd851e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd84c109000 0x7fd84c1a73e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:53170/0 (socket says 172.21.15.28:53170) 2024-04-24T08:16:31.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.649+0000 7fd851400640 1 -- 172.21.15.28:0/1643012799 learned_addr learned my addr 172.21.15.28:0/1643012799 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:31.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.650+0000 7fd853200640 1 -- 172.21.15.28:0/1643012799 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c121080 con 0x7fd84c1106a0 2024-04-24T08:16:31.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.650+0000 7fd853200640 1 -- 172.21.15.28:0/1643012799 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c120f00 con 0x7fd84c109000 2024-04-24T08:16:31.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.650+0000 7fd853200640 1 -- 172.21.15.28:0/1643012799 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c121200 con 0x7fd84c109a80 2024-04-24T08:16:31.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.650+0000 7fd852800640 1 --2- 172.21.15.28:0/1643012799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84c1106a0 0x7fd84c1abc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:31.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.650+0000 7fd851400640 1 -- 172.21.15.28:0/1643012799 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd84c109000 msgr2=0x7fd84c1a73e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:31.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.650+0000 7fd851400640 1 --2- 172.21.15.28:0/1643012799 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd84c109000 0x7fd84c1a73e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.650+0000 7fd851400640 1 -- 172.21.15.28:0/1643012799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84c1106a0 msgr2=0x7fd84c1abc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.650+0000 7fd851400640 1 --2- 172.21.15.28:0/1643012799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84c1106a0 0x7fd84c1abc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.650+0000 7fd851400640 1 -- 172.21.15.28:0/1643012799 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd84c1ac380 con 0x7fd84c109a80 2024-04-24T08:16:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.651+0000 7fd852800640 1 --2- 172.21.15.28:0/1643012799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84c1106a0 0x7fd84c1abc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:16:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.651+0000 7fd851400640 1 --2- 172.21.15.28:0/1643012799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd84c109a80 0x7fd84c1a7920 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd83c02a9c0 tx=0x7fd83c0052f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.651+0000 7fd846a00640 1 -- 172.21.15.28:0/1643012799 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c031600 con 0x7fd84c109a80 2024-04-24T08:16:31.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.651+0000 7fd846a00640 1 -- 172.21.15.28:0/1643012799 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd83c0046c0 con 0x7fd84c109a80 2024-04-24T08:16:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.651+0000 7fd846a00640 1 -- 172.21.15.28:0/1643012799 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd83c03c7c0 con 0x7fd84c109a80 2024-04-24T08:16:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.651+0000 7fd853200640 1 -- 172.21.15.28:0/1643012799 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd84c1ac610 con 0x7fd84c109a80 2024-04-24T08:16:31.654 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.652+0000 7fd853200640 1 -- 172.21.15.28:0/1643012799 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd84c1b3f00 con 0x7fd84c109a80 2024-04-24T08:16:31.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.658+0000 7fd846a00640 1 -- 172.21.15.28:0/1643012799 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fd83c039040 con 0x7fd84c109a80 2024-04-24T08:16:31.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.660+0000 7fd8575ea740 1 Processor -- start 2024-04-24T08:16:31.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.661+0000 7fd8575ea740 1 -- start start 2024-04-24T08:16:31.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.661+0000 7fd8575ea740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f3772eb40 0x563f3772ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.662+0000 7fd8575ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772f460 0x563f377395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.662+0000 7fd852800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f3772eb40 0x563f3772ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:31.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.662+0000 7fd852800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f3772eb40 0x563f3772ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53264/0 (socket says 172.21.15.28:53264) 2024-04-24T08:16:31.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.663+0000 7fd846a00640 1 --2- 172.21.15.28:0/1643012799 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd8340a3e20 0x7fd8340a62e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.664+0000 7fd8575ea740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f37739b30 0x563f3773bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.664+0000 7fd8575ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563f376941d0 con 0x563f3772f460 2024-04-24T08:16:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.664+0000 7fd8575ea740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563f3763a450 con 0x563f37739b30 2024-04-24T08:16:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.664+0000 7fd8575ea740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563f37690190 con 0x563f3772eb40 2024-04-24T08:16:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.665+0000 7fd852800640 1 -- 172.21.15.28:0/1421972893 learned_addr learned my addr 172.21.15.28:0/1421972893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.665+0000 7fd851e00640 1 --2- 172.21.15.28:0/1421972893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772f460 0x563f377395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:31.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.665+0000 7fd852800640 1 --2- 172.21.15.28:0/1643012799 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd8340a3e20 0x7fd8340a62e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.666+0000 7fd851e00640 1 -- 172.21.15.28:0/1421972893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f3772eb40 msgr2=0x563f3772ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.666+0000 7fd851e00640 1 --2- 172.21.15.28:0/1421972893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f3772eb40 0x563f3772ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.666+0000 7fd851e00640 1 -- 172.21.15.28:0/1421972893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f37739b30 msgr2=0x563f3773bf20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:16:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.666+0000 7fd851e00640 1 --2- 172.21.15.28:0/1421972893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f37739b30 0x563f3773bf20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.666+0000 7fd851e00640 1 -- 172.21.15.28:0/1421972893 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f376f7310 con 0x563f3772f460 2024-04-24T08:16:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.666+0000 7fd846a00640 1 -- 172.21.15.28:0/1643012799 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fd83c0f1a70 con 0x7fd84c109a80 2024-04-24T08:16:31.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.667+0000 7fd851e00640 1 --2- 172.21.15.28:0/1421972893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772f460 0x563f377395f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd840009c30 tx=0x7fd8400107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56103e97b0bb3ed1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.668+0000 7fd83b000640 1 -- 172.21.15.28:0/1421972893 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84001c480 con 0x563f3772f460 2024-04-24T08:16:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.668+0000 7fd83b000640 1 -- 172.21.15.28:0/1421972893 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd840011920 con 0x563f3772f460 2024-04-24T08:16:31.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.668+0000 7fd83b000640 1 -- 172.21.15.28:0/1421972893 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84001c480 con 0x563f3772f460 2024-04-24T08:16:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.669+0000 7fd8575ea740 1 -- 172.21.15.28:0/1421972893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772f460 msgr2=0x563f377395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:31.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.669+0000 7fd8575ea740 1 --2- 172.21.15.28:0/1421972893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772f460 0x563f377395f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fd840009c30 tx=0x7fd8400107a0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.669+0000 7fd852800640 1 --2- 172.21.15.28:0/1643012799 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd8340a3e20 0x7fd8340a62e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd848012e00 tx=0x7fd848004ab0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:31.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.670+0000 7fd8575ea740 1 -- 172.21.15.28:0/1421972893 shutdown_connections 2024-04-24T08:16:31.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.670+0000 7fd8575ea740 1 --2- 172.21.15.28:0/1421972893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772f460 0x563f377395f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.670+0000 7fd8575ea740 1 --2- 172.21.15.28:0/1421972893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f37739b30 0x563f3773bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.670+0000 7fd8575ea740 1 --2- 172.21.15.28:0/1421972893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f3772eb40 0x563f3772ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.670+0000 7fd8575ea740 1 -- 172.21.15.28:0/1421972893 >> 172.21.15.28:0/1421972893 conn(0x563f3772ae60 msgr2=0x563f3772b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:31.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.672+0000 7fd8575ea740 1 -- 172.21.15.28:0/1421972893 shutdown_connections 2024-04-24T08:16:31.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.672+0000 7fd8575ea740 1 -- 172.21.15.28:0/1421972893 wait complete. 2024-04-24T08:16:31.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.673+0000 7fd8575ea740 1 -- start start 2024-04-24T08:16:31.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.673+0000 7fd8575ea740 1 Processor -- start 2024-04-24T08:16:31.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.675+0000 7fd8575ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772eb40 0x563f377395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.675+0000 7fd8575ea740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f3772f460 0x563f377368e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.676+0000 7fd8575ea740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f37739b30 0x563f37737020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.676+0000 7fd8575ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563f3773c9c0 con 0x563f3772eb40 2024-04-24T08:16:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.676+0000 7fd8575ea740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563f3764d810 con 0x563f3772f460 2024-04-24T08:16:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.676+0000 7fd8575ea740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563f3773cb40 con 0x563f37739b30 2024-04-24T08:16:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.676+0000 7fd8575ea740 10 client.0 _ll_register_callbacks cb 0x563f3761b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.676+0000 7fd852800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f37739b30 0x563f37737020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.676+0000 7fd852800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f37739b30 0x563f37737020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53280/0 (socket says 172.21.15.28:53280) 2024-04-24T08:16:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.676+0000 7fd852800640 1 -- 172.21.15.28:0/901639861 learned_addr learned my addr 172.21.15.28:0/901639861 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.677+0000 7fd852800640 1 -- 172.21.15.28:0/901639861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f3772f460 msgr2=0x563f377368e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:16:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.677+0000 7fd852800640 1 --2- 172.21.15.28:0/901639861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f3772f460 0x563f377368e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.677+0000 7fd852800640 1 -- 172.21.15.28:0/901639861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772eb40 msgr2=0x563f377395f0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:16:31.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.678+0000 7fd851e00640 1 --2- 172.21.15.28:0/901639861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772eb40 0x563f377395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:31.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.678+0000 7fd852800640 1 --2- 172.21.15.28:0/901639861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772eb40 0x563f377395f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.678+0000 7fd852800640 1 -- 172.21.15.28:0/901639861 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f3765e3f0 con 0x563f37739b30 2024-04-24T08:16:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.678+0000 7fd851e00640 1 --2- 172.21.15.28:0/901639861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772eb40 0x563f377395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:16:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.678+0000 7fd852800640 1 --2- 172.21.15.28:0/901639861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f37739b30 0x563f37737020 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd848012800 tx=0x7fd848004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cf7fdfca6a9fca1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.678+0000 7fd82f400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:16:31.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.680+0000 7fd82f400640 1 -- 172.21.15.28:0/901639861 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd848025070 con 0x563f37739b30 2024-04-24T08:16:31.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.680+0000 7fd82f400640 1 -- 172.21.15.28:0/901639861 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fd848021680 con 0x563f37739b30 2024-04-24T08:16:31.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.680+0000 7fd8575ea740 1 -- 172.21.15.28:0/901639861 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd840011920 con 0x563f37739b30 2024-04-24T08:16:31.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.681+0000 7fd82f400640 1 -- 172.21.15.28:0/901639861 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd84802f050 con 0x563f37739b30 2024-04-24T08:16:31.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.681+0000 7fd8575ea740 10 client.26275 fetch_fsmap learned FSMap version 26 2024-04-24T08:16:31.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.681+0000 7fd8575ea740 1 -- 172.21.15.28:0/901639861 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563f373a63f0 con 0x563f37739b30 2024-04-24T08:16:31.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.682+0000 7fd82f400640 1 -- 172.21.15.28:0/901639861 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd840011920 con 0x563f37739b30 2024-04-24T08:16:31.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.682+0000 7fd82f400640 1 -- 172.21.15.28:0/901639861 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd84801e050 con 0x563f37739b30 2024-04-24T08:16:31.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.683+0000 7fd8575ea740 10 client.26275 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:16:31.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.683+0000 7fd8575ea740 10 client.26275 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:16:31.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.683+0000 7fd8575ea740 20 client.26275 populate_metadata read hostname 'smithi028' 2024-04-24T08:16:31.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.683+0000 7fd8575ea740 1 --2- 172.21.15.28:0/901639861 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563f37749e40 0x563f3774c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.683+0000 7fd8575ea740 4 client.26275 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:16:31.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.683+0000 7fd8575ea740 1 -- 172.21.15.28:0/901639861 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563f376a1bf0 con 0x563f37749e40 2024-04-24T08:16:31.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.683+0000 7fd851e00640 1 --2- 172.21.15.28:0/901639861 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563f37749e40 0x563f3774c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:31.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.688+0000 7fd851e00640 1 --2- 172.21.15.28:0/901639861 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563f37749e40 0x563f3774c2d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e253ef09df98af89 server_cookie=d851b9c359b22b64 in_seq=0 out_seq=0 2024-04-24T08:16:31.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.688+0000 7fd82f400640 10 client.26275 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:16:31.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.689+0000 7fd82f400640 1 -- 172.21.15.28:0/901639861 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563f376a1bf0 con 0x563f37749e40 2024-04-24T08:16:31.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.689+0000 7fd82f400640 10 client.26275 handle_command_reply: tid=0 2024-04-24T08:16:31.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.689+0000 7fd8575ea740 1 client.26275 shutdown 2024-04-24T08:16:31.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.691+0000 7fd8575ea740 1 -- 172.21.15.28:0/901639861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f37739b30 msgr2=0x563f37737020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:31.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.691+0000 7fd8575ea740 1 --2- 172.21.15.28:0/901639861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f37739b30 0x563f37737020 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fd848012800 tx=0x7fd848004a40 comp rx=0 tx=0).stop 2024-04-24T08:16:31.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.691+0000 7fd8575ea740 1 -- 172.21.15.28:0/901639861 shutdown_connections 2024-04-24T08:16:31.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.691+0000 7fd8575ea740 1 --2- 172.21.15.28:0/901639861 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563f37749e40 0x563f3774c2d0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.691+0000 7fd8575ea740 1 --2- 172.21.15.28:0/901639861 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f37739b30 0x563f37737020 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.691+0000 7fd8575ea740 1 --2- 172.21.15.28:0/901639861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f3772f460 0x563f377368e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.691+0000 7fd8575ea740 1 --2- 172.21.15.28:0/901639861 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772eb40 0x563f377395f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.691+0000 7fd8575ea740 1 -- 172.21.15.28:0/901639861 >> 172.21.15.28:0/901639861 conn(0x563f3772ae60 msgr2=0x563f37730b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.691+0000 7fd82f400640 0 client.26275 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:16:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.692+0000 7fd8575ea740 1 -- 172.21.15.28:0/901639861 shutdown_connections 2024-04-24T08:16:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.692+0000 7fd8575ea740 1 -- 172.21.15.28:0/901639861 wait complete. 2024-04-24T08:16:31.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.692+0000 7fd8575ea740 20 client.26275 trim_cache size 0 max 16384 2024-04-24T08:16:31.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.728+0000 7fd8575ea740 1 Processor -- start 2024-04-24T08:16:31.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.728+0000 7fd8575ea740 1 -- start start 2024-04-24T08:16:31.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.728+0000 7fd8575ea740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f3772eb40 0x563f377312e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.728+0000 7fd8575ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772f460 0x563f3772a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.728+0000 7fd8575ea740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f37739b30 0x563f37739120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.728+0000 7fd8575ea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c1bf140 con 0x563f3772f460 2024-04-24T08:16:31.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.728+0000 7fd8575ea740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c1bca10 con 0x563f37739b30 2024-04-24T08:16:31.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.728+0000 7fd8575ea740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c1ba860 con 0x563f3772eb40 2024-04-24T08:16:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.729+0000 7fd851e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f3772eb40 0x563f377312e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.729+0000 7fd851e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f3772eb40 0x563f377312e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53292/0 (socket says 172.21.15.28:53292) 2024-04-24T08:16:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.729+0000 7fd851e00640 1 -- 172.21.15.28:0/1159298780 learned_addr learned my addr 172.21.15.28:0/1159298780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.730+0000 7fd851e00640 1 -- 172.21.15.28:0/1159298780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f37739b30 msgr2=0x563f37739120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.730+0000 7fd851e00640 1 --2- 172.21.15.28:0/1159298780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f37739b30 0x563f37739120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.730+0000 7fd851e00640 1 -- 172.21.15.28:0/1159298780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772f460 msgr2=0x563f3772a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.730+0000 7fd851e00640 1 --2- 172.21.15.28:0/1159298780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772f460 0x563f3772a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.730+0000 7fd851e00640 1 -- 172.21.15.28:0/1159298780 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563f374707d0 con 0x563f3772eb40 2024-04-24T08:16:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.730+0000 7fd851e00640 1 --2- 172.21.15.28:0/1159298780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f3772eb40 0x563f377312e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd840000c00 tx=0x7fd8400097f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ebf18526cb1099a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.731+0000 7fd83a600640 1 -- 172.21.15.28:0/1159298780 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd840005110 con 0x563f3772eb40 2024-04-24T08:16:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.731+0000 7fd83a600640 1 -- 172.21.15.28:0/1159298780 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8400052b0 con 0x563f3772eb40 2024-04-24T08:16:31.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.731+0000 7fd83a600640 1 -- 172.21.15.28:0/1159298780 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd84002a510 con 0x563f3772eb40 2024-04-24T08:16:31.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.732+0000 7fd8575ea740 1 -- 172.21.15.28:0/1159298780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f3772eb40 msgr2=0x563f377312e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:31.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.732+0000 7fd8575ea740 1 --2- 172.21.15.28:0/1159298780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f3772eb40 0x563f377312e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fd840000c00 tx=0x7fd8400097f0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.732+0000 7fd8575ea740 1 -- 172.21.15.28:0/1159298780 shutdown_connections 2024-04-24T08:16:31.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.732+0000 7fd8575ea740 1 --2- 172.21.15.28:0/1159298780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772f460 0x563f3772a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.733+0000 7fd8575ea740 1 --2- 172.21.15.28:0/1159298780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f37739b30 0x563f37739120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.733+0000 7fd8575ea740 1 --2- 172.21.15.28:0/1159298780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f3772eb40 0x563f377312e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.733+0000 7fd8575ea740 1 -- 172.21.15.28:0/1159298780 >> 172.21.15.28:0/1159298780 conn(0x563f3772ae60 msgr2=0x563f3772b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:31.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.733+0000 7fd8575ea740 1 -- 172.21.15.28:0/1159298780 shutdown_connections 2024-04-24T08:16:31.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.733+0000 7fd8575ea740 1 -- 172.21.15.28:0/1159298780 wait complete. 2024-04-24T08:16:31.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.733+0000 7fd8575ea740 1 -- start start 2024-04-24T08:16:31.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.734+0000 7fd8575ea740 1 Processor -- start 2024-04-24T08:16:31.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.737+0000 7fd8575ea740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f3772eb40 0x563f3773b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.737+0000 7fd8575ea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772f460 0x563f37741dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.737+0000 7fd8575ea740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f37739b30 0x563f37733b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.738+0000 7fd851e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f3772eb40 0x563f3773b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.738+0000 7fd851e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f3772eb40 0x563f3773b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:53202/0 (socket says 172.21.15.28:53202) 2024-04-24T08:16:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.738+0000 7fd852800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f37739b30 0x563f37733b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.738+0000 7fd852800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f37739b30 0x563f37733b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53302/0 (socket says 172.21.15.28:53302) 2024-04-24T08:16:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.738+0000 7fd851400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772f460 0x563f37741dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.738+0000 7fd851e00640 1 -- 172.21.15.28:0/1965791452 learned_addr learned my addr 172.21.15.28:0/1965791452 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:31.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.738+0000 7fd8575ea740 1 -- 172.21.15.28:0/1965791452 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c1baed0 con 0x563f3772f460 2024-04-24T08:16:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.739+0000 7fd8575ea740 1 -- 172.21.15.28:0/1965791452 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c1bc340 con 0x563f3772eb40 2024-04-24T08:16:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.739+0000 7fd8575ea740 1 -- 172.21.15.28:0/1965791452 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd84c1bb5a0 con 0x563f37739b30 2024-04-24T08:16:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.740+0000 7fd852800640 1 -- 172.21.15.28:0/1965791452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f3772eb40 msgr2=0x563f3773b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.740+0000 7fd852800640 1 --2- 172.21.15.28:0/1965791452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f3772eb40 0x563f3773b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.740+0000 7fd852800640 1 -- 172.21.15.28:0/1965791452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772f460 msgr2=0x563f37741dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.740+0000 7fd852800640 1 --2- 172.21.15.28:0/1965791452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772f460 0x563f37741dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.740+0000 7fd852800640 1 -- 172.21.15.28:0/1965791452 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563f377340a0 con 0x563f37739b30 2024-04-24T08:16:31.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.740+0000 7fd851e00640 1 --2- 172.21.15.28:0/1965791452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f3772eb40 0x563f3773b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:16:31.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.741+0000 7fd852800640 1 --2- 172.21.15.28:0/1965791452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f37739b30 0x563f37733b60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd848000f80 tx=0x7fd84802a670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=22e83c21ede72d49 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.741+0000 7fd8575ea740 10 client.0 _ll_register_callbacks cb 0x7fd848012110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.741+0000 7fd8575ea740 1 -- 172.21.15.28:0/1965791452 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd8400052b0 con 0x563f37739b30 2024-04-24T08:16:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.741+0000 7fd82f400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:16:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.742+0000 7fd82f400640 1 -- 172.21.15.28:0/1965791452 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd848038070 con 0x563f37739b30 2024-04-24T08:16:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.742+0000 7fd82f400640 1 -- 172.21.15.28:0/1965791452 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fd848004b20 con 0x563f37739b30 2024-04-24T08:16:31.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.742+0000 7fd82f400640 1 -- 172.21.15.28:0/1965791452 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd848025040 con 0x563f37739b30 2024-04-24T08:16:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.742+0000 7fd8575ea740 10 client.26281 fetch_fsmap learned FSMap version 26 2024-04-24T08:16:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.742+0000 7fd8575ea740 1 -- 172.21.15.28:0/1965791452 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({fsmap=26}) -- 0x563f3773a4f0 con 0x563f37739b30 2024-04-24T08:16:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.743+0000 7fd82f400640 1 -- 172.21.15.28:0/1965791452 <== mon.1 v2:172.21.15.194:3300/0 4 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd84801e050 con 0x563f37739b30 2024-04-24T08:16:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.743+0000 7fd8575ea740 10 client.26281 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:16:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.743+0000 7fd8575ea740 10 client.26281 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:16:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.743+0000 7fd8575ea740 20 client.26281 populate_metadata read hostname 'smithi028' 2024-04-24T08:16:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.743+0000 7fd8575ea740 1 --2- 172.21.15.28:0/1965791452 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563f3773aa80 0x563f377b2440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:31.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.744+0000 7fd8575ea740 4 client.26281 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:16:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.744+0000 7fd8575ea740 1 -- 172.21.15.28:0/1965791452 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563f3769ba60 con 0x563f3773aa80 2024-04-24T08:16:31.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.744+0000 7fd851e00640 1 --2- 172.21.15.28:0/1965791452 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563f3773aa80 0x563f377b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:31.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.748+0000 7fd851e00640 1 --2- 172.21.15.28:0/1965791452 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563f3773aa80 0x563f377b2440 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e189516f76611090 server_cookie=a7e0653b5aad2d71 in_seq=0 out_seq=0 2024-04-24T08:16:31.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.750+0000 7fd82f400640 10 client.26281 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:16:31.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.750+0000 7fd82f400640 1 -- 172.21.15.28:0/1965791452 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563f3769ba60 con 0x563f3773aa80 2024-04-24T08:16:31.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.750+0000 7fd82f400640 10 client.26281 handle_command_reply: tid=0 2024-04-24T08:16:31.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.750+0000 7fd8575ea740 1 client.26281 shutdown 2024-04-24T08:16:31.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.750+0000 7fd8575ea740 1 -- 172.21.15.28:0/1965791452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f37739b30 msgr2=0x563f37733b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:31.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.750+0000 7fd8575ea740 1 --2- 172.21.15.28:0/1965791452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f37739b30 0x563f37733b60 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fd848000f80 tx=0x7fd84802a670 comp rx=0 tx=0).stop 2024-04-24T08:16:31.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.750+0000 7fd8575ea740 1 -- 172.21.15.28:0/1965791452 shutdown_connections 2024-04-24T08:16:31.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.750+0000 7fd8575ea740 1 --2- 172.21.15.28:0/1965791452 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563f3773aa80 0x563f377b2440 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.750+0000 7fd8575ea740 1 --2- 172.21.15.28:0/1965791452 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563f3772f460 0x563f37741dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.750+0000 7fd8575ea740 1 --2- 172.21.15.28:0/1965791452 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563f37739b30 0x563f37733b60 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.750+0000 7fd8575ea740 1 --2- 172.21.15.28:0/1965791452 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563f3772eb40 0x563f3773b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.750+0000 7fd82f400640 0 client.26281 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:16:31.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.750+0000 7fd8575ea740 1 -- 172.21.15.28:0/1965791452 >> 172.21.15.28:0/1965791452 conn(0x563f3772ae60 msgr2=0x563f37730b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:31.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.751+0000 7fd8575ea740 1 -- 172.21.15.28:0/1965791452 shutdown_connections 2024-04-24T08:16:31.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.751+0000 7fd8575ea740 1 -- 172.21.15.28:0/1965791452 wait complete. 2024-04-24T08:16:31.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.751+0000 7fd8575ea740 20 client.26281 trim_cache size 0 max 16384 2024-04-24T08:16:31.754 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:16:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.753+0000 7fd853200640 1 -- 172.21.15.28:0/1643012799 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd8340a3e20 msgr2=0x7fd8340a62e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.753+0000 7fd853200640 1 --2- 172.21.15.28:0/1643012799 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd8340a3e20 0x7fd8340a62e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fd848012e00 tx=0x7fd848004ab0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.753+0000 7fd853200640 1 -- 172.21.15.28:0/1643012799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd84c109a80 msgr2=0x7fd84c1a7920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.753+0000 7fd853200640 1 --2- 172.21.15.28:0/1643012799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd84c109a80 0x7fd84c1a7920 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fd83c02a9c0 tx=0x7fd83c0052f0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.753+0000 7fd853200640 1 -- 172.21.15.28:0/1643012799 shutdown_connections 2024-04-24T08:16:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.753+0000 7fd853200640 1 --2- 172.21.15.28:0/1643012799 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd8340a3e20 0x7fd8340a62e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.753+0000 7fd853200640 1 --2- 172.21.15.28:0/1643012799 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd84c1106a0 0x7fd84c1abc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.753+0000 7fd853200640 1 --2- 172.21.15.28:0/1643012799 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd84c109a80 0x7fd84c1a7920 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.753+0000 7fd853200640 1 --2- 172.21.15.28:0/1643012799 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd84c109000 0x7fd84c1a73e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.753+0000 7fd853200640 1 -- 172.21.15.28:0/1643012799 >> 172.21.15.28:0/1643012799 conn(0x7fd84c06cf80 msgr2=0x7fd84c070ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.753+0000 7fd853200640 1 -- 172.21.15.28:0/1643012799 shutdown_connections 2024-04-24T08:16:31.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:31.753+0000 7fd853200640 1 -- 172.21.15.28:0/1643012799 wait complete. 2024-04-24T08:16:31.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:31 smithi028 ceph-mon[27786]: pgmap v188: 97 pgs: 97 active+clean; 21 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 42 MiB/s wr, 8.72k op/s 2024-04-24T08:16:31.783 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:31 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/4109017706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:16:31.788 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:16:31.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:31 smithi117 ceph-mon[32438]: pgmap v188: 97 pgs: 97 active+clean; 21 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 42 MiB/s wr, 8.72k op/s 2024-04-24T08:16:31.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:31 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/4109017706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:16:31.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:31 smithi194 ceph-mon[32836]: pgmap v188: 97 pgs: 97 active+clean; 21 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 42 MiB/s wr, 8.72k op/s 2024-04-24T08:16:31.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:31 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/4109017706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:16:32.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.114+0000 7fee2b800640 1 -- 172.21.15.28:0/1688162002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee24102bd0 msgr2=0x7fee24106c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.114+0000 7fee2b800640 1 --2- 172.21.15.28:0/1688162002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee24102bd0 0x7fee24106c10 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fee14009860 tx=0x7fee1402a340 comp rx=0 tx=0).stop 2024-04-24T08:16:32.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.115+0000 7fee2b800640 1 -- 172.21.15.28:0/1688162002 shutdown_connections 2024-04-24T08:16:32.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.115+0000 7fee2b800640 1 --2- 172.21.15.28:0/1688162002 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee241072f0 0x7fee2410eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.115+0000 7fee2b800640 1 --2- 172.21.15.28:0/1688162002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee24102bd0 0x7fee24106c10 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.115+0000 7fee2b800640 1 --2- 172.21.15.28:0/1688162002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee24102220 0x7fee24102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.115+0000 7fee2b800640 1 -- 172.21.15.28:0/1688162002 >> 172.21.15.28:0/1688162002 conn(0x7fee240fda80 msgr2=0x7fee240ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:32.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.115+0000 7fee2b800640 1 -- 172.21.15.28:0/1688162002 shutdown_connections 2024-04-24T08:16:32.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.115+0000 7fee2b800640 1 -- 172.21.15.28:0/1688162002 wait complete. 2024-04-24T08:16:32.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.117+0000 7fee2b800640 1 Processor -- start 2024-04-24T08:16:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.123+0000 7fee2b800640 1 -- start start 2024-04-24T08:16:32.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.123+0000 7fee2b800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee24102220 0x7fee24073a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.125+0000 7fee2b800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee241072f0 0x7fee2406db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.125+0000 7fee2b800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee2406e040 0x7fee2406e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.125+0000 7fee2b800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee2406c450 con 0x7fee24102220 2024-04-24T08:16:32.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.125+0000 7fee2b800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fee2406c2d0 con 0x7fee241072f0 2024-04-24T08:16:32.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.125+0000 7fee2b800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fee2406c5d0 con 0x7fee2406e040 2024-04-24T08:16:32.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.125+0000 7fee28e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee24102220 0x7fee24073a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.125+0000 7fee23e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee241072f0 0x7fee2406db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.125+0000 7fee23e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee241072f0 0x7fee2406db00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:53218/0 (socket says 172.21.15.28:53218) 2024-04-24T08:16:32.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.125+0000 7fee23e00640 1 -- 172.21.15.28:0/1991601429 learned_addr learned my addr 172.21.15.28:0/1991601429 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.126+0000 7fee23e00640 1 -- 172.21.15.28:0/1991601429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee2406e040 msgr2=0x7fee2406e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.126+0000 7fee23e00640 1 --2- 172.21.15.28:0/1991601429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee2406e040 0x7fee2406e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.126+0000 7fee23e00640 1 -- 172.21.15.28:0/1991601429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee24102220 msgr2=0x7fee24073a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.126+0000 7fee23e00640 1 --2- 172.21.15.28:0/1991601429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee24102220 0x7fee24073a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.126+0000 7fee23e00640 1 -- 172.21.15.28:0/1991601429 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee241ab660 con 0x7fee241072f0 2024-04-24T08:16:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.126+0000 7fee23e00640 1 --2- 172.21.15.28:0/1991601429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee241072f0 0x7fee2406db00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fee14035040 tx=0x7fee14033d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:32.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.126+0000 7fee21600640 1 -- 172.21.15.28:0/1991601429 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee14037040 con 0x7fee241072f0 2024-04-24T08:16:32.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.127+0000 7fee2b800640 1 -- 172.21.15.28:0/1991601429 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee241ab8f0 con 0x7fee241072f0 2024-04-24T08:16:32.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.127+0000 7fee2b800640 1 -- 172.21.15.28:0/1991601429 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee241abe30 con 0x7fee241072f0 2024-04-24T08:16:32.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.127+0000 7fee21600640 1 -- 172.21.15.28:0/1991601429 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee14040070 con 0x7fee241072f0 2024-04-24T08:16:32.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.127+0000 7fee21600640 1 -- 172.21.15.28:0/1991601429 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee140039f0 con 0x7fee241072f0 2024-04-24T08:16:32.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.135+0000 7fee2e675740 1 Processor -- start 2024-04-24T08:16:32.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.136+0000 7fee2e675740 1 -- start start 2024-04-24T08:16:32.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.136+0000 7fee21600640 1 -- 172.21.15.28:0/1991601429 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fee14003b90 con 0x7fee241072f0 2024-04-24T08:16:32.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.137+0000 7fee2e675740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e2231950 0x5558e2231d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.138+0000 7fee29800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e2231950 0x5558e2231d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.138+0000 7fee29800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e2231950 0x5558e2231d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53328/0 (socket says 172.21.15.28:53328) 2024-04-24T08:16:32.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.138+0000 7fee29800640 1 -- 172.21.15.28:0/3456701032 learned_addr learned my addr 172.21.15.28:0/3456701032 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:32.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.139+0000 7fee2e675740 1 --2- 172.21.15.28:0/3456701032 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2232270 0x5558e223c400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.139+0000 7fee2e675740 1 --2- 172.21.15.28:0/3456701032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e223c940 0x5558e223ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.139+0000 7fee2e675740 1 -- 172.21.15.28:0/3456701032 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5558e21971d0 con 0x5558e223c940 2024-04-24T08:16:32.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.139+0000 7fee2e675740 1 -- 172.21.15.28:0/3456701032 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5558e213d450 con 0x5558e2232270 2024-04-24T08:16:32.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.139+0000 7fee2e675740 1 -- 172.21.15.28:0/3456701032 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5558e2193190 con 0x5558e2231950 2024-04-24T08:16:32.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.139+0000 7fee29800640 1 -- 172.21.15.28:0/3456701032 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2232270 msgr2=0x5558e223c400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:16:32.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.139+0000 7fee29800640 1 --2- 172.21.15.28:0/3456701032 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2232270 0x5558e223c400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.139+0000 7fee29800640 1 -- 172.21.15.28:0/3456701032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e223c940 msgr2=0x5558e223ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:32.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.139+0000 7fee23e00640 1 --2- 172.21.15.28:0/3456701032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e223c940 0x5558e223ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.140+0000 7fee21600640 1 --2- 172.21.15.28:0/1991601429 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fee080a3dd0 0x7fee080a6290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.140+0000 7fee29800640 1 --2- 172.21.15.28:0/3456701032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e223c940 0x5558e223ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.140+0000 7fee29800640 1 -- 172.21.15.28:0/3456701032 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558e21fa310 con 0x5558e2231950 2024-04-24T08:16:32.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.140+0000 7fee21600640 1 -- 172.21.15.28:0/1991601429 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fee140f06a0 con 0x7fee241072f0 2024-04-24T08:16:32.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.140+0000 7fee29800640 1 --2- 172.21.15.28:0/1991601429 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fee080a3dd0 0x7fee080a6290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.141+0000 7fee23e00640 1 --2- 172.21.15.28:0/3456701032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e223c940 0x5558e223ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:16:32.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.142+0000 7fee29800640 1 --2- 172.21.15.28:0/3456701032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e2231950 0x5558e2231d30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fee10016490 tx=0x7fee10016950 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aeb1ef468ce2fe43 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:32.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.143+0000 7fee29800640 1 --2- 172.21.15.28:0/1991601429 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fee080a3dd0 0x7fee080a6290 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fee10011000 tx=0x7fee1001d000 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:32.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.143+0000 7fee1a000640 1 -- 172.21.15.28:0/3456701032 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee10022040 con 0x5558e2231950 2024-04-24T08:16:32.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.144+0000 7fee1a000640 1 -- 172.21.15.28:0/3456701032 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee1001ac30 con 0x5558e2231950 2024-04-24T08:16:32.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.144+0000 7fee1a000640 1 -- 172.21.15.28:0/3456701032 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee10029c40 con 0x5558e2231950 2024-04-24T08:16:32.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.145+0000 7fee2e675740 1 -- 172.21.15.28:0/3456701032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e2231950 msgr2=0x5558e2231d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.145+0000 7fee2e675740 1 --2- 172.21.15.28:0/3456701032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e2231950 0x5558e2231d30 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fee10016490 tx=0x7fee10016950 comp rx=0 tx=0).stop 2024-04-24T08:16:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.147+0000 7fee2e675740 1 -- 172.21.15.28:0/3456701032 shutdown_connections 2024-04-24T08:16:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.148+0000 7fee2e675740 1 --2- 172.21.15.28:0/3456701032 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e223c940 0x5558e223ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.148+0000 7fee2e675740 1 --2- 172.21.15.28:0/3456701032 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2232270 0x5558e223c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.148+0000 7fee2e675740 1 --2- 172.21.15.28:0/3456701032 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e2231950 0x5558e2231d30 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.148+0000 7fee2e675740 1 -- 172.21.15.28:0/3456701032 >> 172.21.15.28:0/3456701032 conn(0x5558e222dc70 msgr2=0x5558e222e050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:32.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.149+0000 7fee2e675740 1 -- 172.21.15.28:0/3456701032 shutdown_connections 2024-04-24T08:16:32.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.149+0000 7fee2e675740 1 -- 172.21.15.28:0/3456701032 wait complete. 2024-04-24T08:16:32.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.150+0000 7fee2e675740 1 -- start start 2024-04-24T08:16:32.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.150+0000 7fee2e675740 1 Processor -- start 2024-04-24T08:16:32.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.152+0000 7fee2e675740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2231950 0x5558e223c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.152+0000 7fee2e675740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e2232270 0x5558e22396b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.152+0000 7fee2e675740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e223c940 0x5558e2239df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.152+0000 7fee2e675740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5558e223f7d0 con 0x5558e2232270 2024-04-24T08:16:32.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.152+0000 7fee2e675740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5558e2150810 con 0x5558e2231950 2024-04-24T08:16:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.152+0000 7fee2e675740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5558e223f950 con 0x5558e223c940 2024-04-24T08:16:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.152+0000 7fee2e675740 10 client.0 _ll_register_callbacks cb 0x5558e211e140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.153+0000 7fee28e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2231950 0x5558e223c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.153+0000 7fee29800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e223c940 0x5558e2239df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.153+0000 7fee23e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e2232270 0x5558e22396b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.154+0000 7fee28e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2231950 0x5558e223c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:53230/0 (socket says 172.21.15.28:53230) 2024-04-24T08:16:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.154+0000 7fee28e00640 1 -- 172.21.15.28:0/1513128403 learned_addr learned my addr 172.21.15.28:0/1513128403 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:32.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.153+0000 7fee29800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e223c940 0x5558e2239df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53336/0 (socket says 172.21.15.28:53336) 2024-04-24T08:16:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.155+0000 7fee29800640 1 -- 172.21.15.28:0/1513128403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2231950 msgr2=0x5558e223c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.155+0000 7fee29800640 1 --2- 172.21.15.28:0/1513128403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2231950 0x5558e223c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.155+0000 7fee29800640 1 -- 172.21.15.28:0/1513128403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e2232270 msgr2=0x5558e22396b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.155+0000 7fee29800640 1 --2- 172.21.15.28:0/1513128403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e2232270 0x5558e22396b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.155+0000 7fee29800640 1 -- 172.21.15.28:0/1513128403 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558e21613f0 con 0x5558e223c940 2024-04-24T08:16:32.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.156+0000 7fee23e00640 1 --2- 172.21.15.28:0/1513128403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e2232270 0x5558e22396b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:16:32.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.156+0000 7fee28e00640 1 --2- 172.21.15.28:0/1513128403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2231950 0x5558e223c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:16:32.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.157+0000 7fee29800640 1 --2- 172.21.15.28:0/1513128403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e223c940 0x5558e2239df0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fee10016460 tx=0x7fee10011980 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a8c7aa7675f4a412 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:32.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.157+0000 7fee0ea00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:16:32.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.157+0000 7fee0ea00640 1 -- 172.21.15.28:0/1513128403 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee10036070 con 0x5558e223c940 2024-04-24T08:16:32.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.158+0000 7fee0ea00640 1 -- 172.21.15.28:0/1513128403 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fee10029dd0 con 0x5558e223c940 2024-04-24T08:16:32.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.158+0000 7fee2e675740 1 -- 172.21.15.28:0/1513128403 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee1001ac30 con 0x5558e223c940 2024-04-24T08:16:32.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.159+0000 7fee0ea00640 1 -- 172.21.15.28:0/1513128403 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fee10040050 con 0x5558e223c940 2024-04-24T08:16:32.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.159+0000 7fee2e675740 10 client.26290 fetch_fsmap learned FSMap version 26 2024-04-24T08:16:32.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.160+0000 7fee2e675740 1 -- 172.21.15.28:0/1513128403 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5558e1ea93f0 con 0x5558e223c940 2024-04-24T08:16:32.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.161+0000 7fee0ea00640 1 -- 172.21.15.28:0/1513128403 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee1001ac30 con 0x5558e223c940 2024-04-24T08:16:32.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.162+0000 7fee0ea00640 1 -- 172.21.15.28:0/1513128403 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fee100035f0 con 0x5558e223c940 2024-04-24T08:16:32.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.162+0000 7fee2e675740 10 client.26290 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:16:32.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.162+0000 7fee2e675740 10 client.26290 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:16:32.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.162+0000 7fee2e675740 20 client.26290 populate_metadata read hostname 'smithi028' 2024-04-24T08:16:32.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.162+0000 7fee2e675740 1 --2- 172.21.15.28:0/1513128403 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5558e224cf90 0x5558e224f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.162+0000 7fee2e675740 4 client.26290 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:16:32.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.162+0000 7fee2e675740 1 -- 172.21.15.28:0/1513128403 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5558e21a4bf0 con 0x5558e224cf90 2024-04-24T08:16:32.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.162+0000 7fee28e00640 1 --2- 172.21.15.28:0/1513128403 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5558e224cf90 0x5558e224f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.163+0000 7fee28e00640 1 --2- 172.21.15.28:0/1513128403 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5558e224cf90 0x5558e224f410 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=450d3f2cbd2955ca server_cookie=dad8679bff963810 in_seq=0 out_seq=0 2024-04-24T08:16:32.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.163+0000 7fee0ea00640 10 client.26290 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:16:32.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.164+0000 7fee0ea00640 1 -- 172.21.15.28:0/1513128403 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5558e21a4bf0 con 0x5558e224cf90 2024-04-24T08:16:32.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.164+0000 7fee0ea00640 10 client.26290 handle_command_reply: tid=0 2024-04-24T08:16:32.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.164+0000 7fee2e675740 1 client.26290 shutdown 2024-04-24T08:16:32.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.166+0000 7fee2e675740 1 -- 172.21.15.28:0/1513128403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e223c940 msgr2=0x5558e2239df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.166+0000 7fee2e675740 1 --2- 172.21.15.28:0/1513128403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e223c940 0x5558e2239df0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fee10016460 tx=0x7fee10011980 comp rx=0 tx=0).stop 2024-04-24T08:16:32.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.168+0000 7fee2e675740 1 -- 172.21.15.28:0/1513128403 shutdown_connections 2024-04-24T08:16:32.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.168+0000 7fee2e675740 1 --2- 172.21.15.28:0/1513128403 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5558e224cf90 0x5558e224f410 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.168+0000 7fee2e675740 1 --2- 172.21.15.28:0/1513128403 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e2232270 0x5558e22396b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.168+0000 7fee2e675740 1 --2- 172.21.15.28:0/1513128403 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e223c940 0x5558e2239df0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.168+0000 7fee2e675740 1 --2- 172.21.15.28:0/1513128403 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2231950 0x5558e223c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.168+0000 7fee2e675740 1 -- 172.21.15.28:0/1513128403 >> 172.21.15.28:0/1513128403 conn(0x5558e222dc70 msgr2=0x5558e2233970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:32.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.168+0000 7fee0ea00640 0 client.26290 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:16:32.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.168+0000 7fee2e675740 1 -- 172.21.15.28:0/1513128403 shutdown_connections 2024-04-24T08:16:32.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.169+0000 7fee2e675740 1 -- 172.21.15.28:0/1513128403 wait complete. 2024-04-24T08:16:32.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.169+0000 7fee2e675740 20 client.26290 trim_cache size 0 max 16384 2024-04-24T08:16:32.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.206+0000 7fee2e675740 1 Processor -- start 2024-04-24T08:16:32.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.207+0000 7fee2e675740 1 -- start start 2024-04-24T08:16:32.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.207+0000 7fee2e675740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2231950 0x5558e222d570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.207+0000 7fee2e675740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e2232270 0x5558e2238690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.207+0000 7fee2e675740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e223c940 0x5558e223bef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.207+0000 7fee2e675740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee04011140 con 0x5558e223c940 2024-04-24T08:16:32.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.207+0000 7fee2e675740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fee0400ea10 con 0x5558e2231950 2024-04-24T08:16:32.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.207+0000 7fee2e675740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fee0400c820 con 0x5558e2232270 2024-04-24T08:16:32.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.208+0000 7fee28e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2231950 0x5558e222d570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.208+0000 7fee28e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2231950 0x5558e222d570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:53244/0 (socket says 172.21.15.28:53244) 2024-04-24T08:16:32.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.208+0000 7fee28e00640 1 -- 172.21.15.28:0/1974813176 learned_addr learned my addr 172.21.15.28:0/1974813176 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:32.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.208+0000 7fee28e00640 1 -- 172.21.15.28:0/1974813176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e2232270 msgr2=0x5558e2238690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:32.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.208+0000 7fee28e00640 1 --2- 172.21.15.28:0/1974813176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e2232270 0x5558e2238690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.208+0000 7fee28e00640 1 -- 172.21.15.28:0/1974813176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e223c940 msgr2=0x5558e223bef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:32.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.208+0000 7fee28e00640 1 --2- 172.21.15.28:0/1974813176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e223c940 0x5558e223bef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.208+0000 7fee28e00640 1 -- 172.21.15.28:0/1974813176 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5558e223f1e0 con 0x5558e2231950 2024-04-24T08:16:32.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.208+0000 7fee28e00640 1 --2- 172.21.15.28:0/1974813176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2231950 0x5558e222d570 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x5558e224fc50 tx=0x7fee1c00fd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a5c8dc6fb503a67 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:32.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.209+0000 7fee19600640 1 -- 172.21.15.28:0/1974813176 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c00fea0 con 0x5558e2231950 2024-04-24T08:16:32.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.209+0000 7fee19600640 1 -- 172.21.15.28:0/1974813176 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee1c00c040 con 0x5558e2231950 2024-04-24T08:16:32.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.209+0000 7fee19600640 1 -- 172.21.15.28:0/1974813176 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c0123d0 con 0x5558e2231950 2024-04-24T08:16:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.212+0000 7fee2e675740 1 -- 172.21.15.28:0/1974813176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2231950 msgr2=0x5558e222d570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.212+0000 7fee2e675740 1 --2- 172.21.15.28:0/1974813176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2231950 0x5558e222d570 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x5558e224fc50 tx=0x7fee1c00fd70 comp rx=0 tx=0).stop 2024-04-24T08:16:32.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.213+0000 7fee2e675740 1 -- 172.21.15.28:0/1974813176 shutdown_connections 2024-04-24T08:16:32.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.214+0000 7fee2e675740 1 --2- 172.21.15.28:0/1974813176 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e223c940 0x5558e223bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.214+0000 7fee2e675740 1 --2- 172.21.15.28:0/1974813176 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e2232270 0x5558e2238690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.214+0000 7fee2e675740 1 --2- 172.21.15.28:0/1974813176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e2231950 0x5558e222d570 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.214+0000 7fee2e675740 1 -- 172.21.15.28:0/1974813176 >> 172.21.15.28:0/1974813176 conn(0x5558e222dc70 msgr2=0x5558e2237f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:32.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.217+0000 7fee2e675740 1 -- 172.21.15.28:0/1974813176 shutdown_connections 2024-04-24T08:16:32.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.217+0000 7fee2e675740 1 -- 172.21.15.28:0/1974813176 wait complete. 2024-04-24T08:16:32.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.217+0000 7fee2e675740 1 -- start start 2024-04-24T08:16:32.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.218+0000 7fee2e675740 1 Processor -- start 2024-04-24T08:16:32.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.219+0000 7fee2e675740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e2231950 0x5558e223e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.219+0000 7fee2e675740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e2232270 0x5558e223cd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.219+0000 7fee2e675740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e223c940 0x5558e223d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.219+0000 7fee2e675740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fee0400ced0 con 0x5558e2231950 2024-04-24T08:16:32.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.219+0000 7fee2e675740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fee0400e340 con 0x5558e223c940 2024-04-24T08:16:32.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.219+0000 7fee2e675740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fee0400d5a0 con 0x5558e2232270 2024-04-24T08:16:32.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.219+0000 7fee2e675740 10 client.0 _ll_register_callbacks cb 0x7fee1001f040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:32.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.219+0000 7fee28e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e2231950 0x5558e223e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.220+0000 7fee28e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e2231950 0x5558e223e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39502/0 (socket says 172.21.15.28:39502) 2024-04-24T08:16:32.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.220+0000 7fee28e00640 1 -- 172.21.15.28:0/3897292660 learned_addr learned my addr 172.21.15.28:0/3897292660 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:32.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.221+0000 7fee28e00640 1 -- 172.21.15.28:0/3897292660 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e2232270 msgr2=0x5558e223cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:32.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.221+0000 7fee28e00640 1 --2- 172.21.15.28:0/3897292660 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e2232270 0x5558e223cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.221+0000 7fee29800640 1 --2- 172.21.15.28:0/3897292660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e223c940 0x5558e223d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.221+0000 7fee28e00640 1 -- 172.21.15.28:0/3897292660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e223c940 msgr2=0x5558e223d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:32.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.221+0000 7fee28e00640 1 --2- 172.21.15.28:0/3897292660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e223c940 0x5558e223d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.221+0000 7fee28e00640 1 -- 172.21.15.28:0/3897292660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5558e1f737d0 con 0x5558e2231950 2024-04-24T08:16:32.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.221+0000 7fee29800640 1 --2- 172.21.15.28:0/3897292660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e223c940 0x5558e223d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:16:32.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.222+0000 7fee28e00640 1 --2- 172.21.15.28:0/3897292660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e2231950 0x5558e223e700 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fee1c004a80 tx=0x7fee1c007cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=398ba65b188172e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:32.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.222+0000 7fee0ea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:16:32.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.223+0000 7fee0ea00640 1 -- 172.21.15.28:0/3897292660 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fee1c0225f0 con 0x5558e2231950 2024-04-24T08:16:32.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.223+0000 7fee2e675740 1 -- 172.21.15.28:0/3897292660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fee1c00c040 con 0x5558e2231950 2024-04-24T08:16:32.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.225+0000 7fee0ea00640 1 -- 172.21.15.28:0/3897292660 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fee1c022790 con 0x5558e2231950 2024-04-24T08:16:32.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.227+0000 7fee0ea00640 1 -- 172.21.15.28:0/3897292660 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fee1c014880 con 0x5558e2231950 2024-04-24T08:16:32.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.227+0000 7fee2e675740 10 client.16029 fetch_fsmap learned FSMap version 26 2024-04-24T08:16:32.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.227+0000 7fee2e675740 1 -- 172.21.15.28:0/3897292660 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5558e2244c60 con 0x5558e2231950 2024-04-24T08:16:32.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.228+0000 7fee0ea00640 1 -- 172.21.15.28:0/3897292660 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee1c00c040 con 0x5558e2231950 2024-04-24T08:16:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.228+0000 7fee0ea00640 1 -- 172.21.15.28:0/3897292660 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fee1c011040 con 0x5558e2231950 2024-04-24T08:16:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.228+0000 7fee2e675740 10 client.16029 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:16:32.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.228+0000 7fee2e675740 10 client.16029 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:16:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.228+0000 7fee2e675740 20 client.16029 populate_metadata read hostname 'smithi028' 2024-04-24T08:16:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.228+0000 7fee2e675740 1 --2- 172.21.15.28:0/3897292660 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5558e22b3550 0x5558e22b5940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.228+0000 7fee2e675740 4 client.16029 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:16:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.228+0000 7fee2e675740 1 -- 172.21.15.28:0/3897292660 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5558e219ea60 con 0x5558e22b3550 2024-04-24T08:16:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.228+0000 7fee23e00640 1 --2- 172.21.15.28:0/3897292660 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5558e22b3550 0x5558e22b5940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.229+0000 7fee23e00640 1 --2- 172.21.15.28:0/3897292660 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5558e22b3550 0x5558e22b5940 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=bab4076617d663d5 server_cookie=80835a18dd14e3f2 in_seq=0 out_seq=0 2024-04-24T08:16:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.229+0000 7fee0ea00640 10 client.16029 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:16:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.229+0000 7fee0ea00640 1 -- 172.21.15.28:0/3897292660 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5558e219ea60 con 0x5558e22b3550 2024-04-24T08:16:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.229+0000 7fee0ea00640 10 client.16029 handle_command_reply: tid=0 2024-04-24T08:16:32.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.229+0000 7fee2e675740 1 client.16029 shutdown 2024-04-24T08:16:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.229+0000 7fee2e675740 1 -- 172.21.15.28:0/3897292660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e2231950 msgr2=0x5558e223e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.229+0000 7fee2e675740 1 --2- 172.21.15.28:0/3897292660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e2231950 0x5558e223e700 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fee1c004a80 tx=0x7fee1c007cc0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.230+0000 7fee2e675740 1 -- 172.21.15.28:0/3897292660 shutdown_connections 2024-04-24T08:16:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.230+0000 7fee2e675740 1 --2- 172.21.15.28:0/3897292660 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5558e22b3550 0x5558e22b5940 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.230+0000 7fee2e675740 1 --2- 172.21.15.28:0/3897292660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5558e223c940 0x5558e223d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.230+0000 7fee2e675740 1 --2- 172.21.15.28:0/3897292660 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5558e2232270 0x5558e223cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.230+0000 7fee2e675740 1 --2- 172.21.15.28:0/3897292660 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5558e2231950 0x5558e223e700 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.230+0000 7fee2e675740 1 -- 172.21.15.28:0/3897292660 >> 172.21.15.28:0/3897292660 conn(0x5558e222dc70 msgr2=0x5558e2238810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.230+0000 7fee0ea00640 0 client.16029 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:16:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.230+0000 7fee2e675740 1 -- 172.21.15.28:0/3897292660 shutdown_connections 2024-04-24T08:16:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.230+0000 7fee2e675740 1 -- 172.21.15.28:0/3897292660 wait complete. 2024-04-24T08:16:32.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.231+0000 7fee2e675740 20 client.16029 trim_cache size 0 max 16384 2024-04-24T08:16:32.233 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:16:32.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.232+0000 7fee2b800640 1 -- 172.21.15.28:0/1991601429 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fee080a3dd0 msgr2=0x7fee080a6290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.232+0000 7fee2b800640 1 --2- 172.21.15.28:0/1991601429 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fee080a3dd0 0x7fee080a6290 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fee10011000 tx=0x7fee1001d000 comp rx=0 tx=0).stop 2024-04-24T08:16:32.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.232+0000 7fee2b800640 1 -- 172.21.15.28:0/1991601429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee241072f0 msgr2=0x7fee2406db00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.232+0000 7fee2b800640 1 --2- 172.21.15.28:0/1991601429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee241072f0 0x7fee2406db00 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fee14035040 tx=0x7fee14033d90 comp rx=0 tx=0).stop 2024-04-24T08:16:32.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.232+0000 7fee2b800640 1 -- 172.21.15.28:0/1991601429 shutdown_connections 2024-04-24T08:16:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.232+0000 7fee2b800640 1 --2- 172.21.15.28:0/1991601429 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fee080a3dd0 0x7fee080a6290 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.232+0000 7fee2b800640 1 --2- 172.21.15.28:0/1991601429 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fee2406e040 0x7fee2406e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.232+0000 7fee2b800640 1 --2- 172.21.15.28:0/1991601429 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee241072f0 0x7fee2406db00 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.232+0000 7fee2b800640 1 --2- 172.21.15.28:0/1991601429 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fee24102220 0x7fee24073a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.232+0000 7fee2b800640 1 -- 172.21.15.28:0/1991601429 >> 172.21.15.28:0/1991601429 conn(0x7fee240fda80 msgr2=0x7fee240fef00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.232+0000 7fee2b800640 1 -- 172.21.15.28:0/1991601429 shutdown_connections 2024-04-24T08:16:32.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.232+0000 7fee2b800640 1 -- 172.21.15.28:0/1991601429 wait complete. 2024-04-24T08:16:32.268 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:16:32.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.592+0000 7f2b2b200640 1 -- 172.21.15.28:0/2003907808 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b2c101770 msgr2=0x7f2b2c105c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.592+0000 7f2b2b200640 1 --2- 172.21.15.28:0/2003907808 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b2c101770 0x7f2b2c105c30 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f2b14009860 tx=0x7f2b1402a3c0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.592+0000 7f2b2b200640 1 -- 172.21.15.28:0/2003907808 shutdown_connections 2024-04-24T08:16:32.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.592+0000 7f2b2b200640 1 --2- 172.21.15.28:0/2003907808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b2c106250 0x7f2b2c1066e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.592+0000 7f2b2b200640 1 --2- 172.21.15.28:0/2003907808 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b2c101770 0x7f2b2c105c30 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.592+0000 7f2b2b200640 1 --2- 172.21.15.28:0/2003907808 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b2c100e50 0x7f2b2c101230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.592+0000 7f2b2b200640 1 -- 172.21.15.28:0/2003907808 >> 172.21.15.28:0/2003907808 conn(0x7f2b2c0fc880 msgr2=0x7f2b2c0feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:32.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.592+0000 7f2b2b200640 1 -- 172.21.15.28:0/2003907808 shutdown_connections 2024-04-24T08:16:32.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.592+0000 7f2b2b200640 1 -- 172.21.15.28:0/2003907808 wait complete. 2024-04-24T08:16:32.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.594+0000 7f2b2b200640 1 Processor -- start 2024-04-24T08:16:32.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.594+0000 7f2b2b200640 1 -- start start 2024-04-24T08:16:32.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.595+0000 7f2b2b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b2c100e50 0x7f2b2c0737c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.595+0000 7f2b2b200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b2c101770 0x7f2b2c06c840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.595+0000 7f2b2b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b2c106250 0x7f2b2c06cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.595+0000 7f2b2b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2c118850 con 0x7f2b2c100e50 2024-04-24T08:16:32.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.596+0000 7f2b2a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b2c106250 0x7f2b2c06cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.596+0000 7f2b2a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b2c106250 0x7f2b2c06cd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53378/0 (socket says 172.21.15.28:53378) 2024-04-24T08:16:32.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.596+0000 7f2b2a800640 1 -- 172.21.15.28:0/3535116324 learned_addr learned my addr 172.21.15.28:0/3535116324 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:32.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.596+0000 7f2b29400640 1 --2- 172.21.15.28:0/3535116324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b2c101770 0x7f2b2c06c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.596+0000 7f2b2b200640 1 -- 172.21.15.28:0/3535116324 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2c1186d0 con 0x7f2b2c101770 2024-04-24T08:16:32.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.596+0000 7f2b2b200640 1 -- 172.21.15.28:0/3535116324 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2b2c1189d0 con 0x7f2b2c106250 2024-04-24T08:16:32.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.597+0000 7f2b29400640 1 -- 172.21.15.28:0/3535116324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b2c106250 msgr2=0x7f2b2c06cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.597+0000 7f2b29400640 1 --2- 172.21.15.28:0/3535116324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b2c106250 0x7f2b2c06cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.597+0000 7f2b29e00640 1 --2- 172.21.15.28:0/3535116324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b2c100e50 0x7f2b2c0737c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.597+0000 7f2b29400640 1 -- 172.21.15.28:0/3535116324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b2c100e50 msgr2=0x7f2b2c0737c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.597+0000 7f2b29400640 1 --2- 172.21.15.28:0/3535116324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b2c100e50 0x7f2b2c0737c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.597+0000 7f2b29400640 1 -- 172.21.15.28:0/3535116324 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2b2c06d2f0 con 0x7f2b2c101770 2024-04-24T08:16:32.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.597+0000 7f2b29e00640 1 --2- 172.21.15.28:0/3535116324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b2c100e50 0x7f2b2c0737c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:16:32.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.599+0000 7f2b29400640 1 --2- 172.21.15.28:0/3535116324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b2c101770 0x7f2b2c06c840 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2b14035040 tx=0x7f2b14030430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:32.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.599+0000 7f2b22a00640 1 -- 172.21.15.28:0/3535116324 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b1403f020 con 0x7f2b2c101770 2024-04-24T08:16:32.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.599+0000 7f2b2b200640 1 -- 172.21.15.28:0/3535116324 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2b2c06d580 con 0x7f2b2c101770 2024-04-24T08:16:32.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.599+0000 7f2b2b200640 1 -- 172.21.15.28:0/3535116324 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2b2c071570 con 0x7f2b2c101770 2024-04-24T08:16:32.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.599+0000 7f2b22a00640 1 -- 172.21.15.28:0/3535116324 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b140042c0 con 0x7f2b2c101770 2024-04-24T08:16:32.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.599+0000 7f2b22a00640 1 -- 172.21.15.28:0/3535116324 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b1403f020 con 0x7f2b2c101770 2024-04-24T08:16:32.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.603+0000 7f2b339cb740 1 Processor -- start 2024-04-24T08:16:32.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.604+0000 7f2b339cb740 1 -- start start 2024-04-24T08:16:32.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.604+0000 7f2b22a00640 1 -- 172.21.15.28:0/3535116324 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f2b14042020 con 0x7f2b2c101770 2024-04-24T08:16:32.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.606+0000 7f2b339cb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3d7b40 0x5599fe3d7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.606+0000 7f2b339cb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3d8460 0x5599fe3e25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.606+0000 7f2b22a00640 1 --2- 172.21.15.28:0/3535116324 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2b100a4020 0x7f2b100a64e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.606+0000 7f2b22a00640 1 -- 172.21.15.28:0/3535116324 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f2b140efbd0 con 0x7f2b2c101770 2024-04-24T08:16:32.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.607+0000 7f2b2a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3d7b40 0x5599fe3d7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.607+0000 7f2b2a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3d7b40 0x5599fe3d7f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39516/0 (socket says 172.21.15.28:39516) 2024-04-24T08:16:32.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.607+0000 7f2b339cb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3e2b30 0x5599fe3e4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.607+0000 7f2b339cb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5599fe33d1d0 con 0x5599fe3d7b40 2024-04-24T08:16:32.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.607+0000 7f2b339cb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5599fe2e3450 con 0x5599fe3d8460 2024-04-24T08:16:32.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.607+0000 7f2b339cb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5599fe339190 con 0x5599fe3e2b30 2024-04-24T08:16:32.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.607+0000 7f2b2a800640 1 -- 172.21.15.28:0/184643325 learned_addr learned my addr 172.21.15.28:0/184643325 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:32.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.606+0000 7f2b29e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3d8460 0x5599fe3e25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.608+0000 7f2b29400640 1 --2- 172.21.15.28:0/184643325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3e2b30 0x5599fe3e4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.608+0000 7f2b2a800640 1 --2- 172.21.15.28:0/3535116324 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2b100a4020 0x7f2b100a64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.608+0000 7f2b29e00640 1 -- 172.21.15.28:0/184643325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3e2b30 msgr2=0x5599fe3e4f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:32.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.608+0000 7f2b29e00640 1 --2- 172.21.15.28:0/184643325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3e2b30 0x5599fe3e4f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.608+0000 7f2b29e00640 1 -- 172.21.15.28:0/184643325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3d7b40 msgr2=0x5599fe3d7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:32.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.608+0000 7f2b29e00640 1 --2- 172.21.15.28:0/184643325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3d7b40 0x5599fe3d7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.608+0000 7f2b29e00640 1 -- 172.21.15.28:0/184643325 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599fe3a0310 con 0x5599fe3d8460 2024-04-24T08:16:32.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.609+0000 7f2b29e00640 1 --2- 172.21.15.28:0/184643325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3d8460 0x5599fe3e25f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2b24008e50 tx=0x7f2b2400d710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c84cf3d647085bce server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.609+0000 7f2b2a800640 1 --2- 172.21.15.28:0/3535116324 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2b100a4020 0x7f2b100a64e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c009700 tx=0x7f2b1c004730 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.610+0000 7f2b1b000640 1 -- 172.21.15.28:0/184643325 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2400f830 con 0x5599fe3d8460 2024-04-24T08:16:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.610+0000 7f2b1b000640 1 -- 172.21.15.28:0/184643325 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b24018480 con 0x5599fe3d8460 2024-04-24T08:16:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.610+0000 7f2b1b000640 1 -- 172.21.15.28:0/184643325 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b2400e610 con 0x5599fe3d8460 2024-04-24T08:16:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.611+0000 7f2b339cb740 1 -- 172.21.15.28:0/184643325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3d8460 msgr2=0x5599fe3e25f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.611+0000 7f2b339cb740 1 --2- 172.21.15.28:0/184643325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3d8460 0x5599fe3e25f0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f2b24008e50 tx=0x7f2b2400d710 comp rx=0 tx=0).stop 2024-04-24T08:16:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.611+0000 7f2b339cb740 1 -- 172.21.15.28:0/184643325 shutdown_connections 2024-04-24T08:16:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.611+0000 7f2b339cb740 1 --2- 172.21.15.28:0/184643325 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3e2b30 0x5599fe3e4f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.611+0000 7f2b339cb740 1 --2- 172.21.15.28:0/184643325 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3d8460 0x5599fe3e25f0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.611+0000 7f2b339cb740 1 --2- 172.21.15.28:0/184643325 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3d7b40 0x5599fe3d7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.611+0000 7f2b339cb740 1 -- 172.21.15.28:0/184643325 >> 172.21.15.28:0/184643325 conn(0x5599fe3d3e60 msgr2=0x5599fe3d4240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:32.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.612+0000 7f2b339cb740 1 -- 172.21.15.28:0/184643325 shutdown_connections 2024-04-24T08:16:32.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.612+0000 7f2b339cb740 1 -- 172.21.15.28:0/184643325 wait complete. 2024-04-24T08:16:32.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.613+0000 7f2b339cb740 1 -- start start 2024-04-24T08:16:32.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.613+0000 7f2b339cb740 1 Processor -- start 2024-04-24T08:16:32.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.614+0000 7f2b339cb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3d7b40 0x5599fe3dfab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.614+0000 7f2b339cb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e2b30 0x5599fe3dfff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.614+0000 7f2b339cb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3e0530 0x5599fe3eee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.614+0000 7f2b29e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3d7b40 0x5599fe3dfab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.614+0000 7f2b339cb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5599fe3e59c0 con 0x5599fe3e2b30 2024-04-24T08:16:32.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.614+0000 7f2b339cb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5599fe2f6810 con 0x5599fe3d7b40 2024-04-24T08:16:32.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.614+0000 7f2b339cb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5599fe3e5b40 con 0x5599fe3e0530 2024-04-24T08:16:32.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.614+0000 7f2b29e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3d7b40 0x5599fe3dfab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:53298/0 (socket says 172.21.15.28:53298) 2024-04-24T08:16:32.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.614+0000 7f2b29e00640 1 -- 172.21.15.28:0/2839364047 learned_addr learned my addr 172.21.15.28:0/2839364047 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:32.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.614+0000 7f2b2a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3e0530 0x5599fe3eee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.614+0000 7f2b339cb740 10 client.0 _ll_register_callbacks cb 0x5599fe2c4140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.615+0000 7f2b2a800640 1 -- 172.21.15.28:0/2839364047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3d7b40 msgr2=0x5599fe3dfab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.615+0000 7f2b2a800640 1 --2- 172.21.15.28:0/2839364047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3d7b40 0x5599fe3dfab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.615+0000 7f2b2a800640 1 -- 172.21.15.28:0/2839364047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e2b30 msgr2=0x5599fe3dfff0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:16:32.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.617+0000 7f2b29400640 1 --2- 172.21.15.28:0/2839364047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e2b30 0x5599fe3dfff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.617+0000 7f2b2a800640 1 --2- 172.21.15.28:0/2839364047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e2b30 0x5599fe3dfff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.617+0000 7f2b2a800640 1 -- 172.21.15.28:0/2839364047 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599fe3073f0 con 0x5599fe3e0530 2024-04-24T08:16:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.617+0000 7f2b2a800640 1 --2- 172.21.15.28:0/2839364047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3e0530 0x5599fe3eee90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c00f790 tx=0x7f2b1c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97888717aa9abbeb server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:32.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.617+0000 7f2b0f400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:16:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.618+0000 7f2b0f400640 1 -- 172.21.15.28:0/2839364047 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b1c010e10 con 0x5599fe3e0530 2024-04-24T08:16:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.618+0000 7f2b0f400640 1 -- 172.21.15.28:0/2839364047 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f2b1c005020 con 0x5599fe3e0530 2024-04-24T08:16:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.618+0000 7f2b339cb740 1 -- 172.21.15.28:0/2839364047 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b24018480 con 0x5599fe3e0530 2024-04-24T08:16:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.620+0000 7f2b0f400640 1 -- 172.21.15.28:0/2839364047 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2b1c009b80 con 0x5599fe3e0530 2024-04-24T08:16:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.620+0000 7f2b339cb740 10 client.26302 fetch_fsmap learned FSMap version 26 2024-04-24T08:16:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.620+0000 7f2b339cb740 1 -- 172.21.15.28:0/2839364047 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5599fe04f3f0 con 0x5599fe3e0530 2024-04-24T08:16:32.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.620+0000 7f2b0f400640 1 -- 172.21.15.28:0/2839364047 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b24018480 con 0x5599fe3e0530 2024-04-24T08:16:32.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.623+0000 7f2b0f400640 1 -- 172.21.15.28:0/2839364047 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2b1c005220 con 0x5599fe3e0530 2024-04-24T08:16:32.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.623+0000 7f2b339cb740 10 client.26302 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:16:32.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.623+0000 7f2b339cb740 10 client.26302 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:16:32.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.623+0000 7f2b339cb740 20 client.26302 populate_metadata read hostname 'smithi028' 2024-04-24T08:16:32.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.623+0000 7f2b339cb740 1 --2- 172.21.15.28:0/2839364047 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5599fe3f3390 0x5599fe3f5850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.623+0000 7f2b339cb740 4 client.26302 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:16:32.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.623+0000 7f2b339cb740 1 -- 172.21.15.28:0/2839364047 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5599fe34abf0 con 0x5599fe3f3390 2024-04-24T08:16:32.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.624+0000 7f2b29e00640 1 --2- 172.21.15.28:0/2839364047 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5599fe3f3390 0x5599fe3f5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.625+0000 7f2b0f400640 10 client.26302 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:16:32.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.625+0000 7f2b29e00640 1 --2- 172.21.15.28:0/2839364047 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5599fe3f3390 0x5599fe3f5850 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=adcdc36c087d80b0 server_cookie=acfd105800868012 in_seq=0 out_seq=0 2024-04-24T08:16:32.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.627+0000 7f2b0f400640 1 -- 172.21.15.28:0/2839364047 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5599fe34abf0 con 0x5599fe3f3390 2024-04-24T08:16:32.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.627+0000 7f2b0f400640 10 client.26302 handle_command_reply: tid=0 2024-04-24T08:16:32.628 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.627+0000 7f2b339cb740 1 client.26302 shutdown 2024-04-24T08:16:32.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.629+0000 7f2b339cb740 1 -- 172.21.15.28:0/2839364047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3e0530 msgr2=0x5599fe3eee90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.629+0000 7f2b339cb740 1 --2- 172.21.15.28:0/2839364047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3e0530 0x5599fe3eee90 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c00f790 tx=0x7f2b1c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.630+0000 7f2b339cb740 1 -- 172.21.15.28:0/2839364047 shutdown_connections 2024-04-24T08:16:32.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.630+0000 7f2b339cb740 1 --2- 172.21.15.28:0/2839364047 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5599fe3f3390 0x5599fe3f5850 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.630+0000 7f2b0f400640 0 client.26302 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:16:32.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.630+0000 7f2b339cb740 1 --2- 172.21.15.28:0/2839364047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e2b30 0x5599fe3dfff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.630+0000 7f2b339cb740 1 --2- 172.21.15.28:0/2839364047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3e0530 0x5599fe3eee90 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.630+0000 7f2b339cb740 1 --2- 172.21.15.28:0/2839364047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3d7b40 0x5599fe3dfab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.630+0000 7f2b339cb740 1 -- 172.21.15.28:0/2839364047 >> 172.21.15.28:0/2839364047 conn(0x5599fe3d3e60 msgr2=0x5599fe3d9b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:32.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.630+0000 7f2b339cb740 1 -- 172.21.15.28:0/2839364047 shutdown_connections 2024-04-24T08:16:32.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.630+0000 7f2b339cb740 1 -- 172.21.15.28:0/2839364047 wait complete. 2024-04-24T08:16:32.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.631+0000 7f2b339cb740 20 client.26302 trim_cache size 0 max 16384 2024-04-24T08:16:32.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.661+0000 7f2b339cb740 1 Processor -- start 2024-04-24T08:16:32.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.662+0000 7f2b339cb740 1 -- start start 2024-04-24T08:16:32.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.662+0000 7f2b339cb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3d7b40 0x5599fe3d36d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.664 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.662+0000 7f2b339cb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e0530 0x5599fe3dea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.662+0000 7f2b339cb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3e2b30 0x5599fe3fb170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.662+0000 7f2b339cb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b08011100 con 0x5599fe3e0530 2024-04-24T08:16:32.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.662+0000 7f2b339cb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0800e9d0 con 0x5599fe3e2b30 2024-04-24T08:16:32.665 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.662+0000 7f2b339cb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0800c820 con 0x5599fe3d7b40 2024-04-24T08:16:32.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.663+0000 7f2b29e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3d7b40 0x5599fe3d36d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.663+0000 7f2b2a800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3e2b30 0x5599fe3fb170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.664+0000 7f2b2a800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3e2b30 0x5599fe3fb170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:53306/0 (socket says 172.21.15.28:53306) 2024-04-24T08:16:32.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.664+0000 7f2b2a800640 1 -- 172.21.15.28:0/4260508694 learned_addr learned my addr 172.21.15.28:0/4260508694 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:32.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.664+0000 7f2b29e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3d7b40 0x5599fe3d36d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53396/0 (socket says 172.21.15.28:53396) 2024-04-24T08:16:32.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.663+0000 7f2b29400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e0530 0x5599fe3dea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.664+0000 7f2b29400640 1 -- 172.21.15.28:0/4260508694 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3d7b40 msgr2=0x5599fe3d36d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:32.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.664+0000 7f2b29400640 1 --2- 172.21.15.28:0/4260508694 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3d7b40 0x5599fe3d36d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.664+0000 7f2b29400640 1 -- 172.21.15.28:0/4260508694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3e2b30 msgr2=0x5599fe3fb170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:32.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.664+0000 7f2b29400640 1 --2- 172.21.15.28:0/4260508694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3e2b30 0x5599fe3fb170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.664+0000 7f2b29400640 1 -- 172.21.15.28:0/4260508694 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5599fe1197d0 con 0x5599fe3e0530 2024-04-24T08:16:32.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.665+0000 7f2b29e00640 1 --2- 172.21.15.28:0/4260508694 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3d7b40 0x5599fe3d36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:16:32.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.665+0000 7f2b29400640 1 --2- 172.21.15.28:0/4260508694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e0530 0x5599fe3dea90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2b140ed3b0 tx=0x7f2b1402a3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e809556c596cb55d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:32.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.666+0000 7f2b1a600640 1 -- 172.21.15.28:0/4260508694 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b140d4020 con 0x5599fe3e0530 2024-04-24T08:16:32.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.666+0000 7f2b1a600640 1 -- 172.21.15.28:0/4260508694 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b140bffb0 con 0x5599fe3e0530 2024-04-24T08:16:32.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.666+0000 7f2b1a600640 1 -- 172.21.15.28:0/4260508694 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b14037030 con 0x5599fe3e0530 2024-04-24T08:16:32.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.669+0000 7f2b339cb740 1 -- 172.21.15.28:0/4260508694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e0530 msgr2=0x5599fe3dea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.669+0000 7f2b339cb740 1 --2- 172.21.15.28:0/4260508694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e0530 0x5599fe3dea90 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2b140ed3b0 tx=0x7f2b1402a3c0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.672+0000 7f2b339cb740 1 -- 172.21.15.28:0/4260508694 shutdown_connections 2024-04-24T08:16:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.672+0000 7f2b339cb740 1 --2- 172.21.15.28:0/4260508694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e0530 0x5599fe3dea90 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.672+0000 7f2b339cb740 1 --2- 172.21.15.28:0/4260508694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3e2b30 0x5599fe3fb170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.672+0000 7f2b339cb740 1 --2- 172.21.15.28:0/4260508694 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3d7b40 0x5599fe3d36d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.672+0000 7f2b339cb740 1 -- 172.21.15.28:0/4260508694 >> 172.21.15.28:0/4260508694 conn(0x5599fe3d3e60 msgr2=0x5599fe3de380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:32.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.673+0000 7f2b339cb740 1 -- 172.21.15.28:0/4260508694 shutdown_connections 2024-04-24T08:16:32.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.675+0000 7f2b339cb740 1 -- 172.21.15.28:0/4260508694 wait complete. 2024-04-24T08:16:32.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.675+0000 7f2b339cb740 1 -- start start 2024-04-24T08:16:32.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.676+0000 7f2b339cb740 1 Processor -- start 2024-04-24T08:16:32.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.676+0000 7f2b339cb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3d7b40 0x5599fe3eadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.676+0000 7f2b339cb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e0530 0x5599fe3eb330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.676+0000 7f2b339cb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3e2b30 0x5599fe3dc150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.676+0000 7f2b339cb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0800ce90 con 0x5599fe3e0530 2024-04-24T08:16:32.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.676+0000 7f2b339cb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0800e300 con 0x5599fe3e2b30 2024-04-24T08:16:32.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.676+0000 7f2b339cb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2b0800d560 con 0x5599fe3d7b40 2024-04-24T08:16:32.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.676+0000 7f2b339cb740 10 client.0 _ll_register_callbacks cb 0x5599fe3e2a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:32.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.677+0000 7f2b29e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3d7b40 0x5599fe3eadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.677+0000 7f2b29400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e0530 0x5599fe3eb330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.677+0000 7f2b29e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3d7b40 0x5599fe3eadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53406/0 (socket says 172.21.15.28:53406) 2024-04-24T08:16:32.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.677+0000 7f2b29400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e0530 0x5599fe3eb330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39546/0 (socket says 172.21.15.28:39546) 2024-04-24T08:16:32.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.677+0000 7f2b29e00640 1 -- 172.21.15.28:0/2909679439 learned_addr learned my addr 172.21.15.28:0/2909679439 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:32.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.677+0000 7f2b2a800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3e2b30 0x5599fe3dc150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.678+0000 7f2b29e00640 1 -- 172.21.15.28:0/2909679439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3e2b30 msgr2=0x5599fe3dc150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:32.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.678+0000 7f2b29e00640 1 --2- 172.21.15.28:0/2909679439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3e2b30 0x5599fe3dc150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.678+0000 7f2b29e00640 1 -- 172.21.15.28:0/2909679439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e0530 msgr2=0x5599fe3eb330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:32.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.678+0000 7f2b29e00640 1 --2- 172.21.15.28:0/2909679439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e0530 0x5599fe3eb330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.678+0000 7f2b29e00640 1 -- 172.21.15.28:0/2909679439 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5599fe3dc980 con 0x5599fe3d7b40 2024-04-24T08:16:32.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.678+0000 7f2b29400640 1 --2- 172.21.15.28:0/2909679439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e0530 0x5599fe3eb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:16:32.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.678+0000 7f2b2a800640 1 --2- 172.21.15.28:0/2909679439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3e2b30 0x5599fe3dc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:16:32.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.679+0000 7f2b29e00640 1 --2- 172.21.15.28:0/2909679439 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3d7b40 0x5599fe3eadf0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x5599fe3d9100 tx=0x7f2b24018e10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=173418205d67f459 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:32.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.679+0000 7f2b0f400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:16:32.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.680+0000 7f2b0f400640 1 -- 172.21.15.28:0/2909679439 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2b24004be0 con 0x5599fe3d7b40 2024-04-24T08:16:32.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.680+0000 7f2b0f400640 1 -- 172.21.15.28:0/2909679439 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f2b24004d80 con 0x5599fe3d7b40 2024-04-24T08:16:32.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.681+0000 7f2b339cb740 1 -- 172.21.15.28:0/2909679439 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2b140bffb0 con 0x5599fe3d7b40 2024-04-24T08:16:32.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.681+0000 7f2b0f400640 1 -- 172.21.15.28:0/2909679439 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2b240085f0 con 0x5599fe3d7b40 2024-04-24T08:16:32.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.681+0000 7f2b339cb740 10 client.26305 fetch_fsmap learned FSMap version 26 2024-04-24T08:16:32.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.681+0000 7f2b339cb740 1 -- 172.21.15.28:0/2909679439 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5599fe459490 con 0x5599fe3d7b40 2024-04-24T08:16:32.683 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.682+0000 7f2b0f400640 1 -- 172.21.15.28:0/2909679439 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2b140bffb0 con 0x5599fe3d7b40 2024-04-24T08:16:32.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.682+0000 7f2b0f400640 1 -- 172.21.15.28:0/2909679439 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2b2400fb20 con 0x5599fe3d7b40 2024-04-24T08:16:32.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.683+0000 7f2b339cb740 10 client.26305 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:16:32.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.683+0000 7f2b339cb740 10 client.26305 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:16:32.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.683+0000 7f2b339cb740 20 client.26305 populate_metadata read hostname 'smithi028' 2024-04-24T08:16:32.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.683+0000 7f2b339cb740 1 --2- 172.21.15.28:0/2909679439 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5599fe459980 0x5599fe45bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:32.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.683+0000 7f2b339cb740 4 client.26305 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:16:32.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.683+0000 7f2b339cb740 1 -- 172.21.15.28:0/2909679439 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5599fe344a60 con 0x5599fe459980 2024-04-24T08:16:32.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.683+0000 7f2b29400640 1 --2- 172.21.15.28:0/2909679439 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5599fe459980 0x5599fe45bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:32.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.685+0000 7f2b29400640 1 --2- 172.21.15.28:0/2909679439 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5599fe459980 0x5599fe45bd90 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8c825f37549f154 server_cookie=2112db5e35b52865 in_seq=0 out_seq=0 2024-04-24T08:16:32.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.685+0000 7f2b0f400640 10 client.26305 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:16:32.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.685+0000 7f2b0f400640 1 -- 172.21.15.28:0/2909679439 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5599fe344a60 con 0x5599fe459980 2024-04-24T08:16:32.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.685+0000 7f2b0f400640 10 client.26305 handle_command_reply: tid=0 2024-04-24T08:16:32.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.685+0000 7f2b339cb740 1 client.26305 shutdown 2024-04-24T08:16:32.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.686+0000 7f2b339cb740 1 -- 172.21.15.28:0/2909679439 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3d7b40 msgr2=0x5599fe3eadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.686+0000 7f2b339cb740 1 --2- 172.21.15.28:0/2909679439 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3d7b40 0x5599fe3eadf0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x5599fe3d9100 tx=0x7f2b24018e10 comp rx=0 tx=0).stop 2024-04-24T08:16:32.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.686+0000 7f2b339cb740 1 -- 172.21.15.28:0/2909679439 shutdown_connections 2024-04-24T08:16:32.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.686+0000 7f2b339cb740 1 --2- 172.21.15.28:0/2909679439 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5599fe459980 0x5599fe45bd90 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.686+0000 7f2b339cb740 1 --2- 172.21.15.28:0/2909679439 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5599fe3e0530 0x5599fe3eb330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.686+0000 7f2b0f400640 0 client.26305 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:16:32.688 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.686+0000 7f2b339cb740 1 --2- 172.21.15.28:0/2909679439 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5599fe3e2b30 0x5599fe3dc150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.686+0000 7f2b339cb740 1 --2- 172.21.15.28:0/2909679439 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5599fe3d7b40 0x5599fe3eadf0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.686+0000 7f2b339cb740 1 -- 172.21.15.28:0/2909679439 >> 172.21.15.28:0/2909679439 conn(0x5599fe3d3e60 msgr2=0x5599fe3dec10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:32.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.687+0000 7f2b339cb740 1 -- 172.21.15.28:0/2909679439 shutdown_connections 2024-04-24T08:16:32.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.687+0000 7f2b339cb740 1 -- 172.21.15.28:0/2909679439 wait complete. 2024-04-24T08:16:32.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.687+0000 7f2b339cb740 20 client.26305 trim_cache size 0 max 16384 2024-04-24T08:16:32.691 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:16:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.691+0000 7f2b2b200640 1 -- 172.21.15.28:0/3535116324 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2b100a4020 msgr2=0x7f2b100a64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.691+0000 7f2b2b200640 1 --2- 172.21.15.28:0/3535116324 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2b100a4020 0x7f2b100a64e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f2b1c009700 tx=0x7f2b1c004730 comp rx=0 tx=0).stop 2024-04-24T08:16:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.691+0000 7f2b2b200640 1 -- 172.21.15.28:0/3535116324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b2c101770 msgr2=0x7f2b2c06c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.691+0000 7f2b2b200640 1 --2- 172.21.15.28:0/3535116324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b2c101770 0x7f2b2c06c840 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f2b14035040 tx=0x7f2b14030430 comp rx=0 tx=0).stop 2024-04-24T08:16:32.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.692+0000 7f2b2b200640 1 -- 172.21.15.28:0/3535116324 shutdown_connections 2024-04-24T08:16:32.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.692+0000 7f2b2b200640 1 --2- 172.21.15.28:0/3535116324 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2b100a4020 0x7f2b100a64e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.692+0000 7f2b2b200640 1 --2- 172.21.15.28:0/3535116324 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2b2c106250 0x7f2b2c06cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.692+0000 7f2b2b200640 1 --2- 172.21.15.28:0/3535116324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b2c101770 0x7f2b2c06c840 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.692+0000 7f2b2b200640 1 --2- 172.21.15.28:0/3535116324 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2b2c100e50 0x7f2b2c0737c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:32.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.692+0000 7f2b2b200640 1 -- 172.21.15.28:0/3535116324 >> 172.21.15.28:0/3535116324 conn(0x7f2b2c0fc880 msgr2=0x7f2b2c0fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:32.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.692+0000 7f2b2b200640 1 -- 172.21.15.28:0/3535116324 shutdown_connections 2024-04-24T08:16:32.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:32.692+0000 7f2b2b200640 1 -- 172.21.15.28:0/3535116324 wait complete. 2024-04-24T08:16:33.722 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:16:33.722 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:16:33.723 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:16:33.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:33 smithi028 ceph-mon[27786]: pgmap v189: 97 pgs: 97 active+clean; 21 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 53 MiB/s wr, 11.09k op/s 2024-04-24T08:16:33.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:33 smithi117 ceph-mon[32438]: pgmap v189: 97 pgs: 97 active+clean; 21 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 53 MiB/s wr, 11.09k op/s 2024-04-24T08:16:33.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:33 smithi194 ceph-mon[32836]: pgmap v189: 97 pgs: 97 active+clean; 21 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 53 MiB/s wr, 11.09k op/s 2024-04-24T08:16:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.068+0000 7f278d400640 1 -- 172.21.15.28:0/1342916536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2788109520 msgr2=0x7f27881110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.068+0000 7f278d400640 1 --2- 172.21.15.28:0/1342916536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2788109520 0x7f27881110f0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f278001acb0 tx=0x7f278003b8e0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.069+0000 7f278d400640 1 -- 172.21.15.28:0/1342916536 shutdown_connections 2024-04-24T08:16:34.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.069+0000 7f278d400640 1 --2- 172.21.15.28:0/1342916536 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2788104e00 0x7f2788108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.069+0000 7f278d400640 1 --2- 172.21.15.28:0/1342916536 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2788109520 0x7f27881110f0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.069+0000 7f278d400640 1 --2- 172.21.15.28:0/1342916536 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2788104450 0x7f2788104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.069+0000 7f278d400640 1 -- 172.21.15.28:0/1342916536 >> 172.21.15.28:0/1342916536 conn(0x7f27880ffc70 msgr2=0x7f2788102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.070+0000 7f278d400640 1 -- 172.21.15.28:0/1342916536 shutdown_connections 2024-04-24T08:16:34.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.070+0000 7f278d400640 1 -- 172.21.15.28:0/1342916536 wait complete. 2024-04-24T08:16:34.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.073+0000 7f278d400640 1 Processor -- start 2024-04-24T08:16:34.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.074+0000 7f278d400640 1 -- start start 2024-04-24T08:16:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.074+0000 7f278d400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2788104450 0x7f27881110f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.074+0000 7f278d400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2788109520 0x7f2788073910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.074+0000 7f278d400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2788071a10 0x7f2788071e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.074+0000 7f277fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2788109520 0x7f2788073910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.074+0000 7f277fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2788109520 0x7f2788073910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53430/0 (socket says 172.21.15.28:53430) 2024-04-24T08:16:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.075+0000 7f2786800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2788104450 0x7f27881110f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.075+0000 7f2786800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2788104450 0x7f27881110f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:53338/0 (socket says 172.21.15.28:53338) 2024-04-24T08:16:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.075+0000 7f278d400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f278806e3b0 con 0x7f2788071a10 2024-04-24T08:16:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.075+0000 7f277fe00640 1 -- 172.21.15.28:0/2811624573 learned_addr learned my addr 172.21.15.28:0/2811624573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.075+0000 7f278d400640 1 -- 172.21.15.28:0/2811624573 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f278806e230 con 0x7f2788104450 2024-04-24T08:16:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.075+0000 7f278d400640 1 -- 172.21.15.28:0/2811624573 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f278806e530 con 0x7f2788109520 2024-04-24T08:16:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.075+0000 7f2786800640 1 -- 172.21.15.28:0/2811624573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2788109520 msgr2=0x7f2788073910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.075+0000 7f2786800640 1 --2- 172.21.15.28:0/2811624573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2788109520 0x7f2788073910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.075+0000 7f2786800640 1 -- 172.21.15.28:0/2811624573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2788071a10 msgr2=0x7f2788071e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.075+0000 7f2786800640 1 --2- 172.21.15.28:0/2811624573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2788071a10 0x7f2788071e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.076+0000 7f2786800640 1 -- 172.21.15.28:0/2811624573 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27880724d0 con 0x7f2788104450 2024-04-24T08:16:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.076+0000 7f2786800640 1 --2- 172.21.15.28:0/2811624573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2788104450 0x7f27881110f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f277800c970 tx=0x7f277800ce30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:34.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.077+0000 7f277f400640 1 -- 172.21.15.28:0/2811624573 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778007b30 con 0x7f2788104450 2024-04-24T08:16:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.077+0000 7f277f400640 1 -- 172.21.15.28:0/2811624573 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2778007cd0 con 0x7f2788104450 2024-04-24T08:16:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.077+0000 7f277f400640 1 -- 172.21.15.28:0/2811624573 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778007b30 con 0x7f2788104450 2024-04-24T08:16:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.078+0000 7f278d400640 1 -- 172.21.15.28:0/2811624573 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27880727c0 con 0x7f2788104450 2024-04-24T08:16:34.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.078+0000 7f278d400640 1 -- 172.21.15.28:0/2811624573 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27881a3420 con 0x7f2788104450 2024-04-24T08:16:34.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.087+0000 7f2790380740 1 Processor -- start 2024-04-24T08:16:34.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.087+0000 7f277f400640 1 -- 172.21.15.28:0/2811624573 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f277800f450 con 0x7f2788104450 2024-04-24T08:16:34.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.090+0000 7f277f400640 1 --2- 172.21.15.28:0/2811624573 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f27680a3fd0 0x7f27680a6490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.090+0000 7f2790380740 1 -- start start 2024-04-24T08:16:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.091+0000 7f2790380740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64077090 0x563b64077470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.091+0000 7f2790380740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b640779b0 0x563b64081b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.091+0000 7f2790380740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64082080 0x563b64084470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.091+0000 7f2790380740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563b63fdce70 con 0x563b64082080 2024-04-24T08:16:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.091+0000 7f2790380740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563b63f83380 con 0x563b640779b0 2024-04-24T08:16:34.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.091+0000 7f2790380740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563b63fd8e30 con 0x563b64077090 2024-04-24T08:16:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.092+0000 7f2787200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64082080 0x563b64084470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.092+0000 7f2786800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64077090 0x563b64077470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.092+0000 7f2786800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64077090 0x563b64077470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53442/0 (socket says 172.21.15.28:53442) 2024-04-24T08:16:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.092+0000 7f2786800640 1 -- 172.21.15.28:0/2831303940 learned_addr learned my addr 172.21.15.28:0/2831303940 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.092+0000 7f277fe00640 1 --2- 172.21.15.28:0/2831303940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b640779b0 0x563b64081b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.092+0000 7f2787200640 1 --2- 172.21.15.28:0/2811624573 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f27680a3fd0 0x7f27680a6490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.093+0000 7f2786800640 1 -- 172.21.15.28:0/2831303940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b640779b0 msgr2=0x563b64081b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.093+0000 7f2786800640 1 --2- 172.21.15.28:0/2831303940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b640779b0 0x563b64081b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.093+0000 7f2786800640 1 -- 172.21.15.28:0/2831303940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64082080 msgr2=0x563b64084470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:34.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.093+0000 7f2786800640 1 --2- 172.21.15.28:0/2831303940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64082080 0x563b64084470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.093+0000 7f2786800640 1 -- 172.21.15.28:0/2831303940 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b63db77e0 con 0x563b64077090 2024-04-24T08:16:34.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.093+0000 7f277f400640 1 -- 172.21.15.28:0/2811624573 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f277809a430 con 0x7f2788104450 2024-04-24T08:16:34.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.093+0000 7f2786800640 1 --2- 172.21.15.28:0/2831303940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64077090 0x563b64077470 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f27780997e0 tx=0x7f27780a1860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1cf52c72549dcff4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:34.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.094+0000 7f2777400640 1 -- 172.21.15.28:0/2831303940 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778098dc0 con 0x563b64077090 2024-04-24T08:16:34.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.094+0000 7f2777400640 1 -- 172.21.15.28:0/2831303940 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2778015e70 con 0x563b64077090 2024-04-24T08:16:34.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.094+0000 7f2777400640 1 -- 172.21.15.28:0/2831303940 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778098f60 con 0x563b64077090 2024-04-24T08:16:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.095+0000 7f2787200640 1 --2- 172.21.15.28:0/2811624573 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f27680a3fd0 0x7f27680a6490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f278003be50 tx=0x7f278003bde0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.095+0000 7f2790380740 1 -- 172.21.15.28:0/2831303940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64077090 msgr2=0x563b64077470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.095+0000 7f2790380740 1 --2- 172.21.15.28:0/2831303940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64077090 0x563b64077470 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f27780997e0 tx=0x7f27780a1860 comp rx=0 tx=0).stop 2024-04-24T08:16:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.095+0000 7f2790380740 1 -- 172.21.15.28:0/2831303940 shutdown_connections 2024-04-24T08:16:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.095+0000 7f2790380740 1 --2- 172.21.15.28:0/2831303940 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64082080 0x563b64084470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.095+0000 7f2790380740 1 --2- 172.21.15.28:0/2831303940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b640779b0 0x563b64081b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.095+0000 7f2790380740 1 --2- 172.21.15.28:0/2831303940 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64077090 0x563b64077470 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.095+0000 7f2790380740 1 -- 172.21.15.28:0/2831303940 >> 172.21.15.28:0/2831303940 conn(0x563b640733b0 msgr2=0x563b64073790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:34.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.096+0000 7f2790380740 1 -- 172.21.15.28:0/2831303940 shutdown_connections 2024-04-24T08:16:34.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.096+0000 7f2790380740 1 -- 172.21.15.28:0/2831303940 wait complete. 2024-04-24T08:16:34.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.096+0000 7f2790380740 1 -- start start 2024-04-24T08:16:34.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.097+0000 7f2790380740 1 Processor -- start 2024-04-24T08:16:34.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.098+0000 7f2790380740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b64077090 0x563b6407f080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.098+0000 7f2790380740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b640779b0 0x563b6407f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.099+0000 7f2790380740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64082080 0x563b6407fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.099+0000 7f2786800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b64077090 0x563b6407f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.099+0000 7f2786800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b64077090 0x563b6407f080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:53364/0 (socket says 172.21.15.28:53364) 2024-04-24T08:16:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.099+0000 7f277fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b640779b0 0x563b6407f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.099+0000 7f277fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b640779b0 0x563b6407f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53448/0 (socket says 172.21.15.28:53448) 2024-04-24T08:16:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.099+0000 7f2787200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64082080 0x563b6407fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.099+0000 7f2787200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64082080 0x563b6407fb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39604/0 (socket says 172.21.15.28:39604) 2024-04-24T08:16:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.099+0000 7f2786800640 1 -- 172.21.15.28:0/2657065442 learned_addr learned my addr 172.21.15.28:0/2657065442 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.100+0000 7f2790380740 1 -- 172.21.15.28:0/2657065442 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563b64084ec0 con 0x563b64082080 2024-04-24T08:16:34.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.100+0000 7f2790380740 1 -- 172.21.15.28:0/2657065442 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563b63f96660 con 0x563b64077090 2024-04-24T08:16:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.100+0000 7f2790380740 1 -- 172.21.15.28:0/2657065442 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563b64085040 con 0x563b640779b0 2024-04-24T08:16:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.100+0000 7f2790380740 10 client.0 _ll_register_callbacks cb 0x563b63f84aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.101+0000 7f277fe00640 1 -- 172.21.15.28:0/2657065442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b64077090 msgr2=0x563b6407f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.101+0000 7f277fe00640 1 --2- 172.21.15.28:0/2657065442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b64077090 0x563b6407f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.101+0000 7f277fe00640 1 -- 172.21.15.28:0/2657065442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64082080 msgr2=0x563b6407fb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:34.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.101+0000 7f277fe00640 1 --2- 172.21.15.28:0/2657065442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64082080 0x563b6407fb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.101+0000 7f277fe00640 1 -- 172.21.15.28:0/2657065442 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b63e17f70 con 0x563b640779b0 2024-04-24T08:16:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.101+0000 7f2786800640 1 --2- 172.21.15.28:0/2657065442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b64077090 0x563b6407f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:16:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.101+0000 7f2790380740 1 -- 172.21.15.28:0/2657065442 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2778015e70 con 0x563b640779b0 2024-04-24T08:16:34.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.101+0000 7f2787200640 1 --2- 172.21.15.28:0/2657065442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64082080 0x563b6407fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:16:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.104+0000 7f277fe00640 1 --2- 172.21.15.28:0/2657065442 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b640779b0 0x563b6407f5c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f27880734a0 tx=0x7f2770010c80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e404cd0c5e59882 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.104+0000 7f276fe00640 10 client.26314 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:16:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.105+0000 7f276fe00640 1 -- 172.21.15.28:0/2657065442 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2770004300 con 0x563b640779b0 2024-04-24T08:16:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.105+0000 7f276fe00640 1 -- 172.21.15.28:0/2657065442 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f277000f980 con 0x563b640779b0 2024-04-24T08:16:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.105+0000 7f276fe00640 1 -- 172.21.15.28:0/2657065442 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f277000fba0 con 0x563b640779b0 2024-04-24T08:16:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.105+0000 7f2790380740 10 client.26314 fetch_fsmap learned FSMap version 26 2024-04-24T08:16:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.105+0000 7f2790380740 1 -- 172.21.15.28:0/2657065442 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({fsmap=26}) -- 0x563b63caf170 con 0x563b640779b0 2024-04-24T08:16:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.106+0000 7f276fe00640 1 -- 172.21.15.28:0/2657065442 <== mon.1 v2:172.21.15.194:3300/0 4 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f27700035f0 con 0x563b640779b0 2024-04-24T08:16:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.106+0000 7f2790380740 10 client.26314 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:16:34.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.106+0000 7f2790380740 10 client.26314 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:16:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.106+0000 7f2790380740 20 client.26314 populate_metadata read hostname 'smithi028' 2024-04-24T08:16:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.106+0000 7f2790380740 1 --2- 172.21.15.28:0/2657065442 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563b640923c0 0x563b640947b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.106+0000 7f2790380740 4 client.26314 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:16:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.106+0000 7f2790380740 1 -- 172.21.15.28:0/2657065442 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b63fea890 con 0x563b640923c0 2024-04-24T08:16:34.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.108+0000 7f2786800640 1 --2- 172.21.15.28:0/2657065442 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563b640923c0 0x563b640947b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.112+0000 7f276fe00640 10 client.26314 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:16:34.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.112+0000 7f2786800640 1 --2- 172.21.15.28:0/2657065442 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563b640923c0 0x563b640947b0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c6dd68bcc8abbd1 server_cookie=6b4bb70ac28efed6 in_seq=0 out_seq=0 2024-04-24T08:16:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.114+0000 7f276fe00640 1 -- 172.21.15.28:0/2657065442 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563b63fea890 con 0x563b640923c0 2024-04-24T08:16:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.114+0000 7f276fe00640 10 client.26314 handle_command_reply: tid=0 2024-04-24T08:16:34.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.114+0000 7f2790380740 1 client.26314 shutdown 2024-04-24T08:16:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.114+0000 7f2790380740 1 -- 172.21.15.28:0/2657065442 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b640779b0 msgr2=0x563b6407f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.114+0000 7f2790380740 1 --2- 172.21.15.28:0/2657065442 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b640779b0 0x563b6407f5c0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f27880734a0 tx=0x7f2770010c80 comp rx=0 tx=0).stop 2024-04-24T08:16:34.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.114+0000 7f2790380740 1 -- 172.21.15.28:0/2657065442 shutdown_connections 2024-04-24T08:16:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.114+0000 7f2790380740 1 --2- 172.21.15.28:0/2657065442 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563b640923c0 0x563b640947b0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.114+0000 7f2790380740 1 --2- 172.21.15.28:0/2657065442 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64082080 0x563b6407fb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.114+0000 7f2790380740 1 --2- 172.21.15.28:0/2657065442 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b640779b0 0x563b6407f5c0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.114+0000 7f2790380740 1 --2- 172.21.15.28:0/2657065442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b64077090 0x563b6407f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.114+0000 7f2790380740 1 -- 172.21.15.28:0/2657065442 >> 172.21.15.28:0/2657065442 conn(0x563b640733b0 msgr2=0x563b640791b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.114+0000 7f276fe00640 0 client.26314 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:16:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.115+0000 7f2790380740 1 -- 172.21.15.28:0/2657065442 shutdown_connections 2024-04-24T08:16:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.115+0000 7f2790380740 1 -- 172.21.15.28:0/2657065442 wait complete. 2024-04-24T08:16:34.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.115+0000 7f2790380740 20 client.26314 trim_cache size 0 max 16384 2024-04-24T08:16:34.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.152+0000 7f2790380740 1 Processor -- start 2024-04-24T08:16:34.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.152+0000 7f2790380740 1 -- start start 2024-04-24T08:16:34.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.152+0000 7f2790380740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64077090 0x563b6408dd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.152+0000 7f2790380740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b640779b0 0x563b64079900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.152+0000 7f2790380740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64082080 0x563b6409a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.152+0000 7f2790380740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27881ae430 con 0x563b64077090 2024-04-24T08:16:34.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.152+0000 7f2790380740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f27881abd00 con 0x563b640779b0 2024-04-24T08:16:34.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.152+0000 7f2790380740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f27881a9b50 con 0x563b64082080 2024-04-24T08:16:34.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.153+0000 7f2786800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64077090 0x563b6408dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.153+0000 7f2786800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64077090 0x563b6408dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39616/0 (socket says 172.21.15.28:39616) 2024-04-24T08:16:34.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.153+0000 7f2786800640 1 -- 172.21.15.28:0/2345374000 learned_addr learned my addr 172.21.15.28:0/2345374000 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:34.155 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.153+0000 7f2787200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64082080 0x563b6409a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.153+0000 7f2786800640 1 -- 172.21.15.28:0/2345374000 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64082080 msgr2=0x563b6409a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:34.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.153+0000 7f2786800640 1 --2- 172.21.15.28:0/2345374000 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64082080 0x563b6409a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.153+0000 7f2786800640 1 -- 172.21.15.28:0/2345374000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b640779b0 msgr2=0x563b64079900 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:16:34.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.153+0000 7f2786800640 1 --2- 172.21.15.28:0/2345374000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b640779b0 0x563b64079900 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.154+0000 7f2786800640 1 -- 172.21.15.28:0/2345374000 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b64084470 con 0x563b64077090 2024-04-24T08:16:34.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.154+0000 7f2786800640 1 --2- 172.21.15.28:0/2345374000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64077090 0x563b6408dd80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f27780987f0 tx=0x7f27780a0930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=90913c3f1bd2070b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:34.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.157+0000 7f2776a00640 1 -- 172.21.15.28:0/2345374000 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27780b0070 con 0x563b64077090 2024-04-24T08:16:34.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.157+0000 7f2776a00640 1 -- 172.21.15.28:0/2345374000 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27780a5000 con 0x563b64077090 2024-04-24T08:16:34.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.157+0000 7f2776a00640 1 -- 172.21.15.28:0/2345374000 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778099100 con 0x563b64077090 2024-04-24T08:16:34.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.158+0000 7f2790380740 1 -- 172.21.15.28:0/2345374000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64077090 msgr2=0x563b6408dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.158+0000 7f2790380740 1 --2- 172.21.15.28:0/2345374000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64077090 0x563b6408dd80 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f27780987f0 tx=0x7f27780a0930 comp rx=0 tx=0).stop 2024-04-24T08:16:34.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.159+0000 7f2790380740 1 -- 172.21.15.28:0/2345374000 shutdown_connections 2024-04-24T08:16:34.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.159+0000 7f2790380740 1 --2- 172.21.15.28:0/2345374000 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64082080 0x563b6409a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.159+0000 7f2790380740 1 --2- 172.21.15.28:0/2345374000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b640779b0 0x563b64079900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.159+0000 7f2790380740 1 --2- 172.21.15.28:0/2345374000 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64077090 0x563b6408dd80 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.159+0000 7f2790380740 1 -- 172.21.15.28:0/2345374000 >> 172.21.15.28:0/2345374000 conn(0x563b640733b0 msgr2=0x563b6407f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:34.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.159+0000 7f2790380740 1 -- 172.21.15.28:0/2345374000 shutdown_connections 2024-04-24T08:16:34.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.159+0000 7f2790380740 1 -- 172.21.15.28:0/2345374000 wait complete. 2024-04-24T08:16:34.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.160+0000 7f2790380740 1 -- start start 2024-04-24T08:16:34.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.160+0000 7f2790380740 1 Processor -- start 2024-04-24T08:16:34.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.160+0000 7f2790380740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b640779b0 0x563b6408a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.160+0000 7f2790380740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64082080 0x563b64085ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.160+0000 7f2790380740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64086240 0x563b640f7880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.160+0000 7f2790380740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f27881aa1c0 con 0x563b64086240 2024-04-24T08:16:34.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.160+0000 7f2790380740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f27881ab630 con 0x563b640779b0 2024-04-24T08:16:34.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.160+0000 7f2790380740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f27881aa890 con 0x563b64082080 2024-04-24T08:16:34.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.161+0000 7f2790380740 10 client.0 _ll_register_callbacks cb 0x7f277000a830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:34.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.161+0000 7f2786800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b640779b0 0x563b6408a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.161+0000 7f2786800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b640779b0 0x563b6408a7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:53382/0 (socket says 172.21.15.28:53382) 2024-04-24T08:16:34.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.161+0000 7f2786800640 1 -- 172.21.15.28:0/1344744207 learned_addr learned my addr 172.21.15.28:0/1344744207 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:34.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.161+0000 7f277fe00640 1 --2- 172.21.15.28:0/1344744207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64082080 0x563b64085ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.161+0000 7f2787200640 1 --2- 172.21.15.28:0/1344744207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64086240 0x563b640f7880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.161+0000 7f277fe00640 1 -- 172.21.15.28:0/1344744207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b640779b0 msgr2=0x563b6408a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:34.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.161+0000 7f277fe00640 1 --2- 172.21.15.28:0/1344744207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b640779b0 0x563b6408a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.161+0000 7f277fe00640 1 -- 172.21.15.28:0/1344744207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64086240 msgr2=0x563b640f7880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:34.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.161+0000 7f277fe00640 1 --2- 172.21.15.28:0/1344744207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64086240 0x563b640f7880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.161+0000 7f277fe00640 1 -- 172.21.15.28:0/1344744207 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b640869b0 con 0x563b64082080 2024-04-24T08:16:34.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.161+0000 7f2786800640 1 --2- 172.21.15.28:0/1344744207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b640779b0 0x563b6408a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:16:34.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.162+0000 7f277fe00640 1 --2- 172.21.15.28:0/1344744207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64082080 0x563b64085ba0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2770000c00 tx=0x7f27700046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4aff586258340da server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:34.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.162+0000 7f276fe00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:16:34.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.162+0000 7f276fe00640 1 -- 172.21.15.28:0/1344744207 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2770003d50 con 0x563b64082080 2024-04-24T08:16:34.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.162+0000 7f276fe00640 1 -- 172.21.15.28:0/1344744207 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f2770004300 con 0x563b64082080 2024-04-24T08:16:34.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.163+0000 7f2790380740 1 -- 172.21.15.28:0/1344744207 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27780a5000 con 0x563b64082080 2024-04-24T08:16:34.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.163+0000 7f276fe00640 1 -- 172.21.15.28:0/1344744207 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2770024470 con 0x563b64082080 2024-04-24T08:16:34.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.163+0000 7f2790380740 10 client.26320 fetch_fsmap learned FSMap version 26 2024-04-24T08:16:34.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.163+0000 7f2790380740 1 -- 172.21.15.28:0/1344744207 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563b640f8e30 con 0x563b64082080 2024-04-24T08:16:34.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.163+0000 7f276fe00640 1 -- 172.21.15.28:0/1344744207 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27780a5000 con 0x563b64082080 2024-04-24T08:16:34.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.164+0000 7f276fe00640 1 -- 172.21.15.28:0/1344744207 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2770024bd0 con 0x563b64082080 2024-04-24T08:16:34.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.164+0000 7f2790380740 10 client.26320 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:16:34.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.164+0000 7f2790380740 10 client.26320 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:16:34.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.164+0000 7f2790380740 20 client.26320 populate_metadata read hostname 'smithi028' 2024-04-24T08:16:34.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.164+0000 7f2790380740 1 --2- 172.21.15.28:0/1344744207 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563b640f94f0 0x563b640fb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.164+0000 7f2790380740 4 client.26320 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:16:34.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.164+0000 7f2790380740 1 -- 172.21.15.28:0/1344744207 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x563b63fe4700 con 0x563b640f94f0 2024-04-24T08:16:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.165+0000 7f2786800640 1 --2- 172.21.15.28:0/1344744207 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563b640f94f0 0x563b640fb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.166+0000 7f276fe00640 10 client.26320 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:16:34.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.166+0000 7f2786800640 1 --2- 172.21.15.28:0/1344744207 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563b640f94f0 0x563b640fb8e0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=905ea26928790cfb server_cookie=1d88a671f6acd2e6 in_seq=0 out_seq=0 2024-04-24T08:16:34.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.168+0000 7f276fe00640 1 -- 172.21.15.28:0/1344744207 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x563b63fe4700 con 0x563b640f94f0 2024-04-24T08:16:34.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.168+0000 7f276fe00640 10 client.26320 handle_command_reply: tid=0 2024-04-24T08:16:34.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.168+0000 7f2790380740 1 client.26320 shutdown 2024-04-24T08:16:34.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.169+0000 7f2790380740 1 -- 172.21.15.28:0/1344744207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64082080 msgr2=0x563b64085ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.169+0000 7f2790380740 1 --2- 172.21.15.28:0/1344744207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64082080 0x563b64085ba0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f2770000c00 tx=0x7f27700046c0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.170+0000 7f2790380740 1 -- 172.21.15.28:0/1344744207 shutdown_connections 2024-04-24T08:16:34.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.170+0000 7f2790380740 1 --2- 172.21.15.28:0/1344744207 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563b640f94f0 0x563b640fb8e0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.170+0000 7f2790380740 1 --2- 172.21.15.28:0/1344744207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b64086240 0x563b640f7880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.170+0000 7f2790380740 1 --2- 172.21.15.28:0/1344744207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b64082080 0x563b64085ba0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.170+0000 7f2790380740 1 --2- 172.21.15.28:0/1344744207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b640779b0 0x563b6408a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.170+0000 7f2790380740 1 -- 172.21.15.28:0/1344744207 >> 172.21.15.28:0/1344744207 conn(0x563b640733b0 msgr2=0x563b64071c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:34.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.170+0000 7f276fe00640 0 client.26320 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:16:34.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.172+0000 7f2790380740 1 -- 172.21.15.28:0/1344744207 shutdown_connections 2024-04-24T08:16:34.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.173+0000 7f2790380740 1 -- 172.21.15.28:0/1344744207 wait complete. 2024-04-24T08:16:34.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.174+0000 7f2790380740 20 client.26320 trim_cache size 0 max 16384 2024-04-24T08:16:34.175 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:16:34.175 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-24T08:16:34.176 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "65ba2886-addf-4ddd-8309-5bde5690d338", 2024-04-24T08:16:34.176 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-24T08:16:34.176 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:16:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.175+0000 7f278d400640 1 -- 172.21.15.28:0/2811624573 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f27680a3fd0 msgr2=0x7f27680a6490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.175+0000 7f278d400640 1 --2- 172.21.15.28:0/2811624573 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f27680a3fd0 0x7f27680a6490 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f278003be50 tx=0x7f278003bde0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.175+0000 7f278d400640 1 -- 172.21.15.28:0/2811624573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2788104450 msgr2=0x7f27881110f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.175+0000 7f278d400640 1 --2- 172.21.15.28:0/2811624573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2788104450 0x7f27881110f0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f277800c970 tx=0x7f277800ce30 comp rx=0 tx=0).stop 2024-04-24T08:16:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.179+0000 7f278d400640 1 -- 172.21.15.28:0/2811624573 shutdown_connections 2024-04-24T08:16:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.179+0000 7f278d400640 1 --2- 172.21.15.28:0/2811624573 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f27680a3fd0 0x7f27680a6490 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.179+0000 7f278d400640 1 --2- 172.21.15.28:0/2811624573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2788071a10 0x7f2788071e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.179+0000 7f278d400640 1 --2- 172.21.15.28:0/2811624573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2788109520 0x7f2788073910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.179+0000 7f278d400640 1 --2- 172.21.15.28:0/2811624573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2788104450 0x7f27881110f0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.179+0000 7f278d400640 1 -- 172.21.15.28:0/2811624573 >> 172.21.15.28:0/2811624573 conn(0x7f27880ffc70 msgr2=0x7f2788100790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:34.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.179+0000 7f278d400640 1 -- 172.21.15.28:0/2811624573 shutdown_connections 2024-04-24T08:16:34.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.179+0000 7f278d400640 1 -- 172.21.15.28:0/2811624573 wait complete. 2024-04-24T08:16:34.209 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:16:34.209 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:16:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.489+0000 7f34bfe00640 1 -- 172.21.15.28:0/2791180680 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34b809d260 msgr2=0x7f34b80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.490+0000 7f34bfe00640 1 --2- 172.21.15.28:0/2791180680 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34b809d260 0x7f34b80a4e30 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f34b0009900 tx=0x7f34b002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.491 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.490+0000 7f34bfe00640 1 -- 172.21.15.28:0/2791180680 shutdown_connections 2024-04-24T08:16:34.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.493+0000 7f34bfe00640 1 --2- 172.21.15.28:0/2791180680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b8098b40 0x7f34b809cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.493+0000 7f34bfe00640 1 --2- 172.21.15.28:0/2791180680 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34b809d260 0x7f34b80a4e30 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.493+0000 7f34bfe00640 1 --2- 172.21.15.28:0/2791180680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34b8098190 0x7f34b8098570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.493+0000 7f34bfe00640 1 -- 172.21.15.28:0/2791180680 >> 172.21.15.28:0/2791180680 conn(0x7f34b80939f0 msgr2=0x7f34b8095e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:34.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.494+0000 7f34bfe00640 1 -- 172.21.15.28:0/2791180680 shutdown_connections 2024-04-24T08:16:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.495+0000 7f34bfe00640 1 -- 172.21.15.28:0/2791180680 wait complete. 2024-04-24T08:16:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.496+0000 7f34bfe00640 1 Processor -- start 2024-04-24T08:16:34.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.498+0000 7f34bfe00640 1 -- start start 2024-04-24T08:16:34.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.499+0000 7f34bfe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34b8098190 0x7f34b813b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.499+0000 7f34bfe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34b809d260 0x7f34b813bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.499+0000 7f34bfe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b813fcb0 0x7f34b8140130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.499+0000 7f34bfe00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34b800c690 con 0x7f34b813fcb0 2024-04-24T08:16:34.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.499+0000 7f34bfe00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f34b800c510 con 0x7f34b8098190 2024-04-24T08:16:34.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.499+0000 7f34bfe00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f34b800c810 con 0x7f34b809d260 2024-04-24T08:16:34.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.499+0000 7f34be000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34b809d260 0x7f34b813bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.500+0000 7f34be000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34b809d260 0x7f34b813bdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53486/0 (socket says 172.21.15.28:53486) 2024-04-24T08:16:34.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.500+0000 7f34be000640 1 -- 172.21.15.28:0/1560607899 learned_addr learned my addr 172.21.15.28:0/1560607899 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:34.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.500+0000 7f34bf400640 1 --2- 172.21.15.28:0/1560607899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b813fcb0 0x7f34b8140130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.500+0000 7f34bea00640 1 --2- 172.21.15.28:0/1560607899 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34b8098190 0x7f34b813b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.500+0000 7f34bf400640 1 -- 172.21.15.28:0/1560607899 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34b809d260 msgr2=0x7f34b813bdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.500+0000 7f34bf400640 1 --2- 172.21.15.28:0/1560607899 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34b809d260 0x7f34b813bdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.500+0000 7f34bf400640 1 -- 172.21.15.28:0/1560607899 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34b8098190 msgr2=0x7f34b813b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.500+0000 7f34bf400640 1 --2- 172.21.15.28:0/1560607899 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34b8098190 0x7f34b813b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.500+0000 7f34bf400640 1 -- 172.21.15.28:0/1560607899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34b81407a0 con 0x7f34b813fcb0 2024-04-24T08:16:34.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.501+0000 7f34bf400640 1 --2- 172.21.15.28:0/1560607899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b813fcb0 0x7f34b8140130 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f34b002a910 tx=0x7f34b0003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:34.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.502+0000 7f34c4a00640 1 -- 172.21.15.28:0/1560607899 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b003f070 con 0x7f34b813fcb0 2024-04-24T08:16:34.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.502+0000 7f34bfe00640 1 -- 172.21.15.28:0/1560607899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34b8140a30 con 0x7f34b813fcb0 2024-04-24T08:16:34.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.502+0000 7f34bfe00640 1 -- 172.21.15.28:0/1560607899 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34b81481f0 con 0x7f34b813fcb0 2024-04-24T08:16:34.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.502+0000 7f34c4a00640 1 -- 172.21.15.28:0/1560607899 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34b0031c90 con 0x7f34b813fcb0 2024-04-24T08:16:34.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.502+0000 7f34c4a00640 1 -- 172.21.15.28:0/1560607899 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b003f070 con 0x7f34b813fcb0 2024-04-24T08:16:34.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.510+0000 7f34c8de5740 1 Processor -- start 2024-04-24T08:16:34.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.512+0000 7f34c8de5740 1 -- start start 2024-04-24T08:16:34.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.512+0000 7f34c4a00640 1 -- 172.21.15.28:0/1560607899 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f34b0042070 con 0x7f34b813fcb0 2024-04-24T08:16:34.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.514+0000 7f34c4a00640 1 --2- 172.21.15.28:0/1560607899 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f34a40b5030 0x7f34a40b74f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.515+0000 7f34c8de5740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114527a30 0x55d114527e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.515+0000 7f34c8de5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114528350 0x55d1145324e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.515+0000 7f34c8de5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114532a20 0x55d114534e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.515+0000 7f34c8de5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d11448d1f0 con 0x55d114528350 2024-04-24T08:16:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.515+0000 7f34c8de5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d114433350 con 0x55d114532a20 2024-04-24T08:16:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.515+0000 7f34c8de5740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d1144891b0 con 0x55d114527a30 2024-04-24T08:16:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.516+0000 7f34bea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114527a30 0x55d114527e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.516+0000 7f34bea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114527a30 0x55d114527e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53492/0 (socket says 172.21.15.28:53492) 2024-04-24T08:16:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.516+0000 7f34bea00640 1 -- 172.21.15.28:0/3456552562 learned_addr learned my addr 172.21.15.28:0/3456552562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:34.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.516+0000 7f34bf400640 1 --2- 172.21.15.28:0/3456552562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114532a20 0x55d114534e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.517+0000 7f34c4a00640 1 -- 172.21.15.28:0/1560607899 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f34b00f08d0 con 0x7f34b813fcb0 2024-04-24T08:16:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.517+0000 7f34bf400640 1 -- 172.21.15.28:0/3456552562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114527a30 msgr2=0x55d114527e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.517+0000 7f34bf400640 1 --2- 172.21.15.28:0/3456552562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114527a30 0x55d114527e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.517+0000 7f34bf400640 1 -- 172.21.15.28:0/3456552562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114528350 msgr2=0x55d1145324e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:16:34.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.517+0000 7f34bf400640 1 --2- 172.21.15.28:0/3456552562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114528350 0x55d1145324e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.517+0000 7f34bf400640 1 -- 172.21.15.28:0/3456552562 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d114487f50 con 0x55d114532a20 2024-04-24T08:16:34.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.517+0000 7f34be000640 1 --2- 172.21.15.28:0/1560607899 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f34a40b5030 0x7f34a40b74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.517+0000 7f34bf400640 1 --2- 172.21.15.28:0/3456552562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114532a20 0x55d114534e10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f34b00edfa0 tx=0x7f34b002a3f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=988198eee6baea10 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:34.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.518+0000 7f34abe00640 1 -- 172.21.15.28:0/3456552562 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b00bcef0 con 0x55d114532a20 2024-04-24T08:16:34.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.518+0000 7f34abe00640 1 -- 172.21.15.28:0/3456552562 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34b00bc7f0 con 0x55d114532a20 2024-04-24T08:16:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.519+0000 7f34c8de5740 1 -- 172.21.15.28:0/3456552562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114532a20 msgr2=0x55d114534e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.519+0000 7f34c8de5740 1 --2- 172.21.15.28:0/3456552562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114532a20 0x55d114534e10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f34b00edfa0 tx=0x7f34b002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.520+0000 7f34c8de5740 1 -- 172.21.15.28:0/3456552562 shutdown_connections 2024-04-24T08:16:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.520+0000 7f34c8de5740 1 --2- 172.21.15.28:0/3456552562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114528350 0x55d1145324e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.520+0000 7f34c8de5740 1 --2- 172.21.15.28:0/3456552562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114532a20 0x55d114534e10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.520+0000 7f34c8de5740 1 --2- 172.21.15.28:0/3456552562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114527a30 0x55d114527e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.520+0000 7f34c8de5740 1 -- 172.21.15.28:0/3456552562 >> 172.21.15.28:0/3456552562 conn(0x55d114523d50 msgr2=0x55d114524130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.521+0000 7f34be000640 1 --2- 172.21.15.28:0/1560607899 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f34a40b5030 0x7f34a40b74f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x55d114528ff0 tx=0x7f34ac008040 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:34.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.521+0000 7f34c8de5740 1 -- 172.21.15.28:0/3456552562 shutdown_connections 2024-04-24T08:16:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.524+0000 7f34c8de5740 1 -- 172.21.15.28:0/3456552562 wait complete. 2024-04-24T08:16:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.524+0000 7f34c8de5740 1 -- start start 2024-04-24T08:16:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.525+0000 7f34c8de5740 1 Processor -- start 2024-04-24T08:16:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.527+0000 7f34c8de5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114527a30 0x55d11452f870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.527+0000 7f34c8de5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114528350 0x55d11452fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.527+0000 7f34c8de5740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114532a20 0x55d1145302f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.527+0000 7f34c8de5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d1145355d0 con 0x55d114527a30 2024-04-24T08:16:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.527+0000 7f34c8de5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d114446690 con 0x55d114528350 2024-04-24T08:16:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.527+0000 7f34c8de5740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d114535750 con 0x55d114532a20 2024-04-24T08:16:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.527+0000 7f34c8de5740 10 client.0 _ll_register_callbacks cb 0x55d114482650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.527+0000 7f34bf400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114532a20 0x55d1145302f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.527+0000 7f34bf400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114532a20 0x55d1145302f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53504/0 (socket says 172.21.15.28:53504) 2024-04-24T08:16:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.527+0000 7f34bf400640 1 -- 172.21.15.28:0/3357279119 learned_addr learned my addr 172.21.15.28:0/3357279119 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.527+0000 7f34bea00640 1 --2- 172.21.15.28:0/3357279119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114527a30 0x55d11452f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.528+0000 7f34bf400640 1 -- 172.21.15.28:0/3357279119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114528350 msgr2=0x55d11452fdb0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:16:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.528+0000 7f34be000640 1 --2- 172.21.15.28:0/3357279119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114528350 0x55d11452fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.528+0000 7f34bf400640 1 --2- 172.21.15.28:0/3357279119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114528350 0x55d11452fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.528+0000 7f34bf400640 1 -- 172.21.15.28:0/3357279119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114527a30 msgr2=0x55d11452f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.528+0000 7f34bf400640 1 --2- 172.21.15.28:0/3357279119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114527a30 0x55d11452f870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.528+0000 7f34bf400640 1 -- 172.21.15.28:0/3357279119 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d114445920 con 0x55d114532a20 2024-04-24T08:16:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.528+0000 7f34bea00640 1 --2- 172.21.15.28:0/3357279119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114527a30 0x55d11452f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:16:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.529+0000 7f34be000640 1 --2- 172.21.15.28:0/3357279119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114528350 0x55d11452fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:16:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.530+0000 7f34bf400640 1 --2- 172.21.15.28:0/3357279119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114532a20 0x55d1145302f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f34b0006010 tx=0x7f34b00ee4b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7e9182a38beb797b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.530+0000 7f34a8c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:16:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.530+0000 7f34a8c00640 1 -- 172.21.15.28:0/3357279119 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b003b660 con 0x55d114532a20 2024-04-24T08:16:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.531+0000 7f34a8c00640 1 -- 172.21.15.28:0/3357279119 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f34b00c4fd0 con 0x55d114532a20 2024-04-24T08:16:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.531+0000 7f34c8de5740 1 -- 172.21.15.28:0/3357279119 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34b00bc7f0 con 0x55d114532a20 2024-04-24T08:16:34.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.533+0000 7f34a8c00640 1 -- 172.21.15.28:0/3357279119 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f34b00c0ba0 con 0x55d114532a20 2024-04-24T08:16:34.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.535+0000 7f34c8de5740 10 client.26332 fetch_fsmap learned FSMap version 26 2024-04-24T08:16:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.535+0000 7f34c8de5740 1 -- 172.21.15.28:0/3357279119 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d1142697d0 con 0x55d114532a20 2024-04-24T08:16:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.536+0000 7f34a8c00640 1 -- 172.21.15.28:0/3357279119 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34b00bc7f0 con 0x55d114532a20 2024-04-24T08:16:34.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.536+0000 7f34a8c00640 1 -- 172.21.15.28:0/3357279119 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f34b00317b0 con 0x55d114532a20 2024-04-24T08:16:34.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.536+0000 7f34c8de5740 10 client.26332 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:16:34.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.536+0000 7f34c8de5740 10 client.26332 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:16:34.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.536+0000 7f34c8de5740 20 client.26332 populate_metadata read hostname 'smithi028' 2024-04-24T08:16:34.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.536+0000 7f34c8de5740 1 --2- 172.21.15.28:0/3357279119 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55d114542810 0x55d114544c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.536+0000 7f34c8de5740 4 client.26332 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:16:34.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.536+0000 7f34c8de5740 1 -- 172.21.15.28:0/3357279119 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d11449ac10 con 0x55d114542810 2024-04-24T08:16:34.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.538+0000 7f34bea00640 1 --2- 172.21.15.28:0/3357279119 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55d114542810 0x55d114544c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.539+0000 7f34bea00640 1 --2- 172.21.15.28:0/3357279119 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55d114542810 0x55d114544c30 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b96c81e81ffcf0b server_cookie=5ad4632170202ff3 in_seq=0 out_seq=0 2024-04-24T08:16:34.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.539+0000 7f34a8c00640 10 client.26332 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:16:34.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.542+0000 7f34a8c00640 1 -- 172.21.15.28:0/3357279119 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d11449ac10 con 0x55d114542810 2024-04-24T08:16:34.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.542+0000 7f34a8c00640 10 client.26332 handle_command_reply: tid=0 2024-04-24T08:16:34.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.542+0000 7f34c8de5740 1 client.26332 shutdown 2024-04-24T08:16:34.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.542+0000 7f34c8de5740 1 -- 172.21.15.28:0/3357279119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114532a20 msgr2=0x55d1145302f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.542+0000 7f34c8de5740 1 --2- 172.21.15.28:0/3357279119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114532a20 0x55d1145302f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f34b0006010 tx=0x7f34b00ee4b0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.543+0000 7f34c8de5740 1 -- 172.21.15.28:0/3357279119 shutdown_connections 2024-04-24T08:16:34.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.543+0000 7f34c8de5740 1 --2- 172.21.15.28:0/3357279119 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55d114542810 0x55d114544c30 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.543+0000 7f34c8de5740 1 --2- 172.21.15.28:0/3357279119 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114532a20 0x55d1145302f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.543+0000 7f34c8de5740 1 --2- 172.21.15.28:0/3357279119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114528350 0x55d11452fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.543+0000 7f34c8de5740 1 --2- 172.21.15.28:0/3357279119 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114527a30 0x55d11452f870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.543+0000 7f34c8de5740 1 -- 172.21.15.28:0/3357279119 >> 172.21.15.28:0/3357279119 conn(0x55d114523d50 msgr2=0x55d114522fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:34.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.543+0000 7f34a8c00640 0 client.26332 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:16:34.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.543+0000 7f34c8de5740 1 -- 172.21.15.28:0/3357279119 shutdown_connections 2024-04-24T08:16:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.545+0000 7f34c8de5740 1 -- 172.21.15.28:0/3357279119 wait complete. 2024-04-24T08:16:34.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.546+0000 7f34c8de5740 20 client.26332 trim_cache size 0 max 16384 2024-04-24T08:16:34.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.588+0000 7f34c8de5740 1 Processor -- start 2024-04-24T08:16:34.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.589+0000 7f34c8de5740 1 -- start start 2024-04-24T08:16:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.589+0000 7f34c8de5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114527a30 0x55d11453e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.589+0000 7f34c8de5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114528350 0x55d114523750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.589+0000 7f34c8de5740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114532a20 0x55d11454a4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.589+0000 7f34c8de5740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34a0011220 con 0x55d114528350 2024-04-24T08:16:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.589+0000 7f34c8de5740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f34a000eaf0 con 0x55d114527a30 2024-04-24T08:16:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.589+0000 7f34c8de5740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f34a000c900 con 0x55d114532a20 2024-04-24T08:16:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.590+0000 7f34bf400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114532a20 0x55d11454a4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.590+0000 7f34bf400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114532a20 0x55d11454a4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53520/0 (socket says 172.21.15.28:53520) 2024-04-24T08:16:34.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.590+0000 7f34bf400640 1 -- 172.21.15.28:0/2884904698 learned_addr learned my addr 172.21.15.28:0/2884904698 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.590+0000 7f34bea00640 1 --2- 172.21.15.28:0/2884904698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114527a30 0x55d11453e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.590+0000 7f34be000640 1 --2- 172.21.15.28:0/2884904698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114528350 0x55d114523750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.593+0000 7f34bf400640 1 -- 172.21.15.28:0/2884904698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114527a30 msgr2=0x55d11453e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.593+0000 7f34bf400640 1 --2- 172.21.15.28:0/2884904698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114527a30 0x55d11453e3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.593+0000 7f34bf400640 1 -- 172.21.15.28:0/2884904698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114528350 msgr2=0x55d114523750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:34.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.593+0000 7f34bf400640 1 --2- 172.21.15.28:0/2884904698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114528350 0x55d114523750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.593+0000 7f34bf400640 1 -- 172.21.15.28:0/2884904698 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d11419f3f0 con 0x55d114532a20 2024-04-24T08:16:34.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.593+0000 7f34bea00640 1 --2- 172.21.15.28:0/2884904698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114527a30 0x55d11453e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:16:34.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.593+0000 7f34be000640 1 --2- 172.21.15.28:0/2884904698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114528350 0x55d114523750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:16:34.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.594+0000 7f34bf400640 1 --2- 172.21.15.28:0/2884904698 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114532a20 0x55d11454a4c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f34b00bc780 tx=0x7f34b00bad00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2c8d76cf44b0f5d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:34.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.594+0000 7f34ab400640 1 -- 172.21.15.28:0/2884904698 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b00c0ba0 con 0x55d114532a20 2024-04-24T08:16:34.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.594+0000 7f34ab400640 1 -- 172.21.15.28:0/2884904698 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34b00c5460 con 0x55d114532a20 2024-04-24T08:16:34.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.594+0000 7f34ab400640 1 -- 172.21.15.28:0/2884904698 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34b0004c90 con 0x55d114532a20 2024-04-24T08:16:34.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.595+0000 7f34c8de5740 1 -- 172.21.15.28:0/2884904698 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114532a20 msgr2=0x55d11454a4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.596+0000 7f34c8de5740 1 --2- 172.21.15.28:0/2884904698 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114532a20 0x55d11454a4c0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f34b00bc780 tx=0x7f34b00bad00 comp rx=0 tx=0).stop 2024-04-24T08:16:34.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.597+0000 7f34c8de5740 1 -- 172.21.15.28:0/2884904698 shutdown_connections 2024-04-24T08:16:34.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.597+0000 7f34c8de5740 1 --2- 172.21.15.28:0/2884904698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114528350 0x55d114523750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.597+0000 7f34c8de5740 1 --2- 172.21.15.28:0/2884904698 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114532a20 0x55d11454a4c0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.597+0000 7f34c8de5740 1 --2- 172.21.15.28:0/2884904698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114527a30 0x55d11453e3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.597+0000 7f34c8de5740 1 -- 172.21.15.28:0/2884904698 >> 172.21.15.28:0/2884904698 conn(0x55d114523d50 msgr2=0x55d11452f830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:34.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.599+0000 7f34c8de5740 1 -- 172.21.15.28:0/2884904698 shutdown_connections 2024-04-24T08:16:34.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.599+0000 7f34c8de5740 1 -- 172.21.15.28:0/2884904698 wait complete. 2024-04-24T08:16:34.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.600+0000 7f34c8de5740 1 -- start start 2024-04-24T08:16:34.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.600+0000 7f34c8de5740 1 Processor -- start 2024-04-24T08:16:34.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.601+0000 7f34c8de5740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114527a30 0x55d11453a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.601+0000 7f34c8de5740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114528350 0x55d11453ae30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.602+0000 7f34bea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114527a30 0x55d11453a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.602+0000 7f34bea00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114527a30 0x55d11453a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:39670/0 (socket says 172.21.15.28:39670) 2024-04-24T08:16:34.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.602+0000 7f34be000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114528350 0x55d11453ae30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.602+0000 7f34be000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114528350 0x55d11453ae30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:53526/0 (socket says 172.21.15.28:53526) 2024-04-24T08:16:34.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.602+0000 7f34c8de5740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114532a20 0x55d11452bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.602+0000 7f34be000640 1 -- 172.21.15.28:0/1170416771 learned_addr learned my addr 172.21.15.28:0/1170416771 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:16:34.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.602+0000 7f34c8de5740 1 -- 172.21.15.28:0/1170416771 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f34a000cfb0 con 0x55d114527a30 2024-04-24T08:16:34.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.602+0000 7f34c8de5740 1 -- 172.21.15.28:0/1170416771 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f34a000e420 con 0x55d114532a20 2024-04-24T08:16:34.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.602+0000 7f34c8de5740 1 -- 172.21.15.28:0/1170416771 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f34a000d680 con 0x55d114528350 2024-04-24T08:16:34.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.602+0000 7f34c8de5740 10 client.0 _ll_register_callbacks cb 0x7f34b00bc5c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:16:34.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.603+0000 7f34bf400640 1 --2- 172.21.15.28:0/1170416771 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114532a20 0x55d11452bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.603+0000 7f34be000640 1 -- 172.21.15.28:0/1170416771 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114532a20 msgr2=0x55d11452bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:34.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.603+0000 7f34be000640 1 --2- 172.21.15.28:0/1170416771 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114532a20 0x55d11452bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.603+0000 7f34be000640 1 -- 172.21.15.28:0/1170416771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114527a30 msgr2=0x55d11453a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:16:34.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.603+0000 7f34be000640 1 --2- 172.21.15.28:0/1170416771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114527a30 0x55d11453a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.604+0000 7f34be000640 1 -- 172.21.15.28:0/1170416771 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d11415f0f0 con 0x55d114528350 2024-04-24T08:16:34.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.604+0000 7f34bf400640 1 --2- 172.21.15.28:0/1170416771 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114532a20 0x55d11452bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:16:34.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.604+0000 7f34be000640 1 --2- 172.21.15.28:0/1170416771 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114528350 0x55d11453ae30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f34ac006fd0 tx=0x7f34ac0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c0858cbdfda2fd3e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:16:34.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.604+0000 7f34a8c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:16:34.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.605+0000 7f34a8c00640 1 -- 172.21.15.28:0/1170416771 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34ac024020 con 0x55d114528350 2024-04-24T08:16:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.605+0000 7f34a8c00640 1 -- 172.21.15.28:0/1170416771 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f34ac00f040 con 0x55d114528350 2024-04-24T08:16:34.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.605+0000 7f34c8de5740 1 -- 172.21.15.28:0/1170416771 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34b00c5460 con 0x55d114528350 2024-04-24T08:16:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.608+0000 7f34a8c00640 1 -- 172.21.15.28:0/1170416771 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f34ac00de70 con 0x55d114528350 2024-04-24T08:16:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.608+0000 7f34c8de5740 10 client.26338 fetch_fsmap learned FSMap version 26 2024-04-24T08:16:34.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.608+0000 7f34c8de5740 1 -- 172.21.15.28:0/1170416771 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d114452810 con 0x55d114528350 2024-04-24T08:16:34.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.610+0000 7f34a8c00640 1 -- 172.21.15.28:0/1170416771 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34b00c5460 con 0x55d114528350 2024-04-24T08:16:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.610+0000 7f34a8c00640 1 -- 172.21.15.28:0/1170416771 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f34ac031020 con 0x55d114528350 2024-04-24T08:16:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.610+0000 7f34c8de5740 10 client.26338 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:16:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.610+0000 7f34c8de5740 10 client.26338 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:16:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.610+0000 7f34c8de5740 20 client.26338 populate_metadata read hostname 'smithi028' 2024-04-24T08:16:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.610+0000 7f34c8de5740 1 --2- 172.21.15.28:0/1170416771 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55d1145a8c80 0x55d1145ab0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:16:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.610+0000 7f34c8de5740 4 client.26338 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:16:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.610+0000 7f34c8de5740 1 -- 172.21.15.28:0/1170416771 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55d114494a80 con 0x55d1145a8c80 2024-04-24T08:16:34.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.611+0000 7f34bea00640 1 --2- 172.21.15.28:0/1170416771 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55d1145a8c80 0x55d1145ab0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:16:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.612+0000 7f34a8c00640 10 client.26338 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:16:34.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.612+0000 7f34bea00640 1 --2- 172.21.15.28:0/1170416771 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55d1145a8c80 0x55d1145ab0a0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b95c3fe5c96a37c server_cookie=23a603bec1e8386e in_seq=0 out_seq=0 2024-04-24T08:16:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.616+0000 7f34a8c00640 1 -- 172.21.15.28:0/1170416771 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+267 (crc 0 0 0) 0x55d114494a80 con 0x55d1145a8c80 2024-04-24T08:16:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.616+0000 7f34a8c00640 10 client.26338 handle_command_reply: tid=0 2024-04-24T08:16:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.616+0000 7f34c8de5740 1 client.26338 shutdown 2024-04-24T08:16:34.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.617+0000 7f34c8de5740 1 -- 172.21.15.28:0/1170416771 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114528350 msgr2=0x55d11453ae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.617+0000 7f34c8de5740 1 --2- 172.21.15.28:0/1170416771 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114528350 0x55d11453ae30 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f34ac006fd0 tx=0x7f34ac0046c0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.617+0000 7f34c8de5740 1 -- 172.21.15.28:0/1170416771 shutdown_connections 2024-04-24T08:16:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.617+0000 7f34c8de5740 1 --2- 172.21.15.28:0/1170416771 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55d1145a8c80 0x55d1145ab0a0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.617+0000 7f34c8de5740 1 --2- 172.21.15.28:0/1170416771 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d114532a20 0x55d11452bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.617+0000 7f34c8de5740 1 --2- 172.21.15.28:0/1170416771 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d114528350 0x55d11453ae30 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.617+0000 7f34c8de5740 1 --2- 172.21.15.28:0/1170416771 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d114527a30 0x55d11453a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.617+0000 7f34c8de5740 1 -- 172.21.15.28:0/1170416771 >> 172.21.15.28:0/1170416771 conn(0x55d114523d50 msgr2=0x55d1145300a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:34.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.618+0000 7f34a8c00640 0 client.26338 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:16:34.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.618+0000 7f34c8de5740 1 -- 172.21.15.28:0/1170416771 shutdown_connections 2024-04-24T08:16:34.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.618+0000 7f34c8de5740 1 -- 172.21.15.28:0/1170416771 wait complete. 2024-04-24T08:16:34.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.618+0000 7f34c8de5740 20 client.26338 trim_cache size 0 max 16384 2024-04-24T08:16:34.621 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:16:34.621 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (169 inodes in the stack)", 2024-04-24T08:16:34.622 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:16:34.622 INFO:teuthology.orchestra.run.smithi028.stdout: "65ba2886-addf-4ddd-8309-5bde5690d338": { 2024-04-24T08:16:34.622 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:16:34.622 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "65ba2886-addf-4ddd-8309-5bde5690d338", 2024-04-24T08:16:34.622 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:16:34.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:16:34.622 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:16:34.622 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:16:34.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.621+0000 7f34bfe00640 1 -- 172.21.15.28:0/1560607899 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f34a40b5030 msgr2=0x7f34a40b74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.621+0000 7f34bfe00640 1 --2- 172.21.15.28:0/1560607899 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f34a40b5030 0x7f34a40b74f0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x55d114528ff0 tx=0x7f34ac008040 comp rx=0 tx=0).stop 2024-04-24T08:16:34.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.621+0000 7f34bfe00640 1 -- 172.21.15.28:0/1560607899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b813fcb0 msgr2=0x7f34b8140130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:16:34.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.621+0000 7f34bfe00640 1 --2- 172.21.15.28:0/1560607899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b813fcb0 0x7f34b8140130 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f34b002a910 tx=0x7f34b0003b90 comp rx=0 tx=0).stop 2024-04-24T08:16:34.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.621+0000 7f34bfe00640 1 -- 172.21.15.28:0/1560607899 shutdown_connections 2024-04-24T08:16:34.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.621+0000 7f34bfe00640 1 --2- 172.21.15.28:0/1560607899 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f34a40b5030 0x7f34a40b74f0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.621+0000 7f34bfe00640 1 --2- 172.21.15.28:0/1560607899 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f34b813fcb0 0x7f34b8140130 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.621+0000 7f34bfe00640 1 --2- 172.21.15.28:0/1560607899 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f34b809d260 0x7f34b813bdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.621+0000 7f34bfe00640 1 --2- 172.21.15.28:0/1560607899 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f34b8098190 0x7f34b813b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:16:34.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.621+0000 7f34bfe00640 1 -- 172.21.15.28:0/1560607899 >> 172.21.15.28:0/1560607899 conn(0x7f34b80939f0 msgr2=0x7f34b809bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:16:34.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.622+0000 7f34bfe00640 1 -- 172.21.15.28:0/1560607899 shutdown_connections 2024-04-24T08:16:34.624 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:16:34.622+0000 7f34bfe00640 1 -- 172.21.15.28:0/1560607899 wait complete. 2024-04-24T08:16:34.654 INFO:tasks.cephfs.filesystem:scrub status for tag:65ba2886-addf-4ddd-8309-5bde5690d338 - {'path': '/', 'tag': '65ba2886-addf-4ddd-8309-5bde5690d338', 'options': 'recursive,force'} 2024-04-24T08:16:35.795 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:35 smithi028 ceph-mon[27786]: scrub queued for path: / 2024-04-24T08:16:35.796 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:35 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:35.796 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:35 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:16:35.796 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:35 smithi028 ceph-mon[27786]: pgmap v190: 97 pgs: 97 active+clean; 21 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 51 MiB/s wr, 10.84k op/s 2024-04-24T08:16:35.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:35 smithi117 ceph-mon[32438]: scrub queued for path: / 2024-04-24T08:16:35.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:35 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:35.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:35 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:16:35.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:35 smithi117 ceph-mon[32438]: pgmap v190: 97 pgs: 97 active+clean; 21 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 51 MiB/s wr, 10.84k op/s 2024-04-24T08:16:35.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:35 smithi194 ceph-mon[32836]: scrub queued for path: / 2024-04-24T08:16:35.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:35 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:35.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:35 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:16:35.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:35 smithi194 ceph-mon[32836]: pgmap v190: 97 pgs: 97 active+clean; 21 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 51 MiB/s wr, 10.84k op/s 2024-04-24T08:16:37.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:37 smithi117 ceph-mon[32438]: pgmap v191: 97 pgs: 97 active+clean; 21 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 MiB/s wr, 9.99k op/s 2024-04-24T08:16:37.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:37 smithi194 ceph-mon[32836]: pgmap v191: 97 pgs: 97 active+clean; 21 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 MiB/s wr, 9.99k op/s 2024-04-24T08:16:38.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:37 smithi028 ceph-mon[27786]: pgmap v191: 97 pgs: 97 active+clean; 21 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 47 MiB/s wr, 9.99k op/s 2024-04-24T08:16:39.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:39 smithi117 ceph-mon[32438]: pgmap v192: 97 pgs: 97 active+clean; 22 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 49 MiB/s wr, 10.31k op/s 2024-04-24T08:16:39.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:39 smithi194 ceph-mon[32836]: pgmap v192: 97 pgs: 97 active+clean; 22 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 49 MiB/s wr, 10.31k op/s 2024-04-24T08:16:40.043 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:39 smithi028 ceph-mon[27786]: pgmap v192: 97 pgs: 97 active+clean; 22 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 70 KiB/s rd, 49 MiB/s wr, 10.31k op/s 2024-04-24T08:16:41.847 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:41 smithi028 ceph-mon[27786]: pgmap v193: 97 pgs: 97 active+clean; 22 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 44 MiB/s wr, 9.32k op/s 2024-04-24T08:16:41.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:41 smithi117 ceph-mon[32438]: pgmap v193: 97 pgs: 97 active+clean; 22 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 44 MiB/s wr, 9.32k op/s 2024-04-24T08:16:41.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:41 smithi194 ceph-mon[32836]: pgmap v193: 97 pgs: 97 active+clean; 22 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 44 MiB/s wr, 9.32k op/s 2024-04-24T08:16:43.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:43 smithi117 ceph-mon[32438]: pgmap v194: 97 pgs: 97 active+clean; 22 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 43 MiB/s wr, 9.15k op/s 2024-04-24T08:16:43.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:43 smithi194 ceph-mon[32836]: pgmap v194: 97 pgs: 97 active+clean; 22 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 43 MiB/s wr, 9.15k op/s 2024-04-24T08:16:44.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:43 smithi028 ceph-mon[27786]: pgmap v194: 97 pgs: 97 active+clean; 22 GiB data, 43 GiB used, 1.0 TiB / 1.0 TiB avail; 49 KiB/s rd, 43 MiB/s wr, 9.15k op/s 2024-04-24T08:16:45.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:45 smithi117 ceph-mon[32438]: pgmap v195: 97 pgs: 97 active+clean; 22 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 42 MiB/s wr, 8.93k op/s 2024-04-24T08:16:45.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:45 smithi194 ceph-mon[32836]: pgmap v195: 97 pgs: 97 active+clean; 22 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 42 MiB/s wr, 8.93k op/s 2024-04-24T08:16:46.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:45 smithi028 ceph-mon[27786]: pgmap v195: 97 pgs: 97 active+clean; 22 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 42 MiB/s wr, 8.93k op/s 2024-04-24T08:16:46.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:46 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:16:46.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:46 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:46.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:46 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:16:46.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:46 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:46.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:46 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:16:46.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:46 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:47.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:46 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:16:47.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:46 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:47.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:46 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:16:47.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:46 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:47.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:46 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:16:47.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:46 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:47.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:46 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:16:47.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:46 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:47.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:46 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:16:47.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:46 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:47.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:46 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:16:47.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:46 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:47.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:47 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:47.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:47 smithi194 ceph-mon[32836]: pgmap v196: 97 pgs: 97 active+clean; 22 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 39 MiB/s wr, 8.38k op/s 2024-04-24T08:16:48.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:47 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:48.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:47 smithi028 ceph-mon[27786]: pgmap v196: 97 pgs: 97 active+clean; 22 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 39 MiB/s wr, 8.38k op/s 2024-04-24T08:16:48.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:47 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:16:48.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:47 smithi117 ceph-mon[32438]: pgmap v196: 97 pgs: 97 active+clean; 22 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 50 KiB/s rd, 39 MiB/s wr, 8.38k op/s 2024-04-24T08:16:49.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:49 smithi194 ceph-mon[32836]: pgmap v197: 97 pgs: 97 active+clean; 22 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 38 MiB/s wr, 8.16k op/s 2024-04-24T08:16:50.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:49 smithi028 ceph-mon[27786]: pgmap v197: 97 pgs: 97 active+clean; 22 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 38 MiB/s wr, 8.16k op/s 2024-04-24T08:16:50.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:49 smithi117 ceph-mon[32438]: pgmap v197: 97 pgs: 97 active+clean; 22 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 38 MiB/s wr, 8.16k op/s 2024-04-24T08:16:50.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:50 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:16:51.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:50 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:16:51.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:50 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:16:52.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:51 smithi028 ceph-mon[27786]: pgmap v198: 97 pgs: 97 active+clean; 22 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 30 MiB/s wr, 6.42k op/s 2024-04-24T08:16:52.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:51 smithi117 ceph-mon[32438]: pgmap v198: 97 pgs: 97 active+clean; 22 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 30 MiB/s wr, 6.42k op/s 2024-04-24T08:16:52.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:51 smithi194 ceph-mon[32836]: pgmap v198: 97 pgs: 97 active+clean; 22 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 30 MiB/s wr, 6.42k op/s 2024-04-24T08:16:54.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:53 smithi028 ceph-mon[27786]: pgmap v199: 97 pgs: 97 active+clean; 22 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 34 MiB/s wr, 7.17k op/s 2024-04-24T08:16:54.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:53 smithi117 ceph-mon[32438]: pgmap v199: 97 pgs: 97 active+clean; 22 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 34 MiB/s wr, 7.17k op/s 2024-04-24T08:16:54.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:53 smithi194 ceph-mon[32836]: pgmap v199: 97 pgs: 97 active+clean; 22 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 82 KiB/s rd, 34 MiB/s wr, 7.17k op/s 2024-04-24T08:16:55.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:54 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:16:55.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:54 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:16:55.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:54 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:16:55.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:54 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:16:55.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:54 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:16:55.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:54 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:16:56.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:55 smithi028 ceph-mon[27786]: pgmap v200: 97 pgs: 97 active+clean; 22 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 42 MiB/s wr, 8.97k op/s 2024-04-24T08:16:56.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:55 smithi117 ceph-mon[32438]: pgmap v200: 97 pgs: 97 active+clean; 22 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 42 MiB/s wr, 8.97k op/s 2024-04-24T08:16:56.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:55 smithi194 ceph-mon[32836]: pgmap v200: 97 pgs: 97 active+clean; 22 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 111 KiB/s rd, 42 MiB/s wr, 8.97k op/s 2024-04-24T08:16:58.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:57 smithi028 ceph-mon[27786]: pgmap v201: 97 pgs: 97 active+clean; 22 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 32 MiB/s wr, 6.84k op/s 2024-04-24T08:16:58.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:57 smithi117 ceph-mon[32438]: pgmap v201: 97 pgs: 97 active+clean; 22 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 32 MiB/s wr, 6.84k op/s 2024-04-24T08:16:58.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:57 smithi194 ceph-mon[32836]: pgmap v201: 97 pgs: 97 active+clean; 22 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 79 KiB/s rd, 32 MiB/s wr, 6.84k op/s 2024-04-24T08:17:00.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:16:59 smithi028 ceph-mon[27786]: pgmap v202: 97 pgs: 97 active+clean; 22 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 36 MiB/s wr, 7.75k op/s 2024-04-24T08:17:00.127 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:16:59 smithi117 ceph-mon[32438]: pgmap v202: 97 pgs: 97 active+clean; 22 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 36 MiB/s wr, 7.75k op/s 2024-04-24T08:17:00.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:16:59 smithi194 ceph-mon[32836]: pgmap v202: 97 pgs: 97 active+clean; 22 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 80 KiB/s rd, 36 MiB/s wr, 7.75k op/s 2024-04-24T08:17:02.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:01 smithi028 ceph-mon[27786]: pgmap v203: 97 pgs: 97 active+clean; 22 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 44 MiB/s wr, 9.39k op/s 2024-04-24T08:17:02.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:01 smithi117 ceph-mon[32438]: pgmap v203: 97 pgs: 97 active+clean; 22 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 44 MiB/s wr, 9.39k op/s 2024-04-24T08:17:02.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:01 smithi194 ceph-mon[32836]: pgmap v203: 97 pgs: 97 active+clean; 22 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 44 MiB/s wr, 9.39k op/s 2024-04-24T08:17:02.746 INFO:tasks.workunit.client.0.smithi028.stdout:Wrote -1 instead of 4096 bytes. 2024-04-24T08:17:02.747 INFO:tasks.workunit.client.0.smithi028.stdout:Probably out of disk space 2024-04-24T08:17:02.747 INFO:tasks.workunit.client.0.smithi028.stderr:write: Resource temporarily unavailable 2024-04-24T08:17:04.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:03 smithi028 ceph-mon[27786]: pgmap v204: 97 pgs: 97 active+clean; 22 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 40 MiB/s wr, 8.51k op/s 2024-04-24T08:17:04.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:03 smithi117 ceph-mon[32438]: pgmap v204: 97 pgs: 97 active+clean; 22 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 40 MiB/s wr, 8.51k op/s 2024-04-24T08:17:04.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:03 smithi194 ceph-mon[32836]: pgmap v204: 97 pgs: 97 active+clean; 22 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 77 KiB/s rd, 40 MiB/s wr, 8.51k op/s 2024-04-24T08:17:04.233 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:17:04.233 INFO:tasks.workunit:Stopping ['suites/ffsb.sh'] on client.0... 2024-04-24T08:17:04.233 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-04-24T08:17:04.655 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:17:04.655 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:17:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.981+0000 7f8e62200640 1 -- 172.21.15.28:0/3312660504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c10e760 msgr2=0x7f8e5c110b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:04.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.981+0000 7f8e62200640 1 --2- 172.21.15.28:0/3312660504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c10e760 0x7f8e5c110b50 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c009900 tx=0x7f8e4c02a4b0 comp rx=0 tx=0).stop 2024-04-24T08:17:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.982+0000 7f8e62200640 1 -- 172.21.15.28:0/3312660504 shutdown_connections 2024-04-24T08:17:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.982+0000 7f8e62200640 1 --2- 172.21.15.28:0/3312660504 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c10e760 0x7f8e5c110b50 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.982+0000 7f8e62200640 1 --2- 172.21.15.28:0/3312660504 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e5c074cd0 0x7f8e5c10e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.982+0000 7f8e62200640 1 --2- 172.21.15.28:0/3312660504 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e5c0743b0 0x7f8e5c074790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.982+0000 7f8e62200640 1 -- 172.21.15.28:0/3312660504 >> 172.21.15.28:0/3312660504 conn(0x7f8e5c0fd9d0 msgr2=0x7f8e5c0ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:04.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.982+0000 7f8e62200640 1 -- 172.21.15.28:0/3312660504 shutdown_connections 2024-04-24T08:17:04.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.983+0000 7f8e62200640 1 -- 172.21.15.28:0/3312660504 wait complete. 2024-04-24T08:17:04.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.984+0000 7f8e62200640 1 Processor -- start 2024-04-24T08:17:04.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.985+0000 7f8e62200640 1 -- start start 2024-04-24T08:17:04.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.985+0000 7f8e62200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e5c0743b0 0x7f8e5c073730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:04.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.985+0000 7f8e62200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c074cd0 0x7f8e5c073c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:04.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.985+0000 7f8e62200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e5c10e760 0x7f8e5c06c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:04.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.985+0000 7f8e62200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c114530 con 0x7f8e5c074cd0 2024-04-24T08:17:04.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.985+0000 7f8e62200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c1143b0 con 0x7f8e5c0743b0 2024-04-24T08:17:04.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.985+0000 7f8e62200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c1146b0 con 0x7f8e5c10e760 2024-04-24T08:17:04.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.986+0000 7f8e5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c074cd0 0x7f8e5c073c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:04.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.986+0000 7f8e60e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e5c0743b0 0x7f8e5c073730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:04.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.986+0000 7f8e5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c074cd0 0x7f8e5c073c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52598/0 (socket says 172.21.15.28:52598) 2024-04-24T08:17:04.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.986+0000 7f8e5be00640 1 -- 172.21.15.28:0/388405047 learned_addr learned my addr 172.21.15.28:0/388405047 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:04.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.987+0000 7f8e60e00640 1 -- 172.21.15.28:0/388405047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e5c10e760 msgr2=0x7f8e5c06c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:04.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.987+0000 7f8e60e00640 1 --2- 172.21.15.28:0/388405047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e5c10e760 0x7f8e5c06c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:04.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.987+0000 7f8e60e00640 1 -- 172.21.15.28:0/388405047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c074cd0 msgr2=0x7f8e5c073c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:04.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.987+0000 7f8e60e00640 1 --2- 172.21.15.28:0/388405047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c074cd0 0x7f8e5c073c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:04.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.987+0000 7f8e60e00640 1 -- 172.21.15.28:0/388405047 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e5c06d250 con 0x7f8e5c0743b0 2024-04-24T08:17:04.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.987+0000 7f8e5be00640 1 --2- 172.21.15.28:0/388405047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c074cd0 0x7f8e5c073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:17:04.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.987+0000 7f8e60e00640 1 --2- 172.21.15.28:0/388405047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e5c0743b0 0x7f8e5c073730 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8e5400eb00 tx=0x7f8e5400c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:04.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.988+0000 7f8e59600640 1 -- 172.21.15.28:0/388405047 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e54010430 con 0x7f8e5c0743b0 2024-04-24T08:17:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.988+0000 7f8e59600640 1 -- 172.21.15.28:0/388405047 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e54010ab0 con 0x7f8e5c0743b0 2024-04-24T08:17:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.988+0000 7f8e59600640 1 -- 172.21.15.28:0/388405047 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e54010430 con 0x7f8e5c0743b0 2024-04-24T08:17:04.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.989+0000 7f8e62200640 1 -- 172.21.15.28:0/388405047 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e5c06d540 con 0x7f8e5c0743b0 2024-04-24T08:17:04.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:04.990+0000 7f8e62200640 1 -- 172.21.15.28:0/388405047 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e5c071430 con 0x7f8e5c0743b0 2024-04-24T08:17:05.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.007+0000 7f8e667e8740 1 Processor -- start 2024-04-24T08:17:05.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.008+0000 7f8e667e8740 1 -- start start 2024-04-24T08:17:05.009 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.008+0000 7f8e59600640 1 -- 172.21.15.28:0/388405047 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f8e540105d0 con 0x7f8e5c0743b0 2024-04-24T08:17:05.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.010+0000 7f8e667e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43be240 0x5608d43be620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.010+0000 7f8e667e8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43beb60 0x5608d43c8cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.010+0000 7f8e667e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43c9230 0x5608d43cb620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.010+0000 7f8e667e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5608d431b1f0 con 0x5608d43c9230 2024-04-24T08:17:05.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.010+0000 7f8e667e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5608d42c1350 con 0x5608d43be240 2024-04-24T08:17:05.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.010+0000 7f8e667e8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5608d43171b0 con 0x5608d43beb60 2024-04-24T08:17:05.012 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.011+0000 7f8e5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43c9230 0x5608d43cb620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.011+0000 7f8e61800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43be240 0x5608d43be620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.011+0000 7f8e5be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43c9230 0x5608d43cb620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52614/0 (socket says 172.21.15.28:52614) 2024-04-24T08:17:05.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.013+0000 7f8e5be00640 1 -- 172.21.15.28:0/2426918680 learned_addr learned my addr 172.21.15.28:0/2426918680 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:05.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.013+0000 7f8e60e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43beb60 0x5608d43c8cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.013+0000 7f8e5be00640 1 -- 172.21.15.28:0/2426918680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43beb60 msgr2=0x5608d43c8cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:05.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.013+0000 7f8e5be00640 1 --2- 172.21.15.28:0/2426918680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43beb60 0x5608d43c8cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.013+0000 7f8e5be00640 1 -- 172.21.15.28:0/2426918680 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43be240 msgr2=0x5608d43be620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:05.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.013+0000 7f8e5be00640 1 --2- 172.21.15.28:0/2426918680 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43be240 0x5608d43be620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.013+0000 7f8e5be00640 1 -- 172.21.15.28:0/2426918680 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608d4315f50 con 0x5608d43c9230 2024-04-24T08:17:05.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.015+0000 7f8e59600640 1 --2- 172.21.15.28:0/388405047 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8e400a4050 0x7f8e400a6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.015+0000 7f8e59600640 1 -- 172.21.15.28:0/388405047 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f8e540d0d70 con 0x7f8e5c0743b0 2024-04-24T08:17:05.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.017+0000 7f8e61800640 1 --2- 172.21.15.28:0/388405047 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8e400a4050 0x7f8e400a6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.018+0000 7f8e5be00640 1 --2- 172.21.15.28:0/2426918680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43c9230 0x5608d43cb620 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009e70 tx=0x7f8e4400e5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58666d8377a753c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:05.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.018+0000 7f8e52000640 1 -- 172.21.15.28:0/2426918680 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e44017070 con 0x5608d43c9230 2024-04-24T08:17:05.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.019+0000 7f8e52000640 1 -- 172.21.15.28:0/2426918680 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e44013950 con 0x5608d43c9230 2024-04-24T08:17:05.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.019+0000 7f8e52000640 1 -- 172.21.15.28:0/2426918680 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e44012a70 con 0x5608d43c9230 2024-04-24T08:17:05.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.020+0000 7f8e667e8740 1 -- 172.21.15.28:0/2426918680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43c9230 msgr2=0x5608d43cb620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:05.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.020+0000 7f8e667e8740 1 --2- 172.21.15.28:0/2426918680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43c9230 0x5608d43cb620 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8e44009e70 tx=0x7f8e4400e5a0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.020+0000 7f8e61800640 1 --2- 172.21.15.28:0/388405047 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8e400a4050 0x7f8e400a6510 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c005ab0 tx=0x7f8e4c005a00 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:05.022 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.021+0000 7f8e667e8740 1 -- 172.21.15.28:0/2426918680 shutdown_connections 2024-04-24T08:17:05.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.021+0000 7f8e667e8740 1 --2- 172.21.15.28:0/2426918680 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43c9230 0x5608d43cb620 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.021+0000 7f8e667e8740 1 --2- 172.21.15.28:0/2426918680 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43beb60 0x5608d43c8cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.022+0000 7f8e667e8740 1 --2- 172.21.15.28:0/2426918680 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43be240 0x5608d43be620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.022+0000 7f8e667e8740 1 -- 172.21.15.28:0/2426918680 >> 172.21.15.28:0/2426918680 conn(0x5608d43ba560 msgr2=0x5608d43ba940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:05.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.024+0000 7f8e667e8740 1 -- 172.21.15.28:0/2426918680 shutdown_connections 2024-04-24T08:17:05.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.024+0000 7f8e667e8740 1 -- 172.21.15.28:0/2426918680 wait complete. 2024-04-24T08:17:05.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.025+0000 7f8e667e8740 1 -- start start 2024-04-24T08:17:05.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.026+0000 7f8e667e8740 1 Processor -- start 2024-04-24T08:17:05.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.028+0000 7f8e667e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43be240 0x5608d43c6390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.028+0000 7f8e667e8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43beb60 0x5608d43c68d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.028+0000 7f8e667e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43c6e10 0x5608d43d54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.028+0000 7f8e667e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5608d43cc020 con 0x5608d43c6e10 2024-04-24T08:17:05.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.028+0000 7f8e667e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5608d42d4690 con 0x5608d43be240 2024-04-24T08:17:05.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.028+0000 7f8e667e8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5608d43cc1a0 con 0x5608d43beb60 2024-04-24T08:17:05.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.028+0000 7f8e667e8740 10 client.0 _ll_register_callbacks cb 0x5608d4310650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:05.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.028+0000 7f8e60e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43be240 0x5608d43c6390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.028+0000 7f8e60e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43be240 0x5608d43c6390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33018/0 (socket says 172.21.15.28:33018) 2024-04-24T08:17:05.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.028+0000 7f8e60e00640 1 -- 172.21.15.28:0/1078152480 learned_addr learned my addr 172.21.15.28:0/1078152480 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:05.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.029+0000 7f8e5be00640 1 --2- 172.21.15.28:0/1078152480 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43beb60 0x5608d43c68d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.030+0000 7f8e61800640 1 --2- 172.21.15.28:0/1078152480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43c6e10 0x5608d43d54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.030+0000 7f8e5be00640 1 -- 172.21.15.28:0/1078152480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43be240 msgr2=0x5608d43c6390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.030+0000 7f8e5be00640 1 --2- 172.21.15.28:0/1078152480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43be240 0x5608d43c6390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.030+0000 7f8e5be00640 1 -- 172.21.15.28:0/1078152480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43c6e10 msgr2=0x5608d43d54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.030+0000 7f8e5be00640 1 --2- 172.21.15.28:0/1078152480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43c6e10 0x5608d43d54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.030+0000 7f8e5be00640 1 -- 172.21.15.28:0/1078152480 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608d42d3920 con 0x5608d43beb60 2024-04-24T08:17:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.031+0000 7f8e5be00640 1 --2- 172.21.15.28:0/1078152480 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43beb60 0x5608d43c68d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x5608d43bf9d0 tx=0x7f8e44014d40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dce8dbab03bb7384 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.031+0000 7f8e4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:17:05.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.032+0000 7f8e4aa00640 1 -- 172.21.15.28:0/1078152480 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e44022070 con 0x5608d43beb60 2024-04-24T08:17:05.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.032+0000 7f8e4aa00640 1 -- 172.21.15.28:0/1078152480 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f8e44004450 con 0x5608d43beb60 2024-04-24T08:17:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.032+0000 7f8e667e8740 1 -- 172.21.15.28:0/1078152480 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e44013950 con 0x5608d43beb60 2024-04-24T08:17:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.033+0000 7f8e4aa00640 1 -- 172.21.15.28:0/1078152480 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8e44017040 con 0x5608d43beb60 2024-04-24T08:17:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.033+0000 7f8e667e8740 10 client.26341 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.033+0000 7f8e667e8740 1 -- 172.21.15.28:0/1078152480 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5608d40f77d0 con 0x5608d43beb60 2024-04-24T08:17:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.034+0000 7f8e4aa00640 1 -- 172.21.15.28:0/1078152480 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e44009a50 con 0x5608d43beb60 2024-04-24T08:17:05.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.035+0000 7f8e4aa00640 1 -- 172.21.15.28:0/1078152480 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8e440037b0 con 0x5608d43beb60 2024-04-24T08:17:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.035+0000 7f8e667e8740 10 client.26341 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.035+0000 7f8e667e8740 10 client.26341 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:17:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.035+0000 7f8e667e8740 20 client.26341 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.035+0000 7f8e667e8740 1 --2- 172.21.15.28:0/1078152480 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5608d43d9a90 0x5608d43dbe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.035+0000 7f8e667e8740 4 client.26341 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:17:05.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.035+0000 7f8e667e8740 1 -- 172.21.15.28:0/1078152480 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f8e44013950 con 0x5608d43d9a90 2024-04-24T08:17:05.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.037+0000 7f8e60e00640 1 --2- 172.21.15.28:0/1078152480 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5608d43d9a90 0x5608d43dbe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.040+0000 7f8e60e00640 1 --2- 172.21.15.28:0/1078152480 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5608d43d9a90 0x5608d43dbe80 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e78d5518c169e23b server_cookie=1b12019f8a097406 in_seq=0 out_seq=0 2024-04-24T08:17:05.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.040+0000 7f8e4aa00640 10 client.26341 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:05.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.043+0000 7f8e4aa00640 1 -- 172.21.15.28:0/1078152480 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f8e44013950 con 0x5608d43d9a90 2024-04-24T08:17:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.044+0000 7f8e4aa00640 10 client.26341 handle_command_reply: tid=0 2024-04-24T08:17:05.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.044+0000 7f8e667e8740 1 client.26341 shutdown 2024-04-24T08:17:05.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.045+0000 7f8e667e8740 1 -- 172.21.15.28:0/1078152480 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43beb60 msgr2=0x5608d43c68d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:05.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.045+0000 7f8e667e8740 1 --2- 172.21.15.28:0/1078152480 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43beb60 0x5608d43c68d0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x5608d43bf9d0 tx=0x7f8e44014d40 comp rx=0 tx=0).stop 2024-04-24T08:17:05.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.047+0000 7f8e667e8740 1 -- 172.21.15.28:0/1078152480 shutdown_connections 2024-04-24T08:17:05.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.047+0000 7f8e667e8740 1 --2- 172.21.15.28:0/1078152480 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5608d43d9a90 0x5608d43dbe80 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.047+0000 7f8e667e8740 1 --2- 172.21.15.28:0/1078152480 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43c6e10 0x5608d43d54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.047+0000 7f8e667e8740 1 --2- 172.21.15.28:0/1078152480 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43beb60 0x5608d43c68d0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.047+0000 7f8e667e8740 1 --2- 172.21.15.28:0/1078152480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43be240 0x5608d43c6390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.047+0000 7f8e667e8740 1 -- 172.21.15.28:0/1078152480 >> 172.21.15.28:0/1078152480 conn(0x5608d43ba560 msgr2=0x5608d43b0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:05.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.047+0000 7f8e4aa00640 0 client.26341 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:05.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.047+0000 7f8e667e8740 1 -- 172.21.15.28:0/1078152480 shutdown_connections 2024-04-24T08:17:05.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.047+0000 7f8e667e8740 1 -- 172.21.15.28:0/1078152480 wait complete. 2024-04-24T08:17:05.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.048+0000 7f8e667e8740 20 client.26341 trim_cache size 0 max 16384 2024-04-24T08:17:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.097+0000 7f8e667e8740 1 Processor -- start 2024-04-24T08:17:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.097+0000 7f8e667e8740 1 -- start start 2024-04-24T08:17:05.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.097+0000 7f8e667e8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43be240 0x5608d43c54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.098+0000 7f8e667e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43beb60 0x5608d43c0a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.098+0000 7f8e667e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43c6e10 0x5608d43b0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.098+0000 7f8e667e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c1b6b50 con 0x5608d43beb60 2024-04-24T08:17:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.098+0000 7f8e667e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c1b4420 con 0x5608d43c6e10 2024-04-24T08:17:05.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.098+0000 7f8e667e8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c105ec0 con 0x5608d43be240 2024-04-24T08:17:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.098+0000 7f8e61800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43c6e10 0x5608d43b0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.098+0000 7f8e60e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43be240 0x5608d43c54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.098+0000 7f8e61800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43c6e10 0x5608d43b0e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33032/0 (socket says 172.21.15.28:33032) 2024-04-24T08:17:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.098+0000 7f8e61800640 1 -- 172.21.15.28:0/757428100 learned_addr learned my addr 172.21.15.28:0/757428100 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:05.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.099+0000 7f8e5be00640 1 --2- 172.21.15.28:0/757428100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43beb60 0x5608d43c0a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.100+0000 7f8e60e00640 1 -- 172.21.15.28:0/757428100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43c6e10 msgr2=0x5608d43b0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.100+0000 7f8e60e00640 1 --2- 172.21.15.28:0/757428100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43c6e10 0x5608d43b0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.100+0000 7f8e60e00640 1 -- 172.21.15.28:0/757428100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43beb60 msgr2=0x5608d43c0a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.100+0000 7f8e60e00640 1 --2- 172.21.15.28:0/757428100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43beb60 0x5608d43c0a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.100+0000 7f8e60e00640 1 -- 172.21.15.28:0/757428100 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5608d402d3f0 con 0x5608d43be240 2024-04-24T08:17:05.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.101+0000 7f8e60e00640 1 --2- 172.21.15.28:0/757428100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43be240 0x5608d43c54f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8e540990f0 tx=0x7f8e540bd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ba4d5c6ff0acd4aa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:05.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.102+0000 7f8e51600640 1 -- 172.21.15.28:0/757428100 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e540b5070 con 0x5608d43be240 2024-04-24T08:17:05.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.102+0000 7f8e51600640 1 -- 172.21.15.28:0/757428100 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e540bbe50 con 0x5608d43be240 2024-04-24T08:17:05.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.103+0000 7f8e51600640 1 -- 172.21.15.28:0/757428100 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e540b5070 con 0x5608d43be240 2024-04-24T08:17:05.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.104+0000 7f8e667e8740 1 -- 172.21.15.28:0/757428100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43be240 msgr2=0x5608d43c54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:05.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.105+0000 7f8e667e8740 1 --2- 172.21.15.28:0/757428100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43be240 0x5608d43c54f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f8e540990f0 tx=0x7f8e540bd040 comp rx=0 tx=0).stop 2024-04-24T08:17:05.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.106+0000 7f8e667e8740 1 -- 172.21.15.28:0/757428100 shutdown_connections 2024-04-24T08:17:05.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.106+0000 7f8e667e8740 1 --2- 172.21.15.28:0/757428100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43beb60 0x5608d43c0a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.106+0000 7f8e667e8740 1 --2- 172.21.15.28:0/757428100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43c6e10 0x5608d43b0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.106+0000 7f8e667e8740 1 --2- 172.21.15.28:0/757428100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43be240 0x5608d43c54f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.106+0000 7f8e667e8740 1 -- 172.21.15.28:0/757428100 >> 172.21.15.28:0/757428100 conn(0x5608d43ba560 msgr2=0x5608d43b0740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:05.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.106+0000 7f8e667e8740 1 -- 172.21.15.28:0/757428100 shutdown_connections 2024-04-24T08:17:05.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.106+0000 7f8e667e8740 1 -- 172.21.15.28:0/757428100 wait complete. 2024-04-24T08:17:05.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.107+0000 7f8e667e8740 1 -- start start 2024-04-24T08:17:05.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.107+0000 7f8e667e8740 1 Processor -- start 2024-04-24T08:17:05.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.108+0000 7f8e667e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43be240 0x5608d43d1440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.108+0000 7f8e667e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43beb60 0x5608d43d1980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.109+0000 7f8e667e8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43c6e10 0x5608d43c2600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.110+0000 7f8e5be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43beb60 0x5608d43d1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.110+0000 7f8e60e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43be240 0x5608d43d1440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.110+0000 7f8e5be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43beb60 0x5608d43d1980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33040/0 (socket says 172.21.15.28:33040) 2024-04-24T08:17:05.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.110+0000 7f8e60e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43be240 0x5608d43d1440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52646/0 (socket says 172.21.15.28:52646) 2024-04-24T08:17:05.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.110+0000 7f8e5be00640 1 -- 172.21.15.28:0/538663184 learned_addr learned my addr 172.21.15.28:0/538663184 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:05.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.111+0000 7f8e61800640 1 --2- 172.21.15.28:0/538663184 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43c6e10 0x5608d43c2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.111+0000 7f8e667e8740 1 -- 172.21.15.28:0/538663184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c1b28e0 con 0x5608d43be240 2024-04-24T08:17:05.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.111+0000 7f8e667e8740 1 -- 172.21.15.28:0/538663184 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c1b3d50 con 0x5608d43beb60 2024-04-24T08:17:05.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.111+0000 7f8e667e8740 1 -- 172.21.15.28:0/538663184 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f8e5c1b2fb0 con 0x5608d43c6e10 2024-04-24T08:17:05.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.112+0000 7f8e60e00640 1 -- 172.21.15.28:0/538663184 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43c6e10 msgr2=0x5608d43c2600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:05.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.112+0000 7f8e60e00640 1 --2- 172.21.15.28:0/538663184 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43c6e10 0x5608d43c2600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.112+0000 7f8e60e00640 1 -- 172.21.15.28:0/538663184 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43beb60 msgr2=0x5608d43d1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:05.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.112+0000 7f8e60e00640 1 --2- 172.21.15.28:0/538663184 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43beb60 0x5608d43d1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.112+0000 7f8e60e00640 1 -- 172.21.15.28:0/538663184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5608d3fed0f0 con 0x5608d43be240 2024-04-24T08:17:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.112+0000 7f8e5be00640 1 --2- 172.21.15.28:0/538663184 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43beb60 0x5608d43d1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:17:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.113+0000 7f8e667e8740 10 client.0 _ll_register_callbacks cb 0x7f8e44009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.113+0000 7f8e667e8740 1 -- 172.21.15.28:0/538663184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8e540bbe50 con 0x5608d43be240 2024-04-24T08:17:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.113+0000 7f8e4aa00640 10 client.16077 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:17:05.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.114+0000 7f8e60e00640 1 --2- 172.21.15.28:0/538663184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43be240 0x5608d43d1440 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8e5409cc20 tx=0x7f8e54000f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70ade8e590747eaa server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:05.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.114+0000 7f8e4aa00640 1 -- 172.21.15.28:0/538663184 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e540b5070 con 0x5608d43be240 2024-04-24T08:17:05.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.115+0000 7f8e4aa00640 1 -- 172.21.15.28:0/538663184 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f8e540ce6c0 con 0x5608d43be240 2024-04-24T08:17:05.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.115+0000 7f8e4aa00640 1 -- 172.21.15.28:0/538663184 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f8e540108c0 con 0x5608d43be240 2024-04-24T08:17:05.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.115+0000 7f8e667e8740 10 client.16077 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:05.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.115+0000 7f8e667e8740 1 -- 172.21.15.28:0/538663184 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({fsmap=26}) -- 0x5608d42e0810 con 0x5608d43be240 2024-04-24T08:17:05.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.116+0000 7f8e4aa00640 1 -- 172.21.15.28:0/538663184 <== mon.0 v2:172.21.15.28:3300/0 4 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f8e54099300 con 0x5608d43be240 2024-04-24T08:17:05.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.116+0000 7f8e667e8740 10 client.16077 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:05.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.116+0000 7f8e667e8740 10 client.16077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:17:05.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.116+0000 7f8e667e8740 20 client.16077 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:05.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.116+0000 7f8e667e8740 1 --2- 172.21.15.28:0/538663184 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5608d443f5f0 0x5608d44419e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.116+0000 7f8e667e8740 4 client.16077 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:17:05.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.116+0000 7f8e667e8740 1 -- 172.21.15.28:0/538663184 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5608d4328c10 con 0x5608d443f5f0 2024-04-24T08:17:05.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.117+0000 7f8e5be00640 1 --2- 172.21.15.28:0/538663184 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5608d443f5f0 0x5608d44419e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.120+0000 7f8e5be00640 1 --2- 172.21.15.28:0/538663184 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5608d443f5f0 0x5608d44419e0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d28b13f4076d08fc server_cookie=c92f3a1e52678586 in_seq=0 out_seq=0 2024-04-24T08:17:05.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.120+0000 7f8e4aa00640 10 client.16077 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:05.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.121+0000 7f8e4aa00640 1 -- 172.21.15.28:0/538663184 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5608d4328c10 con 0x5608d443f5f0 2024-04-24T08:17:05.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.121+0000 7f8e4aa00640 10 client.16077 handle_command_reply: tid=0 2024-04-24T08:17:05.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.121+0000 7f8e667e8740 1 client.16077 shutdown 2024-04-24T08:17:05.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.121+0000 7f8e667e8740 1 -- 172.21.15.28:0/538663184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43be240 msgr2=0x5608d43d1440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:05.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.122+0000 7f8e667e8740 1 --2- 172.21.15.28:0/538663184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43be240 0x5608d43d1440 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f8e5409cc20 tx=0x7f8e54000f40 comp rx=0 tx=0).stop 2024-04-24T08:17:05.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.122+0000 7f8e667e8740 1 -- 172.21.15.28:0/538663184 shutdown_connections 2024-04-24T08:17:05.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.122+0000 7f8e667e8740 1 --2- 172.21.15.28:0/538663184 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5608d443f5f0 0x5608d44419e0 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.122+0000 7f8e667e8740 1 --2- 172.21.15.28:0/538663184 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5608d43c6e10 0x5608d43c2600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.122+0000 7f8e667e8740 1 --2- 172.21.15.28:0/538663184 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5608d43beb60 0x5608d43d1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.122+0000 7f8e667e8740 1 --2- 172.21.15.28:0/538663184 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5608d43be240 0x5608d43d1440 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.122+0000 7f8e667e8740 1 -- 172.21.15.28:0/538663184 >> 172.21.15.28:0/538663184 conn(0x5608d43ba560 msgr2=0x5608d43b0fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:05.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.122+0000 7f8e4aa00640 0 client.16077 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:05.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.126+0000 7f8e667e8740 1 -- 172.21.15.28:0/538663184 shutdown_connections 2024-04-24T08:17:05.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.127+0000 7f8e667e8740 1 -- 172.21.15.28:0/538663184 wait complete. 2024-04-24T08:17:05.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.128+0000 7f8e667e8740 20 client.16077 trim_cache size 0 max 16384 2024-04-24T08:17:05.132 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:17:05.132 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-24T08:17:05.132 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-24T08:17:05.132 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:17:05.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.129+0000 7f8e62200640 1 -- 172.21.15.28:0/388405047 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8e400a4050 msgr2=0x7f8e400a6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:05.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.129+0000 7f8e62200640 1 --2- 172.21.15.28:0/388405047 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8e400a4050 0x7f8e400a6510 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f8e4c005ab0 tx=0x7f8e4c005a00 comp rx=0 tx=0).stop 2024-04-24T08:17:05.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.129+0000 7f8e62200640 1 -- 172.21.15.28:0/388405047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e5c0743b0 msgr2=0x7f8e5c073730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:05.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.129+0000 7f8e62200640 1 --2- 172.21.15.28:0/388405047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e5c0743b0 0x7f8e5c073730 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f8e5400eb00 tx=0x7f8e5400c2d0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.131+0000 7f8e62200640 1 -- 172.21.15.28:0/388405047 shutdown_connections 2024-04-24T08:17:05.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.131+0000 7f8e62200640 1 --2- 172.21.15.28:0/388405047 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f8e400a4050 0x7f8e400a6510 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.131+0000 7f8e62200640 1 --2- 172.21.15.28:0/388405047 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f8e5c074cd0 0x7f8e5c073c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.131+0000 7f8e62200640 1 --2- 172.21.15.28:0/388405047 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f8e5c10e760 0x7f8e5c06c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.131+0000 7f8e62200640 1 --2- 172.21.15.28:0/388405047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8e5c0743b0 0x7f8e5c073730 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.131+0000 7f8e62200640 1 -- 172.21.15.28:0/388405047 >> 172.21.15.28:0/388405047 conn(0x7f8e5c0fd9d0 msgr2=0x7f8e5c110950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:05.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.132+0000 7f8e62200640 1 -- 172.21.15.28:0/388405047 shutdown_connections 2024-04-24T08:17:05.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.132+0000 7f8e62200640 1 -- 172.21.15.28:0/388405047 wait complete. 2024-04-24T08:17:05.163 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:17:05.163 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:17:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.507+0000 7feda9e00640 1 -- 172.21.15.28:0/3350867915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda410ef60 msgr2=0x7feda411d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.507+0000 7feda9e00640 1 --2- 172.21.15.28:0/3350867915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda410ef60 0x7feda411d850 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fed9c009be0 tx=0x7fed9c02a6d0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.508+0000 7feda9e00640 1 -- 172.21.15.28:0/3350867915 shutdown_connections 2024-04-24T08:17:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.508+0000 7feda9e00640 1 --2- 172.21.15.28:0/3350867915 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda410ef60 0x7feda411d850 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.509+0000 7feda9e00640 1 --2- 172.21.15.28:0/3350867915 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feda4110bd0 0x7feda410e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.510+0000 7feda9e00640 1 --2- 172.21.15.28:0/3350867915 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feda4110760 0x7feda410e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.510+0000 7feda9e00640 1 -- 172.21.15.28:0/3350867915 >> 172.21.15.28:0/3350867915 conn(0x7feda406d120 msgr2=0x7feda406d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.510+0000 7feda9e00640 1 -- 172.21.15.28:0/3350867915 shutdown_connections 2024-04-24T08:17:05.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.511+0000 7feda9e00640 1 -- 172.21.15.28:0/3350867915 wait complete. 2024-04-24T08:17:05.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.513+0000 7feda9e00640 1 Processor -- start 2024-04-24T08:17:05.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.513+0000 7feda9e00640 1 -- start start 2024-04-24T08:17:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.514+0000 7feda9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda4110760 0x7feda41a7780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.515+0000 7feda9e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feda41a7cc0 0x7feda41abfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.515+0000 7feda9e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feda41ac4e0 0x7feda41ac990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.515+0000 7feda9e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7feda4121180 con 0x7feda4110760 2024-04-24T08:17:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.515+0000 7feda9e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7feda4121000 con 0x7feda41a7cc0 2024-04-24T08:17:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.515+0000 7feda9e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7feda4121300 con 0x7feda41ac4e0 2024-04-24T08:17:05.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.516+0000 7feda2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda4110760 0x7feda41a7780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.516+0000 7feda2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda4110760 0x7feda41a7780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52666/0 (socket says 172.21.15.28:52666) 2024-04-24T08:17:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.516+0000 7feda2a00640 1 -- 172.21.15.28:0/879016093 learned_addr learned my addr 172.21.15.28:0/879016093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.516+0000 7feda3400640 1 --2- 172.21.15.28:0/879016093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feda41ac4e0 0x7feda41ac990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.517+0000 7feda3400640 1 -- 172.21.15.28:0/879016093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feda41a7cc0 msgr2=0x7feda41abfa0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:17:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.517+0000 7feda3400640 1 --2- 172.21.15.28:0/879016093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feda41a7cc0 0x7feda41abfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.517+0000 7feda3400640 1 -- 172.21.15.28:0/879016093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda4110760 msgr2=0x7feda41a7780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.517+0000 7feda3400640 1 --2- 172.21.15.28:0/879016093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda4110760 0x7feda41a7780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.517+0000 7feda3400640 1 -- 172.21.15.28:0/879016093 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feda41b4250 con 0x7feda41ac4e0 2024-04-24T08:17:05.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.517+0000 7feda3400640 1 --2- 172.21.15.28:0/879016093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feda41ac4e0 0x7feda41ac990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fed9c034040 tx=0x7fed9c030640 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:05.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.519+0000 7feda8a00640 1 -- 172.21.15.28:0/879016093 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c041030 con 0x7feda41ac4e0 2024-04-24T08:17:05.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.519+0000 7feda8a00640 1 -- 172.21.15.28:0/879016093 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed9c003e50 con 0x7feda41ac4e0 2024-04-24T08:17:05.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.519+0000 7feda8a00640 1 -- 172.21.15.28:0/879016093 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed9c036040 con 0x7feda41ac4e0 2024-04-24T08:17:05.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.520+0000 7feda9e00640 1 -- 172.21.15.28:0/879016093 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feda4110fc0 con 0x7feda41ac4e0 2024-04-24T08:17:05.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.520+0000 7feda9e00640 1 -- 172.21.15.28:0/879016093 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feda4111430 con 0x7feda41ac4e0 2024-04-24T08:17:05.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.523+0000 7feda9e00640 1 -- 172.21.15.28:0/879016093 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feda4068e20 con 0x7feda41ac4e0 2024-04-24T08:17:05.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.531+0000 7feda8a00640 1 -- 172.21.15.28:0/879016093 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fed9c044070 con 0x7feda41ac4e0 2024-04-24T08:17:05.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.537+0000 7feda8a00640 1 --2- 172.21.15.28:0/879016093 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fed880b4f30 0x7fed880b73f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:05.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.537+0000 7feda8a00640 1 -- 172.21.15.28:0/879016093 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fed9c0f1470 con 0x7feda41ac4e0 2024-04-24T08:17:05.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.538+0000 7feda8a00640 1 -- 172.21.15.28:0/879016093 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fed9c0f1890 con 0x7feda41ac4e0 2024-04-24T08:17:05.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.540+0000 7feda2a00640 1 --2- 172.21.15.28:0/879016093 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fed880b4f30 0x7fed880b73f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:05.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:05.542+0000 7feda2a00640 1 --2- 172.21.15.28:0/879016093 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fed880b4f30 0x7fed880b73f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fed900058c0 tx=0x7fed90009260 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:06.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:05 smithi028 ceph-mon[27786]: pgmap v205: 97 pgs: 97 active+clean; 23 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 44 MiB/s wr, 9.30k op/s 2024-04-24T08:17:06.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:05 smithi117 ceph-mon[32438]: pgmap v205: 97 pgs: 97 active+clean; 23 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 44 MiB/s wr, 9.30k op/s 2024-04-24T08:17:06.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.127+0000 7feda9e00640 1 -- 172.21.15.28:0/879016093 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7feda41a8ad0 con 0x7feda41ac4e0 2024-04-24T08:17:06.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.130+0000 7feda8a00640 1 -- 172.21.15.28:0/879016093 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11871 (secure 0 0 0) 0x7feda41a8ad0 con 0x7feda41ac4e0 2024-04-24T08:17:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:17:06.132 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:14:38.663141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:17:06.133 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-24T08:17:06.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.137+0000 7feda9e00640 1 -- 172.21.15.28:0/879016093 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fed880b4f30 msgr2=0x7fed880b73f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:06.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.137+0000 7feda9e00640 1 --2- 172.21.15.28:0/879016093 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fed880b4f30 0x7fed880b73f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fed900058c0 tx=0x7fed90009260 comp rx=0 tx=0).stop 2024-04-24T08:17:06.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.138+0000 7feda9e00640 1 -- 172.21.15.28:0/879016093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feda41ac4e0 msgr2=0x7feda41ac990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:06.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.138+0000 7feda9e00640 1 --2- 172.21.15.28:0/879016093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feda41ac4e0 0x7feda41ac990 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fed9c034040 tx=0x7fed9c030640 comp rx=0 tx=0).stop 2024-04-24T08:17:06.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.141+0000 7feda9e00640 1 -- 172.21.15.28:0/879016093 shutdown_connections 2024-04-24T08:17:06.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.141+0000 7feda9e00640 1 --2- 172.21.15.28:0/879016093 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fed880b4f30 0x7fed880b73f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.141+0000 7feda9e00640 1 --2- 172.21.15.28:0/879016093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7feda41ac4e0 0x7feda41ac990 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.141+0000 7feda9e00640 1 --2- 172.21.15.28:0/879016093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feda41a7cc0 0x7feda41abfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.141+0000 7feda9e00640 1 --2- 172.21.15.28:0/879016093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7feda4110760 0x7feda41a7780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.141+0000 7feda9e00640 1 -- 172.21.15.28:0/879016093 >> 172.21.15.28:0/879016093 conn(0x7feda406d120 msgr2=0x7feda4071660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:06.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.144+0000 7feda9e00640 1 -- 172.21.15.28:0/879016093 shutdown_connections 2024-04-24T08:17:06.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.144+0000 7feda9e00640 1 -- 172.21.15.28:0/879016093 wait complete. 2024-04-24T08:17:06.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:17:06.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:05 smithi194 ceph-mon[32836]: pgmap v205: 97 pgs: 97 active+clean; 23 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 44 MiB/s wr, 9.30k op/s 2024-04-24T08:17:06.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.537+0000 7f92bf000640 1 -- 172.21.15.28:0/3015788851 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92b0004cb0 msgr2=0x7f92b0005130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:06.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.537+0000 7f92bf000640 1 --2- 172.21.15.28:0/3015788851 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92b0004cb0 0x7f92b0005130 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f92b8067d00 tx=0x7f92b8096960 comp rx=0 tx=0).stop 2024-04-24T08:17:06.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.539+0000 7f92bf000640 1 -- 172.21.15.28:0/3015788851 shutdown_connections 2024-04-24T08:17:06.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.539+0000 7f92bf000640 1 --2- 172.21.15.28:0/3015788851 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b0005670 0x7f92b00a2bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.539+0000 7f92bf000640 1 --2- 172.21.15.28:0/3015788851 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92b0004cb0 0x7f92b0005130 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.539+0000 7f92bf000640 1 --2- 172.21.15.28:0/3015788851 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92b009be40 0x7f92b009c220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.539+0000 7f92bf000640 1 -- 172.21.15.28:0/3015788851 >> 172.21.15.28:0/3015788851 conn(0x7f92b000a770 msgr2=0x7f92b000ab80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:06.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.540+0000 7f92bf000640 1 -- 172.21.15.28:0/3015788851 shutdown_connections 2024-04-24T08:17:06.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.540+0000 7f92bf000640 1 -- 172.21.15.28:0/3015788851 wait complete. 2024-04-24T08:17:06.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.541+0000 7f92bf000640 1 Processor -- start 2024-04-24T08:17:06.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.542+0000 7f92bf000640 1 -- start start 2024-04-24T08:17:06.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.542+0000 7f92bf000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92b0004cb0 0x7f92b012e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.542+0000 7f92bf000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92b0005670 0x7f92b012ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.543+0000 7f92b7e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92b0004cb0 0x7f92b012e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:06.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.543+0000 7f92b7e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92b0004cb0 0x7f92b012e680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33060/0 (socket says 172.21.15.28:33060) 2024-04-24T08:17:06.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.543+0000 7f92b7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92b0005670 0x7f92b012ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:06.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.543+0000 7f92b7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92b0005670 0x7f92b012ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:45518/0 (socket says 172.21.15.28:45518) 2024-04-24T08:17:06.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.543+0000 7f92bf000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b009be40 0x7f92b0132f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.544+0000 7f92bd000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b009be40 0x7f92b0132f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:06.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.544+0000 7f92b7400640 1 -- 172.21.15.28:0/2502308274 learned_addr learned my addr 172.21.15.28:0/2502308274 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:06.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.544+0000 7f92bf000640 1 -- 172.21.15.28:0/2502308274 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f92b00a6180 con 0x7f92b009be40 2024-04-24T08:17:06.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.544+0000 7f92bf000640 1 -- 172.21.15.28:0/2502308274 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f92b00a6000 con 0x7f92b0004cb0 2024-04-24T08:17:06.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.544+0000 7f92bf000640 1 -- 172.21.15.28:0/2502308274 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f92b00a6300 con 0x7f92b0005670 2024-04-24T08:17:06.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.545+0000 7f92b7e00640 1 -- 172.21.15.28:0/2502308274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92b0005670 msgr2=0x7f92b012ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:06.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.545+0000 7f92b7e00640 1 --2- 172.21.15.28:0/2502308274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92b0005670 0x7f92b012ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.545+0000 7f92b7e00640 1 -- 172.21.15.28:0/2502308274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b009be40 msgr2=0x7f92b0132f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:06.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.545+0000 7f92b7e00640 1 --2- 172.21.15.28:0/2502308274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b009be40 0x7f92b0132f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.545+0000 7f92b7e00640 1 -- 172.21.15.28:0/2502308274 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f92b0133670 con 0x7f92b0004cb0 2024-04-24T08:17:06.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.545+0000 7f92bd000640 1 --2- 172.21.15.28:0/2502308274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b009be40 0x7f92b0132f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:17:06.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.546+0000 7f92b7e00640 1 --2- 172.21.15.28:0/2502308274 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92b0004cb0 0x7f92b012e680 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f92a8009fd0 tx=0x7f92a800b870 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:06.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.547+0000 7f92b4c00640 1 -- 172.21.15.28:0/2502308274 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a8015020 con 0x7f92b0004cb0 2024-04-24T08:17:06.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.547+0000 7f92bf000640 1 -- 172.21.15.28:0/2502308274 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f92b0133960 con 0x7f92b0004cb0 2024-04-24T08:17:06.549 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.547+0000 7f92bf000640 1 -- 172.21.15.28:0/2502308274 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f92b013b1f0 con 0x7f92b0004cb0 2024-04-24T08:17:06.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.548+0000 7f92b4c00640 1 -- 172.21.15.28:0/2502308274 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92a8011d10 con 0x7f92b0004cb0 2024-04-24T08:17:06.551 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.548+0000 7f92b4c00640 1 -- 172.21.15.28:0/2502308274 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a8015020 con 0x7f92b0004cb0 2024-04-24T08:17:06.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.560+0000 7f92b4c00640 1 -- 172.21.15.28:0/2502308274 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f92a8018020 con 0x7f92b0004cb0 2024-04-24T08:17:06.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.564+0000 7f92b4c00640 1 --2- 172.21.15.28:0/2502308274 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f92a00a4050 0x7f92a00a6510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.565+0000 7f92b7400640 1 --2- 172.21.15.28:0/2502308274 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f92a00a4050 0x7f92a00a6510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:06.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.567+0000 7f92c1daa740 1 Processor -- start 2024-04-24T08:17:06.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.567+0000 7f92b4c00640 1 -- 172.21.15.28:0/2502308274 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f92a80d0ab0 con 0x7f92b0004cb0 2024-04-24T08:17:06.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.567+0000 7f92c1daa740 1 -- start start 2024-04-24T08:17:06.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.568+0000 7f92c1daa740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004642b40 0x564004642f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.568+0000 7f92c1daa740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004643460 0x56400464d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.568+0000 7f92c1daa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400464db30 0x56400464ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.568+0000 7f92c1daa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5640045a81d0 con 0x56400464db30 2024-04-24T08:17:06.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.568+0000 7f92c1daa740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x56400454e450 con 0x564004642b40 2024-04-24T08:17:06.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.568+0000 7f92c1daa740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5640045a4190 con 0x564004643460 2024-04-24T08:17:06.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.569+0000 7f92b7400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004643460 0x56400464d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:06.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.569+0000 7f92b7e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004642b40 0x564004642f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:06.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.569+0000 7f92b7e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004642b40 0x564004642f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33068/0 (socket says 172.21.15.28:33068) 2024-04-24T08:17:06.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.569+0000 7f92b7e00640 1 -- 172.21.15.28:0/1682711992 learned_addr learned my addr 172.21.15.28:0/1682711992 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:06.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.570+0000 7f92b7e00640 1 -- 172.21.15.28:0/1682711992 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004643460 msgr2=0x56400464d5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:06.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.570+0000 7f92b7e00640 1 --2- 172.21.15.28:0/1682711992 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004643460 0x56400464d5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.570+0000 7f92b7e00640 1 -- 172.21.15.28:0/1682711992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400464db30 msgr2=0x56400464ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:06.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.571+0000 7f92b7e00640 1 --2- 172.21.15.28:0/1682711992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400464db30 0x56400464ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.571+0000 7f92b7e00640 1 -- 172.21.15.28:0/1682711992 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56400460b310 con 0x564004642b40 2024-04-24T08:17:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.571+0000 7f92b7400640 1 --2- 172.21.15.28:0/1682711992 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004643460 0x56400464d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.572+0000 7f92b7e00640 1 --2- 172.21.15.28:0/1682711992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004642b40 0x564004642f20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f92a8006fd0 tx=0x7f92a809c530 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9ac0fd13fd4cce10 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:06.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.573+0000 7f92b7400640 1 --2- 172.21.15.28:0/2502308274 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f92a00a4050 0x7f92a00a6510 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f92b8065520 tx=0x7f92b80684f0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:06.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.573+0000 7f92a5600640 1 -- 172.21.15.28:0/1682711992 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a80a0e90 con 0x564004642b40 2024-04-24T08:17:06.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.573+0000 7f92a5600640 1 -- 172.21.15.28:0/1682711992 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92a80a5750 con 0x564004642b40 2024-04-24T08:17:06.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.573+0000 7f92a5600640 1 -- 172.21.15.28:0/1682711992 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a80105e0 con 0x564004642b40 2024-04-24T08:17:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.575+0000 7f92c1daa740 1 -- 172.21.15.28:0/1682711992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004642b40 msgr2=0x564004642f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:06.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.575+0000 7f92c1daa740 1 --2- 172.21.15.28:0/1682711992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004642b40 0x564004642f20 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f92a8006fd0 tx=0x7f92a809c530 comp rx=0 tx=0).stop 2024-04-24T08:17:06.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.576+0000 7f92c1daa740 1 -- 172.21.15.28:0/1682711992 shutdown_connections 2024-04-24T08:17:06.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.576+0000 7f92c1daa740 1 --2- 172.21.15.28:0/1682711992 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400464db30 0x56400464ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.576+0000 7f92c1daa740 1 --2- 172.21.15.28:0/1682711992 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004643460 0x56400464d5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.576+0000 7f92c1daa740 1 --2- 172.21.15.28:0/1682711992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004642b40 0x564004642f20 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.576+0000 7f92c1daa740 1 -- 172.21.15.28:0/1682711992 >> 172.21.15.28:0/1682711992 conn(0x56400463ee60 msgr2=0x56400463f240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:06.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.576+0000 7f92c1daa740 1 -- 172.21.15.28:0/1682711992 shutdown_connections 2024-04-24T08:17:06.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.577+0000 7f92c1daa740 1 -- 172.21.15.28:0/1682711992 wait complete. 2024-04-24T08:17:06.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.577+0000 7f92c1daa740 1 -- start start 2024-04-24T08:17:06.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.578+0000 7f92c1daa740 1 Processor -- start 2024-04-24T08:17:06.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.581+0000 7f92c1daa740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004642b40 0x56400464a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.581+0000 7f92c1daa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564004643460 0x56400464ae50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.583+0000 7f92b7e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004642b40 0x56400464a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:06.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.583+0000 7f92b7e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004642b40 0x56400464a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33084/0 (socket says 172.21.15.28:33084) 2024-04-24T08:17:06.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.584+0000 7f92c1daa740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56400464db30 0x56400464b390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.584+0000 7f92c1daa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5640046509c0 con 0x564004643460 2024-04-24T08:17:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.584+0000 7f92c1daa740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x564004561810 con 0x564004642b40 2024-04-24T08:17:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.584+0000 7f92c1daa740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x564004650b40 con 0x56400464db30 2024-04-24T08:17:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.584+0000 7f92c1daa740 10 client.0 _ll_register_callbacks cb 0x56400452f140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.584+0000 7f92b7e00640 1 -- 172.21.15.28:0/1307382751 learned_addr learned my addr 172.21.15.28:0/1307382751 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:06.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.585+0000 7f92b7400640 1 --2- 172.21.15.28:0/1307382751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564004643460 0x56400464ae50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:06.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.585+0000 7f92b7e00640 1 -- 172.21.15.28:0/1307382751 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56400464db30 msgr2=0x56400464b390 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-04-24T08:17:06.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.585+0000 7f92b7e00640 1 --2- 172.21.15.28:0/1307382751 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56400464db30 0x56400464b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.585+0000 7f92b7e00640 1 -- 172.21.15.28:0/1307382751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564004643460 msgr2=0x56400464ae50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:06.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.585+0000 7f92b7e00640 1 --2- 172.21.15.28:0/1307382751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564004643460 0x56400464ae50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.585+0000 7f92b7e00640 1 -- 172.21.15.28:0/1307382751 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640045723f0 con 0x564004642b40 2024-04-24T08:17:06.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.585+0000 7f92b7e00640 1 --2- 172.21.15.28:0/1307382751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004642b40 0x56400464a910 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f92a8098f40 tx=0x7f92a8099380 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60bb3e44c6feb8c4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.585+0000 7f929a000640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:17:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.586+0000 7f929a000640 1 -- 172.21.15.28:0/1307382751 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a80b6040 con 0x564004642b40 2024-04-24T08:17:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.586+0000 7f929a000640 1 -- 172.21.15.28:0/1307382751 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f92a8010380 con 0x564004642b40 2024-04-24T08:17:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.586+0000 7f92c1daa740 1 -- 172.21.15.28:0/1307382751 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92a80a5750 con 0x564004642b40 2024-04-24T08:17:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.587+0000 7f929a000640 1 -- 172.21.15.28:0/1307382751 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f92a80ce5e0 con 0x564004642b40 2024-04-24T08:17:06.588 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.587+0000 7f92c1daa740 10 client.26327 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.587+0000 7f92c1daa740 1 -- 172.21.15.28:0/1307382751 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5640042ba3f0 con 0x564004642b40 2024-04-24T08:17:06.589 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.587+0000 7f929a000640 1 -- 172.21.15.28:0/1307382751 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92a80a5750 con 0x564004642b40 2024-04-24T08:17:06.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.590+0000 7f929a000640 1 -- 172.21.15.28:0/1307382751 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f92a80989d0 con 0x564004642b40 2024-04-24T08:17:06.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.590+0000 7f92c1daa740 10 client.26327 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:06.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.590+0000 7f92c1daa740 10 client.26327 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:17:06.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.590+0000 7f92c1daa740 20 client.26327 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:06.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.590+0000 7f92c1daa740 1 --2- 172.21.15.28:0/1307382751 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5640046663e0 0x5640046687d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.592 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.591+0000 7f92c1daa740 4 client.26327 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:17:06.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.591+0000 7f92b7400640 1 --2- 172.21.15.28:0/1307382751 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5640046663e0 0x5640046687d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:06.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.591+0000 7f92c1daa740 1 -- 172.21.15.28:0/1307382751 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640045b5bf0 con 0x5640046663e0 2024-04-24T08:17:06.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.592+0000 7f929a000640 10 client.26327 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:17:06.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.592+0000 7f92b7400640 1 --2- 172.21.15.28:0/1307382751 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5640046663e0 0x5640046687d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3474cd23eebd7183 server_cookie=101e57d9afba038e in_seq=0 out_seq=0 2024-04-24T08:17:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.592+0000 7f929a000640 1 -- 172.21.15.28:0/1307382751 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5640045b5bf0 con 0x5640046663e0 2024-04-24T08:17:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.592+0000 7f929a000640 10 client.26327 handle_command_reply: tid=0 2024-04-24T08:17:06.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.593+0000 7f92c1daa740 1 client.26327 shutdown 2024-04-24T08:17:06.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.595+0000 7f92c1daa740 1 -- 172.21.15.28:0/1307382751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004642b40 msgr2=0x56400464a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:06.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.595+0000 7f92c1daa740 1 --2- 172.21.15.28:0/1307382751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004642b40 0x56400464a910 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f92a8098f40 tx=0x7f92a8099380 comp rx=0 tx=0).stop 2024-04-24T08:17:06.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.595+0000 7f92c1daa740 1 -- 172.21.15.28:0/1307382751 shutdown_connections 2024-04-24T08:17:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.595+0000 7f92c1daa740 1 --2- 172.21.15.28:0/1307382751 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5640046663e0 0x5640046687d0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.595+0000 7f92c1daa740 1 --2- 172.21.15.28:0/1307382751 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x564004643460 0x56400464ae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.595+0000 7f92c1daa740 1 --2- 172.21.15.28:0/1307382751 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x56400464db30 0x56400464b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.595+0000 7f929a000640 0 client.26327 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:17:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.595+0000 7f92c1daa740 1 --2- 172.21.15.28:0/1307382751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004642b40 0x56400464a910 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.595+0000 7f92c1daa740 1 -- 172.21.15.28:0/1307382751 >> 172.21.15.28:0/1307382751 conn(0x56400463ee60 msgr2=0x564004644b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.596+0000 7f92c1daa740 1 -- 172.21.15.28:0/1307382751 shutdown_connections 2024-04-24T08:17:06.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.596+0000 7f92c1daa740 1 -- 172.21.15.28:0/1307382751 wait complete. 2024-04-24T08:17:06.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.596+0000 7f92c1daa740 20 client.26327 trim_cache size 0 max 16384 2024-04-24T08:17:06.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.646+0000 7f92c1daa740 1 Processor -- start 2024-04-24T08:17:06.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.647+0000 7f92c1daa740 1 -- start start 2024-04-24T08:17:06.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.647+0000 7f92c1daa740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004642b40 0x5640046452e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.648+0000 7f92c1daa740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004643460 0x56400464b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.648+0000 7f92c1daa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400464db30 0x564004647b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.648+0000 7f92c1daa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f929c011110 con 0x56400464db30 2024-04-24T08:17:06.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.648+0000 7f92c1daa740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f929c00e9e0 con 0x564004643460 2024-04-24T08:17:06.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.648+0000 7f92c1daa740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f929c00c830 con 0x564004642b40 2024-04-24T08:17:06.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.648+0000 7f92b7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004642b40 0x5640046452e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:06.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.649+0000 7f92b7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004642b40 0x5640046452e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:45542/0 (socket says 172.21.15.28:45542) 2024-04-24T08:17:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.649+0000 7f92b7e00640 1 -- 172.21.15.28:0/2722906128 learned_addr learned my addr 172.21.15.28:0/2722906128 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.649+0000 7f92b7400640 1 --2- 172.21.15.28:0/2722906128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004643460 0x56400464b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.649+0000 7f92b7e00640 1 -- 172.21.15.28:0/2722906128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004643460 msgr2=0x56400464b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.649+0000 7f92b7e00640 1 --2- 172.21.15.28:0/2722906128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004643460 0x56400464b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.649+0000 7f92b7e00640 1 -- 172.21.15.28:0/2722906128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400464db30 msgr2=0x564004647b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.649+0000 7f92b7e00640 1 --2- 172.21.15.28:0/2722906128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400464db30 0x564004647b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.649+0000 7f92b7e00640 1 -- 172.21.15.28:0/2722906128 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640043847d0 con 0x564004642b40 2024-04-24T08:17:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.650+0000 7f92b7e00640 1 --2- 172.21.15.28:0/2722906128 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004642b40 0x5640046452e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f92a809ca70 tx=0x7f92a80bd070 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0262dcfd47d1df1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:06.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.650+0000 7f92a4c00640 1 -- 172.21.15.28:0/2722906128 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a80b6040 con 0x564004642b40 2024-04-24T08:17:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.651+0000 7f92a4c00640 1 -- 172.21.15.28:0/2722906128 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92a80a5750 con 0x564004642b40 2024-04-24T08:17:06.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.652+0000 7f92a4c00640 1 -- 172.21.15.28:0/2722906128 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a8098900 con 0x564004642b40 2024-04-24T08:17:06.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.654+0000 7f92c1daa740 1 -- 172.21.15.28:0/2722906128 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004642b40 msgr2=0x5640046452e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:06.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.654+0000 7f92c1daa740 1 --2- 172.21.15.28:0/2722906128 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004642b40 0x5640046452e0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f92a809ca70 tx=0x7f92a80bd070 comp rx=0 tx=0).stop 2024-04-24T08:17:06.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.654+0000 7f92c1daa740 1 -- 172.21.15.28:0/2722906128 shutdown_connections 2024-04-24T08:17:06.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.654+0000 7f92c1daa740 1 --2- 172.21.15.28:0/2722906128 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400464db30 0x564004647b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.655+0000 7f92c1daa740 1 --2- 172.21.15.28:0/2722906128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004643460 0x56400464b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.655+0000 7f92c1daa740 1 --2- 172.21.15.28:0/2722906128 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004642b40 0x5640046452e0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.655+0000 7f92c1daa740 1 -- 172.21.15.28:0/2722906128 >> 172.21.15.28:0/2722906128 conn(0x56400463ee60 msgr2=0x564004659650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.655+0000 7f92c1daa740 1 -- 172.21.15.28:0/2722906128 shutdown_connections 2024-04-24T08:17:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.655+0000 7f92c1daa740 1 -- 172.21.15.28:0/2722906128 wait complete. 2024-04-24T08:17:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.656+0000 7f92c1daa740 1 -- start start 2024-04-24T08:17:06.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.656+0000 7f92c1daa740 1 Processor -- start 2024-04-24T08:17:06.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.657+0000 7f92c1daa740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004642b40 0x5640046c48d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.657+0000 7f92c1daa740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004643460 0x5640046c4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.657+0000 7f92c1daa740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400464db30 0x564004666b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.657+0000 7f92c1daa740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f929c00cea0 con 0x56400464db30 2024-04-24T08:17:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.657+0000 7f92c1daa740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f929c00e310 con 0x564004643460 2024-04-24T08:17:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.657+0000 7f92c1daa740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f929c00d570 con 0x564004642b40 2024-04-24T08:17:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.657+0000 7f92c1daa740 10 client.0 _ll_register_callbacks cb 0x7f92a80b4070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.658+0000 7f92b7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004642b40 0x5640046c48d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.658+0000 7f92b7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004642b40 0x5640046c48d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:45548/0 (socket says 172.21.15.28:45548) 2024-04-24T08:17:06.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.658+0000 7f92b7e00640 1 -- 172.21.15.28:0/2196769765 learned_addr learned my addr 172.21.15.28:0/2196769765 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.658+0000 7f92bd000640 1 --2- 172.21.15.28:0/2196769765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400464db30 0x564004666b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.658+0000 7f92b7400640 1 --2- 172.21.15.28:0/2196769765 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004643460 0x5640046c4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.658+0000 7f92b7e00640 1 -- 172.21.15.28:0/2196769765 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004643460 msgr2=0x5640046c4e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.658+0000 7f92b7e00640 1 --2- 172.21.15.28:0/2196769765 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004643460 0x5640046c4e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.658+0000 7f92b7e00640 1 -- 172.21.15.28:0/2196769765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400464db30 msgr2=0x564004666b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.660+0000 7f92b7e00640 1 --2- 172.21.15.28:0/2196769765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400464db30 0x564004666b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.660+0000 7f92b7e00640 1 -- 172.21.15.28:0/2196769765 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564004667260 con 0x564004642b40 2024-04-24T08:17:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.660+0000 7f92bd000640 1 --2- 172.21.15.28:0/2196769765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400464db30 0x564004666b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:17:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.662+0000 7f92b7e00640 1 --2- 172.21.15.28:0/2196769765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004642b40 0x5640046c48d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f92a8006fd0 tx=0x7f92a809bc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f8dfd5c07b191c90 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:06.663 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.662+0000 7f929a000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:17:06.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.664+0000 7f929a000640 1 -- 172.21.15.28:0/2196769765 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92a8098b40 con 0x564004642b40 2024-04-24T08:17:06.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.665+0000 7f929a000640 1 -- 172.21.15.28:0/2196769765 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f92a8010380 con 0x564004642b40 2024-04-24T08:17:06.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.666+0000 7f92c1daa740 1 -- 172.21.15.28:0/2196769765 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f92a80a5750 con 0x564004642b40 2024-04-24T08:17:06.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.666+0000 7f929a000640 1 -- 172.21.15.28:0/2196769765 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f92a80b6040 con 0x564004642b40 2024-04-24T08:17:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.666+0000 7f92c1daa740 10 client.26362 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.666+0000 7f92c1daa740 1 -- 172.21.15.28:0/2196769765 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x564004667260 con 0x564004642b40 2024-04-24T08:17:06.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.666+0000 7f929a000640 1 -- 172.21.15.28:0/2196769765 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f92a80a5750 con 0x564004642b40 2024-04-24T08:17:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.667+0000 7f929a000640 1 -- 172.21.15.28:0/2196769765 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f92a80a2230 con 0x564004642b40 2024-04-24T08:17:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.667+0000 7f92c1daa740 10 client.26362 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.667+0000 7f92c1daa740 10 client.26362 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:17:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.667+0000 7f92c1daa740 20 client.26362 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.667+0000 7f92c1daa740 1 --2- 172.21.15.28:0/2196769765 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5640046cc2b0 0x5640046ce6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.667+0000 7f92c1daa740 4 client.26362 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:17:06.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.667+0000 7f92c1daa740 1 -- 172.21.15.28:0/2196769765 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5640045afa60 con 0x5640046cc2b0 2024-04-24T08:17:06.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.668+0000 7f92b7400640 1 --2- 172.21.15.28:0/2196769765 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5640046cc2b0 0x5640046ce6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:06.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.669+0000 7f92b7400640 1 --2- 172.21.15.28:0/2196769765 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5640046cc2b0 0x5640046ce6a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ddeca0053c4c8638 server_cookie=a5c2d2ce1c3310c4 in_seq=0 out_seq=0 2024-04-24T08:17:06.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.669+0000 7f929a000640 10 client.26362 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:17:06.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.669+0000 7f929a000640 1 -- 172.21.15.28:0/2196769765 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5640045afa60 con 0x5640046cc2b0 2024-04-24T08:17:06.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.669+0000 7f929a000640 10 client.26362 handle_command_reply: tid=0 2024-04-24T08:17:06.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.669+0000 7f92c1daa740 1 client.26362 shutdown 2024-04-24T08:17:06.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.670+0000 7f92c1daa740 1 -- 172.21.15.28:0/2196769765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004642b40 msgr2=0x5640046c48d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:06.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.670+0000 7f92c1daa740 1 --2- 172.21.15.28:0/2196769765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004642b40 0x5640046c48d0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f92a8006fd0 tx=0x7f92a809bc50 comp rx=0 tx=0).stop 2024-04-24T08:17:06.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.670+0000 7f92c1daa740 1 -- 172.21.15.28:0/2196769765 shutdown_connections 2024-04-24T08:17:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.670+0000 7f92c1daa740 1 --2- 172.21.15.28:0/2196769765 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5640046cc2b0 0x5640046ce6a0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.670+0000 7f929a000640 0 client.26362 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:17:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.670+0000 7f92c1daa740 1 --2- 172.21.15.28:0/2196769765 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x56400464db30 0x564004666b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.670+0000 7f92c1daa740 1 --2- 172.21.15.28:0/2196769765 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x564004643460 0x5640046c4e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.670+0000 7f92c1daa740 1 --2- 172.21.15.28:0/2196769765 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x564004642b40 0x5640046c48d0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.670+0000 7f92c1daa740 1 -- 172.21.15.28:0/2196769765 >> 172.21.15.28:0/2196769765 conn(0x56400463ee60 msgr2=0x56400463d6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.671+0000 7f92c1daa740 1 -- 172.21.15.28:0/2196769765 shutdown_connections 2024-04-24T08:17:06.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.671+0000 7f92c1daa740 1 -- 172.21.15.28:0/2196769765 wait complete. 2024-04-24T08:17:06.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.674+0000 7f92c1daa740 20 client.26362 trim_cache size 0 max 16384 2024-04-24T08:17:06.677 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:17:06.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.682+0000 7f92bf000640 1 -- 172.21.15.28:0/2502308274 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f92a00a4050 msgr2=0x7f92a00a6510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:06.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.682+0000 7f92bf000640 1 --2- 172.21.15.28:0/2502308274 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f92a00a4050 0x7f92a00a6510 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f92b8065520 tx=0x7f92b80684f0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.682+0000 7f92bf000640 1 -- 172.21.15.28:0/2502308274 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92b0004cb0 msgr2=0x7f92b012e680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:06.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.682+0000 7f92bf000640 1 --2- 172.21.15.28:0/2502308274 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92b0004cb0 0x7f92b012e680 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f92a8009fd0 tx=0x7f92a800b870 comp rx=0 tx=0).stop 2024-04-24T08:17:06.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.682+0000 7f92bf000640 1 -- 172.21.15.28:0/2502308274 shutdown_connections 2024-04-24T08:17:06.684 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.682+0000 7f92bf000640 1 --2- 172.21.15.28:0/2502308274 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f92a00a4050 0x7f92a00a6510 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.682+0000 7f92bf000640 1 --2- 172.21.15.28:0/2502308274 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f92b009be40 0x7f92b0132f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.682+0000 7f92bf000640 1 --2- 172.21.15.28:0/2502308274 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f92b0005670 0x7f92b012ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.682+0000 7f92bf000640 1 --2- 172.21.15.28:0/2502308274 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92b0004cb0 0x7f92b012e680 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:06.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.682+0000 7f92bf000640 1 -- 172.21.15.28:0/2502308274 >> 172.21.15.28:0/2502308274 conn(0x7f92b000a770 msgr2=0x7f92b00a1810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:06.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.682+0000 7f92bf000640 1 -- 172.21.15.28:0/2502308274 shutdown_connections 2024-04-24T08:17:06.685 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:06.682+0000 7f92bf000640 1 -- 172.21.15.28:0/2502308274 wait complete. 2024-04-24T08:17:06.727 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:17:07.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:06 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/879016093' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:17:07.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.078+0000 7f9514000640 1 -- 172.21.15.28:0/1438994306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f950c11b7e0 msgr2=0x7f950c11dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.078+0000 7f9514000640 1 --2- 172.21.15.28:0/1438994306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f950c11b7e0 0x7f950c11dbd0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9500009890 tx=0x7f950002a360 comp rx=0 tx=0).stop 2024-04-24T08:17:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.078+0000 7f9514000640 1 -- 172.21.15.28:0/1438994306 shutdown_connections 2024-04-24T08:17:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.078+0000 7f9514000640 1 --2- 172.21.15.28:0/1438994306 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f950c11b7e0 0x7f950c11dbd0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.078+0000 7f9514000640 1 --2- 172.21.15.28:0/1438994306 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f950c10e880 0x7f950c10ed00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.079+0000 7f9514000640 1 --2- 172.21.15.28:0/1438994306 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f950c110850 0x7f950c10e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.079+0000 7f9514000640 1 -- 172.21.15.28:0/1438994306 >> 172.21.15.28:0/1438994306 conn(0x7f950c06d110 msgr2=0x7f950c06d520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:07.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.080+0000 7f9514000640 1 -- 172.21.15.28:0/1438994306 shutdown_connections 2024-04-24T08:17:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.080+0000 7f9514000640 1 -- 172.21.15.28:0/1438994306 wait complete. 2024-04-24T08:17:07.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.082+0000 7f9514000640 1 Processor -- start 2024-04-24T08:17:07.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.083+0000 7f9514000640 1 -- start start 2024-04-24T08:17:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.084+0000 7f9514000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f950c10e880 0x7f950c118dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.084+0000 7f9514000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f950c110850 0x7f950c111ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.084+0000 7f9514000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f950c11b7e0 0x7f950c1123e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.084+0000 7f9514000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f950c121140 con 0x7f950c10e880 2024-04-24T08:17:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.084+0000 7f9514000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f950c120fc0 con 0x7f950c110850 2024-04-24T08:17:07.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.084+0000 7f9514000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f950c1212c0 con 0x7f950c11b7e0 2024-04-24T08:17:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.085+0000 7f9510c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f950c110850 0x7f950c111ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.085+0000 7f9510c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f950c110850 0x7f950c111ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33112/0 (socket says 172.21.15.28:33112) 2024-04-24T08:17:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.085+0000 7f9510c00640 1 -- 172.21.15.28:0/3577482313 learned_addr learned my addr 172.21.15.28:0/3577482313 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.085+0000 7f9512000640 1 --2- 172.21.15.28:0/3577482313 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f950c11b7e0 0x7f950c1123e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.085+0000 7f9512000640 1 -- 172.21.15.28:0/3577482313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f950c110850 msgr2=0x7f950c111ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.086+0000 7f9512000640 1 --2- 172.21.15.28:0/3577482313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f950c110850 0x7f950c111ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.086+0000 7f9512000640 1 -- 172.21.15.28:0/3577482313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f950c10e880 msgr2=0x7f950c118dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.086+0000 7f9512000640 1 --2- 172.21.15.28:0/3577482313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f950c10e880 0x7f950c118dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.086+0000 7f9512000640 1 -- 172.21.15.28:0/3577482313 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f950c1129d0 con 0x7f950c11b7e0 2024-04-24T08:17:07.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.086+0000 7f9512000640 1 --2- 172.21.15.28:0/3577482313 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f950c11b7e0 0x7f950c1123e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f950002a950 tx=0x7f9500033d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.090+0000 7f950a000640 1 -- 172.21.15.28:0/3577482313 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500037040 con 0x7f950c11b7e0 2024-04-24T08:17:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.090+0000 7f950a000640 1 -- 172.21.15.28:0/3577482313 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9500040070 con 0x7f950c11b7e0 2024-04-24T08:17:07.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.090+0000 7f950a000640 1 -- 172.21.15.28:0/3577482313 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9500003970 con 0x7f950c11b7e0 2024-04-24T08:17:07.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.090+0000 7f9514000640 1 -- 172.21.15.28:0/3577482313 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f950c112c60 con 0x7f950c11b7e0 2024-04-24T08:17:07.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.090+0000 7f9514000640 1 -- 172.21.15.28:0/3577482313 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f950c116b30 con 0x7f950c11b7e0 2024-04-24T08:17:07.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.101+0000 7f9516e67740 1 Processor -- start 2024-04-24T08:17:07.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.101+0000 7f950a000640 1 -- 172.21.15.28:0/3577482313 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9500003b10 con 0x7f950c11b7e0 2024-04-24T08:17:07.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.104+0000 7f950a000640 1 --2- 172.21.15.28:0/3577482313 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f94f40a3e70 0x7f94f40a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.104+0000 7f950a000640 1 -- 172.21.15.28:0/3577482313 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f95000ba4c0 con 0x7f950c11b7e0 2024-04-24T08:17:07.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.105+0000 7f9516e67740 1 -- start start 2024-04-24T08:17:07.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.105+0000 7f9516e67740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104ce950 0x5600104ced30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.105+0000 7f9516e67740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104cf270 0x5600104d9400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.105+0000 7f9516e67740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104d9940 0x5600104dbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.106+0000 7f9516e67740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5600104341d0 con 0x5600104cf270 2024-04-24T08:17:07.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.106+0000 7f9516e67740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5600103da450 con 0x5600104ce950 2024-04-24T08:17:07.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.106+0000 7f9516e67740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560010430190 con 0x5600104d9940 2024-04-24T08:17:07.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.106+0000 7f9512000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104d9940 0x5600104dbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.106+0000 7f9512000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104d9940 0x5600104dbd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:45578/0 (socket says 172.21.15.28:45578) 2024-04-24T08:17:07.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.106+0000 7f9512000640 1 -- 172.21.15.28:0/1194376736 learned_addr learned my addr 172.21.15.28:0/1194376736 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:07.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.106+0000 7f9512000640 1 -- 172.21.15.28:0/1194376736 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104ce950 msgr2=0x5600104ced30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:17:07.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.106+0000 7f9512000640 1 --2- 172.21.15.28:0/1194376736 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104ce950 0x5600104ced30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.106+0000 7f9512000640 1 -- 172.21.15.28:0/1194376736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104cf270 msgr2=0x5600104d9400 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:17:07.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.106+0000 7f9512000640 1 --2- 172.21.15.28:0/1194376736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104cf270 0x5600104d9400 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.106+0000 7f9512000640 1 -- 172.21.15.28:0/1194376736 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560010497310 con 0x5600104d9940 2024-04-24T08:17:07.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.107+0000 7f9512000640 1 --2- 172.21.15.28:0/1194376736 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104d9940 0x5600104dbd30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f95000bc3b0 tx=0x7f95000b6f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=38d14e9a07ba4418 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:07.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.107+0000 7f94fe600640 1 -- 172.21.15.28:0/1194376736 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95000b8e50 con 0x5600104d9940 2024-04-24T08:17:07.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.107+0000 7f94fe600640 1 -- 172.21.15.28:0/1194376736 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9500039ec0 con 0x5600104d9940 2024-04-24T08:17:07.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.107+0000 7f94fe600640 1 -- 172.21.15.28:0/1194376736 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95000b8ff0 con 0x5600104d9940 2024-04-24T08:17:07.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.107+0000 7f9510c00640 1 --2- 172.21.15.28:0/3577482313 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f94f40a3e70 0x7f94f40a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.108+0000 7f9516e67740 1 -- 172.21.15.28:0/1194376736 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104d9940 msgr2=0x5600104dbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.108+0000 7f9516e67740 1 --2- 172.21.15.28:0/1194376736 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104d9940 0x5600104dbd30 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f95000bc3b0 tx=0x7f95000b6f90 comp rx=0 tx=0).stop 2024-04-24T08:17:07.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.110+0000 7f9510c00640 1 --2- 172.21.15.28:0/3577482313 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f94f40a3e70 0x7f94f40a6330 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x5600104cff10 tx=0x7f9504010040 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:07.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.115+0000 7f9516e67740 1 -- 172.21.15.28:0/1194376736 shutdown_connections 2024-04-24T08:17:07.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.115+0000 7f9516e67740 1 --2- 172.21.15.28:0/1194376736 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104cf270 0x5600104d9400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.115+0000 7f9516e67740 1 --2- 172.21.15.28:0/1194376736 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104d9940 0x5600104dbd30 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.115+0000 7f9516e67740 1 --2- 172.21.15.28:0/1194376736 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104ce950 0x5600104ced30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.115+0000 7f9516e67740 1 -- 172.21.15.28:0/1194376736 >> 172.21.15.28:0/1194376736 conn(0x5600104cac70 msgr2=0x5600104cb050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:07.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.115+0000 7f9516e67740 1 -- 172.21.15.28:0/1194376736 shutdown_connections 2024-04-24T08:17:07.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.115+0000 7f9516e67740 1 -- 172.21.15.28:0/1194376736 wait complete. 2024-04-24T08:17:07.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.116+0000 7f9516e67740 1 -- start start 2024-04-24T08:17:07.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.116+0000 7f9516e67740 1 Processor -- start 2024-04-24T08:17:07.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.117+0000 7f9516e67740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104ce950 0x5600104d93a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.117+0000 7f9516e67740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104cf270 0x5600104d6690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.117+0000 7f9516e67740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104d9940 0x5600104d6db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.117+0000 7f9516e67740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5600104dc4f0 con 0x5600104ce950 2024-04-24T08:17:07.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.117+0000 7f9516e67740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5600103ed810 con 0x5600104d9940 2024-04-24T08:17:07.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.117+0000 7f9516e67740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5600104dc670 con 0x5600104cf270 2024-04-24T08:17:07.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.117+0000 7f9516e67740 10 client.0 _ll_register_callbacks cb 0x5600103bb140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:07.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.117+0000 7f9511600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104ce950 0x5600104d93a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.117+0000 7f9511600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104ce950 0x5600104d93a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52748/0 (socket says 172.21.15.28:52748) 2024-04-24T08:17:07.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.117+0000 7f9511600640 1 -- 172.21.15.28:0/3737504994 learned_addr learned my addr 172.21.15.28:0/3737504994 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:07.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.118+0000 7f9512000640 1 --2- 172.21.15.28:0/3737504994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104d9940 0x5600104d6db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.118+0000 7f9510c00640 1 --2- 172.21.15.28:0/3737504994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104cf270 0x5600104d6690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.118+0000 7f9511600640 1 -- 172.21.15.28:0/3737504994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104cf270 msgr2=0x5600104d6690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:07.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.118+0000 7f9511600640 1 --2- 172.21.15.28:0/3737504994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104cf270 0x5600104d6690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.118+0000 7f9511600640 1 -- 172.21.15.28:0/3737504994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104d9940 msgr2=0x5600104d6db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:07.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.118+0000 7f9511600640 1 --2- 172.21.15.28:0/3737504994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104d9940 0x5600104d6db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.118+0000 7f9511600640 1 -- 172.21.15.28:0/3737504994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600103fe3f0 con 0x5600104ce950 2024-04-24T08:17:07.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.118+0000 7f9511600640 1 --2- 172.21.15.28:0/3737504994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104ce950 0x5600104d93a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f94f800bbf0 tx=0x7f94f800d6b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ad04835d529e55c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:07.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.119+0000 7f94f2a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:17:07.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.119+0000 7f94f2a00640 1 -- 172.21.15.28:0/3737504994 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8016070 con 0x5600104ce950 2024-04-24T08:17:07.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.119+0000 7f9516e67740 1 -- 172.21.15.28:0/3737504994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9500039ec0 con 0x5600104ce950 2024-04-24T08:17:07.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.121+0000 7f94f2a00640 1 -- 172.21.15.28:0/3737504994 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f94f8010660 con 0x5600104ce950 2024-04-24T08:17:07.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.121+0000 7f94f2a00640 1 -- 172.21.15.28:0/3737504994 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f94f8022050 con 0x5600104ce950 2024-04-24T08:17:07.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.121+0000 7f9516e67740 10 client.16098 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:07.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.121+0000 7f9516e67740 1 -- 172.21.15.28:0/3737504994 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5600101463f0 con 0x5600104ce950 2024-04-24T08:17:07.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.123+0000 7f94f2a00640 1 -- 172.21.15.28:0/3737504994 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9500039ec0 con 0x5600104ce950 2024-04-24T08:17:07.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.123+0000 7f94f2a00640 1 -- 172.21.15.28:0/3737504994 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f94f800f8d0 con 0x5600104ce950 2024-04-24T08:17:07.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.126+0000 7f9516e67740 10 client.16098 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:07.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.126+0000 7f9516e67740 10 client.16098 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:17:07.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.126+0000 7f9516e67740 20 client.16098 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:07.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.126+0000 7f9516e67740 1 --2- 172.21.15.28:0/3737504994 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5600104e9850 0x5600104ebcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.126+0000 7f9516e67740 4 client.16098 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:17:07.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.126+0000 7f9516e67740 1 -- 172.21.15.28:0/3737504994 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560010441bf0 con 0x5600104e9850 2024-04-24T08:17:07.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:06 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/879016093' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:17:07.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.127+0000 7f9510c00640 1 --2- 172.21.15.28:0/3737504994 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5600104e9850 0x5600104ebcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.129+0000 7f94f2a00640 10 client.16098 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:17:07.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.129+0000 7f9510c00640 1 --2- 172.21.15.28:0/3737504994 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5600104e9850 0x5600104ebcc0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=3ecdbef37030909b server_cookie=bb011ea121024c32 in_seq=0 out_seq=0 2024-04-24T08:17:07.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.131+0000 7f94f2a00640 1 -- 172.21.15.28:0/3737504994 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560010441bf0 con 0x5600104e9850 2024-04-24T08:17:07.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.131+0000 7f94f2a00640 10 client.16098 handle_command_reply: tid=0 2024-04-24T08:17:07.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.132+0000 7f9516e67740 1 client.16098 shutdown 2024-04-24T08:17:07.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.134+0000 7f9516e67740 1 -- 172.21.15.28:0/3737504994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104ce950 msgr2=0x5600104d93a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.134+0000 7f9516e67740 1 --2- 172.21.15.28:0/3737504994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104ce950 0x5600104d93a0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f94f800bbf0 tx=0x7f94f800d6b0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.135+0000 7f9516e67740 1 -- 172.21.15.28:0/3737504994 shutdown_connections 2024-04-24T08:17:07.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.135+0000 7f9516e67740 1 --2- 172.21.15.28:0/3737504994 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x5600104e9850 0x5600104ebcc0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.135+0000 7f9516e67740 1 --2- 172.21.15.28:0/3737504994 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104d9940 0x5600104d6db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.135+0000 7f9516e67740 1 --2- 172.21.15.28:0/3737504994 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104cf270 0x5600104d6690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.135+0000 7f9516e67740 1 --2- 172.21.15.28:0/3737504994 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104ce950 0x5600104d93a0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.135+0000 7f9516e67740 1 -- 172.21.15.28:0/3737504994 >> 172.21.15.28:0/3737504994 conn(0x5600104cac70 msgr2=0x5600104d0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:07.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.135+0000 7f94f2a00640 0 client.16098 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:17:07.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.135+0000 7f9516e67740 1 -- 172.21.15.28:0/3737504994 shutdown_connections 2024-04-24T08:17:07.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.136+0000 7f9516e67740 1 -- 172.21.15.28:0/3737504994 wait complete. 2024-04-24T08:17:07.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.136+0000 7f9516e67740 20 client.16098 trim_cache size 0 max 16384 2024-04-24T08:17:07.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.172+0000 7f9516e67740 1 Processor -- start 2024-04-24T08:17:07.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.172+0000 7f9516e67740 1 -- start start 2024-04-24T08:17:07.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.173+0000 7f9516e67740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104ce950 0x5600104d1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.173+0000 7f9516e67740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104cf270 0x5600104c9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.173+0000 7f9516e67740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104d9940 0x5600104d8ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.173+0000 7f9516e67740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f950c1c7c10 con 0x5600104cf270 2024-04-24T08:17:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.173+0000 7f9516e67740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f950c1c54e0 con 0x5600104d9940 2024-04-24T08:17:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.173+0000 7f9516e67740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f950c1c3330 con 0x5600104ce950 2024-04-24T08:17:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.173+0000 7f9511600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104ce950 0x5600104d1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.173+0000 7f9512000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104d9940 0x5600104d8ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.173+0000 7f9512000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104d9940 0x5600104d8ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33130/0 (socket says 172.21.15.28:33130) 2024-04-24T08:17:07.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.173+0000 7f9512000640 1 -- 172.21.15.28:0/4014214717 learned_addr learned my addr 172.21.15.28:0/4014214717 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:07.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.174+0000 7f9512000640 1 -- 172.21.15.28:0/4014214717 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104ce950 msgr2=0x5600104d1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:07.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.174+0000 7f9512000640 1 --2- 172.21.15.28:0/4014214717 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104ce950 0x5600104d1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.174+0000 7f9512000640 1 -- 172.21.15.28:0/4014214717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104cf270 msgr2=0x5600104c9c70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:17:07.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.174+0000 7f9512000640 1 --2- 172.21.15.28:0/4014214717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104cf270 0x5600104c9c70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.174+0000 7f9512000640 1 -- 172.21.15.28:0/4014214717 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5600104ec030 con 0x5600104d9940 2024-04-24T08:17:07.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.175+0000 7f9512000640 1 --2- 172.21.15.28:0/4014214717 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104d9940 0x5600104d8ed0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9500004650 tx=0x7f9500005740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21ef9ab66b6e80b7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:07.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.176+0000 7f94fdc00640 1 -- 172.21.15.28:0/4014214717 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95000b9190 con 0x5600104d9940 2024-04-24T08:17:07.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.177+0000 7f94fdc00640 1 -- 172.21.15.28:0/4014214717 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95000d5070 con 0x5600104d9940 2024-04-24T08:17:07.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.177+0000 7f94fdc00640 1 -- 172.21.15.28:0/4014214717 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f95000b9190 con 0x5600104d9940 2024-04-24T08:17:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.179+0000 7f9516e67740 1 -- 172.21.15.28:0/4014214717 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104d9940 msgr2=0x5600104d8ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.179+0000 7f9516e67740 1 --2- 172.21.15.28:0/4014214717 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104d9940 0x5600104d8ed0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f9500004650 tx=0x7f9500005740 comp rx=0 tx=0).stop 2024-04-24T08:17:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.179+0000 7f9516e67740 1 -- 172.21.15.28:0/4014214717 shutdown_connections 2024-04-24T08:17:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.179+0000 7f9516e67740 1 --2- 172.21.15.28:0/4014214717 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104cf270 0x5600104c9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.179+0000 7f9516e67740 1 --2- 172.21.15.28:0/4014214717 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104d9940 0x5600104d8ed0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.179+0000 7f9516e67740 1 --2- 172.21.15.28:0/4014214717 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104ce950 0x5600104d1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.179+0000 7f9516e67740 1 -- 172.21.15.28:0/4014214717 >> 172.21.15.28:0/4014214717 conn(0x5600104cac70 msgr2=0x5600104c9560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:07.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.179+0000 7f9516e67740 1 -- 172.21.15.28:0/4014214717 shutdown_connections 2024-04-24T08:17:07.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.181+0000 7f9516e67740 1 -- 172.21.15.28:0/4014214717 wait complete. 2024-04-24T08:17:07.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.182+0000 7f9516e67740 1 -- start start 2024-04-24T08:17:07.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.182+0000 7f9516e67740 1 Processor -- start 2024-04-24T08:17:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.183+0000 7f9516e67740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104ce950 0x5600104e1d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.183+0000 7f9516e67740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104cf270 0x5600104d3930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.183+0000 7f9516e67740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104d9940 0x5600104d3ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.183+0000 7f9516e67740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f950c1c39a0 con 0x5600104cf270 2024-04-24T08:17:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.183+0000 7f9516e67740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f950c1c4e10 con 0x5600104ce950 2024-04-24T08:17:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.183+0000 7f9516e67740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f950c1c4070 con 0x5600104d9940 2024-04-24T08:17:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.183+0000 7f9516e67740 10 client.0 _ll_register_callbacks cb 0x7f94f800b260 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.184+0000 7f9511600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104ce950 0x5600104e1d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.184+0000 7f9510c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104cf270 0x5600104d3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.184+0000 7f9511600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104ce950 0x5600104e1d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33144/0 (socket says 172.21.15.28:33144) 2024-04-24T08:17:07.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.184+0000 7f9511600640 1 -- 172.21.15.28:0/3240862008 learned_addr learned my addr 172.21.15.28:0/3240862008 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:07.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.185+0000 7f9511600640 1 -- 172.21.15.28:0/3240862008 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104d9940 msgr2=0x5600104d3ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:07.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.185+0000 7f9511600640 1 --2- 172.21.15.28:0/3240862008 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104d9940 0x5600104d3ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.185+0000 7f9511600640 1 -- 172.21.15.28:0/3240862008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104cf270 msgr2=0x5600104d3930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:07.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.185+0000 7f9511600640 1 --2- 172.21.15.28:0/3240862008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104cf270 0x5600104d3930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.185+0000 7f9511600640 1 -- 172.21.15.28:0/3240862008 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5600104cf0a0 con 0x5600104ce950 2024-04-24T08:17:07.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.185+0000 7f9511600640 1 --2- 172.21.15.28:0/3240862008 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104ce950 0x5600104e1d10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f94f800b890 tx=0x7f94f8004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c9da81c729a9356 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:07.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.186+0000 7f94f2a00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:17:07.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.186+0000 7f94f2a00640 1 -- 172.21.15.28:0/3240862008 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8016040 con 0x5600104ce950 2024-04-24T08:17:07.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.186+0000 7f9516e67740 1 -- 172.21.15.28:0/3240862008 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f95000d5070 con 0x5600104ce950 2024-04-24T08:17:07.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:06 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/879016093' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:17:07.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.190+0000 7f94f2a00640 1 -- 172.21.15.28:0/3240862008 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f94f800f3c0 con 0x5600104ce950 2024-04-24T08:17:07.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.191+0000 7f94f2a00640 1 -- 172.21.15.28:0/3240862008 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f94f8034050 con 0x5600104ce950 2024-04-24T08:17:07.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.192+0000 7f9516e67740 10 client.26348 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:07.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.192+0000 7f9516e67740 1 -- 172.21.15.28:0/3240862008 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5600102107d0 con 0x5600104ce950 2024-04-24T08:17:07.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.192+0000 7f94f2a00640 1 -- 172.21.15.28:0/3240862008 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f95000d5070 con 0x5600104ce950 2024-04-24T08:17:07.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.193+0000 7f94f2a00640 1 -- 172.21.15.28:0/3240862008 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f94f80035f0 con 0x5600104ce950 2024-04-24T08:17:07.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.194+0000 7f9516e67740 10 client.26348 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:07.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.194+0000 7f9516e67740 10 client.26348 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:17:07.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.194+0000 7f9516e67740 20 client.26348 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:07.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.194+0000 7f9516e67740 1 --2- 172.21.15.28:0/3240862008 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560010550210 0x560010552630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.194+0000 7f9516e67740 4 client.26348 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:17:07.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.194+0000 7f9516e67740 1 -- 172.21.15.28:0/3240862008 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56001043ba60 con 0x560010550210 2024-04-24T08:17:07.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.194+0000 7f9510c00640 1 --2- 172.21.15.28:0/3240862008 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560010550210 0x560010552630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.196+0000 7f9510c00640 1 --2- 172.21.15.28:0/3240862008 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560010550210 0x560010552630 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4ad4c2c4812c2da8 server_cookie=8a25db62ce372b38 in_seq=0 out_seq=0 2024-04-24T08:17:07.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.196+0000 7f94f2a00640 10 client.26348 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:17:07.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.196+0000 7f94f2a00640 1 -- 172.21.15.28:0/3240862008 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56001043ba60 con 0x560010550210 2024-04-24T08:17:07.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.196+0000 7f94f2a00640 10 client.26348 handle_command_reply: tid=0 2024-04-24T08:17:07.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.197+0000 7f9516e67740 1 client.26348 shutdown 2024-04-24T08:17:07.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.197+0000 7f9516e67740 1 -- 172.21.15.28:0/3240862008 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104ce950 msgr2=0x5600104e1d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.198 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.197+0000 7f9516e67740 1 --2- 172.21.15.28:0/3240862008 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104ce950 0x5600104e1d10 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f94f800b890 tx=0x7f94f8004270 comp rx=0 tx=0).stop 2024-04-24T08:17:07.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.201+0000 7f9516e67740 1 -- 172.21.15.28:0/3240862008 shutdown_connections 2024-04-24T08:17:07.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.201+0000 7f9516e67740 1 --2- 172.21.15.28:0/3240862008 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560010550210 0x560010552630 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.201+0000 7f9516e67740 1 --2- 172.21.15.28:0/3240862008 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5600104cf270 0x5600104d3930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.201+0000 7f9516e67740 1 --2- 172.21.15.28:0/3240862008 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5600104d9940 0x5600104d3ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.201+0000 7f9516e67740 1 --2- 172.21.15.28:0/3240862008 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5600104ce950 0x5600104e1d10 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.201+0000 7f94f2a00640 0 client.26348 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:17:07.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.201+0000 7f9516e67740 1 -- 172.21.15.28:0/3240862008 >> 172.21.15.28:0/3240862008 conn(0x5600104cac70 msgr2=0x5600104c9df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:07.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.202+0000 7f9516e67740 1 -- 172.21.15.28:0/3240862008 shutdown_connections 2024-04-24T08:17:07.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.202+0000 7f9516e67740 1 -- 172.21.15.28:0/3240862008 wait complete. 2024-04-24T08:17:07.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.202+0000 7f9516e67740 20 client.26348 trim_cache size 0 max 16384 2024-04-24T08:17:07.204 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:17:07.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.203+0000 7f9514000640 1 -- 172.21.15.28:0/3577482313 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f94f40a3e70 msgr2=0x7f94f40a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.203+0000 7f9514000640 1 --2- 172.21.15.28:0/3577482313 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f94f40a3e70 0x7f94f40a6330 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x5600104cff10 tx=0x7f9504010040 comp rx=0 tx=0).stop 2024-04-24T08:17:07.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.203+0000 7f9514000640 1 -- 172.21.15.28:0/3577482313 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f950c11b7e0 msgr2=0x7f950c1123e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.203+0000 7f9514000640 1 --2- 172.21.15.28:0/3577482313 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f950c11b7e0 0x7f950c1123e0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f950002a950 tx=0x7f9500033d10 comp rx=0 tx=0).stop 2024-04-24T08:17:07.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.204+0000 7f9514000640 1 -- 172.21.15.28:0/3577482313 shutdown_connections 2024-04-24T08:17:07.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.204+0000 7f9514000640 1 --2- 172.21.15.28:0/3577482313 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f94f40a3e70 0x7f94f40a6330 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.204+0000 7f9514000640 1 --2- 172.21.15.28:0/3577482313 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f950c11b7e0 0x7f950c1123e0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.204+0000 7f9514000640 1 --2- 172.21.15.28:0/3577482313 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f950c110850 0x7f950c111ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.204+0000 7f9514000640 1 --2- 172.21.15.28:0/3577482313 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f950c10e880 0x7f950c118dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.204+0000 7f9514000640 1 -- 172.21.15.28:0/3577482313 >> 172.21.15.28:0/3577482313 conn(0x7f950c06d110 msgr2=0x7f950c072b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:07.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.205+0000 7f9514000640 1 -- 172.21.15.28:0/3577482313 shutdown_connections 2024-04-24T08:17:07.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.205+0000 7f9514000640 1 -- 172.21.15.28:0/3577482313 wait complete. 2024-04-24T08:17:07.216 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi028 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1f81fb397ae98da7563d451a78c61574c8f4e6e0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-04-24T08:17:07.472 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=282c113091c842419bfbf9e31405648e Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_1f81fb397ae98da7563d451a78c61574c8f4e6e0/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9481b1d62f50e7d0a4f3dd83adf6945b08d5ff17/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/ffsb.sh) on smithi028 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1f81fb397ae98da7563d451a78c61574c8f4e6e0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh' 2024-04-24T08:17:07.474 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-04-24T08:17:07.485 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-04-24T08:17:07.485 INFO:teuthology.task.exec:Executing custom commands... 2024-04-24T08:17:07.485 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi028.front.sepia.ceph.com 2024-04-24T08:17:07.485 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-04-24T08:17:07.491 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:17:07.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.877+0000 7f88e3000640 1 -- 172.21.15.28:0/3246755762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88dc1063a0 msgr2=0x7f88dc106830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.877+0000 7f88daa00640 1 -- 172.21.15.28:0/3246755762 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88cc031600 con 0x7f88dc1063a0 2024-04-24T08:17:07.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.877+0000 7f88e3000640 1 --2- 172.21.15.28:0/3246755762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88dc1063a0 0x7f88dc106830 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f88cc009860 tx=0x7f88cc02a480 comp rx=0 tx=0).stop 2024-04-24T08:17:07.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.878+0000 7f88e3000640 1 -- 172.21.15.28:0/3246755762 shutdown_connections 2024-04-24T08:17:07.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.878+0000 7f88e3000640 1 --2- 172.21.15.28:0/3246755762 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88dc1063a0 0x7f88dc106830 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.878+0000 7f88e3000640 1 --2- 172.21.15.28:0/3246755762 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88dc1019a0 0x7f88dc105e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.878+0000 7f88e3000640 1 --2- 172.21.15.28:0/3246755762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88dc101080 0x7f88dc101460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.878+0000 7f88e3000640 1 -- 172.21.15.28:0/3246755762 >> 172.21.15.28:0/3246755762 conn(0x7f88dc077280 msgr2=0x7f88dc077690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:07.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.878+0000 7f88e3000640 1 -- 172.21.15.28:0/3246755762 shutdown_connections 2024-04-24T08:17:07.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.878+0000 7f88e3000640 1 -- 172.21.15.28:0/3246755762 wait complete. 2024-04-24T08:17:07.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.880+0000 7f88e3000640 1 Processor -- start 2024-04-24T08:17:07.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.880+0000 7f88e3000640 1 -- start start 2024-04-24T08:17:07.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.881+0000 7f88e3000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88dc101080 0x7f88dc19ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.881+0000 7f88e3000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88dc1019a0 0x7f88dc19f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.881+0000 7f88e3000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88dc1063a0 0x7f88dc1a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.881+0000 7f88e3000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc118a80 con 0x7f88dc101080 2024-04-24T08:17:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.881+0000 7f88e3000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc118900 con 0x7f88dc1019a0 2024-04-24T08:17:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.881+0000 7f88e3000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc118c00 con 0x7f88dc1063a0 2024-04-24T08:17:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.881+0000 7f88db400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88dc1019a0 0x7f88dc19f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.881+0000 7f88dbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88dc101080 0x7f88dc19ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.881+0000 7f88db400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88dc1019a0 0x7f88dc19f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33152/0 (socket says 172.21.15.28:33152) 2024-04-24T08:17:07.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.881+0000 7f88db400640 1 -- 172.21.15.28:0/2889124352 learned_addr learned my addr 172.21.15.28:0/2889124352 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.881+0000 7f88e1000640 1 --2- 172.21.15.28:0/2889124352 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88dc1063a0 0x7f88dc1a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.882+0000 7f88db400640 1 -- 172.21.15.28:0/2889124352 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88dc1063a0 msgr2=0x7f88dc1a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.882+0000 7f88db400640 1 --2- 172.21.15.28:0/2889124352 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88dc1063a0 0x7f88dc1a3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.882+0000 7f88db400640 1 -- 172.21.15.28:0/2889124352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88dc101080 msgr2=0x7f88dc19ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.882+0000 7f88db400640 1 --2- 172.21.15.28:0/2889124352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88dc101080 0x7f88dc19ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.882+0000 7f88db400640 1 -- 172.21.15.28:0/2889124352 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f88dc1a3d30 con 0x7f88dc1019a0 2024-04-24T08:17:07.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.882+0000 7f88db400640 1 --2- 172.21.15.28:0/2889124352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88dc1019a0 0x7f88dc19f280 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f88c800b450 tx=0x7f88c800b910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.883+0000 7f88d8c00640 1 -- 172.21.15.28:0/2889124352 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c8015020 con 0x7f88dc1019a0 2024-04-24T08:17:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.883+0000 7f88d8c00640 1 -- 172.21.15.28:0/2889124352 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88c8011d70 con 0x7f88dc1019a0 2024-04-24T08:17:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.883+0000 7f88d8c00640 1 -- 172.21.15.28:0/2889124352 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c800f340 con 0x7f88dc1019a0 2024-04-24T08:17:07.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.883+0000 7f88e3000640 1 -- 172.21.15.28:0/2889124352 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f88dc1a4020 con 0x7f88dc1019a0 2024-04-24T08:17:07.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.883+0000 7f88e3000640 1 -- 172.21.15.28:0/2889124352 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f88dc1ab8b0 con 0x7f88dc1019a0 2024-04-24T08:17:07.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.888+0000 7f88e5e2f740 1 Processor -- start 2024-04-24T08:17:07.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.888+0000 7f88e5e2f740 1 -- start start 2024-04-24T08:17:07.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.889+0000 7f88e5e2f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54f0950 0x562bb54f0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.889+0000 7f88e5e2f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54f1270 0x562bb54fb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.889+0000 7f88e5e2f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54fb940 0x562bb54fdd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.889+0000 7f88e5e2f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562bb54561d0 con 0x562bb54f1270 2024-04-24T08:17:07.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.889+0000 7f88e5e2f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562bb53fc450 con 0x562bb54f0950 2024-04-24T08:17:07.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.889+0000 7f88e5e2f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562bb5452190 con 0x562bb54fb940 2024-04-24T08:17:07.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.889+0000 7f88e1000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54f0950 0x562bb54f0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.890+0000 7f88e1000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54f0950 0x562bb54f0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33162/0 (socket says 172.21.15.28:33162) 2024-04-24T08:17:07.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.890+0000 7f88e1000640 1 -- 172.21.15.28:0/2855681778 learned_addr learned my addr 172.21.15.28:0/2855681778 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:07.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.890+0000 7f88dbe00640 1 --2- 172.21.15.28:0/2855681778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54f1270 0x562bb54fb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.891+0000 7f88e1000640 1 -- 172.21.15.28:0/2855681778 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54fb940 msgr2=0x562bb54fdd30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:17:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.891+0000 7f88e1000640 1 --2- 172.21.15.28:0/2855681778 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54fb940 0x562bb54fdd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.891+0000 7f88e1000640 1 -- 172.21.15.28:0/2855681778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54f1270 msgr2=0x562bb54fb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.891+0000 7f88e1000640 1 --2- 172.21.15.28:0/2855681778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54f1270 0x562bb54fb400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.891+0000 7f88e1000640 1 -- 172.21.15.28:0/2855681778 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bb54b9310 con 0x562bb54f0950 2024-04-24T08:17:07.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.892+0000 7f88d8c00640 1 -- 172.21.15.28:0/2889124352 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f88c8018020 con 0x7f88dc1019a0 2024-04-24T08:17:07.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.893+0000 7f88e1000640 1 --2- 172.21.15.28:0/2855681778 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54f0950 0x562bb54f0d30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f88cc009780 tx=0x7f88cc030e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=269b5d23a992c1a9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:07.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.893+0000 7f88d1600640 1 -- 172.21.15.28:0/2855681778 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88cc004450 con 0x562bb54f0950 2024-04-24T08:17:07.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.893+0000 7f88d1600640 1 -- 172.21.15.28:0/2855681778 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88cc030880 con 0x562bb54f0950 2024-04-24T08:17:07.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.893+0000 7f88d1600640 1 -- 172.21.15.28:0/2855681778 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88cc003df0 con 0x562bb54f0950 2024-04-24T08:17:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.894+0000 7f88d8c00640 1 --2- 172.21.15.28:0/2889124352 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88c00a3e60 0x7f88c00a6320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.894+0000 7f88d8c00640 1 -- 172.21.15.28:0/2889124352 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f88c80d0150 con 0x7f88dc1019a0 2024-04-24T08:17:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.894+0000 7f88e5e2f740 1 -- 172.21.15.28:0/2855681778 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54f0950 msgr2=0x562bb54f0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.895+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/2855681778 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54f0950 0x562bb54f0d30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f88cc009780 tx=0x7f88cc030e50 comp rx=0 tx=0).stop 2024-04-24T08:17:07.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.895+0000 7f88dbe00640 1 --2- 172.21.15.28:0/2889124352 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88c00a3e60 0x7f88c00a6320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.895+0000 7f88e5e2f740 1 -- 172.21.15.28:0/2855681778 shutdown_connections 2024-04-24T08:17:07.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.895+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/2855681778 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54f1270 0x562bb54fb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.895+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/2855681778 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54fb940 0x562bb54fdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.895+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/2855681778 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54f0950 0x562bb54f0d30 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.895+0000 7f88e5e2f740 1 -- 172.21.15.28:0/2855681778 >> 172.21.15.28:0/2855681778 conn(0x562bb54ecc70 msgr2=0x562bb54ed050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:07.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.895+0000 7f88e5e2f740 1 -- 172.21.15.28:0/2855681778 shutdown_connections 2024-04-24T08:17:07.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.896+0000 7f88e5e2f740 1 -- 172.21.15.28:0/2855681778 wait complete. 2024-04-24T08:17:07.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.897+0000 7f88dbe00640 1 --2- 172.21.15.28:0/2889124352 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88c00a3e60 0x7f88c00a6320 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x562bb54f1f10 tx=0x7f88d40095c0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:07.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.897+0000 7f88e5e2f740 1 -- start start 2024-04-24T08:17:07.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.897+0000 7f88e5e2f740 1 Processor -- start 2024-04-24T08:17:07.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.898+0000 7f88e5e2f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 0x562bb54f8700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.898+0000 7f88e5e2f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54f1270 0x562bb54f8c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.898+0000 7f88e5e2f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54fb940 0x562bb54f9180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.898+0000 7f88e5e2f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562bb550f780 con 0x562bb54f1270 2024-04-24T08:17:07.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.898+0000 7f88e5e2f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562bb540f810 con 0x562bb54fb940 2024-04-24T08:17:07.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.898+0000 7f88e5e2f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562bb550f900 con 0x562bb54f0950 2024-04-24T08:17:07.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.898+0000 7f88e5e2f740 10 client.0 _ll_register_callbacks cb 0x562bb53dd140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:07.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.898+0000 7f88dbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 0x562bb54f8700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.899+0000 7f88dbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 0x562bb54f8700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:45622/0 (socket says 172.21.15.28:45622) 2024-04-24T08:17:07.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.899+0000 7f88dbe00640 1 -- 172.21.15.28:0/1471798080 learned_addr learned my addr 172.21.15.28:0/1471798080 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:07.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.899+0000 7f88e1000640 1 --2- 172.21.15.28:0/1471798080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54fb940 0x562bb54f9180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.899+0000 7f88db400640 1 --2- 172.21.15.28:0/1471798080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54f1270 0x562bb54f8c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.900+0000 7f88db400640 1 -- 172.21.15.28:0/1471798080 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 msgr2=0x562bb54f8700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:07.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.900+0000 7f88db400640 1 --2- 172.21.15.28:0/1471798080 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 0x562bb54f8700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.900+0000 7f88db400640 1 -- 172.21.15.28:0/1471798080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54fb940 msgr2=0x562bb54f9180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:07.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.900+0000 7f88db400640 1 --2- 172.21.15.28:0/1471798080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54fb940 0x562bb54f9180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.900+0000 7f88db400640 1 -- 172.21.15.28:0/1471798080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bb54203f0 con 0x562bb54f1270 2024-04-24T08:17:07.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.901+0000 7f88e1000640 1 --2- 172.21.15.28:0/1471798080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54fb940 0x562bb54f9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:17:07.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.901+0000 7f88dbe00640 1 --2- 172.21.15.28:0/1471798080 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 0x562bb54f8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:17:07.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.901+0000 7f88db400640 1 --2- 172.21.15.28:0/1471798080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54f1270 0x562bb54f8c40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f88c80cd930 tx=0x7f88c809bcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf5610ef598e452f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:07.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.902+0000 7f88c6000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:17:07.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.903+0000 7f88c6000640 1 -- 172.21.15.28:0/1471798080 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88c809c180 con 0x562bb54f1270 2024-04-24T08:17:07.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.903+0000 7f88c6000640 1 -- 172.21.15.28:0/1471798080 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f88c80cdaa0 con 0x562bb54f1270 2024-04-24T08:17:07.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.903+0000 7f88e5e2f740 1 -- 172.21.15.28:0/1471798080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x562bb5463bf0 con 0x562bb54f1270 2024-04-24T08:17:07.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.904+0000 7f88c6000640 1 -- 172.21.15.28:0/1471798080 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f88c80bb050 con 0x562bb54f1270 2024-04-24T08:17:07.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.905+0000 7f88e5e2f740 10 client.16110 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:07.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.905+0000 7f88e5e2f740 1 -- 172.21.15.28:0/1471798080 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562bb51683f0 con 0x562bb54f1270 2024-04-24T08:17:07.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.906+0000 7f88c6000640 1 -- 172.21.15.28:0/1471798080 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x562bb5463bf0 con 0x562bb54f1270 2024-04-24T08:17:07.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.906+0000 7f88c6000640 1 -- 172.21.15.28:0/1471798080 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f88c809b770 con 0x562bb54f1270 2024-04-24T08:17:07.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.908+0000 7f88e5e2f740 10 client.16110 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:07.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.908+0000 7f88e5e2f740 10 client.16110 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:17:07.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.908+0000 7f88e5e2f740 20 client.16110 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:07.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.908+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/1471798080 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x562bb551caf0 0x562bb551eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.908+0000 7f88e5e2f740 4 client.16110 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:17:07.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.908+0000 7f88e5e2f740 1 -- 172.21.15.28:0/1471798080 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562bb545da60 con 0x562bb551caf0 2024-04-24T08:17:07.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.909+0000 7f88dbe00640 1 --2- 172.21.15.28:0/1471798080 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x562bb551caf0 0x562bb551eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.912+0000 7f88dbe00640 1 --2- 172.21.15.28:0/1471798080 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x562bb551caf0 0x562bb551eee0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcb25f19d23da0db server_cookie=846cc626f0d98fa7 in_seq=0 out_seq=0 2024-04-24T08:17:07.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.913+0000 7f88c6000640 10 client.16110 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.913+0000 7f88c6000640 1 -- 172.21.15.28:0/1471798080 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562bb545da60 con 0x562bb551caf0 2024-04-24T08:17:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.913+0000 7f88c6000640 10 client.16110 handle_command_reply: tid=0 2024-04-24T08:17:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.913+0000 7f88e5e2f740 1 client.16110 shutdown 2024-04-24T08:17:07.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.914+0000 7f88e5e2f740 1 -- 172.21.15.28:0/1471798080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54f1270 msgr2=0x562bb54f8c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.914+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/1471798080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54f1270 0x562bb54f8c40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f88c80cd930 tx=0x7f88c809bcc0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.914+0000 7f88e5e2f740 1 -- 172.21.15.28:0/1471798080 shutdown_connections 2024-04-24T08:17:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.914+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/1471798080 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x562bb551caf0 0x562bb551eee0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.914+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/1471798080 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54f1270 0x562bb54f8c40 secure :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f88c80cd930 tx=0x7f88c809bcc0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.914+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/1471798080 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54fb940 0x562bb54f9180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.914+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/1471798080 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 0x562bb54f8700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.914+0000 7f88c6000640 0 client.16110 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:07.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.914+0000 7f88e5e2f740 1 -- 172.21.15.28:0/1471798080 >> 172.21.15.28:0/1471798080 conn(0x562bb54ecc70 msgr2=0x562bb54f2970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.916+0000 7f88e5e2f740 1 -- 172.21.15.28:0/1471798080 shutdown_connections 2024-04-24T08:17:07.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.916+0000 7f88e5e2f740 1 -- 172.21.15.28:0/1471798080 wait complete. 2024-04-24T08:17:07.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.916+0000 7f88e5e2f740 20 client.16110 trim_cache size 0 max 16384 2024-04-24T08:17:07.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.947+0000 7f88e5e2f740 1 Processor -- start 2024-04-24T08:17:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.948+0000 7f88e5e2f740 1 -- start start 2024-04-24T08:17:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.948+0000 7f88e5e2f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 0x562bb5518640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.948+0000 7f88e5e2f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54fb940 0x562bb54ec570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.948+0000 7f88e5e2f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54faf40 0x562bb54f7860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.948+0000 7f88e5e2f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc1adbe0 con 0x562bb54fb940 2024-04-24T08:17:07.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.948+0000 7f88e5e2f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc1ae2b0 con 0x562bb54faf40 2024-04-24T08:17:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.948+0000 7f88e5e2f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc1abb80 con 0x562bb54f0950 2024-04-24T08:17:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.949+0000 7f88e1000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54faf40 0x562bb54f7860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.949+0000 7f88e1000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54faf40 0x562bb54f7860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33190/0 (socket says 172.21.15.28:33190) 2024-04-24T08:17:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.949+0000 7f88e1000640 1 -- 172.21.15.28:0/4007111676 learned_addr learned my addr 172.21.15.28:0/4007111676 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.950+0000 7f88dbe00640 1 --2- 172.21.15.28:0/4007111676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 0x562bb5518640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.950+0000 7f88e1000640 1 -- 172.21.15.28:0/4007111676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 msgr2=0x562bb5518640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:07.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.950+0000 7f88e1000640 1 --2- 172.21.15.28:0/4007111676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 0x562bb5518640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.950+0000 7f88e1000640 1 -- 172.21.15.28:0/4007111676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54fb940 msgr2=0x562bb54ec570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:07.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.950+0000 7f88e1000640 1 --2- 172.21.15.28:0/4007111676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54fb940 0x562bb54ec570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.950+0000 7f88e1000640 1 -- 172.21.15.28:0/4007111676 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562bb52327d0 con 0x562bb54faf40 2024-04-24T08:17:07.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.951+0000 7f88e1000640 1 --2- 172.21.15.28:0/4007111676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54faf40 0x562bb54f7860 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f88cc0039e0 tx=0x7f88cc003a10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=77fe440acc530f82 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:07.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.951+0000 7f88d0c00640 1 -- 172.21.15.28:0/4007111676 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88cc003df0 con 0x562bb54faf40 2024-04-24T08:17:07.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.952+0000 7f88d0c00640 1 -- 172.21.15.28:0/4007111676 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88cc004490 con 0x562bb54faf40 2024-04-24T08:17:07.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.952+0000 7f88d0c00640 1 -- 172.21.15.28:0/4007111676 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88cc003df0 con 0x562bb54faf40 2024-04-24T08:17:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.953+0000 7f88e5e2f740 1 -- 172.21.15.28:0/4007111676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54faf40 msgr2=0x562bb54f7860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.953+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/4007111676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54faf40 0x562bb54f7860 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f88cc0039e0 tx=0x7f88cc003a10 comp rx=0 tx=0).stop 2024-04-24T08:17:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.953+0000 7f88e5e2f740 1 -- 172.21.15.28:0/4007111676 shutdown_connections 2024-04-24T08:17:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.953+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/4007111676 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54fb940 0x562bb54ec570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.953+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/4007111676 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54faf40 0x562bb54f7860 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.953+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/4007111676 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 0x562bb5518640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.953+0000 7f88e5e2f740 1 -- 172.21.15.28:0/4007111676 >> 172.21.15.28:0/4007111676 conn(0x562bb54ecc70 msgr2=0x562bb54f86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:07.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.954+0000 7f88e5e2f740 1 -- 172.21.15.28:0/4007111676 shutdown_connections 2024-04-24T08:17:07.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.954+0000 7f88e5e2f740 1 -- 172.21.15.28:0/4007111676 wait complete. 2024-04-24T08:17:07.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.954+0000 7f88e5e2f740 1 -- start start 2024-04-24T08:17:07.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.955+0000 7f88e5e2f740 1 Processor -- start 2024-04-24T08:17:07.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.956+0000 7f88e5e2f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 0x562bb54fc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.956+0000 7f88e5e2f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54faf40 0x562bb54fc670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.957+0000 7f88e5e2f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54fb940 0x562bb5513b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.957+0000 7f88e5e2f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc10f320 con 0x562bb54faf40 2024-04-24T08:17:07.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.957+0000 7f88e5e2f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc110790 con 0x562bb54fb940 2024-04-24T08:17:07.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.957+0000 7f88e5e2f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f88dc10f9f0 con 0x562bb54f0950 2024-04-24T08:17:07.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.958+0000 7f88db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54faf40 0x562bb54fc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.958+0000 7f88e5e2f740 10 client.0 _ll_register_callbacks cb 0x7f88c8004700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:07.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.958+0000 7f88db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54faf40 0x562bb54fc670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52824/0 (socket says 172.21.15.28:52824) 2024-04-24T08:17:07.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.958+0000 7f88db400640 1 -- 172.21.15.28:0/4231867832 learned_addr learned my addr 172.21.15.28:0/4231867832 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:07.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.958+0000 7f88dbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 0x562bb54fc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.959+0000 7f88e1000640 1 --2- 172.21.15.28:0/4231867832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54fb940 0x562bb5513b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.959+0000 7f88dbe00640 1 -- 172.21.15.28:0/4231867832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54fb940 msgr2=0x562bb5513b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.959+0000 7f88dbe00640 1 --2- 172.21.15.28:0/4231867832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54fb940 0x562bb5513b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.959+0000 7f88dbe00640 1 -- 172.21.15.28:0/4231867832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54faf40 msgr2=0x562bb54fc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.959+0000 7f88dbe00640 1 --2- 172.21.15.28:0/4231867832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54faf40 0x562bb54fc670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.960+0000 7f88dbe00640 1 -- 172.21.15.28:0/4231867832 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562bb5514090 con 0x562bb54f0950 2024-04-24T08:17:07.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.961+0000 7f88dbe00640 1 --2- 172.21.15.28:0/4231867832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 0x562bb54fc130 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f88c809a430 tx=0x7f88d4004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6db41d93e135018e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:07.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.961+0000 7f88c6000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:17:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.963+0000 7f88c6000640 1 -- 172.21.15.28:0/4231867832 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f88d402b2f0 con 0x562bb54f0950 2024-04-24T08:17:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.964+0000 7f88c6000640 1 -- 172.21.15.28:0/4231867832 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7f88d402b490 con 0x562bb54f0950 2024-04-24T08:17:07.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.964+0000 7f88e5e2f740 1 -- 172.21.15.28:0/4231867832 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f88cc004490 con 0x562bb54f0950 2024-04-24T08:17:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.967+0000 7f88c6000640 1 -- 172.21.15.28:0/4231867832 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f88d402b6f0 con 0x562bb54f0950 2024-04-24T08:17:07.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.967+0000 7f88e5e2f740 10 client.26389 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.967+0000 7f88e5e2f740 1 -- 172.21.15.28:0/4231867832 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562bb55153b0 con 0x562bb54f0950 2024-04-24T08:17:07.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.968+0000 7f88c6000640 1 -- 172.21.15.28:0/4231867832 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f88cc004490 con 0x562bb54f0950 2024-04-24T08:17:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.968+0000 7f88c6000640 1 -- 172.21.15.28:0/4231867832 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f88d400b040 con 0x562bb54f0950 2024-04-24T08:17:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.968+0000 7f88e5e2f740 10 client.26389 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.968+0000 7f88e5e2f740 10 client.26389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:17:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.968+0000 7f88e5e2f740 20 client.26389 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.968+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/4231867832 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x562bb5583320 0x562bb5585710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:07.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.968+0000 7f88e5e2f740 4 client.26389 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:17:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.968+0000 7f88e5e2f740 1 -- 172.21.15.28:0/4231867832 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562bb53d94d0 con 0x562bb5583320 2024-04-24T08:17:07.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.970+0000 7f88db400640 1 --2- 172.21.15.28:0/4231867832 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x562bb5583320 0x562bb5585710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:07.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.971+0000 7f88db400640 1 --2- 172.21.15.28:0/4231867832 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x562bb5583320 0x562bb5585710 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3746ac022473bc39 server_cookie=d1c9a2f652b0a66f in_seq=0 out_seq=0 2024-04-24T08:17:07.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.971+0000 7f88c6000640 10 client.26389 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:07.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.975+0000 7f88c6000640 1 -- 172.21.15.28:0/4231867832 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562bb53d94d0 con 0x562bb5583320 2024-04-24T08:17:07.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.975+0000 7f88c6000640 10 client.26389 handle_command_reply: tid=0 2024-04-24T08:17:07.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.975+0000 7f88e5e2f740 1 client.26389 shutdown 2024-04-24T08:17:07.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.976+0000 7f88e5e2f740 1 -- 172.21.15.28:0/4231867832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 msgr2=0x562bb54fc130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.976+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/4231867832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 0x562bb54fc130 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f88c809a430 tx=0x7f88d4004ab0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.976+0000 7f88e5e2f740 1 -- 172.21.15.28:0/4231867832 shutdown_connections 2024-04-24T08:17:07.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.976+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/4231867832 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x562bb5583320 0x562bb5585710 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.976+0000 7f88c6000640 0 client.26389 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:07.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.976+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/4231867832 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562bb54faf40 0x562bb54fc670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.976+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/4231867832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562bb54fb940 0x562bb5513b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.976+0000 7f88e5e2f740 1 --2- 172.21.15.28:0/4231867832 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562bb54f0950 0x562bb54fc130 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.976+0000 7f88e5e2f740 1 -- 172.21.15.28:0/4231867832 >> 172.21.15.28:0/4231867832 conn(0x562bb54ecc70 msgr2=0x562bb54fa7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:07.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.977+0000 7f88e5e2f740 1 -- 172.21.15.28:0/4231867832 shutdown_connections 2024-04-24T08:17:07.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.977+0000 7f88e5e2f740 1 -- 172.21.15.28:0/4231867832 wait complete. 2024-04-24T08:17:07.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.978+0000 7f88e5e2f740 20 client.26389 trim_cache size 0 max 16384 2024-04-24T08:17:07.980 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:17:07.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.979+0000 7f88e3000640 1 -- 172.21.15.28:0/2889124352 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88c00a3e60 msgr2=0x7f88c00a6320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.979+0000 7f88e3000640 1 --2- 172.21.15.28:0/2889124352 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88c00a3e60 0x7f88c00a6320 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x562bb54f1f10 tx=0x7f88d40095c0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.979+0000 7f88e3000640 1 -- 172.21.15.28:0/2889124352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88dc1019a0 msgr2=0x7f88dc19f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:07.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.979+0000 7f88e3000640 1 --2- 172.21.15.28:0/2889124352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88dc1019a0 0x7f88dc19f280 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f88c800b450 tx=0x7f88c800b910 comp rx=0 tx=0).stop 2024-04-24T08:17:07.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.982+0000 7f88e3000640 1 -- 172.21.15.28:0/2889124352 shutdown_connections 2024-04-24T08:17:07.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.982+0000 7f88e3000640 1 --2- 172.21.15.28:0/2889124352 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f88c00a3e60 0x7f88c00a6320 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.982+0000 7f88e3000640 1 --2- 172.21.15.28:0/2889124352 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f88dc1063a0 0x7f88dc1a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.982+0000 7f88e3000640 1 --2- 172.21.15.28:0/2889124352 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88dc1019a0 0x7f88dc19f280 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.982+0000 7f88e3000640 1 --2- 172.21.15.28:0/2889124352 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f88dc101080 0x7f88dc19ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:07.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.982+0000 7f88e3000640 1 -- 172.21.15.28:0/2889124352 >> 172.21.15.28:0/2889124352 conn(0x7f88dc077280 msgr2=0x7f88dc103ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:07.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.983+0000 7f88e3000640 1 -- 172.21.15.28:0/2889124352 shutdown_connections 2024-04-24T08:17:07.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:07.983+0000 7f88e3000640 1 -- 172.21.15.28:0/2889124352 wait complete. 2024-04-24T08:17:08.012 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:07 smithi028 ceph-mon[27786]: pgmap v206: 97 pgs: 97 active+clean; 23 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 40 MiB/s wr, 8.50k op/s 2024-04-24T08:17:08.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:07 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:08.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:07 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:08.013 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:07 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:17:08.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:07 smithi117 ceph-mon[32438]: pgmap v206: 97 pgs: 97 active+clean; 23 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 40 MiB/s wr, 8.50k op/s 2024-04-24T08:17:08.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:07 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:08.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:07 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:08.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:07 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:17:08.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:07 smithi194 ceph-mon[32836]: pgmap v206: 97 pgs: 97 active+clean; 23 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 85 KiB/s rd, 40 MiB/s wr, 8.50k op/s 2024-04-24T08:17:08.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:07 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:17:08.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:07 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:17:08.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:07 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:17:09.016 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:17:09.016 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:09.017 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:17:09.141 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-04-24T08:17:09.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.359+0000 7fb35a800640 1 -- 172.21.15.28:0/68029970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34c0968e0 msgr2=0x7fb34c0a2b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.359+0000 7fb35a800640 1 --2- 172.21.15.28:0/68029970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34c0968e0 0x7fb34c0a2b70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb354067e40 tx=0x7fb3540968d0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.362+0000 7fb35a800640 1 -- 172.21.15.28:0/68029970 shutdown_connections 2024-04-24T08:17:09.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.362+0000 7fb35a800640 1 --2- 172.21.15.28:0/68029970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb34c095f20 0x7fb34c0963a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.362+0000 7fb35a800640 1 --2- 172.21.15.28:0/68029970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34c0968e0 0x7fb34c0a2b70 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.362+0000 7fb35a800640 1 --2- 172.21.15.28:0/68029970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb34c09ace0 0x7fb34c09b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.362+0000 7fb35a800640 1 -- 172.21.15.28:0/68029970 >> 172.21.15.28:0/68029970 conn(0x7fb34c00cb30 msgr2=0x7fb34c00cf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.362+0000 7fb35a800640 1 -- 172.21.15.28:0/68029970 shutdown_connections 2024-04-24T08:17:09.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.363+0000 7fb35a800640 1 -- 172.21.15.28:0/68029970 wait complete. 2024-04-24T08:17:09.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.366+0000 7fb35a800640 1 Processor -- start 2024-04-24T08:17:09.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.366+0000 7fb35a800640 1 -- start start 2024-04-24T08:17:09.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.368+0000 7fb35a800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb34c095f20 0x7fb34c0aa1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.368+0000 7fb35a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb34c0968e0 0x7fb34c0aa720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.369+0000 7fb35a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34c09ace0 0x7fb34c0afef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.369+0000 7fb35a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb34c008d10 con 0x7fb34c0968e0 2024-04-24T08:17:09.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.369+0000 7fb35a800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb34c008b90 con 0x7fb34c095f20 2024-04-24T08:17:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.369+0000 7fb359400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb34c095f20 0x7fb34c0aa1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.369+0000 7fb35a800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb34c0a4040 con 0x7fb34c09ace0 2024-04-24T08:17:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.369+0000 7fb359400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb34c095f20 0x7fb34c0aa1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33208/0 (socket says 172.21.15.28:33208) 2024-04-24T08:17:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.369+0000 7fb359400640 1 -- 172.21.15.28:0/3532960909 learned_addr learned my addr 172.21.15.28:0/3532960909 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:09.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.369+0000 7fb359e00640 1 --2- 172.21.15.28:0/3532960909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34c09ace0 0x7fb34c0afef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.369+0000 7fb358a00640 1 --2- 172.21.15.28:0/3532960909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb34c0968e0 0x7fb34c0aa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.371+0000 7fb359e00640 1 -- 172.21.15.28:0/3532960909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb34c095f20 msgr2=0x7fb34c0aa1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.371+0000 7fb359e00640 1 --2- 172.21.15.28:0/3532960909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb34c095f20 0x7fb34c0aa1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.371+0000 7fb359e00640 1 -- 172.21.15.28:0/3532960909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb34c0968e0 msgr2=0x7fb34c0aa720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.371+0000 7fb359e00640 1 --2- 172.21.15.28:0/3532960909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb34c0968e0 0x7fb34c0aa720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.371+0000 7fb359e00640 1 -- 172.21.15.28:0/3532960909 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb34c0b07b0 con 0x7fb34c09ace0 2024-04-24T08:17:09.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.371+0000 7fb359400640 1 --2- 172.21.15.28:0/3532960909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb34c095f20 0x7fb34c0aa1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:17:09.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.372+0000 7fb359e00640 1 --2- 172.21.15.28:0/3532960909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34c09ace0 0x7fb34c0afef0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb354065520 tx=0x7fb3540686e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.374+0000 7fb352000640 1 -- 172.21.15.28:0/3532960909 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3540ab020 con 0x7fb34c09ace0 2024-04-24T08:17:09.375 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.374+0000 7fb352000640 1 -- 172.21.15.28:0/3532960909 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb354068e40 con 0x7fb34c09ace0 2024-04-24T08:17:09.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.374+0000 7fb352000640 1 -- 172.21.15.28:0/3532960909 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3540a2040 con 0x7fb34c09ace0 2024-04-24T08:17:09.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.374+0000 7fb35a800640 1 -- 172.21.15.28:0/3532960909 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb34c0b0a40 con 0x7fb34c09ace0 2024-04-24T08:17:09.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.374+0000 7fb35a800640 1 -- 172.21.15.28:0/3532960909 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb34c0a5590 con 0x7fb34c09ace0 2024-04-24T08:17:09.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.382+0000 7fb352000640 1 -- 172.21.15.28:0/3532960909 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb3540ae020 con 0x7fb34c09ace0 2024-04-24T08:17:09.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.386+0000 7fb35ed81740 1 Processor -- start 2024-04-24T08:17:09.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.387+0000 7fb352000640 1 --2- 172.21.15.28:0/3532960909 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb33c0a3ea0 0x7fb33c0a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.388+0000 7fb352000640 1 -- 172.21.15.28:0/3532960909 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fb35415cc90 con 0x7fb34c09ace0 2024-04-24T08:17:09.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.389+0000 7fb358a00640 1 --2- 172.21.15.28:0/3532960909 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb33c0a3ea0 0x7fb33c0a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.389+0000 7fb35ed81740 1 -- start start 2024-04-24T08:17:09.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.392+0000 7fb35ed81740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8608270 0x55aea8608650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.392+0000 7fb35ed81740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608b90 0x55aea8612d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.392+0000 7fb35ed81740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8613260 0x55aea8615650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.392+0000 7fb35ed81740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aea856de70 con 0x55aea8608270 2024-04-24T08:17:09.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.392+0000 7fb35ed81740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55aea8514380 con 0x55aea8608b90 2024-04-24T08:17:09.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.392+0000 7fb35ed81740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55aea8569e30 con 0x55aea8613260 2024-04-24T08:17:09.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.395+0000 7fb358a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608b90 0x55aea8612d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.395+0000 7fb359e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8613260 0x55aea8615650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.395+0000 7fb359e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8613260 0x55aea8615650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:45656/0 (socket says 172.21.15.28:45656) 2024-04-24T08:17:09.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.396+0000 7fb359e00640 1 -- 172.21.15.28:0/2939819972 learned_addr learned my addr 172.21.15.28:0/2939819972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:09.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.398+0000 7fb359400640 1 --2- 172.21.15.28:0/2939819972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8608270 0x55aea8608650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.398+0000 7fb359e00640 1 -- 172.21.15.28:0/2939819972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608b90 msgr2=0x55aea8612d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.398+0000 7fb359e00640 1 --2- 172.21.15.28:0/2939819972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608b90 0x55aea8612d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.398+0000 7fb359e00640 1 -- 172.21.15.28:0/2939819972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8608270 msgr2=0x55aea8608650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.398+0000 7fb359e00640 1 --2- 172.21.15.28:0/2939819972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8608270 0x55aea8608650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.398+0000 7fb359e00640 1 -- 172.21.15.28:0/2939819972 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aea83487e0 con 0x55aea8613260 2024-04-24T08:17:09.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.399+0000 7fb359e00640 1 --2- 172.21.15.28:0/2939819972 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8613260 0x55aea8615650 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb354125070 tx=0x7fb354128630 comp rx=0 tx=0).ready entity=mon.1 client_cookie=79b340e2e806afb2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.400+0000 7fb346600640 1 -- 172.21.15.28:0/2939819972 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb354129310 con 0x55aea8613260 2024-04-24T08:17:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.400+0000 7fb346600640 1 -- 172.21.15.28:0/2939819972 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb354128c10 con 0x55aea8613260 2024-04-24T08:17:09.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.400+0000 7fb346600640 1 -- 172.21.15.28:0/2939819972 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3541490f0 con 0x55aea8613260 2024-04-24T08:17:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.400+0000 7fb358a00640 1 --2- 172.21.15.28:0/3532960909 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb33c0a3ea0 0x7fb33c0a6360 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb340005fd0 tx=0x7fb34000b7d0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.403+0000 7fb35ed81740 1 -- 172.21.15.28:0/2939819972 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8613260 msgr2=0x55aea8615650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.403+0000 7fb35ed81740 1 --2- 172.21.15.28:0/2939819972 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8613260 0x55aea8615650 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fb354125070 tx=0x7fb354128630 comp rx=0 tx=0).stop 2024-04-24T08:17:09.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.403+0000 7fb35ed81740 1 -- 172.21.15.28:0/2939819972 shutdown_connections 2024-04-24T08:17:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.404+0000 7fb35ed81740 1 --2- 172.21.15.28:0/2939819972 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8613260 0x55aea8615650 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.405+0000 7fb35ed81740 1 --2- 172.21.15.28:0/2939819972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608b90 0x55aea8612d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.405+0000 7fb35ed81740 1 --2- 172.21.15.28:0/2939819972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8608270 0x55aea8608650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.405+0000 7fb35ed81740 1 -- 172.21.15.28:0/2939819972 >> 172.21.15.28:0/2939819972 conn(0x55aea8604590 msgr2=0x55aea8604970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.405+0000 7fb35ed81740 1 -- 172.21.15.28:0/2939819972 shutdown_connections 2024-04-24T08:17:09.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.405+0000 7fb35ed81740 1 -- 172.21.15.28:0/2939819972 wait complete. 2024-04-24T08:17:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.406+0000 7fb35ed81740 1 -- start start 2024-04-24T08:17:09.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.406+0000 7fb35ed81740 1 Processor -- start 2024-04-24T08:17:09.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.410+0000 7fb35ed81740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8608270 0x55aea86101d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.411+0000 7fb35ed81740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8608b90 0x55aea8610710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.411+0000 7fb359400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8608270 0x55aea86101d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.411+0000 7fb359400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8608270 0x55aea86101d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:45658/0 (socket says 172.21.15.28:45658) 2024-04-24T08:17:09.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.412+0000 7fb359400640 1 -- 172.21.15.28:0/3971161782 learned_addr learned my addr 172.21.15.28:0/3971161782 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.412+0000 7fb35ed81740 1 --2- 172.21.15.28:0/3971161782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8613260 0x55aea8610c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.412+0000 7fb358a00640 1 --2- 172.21.15.28:0/3971161782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8608b90 0x55aea8610710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.412+0000 7fb35ed81740 1 -- 172.21.15.28:0/3971161782 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55aea8569e30 con 0x55aea8608b90 2024-04-24T08:17:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.412+0000 7fb35ed81740 1 -- 172.21.15.28:0/3971161782 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55aea8527660 con 0x55aea8613260 2024-04-24T08:17:09.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.412+0000 7fb35ed81740 1 -- 172.21.15.28:0/3971161782 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55aea8615e10 con 0x55aea8608270 2024-04-24T08:17:09.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.412+0000 7fb359e00640 1 --2- 172.21.15.28:0/3971161782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8613260 0x55aea8610c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.413+0000 7fb359e00640 1 -- 172.21.15.28:0/3971161782 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8608270 msgr2=0x55aea86101d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.413+0000 7fb359e00640 1 --2- 172.21.15.28:0/3971161782 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8608270 0x55aea86101d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.413+0000 7fb359e00640 1 -- 172.21.15.28:0/3971161782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8608b90 msgr2=0x55aea8610710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.413+0000 7fb359e00640 1 --2- 172.21.15.28:0/3971161782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8608b90 0x55aea8610710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.413+0000 7fb359e00640 1 -- 172.21.15.28:0/3971161782 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aea83a8f70 con 0x55aea8613260 2024-04-24T08:17:09.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.413+0000 7fb35ed81740 10 client.0 _ll_register_callbacks cb 0x55aea8515aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:09.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.413+0000 7fb35ed81740 1 -- 172.21.15.28:0/3971161782 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb354128c10 con 0x55aea8613260 2024-04-24T08:17:09.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.413+0000 7fb359400640 1 --2- 172.21.15.28:0/3971161782 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8608270 0x55aea86101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:09.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.413+0000 7fb358a00640 1 --2- 172.21.15.28:0/3971161782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8608b90 0x55aea8610710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:09.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.414+0000 7fb359e00640 1 --2- 172.21.15.28:0/3971161782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8613260 0x55aea8610c50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb354067e40 tx=0x7fb354142230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7354dce543930963 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.414+0000 7fb33aa00640 10 client.26378 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:17:09.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.415+0000 7fb33aa00640 1 -- 172.21.15.28:0/3971161782 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb3541315f0 con 0x55aea8613260 2024-04-24T08:17:09.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.415+0000 7fb33aa00640 1 -- 172.21.15.28:0/3971161782 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fb354128060 con 0x55aea8613260 2024-04-24T08:17:09.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.415+0000 7fb33aa00640 1 -- 172.21.15.28:0/3971161782 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb3541282c0 con 0x55aea8613260 2024-04-24T08:17:09.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.416+0000 7fb35ed81740 10 client.26378 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:09.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.416+0000 7fb35ed81740 1 -- 172.21.15.28:0/3971161782 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({fsmap=26}) -- 0x55aea8240170 con 0x55aea8613260 2024-04-24T08:17:09.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.417+0000 7fb33aa00640 1 -- 172.21.15.28:0/3971161782 <== mon.2 v2:172.21.15.117:3300/0 4 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fb35415ab10 con 0x55aea8613260 2024-04-24T08:17:09.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.417+0000 7fb35ed81740 10 client.26378 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:09.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.417+0000 7fb35ed81740 10 client.26378 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:17:09.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.417+0000 7fb35ed81740 20 client.26378 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:09.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.417+0000 7fb35ed81740 1 --2- 172.21.15.28:0/3971161782 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55aea8622e90 0x55aea8625280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.417+0000 7fb35ed81740 4 client.26378 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:17:09.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.417+0000 7fb35ed81740 1 -- 172.21.15.28:0/3971161782 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55aea857b890 con 0x55aea8622e90 2024-04-24T08:17:09.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.420+0000 7fb359400640 1 --2- 172.21.15.28:0/3971161782 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55aea8622e90 0x55aea8625280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.423+0000 7fb359400640 1 --2- 172.21.15.28:0/3971161782 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55aea8622e90 0x55aea8625280 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ce0900e42a16ef7 server_cookie=b85114d045198006 in_seq=0 out_seq=0 2024-04-24T08:17:09.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.423+0000 7fb33aa00640 10 client.26378 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:09.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.426+0000 7fb33aa00640 1 -- 172.21.15.28:0/3971161782 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55aea857b890 con 0x55aea8622e90 2024-04-24T08:17:09.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.426+0000 7fb33aa00640 10 client.26378 handle_command_reply: tid=0 2024-04-24T08:17:09.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.426+0000 7fb35ed81740 1 client.26378 shutdown 2024-04-24T08:17:09.429 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.427+0000 7fb35ed81740 1 -- 172.21.15.28:0/3971161782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8613260 msgr2=0x55aea8610c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.427+0000 7fb35ed81740 1 --2- 172.21.15.28:0/3971161782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8613260 0x55aea8610c50 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fb354067e40 tx=0x7fb354142230 comp rx=0 tx=0).stop 2024-04-24T08:17:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.427+0000 7fb35ed81740 1 -- 172.21.15.28:0/3971161782 shutdown_connections 2024-04-24T08:17:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.427+0000 7fb35ed81740 1 --2- 172.21.15.28:0/3971161782 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55aea8622e90 0x55aea8625280 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.427+0000 7fb33aa00640 0 client.26378 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.427+0000 7fb35ed81740 1 --2- 172.21.15.28:0/3971161782 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8608b90 0x55aea8610710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.427+0000 7fb35ed81740 1 --2- 172.21.15.28:0/3971161782 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8613260 0x55aea8610c50 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.430 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.427+0000 7fb35ed81740 1 --2- 172.21.15.28:0/3971161782 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8608270 0x55aea86101d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.427+0000 7fb35ed81740 1 -- 172.21.15.28:0/3971161782 >> 172.21.15.28:0/3971161782 conn(0x55aea8604590 msgr2=0x55aea860a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.429+0000 7fb35ed81740 1 -- 172.21.15.28:0/3971161782 shutdown_connections 2024-04-24T08:17:09.431 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.429+0000 7fb35ed81740 1 -- 172.21.15.28:0/3971161782 wait complete. 2024-04-24T08:17:09.432 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.430+0000 7fb35ed81740 20 client.26378 trim_cache size 0 max 16384 2024-04-24T08:17:09.479 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.477+0000 7fb35ed81740 1 Processor -- start 2024-04-24T08:17:09.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.479+0000 7fb35ed81740 1 -- start start 2024-04-24T08:17:09.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.479+0000 7fb35ed81740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608270 0x55aea861eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.479+0000 7fb35ed81740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8608b90 0x55aea8610a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.481 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.479+0000 7fb35ed81740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8613260 0x55aea860a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.479+0000 7fb359400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608270 0x55aea861eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.480+0000 7fb359400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608270 0x55aea861eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33234/0 (socket says 172.21.15.28:33234) 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.480+0000 7fb35ed81740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb34c1530b0 con 0x55aea8613260 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.480+0000 7fb359400640 1 -- 172.21.15.28:0/1046844698 learned_addr learned my addr 172.21.15.28:0/1046844698 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.480+0000 7fb35ed81740 1 -- 172.21.15.28:0/1046844698 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb34c150980 con 0x55aea8608270 2024-04-24T08:17:09.482 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.480+0000 7fb35ed81740 1 -- 172.21.15.28:0/1046844698 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb34c14e7d0 con 0x55aea8608b90 2024-04-24T08:17:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.480+0000 7fb358a00640 1 --2- 172.21.15.28:0/1046844698 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8608b90 0x55aea8610a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.480+0000 7fb359e00640 1 --2- 172.21.15.28:0/1046844698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8613260 0x55aea860a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.481+0000 7fb359400640 1 -- 172.21.15.28:0/1046844698 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8608b90 msgr2=0x55aea8610a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.481+0000 7fb359400640 1 --2- 172.21.15.28:0/1046844698 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8608b90 0x55aea8610a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.481+0000 7fb359400640 1 -- 172.21.15.28:0/1046844698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8613260 msgr2=0x55aea860a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.481+0000 7fb359400640 1 --2- 172.21.15.28:0/1046844698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8613260 0x55aea860a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.483 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.481+0000 7fb359400640 1 -- 172.21.15.28:0/1046844698 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55aea860f180 con 0x55aea8608270 2024-04-24T08:17:09.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.481+0000 7fb358a00640 1 --2- 172.21.15.28:0/1046844698 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8608b90 0x55aea8610a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:17:09.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.481+0000 7fb359400640 1 --2- 172.21.15.28:0/1046844698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608270 0x55aea861eb80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb348009940 tx=0x7fb348009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54c9fd1adee95860 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.483+0000 7fb345c00640 1 -- 172.21.15.28:0/1046844698 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348003e00 con 0x55aea8608270 2024-04-24T08:17:09.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.483+0000 7fb345c00640 1 -- 172.21.15.28:0/1046844698 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb348003fa0 con 0x55aea8608270 2024-04-24T08:17:09.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.484+0000 7fb35ed81740 1 -- 172.21.15.28:0/1046844698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608270 msgr2=0x55aea861eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.484+0000 7fb35ed81740 1 --2- 172.21.15.28:0/1046844698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608270 0x55aea861eb80 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fb348009940 tx=0x7fb348009ed0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.483+0000 7fb345c00640 1 -- 172.21.15.28:0/1046844698 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb348010430 con 0x55aea8608270 2024-04-24T08:17:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.485+0000 7fb35ed81740 1 -- 172.21.15.28:0/1046844698 shutdown_connections 2024-04-24T08:17:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.485+0000 7fb35ed81740 1 --2- 172.21.15.28:0/1046844698 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8613260 0x55aea860a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.485+0000 7fb35ed81740 1 --2- 172.21.15.28:0/1046844698 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8608b90 0x55aea8610a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.485+0000 7fb35ed81740 1 --2- 172.21.15.28:0/1046844698 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608270 0x55aea861eb80 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.485+0000 7fb35ed81740 1 -- 172.21.15.28:0/1046844698 >> 172.21.15.28:0/1046844698 conn(0x55aea8604590 msgr2=0x55aea86099c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.485+0000 7fb35ed81740 1 -- 172.21.15.28:0/1046844698 shutdown_connections 2024-04-24T08:17:09.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.485+0000 7fb35ed81740 1 -- 172.21.15.28:0/1046844698 wait complete. 2024-04-24T08:17:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.486+0000 7fb35ed81740 1 -- start start 2024-04-24T08:17:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.486+0000 7fb35ed81740 1 Processor -- start 2024-04-24T08:17:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.488+0000 7fb35ed81740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8608270 0x55aea860d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.488+0000 7fb35ed81740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608b90 0x55aea860d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.488+0000 7fb35ed81740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8613260 0x55aea86883d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.488+0000 7fb359400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8608270 0x55aea860d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.489+0000 7fb359400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8608270 0x55aea860d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52886/0 (socket says 172.21.15.28:52886) 2024-04-24T08:17:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.489+0000 7fb358a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608b90 0x55aea860d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.489+0000 7fb359e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8613260 0x55aea86883d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.489+0000 7fb359e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8613260 0x55aea86883d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:45678/0 (socket says 172.21.15.28:45678) 2024-04-24T08:17:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.489+0000 7fb358a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608b90 0x55aea860d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33240/0 (socket says 172.21.15.28:33240) 2024-04-24T08:17:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.489+0000 7fb359400640 1 -- 172.21.15.28:0/2198305845 learned_addr learned my addr 172.21.15.28:0/2198305845 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:09.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.489+0000 7fb35ed81740 1 -- 172.21.15.28:0/2198305845 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb34c14ee40 con 0x55aea8608270 2024-04-24T08:17:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.489+0000 7fb35ed81740 1 -- 172.21.15.28:0/2198305845 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb34c1502b0 con 0x55aea8608b90 2024-04-24T08:17:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.489+0000 7fb35ed81740 1 -- 172.21.15.28:0/2198305845 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb34c14f510 con 0x55aea8613260 2024-04-24T08:17:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.489+0000 7fb35ed81740 10 client.0 _ll_register_callbacks cb 0x7fb3480096e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.490+0000 7fb358a00640 1 -- 172.21.15.28:0/2198305845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8613260 msgr2=0x55aea86883d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.490+0000 7fb358a00640 1 --2- 172.21.15.28:0/2198305845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8613260 0x55aea86883d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.490+0000 7fb358a00640 1 -- 172.21.15.28:0/2198305845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8608270 msgr2=0x55aea860d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.490+0000 7fb358a00640 1 --2- 172.21.15.28:0/2198305845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8608270 0x55aea860d2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.496 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.490+0000 7fb358a00640 1 -- 172.21.15.28:0/2198305845 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55aea8688910 con 0x55aea8608b90 2024-04-24T08:17:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.491+0000 7fb358a00640 1 --2- 172.21.15.28:0/2198305845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608b90 0x55aea860d7e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x55aea860fa30 tx=0x7fb340004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f57e97dd3e520c7b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.491+0000 7fb33aa00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:17:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.492+0000 7fb33aa00640 1 -- 172.21.15.28:0/2198305845 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb34001fce0 con 0x55aea8608b90 2024-04-24T08:17:09.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.492+0000 7fb33aa00640 1 -- 172.21.15.28:0/2198305845 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fb3400203f0 con 0x55aea8608b90 2024-04-24T08:17:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.492+0000 7fb35ed81740 1 -- 172.21.15.28:0/2198305845 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55aea8575700 con 0x55aea8608b90 2024-04-24T08:17:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.492+0000 7fb33aa00640 1 -- 172.21.15.28:0/2198305845 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb34001fe80 con 0x55aea8608b90 2024-04-24T08:17:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.493+0000 7fb35ed81740 10 client.26384 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.493+0000 7fb35ed81740 1 -- 172.21.15.28:0/2198305845 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55aea8689c70 con 0x55aea8608b90 2024-04-24T08:17:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.493+0000 7fb33aa00640 1 -- 172.21.15.28:0/2198305845 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55aea8575700 con 0x55aea8608b90 2024-04-24T08:17:09.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.493+0000 7fb33aa00640 1 -- 172.21.15.28:0/2198305845 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fb34002b6f0 con 0x55aea8608b90 2024-04-24T08:17:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.494+0000 7fb35ed81740 10 client.26384 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.495+0000 7fb35ed81740 10 client.26384 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:17:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.495+0000 7fb35ed81740 20 client.26384 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.495+0000 7fb35ed81740 1 --2- 172.21.15.28:0/2198305845 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55aea868a320 0x55aea868c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.495+0000 7fb35ed81740 4 client.26384 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:17:09.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.495+0000 7fb35ed81740 1 -- 172.21.15.28:0/2198305845 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55aea84f1980 con 0x55aea868a320 2024-04-24T08:17:09.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.517+0000 7fb359400640 1 --2- 172.21.15.28:0/2198305845 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55aea868a320 0x55aea868c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.518+0000 7fb359400640 1 --2- 172.21.15.28:0/2198305845 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55aea868a320 0x55aea868c710 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c972a69be66f008 server_cookie=d5e63a8ba485bab8 in_seq=0 out_seq=0 2024-04-24T08:17:09.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.518+0000 7fb33aa00640 10 client.26384 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:09.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.522+0000 7fb33aa00640 1 -- 172.21.15.28:0/2198305845 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55aea84f1980 con 0x55aea868a320 2024-04-24T08:17:09.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.522+0000 7fb33aa00640 10 client.26384 handle_command_reply: tid=0 2024-04-24T08:17:09.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.523+0000 7fb35ed81740 1 client.26384 shutdown 2024-04-24T08:17:09.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.525+0000 7fb35ed81740 1 -- 172.21.15.28:0/2198305845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608b90 msgr2=0x55aea860d7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.525+0000 7fb35ed81740 1 --2- 172.21.15.28:0/2198305845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608b90 0x55aea860d7e0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x55aea860fa30 tx=0x7fb340004730 comp rx=0 tx=0).stop 2024-04-24T08:17:09.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.525+0000 7fb35ed81740 1 -- 172.21.15.28:0/2198305845 shutdown_connections 2024-04-24T08:17:09.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.525+0000 7fb35ed81740 1 --2- 172.21.15.28:0/2198305845 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55aea868a320 0x55aea868c710 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.525+0000 7fb35ed81740 1 --2- 172.21.15.28:0/2198305845 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55aea8613260 0x55aea86883d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.525+0000 7fb35ed81740 1 --2- 172.21.15.28:0/2198305845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55aea8608b90 0x55aea860d7e0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.525+0000 7fb35ed81740 1 --2- 172.21.15.28:0/2198305845 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55aea8608270 0x55aea860d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.525+0000 7fb35ed81740 1 -- 172.21.15.28:0/2198305845 >> 172.21.15.28:0/2198305845 conn(0x55aea8604590 msgr2=0x55aea860a250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.525+0000 7fb33aa00640 0 client.26384 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:09.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.526+0000 7fb35ed81740 1 -- 172.21.15.28:0/2198305845 shutdown_connections 2024-04-24T08:17:09.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.526+0000 7fb35ed81740 1 -- 172.21.15.28:0/2198305845 wait complete. 2024-04-24T08:17:09.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.526+0000 7fb35ed81740 20 client.26384 trim_cache size 0 max 16384 2024-04-24T08:17:09.530 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:17:09.530 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-24T08:17:09.530 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "d52a19ca-6482-49d7-a4d1-d1bcd5ceb884", 2024-04-24T08:17:09.530 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-24T08:17:09.530 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:17:09.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.527+0000 7fb35a800640 1 -- 172.21.15.28:0/3532960909 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb33c0a3ea0 msgr2=0x7fb33c0a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.527+0000 7fb35a800640 1 --2- 172.21.15.28:0/3532960909 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb33c0a3ea0 0x7fb33c0a6360 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fb340005fd0 tx=0x7fb34000b7d0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.527+0000 7fb35a800640 1 -- 172.21.15.28:0/3532960909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34c09ace0 msgr2=0x7fb34c0afef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.527+0000 7fb35a800640 1 --2- 172.21.15.28:0/3532960909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34c09ace0 0x7fb34c0afef0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fb354065520 tx=0x7fb3540686e0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.527+0000 7fb35a800640 1 -- 172.21.15.28:0/3532960909 shutdown_connections 2024-04-24T08:17:09.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.527+0000 7fb35a800640 1 --2- 172.21.15.28:0/3532960909 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb33c0a3ea0 0x7fb33c0a6360 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.527+0000 7fb35a800640 1 --2- 172.21.15.28:0/3532960909 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb34c0968e0 0x7fb34c0aa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.527+0000 7fb35a800640 1 --2- 172.21.15.28:0/3532960909 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb34c09ace0 0x7fb34c0afef0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.527+0000 7fb35a800640 1 --2- 172.21.15.28:0/3532960909 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb34c095f20 0x7fb34c0aa1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.527+0000 7fb35a800640 1 -- 172.21.15.28:0/3532960909 >> 172.21.15.28:0/3532960909 conn(0x7fb34c00cb30 msgr2=0x7fb34c093960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.529+0000 7fb35a800640 1 -- 172.21.15.28:0/3532960909 shutdown_connections 2024-04-24T08:17:09.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.530+0000 7fb35a800640 1 -- 172.21.15.28:0/3532960909 wait complete. 2024-04-24T08:17:09.563 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:17:09.563 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:17:09.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.880+0000 7fca1b000640 1 -- 172.21.15.28:0/301410208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca14104cc0 msgr2=0x7fca14108d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.880+0000 7fca1b000640 1 --2- 172.21.15.28:0/301410208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca14104cc0 0x7fca14108d00 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fca00009860 tx=0x7fca0002a340 comp rx=0 tx=0).stop 2024-04-24T08:17:09.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.880+0000 7fca1b000640 1 -- 172.21.15.28:0/301410208 shutdown_connections 2024-04-24T08:17:09.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.880+0000 7fca1b000640 1 --2- 172.21.15.28:0/301410208 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca141093e0 0x7fca14110fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.880+0000 7fca1b000640 1 --2- 172.21.15.28:0/301410208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca14104cc0 0x7fca14108d00 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.880+0000 7fca1b000640 1 --2- 172.21.15.28:0/301410208 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca14104310 0x7fca141046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.880+0000 7fca1b000640 1 -- 172.21.15.28:0/301410208 >> 172.21.15.28:0/301410208 conn(0x7fca140ffb70 msgr2=0x7fca14101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.882+0000 7fca1b000640 1 -- 172.21.15.28:0/301410208 shutdown_connections 2024-04-24T08:17:09.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.882+0000 7fca1b000640 1 -- 172.21.15.28:0/301410208 wait complete. 2024-04-24T08:17:09.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.883+0000 7fca1b000640 1 Processor -- start 2024-04-24T08:17:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.884+0000 7fca1b000640 1 -- start start 2024-04-24T08:17:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.884+0000 7fca1b000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca14104310 0x7fca14110f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.884+0000 7fca1b000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca141093e0 0x7fca1410dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.884+0000 7fca1b000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca1410e530 0x7fca1410e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.884+0000 7fca1b000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fca14114470 con 0x7fca1410e530 2024-04-24T08:17:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.884+0000 7fca1b000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fca141142f0 con 0x7fca14104310 2024-04-24T08:17:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.884+0000 7fca1b000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fca141145f0 con 0x7fca141093e0 2024-04-24T08:17:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.885+0000 7fca19200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca141093e0 0x7fca1410dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.885+0000 7fca19200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca141093e0 0x7fca1410dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:48276/0 (socket says 172.21.15.28:48276) 2024-04-24T08:17:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.885+0000 7fca19200640 1 -- 172.21.15.28:0/3819198384 learned_addr learned my addr 172.21.15.28:0/3819198384 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.885+0000 7fca1a600640 1 --2- 172.21.15.28:0/3819198384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca1410e530 0x7fca1410e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.885+0000 7fca19200640 1 -- 172.21.15.28:0/3819198384 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca14104310 msgr2=0x7fca14110f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.885+0000 7fca19200640 1 --2- 172.21.15.28:0/3819198384 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca14104310 0x7fca14110f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.885+0000 7fca19200640 1 -- 172.21.15.28:0/3819198384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca1410e530 msgr2=0x7fca1410e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.885+0000 7fca19200640 1 --2- 172.21.15.28:0/3819198384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca1410e530 0x7fca1410e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.885+0000 7fca19200640 1 -- 172.21.15.28:0/3819198384 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fca141ab440 con 0x7fca141093e0 2024-04-24T08:17:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.886+0000 7fca19200640 1 --2- 172.21.15.28:0/3819198384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca141093e0 0x7fca1410dff0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fca00000c00 tx=0x7fca00033d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.887+0000 7fca12000640 1 -- 172.21.15.28:0/3819198384 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0003f020 con 0x7fca141093e0 2024-04-24T08:17:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.887+0000 7fca12000640 1 -- 172.21.15.28:0/3819198384 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca00032770 con 0x7fca141093e0 2024-04-24T08:17:09.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.887+0000 7fca12000640 1 -- 172.21.15.28:0/3819198384 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca00037040 con 0x7fca141093e0 2024-04-24T08:17:09.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.887+0000 7fca1b000640 1 -- 172.21.15.28:0/3819198384 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fca141ab6d0 con 0x7fca141093e0 2024-04-24T08:17:09.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.887+0000 7fca1b000640 1 -- 172.21.15.28:0/3819198384 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fca141abb70 con 0x7fca141093e0 2024-04-24T08:17:09.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.891+0000 7fca1f38b740 1 Processor -- start 2024-04-24T08:17:09.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.893+0000 7fca12000640 1 -- 172.21.15.28:0/3819198384 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fca00042020 con 0x7fca141093e0 2024-04-24T08:17:09.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.893+0000 7fca1f38b740 1 -- start start 2024-04-24T08:17:09.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.894+0000 7fca1f38b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b43a50 0x561bc2b43e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.895+0000 7fca1f38b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b44370 0x561bc2b4e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.895+0000 7fca1f38b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b4ea40 0x561bc2b50e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.895+0000 7fca1f38b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561bc2aa91f0 con 0x561bc2b43a50 2024-04-24T08:17:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.895+0000 7fca1f38b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561bc2a4f350 con 0x561bc2b4ea40 2024-04-24T08:17:09.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.895+0000 7fca1f38b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561bc2aa51b0 con 0x561bc2b44370 2024-04-24T08:17:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.895+0000 7fca12000640 1 --2- 172.21.15.28:0/3819198384 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc9fc0a3e70 0x7fc9fc0a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.896+0000 7fca12000640 1 -- 172.21.15.28:0/3819198384 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fca000f0810 con 0x7fca141093e0 2024-04-24T08:17:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.896+0000 7fca19c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b44370 0x561bc2b4e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.897+0000 7fca19c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b44370 0x561bc2b4e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:48290/0 (socket says 172.21.15.28:48290) 2024-04-24T08:17:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.897+0000 7fca19c00640 1 -- 172.21.15.28:0/1554694893 learned_addr learned my addr 172.21.15.28:0/1554694893 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:09.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.897+0000 7fca1a600640 1 --2- 172.21.15.28:0/1554694893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b43a50 0x561bc2b43e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.898+0000 7fca19c00640 1 -- 172.21.15.28:0/1554694893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b4ea40 msgr2=0x561bc2b50e30 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-04-24T08:17:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.898+0000 7fca19c00640 1 --2- 172.21.15.28:0/1554694893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b4ea40 0x561bc2b50e30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.898+0000 7fca19c00640 1 -- 172.21.15.28:0/1554694893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b43a50 msgr2=0x561bc2b43e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.898+0000 7fca19c00640 1 --2- 172.21.15.28:0/1554694893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b43a50 0x561bc2b43e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.898+0000 7fca19c00640 1 -- 172.21.15.28:0/1554694893 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bc2aa3f50 con 0x561bc2b44370 2024-04-24T08:17:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.899+0000 7fca1a600640 1 --2- 172.21.15.28:0/3819198384 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc9fc0a3e70 0x7fc9fc0a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.899+0000 7fca19c00640 1 --2- 172.21.15.28:0/1554694893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b44370 0x561bc2b4e500 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fca08009b30 tx=0x7fca0800e350 comp rx=0 tx=0).ready entity=mon.1 client_cookie=81c55b3380ba080b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.900+0000 7fca06a00640 1 -- 172.21.15.28:0/1554694893 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0800ed80 con 0x561bc2b44370 2024-04-24T08:17:09.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.901+0000 7fca06a00640 1 -- 172.21.15.28:0/1554694893 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca08005500 con 0x561bc2b44370 2024-04-24T08:17:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.902+0000 7fca1f38b740 1 -- 172.21.15.28:0/1554694893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b44370 msgr2=0x561bc2b4e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.902+0000 7fca1f38b740 1 --2- 172.21.15.28:0/1554694893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b44370 0x561bc2b4e500 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fca08009b30 tx=0x7fca0800e350 comp rx=0 tx=0).stop 2024-04-24T08:17:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.902+0000 7fca1a600640 1 --2- 172.21.15.28:0/3819198384 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc9fc0a3e70 0x7fc9fc0a6330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fca0c005fd0 tx=0x7fca0c004ab0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.902+0000 7fca1f38b740 1 -- 172.21.15.28:0/1554694893 shutdown_connections 2024-04-24T08:17:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.902+0000 7fca1f38b740 1 --2- 172.21.15.28:0/1554694893 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b4ea40 0x561bc2b50e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.903+0000 7fca1f38b740 1 --2- 172.21.15.28:0/1554694893 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b44370 0x561bc2b4e500 secure :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fca08009b30 tx=0x7fca0800e350 comp rx=0 tx=0).stop 2024-04-24T08:17:09.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.903+0000 7fca1f38b740 1 --2- 172.21.15.28:0/1554694893 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b43a50 0x561bc2b43e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.903+0000 7fca1f38b740 1 -- 172.21.15.28:0/1554694893 >> 172.21.15.28:0/1554694893 conn(0x561bc2b3fd70 msgr2=0x561bc2b40150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.904+0000 7fca1f38b740 1 -- 172.21.15.28:0/1554694893 shutdown_connections 2024-04-24T08:17:09.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.905+0000 7fca1f38b740 1 -- 172.21.15.28:0/1554694893 wait complete. 2024-04-24T08:17:09.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.906+0000 7fca1f38b740 1 -- start start 2024-04-24T08:17:09.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.907+0000 7fca1f38b740 1 Processor -- start 2024-04-24T08:17:09.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.908+0000 7fca1f38b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b43a50 0x561bc2b4bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.908+0000 7fca1f38b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b4ea40 0x561bc2b4c1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.908+0000 7fca1f38b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b4c880 0x561bc2b5ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.908+0000 7fca1f38b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561bc2b518d0 con 0x561bc2b4c880 2024-04-24T08:17:09.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.908+0000 7fca1f38b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561bc2a62690 con 0x561bc2b4ea40 2024-04-24T08:17:09.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.908+0000 7fca1f38b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561bc2b51a50 con 0x561bc2b43a50 2024-04-24T08:17:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.908+0000 7fca1f38b740 10 client.0 _ll_register_callbacks cb 0x561bc2a9e650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.909+0000 7fca1a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b4c880 0x561bc2b5ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.909+0000 7fca1a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b4c880 0x561bc2b5ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:50468/0 (socket says 172.21.15.28:50468) 2024-04-24T08:17:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.909+0000 7fca1a600640 1 -- 172.21.15.28:0/4261003740 learned_addr learned my addr 172.21.15.28:0/4261003740 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:09.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.910+0000 7fca19200640 1 --2- 172.21.15.28:0/4261003740 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b4ea40 0x561bc2b4c1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.910+0000 7fca19200640 1 -- 172.21.15.28:0/4261003740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b43a50 msgr2=0x561bc2b4bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.910+0000 7fca19200640 1 --2- 172.21.15.28:0/4261003740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b43a50 0x561bc2b4bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.910+0000 7fca19200640 1 -- 172.21.15.28:0/4261003740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b4c880 msgr2=0x561bc2b5ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.910+0000 7fca19200640 1 --2- 172.21.15.28:0/4261003740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b4c880 0x561bc2b5ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.910+0000 7fca19200640 1 -- 172.21.15.28:0/4261003740 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bc2a61920 con 0x561bc2b4ea40 2024-04-24T08:17:09.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.911+0000 7fca1a600640 1 --2- 172.21.15.28:0/4261003740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b4c880 0x561bc2b5ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:17:09.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.911+0000 7fca19200640 1 --2- 172.21.15.28:0/4261003740 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b4ea40 0x561bc2b4c1e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fca000ee050 tx=0x7fca000bc3e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df270e9e7dc8cab4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.912+0000 7fc9fb400640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:17:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.912+0000 7fc9fb400640 1 -- 172.21.15.28:0/4261003740 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca000bc8a0 con 0x561bc2b4ea40 2024-04-24T08:17:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.912+0000 7fc9fb400640 1 -- 172.21.15.28:0/4261003740 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fca000bcb00 con 0x561bc2b4ea40 2024-04-24T08:17:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.912+0000 7fca1f38b740 1 -- 172.21.15.28:0/4261003740 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561bc2ab6c10 con 0x561bc2b4ea40 2024-04-24T08:17:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.912+0000 7fc9fb400640 1 -- 172.21.15.28:0/4261003740 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fca000c0c50 con 0x561bc2b4ea40 2024-04-24T08:17:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.912+0000 7fca1f38b740 10 client.26390 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.912+0000 7fca1f38b740 1 -- 172.21.15.28:0/4261003740 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561bc28857d0 con 0x561bc2b4ea40 2024-04-24T08:17:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.913+0000 7fc9fb400640 1 -- 172.21.15.28:0/4261003740 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561bc2ab6c10 con 0x561bc2b4ea40 2024-04-24T08:17:09.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.913+0000 7fc9fb400640 1 -- 172.21.15.28:0/4261003740 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fca0003a030 con 0x561bc2b4ea40 2024-04-24T08:17:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.913+0000 7fca1f38b740 10 client.26390 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.913+0000 7fca1f38b740 10 client.26390 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:17:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.913+0000 7fca1f38b740 20 client.26390 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.913+0000 7fca1f38b740 1 --2- 172.21.15.28:0/4261003740 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x561bc2b705b0 0x561bc2b72a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.913+0000 7fca1f38b740 4 client.26390 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:17:09.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.913+0000 7fca1f38b740 1 -- 172.21.15.28:0/4261003740 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561bc2ab0a80 con 0x561bc2b705b0 2024-04-24T08:17:09.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.919+0000 7fca19c00640 1 --2- 172.21.15.28:0/4261003740 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x561bc2b705b0 0x561bc2b72a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.922+0000 7fca19c00640 1 --2- 172.21.15.28:0/4261003740 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x561bc2b705b0 0x561bc2b72a70 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ff2816a345505d6 server_cookie=ae6bd66499e426a7 in_seq=0 out_seq=0 2024-04-24T08:17:09.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.922+0000 7fc9fb400640 10 client.26390 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.923+0000 7fc9fb400640 1 -- 172.21.15.28:0/4261003740 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561bc2ab0a80 con 0x561bc2b705b0 2024-04-24T08:17:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.923+0000 7fc9fb400640 10 client.26390 handle_command_reply: tid=0 2024-04-24T08:17:09.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.923+0000 7fca1f38b740 1 client.26390 shutdown 2024-04-24T08:17:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.923+0000 7fca1f38b740 1 -- 172.21.15.28:0/4261003740 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b4ea40 msgr2=0x561bc2b4c1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.923+0000 7fca1f38b740 1 --2- 172.21.15.28:0/4261003740 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b4ea40 0x561bc2b4c1e0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fca000ee050 tx=0x7fca000bc3e0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.925+0000 7fca1f38b740 1 -- 172.21.15.28:0/4261003740 shutdown_connections 2024-04-24T08:17:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.925+0000 7fca1f38b740 1 --2- 172.21.15.28:0/4261003740 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x561bc2b705b0 0x561bc2b72a70 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.925+0000 7fca1f38b740 1 --2- 172.21.15.28:0/4261003740 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b4c880 0x561bc2b5ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.925+0000 7fca1f38b740 1 --2- 172.21.15.28:0/4261003740 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b4ea40 0x561bc2b4c1e0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.925+0000 7fca1f38b740 1 --2- 172.21.15.28:0/4261003740 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b43a50 0x561bc2b4bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.925+0000 7fca1f38b740 1 -- 172.21.15.28:0/4261003740 >> 172.21.15.28:0/4261003740 conn(0x561bc2b3fd70 msgr2=0x561bc2b45b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.925+0000 7fc9fb400640 0 client.26390 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.927+0000 7fca1f38b740 1 -- 172.21.15.28:0/4261003740 shutdown_connections 2024-04-24T08:17:09.929 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.928+0000 7fca1f38b740 1 -- 172.21.15.28:0/4261003740 wait complete. 2024-04-24T08:17:09.930 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.928+0000 7fca1f38b740 20 client.26390 trim_cache size 0 max 16384 2024-04-24T08:17:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.977+0000 7fca1f38b740 1 Processor -- start 2024-04-24T08:17:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.978+0000 7fca1f38b740 1 -- start start 2024-04-24T08:17:09.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.978+0000 7fca1f38b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b3fd70 0x561bc2b3f750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.978+0000 7fca1f38b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b43a50 0x561bc2b78380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.978+0000 7fca1f38b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b4c880 0x561bc2b788c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.978+0000 7fca1f38b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fca141ae2e0 con 0x561bc2b4c880 2024-04-24T08:17:09.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.978+0000 7fca1f38b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fca141a1f20 con 0x561bc2b43a50 2024-04-24T08:17:09.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.978+0000 7fca1f38b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fca1419fd70 con 0x561bc2b3fd70 2024-04-24T08:17:09.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.979+0000 7fca1a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b4c880 0x561bc2b788c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.979+0000 7fca19c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b3fd70 0x561bc2b3f750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.979+0000 7fca19200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b43a50 0x561bc2b78380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.979+0000 7fca19200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b43a50 0x561bc2b78380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:56260/0 (socket says 172.21.15.28:56260) 2024-04-24T08:17:09.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.979+0000 7fca19200640 1 -- 172.21.15.28:0/3371819103 learned_addr learned my addr 172.21.15.28:0/3371819103 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:09.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.980+0000 7fca19200640 1 -- 172.21.15.28:0/3371819103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b3fd70 msgr2=0x561bc2b3f750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.980+0000 7fca19200640 1 --2- 172.21.15.28:0/3371819103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b3fd70 0x561bc2b3f750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.980+0000 7fca19200640 1 -- 172.21.15.28:0/3371819103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b4c880 msgr2=0x561bc2b788c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.980+0000 7fca19200640 1 --2- 172.21.15.28:0/3371819103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b4c880 0x561bc2b788c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.980+0000 7fca19200640 1 -- 172.21.15.28:0/3371819103 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561bc27bb3f0 con 0x561bc2b43a50 2024-04-24T08:17:09.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.980+0000 7fca1a600640 1 --2- 172.21.15.28:0/3371819103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b4c880 0x561bc2b788c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:09.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.981+0000 7fca19200640 1 --2- 172.21.15.28:0/3371819103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b43a50 0x561bc2b78380 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fca00033fb0 tx=0x7fca00005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7180e8d7c66b931c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.983+0000 7fca06000640 1 -- 172.21.15.28:0/3371819103 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca000ca080 con 0x561bc2b43a50 2024-04-24T08:17:09.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.983+0000 7fca06000640 1 -- 172.21.15.28:0/3371819103 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca000c5510 con 0x561bc2b43a50 2024-04-24T08:17:09.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.983+0000 7fca06000640 1 -- 172.21.15.28:0/3371819103 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca00003c60 con 0x561bc2b43a50 2024-04-24T08:17:09.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.984+0000 7fca1f38b740 1 -- 172.21.15.28:0/3371819103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b43a50 msgr2=0x561bc2b78380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:09.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.984+0000 7fca1f38b740 1 --2- 172.21.15.28:0/3371819103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b43a50 0x561bc2b78380 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fca00033fb0 tx=0x7fca00005f90 comp rx=0 tx=0).stop 2024-04-24T08:17:09.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.984+0000 7fca1f38b740 1 -- 172.21.15.28:0/3371819103 shutdown_connections 2024-04-24T08:17:09.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.984+0000 7fca1f38b740 1 --2- 172.21.15.28:0/3371819103 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b4c880 0x561bc2b788c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.984+0000 7fca1f38b740 1 --2- 172.21.15.28:0/3371819103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b43a50 0x561bc2b78380 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.984+0000 7fca1f38b740 1 --2- 172.21.15.28:0/3371819103 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b3fd70 0x561bc2b3f750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.984+0000 7fca1f38b740 1 -- 172.21.15.28:0/3371819103 >> 172.21.15.28:0/3371819103 conn(0x561bc2b4c340 msgr2=0x561bc2b4a5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:09.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.985+0000 7fca1f38b740 1 -- 172.21.15.28:0/3371819103 shutdown_connections 2024-04-24T08:17:09.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.985+0000 7fca1f38b740 1 -- 172.21.15.28:0/3371819103 wait complete. 2024-04-24T08:17:09.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.986+0000 7fca1f38b740 1 -- start start 2024-04-24T08:17:09.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.986+0000 7fca1f38b740 1 Processor -- start 2024-04-24T08:17:09.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.987+0000 7fca1f38b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b3fd70 0x561bc2b56cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.987+0000 7fca1f38b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b43a50 0x561bc2b57200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.989 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.987+0000 7fca1f38b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b4c880 0x561bc2b47f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:09.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.988+0000 7fca19c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b3fd70 0x561bc2b56cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.988+0000 7fca1a600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b4c880 0x561bc2b47f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:09.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.988+0000 7fca19c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b3fd70 0x561bc2b56cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:48332/0 (socket says 172.21.15.28:48332) 2024-04-24T08:17:09.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.988+0000 7fca19c00640 1 -- 172.21.15.28:0/329617841 learned_addr learned my addr 172.21.15.28:0/329617841 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:09.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.988+0000 7fca1f38b740 1 -- 172.21.15.28:0/329617841 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fca141a03e0 con 0x561bc2b43a50 2024-04-24T08:17:09.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.988+0000 7fca1f38b740 1 -- 172.21.15.28:0/329617841 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fca141a1850 con 0x561bc2b4c880 2024-04-24T08:17:09.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.988+0000 7fca1f38b740 1 -- 172.21.15.28:0/329617841 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fca141a0ab0 con 0x561bc2b3fd70 2024-04-24T08:17:09.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.989+0000 7fca1f38b740 10 client.0 _ll_register_callbacks cb 0x7fca00033f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:09.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.990+0000 7fca1a600640 1 -- 172.21.15.28:0/329617841 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b3fd70 msgr2=0x561bc2b56cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.993+0000 7fca1a600640 1 --2- 172.21.15.28:0/329617841 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b3fd70 0x561bc2b56cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.993+0000 7fca1a600640 1 -- 172.21.15.28:0/329617841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b43a50 msgr2=0x561bc2b57200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:09.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.993+0000 7fca1a600640 1 --2- 172.21.15.28:0/329617841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b43a50 0x561bc2b57200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:09.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.993+0000 7fca1a600640 1 -- 172.21.15.28:0/329617841 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561bc277b0f0 con 0x561bc2b4c880 2024-04-24T08:17:09.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.993+0000 7fca19c00640 1 --2- 172.21.15.28:0/329617841 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b3fd70 0x561bc2b56cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:17:09.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.994+0000 7fc9fb400640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:17:09.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.994+0000 7fca1a600640 1 --2- 172.21.15.28:0/329617841 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b4c880 0x561bc2b47f50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fca0c012eb0 tx=0x7fca0c004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2a158bf130d77521 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:09.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.995+0000 7fc9fb400640 1 -- 172.21.15.28:0/329617841 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca0c018070 con 0x561bc2b4c880 2024-04-24T08:17:09.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.995+0000 7fca1f38b740 1 -- 172.21.15.28:0/329617841 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561bc2a2c360 con 0x561bc2b4c880 2024-04-24T08:17:09.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.995+0000 7fc9fb400640 1 -- 172.21.15.28:0/329617841 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 8556+0+0 (secure 0 0 0) 0x7fca0c004b20 con 0x561bc2b4c880 2024-04-24T08:17:09.997 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.996+0000 7fc9fb400640 1 -- 172.21.15.28:0/329617841 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fca0c018070 con 0x561bc2b4c880 2024-04-24T08:17:09.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.997+0000 7fca1f38b740 10 client.26396 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:09.999 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.997+0000 7fca1f38b740 1 -- 172.21.15.28:0/329617841 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561bc2a6e810 con 0x561bc2b4c880 2024-04-24T08:17:10.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:09.999+0000 7fc9fb400640 1 -- 172.21.15.28:0/329617841 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561bc2a2c360 con 0x561bc2b4c880 2024-04-24T08:17:10.005 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.004+0000 7fc9fb400640 1 -- 172.21.15.28:0/329617841 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fca0c029030 con 0x561bc2b4c880 2024-04-24T08:17:10.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.004+0000 7fca1f38b740 10 client.26396 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:10.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.004+0000 7fca1f38b740 10 client.26396 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:17:10.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.004+0000 7fca1f38b740 20 client.26396 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:10.006 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.004+0000 7fca1f38b740 1 --2- 172.21.15.28:0/329617841 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x561bc2b4ea40 0x561bc2bd8800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:10.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.004+0000 7fca1f38b740 4 client.26396 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:17:10.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.004+0000 7fca1f38b740 1 -- 172.21.15.28:0/329617841 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x561bc2a79a10 con 0x561bc2b4ea40 2024-04-24T08:17:10.008 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.007+0000 7fca19c00640 1 --2- 172.21.15.28:0/329617841 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x561bc2b4ea40 0x561bc2bd8800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:10.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.009+0000 7fca19c00640 1 --2- 172.21.15.28:0/329617841 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x561bc2b4ea40 0x561bc2bd8800 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=759037f7c855d686 server_cookie=11f51dd01f1c6232 in_seq=0 out_seq=0 2024-04-24T08:17:10.011 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.009+0000 7fc9fb400640 10 client.26396 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:10.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.011+0000 7fc9fb400640 1 -- 172.21.15.28:0/329617841 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+266 (crc 0 0 0) 0x561bc2a79a10 con 0x561bc2b4ea40 2024-04-24T08:17:10.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.011+0000 7fc9fb400640 10 client.26396 handle_command_reply: tid=0 2024-04-24T08:17:10.013 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.011+0000 7fca1f38b740 1 client.26396 shutdown 2024-04-24T08:17:10.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.012+0000 7fca1f38b740 1 -- 172.21.15.28:0/329617841 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b4c880 msgr2=0x561bc2b47f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:10.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.012+0000 7fca1f38b740 1 --2- 172.21.15.28:0/329617841 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b4c880 0x561bc2b47f50 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fca0c012eb0 tx=0x7fca0c004a40 comp rx=0 tx=0).stop 2024-04-24T08:17:10.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.012+0000 7fca1f38b740 1 -- 172.21.15.28:0/329617841 shutdown_connections 2024-04-24T08:17:10.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.012+0000 7fca1f38b740 1 --2- 172.21.15.28:0/329617841 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x561bc2b4ea40 0x561bc2bd8800 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.014 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.012+0000 7fca1f38b740 1 --2- 172.21.15.28:0/329617841 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561bc2b43a50 0x561bc2b57200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.013+0000 7fc9fb400640 0 client.26396 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.012+0000 7fca1f38b740 1 --2- 172.21.15.28:0/329617841 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561bc2b4c880 0x561bc2b47f50 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.013+0000 7fca1f38b740 1 --2- 172.21.15.28:0/329617841 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561bc2b3fd70 0x561bc2b56cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.013+0000 7fca1f38b740 1 -- 172.21.15.28:0/329617841 >> 172.21.15.28:0/329617841 conn(0x561bc2b4c340 msgr2=0x561bc2b451e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:10.015 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.013+0000 7fca1f38b740 1 -- 172.21.15.28:0/329617841 shutdown_connections 2024-04-24T08:17:10.016 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.013+0000 7fca1f38b740 1 -- 172.21.15.28:0/329617841 wait complete. 2024-04-24T08:17:10.016 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:17:10.016 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "scrub active (90 inodes in the stack)", 2024-04-24T08:17:10.016 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:17:10.016 INFO:teuthology.orchestra.run.smithi028.stdout: "d52a19ca-6482-49d7-a4d1-d1bcd5ceb884": { 2024-04-24T08:17:10.016 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:17:10.017 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "d52a19ca-6482-49d7-a4d1-d1bcd5ceb884", 2024-04-24T08:17:10.017 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:17:10.017 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:17:10.017 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:17:10.017 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:17:10.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.013+0000 7fca1f38b740 20 client.26396 trim_cache size 0 max 16384 2024-04-24T08:17:10.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.014+0000 7fca1b000640 1 -- 172.21.15.28:0/3819198384 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc9fc0a3e70 msgr2=0x7fc9fc0a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:10.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.014+0000 7fca1b000640 1 --2- 172.21.15.28:0/3819198384 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc9fc0a3e70 0x7fc9fc0a6330 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fca0c005fd0 tx=0x7fca0c004ab0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.014+0000 7fca1b000640 1 -- 172.21.15.28:0/3819198384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca141093e0 msgr2=0x7fca1410dff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:10.018 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.014+0000 7fca1b000640 1 --2- 172.21.15.28:0/3819198384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca141093e0 0x7fca1410dff0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fca00000c00 tx=0x7fca00033d10 comp rx=0 tx=0).stop 2024-04-24T08:17:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.015+0000 7fca1b000640 1 -- 172.21.15.28:0/3819198384 shutdown_connections 2024-04-24T08:17:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.015+0000 7fca1b000640 1 --2- 172.21.15.28:0/3819198384 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc9fc0a3e70 0x7fc9fc0a6330 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.015+0000 7fca1b000640 1 --2- 172.21.15.28:0/3819198384 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fca1410e530 0x7fca1410e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.015+0000 7fca1b000640 1 --2- 172.21.15.28:0/3819198384 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fca141093e0 0x7fca1410dff0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.015+0000 7fca1b000640 1 --2- 172.21.15.28:0/3819198384 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca14104310 0x7fca14110f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:10.019 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.015+0000 7fca1b000640 1 -- 172.21.15.28:0/3819198384 >> 172.21.15.28:0/3819198384 conn(0x7fca140ffb70 msgr2=0x7fca14100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.015+0000 7fca1b000640 1 -- 172.21.15.28:0/3819198384 shutdown_connections 2024-04-24T08:17:10.020 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:10.015+0000 7fca1b000640 1 -- 172.21.15.28:0/3819198384 wait complete. 2024-04-24T08:17:10.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:09 smithi028 ceph-mon[27786]: pgmap v207: 97 pgs: 97 active+clean; 23 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 42 MiB/s wr, 8.78k op/s 2024-04-24T08:17:10.045 INFO:tasks.cephfs.filesystem:scrub status for tag:d52a19ca-6482-49d7-a4d1-d1bcd5ceb884 - {'path': '/', 'tag': 'd52a19ca-6482-49d7-a4d1-d1bcd5ceb884', 'options': 'recursive,force'} 2024-04-24T08:17:10.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:09 smithi117 ceph-mon[32438]: pgmap v207: 97 pgs: 97 active+clean; 23 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 42 MiB/s wr, 8.78k op/s 2024-04-24T08:17:10.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:09 smithi194 ceph-mon[32836]: pgmap v207: 97 pgs: 97 active+clean; 23 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 42 MiB/s wr, 8.78k op/s 2024-04-24T08:17:11.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:10 smithi117 ceph-mon[32438]: scrub queued for path: / 2024-04-24T08:17:11.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:10 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:11.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:10 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:17:11.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:10 smithi194 ceph-mon[32836]: scrub queued for path: / 2024-04-24T08:17:11.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:10 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:11.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:10 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:17:11.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:10 smithi028 ceph-mon[27786]: scrub queued for path: / 2024-04-24T08:17:11.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:10 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:11.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:10 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:17:12.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:11 smithi117 ceph-mon[32438]: pgmap v208: 97 pgs: 97 active+clean; 23 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 103 KiB/s rd, 41 MiB/s wr, 8.60k op/s 2024-04-24T08:17:12.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:11 smithi194 ceph-mon[32836]: pgmap v208: 97 pgs: 97 active+clean; 23 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 103 KiB/s rd, 41 MiB/s wr, 8.60k op/s 2024-04-24T08:17:12.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:11 smithi028 ceph-mon[27786]: pgmap v208: 97 pgs: 97 active+clean; 23 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 103 KiB/s rd, 41 MiB/s wr, 8.60k op/s 2024-04-24T08:17:14.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:13 smithi117 ceph-mon[32438]: pgmap v209: 97 pgs: 97 active+clean; 23 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 55 KiB/s rd, 39 MiB/s wr, 8.22k op/s 2024-04-24T08:17:14.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:13 smithi194 ceph-mon[32836]: pgmap v209: 97 pgs: 97 active+clean; 23 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 55 KiB/s rd, 39 MiB/s wr, 8.22k op/s 2024-04-24T08:17:14.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:13 smithi028 ceph-mon[27786]: pgmap v209: 97 pgs: 97 active+clean; 23 GiB data, 49 GiB used, 1024 GiB / 1.0 TiB avail; 55 KiB/s rd, 39 MiB/s wr, 8.22k op/s 2024-04-24T08:17:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:14 smithi117 ceph-mon[32438]: osdmap e92: 12 total, 12 up, 12 in 2024-04-24T08:17:15.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:14 smithi194 ceph-mon[32836]: osdmap e92: 12 total, 12 up, 12 in 2024-04-24T08:17:15.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:14 smithi028 ceph-mon[27786]: osdmap e92: 12 total, 12 up, 12 in 2024-04-24T08:17:16.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:15 smithi117 ceph-mon[32438]: pgmap v211: 97 pgs: 97 active+clean; 23 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 65 KiB/s rd, 40 MiB/s wr, 8.27k op/s 2024-04-24T08:17:16.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:15 smithi194 ceph-mon[32836]: pgmap v211: 97 pgs: 97 active+clean; 23 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 65 KiB/s rd, 40 MiB/s wr, 8.27k op/s 2024-04-24T08:17:16.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:15 smithi028 ceph-mon[27786]: pgmap v211: 97 pgs: 97 active+clean; 23 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 65 KiB/s rd, 40 MiB/s wr, 8.27k op/s 2024-04-24T08:17:18.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:17 smithi117 ceph-mon[32438]: pgmap v212: 97 pgs: 97 active+clean; 23 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 57 KiB/s rd, 33 MiB/s wr, 6.76k op/s 2024-04-24T08:17:18.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:17 smithi194 ceph-mon[32836]: pgmap v212: 97 pgs: 97 active+clean; 23 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 57 KiB/s rd, 33 MiB/s wr, 6.76k op/s 2024-04-24T08:17:18.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:17 smithi028 ceph-mon[27786]: pgmap v212: 97 pgs: 97 active+clean; 23 GiB data, 50 GiB used, 1023 GiB / 1.0 TiB avail; 57 KiB/s rd, 33 MiB/s wr, 6.76k op/s 2024-04-24T08:17:20.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:19 smithi117 ceph-mon[32438]: pgmap v213: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 23 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 58 KiB/s rd, 42 MiB/s wr, 8.50k op/s 2024-04-24T08:17:20.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:19 smithi194 ceph-mon[32836]: pgmap v213: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 23 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 58 KiB/s rd, 42 MiB/s wr, 8.50k op/s 2024-04-24T08:17:20.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:19 smithi028 ceph-mon[27786]: pgmap v213: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 23 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 58 KiB/s rd, 42 MiB/s wr, 8.50k op/s 2024-04-24T08:17:22.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:21 smithi117 ceph-mon[32438]: pgmap v214: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 23 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 58 KiB/s rd, 41 MiB/s wr, 8.21k op/s 2024-04-24T08:17:22.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:21 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:17:22.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:21 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:22.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:21 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:17:22.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:21 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:22.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:21 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:17:22.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:21 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:22.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:21 smithi194 ceph-mon[32836]: pgmap v214: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 23 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 58 KiB/s rd, 41 MiB/s wr, 8.21k op/s 2024-04-24T08:17:22.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:21 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:17:22.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:21 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:22.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:21 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:17:22.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:21 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:22.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:21 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:17:22.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:21 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:22.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:21 smithi028 ceph-mon[27786]: pgmap v214: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 23 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 58 KiB/s rd, 41 MiB/s wr, 8.21k op/s 2024-04-24T08:17:22.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:21 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:17:22.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:21 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:22.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:21 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:17:22.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:21 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:22.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:21 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:17:22.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:21 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:23.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:22 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:23.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:22 smithi194 ceph-mon[32836]: pgmap v215: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 23 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 58 KiB/s rd, 36 MiB/s wr, 7.09k op/s 2024-04-24T08:17:23.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:22 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:23.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:22 smithi028 ceph-mon[27786]: pgmap v215: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 23 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 58 KiB/s rd, 36 MiB/s wr, 7.09k op/s 2024-04-24T08:17:23.377 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:22 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:23.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:22 smithi117 ceph-mon[32438]: pgmap v215: 97 pgs: 1 active+clean+snaptrim, 96 active+clean; 23 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 58 KiB/s rd, 36 MiB/s wr, 7.09k op/s 2024-04-24T08:17:25.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:25 smithi194 ceph-mon[32836]: pgmap v216: 97 pgs: 97 active+clean; 23 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 90 KiB/s rd, 44 MiB/s wr, 8.70k op/s 2024-04-24T08:17:25.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:25 smithi028 ceph-mon[27786]: pgmap v216: 97 pgs: 97 active+clean; 23 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 90 KiB/s rd, 44 MiB/s wr, 8.70k op/s 2024-04-24T08:17:25.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:25 smithi117 ceph-mon[32438]: pgmap v216: 97 pgs: 97 active+clean; 23 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 90 KiB/s rd, 44 MiB/s wr, 8.70k op/s 2024-04-24T08:17:27.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:27 smithi194 ceph-mon[32836]: pgmap v217: 97 pgs: 97 active+clean; 23 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 51 KiB/s rd, 36 MiB/s wr, 7.27k op/s 2024-04-24T08:17:27.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:27 smithi028 ceph-mon[27786]: pgmap v217: 97 pgs: 97 active+clean; 23 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 51 KiB/s rd, 36 MiB/s wr, 7.27k op/s 2024-04-24T08:17:27.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:27 smithi117 ceph-mon[32438]: pgmap v217: 97 pgs: 97 active+clean; 23 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 51 KiB/s rd, 36 MiB/s wr, 7.27k op/s 2024-04-24T08:17:29.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:29 smithi028 ceph-mon[27786]: pgmap v218: 97 pgs: 97 active+clean; 23 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 51 KiB/s rd, 40 MiB/s wr, 8.07k op/s 2024-04-24T08:17:29.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:29 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:17:29.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:29 smithi028 ceph-mon[27786]: osdmap e93: 12 total, 12 up, 12 in 2024-04-24T08:17:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:29 smithi117 ceph-mon[32438]: pgmap v218: 97 pgs: 97 active+clean; 23 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 51 KiB/s rd, 40 MiB/s wr, 8.07k op/s 2024-04-24T08:17:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:29 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:17:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:29 smithi117 ceph-mon[32438]: osdmap e93: 12 total, 12 up, 12 in 2024-04-24T08:17:29.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:29 smithi194 ceph-mon[32836]: pgmap v218: 97 pgs: 97 active+clean; 23 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 51 KiB/s rd, 40 MiB/s wr, 8.07k op/s 2024-04-24T08:17:29.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:29 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:17:29.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:29 smithi194 ceph-mon[32836]: osdmap e93: 12 total, 12 up, 12 in 2024-04-24T08:17:30.788 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:30 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:17:30.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:30 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:17:30.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:30 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:17:31.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:31 smithi028 ceph-mon[27786]: pgmap v220: 97 pgs: 97 active+clean; 23 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 40 KiB/s rd, 35 MiB/s wr, 7.23k op/s 2024-04-24T08:17:31.842 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:31 smithi194 ceph-mon[32836]: pgmap v220: 97 pgs: 97 active+clean; 23 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 40 KiB/s rd, 35 MiB/s wr, 7.23k op/s 2024-04-24T08:17:31.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:31 smithi117 ceph-mon[32438]: pgmap v220: 97 pgs: 97 active+clean; 23 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 40 KiB/s rd, 35 MiB/s wr, 7.23k op/s 2024-04-24T08:17:33.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:33 smithi028 ceph-mon[27786]: pgmap v221: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 40 KiB/s rd, 38 MiB/s wr, 8.18k op/s 2024-04-24T08:17:33.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:33 smithi117 ceph-mon[32438]: pgmap v221: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 40 KiB/s rd, 38 MiB/s wr, 8.18k op/s 2024-04-24T08:17:33.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:33 smithi194 ceph-mon[32836]: pgmap v221: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 40 KiB/s rd, 38 MiB/s wr, 8.18k op/s 2024-04-24T08:17:35.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:35 smithi028 ceph-mon[27786]: pgmap v222: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 27 MiB/s wr, 5.91k op/s 2024-04-24T08:17:35.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:35 smithi117 ceph-mon[32438]: pgmap v222: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 27 MiB/s wr, 5.91k op/s 2024-04-24T08:17:35.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:35 smithi194 ceph-mon[32836]: pgmap v222: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 2.2 KiB/s rd, 27 MiB/s wr, 5.91k op/s 2024-04-24T08:17:37.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:37 smithi028 ceph-mon[27786]: pgmap v223: 97 pgs: 97 active+clean; 24 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 51 KiB/s rd, 28 MiB/s wr, 6.26k op/s 2024-04-24T08:17:37.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:37 smithi117 ceph-mon[32438]: pgmap v223: 97 pgs: 97 active+clean; 24 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 51 KiB/s rd, 28 MiB/s wr, 6.26k op/s 2024-04-24T08:17:37.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:37 smithi194 ceph-mon[32836]: pgmap v223: 97 pgs: 97 active+clean; 24 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 51 KiB/s rd, 28 MiB/s wr, 6.26k op/s 2024-04-24T08:17:39.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:39 smithi028 ceph-mon[27786]: pgmap v224: 97 pgs: 97 active+clean; 24 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 51 KiB/s rd, 23 MiB/s wr, 5.18k op/s 2024-04-24T08:17:39.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:39 smithi117 ceph-mon[32438]: pgmap v224: 97 pgs: 97 active+clean; 24 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 51 KiB/s rd, 23 MiB/s wr, 5.18k op/s 2024-04-24T08:17:39.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:39 smithi194 ceph-mon[32836]: pgmap v224: 97 pgs: 97 active+clean; 24 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 51 KiB/s rd, 23 MiB/s wr, 5.18k op/s 2024-04-24T08:17:40.046 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:17:40.046 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:17:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.361+0000 7f106b400640 1 -- 172.21.15.28:0/3260401871 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1064101080 msgr2=0x7f1064101460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.361+0000 7f106b400640 1 --2- 172.21.15.28:0/3260401871 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1064101080 0x7f1064101460 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1058009900 tx=0x7f105802a510 comp rx=0 tx=0).stop 2024-04-24T08:17:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.361+0000 7f106b400640 1 -- 172.21.15.28:0/3260401871 shutdown_connections 2024-04-24T08:17:40.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.361+0000 7f106b400640 1 --2- 172.21.15.28:0/3260401871 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10641019a0 0x7f1064105e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.361+0000 7f106b400640 1 --2- 172.21.15.28:0/3260401871 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10641063a0 0x7f1064106830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.361+0000 7f106b400640 1 --2- 172.21.15.28:0/3260401871 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1064101080 0x7f1064101460 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.361+0000 7f106b400640 1 -- 172.21.15.28:0/3260401871 >> 172.21.15.28:0/3260401871 conn(0x7f1064077280 msgr2=0x7f1064077690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.361+0000 7f106b400640 1 -- 172.21.15.28:0/3260401871 shutdown_connections 2024-04-24T08:17:40.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.361+0000 7f106b400640 1 -- 172.21.15.28:0/3260401871 wait complete. 2024-04-24T08:17:40.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.363+0000 7f106b400640 1 Processor -- start 2024-04-24T08:17:40.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.363+0000 7f106b400640 1 -- start start 2024-04-24T08:17:40.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.364+0000 7f106b400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1064101080 0x7f106419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.364+0000 7f106b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10641019a0 0x7f106419f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.364+0000 7f106b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10641063a0 0x7f10641a37d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.364+0000 7f106b400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1064118a20 con 0x7f10641063a0 2024-04-24T08:17:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.364+0000 7f106b400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f10641188a0 con 0x7f1064101080 2024-04-24T08:17:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.364+0000 7f106b400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1064118ba0 con 0x7f10641019a0 2024-04-24T08:17:40.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.364+0000 7f1069400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10641063a0 0x7f10641a37d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.364+0000 7f1068a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1064101080 0x7f106419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.364+0000 7f1063e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10641019a0 0x7f106419f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.364+0000 7f1068a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1064101080 0x7f106419ef60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:56248/0 (socket says 172.21.15.28:56248) 2024-04-24T08:17:40.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.364+0000 7f1068a00640 1 -- 172.21.15.28:0/4151100731 learned_addr learned my addr 172.21.15.28:0/4151100731 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:40.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.364+0000 7f1063e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10641019a0 0x7f106419f4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60174/0 (socket says 172.21.15.28:60174) 2024-04-24T08:17:40.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.365+0000 7f1068a00640 1 -- 172.21.15.28:0/4151100731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10641019a0 msgr2=0x7f106419f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:40.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.365+0000 7f1068a00640 1 --2- 172.21.15.28:0/4151100731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10641019a0 0x7f106419f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.365+0000 7f1068a00640 1 -- 172.21.15.28:0/4151100731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10641063a0 msgr2=0x7f10641a37d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:40.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.365+0000 7f1068a00640 1 --2- 172.21.15.28:0/4151100731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10641063a0 0x7f10641a37d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.365+0000 7f1068a00640 1 -- 172.21.15.28:0/4151100731 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f10641a3f00 con 0x7f1064101080 2024-04-24T08:17:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.366+0000 7f1068a00640 1 --2- 172.21.15.28:0/4151100731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1064101080 0x7f106419ef60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f105802aa80 tx=0x7f1058003b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.367+0000 7f1061600640 1 -- 172.21.15.28:0/4151100731 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105803f070 con 0x7f1064101080 2024-04-24T08:17:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.367+0000 7f1061600640 1 -- 172.21.15.28:0/4151100731 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1058031c90 con 0x7f1064101080 2024-04-24T08:17:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.367+0000 7f1061600640 1 -- 172.21.15.28:0/4151100731 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1058035030 con 0x7f1064101080 2024-04-24T08:17:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.367+0000 7f106b400640 1 -- 172.21.15.28:0/4151100731 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f10641a4190 con 0x7f1064101080 2024-04-24T08:17:40.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.367+0000 7f106b400640 1 -- 172.21.15.28:0/4151100731 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1064107430 con 0x7f1064101080 2024-04-24T08:17:40.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.373+0000 7f1061600640 1 -- 172.21.15.28:0/4151100731 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1058042070 con 0x7f1064101080 2024-04-24T08:17:40.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.375+0000 7f1061600640 1 --2- 172.21.15.28:0/4151100731 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f104c0a3ea0 0x7f104c0a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.377+0000 7f106e3e8740 1 Processor -- start 2024-04-24T08:17:40.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.377+0000 7f1061600640 1 -- 172.21.15.28:0/4151100731 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7f10580f0bb0 con 0x7f1064101080 2024-04-24T08:17:40.379 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.378+0000 7f1063e00640 1 --2- 172.21.15.28:0/4151100731 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f104c0a3ea0 0x7f104c0a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.378+0000 7f106e3e8740 1 -- start start 2024-04-24T08:17:40.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.380+0000 7f1063e00640 1 --2- 172.21.15.28:0/4151100731 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f104c0a3ea0 0x7f104c0a6360 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1064077b00 tx=0x7f105000b410 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:40.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.381+0000 7f106e3e8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b7f860 0x555a52b7fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.381+0000 7f106e3e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 0x555a52b8a310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.381+0000 7f106e3e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b8a850 0x555a52b8cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.382+0000 7f106e3e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555a52ae51f0 con 0x555a52b80180 2024-04-24T08:17:40.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.382+0000 7f106e3e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555a52a8b350 con 0x555a52b8a850 2024-04-24T08:17:40.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.382+0000 7f1069400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b8a850 0x555a52b8cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.382+0000 7f1069400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b8a850 0x555a52b8cc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:56252/0 (socket says 172.21.15.28:56252) 2024-04-24T08:17:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.382+0000 7f1069400640 1 -- 172.21.15.28:0/2777452972 learned_addr learned my addr 172.21.15.28:0/2777452972 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.382+0000 7f106e3e8740 1 -- 172.21.15.28:0/2777452972 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555a52ae11b0 con 0x555a52b7f860 2024-04-24T08:17:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.382+0000 7f1068a00640 1 --2- 172.21.15.28:0/2777452972 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b7f860 0x555a52b7fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.382+0000 7f1063e00640 1 --2- 172.21.15.28:0/2777452972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 0x555a52b8a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.383+0000 7f1069400640 1 -- 172.21.15.28:0/2777452972 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b7f860 msgr2=0x555a52b7fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.383+0000 7f1069400640 1 --2- 172.21.15.28:0/2777452972 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b7f860 0x555a52b7fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.383+0000 7f1069400640 1 -- 172.21.15.28:0/2777452972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 msgr2=0x555a52b8a310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:40.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.383+0000 7f1069400640 1 --2- 172.21.15.28:0/2777452972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 0x555a52b8a310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.383+0000 7f1069400640 1 -- 172.21.15.28:0/2777452972 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a52adff50 con 0x555a52b8a850 2024-04-24T08:17:40.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.383+0000 7f1063e00640 1 --2- 172.21.15.28:0/2777452972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 0x555a52b8a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.383+0000 7f1069400640 1 --2- 172.21.15.28:0/2777452972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b8a850 0x555a52b8cc40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f105c009f40 tx=0x7f105c010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b74c76b71ce67dc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.385+0000 7f1055c00640 1 -- 172.21.15.28:0/2777452972 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c003b70 con 0x555a52b8a850 2024-04-24T08:17:40.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.385+0000 7f1055c00640 1 -- 172.21.15.28:0/2777452972 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f105c003d10 con 0x555a52b8a850 2024-04-24T08:17:40.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.385+0000 7f1055c00640 1 -- 172.21.15.28:0/2777452972 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c011af0 con 0x555a52b8a850 2024-04-24T08:17:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.386+0000 7f106e3e8740 1 -- 172.21.15.28:0/2777452972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b8a850 msgr2=0x555a52b8cc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.386+0000 7f106e3e8740 1 --2- 172.21.15.28:0/2777452972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b8a850 0x555a52b8cc40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f105c009f40 tx=0x7f105c010860 comp rx=0 tx=0).stop 2024-04-24T08:17:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.386+0000 7f106e3e8740 1 -- 172.21.15.28:0/2777452972 shutdown_connections 2024-04-24T08:17:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.386+0000 7f106e3e8740 1 --2- 172.21.15.28:0/2777452972 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 0x555a52b8a310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.386+0000 7f106e3e8740 1 --2- 172.21.15.28:0/2777452972 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b8a850 0x555a52b8cc40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.386+0000 7f106e3e8740 1 --2- 172.21.15.28:0/2777452972 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b7f860 0x555a52b7fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.386+0000 7f106e3e8740 1 -- 172.21.15.28:0/2777452972 >> 172.21.15.28:0/2777452972 conn(0x555a52b7bb80 msgr2=0x555a52b7bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.387+0000 7f106e3e8740 1 -- 172.21.15.28:0/2777452972 shutdown_connections 2024-04-24T08:17:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.387+0000 7f106e3e8740 1 -- 172.21.15.28:0/2777452972 wait complete. 2024-04-24T08:17:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.387+0000 7f106e3e8740 1 -- start start 2024-04-24T08:17:40.390 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.388+0000 7f106e3e8740 1 Processor -- start 2024-04-24T08:17:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.389+0000 7f106e3e8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b7f860 0x555a52b87800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.389+0000 7f106e3e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 0x555a52b87d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.389+0000 7f106e3e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b8a850 0x555a52b88280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.389+0000 7f106e3e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555a52b8d640 con 0x555a52b80180 2024-04-24T08:17:40.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.389+0000 7f106e3e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555a52a9e690 con 0x555a52b8a850 2024-04-24T08:17:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.389+0000 7f106e3e8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555a52b8d7c0 con 0x555a52b7f860 2024-04-24T08:17:40.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.389+0000 7f106e3e8740 10 client.0 _ll_register_callbacks cb 0x555a52ada650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.390+0000 7f1068a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b7f860 0x555a52b87800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.390+0000 7f1068a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b7f860 0x555a52b87800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60192/0 (socket says 172.21.15.28:60192) 2024-04-24T08:17:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.390+0000 7f1063e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 0x555a52b87d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.390+0000 7f1068a00640 1 -- 172.21.15.28:0/159384917 learned_addr learned my addr 172.21.15.28:0/159384917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:40.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.390+0000 7f1069400640 1 --2- 172.21.15.28:0/159384917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b8a850 0x555a52b88280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.391+0000 7f1063e00640 1 -- 172.21.15.28:0/159384917 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b7f860 msgr2=0x555a52b87800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.391+0000 7f1063e00640 1 --2- 172.21.15.28:0/159384917 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b7f860 0x555a52b87800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.391+0000 7f1063e00640 1 -- 172.21.15.28:0/159384917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b8a850 msgr2=0x555a52b88280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.391+0000 7f1063e00640 1 --2- 172.21.15.28:0/159384917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b8a850 0x555a52b88280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.391+0000 7f1063e00640 1 -- 172.21.15.28:0/159384917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a52a9d920 con 0x555a52b80180 2024-04-24T08:17:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.392+0000 7f1063e00640 1 --2- 172.21.15.28:0/159384917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 0x555a52b87d40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1050005d00 tx=0x7f1050004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc3cd8e0ac94f461 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.393+0000 7f104a000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:17:40.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.393+0000 7f104a000640 1 -- 172.21.15.28:0/159384917 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105002e900 con 0x555a52b80180 2024-04-24T08:17:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.393+0000 7f106e3e8740 1 -- 172.21.15.28:0/159384917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f105c003d10 con 0x555a52b80180 2024-04-24T08:17:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.393+0000 7f104a000640 1 -- 172.21.15.28:0/159384917 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7f105002eaa0 con 0x555a52b80180 2024-04-24T08:17:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.394+0000 7f104a000640 1 -- 172.21.15.28:0/159384917 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f105002eca0 con 0x555a52b80180 2024-04-24T08:17:40.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.396+0000 7f106e3e8740 10 client.16149 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:40.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.396+0000 7f106e3e8740 1 -- 172.21.15.28:0/159384917 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555a528c17d0 con 0x555a52b80180 2024-04-24T08:17:40.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.398+0000 7f104a000640 1 -- 172.21.15.28:0/159384917 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f105c003d10 con 0x555a52b80180 2024-04-24T08:17:40.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.398+0000 7f104a000640 1 -- 172.21.15.28:0/159384917 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f105001f040 con 0x555a52b80180 2024-04-24T08:17:40.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.400+0000 7f106e3e8740 10 client.16149 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.400+0000 7f106e3e8740 10 client.16149 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:17:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.400+0000 7f106e3e8740 20 client.16149 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.400+0000 7f106e3e8740 1 --2- 172.21.15.28:0/159384917 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x555a52b9abb0 0x555a52b9cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.400+0000 7f106e3e8740 4 client.16149 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:17:40.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.400+0000 7f106e3e8740 1 -- 172.21.15.28:0/159384917 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a52af2c10 con 0x555a52b9abb0 2024-04-24T08:17:40.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.401+0000 7f1068a00640 1 --2- 172.21.15.28:0/159384917 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x555a52b9abb0 0x555a52b9cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.402+0000 7f1068a00640 1 --2- 172.21.15.28:0/159384917 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x555a52b9abb0 0x555a52b9cfa0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22c49144a328b243 server_cookie=f66b76c1b8d968fb in_seq=0 out_seq=0 2024-04-24T08:17:40.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.402+0000 7f104a000640 10 client.16149 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.404+0000 7f104a000640 1 -- 172.21.15.28:0/159384917 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555a52af2c10 con 0x555a52b9abb0 2024-04-24T08:17:40.405 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.404+0000 7f104a000640 10 client.16149 handle_command_reply: tid=0 2024-04-24T08:17:40.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.404+0000 7f106e3e8740 1 client.16149 shutdown 2024-04-24T08:17:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.406+0000 7f106e3e8740 1 -- 172.21.15.28:0/159384917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 msgr2=0x555a52b87d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.406+0000 7f106e3e8740 1 --2- 172.21.15.28:0/159384917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 0x555a52b87d40 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f1050005d00 tx=0x7f1050004730 comp rx=0 tx=0).stop 2024-04-24T08:17:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.406+0000 7f106e3e8740 1 -- 172.21.15.28:0/159384917 shutdown_connections 2024-04-24T08:17:40.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.406+0000 7f106e3e8740 1 --2- 172.21.15.28:0/159384917 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x555a52b9abb0 0x555a52b9cfa0 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.406+0000 7f106e3e8740 1 --2- 172.21.15.28:0/159384917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 0x555a52b87d40 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.406+0000 7f106e3e8740 1 --2- 172.21.15.28:0/159384917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b8a850 0x555a52b88280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.406+0000 7f106e3e8740 1 --2- 172.21.15.28:0/159384917 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b7f860 0x555a52b87800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.406+0000 7f104a000640 0 client.16149 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:40.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.406+0000 7f106e3e8740 1 -- 172.21.15.28:0/159384917 >> 172.21.15.28:0/159384917 conn(0x555a52b7bb80 msgr2=0x555a52b81980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:40.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.407+0000 7f106e3e8740 1 -- 172.21.15.28:0/159384917 shutdown_connections 2024-04-24T08:17:40.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.407+0000 7f106e3e8740 1 -- 172.21.15.28:0/159384917 wait complete. 2024-04-24T08:17:40.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.407+0000 7f106e3e8740 20 client.16149 trim_cache size 0 max 16384 2024-04-24T08:17:40.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.449+0000 7f106e3e8740 1 Processor -- start 2024-04-24T08:17:40.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.451+0000 7f106e3e8740 1 -- start start 2024-04-24T08:17:40.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.451+0000 7f106e3e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b7bb80 0x555a52b880a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.451+0000 7f106e3e8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b7f860 0x555a52ba2870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.451+0000 7f106e3e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 0x555a52ba2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.451+0000 7f106e3e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f10641ae610 con 0x555a52b80180 2024-04-24T08:17:40.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.451+0000 7f106e3e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f10641abee0 con 0x555a52b7bb80 2024-04-24T08:17:40.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.451+0000 7f106e3e8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1064069270 con 0x555a52b7f860 2024-04-24T08:17:40.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.452+0000 7f1068a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b7bb80 0x555a52b880a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.454 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.452+0000 7f1068a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b7bb80 0x555a52b880a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:56272/0 (socket says 172.21.15.28:56272) 2024-04-24T08:17:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.452+0000 7f1068a00640 1 -- 172.21.15.28:0/4261335820 learned_addr learned my addr 172.21.15.28:0/4261335820 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.452+0000 7f1063e00640 1 --2- 172.21.15.28:0/4261335820 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b7f860 0x555a52ba2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.452+0000 7f1068a00640 1 -- 172.21.15.28:0/4261335820 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b7f860 msgr2=0x555a52ba2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.452+0000 7f1068a00640 1 --2- 172.21.15.28:0/4261335820 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b7f860 0x555a52ba2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.452+0000 7f1068a00640 1 -- 172.21.15.28:0/4261335820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 msgr2=0x555a52ba2db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.452+0000 7f1068a00640 1 --2- 172.21.15.28:0/4261335820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 0x555a52ba2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.452+0000 7f1068a00640 1 -- 172.21.15.28:0/4261335820 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a52b82620 con 0x555a52b7bb80 2024-04-24T08:17:40.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.453+0000 7f1068a00640 1 --2- 172.21.15.28:0/4261335820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b7bb80 0x555a52b880a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1058000c00 tx=0x7f10580dd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=168489b654c88b01 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:40.456 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.453+0000 7f1055200640 1 -- 172.21.15.28:0/4261335820 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1058049040 con 0x555a52b7bb80 2024-04-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.453+0000 7f1055200640 1 -- 172.21.15.28:0/4261335820 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f10580c0eb0 con 0x555a52b7bb80 2024-04-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.453+0000 7f1055200640 1 -- 172.21.15.28:0/4261335820 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f10580ee530 con 0x555a52b7bb80 2024-04-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.454+0000 7f106e3e8740 1 -- 172.21.15.28:0/4261335820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b7bb80 msgr2=0x555a52b880a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.454+0000 7f106e3e8740 1 --2- 172.21.15.28:0/4261335820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b7bb80 0x555a52b880a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1058000c00 tx=0x7f10580dd040 comp rx=0 tx=0).stop 2024-04-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.455+0000 7f106e3e8740 1 -- 172.21.15.28:0/4261335820 shutdown_connections 2024-04-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.455+0000 7f106e3e8740 1 --2- 172.21.15.28:0/4261335820 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 0x555a52ba2db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.455+0000 7f106e3e8740 1 --2- 172.21.15.28:0/4261335820 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b7f860 0x555a52ba2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.457 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.455+0000 7f106e3e8740 1 --2- 172.21.15.28:0/4261335820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b7bb80 0x555a52b880a0 secure :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f1058000c00 tx=0x7f10580dd040 comp rx=0 tx=0).stop 2024-04-24T08:17:40.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.455+0000 7f106e3e8740 1 -- 172.21.15.28:0/4261335820 >> 172.21.15.28:0/4261335820 conn(0x555a52b86440 msgr2=0x555a52b810c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:40.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.456+0000 7f106e3e8740 1 -- 172.21.15.28:0/4261335820 shutdown_connections 2024-04-24T08:17:40.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.456+0000 7f106e3e8740 1 -- 172.21.15.28:0/4261335820 wait complete. 2024-04-24T08:17:40.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.457+0000 7f106e3e8740 1 -- start start 2024-04-24T08:17:40.460 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.459+0000 7f106e3e8740 1 Processor -- start 2024-04-24T08:17:40.461 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.460+0000 7f106e3e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b7f860 0x555a52b92a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.460+0000 7f106e3e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 0x555a52b92f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.460+0000 7f106e3e8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b8a850 0x555a52b8f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.460+0000 7f106e3e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f106410f6f0 con 0x555a52b80180 2024-04-24T08:17:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.460+0000 7f106e3e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1064110b40 con 0x555a52b7f860 2024-04-24T08:17:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.460+0000 7f106e3e8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f106410fda0 con 0x555a52b8a850 2024-04-24T08:17:40.462 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.460+0000 7f106e3e8740 10 client.0 _ll_register_callbacks cb 0x7f1050005d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.461+0000 7f1068a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b7f860 0x555a52b92a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.461+0000 7f1069400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b8a850 0x555a52b8f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.461+0000 7f1069400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b8a850 0x555a52b8f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60218/0 (socket says 172.21.15.28:60218) 2024-04-24T08:17:40.463 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.461+0000 7f1069400640 1 -- 172.21.15.28:0/3594590441 learned_addr learned my addr 172.21.15.28:0/3594590441 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.462+0000 7f1063e00640 1 --2- 172.21.15.28:0/3594590441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 0x555a52b92f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.462+0000 7f1069400640 1 -- 172.21.15.28:0/3594590441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b7f860 msgr2=0x555a52b92a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.462+0000 7f1069400640 1 --2- 172.21.15.28:0/3594590441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b7f860 0x555a52b92a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.462+0000 7f1069400640 1 -- 172.21.15.28:0/3594590441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 msgr2=0x555a52b92f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.462+0000 7f1069400640 1 --2- 172.21.15.28:0/3594590441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 0x555a52b92f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.462+0000 7f1069400640 1 -- 172.21.15.28:0/3594590441 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a527f73f0 con 0x555a52b8a850 2024-04-24T08:17:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.462+0000 7f104a000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:17:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.462+0000 7f1069400640 1 --2- 172.21.15.28:0/3594590441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b8a850 0x555a52b8f9e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f105c005cc0 tx=0x7f105c003d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c9dec7b55aa4fad server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.463+0000 7f104a000640 1 -- 172.21.15.28:0/3594590441 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f105c02a060 con 0x555a52b8a850 2024-04-24T08:17:40.465 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.463+0000 7f104a000640 1 -- 172.21.15.28:0/3594590441 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7f105c011710 con 0x555a52b8a850 2024-04-24T08:17:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.463+0000 7f106e3e8740 1 -- 172.21.15.28:0/3594590441 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f10580c0eb0 con 0x555a52b8a850 2024-04-24T08:17:40.466 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.464+0000 7f104a000640 1 -- 172.21.15.28:0/3594590441 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f105c02a060 con 0x555a52b8a850 2024-04-24T08:17:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.464+0000 7f106e3e8740 10 client.26434 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.464+0000 7f106e3e8740 1 -- 172.21.15.28:0/3594590441 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555a527b70f0 con 0x555a52b8a850 2024-04-24T08:17:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.464+0000 7f104a000640 1 -- 172.21.15.28:0/3594590441 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f105c011df0 con 0x555a52b8a850 2024-04-24T08:17:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.465+0000 7f104a000640 1 -- 172.21.15.28:0/3594590441 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f105c005040 con 0x555a52b8a850 2024-04-24T08:17:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.465+0000 7f106e3e8740 10 client.26434 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.465+0000 7f106e3e8740 10 client.26434 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:17:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.465+0000 7f106e3e8740 20 client.26434 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.465+0000 7f106e3e8740 1 --2- 172.21.15.28:0/3594590441 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x555a52c01d80 0x555a52c041a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.467 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.465+0000 7f106e3e8740 4 client.26434 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:17:40.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.465+0000 7f106e3e8740 1 -- 172.21.15.28:0/3594590441 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7f10580c0eb0 con 0x555a52c01d80 2024-04-24T08:17:40.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.466+0000 7f1068a00640 1 --2- 172.21.15.28:0/3594590441 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x555a52c01d80 0x555a52c041a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.468 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.467+0000 7f104a000640 10 client.26434 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:40.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.467+0000 7f1068a00640 1 --2- 172.21.15.28:0/3594590441 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x555a52c01d80 0x555a52c041a0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5b4674cf7028ac3 server_cookie=71f33b51e7abd8d5 in_seq=0 out_seq=0 2024-04-24T08:17:40.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.468+0000 7f104a000640 1 -- 172.21.15.28:0/3594590441 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x7f10580c0eb0 con 0x555a52c01d80 2024-04-24T08:17:40.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.468+0000 7f104a000640 10 client.26434 handle_command_reply: tid=0 2024-04-24T08:17:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.468+0000 7f106e3e8740 1 client.26434 shutdown 2024-04-24T08:17:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.468+0000 7f106e3e8740 1 -- 172.21.15.28:0/3594590441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b8a850 msgr2=0x555a52b8f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.468+0000 7f106e3e8740 1 --2- 172.21.15.28:0/3594590441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b8a850 0x555a52b8f9e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f105c005cc0 tx=0x7f105c003d70 comp rx=0 tx=0).stop 2024-04-24T08:17:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.468+0000 7f106e3e8740 1 -- 172.21.15.28:0/3594590441 shutdown_connections 2024-04-24T08:17:40.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.468+0000 7f106e3e8740 1 --2- 172.21.15.28:0/3594590441 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x555a52c01d80 0x555a52c041a0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.468+0000 7f106e3e8740 1 --2- 172.21.15.28:0/3594590441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555a52b80180 0x555a52b92f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.468+0000 7f106e3e8740 1 --2- 172.21.15.28:0/3594590441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555a52b8a850 0x555a52b8f9e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.468+0000 7f104a000640 0 client.26434 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.468+0000 7f106e3e8740 1 --2- 172.21.15.28:0/3594590441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555a52b7f860 0x555a52b92a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.468+0000 7f106e3e8740 1 -- 172.21.15.28:0/3594590441 >> 172.21.15.28:0/3594590441 conn(0x555a52b86440 msgr2=0x555a52b80fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:40.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.469+0000 7f106e3e8740 1 -- 172.21.15.28:0/3594590441 shutdown_connections 2024-04-24T08:17:40.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.470+0000 7f106e3e8740 1 -- 172.21.15.28:0/3594590441 wait complete. 2024-04-24T08:17:40.472 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.470+0000 7f106e3e8740 20 client.26434 trim_cache size 0 max 16384 2024-04-24T08:17:40.472 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:17:40.473 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-24T08:17:40.473 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-24T08:17:40.473 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:17:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.471+0000 7f106b400640 1 -- 172.21.15.28:0/4151100731 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f104c0a3ea0 msgr2=0x7f104c0a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.471+0000 7f106b400640 1 --2- 172.21.15.28:0/4151100731 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f104c0a3ea0 0x7f104c0a6360 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f1064077b00 tx=0x7f105000b410 comp rx=0 tx=0).stop 2024-04-24T08:17:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.471+0000 7f106b400640 1 -- 172.21.15.28:0/4151100731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1064101080 msgr2=0x7f106419ef60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.471+0000 7f106b400640 1 --2- 172.21.15.28:0/4151100731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1064101080 0x7f106419ef60 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f105802aa80 tx=0x7f1058003b90 comp rx=0 tx=0).stop 2024-04-24T08:17:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.472+0000 7f106b400640 1 -- 172.21.15.28:0/4151100731 shutdown_connections 2024-04-24T08:17:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.472+0000 7f106b400640 1 --2- 172.21.15.28:0/4151100731 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f104c0a3ea0 0x7f104c0a6360 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.474 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.472+0000 7f106b400640 1 --2- 172.21.15.28:0/4151100731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f10641063a0 0x7f10641a37d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.472+0000 7f106b400640 1 --2- 172.21.15.28:0/4151100731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f10641019a0 0x7f106419f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.472+0000 7f106b400640 1 --2- 172.21.15.28:0/4151100731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1064101080 0x7f106419ef60 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.472+0000 7f106b400640 1 -- 172.21.15.28:0/4151100731 >> 172.21.15.28:0/4151100731 conn(0x7f1064077280 msgr2=0x7f1064103e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.472+0000 7f106b400640 1 -- 172.21.15.28:0/4151100731 shutdown_connections 2024-04-24T08:17:40.475 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.472+0000 7f106b400640 1 -- 172.21.15.28:0/4151100731 wait complete. 2024-04-24T08:17:40.504 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:17:40.504 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:17:40.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.816+0000 7f7270a00640 1 -- 172.21.15.28:0/3900482957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f726c073d50 msgr2=0x7f726c070dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:40.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.816+0000 7f7270a00640 1 --2- 172.21.15.28:0/3900482957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f726c073d50 0x7f726c070dd0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f725c009900 tx=0x7f725c02a480 comp rx=0 tx=0).stop 2024-04-24T08:17:40.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.816+0000 7f7270a00640 1 -- 172.21.15.28:0/3900482957 shutdown_connections 2024-04-24T08:17:40.818 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.816+0000 7f7270a00640 1 --2- 172.21.15.28:0/3900482957 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f726c06bb30 0x7f726c06bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.816+0000 7f7270a00640 1 --2- 172.21.15.28:0/3900482957 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f726c073d50 0x7f726c070dd0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.816+0000 7f7270a00640 1 --2- 172.21.15.28:0/3900482957 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f726c0684b0 0x7f726c070890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.816+0000 7f7270a00640 1 -- 172.21.15.28:0/3900482957 >> 172.21.15.28:0/3900482957 conn(0x7f726c077060 msgr2=0x7f726c0fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:40.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.817+0000 7f7270a00640 1 -- 172.21.15.28:0/3900482957 shutdown_connections 2024-04-24T08:17:40.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.817+0000 7f7270a00640 1 -- 172.21.15.28:0/3900482957 wait complete. 2024-04-24T08:17:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.819+0000 7f7270a00640 1 Processor -- start 2024-04-24T08:17:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.819+0000 7f7270a00640 1 -- start start 2024-04-24T08:17:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.820+0000 7f7270a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f726c0684b0 0x7f726c1a32c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.820+0000 7f7270a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f726c06bb30 0x7f726c1a3800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.820+0000 7f7270a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f726c073d50 0x7f726c1a7b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.820+0000 7f7270a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f726c119a00 con 0x7f726c0684b0 2024-04-24T08:17:40.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.820+0000 7f7270a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f726c119880 con 0x7f726c073d50 2024-04-24T08:17:40.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.820+0000 7f7270a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f726c119b80 con 0x7f726c06bb30 2024-04-24T08:17:40.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.821+0000 7f7269400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f726c06bb30 0x7f726c1a3800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.821+0000 7f7269400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f726c06bb30 0x7f726c1a3800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60238/0 (socket says 172.21.15.28:60238) 2024-04-24T08:17:40.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.821+0000 7f7269400640 1 -- 172.21.15.28:0/897217917 learned_addr learned my addr 172.21.15.28:0/897217917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:40.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.822+0000 7f726a800640 1 --2- 172.21.15.28:0/897217917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f726c073d50 0x7f726c1a7b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.822+0000 7f7269400640 1 -- 172.21.15.28:0/897217917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f726c073d50 msgr2=0x7f726c1a7b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:40.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.822+0000 7f7269400640 1 --2- 172.21.15.28:0/897217917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f726c073d50 0x7f726c1a7b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.824 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.822+0000 7f7269400640 1 -- 172.21.15.28:0/897217917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f726c0684b0 msgr2=0x7f726c1a32c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.822+0000 7f7269400640 1 --2- 172.21.15.28:0/897217917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f726c0684b0 0x7f726c1a32c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.822+0000 7f7269400640 1 -- 172.21.15.28:0/897217917 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f726c1a82b0 con 0x7f726c06bb30 2024-04-24T08:17:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.823+0000 7f7269400640 1 --2- 172.21.15.28:0/897217917 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f726c06bb30 0x7f726c1a3800 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f725c006010 tx=0x7f725c0042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.823+0000 7f7262a00640 1 -- 172.21.15.28:0/897217917 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c03f030 con 0x7f726c06bb30 2024-04-24T08:17:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.823+0000 7f7262a00640 1 -- 172.21.15.28:0/897217917 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f725c003ce0 con 0x7f726c06bb30 2024-04-24T08:17:40.825 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.823+0000 7f7262a00640 1 -- 172.21.15.28:0/897217917 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f725c035070 con 0x7f726c06bb30 2024-04-24T08:17:40.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.824+0000 7f7270a00640 1 -- 172.21.15.28:0/897217917 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f726c1a85a0 con 0x7f726c06bb30 2024-04-24T08:17:40.826 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.825+0000 7f7270a00640 1 -- 172.21.15.28:0/897217917 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f726c1afe30 con 0x7f726c06bb30 2024-04-24T08:17:40.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.827+0000 7f7270a00640 1 -- 172.21.15.28:0/897217917 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f726c069270 con 0x7f726c06bb30 2024-04-24T08:17:40.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.830+0000 7f7262a00640 1 -- 172.21.15.28:0/897217917 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f725c042030 con 0x7f726c06bb30 2024-04-24T08:17:40.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.835+0000 7f7262a00640 1 --2- 172.21.15.28:0/897217917 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f72500a3da0 0x7f72500a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:40.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.835+0000 7f7262a00640 1 -- 172.21.15.28:0/897217917 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7f725c0f1320 con 0x7f726c06bb30 2024-04-24T08:17:40.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.836+0000 7f7269e00640 1 --2- 172.21.15.28:0/897217917 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f72500a3da0 0x7f72500a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:40.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.838+0000 7f7269e00640 1 --2- 172.21.15.28:0/897217917 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f72500a3da0 0x7f72500a6260 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7264005fd0 tx=0x7f72640071c0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:40.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:40.841+0000 7f7262a00640 1 -- 172.21.15.28:0/897217917 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f725c049020 con 0x7f726c06bb30 2024-04-24T08:17:41.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.318+0000 7f7270a00640 1 -- 172.21.15.28:0/897217917 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f726c1b0030 con 0x7f726c06bb30 2024-04-24T08:17:41.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.319+0000 7f7262a00640 1 -- 172.21.15.28:0/897217917 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11871 (secure 0 0 0) 0x7f725c0bd970 con 0x7f726c06bb30 2024-04-24T08:17:41.324 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:17:41.324 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:14:38.663141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:17:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-24T08:17:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.322+0000 7f7270a00640 1 -- 172.21.15.28:0/897217917 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f72500a3da0 msgr2=0x7f72500a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.322+0000 7f7270a00640 1 --2- 172.21.15.28:0/897217917 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f72500a3da0 0x7f72500a6260 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7264005fd0 tx=0x7f72640071c0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.322+0000 7f7270a00640 1 -- 172.21.15.28:0/897217917 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f726c06bb30 msgr2=0x7f726c1a3800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.322+0000 7f7270a00640 1 --2- 172.21.15.28:0/897217917 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f726c06bb30 0x7f726c1a3800 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f725c006010 tx=0x7f725c0042f0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.323+0000 7f7270a00640 1 -- 172.21.15.28:0/897217917 shutdown_connections 2024-04-24T08:17:41.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.323+0000 7f7270a00640 1 --2- 172.21.15.28:0/897217917 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f72500a3da0 0x7f72500a6260 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.323+0000 7f7270a00640 1 --2- 172.21.15.28:0/897217917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f726c073d50 0x7f726c1a7b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.323+0000 7f7270a00640 1 --2- 172.21.15.28:0/897217917 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f726c06bb30 0x7f726c1a3800 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.323+0000 7f7270a00640 1 --2- 172.21.15.28:0/897217917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f726c0684b0 0x7f726c1a32c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.323+0000 7f7270a00640 1 -- 172.21.15.28:0/897217917 >> 172.21.15.28:0/897217917 conn(0x7f726c077060 msgr2=0x7f726c115d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:41.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.323+0000 7f7270a00640 1 -- 172.21.15.28:0/897217917 shutdown_connections 2024-04-24T08:17:41.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.323+0000 7f7270a00640 1 -- 172.21.15.28:0/897217917 wait complete. 2024-04-24T08:17:41.361 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:17:41.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.671+0000 7f1981800640 1 -- 172.21.15.28:0/366176386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f197c104e60 msgr2=0x7f197c108ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:41.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.671+0000 7f1981800640 1 --2- 172.21.15.28:0/366176386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f197c104e60 0x7f197c108ea0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f1968009860 tx=0x7f196802a340 comp rx=0 tx=0).stop 2024-04-24T08:17:41.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.673+0000 7f1981800640 1 -- 172.21.15.28:0/366176386 shutdown_connections 2024-04-24T08:17:41.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.673+0000 7f1981800640 1 --2- 172.21.15.28:0/366176386 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f197c109580 0x7f197c111150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.673+0000 7f1981800640 1 --2- 172.21.15.28:0/366176386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f197c104e60 0x7f197c108ea0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.673+0000 7f1981800640 1 --2- 172.21.15.28:0/366176386 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f197c1044b0 0x7f197c104890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.673+0000 7f1981800640 1 -- 172.21.15.28:0/366176386 >> 172.21.15.28:0/366176386 conn(0x7f197c0ffd90 msgr2=0x7f197c1021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.674+0000 7f1981800640 1 -- 172.21.15.28:0/366176386 shutdown_connections 2024-04-24T08:17:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.674+0000 7f1981800640 1 -- 172.21.15.28:0/366176386 wait complete. 2024-04-24T08:17:41.677 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.675+0000 7f1981800640 1 Processor -- start 2024-04-24T08:17:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.676+0000 7f1981800640 1 -- start start 2024-04-24T08:17:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.677+0000 7f1981800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f197c1044b0 0x7f197c0738a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.678 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.677+0000 7f1981800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f197c109580 0x7f197c06c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.677+0000 7f1981800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f197c06ceb0 0x7f197c06d360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.677+0000 7f1981800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f197c1145f0 con 0x7f197c1044b0 2024-04-24T08:17:41.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.677+0000 7f1981800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f197c114470 con 0x7f197c06ceb0 2024-04-24T08:17:41.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.677+0000 7f1981800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f197c114770 con 0x7f197c109580 2024-04-24T08:17:41.679 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.677+0000 7f197aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f197c1044b0 0x7f197c0738a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.677+0000 7f197a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f197c109580 0x7f197c06c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.677+0000 7f197aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f197c1044b0 0x7f197c0738a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36270/0 (socket says 172.21.15.28:36270) 2024-04-24T08:17:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.677+0000 7f197aa00640 1 -- 172.21.15.28:0/1315402917 learned_addr learned my addr 172.21.15.28:0/1315402917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.678+0000 7f197b400640 1 --2- 172.21.15.28:0/1315402917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f197c06ceb0 0x7f197c06d360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.678+0000 7f197b400640 1 -- 172.21.15.28:0/1315402917 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f197c109580 msgr2=0x7f197c06c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:41.680 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.678+0000 7f197b400640 1 --2- 172.21.15.28:0/1315402917 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f197c109580 0x7f197c06c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.678+0000 7f197b400640 1 -- 172.21.15.28:0/1315402917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f197c1044b0 msgr2=0x7f197c0738a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:41.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.678+0000 7f197b400640 1 --2- 172.21.15.28:0/1315402917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f197c1044b0 0x7f197c0738a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.678+0000 7f197b400640 1 -- 172.21.15.28:0/1315402917 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197c0713c0 con 0x7f197c06ceb0 2024-04-24T08:17:41.681 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.679+0000 7f197b400640 1 --2- 172.21.15.28:0/1315402917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f197c06ceb0 0x7f197c06d360 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f197400ef00 tx=0x7f197400c740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:41.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.679+0000 7f1973400640 1 -- 172.21.15.28:0/1315402917 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974004b20 con 0x7f197c06ceb0 2024-04-24T08:17:41.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.679+0000 7f1973400640 1 -- 172.21.15.28:0/1315402917 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1974003e50 con 0x7f197c06ceb0 2024-04-24T08:17:41.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.679+0000 7f1973400640 1 -- 172.21.15.28:0/1315402917 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f197400b570 con 0x7f197c06ceb0 2024-04-24T08:17:41.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.679+0000 7f1981800640 1 -- 172.21.15.28:0/1315402917 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f197c0716b0 con 0x7f197c06ceb0 2024-04-24T08:17:41.682 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.680+0000 7f1981800640 1 -- 172.21.15.28:0/1315402917 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f197c071b80 con 0x7f197c06ceb0 2024-04-24T08:17:41.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.685+0000 7f1973400640 1 -- 172.21.15.28:0/1315402917 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f197401d020 con 0x7f197c06ceb0 2024-04-24T08:17:41.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.688+0000 7f198465f740 1 Processor -- start 2024-04-24T08:17:41.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.689+0000 7f1973400640 1 --2- 172.21.15.28:0/1315402917 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f19600a3ef0 0x7f19600a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:41 smithi028 ceph-mon[27786]: pgmap v225: 97 pgs: 97 active+clean; 24 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 46 KiB/s rd, 19 MiB/s wr, 4.35k op/s 2024-04-24T08:17:41.691 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:41 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/897217917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:17:41.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.689+0000 7f1973400640 1 -- 172.21.15.28:0/1315402917 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7f19740d05a0 con 0x7f197c06ceb0 2024-04-24T08:17:41.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.690+0000 7f198465f740 1 -- start start 2024-04-24T08:17:41.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.690+0000 7f197a000640 1 --2- 172.21.15.28:0/1315402917 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f19600a3ef0 0x7f19600a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:41.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.690+0000 7f198465f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144e950 0x55c4c144ed30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.690+0000 7f198465f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c144f270 0x55c4c1459400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.690+0000 7f198465f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459940 0x55c4c145bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.690+0000 7f198465f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c4c13b41d0 con 0x55c4c144f270 2024-04-24T08:17:41.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.690+0000 7f198465f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c4c135a450 con 0x55c4c144e950 2024-04-24T08:17:41.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.690+0000 7f198465f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c4c13b0190 con 0x55c4c1459940 2024-04-24T08:17:41.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.690+0000 7f197b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459940 0x55c4c145bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:41.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.690+0000 7f197b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459940 0x55c4c145bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60258/0 (socket says 172.21.15.28:60258) 2024-04-24T08:17:41.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.691+0000 7f197b400640 1 -- 172.21.15.28:0/3021468338 learned_addr learned my addr 172.21.15.28:0/3021468338 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:41.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.691+0000 7f197aa00640 1 --2- 172.21.15.28:0/3021468338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144e950 0x55c4c144ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:41.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.691+0000 7f197aa00640 1 -- 172.21.15.28:0/3021468338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459940 msgr2=0x55c4c145bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:41.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.691+0000 7f197aa00640 1 --2- 172.21.15.28:0/3021468338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459940 0x55c4c145bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.691+0000 7f197aa00640 1 -- 172.21.15.28:0/3021468338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c144f270 msgr2=0x55c4c1459400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:41.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.691+0000 7f197aa00640 1 --2- 172.21.15.28:0/3021468338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c144f270 0x55c4c1459400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.691+0000 7f197aa00640 1 -- 172.21.15.28:0/3021468338 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4c1417310 con 0x55c4c144e950 2024-04-24T08:17:41.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.692+0000 7f197aa00640 1 --2- 172.21.15.28:0/3021468338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144e950 0x55c4c144ed30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f196c009c80 tx=0x7f196c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b5936e91ee3573a5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:41.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.692+0000 7f197a000640 1 --2- 172.21.15.28:0/1315402917 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f19600a3ef0 0x7f19600a63b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1968009830 tx=0x7f1968030ca0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:41.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.692+0000 7f1967e00640 1 -- 172.21.15.28:0/3021468338 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c004230 con 0x55c4c144e950 2024-04-24T08:17:41.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.692+0000 7f1967e00640 1 -- 172.21.15.28:0/3021468338 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f196c0043d0 con 0x55c4c144e950 2024-04-24T08:17:41.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.692+0000 7f1967e00640 1 -- 172.21.15.28:0/3021468338 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c004d80 con 0x55c4c144e950 2024-04-24T08:17:41.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.694+0000 7f198465f740 1 -- 172.21.15.28:0/3021468338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144e950 msgr2=0x55c4c144ed30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:41.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.694+0000 7f198465f740 1 --2- 172.21.15.28:0/3021468338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144e950 0x55c4c144ed30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f196c009c80 tx=0x7f196c011810 comp rx=0 tx=0).stop 2024-04-24T08:17:41.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.696+0000 7f198465f740 1 -- 172.21.15.28:0/3021468338 shutdown_connections 2024-04-24T08:17:41.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.696+0000 7f198465f740 1 --2- 172.21.15.28:0/3021468338 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c144f270 0x55c4c1459400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.696+0000 7f198465f740 1 --2- 172.21.15.28:0/3021468338 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459940 0x55c4c145bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.696+0000 7f198465f740 1 --2- 172.21.15.28:0/3021468338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144e950 0x55c4c144ed30 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.696+0000 7f198465f740 1 -- 172.21.15.28:0/3021468338 >> 172.21.15.28:0/3021468338 conn(0x55c4c144ac70 msgr2=0x55c4c144b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:41.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.696+0000 7f198465f740 1 -- 172.21.15.28:0/3021468338 shutdown_connections 2024-04-24T08:17:41.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.696+0000 7f198465f740 1 -- 172.21.15.28:0/3021468338 wait complete. 2024-04-24T08:17:41.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.698+0000 7f198465f740 1 -- start start 2024-04-24T08:17:41.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.698+0000 7f198465f740 1 Processor -- start 2024-04-24T08:17:41.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.698+0000 7f198465f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144e950 0x55c4c1459350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.698+0000 7f198465f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c144f270 0x55c4c14566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.699+0000 7f197a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c144f270 0x55c4c14566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:41.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.699+0000 7f197a000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c144f270 0x55c4c14566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36294/0 (socket says 172.21.15.28:36294) 2024-04-24T08:17:41.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.700+0000 7f198465f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459940 0x55c4c1456d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.700+0000 7f197aa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144e950 0x55c4c1459350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.700+0000 7f197aa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144e950 0x55c4c1459350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:56336/0 (socket says 172.21.15.28:56336) 2024-04-24T08:17:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.700+0000 7f198465f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55c4c1464fe0 con 0x55c4c144f270 2024-04-24T08:17:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.700+0000 7f197a000640 1 -- 172.21.15.28:0/3275709092 learned_addr learned my addr 172.21.15.28:0/3275709092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.700+0000 7f198465f740 1 -- 172.21.15.28:0/3275709092 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55c4c136d810 con 0x55c4c144e950 2024-04-24T08:17:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.700+0000 7f198465f740 1 -- 172.21.15.28:0/3275709092 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55c4c1465160 con 0x55c4c1459940 2024-04-24T08:17:41.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.700+0000 7f198465f740 10 client.0 _ll_register_callbacks cb 0x55c4c133b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.701+0000 7f197aa00640 1 -- 172.21.15.28:0/3275709092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459940 msgr2=0x55c4c1456d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.701+0000 7f197aa00640 1 --2- 172.21.15.28:0/3275709092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459940 0x55c4c1456d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.701+0000 7f197aa00640 1 -- 172.21.15.28:0/3275709092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c144f270 msgr2=0x55c4c14566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.701+0000 7f197aa00640 1 --2- 172.21.15.28:0/3275709092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c144f270 0x55c4c14566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.701+0000 7f197aa00640 1 -- 172.21.15.28:0/3275709092 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4c137e3f0 con 0x55c4c144e950 2024-04-24T08:17:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.701+0000 7f197a000640 1 --2- 172.21.15.28:0/3275709092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c144f270 0x55c4c14566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:17:41.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.702+0000 7f197aa00640 1 --2- 172.21.15.28:0/3275709092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144e950 0x55c4c1459350 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f196c004880 tx=0x7f196c0199a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e29cf50b52c730c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.702+0000 7f1964c00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:17:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.703+0000 7f1964c00640 1 -- 172.21.15.28:0/3275709092 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c028020 con 0x55c4c144e950 2024-04-24T08:17:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.703+0000 7f1964c00640 1 -- 172.21.15.28:0/3275709092 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7f196c024420 con 0x55c4c144e950 2024-04-24T08:17:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.703+0000 7f198465f740 1 -- 172.21.15.28:0/3275709092 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f196c0043d0 con 0x55c4c144e950 2024-04-24T08:17:41.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.703+0000 7f1964c00640 1 -- 172.21.15.28:0/3275709092 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f196c033050 con 0x55c4c144e950 2024-04-24T08:17:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.704+0000 7f198465f740 10 client.26429 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.704+0000 7f198465f740 1 -- 172.21.15.28:0/3275709092 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c4c10c63f0 con 0x55c4c144e950 2024-04-24T08:17:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.704+0000 7f1964c00640 1 -- 172.21.15.28:0/3275709092 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f196c0043d0 con 0x55c4c144e950 2024-04-24T08:17:41.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.704+0000 7f1964c00640 1 -- 172.21.15.28:0/3275709092 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f196c018070 con 0x55c4c144e950 2024-04-24T08:17:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.704+0000 7f198465f740 10 client.26429 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.704+0000 7f198465f740 10 client.26429 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:17:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.704+0000 7f198465f740 20 client.26429 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.705+0000 7f198465f740 1 --2- 172.21.15.28:0/3275709092 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55c4c14722e0 0x55c4c14746d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.705+0000 7f198465f740 4 client.26429 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:17:41.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.705+0000 7f198465f740 1 -- 172.21.15.28:0/3275709092 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c4c13c1bf0 con 0x55c4c14722e0 2024-04-24T08:17:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.706+0000 7f197a000640 1 --2- 172.21.15.28:0/3275709092 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55c4c14722e0 0x55c4c14746d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:41.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.707+0000 7f197a000640 1 --2- 172.21.15.28:0/3275709092 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55c4c14722e0 0x55c4c14746d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b1df8fc0d1339e6 server_cookie=9927c67ba0133c7d in_seq=0 out_seq=0 2024-04-24T08:17:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.707+0000 7f1964c00640 10 client.26429 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:17:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.708+0000 7f1964c00640 1 -- 172.21.15.28:0/3275709092 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c4c13c1bf0 con 0x55c4c14722e0 2024-04-24T08:17:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.708+0000 7f1964c00640 10 client.26429 handle_command_reply: tid=0 2024-04-24T08:17:41.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.708+0000 7f198465f740 1 client.26429 shutdown 2024-04-24T08:17:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.708+0000 7f198465f740 1 -- 172.21.15.28:0/3275709092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144e950 msgr2=0x55c4c1459350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.708+0000 7f198465f740 1 --2- 172.21.15.28:0/3275709092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144e950 0x55c4c1459350 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f196c004880 tx=0x7f196c0199a0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.708+0000 7f198465f740 1 -- 172.21.15.28:0/3275709092 shutdown_connections 2024-04-24T08:17:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.709+0000 7f198465f740 1 --2- 172.21.15.28:0/3275709092 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55c4c14722e0 0x55c4c14746d0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.709+0000 7f198465f740 1 --2- 172.21.15.28:0/3275709092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c144f270 0x55c4c14566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.709+0000 7f198465f740 1 --2- 172.21.15.28:0/3275709092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459940 0x55c4c1456d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.709+0000 7f1964c00640 0 client.26429 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:17:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.709+0000 7f198465f740 1 --2- 172.21.15.28:0/3275709092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144e950 0x55c4c1459350 secure :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f196c004880 tx=0x7f196c0199a0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.709+0000 7f198465f740 1 -- 172.21.15.28:0/3275709092 >> 172.21.15.28:0/3275709092 conn(0x55c4c144ac70 msgr2=0x55c4c1450970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:41.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.711+0000 7f198465f740 1 -- 172.21.15.28:0/3275709092 shutdown_connections 2024-04-24T08:17:41.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.711+0000 7f198465f740 1 -- 172.21.15.28:0/3275709092 wait complete. 2024-04-24T08:17:41.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.712+0000 7f198465f740 20 client.26429 trim_cache size 0 max 16384 2024-04-24T08:17:41.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.744+0000 7f198465f740 1 Processor -- start 2024-04-24T08:17:41.745 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.744+0000 7f198465f740 1 -- start start 2024-04-24T08:17:41.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.744+0000 7f198465f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144f270 0x55c4c146dea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.744+0000 7f198465f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459940 0x55c4c1458ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.744+0000 7f198465f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c1459430 0x55c4c1479ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.744+0000 7f198465f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f197c1b6e40 con 0x55c4c1459430 2024-04-24T08:17:41.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.744+0000 7f198465f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f197c1b4710 con 0x55c4c144f270 2024-04-24T08:17:41.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.744+0000 7f198465f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f197c1b2620 con 0x55c4c1459940 2024-04-24T08:17:41.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.745+0000 7f197a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459940 0x55c4c1458ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:41.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.745+0000 7f197a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459940 0x55c4c1458ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60274/0 (socket says 172.21.15.28:60274) 2024-04-24T08:17:41.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.745+0000 7f197a000640 1 -- 172.21.15.28:0/787372942 learned_addr learned my addr 172.21.15.28:0/787372942 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:41.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.745+0000 7f197aa00640 1 --2- 172.21.15.28:0/787372942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144f270 0x55c4c146dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:41.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.745+0000 7f197b400640 1 --2- 172.21.15.28:0/787372942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c1459430 0x55c4c1479ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:41.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.745+0000 7f197a000640 1 -- 172.21.15.28:0/787372942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144f270 msgr2=0x55c4c146dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:41.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.745+0000 7f197a000640 1 --2- 172.21.15.28:0/787372942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144f270 0x55c4c146dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.745+0000 7f197a000640 1 -- 172.21.15.28:0/787372942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c1459430 msgr2=0x55c4c1479ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:41.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.745+0000 7f197a000640 1 --2- 172.21.15.28:0/787372942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c1459430 0x55c4c1479ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.748 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.745+0000 7f197a000640 1 -- 172.21.15.28:0/787372942 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c4c11907d0 con 0x55c4c1459940 2024-04-24T08:17:41.749 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.746+0000 7f197a000640 1 --2- 172.21.15.28:0/787372942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459940 0x55c4c1458ef0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1968035040 tx=0x7f196802c000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6bcbb11fe4864a11 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:41.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.746+0000 7f1967400640 1 -- 172.21.15.28:0/787372942 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1968057070 con 0x55c4c1459940 2024-04-24T08:17:41.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.746+0000 7f1967400640 1 -- 172.21.15.28:0/787372942 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1968042040 con 0x55c4c1459940 2024-04-24T08:17:41.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.746+0000 7f1967400640 1 -- 172.21.15.28:0/787372942 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1968037040 con 0x55c4c1459940 2024-04-24T08:17:41.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.747+0000 7f198465f740 1 -- 172.21.15.28:0/787372942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459940 msgr2=0x55c4c1458ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:41.750 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.747+0000 7f198465f740 1 --2- 172.21.15.28:0/787372942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459940 0x55c4c1458ef0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1968035040 tx=0x7f196802c000 comp rx=0 tx=0).stop 2024-04-24T08:17:41.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.748+0000 7f198465f740 1 -- 172.21.15.28:0/787372942 shutdown_connections 2024-04-24T08:17:41.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.748+0000 7f198465f740 1 --2- 172.21.15.28:0/787372942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c1459430 0x55c4c1479ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.748+0000 7f198465f740 1 --2- 172.21.15.28:0/787372942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459940 0x55c4c1458ef0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.748+0000 7f198465f740 1 --2- 172.21.15.28:0/787372942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144f270 0x55c4c146dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.748+0000 7f198465f740 1 -- 172.21.15.28:0/787372942 >> 172.21.15.28:0/787372942 conn(0x55c4c144ac70 msgr2=0x55c4c144b050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:41.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.748+0000 7f198465f740 1 -- 172.21.15.28:0/787372942 shutdown_connections 2024-04-24T08:17:41.751 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.749+0000 7f198465f740 1 -- 172.21.15.28:0/787372942 wait complete. 2024-04-24T08:17:41.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.749+0000 7f198465f740 1 -- start start 2024-04-24T08:17:41.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.750+0000 7f198465f740 1 Processor -- start 2024-04-24T08:17:41.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.750+0000 7f198465f740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144f270 0x55c4c145a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.752 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.750+0000 7f198465f740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459430 0x55c4c145a840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.753 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.750+0000 7f198465f740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c1459940 0x55c4c14693b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.751+0000 7f198465f740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f197c1b2bd0 con 0x55c4c1459940 2024-04-24T08:17:41.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.751+0000 7f198465f740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f197c1b4040 con 0x55c4c144f270 2024-04-24T08:17:41.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.751+0000 7f198465f740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f197c1b32a0 con 0x55c4c1459430 2024-04-24T08:17:41.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.751+0000 7f198465f740 10 client.0 _ll_register_callbacks cb 0x7f196c009b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:41.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.752+0000 7f197a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459430 0x55c4c145a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:41.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.752+0000 7f197a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459430 0x55c4c145a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60288/0 (socket says 172.21.15.28:60288) 2024-04-24T08:17:41.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.752+0000 7f197a000640 1 -- 172.21.15.28:0/1988202053 learned_addr learned my addr 172.21.15.28:0/1988202053 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:41.754 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.752+0000 7f197aa00640 1 --2- 172.21.15.28:0/1988202053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144f270 0x55c4c145a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:41.755 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.753+0000 7f197b400640 1 --2- 172.21.15.28:0/1988202053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c1459940 0x55c4c14693b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:41.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.753+0000 7f197a000640 1 -- 172.21.15.28:0/1988202053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144f270 msgr2=0x55c4c145a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:41.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.753+0000 7f197a000640 1 --2- 172.21.15.28:0/1988202053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144f270 0x55c4c145a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.753+0000 7f197a000640 1 -- 172.21.15.28:0/1988202053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c1459940 msgr2=0x55c4c14693b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:41.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.753+0000 7f197a000640 1 --2- 172.21.15.28:0/1988202053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c1459940 0x55c4c14693b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.753+0000 7f197a000640 1 -- 172.21.15.28:0/1988202053 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c4c1469920 con 0x55c4c1459430 2024-04-24T08:17:41.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.754+0000 7f197a000640 1 --2- 172.21.15.28:0/1988202053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459430 0x55c4c145a840 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1968005e00 tx=0x7f1968051a70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b063187a322e4362 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:41.756 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.754+0000 7f1964c00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:17:41.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.755+0000 7f1964c00640 1 -- 172.21.15.28:0/1988202053 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1968057020 con 0x55c4c1459430 2024-04-24T08:17:41.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.755+0000 7f1964c00640 1 -- 172.21.15.28:0/1988202053 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7f1968051450 con 0x55c4c1459430 2024-04-24T08:17:41.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.755+0000 7f198465f740 1 -- 172.21.15.28:0/1988202053 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1968042040 con 0x55c4c1459430 2024-04-24T08:17:41.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.756+0000 7f1964c00640 1 -- 172.21.15.28:0/1988202053 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1968062050 con 0x55c4c1459430 2024-04-24T08:17:41.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.756+0000 7f198465f740 10 client.26455 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:41.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.756+0000 7f198465f740 1 -- 172.21.15.28:0/1988202053 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55c4c146ac20 con 0x55c4c1459430 2024-04-24T08:17:41.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.757+0000 7f1964c00640 1 -- 172.21.15.28:0/1988202053 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1968042040 con 0x55c4c1459430 2024-04-24T08:17:41.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.758+0000 7f1964c00640 1 -- 172.21.15.28:0/1988202053 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1968047050 con 0x55c4c1459430 2024-04-24T08:17:41.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.758+0000 7f198465f740 10 client.26455 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:41.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.758+0000 7f198465f740 10 client.26455 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:17:41.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.758+0000 7f198465f740 20 client.26455 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:41.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.758+0000 7f198465f740 1 --2- 172.21.15.28:0/1988202053 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55c4c14d8c50 0x55c4c14db070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:41.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.758+0000 7f198465f740 4 client.26455 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:17:41.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.758+0000 7f198465f740 1 -- 172.21.15.28:0/1988202053 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55c4c13bba60 con 0x55c4c14d8c50 2024-04-24T08:17:41.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.758+0000 7f197aa00640 1 --2- 172.21.15.28:0/1988202053 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55c4c14d8c50 0x55c4c14db070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:41.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.759+0000 7f197aa00640 1 --2- 172.21.15.28:0/1988202053 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55c4c14d8c50 0x55c4c14db070 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3150fa2fb16f561d server_cookie=f4f3069baaa548d7 in_seq=0 out_seq=0 2024-04-24T08:17:41.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.759+0000 7f1964c00640 10 client.26455 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:17:41.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.759+0000 7f1964c00640 1 -- 172.21.15.28:0/1988202053 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55c4c13bba60 con 0x55c4c14d8c50 2024-04-24T08:17:41.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.759+0000 7f1964c00640 10 client.26455 handle_command_reply: tid=0 2024-04-24T08:17:41.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.759+0000 7f198465f740 1 client.26455 shutdown 2024-04-24T08:17:41.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.760+0000 7f198465f740 1 -- 172.21.15.28:0/1988202053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459430 msgr2=0x55c4c145a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:41.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.760+0000 7f198465f740 1 --2- 172.21.15.28:0/1988202053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459430 0x55c4c145a840 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1968005e00 tx=0x7f1968051a70 comp rx=0 tx=0).stop 2024-04-24T08:17:41.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.760+0000 7f198465f740 1 -- 172.21.15.28:0/1988202053 shutdown_connections 2024-04-24T08:17:41.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.760+0000 7f198465f740 1 --2- 172.21.15.28:0/1988202053 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55c4c14d8c50 0x55c4c14db070 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.760+0000 7f198465f740 1 --2- 172.21.15.28:0/1988202053 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55c4c1459940 0x55c4c14693b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.760+0000 7f198465f740 1 --2- 172.21.15.28:0/1988202053 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55c4c1459430 0x55c4c145a840 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.760+0000 7f198465f740 1 --2- 172.21.15.28:0/1988202053 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55c4c144f270 0x55c4c145a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.761+0000 7f198465f740 1 -- 172.21.15.28:0/1988202053 >> 172.21.15.28:0/1988202053 conn(0x55c4c144ac70 msgr2=0x55c4c1449560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:41.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.761+0000 7f1964c00640 0 client.26455 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:17:41.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.761+0000 7f198465f740 1 -- 172.21.15.28:0/1988202053 shutdown_connections 2024-04-24T08:17:41.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.762+0000 7f198465f740 1 -- 172.21.15.28:0/1988202053 wait complete. 2024-04-24T08:17:41.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.762+0000 7f198465f740 20 client.26455 trim_cache size 0 max 16384 2024-04-24T08:17:41.764 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:17:41.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.763+0000 7f1981800640 1 -- 172.21.15.28:0/1315402917 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f19600a3ef0 msgr2=0x7f19600a63b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:41.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.763+0000 7f1981800640 1 --2- 172.21.15.28:0/1315402917 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f19600a3ef0 0x7f19600a63b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f1968009830 tx=0x7f1968030ca0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.763+0000 7f1981800640 1 -- 172.21.15.28:0/1315402917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f197c06ceb0 msgr2=0x7f197c06d360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:41.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.763+0000 7f1981800640 1 --2- 172.21.15.28:0/1315402917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f197c06ceb0 0x7f197c06d360 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f197400ef00 tx=0x7f197400c740 comp rx=0 tx=0).stop 2024-04-24T08:17:41.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.763+0000 7f1981800640 1 -- 172.21.15.28:0/1315402917 shutdown_connections 2024-04-24T08:17:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.763+0000 7f1981800640 1 --2- 172.21.15.28:0/1315402917 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f19600a3ef0 0x7f19600a63b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.763+0000 7f1981800640 1 --2- 172.21.15.28:0/1315402917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f197c06ceb0 0x7f197c06d360 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.763+0000 7f1981800640 1 --2- 172.21.15.28:0/1315402917 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f197c109580 0x7f197c06c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.763+0000 7f1981800640 1 --2- 172.21.15.28:0/1315402917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f197c1044b0 0x7f197c0738a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:41.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.763+0000 7f1981800640 1 -- 172.21.15.28:0/1315402917 >> 172.21.15.28:0/1315402917 conn(0x7f197c0ffd90 msgr2=0x7f197c101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:41.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.767+0000 7f1981800640 1 -- 172.21.15.28:0/1315402917 shutdown_connections 2024-04-24T08:17:41.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:41.767+0000 7f1981800640 1 -- 172.21.15.28:0/1315402917 wait complete. 2024-04-24T08:17:41.797 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:17:41.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:41 smithi117 ceph-mon[32438]: pgmap v225: 97 pgs: 97 active+clean; 24 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 46 KiB/s rd, 19 MiB/s wr, 4.35k op/s 2024-04-24T08:17:41.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:41 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/897217917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:17:41.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:41 smithi194 ceph-mon[32836]: pgmap v225: 97 pgs: 97 active+clean; 24 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 46 KiB/s rd, 19 MiB/s wr, 4.35k op/s 2024-04-24T08:17:41.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:41 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/897217917' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:17:42.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.071+0000 7fe49aa00640 1 -- 172.21.15.28:0/3277024075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe494104450 msgr2=0x7fe494104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.071+0000 7fe49aa00640 1 --2- 172.21.15.28:0/3277024075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe494104450 0x7fe494104830 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe484009900 tx=0x7fe48402a420 comp rx=0 tx=0).stop 2024-04-24T08:17:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.071+0000 7fe49aa00640 1 -- 172.21.15.28:0/3277024075 shutdown_connections 2024-04-24T08:17:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.071+0000 7fe49aa00640 1 --2- 172.21.15.28:0/3277024075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe494104e00 0x7fe494108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.071+0000 7fe49aa00640 1 --2- 172.21.15.28:0/3277024075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe494109520 0x7fe4941110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.071+0000 7fe49aa00640 1 --2- 172.21.15.28:0/3277024075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe494104450 0x7fe494104830 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.071+0000 7fe49aa00640 1 -- 172.21.15.28:0/3277024075 >> 172.21.15.28:0/3277024075 conn(0x7fe4940ffc70 msgr2=0x7fe494102090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:42.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.071+0000 7fe49aa00640 1 -- 172.21.15.28:0/3277024075 shutdown_connections 2024-04-24T08:17:42.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.071+0000 7fe49aa00640 1 -- 172.21.15.28:0/3277024075 wait complete. 2024-04-24T08:17:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.072+0000 7fe49aa00640 1 Processor -- start 2024-04-24T08:17:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.073+0000 7fe49aa00640 1 -- start start 2024-04-24T08:17:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.073+0000 7fe49aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe494104450 0x7fe494111060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.073+0000 7fe49aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe494109520 0x7fe49410e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.073+0000 7fe49aa00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe49410e840 0x7fe4941ab460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.073+0000 7fe49aa00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe494114720 con 0x7fe494109520 2024-04-24T08:17:42.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.073+0000 7fe49aa00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe4941145a0 con 0x7fe49410e840 2024-04-24T08:17:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.073+0000 7fe49aa00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe4941148a0 con 0x7fe494104450 2024-04-24T08:17:42.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.074+0000 7fe493e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe494104450 0x7fe494111060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.074+0000 7fe493e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe494104450 0x7fe494111060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60318/0 (socket says 172.21.15.28:60318) 2024-04-24T08:17:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.074+0000 7fe493e00640 1 -- 172.21.15.28:0/2803292426 learned_addr learned my addr 172.21.15.28:0/2803292426 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.074+0000 7fe493400640 1 --2- 172.21.15.28:0/2803292426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe494109520 0x7fe49410e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.074+0000 7fe498a00640 1 --2- 172.21.15.28:0/2803292426 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe49410e840 0x7fe4941ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.074+0000 7fe493e00640 1 -- 172.21.15.28:0/2803292426 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe49410e840 msgr2=0x7fe4941ab460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.074+0000 7fe493e00640 1 --2- 172.21.15.28:0/2803292426 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe49410e840 0x7fe4941ab460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.074+0000 7fe493e00640 1 -- 172.21.15.28:0/2803292426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe494109520 msgr2=0x7fe49410e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.074+0000 7fe493e00640 1 --2- 172.21.15.28:0/2803292426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe494109520 0x7fe49410e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.074+0000 7fe493e00640 1 -- 172.21.15.28:0/2803292426 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe49410efe0 con 0x7fe494104450 2024-04-24T08:17:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.075+0000 7fe493e00640 1 --2- 172.21.15.28:0/2803292426 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe494104450 0x7fe494111060 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe48402a990 tx=0x7fe484003c20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.075+0000 7fe490c00640 1 -- 172.21.15.28:0/2803292426 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48403f070 con 0x7fe494104450 2024-04-24T08:17:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.075+0000 7fe490c00640 1 -- 172.21.15.28:0/2803292426 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe48403b460 con 0x7fe494104450 2024-04-24T08:17:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.075+0000 7fe490c00640 1 -- 172.21.15.28:0/2803292426 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe484035030 con 0x7fe494104450 2024-04-24T08:17:42.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.076+0000 7fe49aa00640 1 -- 172.21.15.28:0/2803292426 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4941aba00 con 0x7fe494104450 2024-04-24T08:17:42.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.076+0000 7fe49aa00640 1 -- 172.21.15.28:0/2803292426 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4941abf00 con 0x7fe494104450 2024-04-24T08:17:42.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.080+0000 7fe49d7a8740 1 Processor -- start 2024-04-24T08:17:42.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.081+0000 7fe49d7a8740 1 -- start start 2024-04-24T08:17:42.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.081+0000 7fe49d7a8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf0950 0x55be5bbf0d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.081+0000 7fe49d7a8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf1270 0x55be5bbfb400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.081+0000 7fe49d7a8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbfb940 0x55be5bbfdd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.081+0000 7fe49d7a8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55be5bb561d0 con 0x55be5bbfb940 2024-04-24T08:17:42.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.081+0000 7fe49d7a8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55be5bafc450 con 0x55be5bbf1270 2024-04-24T08:17:42.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.081+0000 7fe49d7a8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55be5bb52190 con 0x55be5bbf0950 2024-04-24T08:17:42.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.081+0000 7fe490c00640 1 -- 172.21.15.28:0/2803292426 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe484042070 con 0x7fe494104450 2024-04-24T08:17:42.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.082+0000 7fe493400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbfb940 0x55be5bbfdd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.082+0000 7fe493400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbfb940 0x55be5bbfdd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36332/0 (socket says 172.21.15.28:36332) 2024-04-24T08:17:42.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.082+0000 7fe493400640 1 -- 172.21.15.28:0/2979744117 learned_addr learned my addr 172.21.15.28:0/2979744117 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:42.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.082+0000 7fe498a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf0950 0x55be5bbf0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.083+0000 7fe493e00640 1 --2- 172.21.15.28:0/2979744117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf1270 0x55be5bbfb400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.083+0000 7fe498a00640 1 -- 172.21.15.28:0/2979744117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf1270 msgr2=0x55be5bbfb400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:42.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.083+0000 7fe498a00640 1 --2- 172.21.15.28:0/2979744117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf1270 0x55be5bbfb400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.083+0000 7fe498a00640 1 -- 172.21.15.28:0/2979744117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbfb940 msgr2=0x55be5bbfdd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:42.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.083+0000 7fe498a00640 1 --2- 172.21.15.28:0/2979744117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbfb940 0x55be5bbfdd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.083+0000 7fe498a00640 1 -- 172.21.15.28:0/2979744117 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be5bbb9310 con 0x55be5bbf0950 2024-04-24T08:17:42.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.083+0000 7fe493e00640 1 --2- 172.21.15.28:0/2979744117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf1270 0x55be5bbfb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:17:42.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.083+0000 7fe493400640 1 --2- 172.21.15.28:0/2979744117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbfb940 0x55be5bbfdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:17:42.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.085+0000 7fe490c00640 1 --2- 172.21.15.28:0/2803292426 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4780a3ea0 0x7fe4780a6360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.085+0000 7fe490c00640 1 -- 172.21.15.28:0/2803292426 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7fe4840f0f40 con 0x7fe494104450 2024-04-24T08:17:42.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.085+0000 7fe493e00640 1 --2- 172.21.15.28:0/2803292426 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4780a3ea0 0x7fe4780a6360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.087 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.085+0000 7fe498a00640 1 --2- 172.21.15.28:0/2979744117 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf0950 0x55be5bbf0d30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009e80 tx=0x7fe48c00c7d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0789f5fbc6eb24f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.086+0000 7fe489200640 1 -- 172.21.15.28:0/2979744117 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48c01a070 con 0x55be5bbf0950 2024-04-24T08:17:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.086+0000 7fe489200640 1 -- 172.21.15.28:0/2979744117 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe48c00aad0 con 0x55be5bbf0950 2024-04-24T08:17:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.086+0000 7fe489200640 1 -- 172.21.15.28:0/2979744117 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48c015c30 con 0x55be5bbf0950 2024-04-24T08:17:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.087+0000 7fe493e00640 1 --2- 172.21.15.28:0/2803292426 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4780a3ea0 0x7fe4780a6360 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe4840ee6f0 tx=0x7fe484009260 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:42.088 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.087+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2979744117 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf0950 msgr2=0x55be5bbf0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.087+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/2979744117 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf0950 0x55be5bbf0d30 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009e80 tx=0x7fe48c00c7d0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.089 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.088+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2979744117 shutdown_connections 2024-04-24T08:17:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.088+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/2979744117 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbfb940 0x55be5bbfdd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.088+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/2979744117 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf1270 0x55be5bbfb400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.088+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/2979744117 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf0950 0x55be5bbf0d30 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.088+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2979744117 >> 172.21.15.28:0/2979744117 conn(0x55be5bbecc70 msgr2=0x55be5bbed050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.088+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2979744117 shutdown_connections 2024-04-24T08:17:42.090 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.088+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2979744117 wait complete. 2024-04-24T08:17:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.089+0000 7fe49d7a8740 1 -- start start 2024-04-24T08:17:42.091 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.089+0000 7fe49d7a8740 1 Processor -- start 2024-04-24T08:17:42.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.089+0000 7fe49d7a8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf0950 0x55be5bbf8710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.090+0000 7fe49d7a8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbf1270 0x55be5bbf8c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.092 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.090+0000 7fe49d7a8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbfb940 0x55be5bbf9190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.090+0000 7fe49d7a8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55be5bc0f7e0 con 0x55be5bbf1270 2024-04-24T08:17:42.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.090+0000 7fe49d7a8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55be5bb0f810 con 0x55be5bbf0950 2024-04-24T08:17:42.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.090+0000 7fe49d7a8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55be5bc0f960 con 0x55be5bbfb940 2024-04-24T08:17:42.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.090+0000 7fe49d7a8740 10 client.0 _ll_register_callbacks cb 0x55be5badd140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:42.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.090+0000 7fe498a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf0950 0x55be5bbf8710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.090+0000 7fe498a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf0950 0x55be5bbf8710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:56382/0 (socket says 172.21.15.28:56382) 2024-04-24T08:17:42.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.090+0000 7fe498a00640 1 -- 172.21.15.28:0/1460037375 learned_addr learned my addr 172.21.15.28:0/1460037375 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:42.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.090+0000 7fe498a00640 1 --2- 172.21.15.28:0/1460037375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbfb940 0x55be5bbf9190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.093 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.090+0000 7fe498a00640 1 -- 172.21.15.28:0/1460037375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbfb940 msgr2=0x55be5bbf9190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:42.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.090+0000 7fe498a00640 1 --2- 172.21.15.28:0/1460037375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbfb940 0x55be5bbf9190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.091+0000 7fe498a00640 1 -- 172.21.15.28:0/1460037375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbf1270 msgr2=0x55be5bbf8c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:42.094 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.091+0000 7fe498a00640 1 --2- 172.21.15.28:0/1460037375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbf1270 0x55be5bbf8c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.091+0000 7fe498a00640 1 -- 172.21.15.28:0/1460037375 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be5bb203f0 con 0x55be5bbf0950 2024-04-24T08:17:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.091+0000 7fe498a00640 1 --2- 172.21.15.28:0/1460037375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf0950 0x55be5bbf8710 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe48c015530 tx=0x7fe48c0157d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4527ccc75b4c540e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:42.095 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.091+0000 7fe47d600640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:17:42.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.091+0000 7fe47d600640 1 -- 172.21.15.28:0/1460037375 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48c01a070 con 0x55be5bbf0950 2024-04-24T08:17:42.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.091+0000 7fe49d7a8740 1 -- 172.21.15.28:0/1460037375 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe48c00aad0 con 0x55be5bbf0950 2024-04-24T08:17:42.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.091+0000 7fe47d600640 1 -- 172.21.15.28:0/1460037375 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7fe48c024400 con 0x55be5bbf0950 2024-04-24T08:17:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.092+0000 7fe47d600640 1 -- 172.21.15.28:0/1460037375 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe48c00adc0 con 0x55be5bbf0950 2024-04-24T08:17:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.092+0000 7fe49d7a8740 10 client.26441 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.092+0000 7fe49d7a8740 1 -- 172.21.15.28:0/1460037375 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55be5b8683f0 con 0x55be5bbf0950 2024-04-24T08:17:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.092+0000 7fe47d600640 1 -- 172.21.15.28:0/1460037375 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe48c00aad0 con 0x55be5bbf0950 2024-04-24T08:17:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.092+0000 7fe47d600640 1 -- 172.21.15.28:0/1460037375 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fe48c02c5a0 con 0x55be5bbf0950 2024-04-24T08:17:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.092+0000 7fe49d7a8740 10 client.26441 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.092+0000 7fe49d7a8740 10 client.26441 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:17:42.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.092+0000 7fe49d7a8740 20 client.26441 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:42.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.092+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/1460037375 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55be5bc1ca80 0x55be5bc1ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.092+0000 7fe49d7a8740 4 client.26441 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:17:42.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.092+0000 7fe49d7a8740 1 -- 172.21.15.28:0/1460037375 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be5bb63bf0 con 0x55be5bc1ca80 2024-04-24T08:17:42.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.093+0000 7fe493400640 1 --2- 172.21.15.28:0/1460037375 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55be5bc1ca80 0x55be5bc1ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.094+0000 7fe47d600640 10 client.26441 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:17:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.097+0000 7fe493400640 1 --2- 172.21.15.28:0/1460037375 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55be5bc1ca80 0x55be5bc1ee70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=45e6395d83e8bbd3 server_cookie=d72f18e04d5b7f9c in_seq=0 out_seq=0 2024-04-24T08:17:42.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.098+0000 7fe47d600640 1 -- 172.21.15.28:0/1460037375 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55be5bb63bf0 con 0x55be5bc1ca80 2024-04-24T08:17:42.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.098+0000 7fe47d600640 10 client.26441 handle_command_reply: tid=0 2024-04-24T08:17:42.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.098+0000 7fe49d7a8740 1 client.26441 shutdown 2024-04-24T08:17:42.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.098+0000 7fe49d7a8740 1 -- 172.21.15.28:0/1460037375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf0950 msgr2=0x55be5bbf8710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.098+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/1460037375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf0950 0x55be5bbf8710 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fe48c015530 tx=0x7fe48c0157d0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.098+0000 7fe49d7a8740 1 -- 172.21.15.28:0/1460037375 shutdown_connections 2024-04-24T08:17:42.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.098+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/1460037375 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55be5bc1ca80 0x55be5bc1ee70 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.098+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/1460037375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbf1270 0x55be5bbf8c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.098+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/1460037375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbfb940 0x55be5bbf9190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.098+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/1460037375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf0950 0x55be5bbf8710 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.098+0000 7fe49d7a8740 1 -- 172.21.15.28:0/1460037375 >> 172.21.15.28:0/1460037375 conn(0x55be5bbecc70 msgr2=0x55be5bbf2a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.098+0000 7fe47d600640 0 client.26441 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:17:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.099+0000 7fe49d7a8740 1 -- 172.21.15.28:0/1460037375 shutdown_connections 2024-04-24T08:17:42.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.099+0000 7fe49d7a8740 1 -- 172.21.15.28:0/1460037375 wait complete. 2024-04-24T08:17:42.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.099+0000 7fe49d7a8740 20 client.26441 trim_cache size 0 max 16384 2024-04-24T08:17:42.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.117+0000 7fe49d7a8740 1 Processor -- start 2024-04-24T08:17:42.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.117+0000 7fe49d7a8740 1 -- start start 2024-04-24T08:17:42.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.117+0000 7fe49d7a8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf0950 0x55be5bc186a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.117+0000 7fe49d7a8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf1270 0x55be5bbf8fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.117+0000 7fe49d7a8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbfb940 0x55be5bbebc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.117+0000 7fe49d7a8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4941ae5f0 con 0x55be5bbfb940 2024-04-24T08:17:42.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.117+0000 7fe49d7a8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe4941a2180 con 0x55be5bbf0950 2024-04-24T08:17:42.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.117+0000 7fe49d7a8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe49419fff0 con 0x55be5bbf1270 2024-04-24T08:17:42.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.118+0000 7fe498a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf0950 0x55be5bc186a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.118+0000 7fe493400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf1270 0x55be5bbf8fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.118+0000 7fe498a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf0950 0x55be5bc186a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:56392/0 (socket says 172.21.15.28:56392) 2024-04-24T08:17:42.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.118+0000 7fe493400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf1270 0x55be5bbf8fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60334/0 (socket says 172.21.15.28:60334) 2024-04-24T08:17:42.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.118+0000 7fe498a00640 1 -- 172.21.15.28:0/2874458448 learned_addr learned my addr 172.21.15.28:0/2874458448 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:42.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.118+0000 7fe498a00640 1 --2- 172.21.15.28:0/2874458448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbfb940 0x55be5bbebc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.118+0000 7fe498a00640 1 -- 172.21.15.28:0/2874458448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf1270 msgr2=0x55be5bbf8fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:42.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.118+0000 7fe498a00640 1 --2- 172.21.15.28:0/2874458448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf1270 0x55be5bbf8fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.118+0000 7fe498a00640 1 -- 172.21.15.28:0/2874458448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbfb940 msgr2=0x55be5bbebc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.118+0000 7fe498a00640 1 --2- 172.21.15.28:0/2874458448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbfb940 0x55be5bbebc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.118+0000 7fe498a00640 1 -- 172.21.15.28:0/2874458448 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be5bbf10a0 con 0x55be5bbf0950 2024-04-24T08:17:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.118+0000 7fe493400640 1 --2- 172.21.15.28:0/2874458448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf1270 0x55be5bbf8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:17:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.118+0000 7fe498a00640 1 --2- 172.21.15.28:0/2874458448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf0950 0x55be5bc186a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0022f0 tx=0x7fe48c00c7d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43558fe33f5e0c48 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.119+0000 7fe47fe00640 1 -- 172.21.15.28:0/2874458448 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48c01a070 con 0x55be5bbf0950 2024-04-24T08:17:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.119+0000 7fe47fe00640 1 -- 172.21.15.28:0/2874458448 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe48c00aa30 con 0x55be5bbf0950 2024-04-24T08:17:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.119+0000 7fe47fe00640 1 -- 172.21.15.28:0/2874458448 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48c00a240 con 0x55be5bbf0950 2024-04-24T08:17:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.119+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2874458448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf0950 msgr2=0x55be5bc186a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.119+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/2874458448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf0950 0x55be5bc186a0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7fe48c0022f0 tx=0x7fe48c00c7d0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.119+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2874458448 shutdown_connections 2024-04-24T08:17:42.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.119+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/2874458448 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbfb940 0x55be5bbebc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.119+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/2874458448 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf1270 0x55be5bbf8fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.119+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/2874458448 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbf0950 0x55be5bc186a0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.119+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2874458448 >> 172.21.15.28:0/2874458448 conn(0x55be5bbecc70 msgr2=0x55be5bbeb560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:42.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.120+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2874458448 shutdown_connections 2024-04-24T08:17:42.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.120+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2874458448 wait complete. 2024-04-24T08:17:42.124 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.120+0000 7fe49d7a8740 1 -- start start 2024-04-24T08:17:42.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.120+0000 7fe49d7a8740 1 Processor -- start 2024-04-24T08:17:42.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.121+0000 7fe49d7a8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf0950 0x55be5bc14bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.121+0000 7fe49d7a8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbf1270 0x55be5bc150f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.121+0000 7fe49d7a8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbfb940 0x55be5bbf5990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.121+0000 7fe49d7a8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4941a0640 con 0x55be5bbf1270 2024-04-24T08:17:42.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.121+0000 7fe49d7a8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe4941a1ab0 con 0x55be5bbfb940 2024-04-24T08:17:42.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.121+0000 7fe49d7a8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe4941a0d10 con 0x55be5bbf0950 2024-04-24T08:17:42.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.121+0000 7fe49d7a8740 10 client.0 _ll_register_callbacks cb 0x55be5bbfb890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:42.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.121+0000 7fe498a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf0950 0x55be5bc14bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.121+0000 7fe498a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf0950 0x55be5bc14bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60340/0 (socket says 172.21.15.28:60340) 2024-04-24T08:17:42.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.121+0000 7fe498a00640 1 -- 172.21.15.28:0/2435870296 learned_addr learned my addr 172.21.15.28:0/2435870296 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:42.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.121+0000 7fe498a00640 1 --2- 172.21.15.28:0/2435870296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbfb940 0x55be5bbf5990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.121+0000 7fe498a00640 1 -- 172.21.15.28:0/2435870296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbfb940 msgr2=0x55be5bbf5990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:42.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.121+0000 7fe498a00640 1 --2- 172.21.15.28:0/2435870296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbfb940 0x55be5bbf5990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.121+0000 7fe498a00640 1 -- 172.21.15.28:0/2435870296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbf1270 msgr2=0x55be5bc150f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:42.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.121+0000 7fe498a00640 1 --2- 172.21.15.28:0/2435870296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbf1270 0x55be5bc150f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.121+0000 7fe498a00640 1 -- 172.21.15.28:0/2435870296 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be5b9327d0 con 0x55be5bbf0950 2024-04-24T08:17:42.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.122+0000 7fe498a00640 1 --2- 172.21.15.28:0/2435870296 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf0950 0x55be5bc14bb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009c50 tx=0x7fe48c02f010 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86ec1de4d38ee77d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:42.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.122+0000 7fe47d600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:17:42.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.122+0000 7fe47d600640 1 -- 172.21.15.28:0/2435870296 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe48c01a070 con 0x55be5bbf0950 2024-04-24T08:17:42.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.122+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2435870296 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe48c00aa30 con 0x55be5bbf0950 2024-04-24T08:17:42.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.122+0000 7fe47d600640 1 -- 172.21.15.28:0/2435870296 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7fe48c00a1e0 con 0x55be5bbf0950 2024-04-24T08:17:42.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.122+0000 7fe47d600640 1 -- 172.21.15.28:0/2435870296 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe48c00abe0 con 0x55be5bbf0950 2024-04-24T08:17:42.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.122+0000 7fe49d7a8740 10 client.26473 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:42.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.122+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2435870296 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55be5bbfc640 con 0x55be5bbf0950 2024-04-24T08:17:42.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.122+0000 7fe47d600640 1 -- 172.21.15.28:0/2435870296 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe48c00aa30 con 0x55be5bbf0950 2024-04-24T08:17:42.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.123+0000 7fe47d600640 1 -- 172.21.15.28:0/2435870296 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fe48c0041f0 con 0x55be5bbf0950 2024-04-24T08:17:42.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.123+0000 7fe49d7a8740 10 client.26473 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:42.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.123+0000 7fe49d7a8740 10 client.26473 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:17:42.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.123+0000 7fe49d7a8740 20 client.26473 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:42.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.123+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/2435870296 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55be5bc82e20 0x55be5bc85210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.123+0000 7fe49d7a8740 4 client.26473 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:17:42.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.123+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2435870296 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55be5bb5da60 con 0x55be5bc82e20 2024-04-24T08:17:42.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.123+0000 7fe493400640 1 --2- 172.21.15.28:0/2435870296 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55be5bc82e20 0x55be5bc85210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.124+0000 7fe493400640 1 --2- 172.21.15.28:0/2435870296 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55be5bc82e20 0x55be5bc85210 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4243406a916868da server_cookie=a4ce72054807b77 in_seq=0 out_seq=0 2024-04-24T08:17:42.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.124+0000 7fe47d600640 10 client.26473 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:17:42.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.124+0000 7fe47d600640 1 -- 172.21.15.28:0/2435870296 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55be5bb5da60 con 0x55be5bc82e20 2024-04-24T08:17:42.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.124+0000 7fe47d600640 10 client.26473 handle_command_reply: tid=0 2024-04-24T08:17:42.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.124+0000 7fe49d7a8740 1 client.26473 shutdown 2024-04-24T08:17:42.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.124+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2435870296 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf0950 msgr2=0x55be5bc14bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.124+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/2435870296 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf0950 0x55be5bc14bb0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7fe48c009c50 tx=0x7fe48c02f010 comp rx=0 tx=0).stop 2024-04-24T08:17:42.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.124+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2435870296 shutdown_connections 2024-04-24T08:17:42.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.124+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/2435870296 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55be5bc82e20 0x55be5bc85210 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.124+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/2435870296 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be5bbf1270 0x55be5bc150f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.124+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/2435870296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be5bbfb940 0x55be5bbf5990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.124+0000 7fe49d7a8740 1 --2- 172.21.15.28:0/2435870296 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be5bbf0950 0x55be5bc14bb0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.124+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2435870296 >> 172.21.15.28:0/2435870296 conn(0x55be5bbecc70 msgr2=0x55be5bbebdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:42.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.124+0000 7fe47d600640 0 client.26473 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:17:42.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.124+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2435870296 shutdown_connections 2024-04-24T08:17:42.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.124+0000 7fe49d7a8740 1 -- 172.21.15.28:0/2435870296 wait complete. 2024-04-24T08:17:42.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.125+0000 7fe49d7a8740 20 client.26473 trim_cache size 0 max 16384 2024-04-24T08:17:42.135 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:17:42.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.125+0000 7fe49aa00640 1 -- 172.21.15.28:0/2803292426 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4780a3ea0 msgr2=0x7fe4780a6360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.125+0000 7fe49aa00640 1 --2- 172.21.15.28:0/2803292426 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4780a3ea0 0x7fe4780a6360 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fe4840ee6f0 tx=0x7fe484009260 comp rx=0 tx=0).stop 2024-04-24T08:17:42.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.125+0000 7fe49aa00640 1 -- 172.21.15.28:0/2803292426 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe494104450 msgr2=0x7fe494111060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.125+0000 7fe49aa00640 1 --2- 172.21.15.28:0/2803292426 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe494104450 0x7fe494111060 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fe48402a990 tx=0x7fe484003c20 comp rx=0 tx=0).stop 2024-04-24T08:17:42.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.125+0000 7fe49aa00640 1 -- 172.21.15.28:0/2803292426 shutdown_connections 2024-04-24T08:17:42.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.125+0000 7fe49aa00640 1 --2- 172.21.15.28:0/2803292426 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4780a3ea0 0x7fe4780a6360 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.125+0000 7fe49aa00640 1 --2- 172.21.15.28:0/2803292426 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe494109520 0x7fe49410e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.125+0000 7fe49aa00640 1 --2- 172.21.15.28:0/2803292426 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe49410e840 0x7fe4941ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.125+0000 7fe49aa00640 1 --2- 172.21.15.28:0/2803292426 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe494104450 0x7fe494111060 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.125+0000 7fe49aa00640 1 -- 172.21.15.28:0/2803292426 >> 172.21.15.28:0/2803292426 conn(0x7fe4940ffc70 msgr2=0x7fe4941010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:42.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.126+0000 7fe49aa00640 1 -- 172.21.15.28:0/2803292426 shutdown_connections 2024-04-24T08:17:42.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.126+0000 7fe49aa00640 1 -- 172.21.15.28:0/2803292426 wait complete. 2024-04-24T08:17:42.146 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:17:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.319+0000 7fed26c00640 1 -- 172.21.15.28:0/449392694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed2010d500 msgr2=0x7fed2010d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.319+0000 7fed26c00640 1 --2- 172.21.15.28:0/449392694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed2010d500 0x7fed2010d8e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fed14009930 tx=0x7fed1402a420 comp rx=0 tx=0).stop 2024-04-24T08:17:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.319+0000 7fed26c00640 1 -- 172.21.15.28:0/449392694 shutdown_connections 2024-04-24T08:17:42.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.319+0000 7fed26c00640 1 --2- 172.21.15.28:0/449392694 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed20108f40 0x7fed201153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.319+0000 7fed26c00640 1 --2- 172.21.15.28:0/449392694 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed20108580 0x7fed20108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.319+0000 7fed26c00640 1 --2- 172.21.15.28:0/449392694 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed2010d500 0x7fed2010d8e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.319+0000 7fed26c00640 1 -- 172.21.15.28:0/449392694 >> 172.21.15.28:0/449392694 conn(0x7fed20077e20 msgr2=0x7fed20106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:42.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.319+0000 7fed26c00640 1 -- 172.21.15.28:0/449392694 shutdown_connections 2024-04-24T08:17:42.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.319+0000 7fed26c00640 1 -- 172.21.15.28:0/449392694 wait complete. 2024-04-24T08:17:42.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.320+0000 7fed26c00640 1 Processor -- start 2024-04-24T08:17:42.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.320+0000 7fed26c00640 1 -- start start 2024-04-24T08:17:42.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.321+0000 7fed26c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed20108580 0x7fed2019ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.321+0000 7fed26c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed20108f40 0x7fed2019f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.321+0000 7fed26c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed2010d500 0x7fed201a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.321+0000 7fed26c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed201188c0 con 0x7fed20108f40 2024-04-24T08:17:42.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.321+0000 7fed26c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fed20118740 con 0x7fed20108580 2024-04-24T08:17:42.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.321+0000 7fed26c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fed20118a40 con 0x7fed2010d500 2024-04-24T08:17:42.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.321+0000 7fed26200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed2010d500 0x7fed201a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.321+0000 7fed24e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed20108f40 0x7fed2019f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.321+0000 7fed24e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed20108f40 0x7fed2019f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36376/0 (socket says 172.21.15.28:36376) 2024-04-24T08:17:42.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.321+0000 7fed24e00640 1 -- 172.21.15.28:0/1114747568 learned_addr learned my addr 172.21.15.28:0/1114747568 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:42.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.321+0000 7fed26200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed2010d500 0x7fed201a3600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60358/0 (socket says 172.21.15.28:60358) 2024-04-24T08:17:42.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.321+0000 7fed25800640 1 --2- 172.21.15.28:0/1114747568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed20108580 0x7fed2019ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.321+0000 7fed24e00640 1 -- 172.21.15.28:0/1114747568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed2010d500 msgr2=0x7fed201a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.321+0000 7fed24e00640 1 --2- 172.21.15.28:0/1114747568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed2010d500 0x7fed201a3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.321+0000 7fed24e00640 1 -- 172.21.15.28:0/1114747568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed20108580 msgr2=0x7fed2019ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.321+0000 7fed24e00640 1 --2- 172.21.15.28:0/1114747568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed20108580 0x7fed2019ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.322+0000 7fed24e00640 1 -- 172.21.15.28:0/1114747568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed201a3d30 con 0x7fed20108f40 2024-04-24T08:17:42.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.322+0000 7fed24e00640 1 --2- 172.21.15.28:0/1114747568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed20108f40 0x7fed2019f280 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00e9f0 tx=0x7fed0c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:42.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.322+0000 7fed1e000640 1 -- 172.21.15.28:0/1114747568 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c00cd80 con 0x7fed20108f40 2024-04-24T08:17:42.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.322+0000 7fed26c00640 1 -- 172.21.15.28:0/1114747568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed201a4020 con 0x7fed20108f40 2024-04-24T08:17:42.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.322+0000 7fed1e000640 1 -- 172.21.15.28:0/1114747568 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed0c005500 con 0x7fed20108f40 2024-04-24T08:17:42.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.322+0000 7fed1e000640 1 -- 172.21.15.28:0/1114747568 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c010690 con 0x7fed20108f40 2024-04-24T08:17:42.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.322+0000 7fed26c00640 1 -- 172.21.15.28:0/1114747568 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed201ab8b0 con 0x7fed20108f40 2024-04-24T08:17:42.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.325+0000 7fed2b187740 1 Processor -- start 2024-04-24T08:17:42.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.325+0000 7fed2b187740 1 -- start start 2024-04-24T08:17:42.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.326+0000 7fed2b187740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87181b40 0x55ce87181f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.326+0000 7fed2b187740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 0x55ce8718c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.326+0000 7fed1e000640 1 -- 172.21.15.28:0/1114747568 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fed0c0046c0 con 0x7fed20108f40 2024-04-24T08:17:42.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.326+0000 7fed2b187740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 0x55ce8718ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.326+0000 7fed2b187740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce870e71d0 con 0x55ce8718cb30 2024-04-24T08:17:42.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.326+0000 7fed2b187740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ce8708d450 con 0x55ce87182460 2024-04-24T08:17:42.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.326+0000 7fed2b187740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ce870e3190 con 0x55ce87181b40 2024-04-24T08:17:42.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.326+0000 7fed25800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 0x55ce8718c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.326+0000 7fed26200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87181b40 0x55ce87181f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.326+0000 7fed24e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 0x55ce8718ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.326+0000 7fed26200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87181b40 0x55ce87181f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60364/0 (socket says 172.21.15.28:60364) 2024-04-24T08:17:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.326+0000 7fed26200640 1 -- 172.21.15.28:0/1901752544 learned_addr learned my addr 172.21.15.28:0/1901752544 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.326+0000 7fed25800640 1 --2- 172.21.15.28:0/1901752544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 0x55ce8718c5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:56422/0 (socket says 172.21.15.28:56422) 2024-04-24T08:17:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.326+0000 7fed26200640 1 -- 172.21.15.28:0/1901752544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 msgr2=0x55ce8718c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.326+0000 7fed26200640 1 --2- 172.21.15.28:0/1901752544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 0x55ce8718c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.326+0000 7fed26200640 1 -- 172.21.15.28:0/1901752544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 msgr2=0x55ce8718ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:42.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.326+0000 7fed26200640 1 --2- 172.21.15.28:0/1901752544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 0x55ce8718ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.327+0000 7fed26200640 1 -- 172.21.15.28:0/1901752544 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce8714a310 con 0x55ce87181b40 2024-04-24T08:17:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.327+0000 7fed26200640 1 --2- 172.21.15.28:0/1901752544 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87181b40 0x55ce87181f20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fed18009fc0 tx=0x7fed180118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68bfb233df9bd2b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.327+0000 7fed12a00640 1 -- 172.21.15.28:0/1901752544 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1801e070 con 0x55ce87181b40 2024-04-24T08:17:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.327+0000 7fed12a00640 1 -- 172.21.15.28:0/1901752544 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed18003c30 con 0x55ce87181b40 2024-04-24T08:17:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.327+0000 7fed12a00640 1 -- 172.21.15.28:0/1901752544 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1800f9e0 con 0x55ce87181b40 2024-04-24T08:17:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.327+0000 7fed1e000640 1 --2- 172.21.15.28:0/1114747568 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fed080a3d20 0x7fed080a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.328+0000 7fed1e000640 1 -- 172.21.15.28:0/1114747568 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7fed0c0d0ca0 con 0x7fed20108f40 2024-04-24T08:17:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.328+0000 7fed25800640 1 --2- 172.21.15.28:0/1114747568 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fed080a3d20 0x7fed080a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.328+0000 7fed2b187740 1 -- 172.21.15.28:0/1901752544 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87181b40 msgr2=0x55ce87181f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.328+0000 7fed2b187740 1 --2- 172.21.15.28:0/1901752544 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87181b40 0x55ce87181f20 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7fed18009fc0 tx=0x7fed180118d0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.328+0000 7fed2b187740 1 -- 172.21.15.28:0/1901752544 shutdown_connections 2024-04-24T08:17:42.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.328+0000 7fed2b187740 1 --2- 172.21.15.28:0/1901752544 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 0x55ce8718ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.328+0000 7fed2b187740 1 --2- 172.21.15.28:0/1901752544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 0x55ce8718c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.328+0000 7fed2b187740 1 --2- 172.21.15.28:0/1901752544 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87181b40 0x55ce87181f20 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.328+0000 7fed2b187740 1 -- 172.21.15.28:0/1901752544 >> 172.21.15.28:0/1901752544 conn(0x55ce8717de60 msgr2=0x55ce8717e240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:42.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.328+0000 7fed2b187740 1 -- 172.21.15.28:0/1901752544 shutdown_connections 2024-04-24T08:17:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.328+0000 7fed2b187740 1 -- 172.21.15.28:0/1901752544 wait complete. 2024-04-24T08:17:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.328+0000 7fed25800640 1 --2- 172.21.15.28:0/1114747568 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fed080a3d20 0x7fed080a61e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fed14004770 tx=0x7fed140058d0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.329+0000 7fed2b187740 1 -- start start 2024-04-24T08:17:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.329+0000 7fed2b187740 1 Processor -- start 2024-04-24T08:17:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.329+0000 7fed2b187740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87181b40 0x55ce8718c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.329+0000 7fed2b187740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87182460 0x55ce871898a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.329+0000 7fed2b187740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 0x55ce87189fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.329+0000 7fed2b187740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ce8718f9c0 con 0x55ce8718cb30 2024-04-24T08:17:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.329+0000 7fed2b187740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ce870a0810 con 0x55ce87181b40 2024-04-24T08:17:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.329+0000 7fed2b187740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ce8718fb40 con 0x55ce87182460 2024-04-24T08:17:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.329+0000 7fed2b187740 10 client.0 _ll_register_callbacks cb 0x55ce8706e140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.330+0000 7fed26200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 0x55ce87189fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.330+0000 7fed25800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87181b40 0x55ce8718c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.330+0000 7fed24e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87182460 0x55ce871898a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.330+0000 7fed25800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87181b40 0x55ce8718c5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:56424/0 (socket says 172.21.15.28:56424) 2024-04-24T08:17:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.330+0000 7fed25800640 1 -- 172.21.15.28:0/4222634677 learned_addr learned my addr 172.21.15.28:0/4222634677 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:42.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.330+0000 7fed26200640 1 -- 172.21.15.28:0/4222634677 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87182460 msgr2=0x55ce871898a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:42.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.330+0000 7fed26200640 1 --2- 172.21.15.28:0/4222634677 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87182460 0x55ce871898a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.330+0000 7fed26200640 1 -- 172.21.15.28:0/4222634677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87181b40 msgr2=0x55ce8718c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:42.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.330+0000 7fed26200640 1 --2- 172.21.15.28:0/4222634677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87181b40 0x55ce8718c5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.330+0000 7fed26200640 1 -- 172.21.15.28:0/4222634677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce870b13f0 con 0x55ce8718cb30 2024-04-24T08:17:42.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.330+0000 7fed26200640 1 --2- 172.21.15.28:0/4222634677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 0x55ce87189fe0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fed1801b000 tx=0x7fed1800f6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2c3e5304aea8405 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:42.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.330+0000 7fed07400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:17:42.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.330+0000 7fed07400640 1 -- 172.21.15.28:0/4222634677 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed1801e040 con 0x55ce8718cb30 2024-04-24T08:17:42.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.331+0000 7fed2b187740 1 -- 172.21.15.28:0/4222634677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed18003c30 con 0x55ce8718cb30 2024-04-24T08:17:42.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.331+0000 7fed07400640 1 -- 172.21.15.28:0/4222634677 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7fed18009bd0 con 0x55ce8718cb30 2024-04-24T08:17:42.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.331+0000 7fed07400640 1 -- 172.21.15.28:0/4222634677 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fed18032050 con 0x55ce8718cb30 2024-04-24T08:17:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.331+0000 7fed2b187740 10 client.16179 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.331+0000 7fed2b187740 1 -- 172.21.15.28:0/4222634677 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ce86df93f0 con 0x55ce8718cb30 2024-04-24T08:17:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.331+0000 7fed07400640 1 -- 172.21.15.28:0/4222634677 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed18003c30 con 0x55ce8718cb30 2024-04-24T08:17:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.331+0000 7fed07400640 1 -- 172.21.15.28:0/4222634677 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fed18021030 con 0x55ce8718cb30 2024-04-24T08:17:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.331+0000 7fed2b187740 10 client.16179 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.331+0000 7fed2b187740 10 client.16179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:17:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.331+0000 7fed2b187740 20 client.16179 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:42.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.331+0000 7fed2b187740 1 --2- 172.21.15.28:0/4222634677 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ce8719ccb0 0x55ce8719f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.331+0000 7fed2b187740 4 client.16179 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:17:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.332+0000 7fed2b187740 1 -- 172.21.15.28:0/4222634677 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ce870f4bf0 con 0x55ce8719ccb0 2024-04-24T08:17:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.332+0000 7fed25800640 1 --2- 172.21.15.28:0/4222634677 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ce8719ccb0 0x55ce8719f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.332+0000 7fed25800640 1 --2- 172.21.15.28:0/4222634677 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ce8719ccb0 0x55ce8719f0a0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=baa2efc753cf32cd server_cookie=728b80bbff352815 in_seq=0 out_seq=0 2024-04-24T08:17:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.332+0000 7fed07400640 10 client.16179 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.333+0000 7fed07400640 1 -- 172.21.15.28:0/4222634677 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ce870f4bf0 con 0x55ce8719ccb0 2024-04-24T08:17:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.333+0000 7fed07400640 10 client.16179 handle_command_reply: tid=0 2024-04-24T08:17:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.333+0000 7fed2b187740 1 client.16179 shutdown 2024-04-24T08:17:42.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.333+0000 7fed2b187740 1 -- 172.21.15.28:0/4222634677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 msgr2=0x55ce87189fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.333+0000 7fed2b187740 1 --2- 172.21.15.28:0/4222634677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 0x55ce87189fe0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fed1801b000 tx=0x7fed1800f6c0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.333+0000 7fed2b187740 1 -- 172.21.15.28:0/4222634677 shutdown_connections 2024-04-24T08:17:42.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.333+0000 7fed2b187740 1 --2- 172.21.15.28:0/4222634677 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ce8719ccb0 0x55ce8719f0a0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.333+0000 7fed2b187740 1 --2- 172.21.15.28:0/4222634677 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 0x55ce87189fe0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.333+0000 7fed2b187740 1 --2- 172.21.15.28:0/4222634677 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87182460 0x55ce871898a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.333+0000 7fed2b187740 1 --2- 172.21.15.28:0/4222634677 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87181b40 0x55ce8718c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.333+0000 7fed2b187740 1 -- 172.21.15.28:0/4222634677 >> 172.21.15.28:0/4222634677 conn(0x55ce8717de60 msgr2=0x55ce87183c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:42.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.333+0000 7fed07400640 0 client.16179 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:42.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.334+0000 7fed2b187740 1 -- 172.21.15.28:0/4222634677 shutdown_connections 2024-04-24T08:17:42.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.334+0000 7fed2b187740 1 -- 172.21.15.28:0/4222634677 wait complete. 2024-04-24T08:17:42.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.334+0000 7fed2b187740 20 client.16179 trim_cache size 0 max 16384 2024-04-24T08:17:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.348+0000 7fed2b187740 1 Processor -- start 2024-04-24T08:17:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.348+0000 7fed2b187740 1 -- start start 2024-04-24T08:17:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.348+0000 7fed2b187740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87181b40 0x55ce871988b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.349+0000 7fed2b187740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 0x55ce87183ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.349+0000 7fed2b187740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 0x55ce8718c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.349+0000 7fed2b187740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed201b6ac0 con 0x55ce8718cb30 2024-04-24T08:17:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.349+0000 7fed2b187740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fed201b4390 con 0x55ce87182460 2024-04-24T08:17:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.349+0000 7fed2b187740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fed201b2200 con 0x55ce87181b40 2024-04-24T08:17:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.349+0000 7fed24e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 0x55ce87183ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.349+0000 7fed26200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 0x55ce8718c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.349+0000 7fed24e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 0x55ce87183ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:56426/0 (socket says 172.21.15.28:56426) 2024-04-24T08:17:42.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.349+0000 7fed24e00640 1 -- 172.21.15.28:0/3300295634 learned_addr learned my addr 172.21.15.28:0/3300295634 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.349+0000 7fed25800640 1 --2- 172.21.15.28:0/3300295634 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87181b40 0x55ce871988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.349+0000 7fed24e00640 1 -- 172.21.15.28:0/3300295634 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87181b40 msgr2=0x55ce871988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.349+0000 7fed24e00640 1 --2- 172.21.15.28:0/3300295634 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87181b40 0x55ce871988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.349+0000 7fed24e00640 1 -- 172.21.15.28:0/3300295634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 msgr2=0x55ce8718c0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.349+0000 7fed24e00640 1 --2- 172.21.15.28:0/3300295634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 0x55ce8718c0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.349+0000 7fed24e00640 1 -- 172.21.15.28:0/3300295634 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ce86ec37d0 con 0x55ce87182460 2024-04-24T08:17:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.349+0000 7fed24e00640 1 --2- 172.21.15.28:0/3300295634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 0x55ce87183ab0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fed0c099560 tx=0x7fed0c0d4750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aaef3a3953f4bc00 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.350+0000 7fed12000640 1 -- 172.21.15.28:0/3300295634 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c098e30 con 0x55ce87182460 2024-04-24T08:17:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.350+0000 7fed12000640 1 -- 172.21.15.28:0/3300295634 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed0c09d290 con 0x55ce87182460 2024-04-24T08:17:42.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.350+0000 7fed12000640 1 -- 172.21.15.28:0/3300295634 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c09c4f0 con 0x55ce87182460 2024-04-24T08:17:42.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.350+0000 7fed2b187740 1 -- 172.21.15.28:0/3300295634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 msgr2=0x55ce87183ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.350+0000 7fed2b187740 1 --2- 172.21.15.28:0/3300295634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 0x55ce87183ab0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fed0c099560 tx=0x7fed0c0d4750 comp rx=0 tx=0).stop 2024-04-24T08:17:42.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.350+0000 7fed2b187740 1 -- 172.21.15.28:0/3300295634 shutdown_connections 2024-04-24T08:17:42.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.350+0000 7fed2b187740 1 --2- 172.21.15.28:0/3300295634 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 0x55ce8718c0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.350+0000 7fed2b187740 1 --2- 172.21.15.28:0/3300295634 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 0x55ce87183ab0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.350+0000 7fed2b187740 1 --2- 172.21.15.28:0/3300295634 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87181b40 0x55ce871988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.350+0000 7fed2b187740 1 -- 172.21.15.28:0/3300295634 >> 172.21.15.28:0/3300295634 conn(0x55ce8717de60 msgr2=0x55ce871833a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:42.359 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.351+0000 7fed2b187740 1 -- 172.21.15.28:0/3300295634 shutdown_connections 2024-04-24T08:17:42.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.351+0000 7fed2b187740 1 -- 172.21.15.28:0/3300295634 wait complete. 2024-04-24T08:17:42.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.351+0000 7fed2b187740 1 -- start start 2024-04-24T08:17:42.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.351+0000 7fed2b187740 1 Processor -- start 2024-04-24T08:17:42.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.351+0000 7fed2b187740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87181b40 0x55ce8718d510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.351+0000 7fed2b187740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 0x55ce8718da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.351+0000 7fed2b187740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 0x55ce87193dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.352+0000 7fed2b187740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fed201b2850 con 0x55ce8718cb30 2024-04-24T08:17:42.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.352+0000 7fed2b187740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fed201b3cc0 con 0x55ce87182460 2024-04-24T08:17:42.360 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.352+0000 7fed2b187740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fed201b2f20 con 0x55ce87181b40 2024-04-24T08:17:42.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.352+0000 7fed2b187740 10 client.0 _ll_register_callbacks cb 0x7fed1801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:42.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.352+0000 7fed24e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 0x55ce8718da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.352+0000 7fed24e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 0x55ce8718da50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:56430/0 (socket says 172.21.15.28:56430) 2024-04-24T08:17:42.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.352+0000 7fed24e00640 1 -- 172.21.15.28:0/3292847539 learned_addr learned my addr 172.21.15.28:0/3292847539 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:42.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.352+0000 7fed26200640 1 --2- 172.21.15.28:0/3292847539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 0x55ce87193dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.352+0000 7fed25800640 1 --2- 172.21.15.28:0/3292847539 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87181b40 0x55ce8718d510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.361 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.352+0000 7fed24e00640 1 -- 172.21.15.28:0/3292847539 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87181b40 msgr2=0x55ce8718d510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:42.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.352+0000 7fed24e00640 1 --2- 172.21.15.28:0/3292847539 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87181b40 0x55ce8718d510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.352+0000 7fed24e00640 1 -- 172.21.15.28:0/3292847539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 msgr2=0x55ce87193dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:42.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.352+0000 7fed24e00640 1 --2- 172.21.15.28:0/3292847539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 0x55ce87193dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.352+0000 7fed24e00640 1 -- 172.21.15.28:0/3292847539 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ce87194380 con 0x55ce87182460 2024-04-24T08:17:42.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.352+0000 7fed24e00640 1 --2- 172.21.15.28:0/3292847539 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 0x55ce8718da50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00e9c0 tx=0x7fed0c0d4280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0385fd9095c2c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:42.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.352+0000 7fed07400640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:17:42.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.352+0000 7fed07400640 1 -- 172.21.15.28:0/3292847539 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0c09c6d0 con 0x55ce87182460 2024-04-24T08:17:42.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.353+0000 7fed07400640 1 -- 172.21.15.28:0/3292847539 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7fed0c0a5370 con 0x55ce87182460 2024-04-24T08:17:42.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.353+0000 7fed2b187740 1 -- 172.21.15.28:0/3292847539 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fed0c09d290 con 0x55ce87182460 2024-04-24T08:17:42.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.353+0000 7fed07400640 1 -- 172.21.15.28:0/3292847539 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fed0c09c960 con 0x55ce87182460 2024-04-24T08:17:42.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.353+0000 7fed2b187740 10 client.26465 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:42.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.353+0000 7fed2b187740 1 -- 172.21.15.28:0/3292847539 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ce87195660 con 0x55ce87182460 2024-04-24T08:17:42.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.353+0000 7fed07400640 1 -- 172.21.15.28:0/3292847539 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed0c09d290 con 0x55ce87182460 2024-04-24T08:17:42.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.353+0000 7fed07400640 1 -- 172.21.15.28:0/3292847539 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fed0c0d6030 con 0x55ce87182460 2024-04-24T08:17:42.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.353+0000 7fed2b187740 10 client.26465 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:42.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.353+0000 7fed2b187740 10 client.26465 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:17:42.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.353+0000 7fed2b187740 20 client.26465 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:42.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.353+0000 7fed2b187740 1 --2- 172.21.15.28:0/3292847539 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ce87203650 0x55ce87205a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:42.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.353+0000 7fed2b187740 4 client.26465 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:17:42.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.353+0000 7fed2b187740 1 -- 172.21.15.28:0/3292847539 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55ce870eea60 con 0x55ce87203650 2024-04-24T08:17:42.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.353+0000 7fed25800640 1 --2- 172.21.15.28:0/3292847539 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ce87203650 0x55ce87205a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:42.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.354+0000 7fed25800640 1 --2- 172.21.15.28:0/3292847539 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ce87203650 0x55ce87205a70 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4b8824e28065b1c server_cookie=217a9e28ec6116d8 in_seq=0 out_seq=0 2024-04-24T08:17:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.354+0000 7fed07400640 10 client.26465 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.354+0000 7fed07400640 1 -- 172.21.15.28:0/3292847539 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55ce870eea60 con 0x55ce87203650 2024-04-24T08:17:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.354+0000 7fed07400640 10 client.26465 handle_command_reply: tid=0 2024-04-24T08:17:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.354+0000 7fed2b187740 1 client.26465 shutdown 2024-04-24T08:17:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.354+0000 7fed2b187740 1 -- 172.21.15.28:0/3292847539 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 msgr2=0x55ce8718da50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.354+0000 7fed2b187740 1 --2- 172.21.15.28:0/3292847539 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 0x55ce8718da50 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00e9c0 tx=0x7fed0c0d4280 comp rx=0 tx=0).stop 2024-04-24T08:17:42.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.354+0000 7fed2b187740 1 -- 172.21.15.28:0/3292847539 shutdown_connections 2024-04-24T08:17:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.354+0000 7fed2b187740 1 --2- 172.21.15.28:0/3292847539 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ce87203650 0x55ce87205a70 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.355+0000 7fed2b187740 1 --2- 172.21.15.28:0/3292847539 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ce8718cb30 0x55ce87193dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.355+0000 7fed2b187740 1 --2- 172.21.15.28:0/3292847539 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ce87182460 0x55ce8718da50 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.355+0000 7fed2b187740 1 --2- 172.21.15.28:0/3292847539 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ce87181b40 0x55ce8718d510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.355+0000 7fed2b187740 1 -- 172.21.15.28:0/3292847539 >> 172.21.15.28:0/3292847539 conn(0x55ce8717de60 msgr2=0x55ce8717cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.355+0000 7fed07400640 0 client.26465 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.355+0000 7fed2b187740 1 -- 172.21.15.28:0/3292847539 shutdown_connections 2024-04-24T08:17:42.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.355+0000 7fed2b187740 1 -- 172.21.15.28:0/3292847539 wait complete. 2024-04-24T08:17:42.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.355+0000 7fed2b187740 20 client.26465 trim_cache size 0 max 16384 2024-04-24T08:17:42.368 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.355+0000 7fed26c00640 1 -- 172.21.15.28:0/1114747568 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fed080a3d20 msgr2=0x7fed080a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.355+0000 7fed26c00640 1 --2- 172.21.15.28:0/1114747568 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fed080a3d20 0x7fed080a61e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fed14004770 tx=0x7fed140058d0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.355+0000 7fed26c00640 1 -- 172.21.15.28:0/1114747568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed20108f40 msgr2=0x7fed2019f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.355+0000 7fed26c00640 1 --2- 172.21.15.28:0/1114747568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed20108f40 0x7fed2019f280 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fed0c00e9f0 tx=0x7fed0c00edf0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.355+0000 7fed26c00640 1 -- 172.21.15.28:0/1114747568 shutdown_connections 2024-04-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.355+0000 7fed26c00640 1 --2- 172.21.15.28:0/1114747568 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fed080a3d20 0x7fed080a61e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.356+0000 7fed26c00640 1 --2- 172.21.15.28:0/1114747568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fed20108f40 0x7fed2019f280 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.356+0000 7fed26c00640 1 --2- 172.21.15.28:0/1114747568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fed2010d500 0x7fed201a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.356+0000 7fed26c00640 1 --2- 172.21.15.28:0/1114747568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed20108580 0x7fed2019ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.356+0000 7fed26c00640 1 -- 172.21.15.28:0/1114747568 >> 172.21.15.28:0/1114747568 conn(0x7fed20077e20 msgr2=0x7fed20078340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.356+0000 7fed26c00640 1 -- 172.21.15.28:0/1114747568 shutdown_connections 2024-04-24T08:17:42.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:42.356+0000 7fed26c00640 1 -- 172.21.15.28:0/1114747568 wait complete. 2024-04-24T08:17:43.372 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:17:43.373 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:17:43.373 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:17:43.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.596+0000 7f1bb9400640 1 -- 172.21.15.28:0/3088614020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bb41042f0 msgr2=0x7f1bb41046d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.596+0000 7f1bb9400640 1 --2- 172.21.15.28:0/3088614020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bb41042f0 0x7f1bb41046d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4009900 tx=0x7f1ba402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.596+0000 7f1bb9400640 1 -- 172.21.15.28:0/3088614020 shutdown_connections 2024-04-24T08:17:43.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.596+0000 7f1bb9400640 1 --2- 172.21.15.28:0/3088614020 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bb41093c0 0x7f1bb4110f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.596+0000 7f1bb9400640 1 --2- 172.21.15.28:0/3088614020 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bb4104ca0 0x7f1bb4108ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.596+0000 7f1bb9400640 1 --2- 172.21.15.28:0/3088614020 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bb41042f0 0x7f1bb41046d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.596+0000 7f1bb9400640 1 -- 172.21.15.28:0/3088614020 >> 172.21.15.28:0/3088614020 conn(0x7f1bb40ffb10 msgr2=0x7f1bb4101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:43.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.596+0000 7f1bb9400640 1 -- 172.21.15.28:0/3088614020 shutdown_connections 2024-04-24T08:17:43.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.596+0000 7f1bb9400640 1 -- 172.21.15.28:0/3088614020 wait complete. 2024-04-24T08:17:43.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.598+0000 7f1bb9400640 1 Processor -- start 2024-04-24T08:17:43.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.598+0000 7f1bb9400640 1 -- start start 2024-04-24T08:17:43.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.598+0000 7f1bb9400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bb41042f0 0x7f1bb41a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.598+0000 7f1bb9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bb41093c0 0x7f1bb41a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.598+0000 7f1bb9400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bb419a800 0x7f1bb419acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.598+0000 7f1bb9400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb4114480 con 0x7f1bb41093c0 2024-04-24T08:17:43.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.598+0000 7f1bb9400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb4114300 con 0x7f1bb41042f0 2024-04-24T08:17:43.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.598+0000 7f1bb9400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb4114600 con 0x7f1bb419a800 2024-04-24T08:17:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.599+0000 7f1bb3e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bb41042f0 0x7f1bb41a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.599+0000 7f1bb3e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bb41042f0 0x7f1bb41a0690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:56462/0 (socket says 172.21.15.28:56462) 2024-04-24T08:17:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.599+0000 7f1bb3e00640 1 -- 172.21.15.28:0/1446017187 learned_addr learned my addr 172.21.15.28:0/1446017187 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.599+0000 7f1bb3400640 1 --2- 172.21.15.28:0/1446017187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bb41093c0 0x7f1bb41a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.599+0000 7f1bb8a00640 1 --2- 172.21.15.28:0/1446017187 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bb419a800 0x7f1bb419acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.599+0000 7f1bb3e00640 1 -- 172.21.15.28:0/1446017187 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bb419a800 msgr2=0x7f1bb419acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.599+0000 7f1bb3e00640 1 --2- 172.21.15.28:0/1446017187 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bb419a800 0x7f1bb419acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.599+0000 7f1bb3e00640 1 -- 172.21.15.28:0/1446017187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bb41093c0 msgr2=0x7f1bb41a0bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.599+0000 7f1bb3e00640 1 --2- 172.21.15.28:0/1446017187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bb41093c0 0x7f1bb41a0bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.599+0000 7f1bb3e00640 1 -- 172.21.15.28:0/1446017187 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bb419b390 con 0x7f1bb41042f0 2024-04-24T08:17:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.600+0000 7f1bb3e00640 1 --2- 172.21.15.28:0/1446017187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bb41042f0 0x7f1bb41a0690 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4006010 tx=0x7f1ba4004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.600+0000 7f1bb0c00640 1 -- 172.21.15.28:0/1446017187 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba403f030 con 0x7f1bb41042f0 2024-04-24T08:17:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.600+0000 7f1bb0c00640 1 -- 172.21.15.28:0/1446017187 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ba4004440 con 0x7f1bb41042f0 2024-04-24T08:17:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.600+0000 7f1bb0c00640 1 -- 172.21.15.28:0/1446017187 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba4035070 con 0x7f1bb41042f0 2024-04-24T08:17:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.600+0000 7f1bb9400640 1 -- 172.21.15.28:0/1446017187 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bb419b620 con 0x7f1bb41042f0 2024-04-24T08:17:43.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.600+0000 7f1bb9400640 1 -- 172.21.15.28:0/1446017187 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bb41a77d0 con 0x7f1bb41042f0 2024-04-24T08:17:43.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.604+0000 7f1bbd3fd740 1 Processor -- start 2024-04-24T08:17:43.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.604+0000 7f1bbd3fd740 1 -- start start 2024-04-24T08:17:43.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.604+0000 7f1bbd3fd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7ef270 0x5582ae7ef650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.604+0000 7f1bbd3fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7efb90 0x5582ae7f9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.604+0000 7f1bbd3fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7fa260 0x5582ae7fc650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.605+0000 7f1bbd3fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5582ae754e70 con 0x5582ae7efb90 2024-04-24T08:17:43.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.605+0000 7f1bbd3fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5582ae6fb380 con 0x5582ae7fa260 2024-04-24T08:17:43.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.605+0000 7f1bbd3fd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5582ae750e30 con 0x5582ae7ef270 2024-04-24T08:17:43.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.605+0000 7f1bb0c00640 1 -- 172.21.15.28:0/1446017187 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1ba4042030 con 0x7f1bb41042f0 2024-04-24T08:17:43.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.605+0000 7f1bb8a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7ef270 0x5582ae7ef650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.605+0000 7f1bb8a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7ef270 0x5582ae7ef650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60422/0 (socket says 172.21.15.28:60422) 2024-04-24T08:17:43.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.605+0000 7f1bb8a00640 1 -- 172.21.15.28:0/4089978747 learned_addr learned my addr 172.21.15.28:0/4089978747 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:43.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.605+0000 7f1bb3400640 1 --2- 172.21.15.28:0/4089978747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7fa260 0x5582ae7fc650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.605+0000 7f1bb3e00640 1 --2- 172.21.15.28:0/4089978747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7efb90 0x5582ae7f9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.605+0000 7f1bb8a00640 1 -- 172.21.15.28:0/4089978747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7fa260 msgr2=0x5582ae7fc650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.605+0000 7f1bb8a00640 1 --2- 172.21.15.28:0/4089978747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7fa260 0x5582ae7fc650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.605+0000 7f1bb8a00640 1 -- 172.21.15.28:0/4089978747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7efb90 msgr2=0x5582ae7f9d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.605+0000 7f1bb8a00640 1 --2- 172.21.15.28:0/4089978747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7efb90 0x5582ae7f9d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.605+0000 7f1bb8a00640 1 -- 172.21.15.28:0/4089978747 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582ae52f7e0 con 0x5582ae7ef270 2024-04-24T08:17:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.606+0000 7f1bb8a00640 1 --2- 172.21.15.28:0/4089978747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7ef270 0x5582ae7ef650 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009f50 tx=0x7f1bac0118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec33078de3a19d77 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.606+0000 7f1ba9200640 1 -- 172.21.15.28:0/4089978747 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac004d20 con 0x5582ae7ef270 2024-04-24T08:17:43.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.606+0000 7f1ba9200640 1 -- 172.21.15.28:0/4089978747 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bac004050 con 0x5582ae7ef270 2024-04-24T08:17:43.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.606+0000 7f1ba9200640 1 -- 172.21.15.28:0/4089978747 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bac005280 con 0x5582ae7ef270 2024-04-24T08:17:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.606+0000 7f1bb0c00640 1 --2- 172.21.15.28:0/1446017187 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1b980a3d20 0x7f1b980a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.606+0000 7f1bb0c00640 1 -- 172.21.15.28:0/1446017187 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7f1ba40f0ba0 con 0x7f1bb41042f0 2024-04-24T08:17:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.607+0000 7f1bb3e00640 1 --2- 172.21.15.28:0/1446017187 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1b980a3d20 0x7f1b980a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.607+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/4089978747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7ef270 msgr2=0x5582ae7ef650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.607+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/4089978747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7ef270 0x5582ae7ef650 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f1bac009f50 tx=0x7f1bac0118d0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.607+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/4089978747 shutdown_connections 2024-04-24T08:17:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.607+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/4089978747 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7efb90 0x5582ae7f9d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.615 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.607+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/4089978747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7fa260 0x5582ae7fc650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.607+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/4089978747 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7ef270 0x5582ae7ef650 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.607+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/4089978747 >> 172.21.15.28:0/4089978747 conn(0x5582ae7eb590 msgr2=0x5582ae7eb970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:43.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.607+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/4089978747 shutdown_connections 2024-04-24T08:17:43.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.607+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/4089978747 wait complete. 2024-04-24T08:17:43.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.607+0000 7f1bb3e00640 1 --2- 172.21.15.28:0/1446017187 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1b980a3d20 0x7f1b980a61e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x5582ae7f0830 tx=0x7f1ba402a3f0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:43.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.607+0000 7f1bbd3fd740 1 -- start start 2024-04-24T08:17:43.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.608+0000 7f1bbd3fd740 1 Processor -- start 2024-04-24T08:17:43.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.608+0000 7f1bbd3fd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7ef270 0x5582ae7f7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.608+0000 7f1bbd3fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7efb90 0x5582ae7f76b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.608+0000 7f1bbd3fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7fa260 0x5582ae7f7bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.608+0000 7f1bbd3fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5582ae7fd050 con 0x5582ae7efb90 2024-04-24T08:17:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.608+0000 7f1bbd3fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5582ae70e660 con 0x5582ae7fa260 2024-04-24T08:17:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.608+0000 7f1bbd3fd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5582ae7fd1d0 con 0x5582ae7ef270 2024-04-24T08:17:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.608+0000 7f1bbd3fd740 10 client.0 _ll_register_callbacks cb 0x5582ae6fcaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.608+0000 7f1bb3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7efb90 0x5582ae7f76b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.608+0000 7f1bb8a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7ef270 0x5582ae7f7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.608+0000 7f1bb8a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7ef270 0x5582ae7f7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60426/0 (socket says 172.21.15.28:60426) 2024-04-24T08:17:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.608+0000 7f1bb3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7efb90 0x5582ae7f76b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36454/0 (socket says 172.21.15.28:36454) 2024-04-24T08:17:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.608+0000 7f1bb8a00640 1 -- 172.21.15.28:0/4027287758 learned_addr learned my addr 172.21.15.28:0/4027287758 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.609+0000 7f1bb8a00640 1 --2- 172.21.15.28:0/4027287758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7fa260 0x5582ae7f7bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.609+0000 7f1bb3400640 1 -- 172.21.15.28:0/4027287758 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7ef270 msgr2=0x5582ae7f7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.609+0000 7f1bb3400640 1 --2- 172.21.15.28:0/4027287758 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7ef270 0x5582ae7f7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.609+0000 7f1bb3400640 1 -- 172.21.15.28:0/4027287758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7fa260 msgr2=0x5582ae7f7bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.609+0000 7f1bb3400640 1 --2- 172.21.15.28:0/4027287758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7fa260 0x5582ae7f7bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.618 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.609+0000 7f1bb3400640 1 -- 172.21.15.28:0/4027287758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582ae58ff70 con 0x5582ae7efb90 2024-04-24T08:17:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.609+0000 7f1bb3400640 1 --2- 172.21.15.28:0/4027287758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7efb90 0x5582ae7f76b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1bb419bb50 tx=0x7f1ba0012b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0b17102a02258c7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.609+0000 7f1b9d600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:17:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.609+0000 7f1b9d600640 1 -- 172.21.15.28:0/4027287758 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba000f8f0 con 0x5582ae7efb90 2024-04-24T08:17:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.609+0000 7f1b9d600640 1 -- 172.21.15.28:0/4027287758 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7f1ba000fa90 con 0x5582ae7efb90 2024-04-24T08:17:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.609+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/4027287758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bac004050 con 0x5582ae7efb90 2024-04-24T08:17:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.610+0000 7f1b9d600640 1 -- 172.21.15.28:0/4027287758 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1ba000fc90 con 0x5582ae7efb90 2024-04-24T08:17:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.610+0000 7f1bbd3fd740 10 client.16197 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:43.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.610+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/4027287758 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5582ae427170 con 0x5582ae7efb90 2024-04-24T08:17:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.610+0000 7f1b9d600640 1 -- 172.21.15.28:0/4027287758 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bac004050 con 0x5582ae7efb90 2024-04-24T08:17:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.610+0000 7f1b9d600640 1 -- 172.21.15.28:0/4027287758 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1ba001c420 con 0x5582ae7efb90 2024-04-24T08:17:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.610+0000 7f1bbd3fd740 10 client.16197 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.610+0000 7f1bbd3fd740 10 client.16197 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:17:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.610+0000 7f1bbd3fd740 20 client.16197 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.610+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/4027287758 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5582ae80a690 0x5582ae80cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.610+0000 7f1bbd3fd740 4 client.16197 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:17:43.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.610+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/4027287758 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582ae762890 con 0x5582ae80a690 2024-04-24T08:17:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.611+0000 7f1bb8a00640 1 --2- 172.21.15.28:0/4027287758 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5582ae80a690 0x5582ae80cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.611+0000 7f1bb8a00640 1 --2- 172.21.15.28:0/4027287758 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5582ae80a690 0x5582ae80cab0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3f7e55ff80875ec8 server_cookie=bc2a744a9dcacce2 in_seq=0 out_seq=0 2024-04-24T08:17:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.611+0000 7f1b9d600640 10 client.16197 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.612+0000 7f1b9d600640 1 -- 172.21.15.28:0/4027287758 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5582ae762890 con 0x5582ae80a690 2024-04-24T08:17:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.612+0000 7f1b9d600640 10 client.16197 handle_command_reply: tid=0 2024-04-24T08:17:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.612+0000 7f1bbd3fd740 1 client.16197 shutdown 2024-04-24T08:17:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.612+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/4027287758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7efb90 msgr2=0x5582ae7f76b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.612+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/4027287758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7efb90 0x5582ae7f76b0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f1bb419bb50 tx=0x7f1ba0012b60 comp rx=0 tx=0).stop 2024-04-24T08:17:43.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.612+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/4027287758 shutdown_connections 2024-04-24T08:17:43.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.612+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/4027287758 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5582ae80a690 0x5582ae80cab0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.612+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/4027287758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7efb90 0x5582ae7f76b0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.612+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/4027287758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7fa260 0x5582ae7f7bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.612+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/4027287758 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7ef270 0x5582ae7f7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.612+0000 7f1b9d600640 0 client.16197 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:43.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.612+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/4027287758 >> 172.21.15.28:0/4027287758 conn(0x5582ae7eb590 msgr2=0x5582ae7f1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:43.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.612+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/4027287758 shutdown_connections 2024-04-24T08:17:43.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.612+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/4027287758 wait complete. 2024-04-24T08:17:43.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.612+0000 7f1bbd3fd740 20 client.16197 trim_cache size 0 max 16384 2024-04-24T08:17:43.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.632+0000 7f1bbd3fd740 1 Processor -- start 2024-04-24T08:17:43.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.632+0000 7f1bbd3fd740 1 -- start start 2024-04-24T08:17:43.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.632+0000 7f1bbd3fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7ef270 0x5582ae805f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.632+0000 7f1bbd3fd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7efb90 0x5582ae7f7a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.632+0000 7f1bbd3fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7fa260 0x5582ae8123c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.632+0000 7f1bbd3fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb41b27c0 con 0x5582ae7ef270 2024-04-24T08:17:43.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.632+0000 7f1bbd3fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb41b0090 con 0x5582ae7fa260 2024-04-24T08:17:43.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.632+0000 7f1bbd3fd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb41adf00 con 0x5582ae7efb90 2024-04-24T08:17:43.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.632+0000 7f1bb8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7ef270 0x5582ae805f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.633+0000 7f1bb8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7ef270 0x5582ae805f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36468/0 (socket says 172.21.15.28:36468) 2024-04-24T08:17:43.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.633+0000 7f1bb8a00640 1 -- 172.21.15.28:0/3578707956 learned_addr learned my addr 172.21.15.28:0/3578707956 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:43.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.633+0000 7f1bb3400640 1 --2- 172.21.15.28:0/3578707956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7efb90 0x5582ae7f7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.633+0000 7f1bb8a00640 1 --2- 172.21.15.28:0/3578707956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7fa260 0x5582ae8123c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.633+0000 7f1bb3400640 1 -- 172.21.15.28:0/3578707956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7fa260 msgr2=0x5582ae8123c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:43.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.633+0000 7f1bb3400640 1 --2- 172.21.15.28:0/3578707956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7fa260 0x5582ae8123c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.633+0000 7f1bb3400640 1 -- 172.21.15.28:0/3578707956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7ef270 msgr2=0x5582ae805f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:43.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.633+0000 7f1bb3400640 1 --2- 172.21.15.28:0/3578707956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7ef270 0x5582ae805f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.633+0000 7f1bb3400640 1 -- 172.21.15.28:0/3578707956 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582ae7f56c0 con 0x5582ae7efb90 2024-04-24T08:17:43.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.633+0000 7f1bb3400640 1 --2- 172.21.15.28:0/3578707956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7efb90 0x5582ae7f7a10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1ba001ca00 tx=0x7f1ba0009370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fb698e9a30801c2f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:43.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.634+0000 7f1b9fe00640 1 -- 172.21.15.28:0/3578707956 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0009dc0 con 0x5582ae7efb90 2024-04-24T08:17:43.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.634+0000 7f1b9fe00640 1 -- 172.21.15.28:0/3578707956 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ba000f8f0 con 0x5582ae7efb90 2024-04-24T08:17:43.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.634+0000 7f1b9fe00640 1 -- 172.21.15.28:0/3578707956 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba0005040 con 0x5582ae7efb90 2024-04-24T08:17:43.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.634+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/3578707956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7efb90 msgr2=0x5582ae7f7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.634+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/3578707956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7efb90 0x5582ae7f7a10 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f1ba001ca00 tx=0x7f1ba0009370 comp rx=0 tx=0).stop 2024-04-24T08:17:43.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.634+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/3578707956 shutdown_connections 2024-04-24T08:17:43.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.634+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/3578707956 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7fa260 0x5582ae8123c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.634+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/3578707956 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7efb90 0x5582ae7f7a10 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.634+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/3578707956 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7ef270 0x5582ae805f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.634+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/3578707956 >> 172.21.15.28:0/3578707956 conn(0x5582ae7eb590 msgr2=0x5582ae7f09c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:43.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.635+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/3578707956 shutdown_connections 2024-04-24T08:17:43.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.635+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/3578707956 wait complete. 2024-04-24T08:17:43.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.635+0000 7f1bbd3fd740 1 -- start start 2024-04-24T08:17:43.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.635+0000 7f1bbd3fd740 1 Processor -- start 2024-04-24T08:17:43.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.635+0000 7f1bbd3fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7ef270 0x5582ae802740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.636+0000 7f1bbd3fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7efb90 0x5582ae7f3a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.636+0000 7f1bbd3fd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7fa260 0x5582ae7f3fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.636+0000 7f1bbd3fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb41ae550 con 0x5582ae7ef270 2024-04-24T08:17:43.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.636+0000 7f1bbd3fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb41af9c0 con 0x5582ae7efb90 2024-04-24T08:17:43.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.636+0000 7f1bbd3fd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1bb41aec20 con 0x5582ae7fa260 2024-04-24T08:17:43.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.636+0000 7f1bbd3fd740 10 client.0 _ll_register_callbacks cb 0x7f1ba0009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:43.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.636+0000 7f1bb3400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7efb90 0x5582ae7f3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.636+0000 7f1bb8a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7ef270 0x5582ae802740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.636+0000 7f1bb3400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7efb90 0x5582ae7f3a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:56516/0 (socket says 172.21.15.28:56516) 2024-04-24T08:17:43.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.636+0000 7f1bb3400640 1 -- 172.21.15.28:0/3929603013 learned_addr learned my addr 172.21.15.28:0/3929603013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:43.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.636+0000 7f1bb8a00640 1 --2- 172.21.15.28:0/3929603013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7fa260 0x5582ae7f3fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.636+0000 7f1bb3400640 1 -- 172.21.15.28:0/3929603013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7fa260 msgr2=0x5582ae7f3fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:43.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.636+0000 7f1bb3400640 1 --2- 172.21.15.28:0/3929603013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7fa260 0x5582ae7f3fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.636+0000 7f1bb3400640 1 -- 172.21.15.28:0/3929603013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7ef270 msgr2=0x5582ae802740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:43.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.636+0000 7f1bb3400640 1 --2- 172.21.15.28:0/3929603013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7ef270 0x5582ae802740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.636+0000 7f1bb3400640 1 -- 172.21.15.28:0/3929603013 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582ae7f4660 con 0x5582ae7efb90 2024-04-24T08:17:43.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.636+0000 7f1bb8a00640 1 --2- 172.21.15.28:0/3929603013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7ef270 0x5582ae802740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:17:43.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.637+0000 7f1bb3400640 1 --2- 172.21.15.28:0/3929603013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7efb90 0x5582ae7f3a60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0012ba0 tx=0x7f1ba001c890 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3463e6a643e898ab server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:43.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.637+0000 7f1b9d600640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:17:43.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.637+0000 7f1b9d600640 1 -- 172.21.15.28:0/3929603013 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1ba001c420 con 0x5582ae7efb90 2024-04-24T08:17:43.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.637+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/3929603013 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1ba000f8f0 con 0x5582ae7efb90 2024-04-24T08:17:43.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.637+0000 7f1b9d600640 1 -- 172.21.15.28:0/3929603013 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7f1ba0009dc0 con 0x5582ae7efb90 2024-04-24T08:17:43.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.637+0000 7f1b9d600640 1 -- 172.21.15.28:0/3929603013 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1ba003b050 con 0x5582ae7efb90 2024-04-24T08:17:43.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.637+0000 7f1bbd3fd740 10 client.26483 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:43.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.637+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/3929603013 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5582ae870b60 con 0x5582ae7efb90 2024-04-24T08:17:43.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.638+0000 7f1b9d600640 1 -- 172.21.15.28:0/3929603013 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1ba000f8f0 con 0x5582ae7efb90 2024-04-24T08:17:43.645 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.638+0000 7f1b9d600640 1 -- 172.21.15.28:0/3929603013 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1ba0003bd0 con 0x5582ae7efb90 2024-04-24T08:17:43.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.638+0000 7f1bbd3fd740 10 client.26483 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:43.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.638+0000 7f1bbd3fd740 10 client.26483 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:17:43.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.638+0000 7f1bbd3fd740 20 client.26483 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:43.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.638+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/3929603013 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5582ae8711e0 0x5582ae873600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.638+0000 7f1bbd3fd740 4 client.26483 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:17:43.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.638+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/3929603013 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5582ae75c700 con 0x5582ae8711e0 2024-04-24T08:17:43.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.638+0000 7f1bb8a00640 1 --2- 172.21.15.28:0/3929603013 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5582ae8711e0 0x5582ae873600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.639+0000 7f1bb8a00640 1 --2- 172.21.15.28:0/3929603013 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5582ae8711e0 0x5582ae873600 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8af7ee9c0ac7272f server_cookie=18c884fcc6044335 in_seq=0 out_seq=0 2024-04-24T08:17:43.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.639+0000 7f1b9d600640 10 client.26483 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.640+0000 7f1b9d600640 1 -- 172.21.15.28:0/3929603013 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5582ae75c700 con 0x5582ae8711e0 2024-04-24T08:17:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.640+0000 7f1b9d600640 10 client.26483 handle_command_reply: tid=0 2024-04-24T08:17:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.640+0000 7f1bbd3fd740 1 client.26483 shutdown 2024-04-24T08:17:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.640+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/3929603013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7efb90 msgr2=0x5582ae7f3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.640+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/3929603013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7efb90 0x5582ae7f3a60 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1ba0012ba0 tx=0x7f1ba001c890 comp rx=0 tx=0).stop 2024-04-24T08:17:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.640+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/3929603013 shutdown_connections 2024-04-24T08:17:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.640+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/3929603013 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5582ae8711e0 0x5582ae873600 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.640+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/3929603013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5582ae7fa260 0x5582ae7f3fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.640+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/3929603013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5582ae7efb90 0x5582ae7f3a60 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.640+0000 7f1bbd3fd740 1 --2- 172.21.15.28:0/3929603013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5582ae7ef270 0x5582ae802740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.640+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/3929603013 >> 172.21.15.28:0/3929603013 conn(0x5582ae7eb590 msgr2=0x5582ae7f1250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:43.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.640+0000 7f1b9d600640 0 client.26483 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:43.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.641+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/3929603013 shutdown_connections 2024-04-24T08:17:43.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.641+0000 7f1bbd3fd740 1 -- 172.21.15.28:0/3929603013 wait complete. 2024-04-24T08:17:43.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.641+0000 7f1bbd3fd740 20 client.26483 trim_cache size 0 max 16384 2024-04-24T08:17:43.649 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:17:43.649 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-24T08:17:43.649 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "b1f722fa-8c8f-43ee-bd6f-a8c7d4a3dabc", 2024-04-24T08:17:43.649 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-24T08:17:43.649 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:17:43.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.641+0000 7f1bb9400640 1 -- 172.21.15.28:0/1446017187 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1b980a3d20 msgr2=0x7f1b980a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.641+0000 7f1bb9400640 1 --2- 172.21.15.28:0/1446017187 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1b980a3d20 0x7f1b980a61e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x5582ae7f0830 tx=0x7f1ba402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.641+0000 7f1bb9400640 1 -- 172.21.15.28:0/1446017187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bb41042f0 msgr2=0x7f1bb41a0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.642+0000 7f1bb9400640 1 --2- 172.21.15.28:0/1446017187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bb41042f0 0x7f1bb41a0690 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f1ba4006010 tx=0x7f1ba4004040 comp rx=0 tx=0).stop 2024-04-24T08:17:43.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.642+0000 7f1bb9400640 1 -- 172.21.15.28:0/1446017187 shutdown_connections 2024-04-24T08:17:43.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.642+0000 7f1bb9400640 1 --2- 172.21.15.28:0/1446017187 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1b980a3d20 0x7f1b980a61e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.642+0000 7f1bb9400640 1 --2- 172.21.15.28:0/1446017187 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bb41093c0 0x7f1bb41a0bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.642+0000 7f1bb9400640 1 --2- 172.21.15.28:0/1446017187 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bb419a800 0x7f1bb419acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.642+0000 7f1bb9400640 1 --2- 172.21.15.28:0/1446017187 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bb41042f0 0x7f1bb41a0690 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.642+0000 7f1bb9400640 1 -- 172.21.15.28:0/1446017187 >> 172.21.15.28:0/1446017187 conn(0x7f1bb40ffb10 msgr2=0x7f1bb4101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:43.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.642+0000 7f1bb9400640 1 -- 172.21.15.28:0/1446017187 shutdown_connections 2024-04-24T08:17:43.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.642+0000 7f1bb9400640 1 -- 172.21.15.28:0/1446017187 wait complete. 2024-04-24T08:17:43.670 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:17:43.670 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:17:43.738 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:43 smithi028 ceph-mon[27786]: pgmap v226: 97 pgs: 97 active+clean; 24 GiB data, 52 GiB used, 1020 GiB / 1.0 TiB avail; 46 KiB/s rd, 18 MiB/s wr, 3.95k op/s 2024-04-24T08:17:43.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.869+0000 7f41fb000640 1 -- 172.21.15.28:0/76534310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41f4101400 msgr2=0x7f41f4106b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.869+0000 7f41fb000640 1 --2- 172.21.15.28:0/76534310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41f4101400 0x7f41f4106b30 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f41e0009900 tx=0x7f41e002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.869+0000 7f41fb000640 1 -- 172.21.15.28:0/76534310 shutdown_connections 2024-04-24T08:17:43.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.869+0000 7f41fb000640 1 --2- 172.21.15.28:0/76534310 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41f4101400 0x7f41f4106b30 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.869+0000 7f41fb000640 1 --2- 172.21.15.28:0/76534310 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41f4107260 0x7f41f410ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.869+0000 7f41fb000640 1 --2- 172.21.15.28:0/76534310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41f4100a50 0x7f41f4100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.869+0000 7f41fb000640 1 -- 172.21.15.28:0/76534310 >> 172.21.15.28:0/76534310 conn(0x7f41f4076090 msgr2=0x7f41f40764a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:43.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.869+0000 7f41fb000640 1 -- 172.21.15.28:0/76534310 shutdown_connections 2024-04-24T08:17:43.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.869+0000 7f41fb000640 1 -- 172.21.15.28:0/76534310 wait complete. 2024-04-24T08:17:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.870+0000 7f41fb000640 1 Processor -- start 2024-04-24T08:17:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.870+0000 7f41fb000640 1 -- start start 2024-04-24T08:17:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.871+0000 7f41fb000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41f4100a50 0x7f41f41a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.871+0000 7f41fb000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41f4107260 0x7f41f41a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.871+0000 7f41fb000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41f419ac40 0x7f41f419b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.871+0000 7f41fb000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41f4102050 con 0x7f41f419ac40 2024-04-24T08:17:43.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.871+0000 7f41fb000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f41f4101ed0 con 0x7f41f4100a50 2024-04-24T08:17:43.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.871+0000 7f41fb000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f41f41021d0 con 0x7f41f4107260 2024-04-24T08:17:43.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.871+0000 7f41f9200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41f4107260 0x7f41f41a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.871+0000 7f41fa600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41f419ac40 0x7f41f419b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.871+0000 7f41f9200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41f4107260 0x7f41f41a0bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60442/0 (socket says 172.21.15.28:60442) 2024-04-24T08:17:43.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.871+0000 7f41f9200640 1 -- 172.21.15.28:0/3107849579 learned_addr learned my addr 172.21.15.28:0/3107849579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.871+0000 7f41f9c00640 1 --2- 172.21.15.28:0/3107849579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41f4100a50 0x7f41f41a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.872+0000 7f41fa600640 1 -- 172.21.15.28:0/3107849579 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41f4107260 msgr2=0x7f41f41a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.872+0000 7f41fa600640 1 --2- 172.21.15.28:0/3107849579 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41f4107260 0x7f41f41a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.872+0000 7f41fa600640 1 -- 172.21.15.28:0/3107849579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41f4100a50 msgr2=0x7f41f41a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.872+0000 7f41fa600640 1 --2- 172.21.15.28:0/3107849579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41f4100a50 0x7f41f41a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.872+0000 7f41fa600640 1 -- 172.21.15.28:0/3107849579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41f419b7f0 con 0x7f41f419ac40 2024-04-24T08:17:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.872+0000 7f41f9200640 1 --2- 172.21.15.28:0/3107849579 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41f4107260 0x7f41f41a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:17:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.872+0000 7f41fa600640 1 --2- 172.21.15.28:0/3107849579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41f419ac40 0x7f41f419b0c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f41ec00e970 tx=0x7f41ec00ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:43.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.872+0000 7f41f2000640 1 -- 172.21.15.28:0/3107849579 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ec00ccc0 con 0x7f41f419ac40 2024-04-24T08:17:43.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.872+0000 7f41f2000640 1 -- 172.21.15.28:0/3107849579 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41ec005020 con 0x7f41f419ac40 2024-04-24T08:17:43.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.872+0000 7f41f2000640 1 -- 172.21.15.28:0/3107849579 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41ec004a00 con 0x7f41f419ac40 2024-04-24T08:17:43.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.872+0000 7f41fb000640 1 -- 172.21.15.28:0/3107849579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41f41a74c0 con 0x7f41f419ac40 2024-04-24T08:17:43.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.873+0000 7f41fb000640 1 -- 172.21.15.28:0/3107849579 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41f41a7870 con 0x7f41f419ac40 2024-04-24T08:17:43.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.876+0000 7f41ff377740 1 Processor -- start 2024-04-24T08:17:43.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.876+0000 7f41f2000640 1 -- 172.21.15.28:0/3107849579 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f41ec0051d0 con 0x7f41f419ac40 2024-04-24T08:17:43.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.876+0000 7f41ff377740 1 -- start start 2024-04-24T08:17:43.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.877+0000 7f41ff377740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb43919a30 0x55eb43919e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.877+0000 7f41ff377740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb4391a350 0x55eb439244e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.877+0000 7f41ff377740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb43924a20 0x55eb43926e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.877+0000 7f41ff377740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55eb4387f1f0 con 0x55eb43919a30 2024-04-24T08:17:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.877+0000 7f41ff377740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55eb43825350 con 0x55eb4391a350 2024-04-24T08:17:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.877+0000 7f41ff377740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55eb4387b1b0 con 0x55eb43924a20 2024-04-24T08:17:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.877+0000 7f41f9c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb4391a350 0x55eb439244e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.877+0000 7f41f9c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb4391a350 0x55eb439244e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:56550/0 (socket says 172.21.15.28:56550) 2024-04-24T08:17:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.877+0000 7f41f9c00640 1 -- 172.21.15.28:0/501800267 learned_addr learned my addr 172.21.15.28:0/501800267 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.877+0000 7f41f9200640 1 --2- 172.21.15.28:0/501800267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb43919a30 0x55eb43919e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.877+0000 7f41f9c00640 1 -- 172.21.15.28:0/501800267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb43924a20 msgr2=0x55eb43926e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:43.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.877+0000 7f41f9c00640 1 --2- 172.21.15.28:0/501800267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb43924a20 0x55eb43926e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.877+0000 7f41f9c00640 1 -- 172.21.15.28:0/501800267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb43919a30 msgr2=0x55eb43919e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.877+0000 7f41f9c00640 1 --2- 172.21.15.28:0/501800267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb43919a30 0x55eb43919e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.877+0000 7f41f9c00640 1 -- 172.21.15.28:0/501800267 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb43879f50 con 0x55eb4391a350 2024-04-24T08:17:43.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.878+0000 7f41f9c00640 1 --2- 172.21.15.28:0/501800267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb4391a350 0x55eb439244e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f41e40114f0 tx=0x7f41e40119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9fd235eec493624 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:43.882 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:43 smithi117 ceph-mon[32438]: pgmap v226: 97 pgs: 97 active+clean; 24 GiB data, 52 GiB used, 1020 GiB / 1.0 TiB avail; 46 KiB/s rd, 18 MiB/s wr, 3.95k op/s 2024-04-24T08:17:43.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.878+0000 7f41eaa00640 1 -- 172.21.15.28:0/501800267 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e401e070 con 0x55eb4391a350 2024-04-24T08:17:43.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.878+0000 7f41eaa00640 1 -- 172.21.15.28:0/501800267 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41e4003c30 con 0x55eb4391a350 2024-04-24T08:17:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.878+0000 7f41eaa00640 1 -- 172.21.15.28:0/501800267 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e400fa80 con 0x55eb4391a350 2024-04-24T08:17:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.878+0000 7f41f2000640 1 --2- 172.21.15.28:0/3107849579 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f41d80a3d20 0x7f41d80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.879+0000 7f41f2000640 1 -- 172.21.15.28:0/3107849579 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7f41ec0d0730 con 0x7f41f419ac40 2024-04-24T08:17:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.879+0000 7f41f9200640 1 --2- 172.21.15.28:0/3107849579 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f41d80a3d20 0x7f41d80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.879+0000 7f41ff377740 1 -- 172.21.15.28:0/501800267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb4391a350 msgr2=0x55eb439244e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.879+0000 7f41ff377740 1 --2- 172.21.15.28:0/501800267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb4391a350 0x55eb439244e0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f41e40114f0 tx=0x7f41e40119b0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.879+0000 7f41ff377740 1 -- 172.21.15.28:0/501800267 shutdown_connections 2024-04-24T08:17:43.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.879+0000 7f41ff377740 1 --2- 172.21.15.28:0/501800267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb43924a20 0x55eb43926e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.879+0000 7f41ff377740 1 --2- 172.21.15.28:0/501800267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb4391a350 0x55eb439244e0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.879+0000 7f41ff377740 1 --2- 172.21.15.28:0/501800267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb43919a30 0x55eb43919e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.879+0000 7f41ff377740 1 -- 172.21.15.28:0/501800267 >> 172.21.15.28:0/501800267 conn(0x55eb43915d50 msgr2=0x55eb43916130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.879+0000 7f41ff377740 1 -- 172.21.15.28:0/501800267 shutdown_connections 2024-04-24T08:17:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.879+0000 7f41ff377740 1 -- 172.21.15.28:0/501800267 wait complete. 2024-04-24T08:17:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.879+0000 7f41f9200640 1 --2- 172.21.15.28:0/3107849579 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f41d80a3d20 0x7f41d80a61e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f41f410f200 tx=0x7f41e00314e0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.880+0000 7f41ff377740 1 -- start start 2024-04-24T08:17:43.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.880+0000 7f41ff377740 1 Processor -- start 2024-04-24T08:17:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.880+0000 7f41ff377740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb43919a30 0x55eb439219a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.880+0000 7f41ff377740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb4391a350 0x55eb43921ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.880+0000 7f41ff377740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb43924a20 0x55eb43922420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.880+0000 7f41ff377740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55eb43927810 con 0x55eb43924a20 2024-04-24T08:17:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.880+0000 7f41ff377740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55eb43838690 con 0x55eb4391a350 2024-04-24T08:17:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.880+0000 7f41ff377740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55eb43927990 con 0x55eb43919a30 2024-04-24T08:17:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.881+0000 7f41ff377740 10 client.0 _ll_register_callbacks cb 0x55eb43874650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.881+0000 7f41f9200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb4391a350 0x55eb43921ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.881+0000 7f41f9c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb43919a30 0x55eb439219a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.881+0000 7f41f9200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb4391a350 0x55eb43921ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:56562/0 (socket says 172.21.15.28:56562) 2024-04-24T08:17:43.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.881+0000 7f41f9200640 1 -- 172.21.15.28:0/3486742068 learned_addr learned my addr 172.21.15.28:0/3486742068 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:43.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.881+0000 7f41fa600640 1 --2- 172.21.15.28:0/3486742068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb43924a20 0x55eb43922420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.881+0000 7f41f9c00640 1 -- 172.21.15.28:0/3486742068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb4391a350 msgr2=0x55eb43921ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:43.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.881+0000 7f41f9c00640 1 --2- 172.21.15.28:0/3486742068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb4391a350 0x55eb43921ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.881+0000 7f41f9c00640 1 -- 172.21.15.28:0/3486742068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb43924a20 msgr2=0x55eb43922420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:43.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.881+0000 7f41f9c00640 1 --2- 172.21.15.28:0/3486742068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb43924a20 0x55eb43922420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.881+0000 7f41f9c00640 1 -- 172.21.15.28:0/3486742068 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb43837920 con 0x55eb43919a30 2024-04-24T08:17:43.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.882+0000 7f41f9c00640 1 --2- 172.21.15.28:0/3486742068 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb43919a30 0x55eb439219a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f41e4011eb0 tx=0x7f41e40092a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e78fefa1973f2c3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:43.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.882+0000 7f41df400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:17:43.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.882+0000 7f41df400640 1 -- 172.21.15.28:0/3486742068 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e401e050 con 0x55eb43919a30 2024-04-24T08:17:43.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.882+0000 7f41ff377740 1 -- 172.21.15.28:0/3486742068 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41e4003c30 con 0x55eb43919a30 2024-04-24T08:17:43.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.882+0000 7f41df400640 1 -- 172.21.15.28:0/3486742068 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7f41e4003a30 con 0x55eb43919a30 2024-04-24T08:17:43.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.882+0000 7f41df400640 1 -- 172.21.15.28:0/3486742068 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f41e4032050 con 0x55eb43919a30 2024-04-24T08:17:43.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.882+0000 7f41ff377740 10 client.26518 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:43.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.882+0000 7f41ff377740 1 -- 172.21.15.28:0/3486742068 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55eb4365b7d0 con 0x55eb43919a30 2024-04-24T08:17:43.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.883+0000 7f41df400640 1 -- 172.21.15.28:0/3486742068 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41e4003c30 con 0x55eb43919a30 2024-04-24T08:17:43.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.883+0000 7f41df400640 1 -- 172.21.15.28:0/3486742068 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f41e4005040 con 0x55eb43919a30 2024-04-24T08:17:43.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.883+0000 7f41ff377740 10 client.26518 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:43.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.883+0000 7f41ff377740 10 client.26518 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:17:43.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.883+0000 7f41ff377740 20 client.26518 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:43.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.883+0000 7f41ff377740 1 --2- 172.21.15.28:0/3486742068 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55eb43934c70 0x55eb43937060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.883+0000 7f41ff377740 4 client.26518 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:17:43.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.883+0000 7f41ff377740 1 -- 172.21.15.28:0/3486742068 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55eb4388cc10 con 0x55eb43934c70 2024-04-24T08:17:43.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.883+0000 7f41f9200640 1 --2- 172.21.15.28:0/3486742068 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55eb43934c70 0x55eb43937060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.884+0000 7f41f9200640 1 --2- 172.21.15.28:0/3486742068 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55eb43934c70 0x55eb43937060 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1010dda046498407 server_cookie=2f4380c43468386f in_seq=0 out_seq=0 2024-04-24T08:17:43.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.884+0000 7f41df400640 10 client.26518 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:43.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.885+0000 7f41df400640 1 -- 172.21.15.28:0/3486742068 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55eb4388cc10 con 0x55eb43934c70 2024-04-24T08:17:43.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.885+0000 7f41df400640 10 client.26518 handle_command_reply: tid=0 2024-04-24T08:17:43.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.885+0000 7f41ff377740 1 client.26518 shutdown 2024-04-24T08:17:43.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.885+0000 7f41ff377740 1 -- 172.21.15.28:0/3486742068 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb43919a30 msgr2=0x55eb439219a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.885+0000 7f41ff377740 1 --2- 172.21.15.28:0/3486742068 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb43919a30 0x55eb439219a0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f41e4011eb0 tx=0x7f41e40092a0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.885+0000 7f41ff377740 1 -- 172.21.15.28:0/3486742068 shutdown_connections 2024-04-24T08:17:43.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.885+0000 7f41ff377740 1 --2- 172.21.15.28:0/3486742068 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55eb43934c70 0x55eb43937060 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.885+0000 7f41ff377740 1 --2- 172.21.15.28:0/3486742068 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb43924a20 0x55eb43922420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.885+0000 7f41df400640 0 client.26518 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:43.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.885+0000 7f41ff377740 1 --2- 172.21.15.28:0/3486742068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb4391a350 0x55eb43921ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.885+0000 7f41ff377740 1 --2- 172.21.15.28:0/3486742068 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb43919a30 0x55eb439219a0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.885+0000 7f41ff377740 1 -- 172.21.15.28:0/3486742068 >> 172.21.15.28:0/3486742068 conn(0x55eb43915d50 msgr2=0x55eb43914fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:43.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.885+0000 7f41ff377740 1 -- 172.21.15.28:0/3486742068 shutdown_connections 2024-04-24T08:17:43.894 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.885+0000 7f41ff377740 1 -- 172.21.15.28:0/3486742068 wait complete. 2024-04-24T08:17:43.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.886+0000 7f41ff377740 20 client.26518 trim_cache size 0 max 16384 2024-04-24T08:17:43.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.905+0000 7f41ff377740 1 Processor -- start 2024-04-24T08:17:43.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.905+0000 7f41ff377740 1 -- start start 2024-04-24T08:17:43.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.906+0000 7f41ff377740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb43919a30 0x55eb439306d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.906+0000 7f41ff377740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb4391a350 0x55eb43914e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.906+0000 7f41ff377740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb43924a20 0x55eb4393c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.906+0000 7f41ff377740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41f41b2860 con 0x55eb43924a20 2024-04-24T08:17:43.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.906+0000 7f41ff377740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f41f41b0130 con 0x55eb43919a30 2024-04-24T08:17:43.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.906+0000 7f41ff377740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f41f41adfa0 con 0x55eb4391a350 2024-04-24T08:17:43.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.906+0000 7f41fa600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb43924a20 0x55eb4393c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.906+0000 7f41fa600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb43924a20 0x55eb4393c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36518/0 (socket says 172.21.15.28:36518) 2024-04-24T08:17:43.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.906+0000 7f41fa600640 1 -- 172.21.15.28:0/2557450761 learned_addr learned my addr 172.21.15.28:0/2557450761 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:43.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.906+0000 7f41f9c00640 1 --2- 172.21.15.28:0/2557450761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb43919a30 0x55eb439306d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.906+0000 7f41f9200640 1 --2- 172.21.15.28:0/2557450761 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb4391a350 0x55eb43914e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.906+0000 7f41f9c00640 1 -- 172.21.15.28:0/2557450761 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb4391a350 msgr2=0x55eb43914e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:43.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.906+0000 7f41f9c00640 1 --2- 172.21.15.28:0/2557450761 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb4391a350 0x55eb43914e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.906+0000 7f41f9c00640 1 -- 172.21.15.28:0/2557450761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb43924a20 msgr2=0x55eb4393c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:43.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.906+0000 7f41f9c00640 1 --2- 172.21.15.28:0/2557450761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb43924a20 0x55eb4393c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.906+0000 7f41f9c00640 1 -- 172.21.15.28:0/2557450761 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55eb435913f0 con 0x55eb43919a30 2024-04-24T08:17:43.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.907+0000 7f41f9c00640 1 --2- 172.21.15.28:0/2557450761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb43919a30 0x55eb439306d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f41e4006010 tx=0x7f41e4009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e3eb7bede2801713 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.907+0000 7f41ea000640 1 -- 172.21.15.28:0/2557450761 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e401e040 con 0x55eb43919a30 2024-04-24T08:17:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.907+0000 7f41ea000640 1 -- 172.21.15.28:0/2557450761 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41e401f040 con 0x55eb43919a30 2024-04-24T08:17:43.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.907+0000 7f41ea000640 1 -- 172.21.15.28:0/2557450761 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e4005060 con 0x55eb43919a30 2024-04-24T08:17:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.908+0000 7f41ff377740 1 -- 172.21.15.28:0/2557450761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb43919a30 msgr2=0x55eb439306d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.908+0000 7f41ff377740 1 --2- 172.21.15.28:0/2557450761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb43919a30 0x55eb439306d0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f41e4006010 tx=0x7f41e4009900 comp rx=0 tx=0).stop 2024-04-24T08:17:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.908+0000 7f41ff377740 1 -- 172.21.15.28:0/2557450761 shutdown_connections 2024-04-24T08:17:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.908+0000 7f41ff377740 1 --2- 172.21.15.28:0/2557450761 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb43924a20 0x55eb4393c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.908+0000 7f41ff377740 1 --2- 172.21.15.28:0/2557450761 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb4391a350 0x55eb43914e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.908+0000 7f41ff377740 1 --2- 172.21.15.28:0/2557450761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb43919a30 0x55eb439306d0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.908+0000 7f41ff377740 1 -- 172.21.15.28:0/2557450761 >> 172.21.15.28:0/2557450761 conn(0x55eb43915d50 msgr2=0x55eb43914740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:43.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.908+0000 7f41ff377740 1 -- 172.21.15.28:0/2557450761 shutdown_connections 2024-04-24T08:17:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.908+0000 7f41ff377740 1 -- 172.21.15.28:0/2557450761 wait complete. 2024-04-24T08:17:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.908+0000 7f41ff377740 1 -- start start 2024-04-24T08:17:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.908+0000 7f41ff377740 1 Processor -- start 2024-04-24T08:17:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.909+0000 7f41ff377740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb43919a30 0x55eb4391e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.909+0000 7f41ff377740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb4391a350 0x55eb4391eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.909+0000 7f41ff377740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb43924a20 0x55eb4391f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.909+0000 7f41ff377740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f41f41ae5f0 con 0x55eb4391a350 2024-04-24T08:17:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.909+0000 7f41ff377740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f41f41afa60 con 0x55eb43919a30 2024-04-24T08:17:43.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.909+0000 7f41ff377740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f41f41aecc0 con 0x55eb43924a20 2024-04-24T08:17:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.909+0000 7f41ff377740 10 client.0 _ll_register_callbacks cb 0x7f41e401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:17:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.909+0000 7f41f9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb4391a350 0x55eb4391eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.909+0000 7f41f9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb4391a350 0x55eb4391eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:36530/0 (socket says 172.21.15.28:36530) 2024-04-24T08:17:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.909+0000 7f41f9200640 1 -- 172.21.15.28:0/1518882319 learned_addr learned my addr 172.21.15.28:0/1518882319 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:17:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.909+0000 7f41f9c00640 1 --2- 172.21.15.28:0/1518882319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb43919a30 0x55eb4391e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.910+0000 7f41f9200640 1 -- 172.21.15.28:0/1518882319 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb43924a20 msgr2=0x55eb4391f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.910+0000 7f41f9200640 1 --2- 172.21.15.28:0/1518882319 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb43924a20 0x55eb4391f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.910+0000 7f41f9200640 1 -- 172.21.15.28:0/1518882319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb43919a30 msgr2=0x55eb4391e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:17:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.910+0000 7f41f9200640 1 --2- 172.21.15.28:0/1518882319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb43919a30 0x55eb4391e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.910+0000 7f41f9200640 1 -- 172.21.15.28:0/1518882319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55eb435510f0 con 0x55eb4391a350 2024-04-24T08:17:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.910+0000 7f41f9c00640 1 --2- 172.21.15.28:0/1518882319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb43919a30 0x55eb4391e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:17:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.910+0000 7f41f9200640 1 --2- 172.21.15.28:0/1518882319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb4391a350 0x55eb4391eb70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f41e0005940 tx=0x7f41e0009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb32cc7b9b5663a6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:17:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.910+0000 7f41df400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:17:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.910+0000 7f41df400640 1 -- 172.21.15.28:0/1518882319 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e0043070 con 0x55eb4391a350 2024-04-24T08:17:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.910+0000 7f41df400640 1 -- 172.21.15.28:0/1518882319 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 8600+0+0 (secure 0 0 0) 0x7f41e0005250 con 0x55eb4391a350 2024-04-24T08:17:43.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.910+0000 7f41ff377740 1 -- 172.21.15.28:0/1518882319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f41e401f040 con 0x55eb4391a350 2024-04-24T08:17:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.911+0000 7f41df400640 1 -- 172.21.15.28:0/1518882319 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f41e0061050 con 0x55eb4391a350 2024-04-24T08:17:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.911+0000 7f41ff377740 10 client.16221 fetch_fsmap learned FSMap version 26 2024-04-24T08:17:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.911+0000 7f41ff377740 1 -- 172.21.15.28:0/1518882319 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55eb43844810 con 0x55eb4391a350 2024-04-24T08:17:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.911+0000 7f41df400640 1 -- 172.21.15.28:0/1518882319 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41e401f040 con 0x55eb4391a350 2024-04-24T08:17:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.911+0000 7f41df400640 1 -- 172.21.15.28:0/1518882319 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f41e0004870 con 0x55eb4391a350 2024-04-24T08:17:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.911+0000 7f41ff377740 10 client.16221 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:17:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.911+0000 7f41ff377740 10 client.16221 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:17:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.911+0000 7f41ff377740 20 client.16221 populate_metadata read hostname 'smithi028' 2024-04-24T08:17:43.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.911+0000 7f41ff377740 1 --2- 172.21.15.28:0/1518882319 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55eb4399b7b0 0x55eb4399dbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:17:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.911+0000 7f41ff377740 4 client.16221 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:17:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.911+0000 7f41ff377740 1 -- 172.21.15.28:0/1518882319 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55eb43886a80 con 0x55eb4399b7b0 2024-04-24T08:17:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.911+0000 7f41f9c00640 1 --2- 172.21.15.28:0/1518882319 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55eb4399b7b0 0x55eb4399dbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:17:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.912+0000 7f41f9c00640 1 --2- 172.21.15.28:0/1518882319 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55eb4399b7b0 0x55eb4399dbf0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50d00061cc539bd2 server_cookie=3b45f5e46f0dc1a6 in_seq=0 out_seq=0 2024-04-24T08:17:43.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.912+0000 7f41df400640 10 client.16221 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.912+0000 7f41df400640 1 -- 172.21.15.28:0/1518882319 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55eb43886a80 con 0x55eb4399b7b0 2024-04-24T08:17:43.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.912+0000 7f41df400640 10 client.16221 handle_command_reply: tid=0 2024-04-24T08:17:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.912+0000 7f41ff377740 1 client.16221 shutdown 2024-04-24T08:17:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.912+0000 7f41ff377740 1 -- 172.21.15.28:0/1518882319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb4391a350 msgr2=0x55eb4391eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.912+0000 7f41ff377740 1 --2- 172.21.15.28:0/1518882319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb4391a350 0x55eb4391eb70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f41e0005940 tx=0x7f41e0009260 comp rx=0 tx=0).stop 2024-04-24T08:17:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.913+0000 7f41ff377740 1 -- 172.21.15.28:0/1518882319 shutdown_connections 2024-04-24T08:17:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.913+0000 7f41ff377740 1 --2- 172.21.15.28:0/1518882319 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55eb4399b7b0 0x55eb4399dbf0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.913+0000 7f41ff377740 1 --2- 172.21.15.28:0/1518882319 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55eb4391a350 0x55eb4391eb70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.913+0000 7f41ff377740 1 --2- 172.21.15.28:0/1518882319 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55eb43924a20 0x55eb4391f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.921 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.913+0000 7f41ff377740 1 --2- 172.21.15.28:0/1518882319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55eb43919a30 0x55eb4391e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.913+0000 7f41ff377740 1 -- 172.21.15.28:0/1518882319 >> 172.21.15.28:0/1518882319 conn(0x55eb43915d50 msgr2=0x55eb43920780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:43.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.913+0000 7f41df400640 0 client.16221 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:17:43.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.913+0000 7f41ff377740 1 -- 172.21.15.28:0/1518882319 shutdown_connections 2024-04-24T08:17:43.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.913+0000 7f41ff377740 1 -- 172.21.15.28:0/1518882319 wait complete. 2024-04-24T08:17:43.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.913+0000 7f41ff377740 20 client.16221 trim_cache size 0 max 16384 2024-04-24T08:17:43.922 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:17:43.923 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:17:43.923 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:17:43.923 INFO:teuthology.orchestra.run.smithi028.stdout: "b1f722fa-8c8f-43ee-bd6f-a8c7d4a3dabc": { 2024-04-24T08:17:43.923 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:17:43.923 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "b1f722fa-8c8f-43ee-bd6f-a8c7d4a3dabc", 2024-04-24T08:17:43.923 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:17:43.923 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:17:43.924 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:17:43.924 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:17:43.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.914+0000 7f41fb000640 1 -- 172.21.15.28:0/3107849579 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f41d80a3d20 msgr2=0x7f41d80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.914+0000 7f41fb000640 1 --2- 172.21.15.28:0/3107849579 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f41d80a3d20 0x7f41d80a61e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f41f410f200 tx=0x7f41e00314e0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.914+0000 7f41fb000640 1 -- 172.21.15.28:0/3107849579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41f419ac40 msgr2=0x7f41f419b0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:17:43.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.914+0000 7f41fb000640 1 --2- 172.21.15.28:0/3107849579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41f419ac40 0x7f41f419b0c0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f41ec00e970 tx=0x7f41ec00ee30 comp rx=0 tx=0).stop 2024-04-24T08:17:43.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.914+0000 7f41fb000640 1 -- 172.21.15.28:0/3107849579 shutdown_connections 2024-04-24T08:17:43.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.914+0000 7f41fb000640 1 --2- 172.21.15.28:0/3107849579 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f41d80a3d20 0x7f41d80a61e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.914+0000 7f41fb000640 1 --2- 172.21.15.28:0/3107849579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f41f419ac40 0x7f41f419b0c0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.914+0000 7f41fb000640 1 --2- 172.21.15.28:0/3107849579 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f41f4107260 0x7f41f41a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.914+0000 7f41fb000640 1 --2- 172.21.15.28:0/3107849579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41f4100a50 0x7f41f41a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:17:43.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.914+0000 7f41fb000640 1 -- 172.21.15.28:0/3107849579 >> 172.21.15.28:0/3107849579 conn(0x7f41f4076090 msgr2=0x7f41f40fe760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:17:43.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.914+0000 7f41fb000640 1 -- 172.21.15.28:0/3107849579 shutdown_connections 2024-04-24T08:17:43.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:17:43.914+0000 7f41fb000640 1 -- 172.21.15.28:0/3107849579 wait complete. 2024-04-24T08:17:43.933 INFO:tasks.cephfs.filesystem:scrub status for tag:b1f722fa-8c8f-43ee-bd6f-a8c7d4a3dabc - {'path': '/', 'tag': 'b1f722fa-8c8f-43ee-bd6f-a8c7d4a3dabc', 'options': 'recursive,force'} 2024-04-24T08:17:43.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:43 smithi194 ceph-mon[32836]: pgmap v226: 97 pgs: 97 active+clean; 24 GiB data, 52 GiB used, 1020 GiB / 1.0 TiB avail; 46 KiB/s rd, 18 MiB/s wr, 3.95k op/s 2024-04-24T08:17:44.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:44 smithi028 ceph-mon[27786]: scrub queued for path: / 2024-04-24T08:17:44.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:44 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:44.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:44 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:17:44.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:44 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:44.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:44 smithi117 ceph-mon[32438]: scrub queued for path: / 2024-04-24T08:17:44.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:44 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:44.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:44 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:17:44.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:44 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:44.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:44 smithi194 ceph-mon[32836]: scrub queued for path: / 2024-04-24T08:17:44.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:44 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:44.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:44 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:17:44.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:44 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:45.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:45 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:17:45.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:45 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:45.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:45 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:17:45.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:45 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:45.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:45 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:17:45.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:45 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:45.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:45 smithi117 ceph-mon[32438]: pgmap v227: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 47 KiB/s rd, 15 MiB/s wr, 3.24k op/s 2024-04-24T08:17:45.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:45 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:17:45.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:45 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:45.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:45 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:17:45.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:45 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:45.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:45 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:17:45.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:45 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:45.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:45 smithi194 ceph-mon[32836]: pgmap v227: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 47 KiB/s rd, 15 MiB/s wr, 3.24k op/s 2024-04-24T08:17:46.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:45 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:17:46.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:45 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:46.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:45 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:17:46.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:45 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:46.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:45 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:17:46.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:45 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:17:46.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:45 smithi028 ceph-mon[27786]: pgmap v227: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 47 KiB/s rd, 15 MiB/s wr, 3.24k op/s 2024-04-24T08:17:47.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:47 smithi117 ceph-mon[32438]: pgmap v228: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 46 KiB/s rd, 15 MiB/s wr, 3.18k op/s 2024-04-24T08:17:47.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:47 smithi194 ceph-mon[32836]: pgmap v228: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 46 KiB/s rd, 15 MiB/s wr, 3.18k op/s 2024-04-24T08:17:48.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:47 smithi028 ceph-mon[27786]: pgmap v228: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 46 KiB/s rd, 15 MiB/s wr, 3.18k op/s 2024-04-24T08:17:49.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:49 smithi117 ceph-mon[32438]: pgmap v229: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 24 KiB/s rd, 18 MiB/s wr, 3.73k op/s 2024-04-24T08:17:49.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:49 smithi194 ceph-mon[32836]: pgmap v229: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 24 KiB/s rd, 18 MiB/s wr, 3.73k op/s 2024-04-24T08:17:50.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:49 smithi028 ceph-mon[27786]: pgmap v229: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1020 GiB / 1.0 TiB avail; 24 KiB/s rd, 18 MiB/s wr, 3.73k op/s 2024-04-24T08:17:51.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:51 smithi117 ceph-mon[32438]: pgmap v230: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1019 GiB / 1.0 TiB avail; 24 KiB/s rd, 21 MiB/s wr, 4.54k op/s 2024-04-24T08:17:51.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:51 smithi194 ceph-mon[32836]: pgmap v230: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1019 GiB / 1.0 TiB avail; 24 KiB/s rd, 21 MiB/s wr, 4.54k op/s 2024-04-24T08:17:52.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:51 smithi028 ceph-mon[27786]: pgmap v230: 97 pgs: 97 active+clean; 24 GiB data, 53 GiB used, 1019 GiB / 1.0 TiB avail; 24 KiB/s rd, 21 MiB/s wr, 4.54k op/s 2024-04-24T08:17:53.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:53 smithi117 ceph-mon[32438]: pgmap v231: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 40 KiB/s rd, 23 MiB/s wr, 4.85k op/s 2024-04-24T08:17:53.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:53 smithi194 ceph-mon[32836]: pgmap v231: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 40 KiB/s rd, 23 MiB/s wr, 4.85k op/s 2024-04-24T08:17:54.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:53 smithi028 ceph-mon[27786]: pgmap v231: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 40 KiB/s rd, 23 MiB/s wr, 4.85k op/s 2024-04-24T08:17:54.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:54 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:17:54.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:54 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:17:55.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:54 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:17:55.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:55 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:17:55.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:55 smithi117 ceph-mon[32438]: pgmap v232: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 37 KiB/s rd, 17 MiB/s wr, 3.62k op/s 2024-04-24T08:17:55.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:55 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:17:55.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:55 smithi194 ceph-mon[32836]: pgmap v232: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 37 KiB/s rd, 17 MiB/s wr, 3.62k op/s 2024-04-24T08:17:56.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:55 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:17:56.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:55 smithi028 ceph-mon[27786]: pgmap v232: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 37 KiB/s rd, 17 MiB/s wr, 3.62k op/s 2024-04-24T08:17:57.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:57 smithi117 ceph-mon[32438]: pgmap v233: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 36 KiB/s rd, 14 MiB/s wr, 2.97k op/s 2024-04-24T08:17:57.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:57 smithi194 ceph-mon[32836]: pgmap v233: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 36 KiB/s rd, 14 MiB/s wr, 2.97k op/s 2024-04-24T08:17:58.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:57 smithi028 ceph-mon[27786]: pgmap v233: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 36 KiB/s rd, 14 MiB/s wr, 2.97k op/s 2024-04-24T08:17:59.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:17:59 smithi117 ceph-mon[32438]: pgmap v234: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 41 KiB/s rd, 12 MiB/s wr, 2.70k op/s 2024-04-24T08:17:59.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:17:59 smithi194 ceph-mon[32836]: pgmap v234: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 41 KiB/s rd, 12 MiB/s wr, 2.70k op/s 2024-04-24T08:18:00.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:17:59 smithi028 ceph-mon[27786]: pgmap v234: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 41 KiB/s rd, 12 MiB/s wr, 2.70k op/s 2024-04-24T08:18:01.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:01 smithi117 ceph-mon[32438]: pgmap v235: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 24 KiB/s rd, 6.5 MiB/s wr, 1.42k op/s 2024-04-24T08:18:01.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:01 smithi194 ceph-mon[32836]: pgmap v235: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 24 KiB/s rd, 6.5 MiB/s wr, 1.42k op/s 2024-04-24T08:18:02.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:01 smithi028 ceph-mon[27786]: pgmap v235: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 24 KiB/s rd, 6.5 MiB/s wr, 1.42k op/s 2024-04-24T08:18:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:03 smithi117 ceph-mon[32438]: pgmap v236: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 33 KiB/s rd, 2.9 MiB/s wr, 622 op/s 2024-04-24T08:18:03.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:03 smithi194 ceph-mon[32836]: pgmap v236: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 33 KiB/s rd, 2.9 MiB/s wr, 622 op/s 2024-04-24T08:18:04.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:03 smithi028 ceph-mon[27786]: pgmap v236: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 33 KiB/s rd, 2.9 MiB/s wr, 622 op/s 2024-04-24T08:18:05.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:05 smithi117 ceph-mon[32438]: pgmap v237: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 17 KiB/s rd, 760 KiB/s wr, 156 op/s 2024-04-24T08:18:05.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:05 smithi194 ceph-mon[32836]: pgmap v237: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 17 KiB/s rd, 760 KiB/s wr, 156 op/s 2024-04-24T08:18:06.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:05 smithi028 ceph-mon[27786]: pgmap v237: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 17 KiB/s rd, 760 KiB/s wr, 156 op/s 2024-04-24T08:18:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:07 smithi117 ceph-mon[32438]: pgmap v238: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 17 KiB/s rd, 782 KiB/s wr, 157 op/s 2024-04-24T08:18:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:07 smithi194 ceph-mon[32836]: pgmap v238: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 17 KiB/s rd, 782 KiB/s wr, 157 op/s 2024-04-24T08:18:08.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:07 smithi028 ceph-mon[27786]: pgmap v238: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 17 KiB/s rd, 782 KiB/s wr, 157 op/s 2024-04-24T08:18:08.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:08 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:18:08.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:08 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:18:09.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:08 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:18:09.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:09 smithi194 ceph-mon[32836]: pgmap v239: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 65 KiB/s rd, 829 KiB/s wr, 159 op/s 2024-04-24T08:18:09.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:09 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:09.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:09 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:18:09.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:09 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:18:10.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:09 smithi028 ceph-mon[27786]: pgmap v239: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 65 KiB/s rd, 829 KiB/s wr, 159 op/s 2024-04-24T08:18:10.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:09 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:10.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:09 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:18:10.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:09 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:18:10.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:09 smithi117 ceph-mon[32438]: pgmap v239: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 65 KiB/s rd, 829 KiB/s wr, 159 op/s 2024-04-24T08:18:10.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:09 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:18:10.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:09 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:18:10.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:09 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:18:10.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:10 smithi194 ceph-mon[32836]: osdmap e94: 12 total, 12 up, 12 in 2024-04-24T08:18:11.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:10 smithi028 ceph-mon[27786]: osdmap e94: 12 total, 12 up, 12 in 2024-04-24T08:18:11.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:10 smithi117 ceph-mon[32438]: osdmap e94: 12 total, 12 up, 12 in 2024-04-24T08:18:11.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:11 smithi194 ceph-mon[32836]: pgmap v241: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 72 KiB/s rd, 110 KiB/s wr, 27 op/s 2024-04-24T08:18:11.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:11 smithi028 ceph-mon[27786]: pgmap v241: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 72 KiB/s rd, 110 KiB/s wr, 27 op/s 2024-04-24T08:18:12.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:11 smithi117 ceph-mon[32438]: pgmap v241: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 72 KiB/s rd, 110 KiB/s wr, 27 op/s 2024-04-24T08:18:13.935 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:18:13.935 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:18:13.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:13 smithi194 ceph-mon[32836]: pgmap v242: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 67 KiB/s rd, 90 KiB/s wr, 11 op/s 2024-04-24T08:18:14.017 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:13 smithi028 ceph-mon[27786]: pgmap v242: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 67 KiB/s rd, 90 KiB/s wr, 11 op/s 2024-04-24T08:18:14.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:13 smithi117 ceph-mon[32438]: pgmap v242: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 67 KiB/s rd, 90 KiB/s wr, 11 op/s 2024-04-24T08:18:14.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.160+0000 7f5c98a00640 1 -- 172.21.15.28:0/2412379397 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c940684b0 msgr2=0x7f5c94104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.160+0000 7f5c98a00640 1 --2- 172.21.15.28:0/2412379397 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c940684b0 0x7f5c94104d70 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f5c84009900 tx=0x7f5c8402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.160+0000 7f5c98a00640 1 -- 172.21.15.28:0/2412379397 shutdown_connections 2024-04-24T08:18:14.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.160+0000 7f5c98a00640 1 --2- 172.21.15.28:0/2412379397 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c9410b010 0x7f5c9410d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.160+0000 7f5c98a00640 1 --2- 172.21.15.28:0/2412379397 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c94100f40 0x7f5c941052b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.160+0000 7f5c98a00640 1 --2- 172.21.15.28:0/2412379397 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c940684b0 0x7f5c94104d70 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.160+0000 7f5c98a00640 1 -- 172.21.15.28:0/2412379397 >> 172.21.15.28:0/2412379397 conn(0x7f5c94075d00 msgr2=0x7f5c94076110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.161+0000 7f5c98a00640 1 -- 172.21.15.28:0/2412379397 shutdown_connections 2024-04-24T08:18:14.163 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.161+0000 7f5c98a00640 1 -- 172.21.15.28:0/2412379397 wait complete. 2024-04-24T08:18:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.162+0000 7f5c98a00640 1 Processor -- start 2024-04-24T08:18:14.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.162+0000 7f5c98a00640 1 -- start start 2024-04-24T08:18:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.162+0000 7f5c98a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c940684b0 0x7f5c941a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.162+0000 7f5c98a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c94100f40 0x7f5c941a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.162+0000 7f5c98a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c9410b010 0x7f5c94109bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.162+0000 7f5c98a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c94112340 con 0x7f5c9410b010 2024-04-24T08:18:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.162+0000 7f5c98a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5c941121c0 con 0x7f5c94100f40 2024-04-24T08:18:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.162+0000 7f5c98a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5c941124c0 con 0x7f5c940684b0 2024-04-24T08:18:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.163+0000 7f5c93e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c9410b010 0x7f5c94109bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.163+0000 7f5c93e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c9410b010 0x7f5c94109bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40444/0 (socket says 172.21.15.28:40444) 2024-04-24T08:18:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.163+0000 7f5c93e00640 1 -- 172.21.15.28:0/3230725264 learned_addr learned my addr 172.21.15.28:0/3230725264 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.163+0000 7f5c93400640 1 --2- 172.21.15.28:0/3230725264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c940684b0 0x7f5c941a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.163+0000 7f5c92a00640 1 --2- 172.21.15.28:0/3230725264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c94100f40 0x7f5c941a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.163+0000 7f5c93e00640 1 -- 172.21.15.28:0/3230725264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c940684b0 msgr2=0x7f5c941a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.163+0000 7f5c93e00640 1 --2- 172.21.15.28:0/3230725264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c940684b0 0x7f5c941a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.163+0000 7f5c93e00640 1 -- 172.21.15.28:0/3230725264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c94100f40 msgr2=0x7f5c941a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.163+0000 7f5c93e00640 1 --2- 172.21.15.28:0/3230725264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c94100f40 0x7f5c941a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.163+0000 7f5c93e00640 1 -- 172.21.15.28:0/3230725264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c9410a3c0 con 0x7f5c9410b010 2024-04-24T08:18:14.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.163+0000 7f5c93e00640 1 --2- 172.21.15.28:0/3230725264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c9410b010 0x7f5c94109bf0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00eb40 tx=0x7f5c8c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.164+0000 7f5c8be00640 1 -- 172.21.15.28:0/3230725264 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c004b20 con 0x7f5c9410b010 2024-04-24T08:18:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.164+0000 7f5c8be00640 1 -- 172.21.15.28:0/3230725264 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c8c004cc0 con 0x7f5c9410b010 2024-04-24T08:18:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.164+0000 7f5c98a00640 1 -- 172.21.15.28:0/3230725264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c9410a6b0 con 0x7f5c9410b010 2024-04-24T08:18:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.164+0000 7f5c8be00640 1 -- 172.21.15.28:0/3230725264 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c010850 con 0x7f5c9410b010 2024-04-24T08:18:14.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.164+0000 7f5c98a00640 1 -- 172.21.15.28:0/3230725264 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c941a7490 con 0x7f5c9410b010 2024-04-24T08:18:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.167+0000 7f5c9cdeb740 1 Processor -- start 2024-04-24T08:18:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.167+0000 7f5c8be00640 1 -- 172.21.15.28:0/3230725264 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f5c8c005020 con 0x7f5c9410b010 2024-04-24T08:18:14.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.167+0000 7f5c9cdeb740 1 -- start start 2024-04-24T08:18:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.168+0000 7f5c9cdeb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73ca30 0x563b3a73ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.168+0000 7f5c9cdeb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a73d350 0x563b3a7474e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.168+0000 7f5c9cdeb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a747a20 0x563b3a749e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.168+0000 7f5c9cdeb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563b3a6a21f0 con 0x563b3a747a20 2024-04-24T08:18:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.168+0000 7f5c9cdeb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563b3a648350 con 0x563b3a73d350 2024-04-24T08:18:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.168+0000 7f5c9cdeb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563b3a69e1b0 con 0x563b3a73ca30 2024-04-24T08:18:14.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.168+0000 7f5c92a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a747a20 0x563b3a749e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.168+0000 7f5c92a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73ca30 0x563b3a73ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.168+0000 7f5c93400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a73d350 0x563b3a7474e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.168+0000 7f5c93400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a73d350 0x563b3a7474e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:39974/0 (socket says 172.21.15.28:39974) 2024-04-24T08:18:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.168+0000 7f5c92a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73ca30 0x563b3a73ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:50526/0 (socket says 172.21.15.28:50526) 2024-04-24T08:18:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.168+0000 7f5c93400640 1 -- 172.21.15.28:0/1564114135 learned_addr learned my addr 172.21.15.28:0/1564114135 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.169+0000 7f5c93400640 1 -- 172.21.15.28:0/1564114135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73ca30 msgr2=0x563b3a73ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.169+0000 7f5c93400640 1 --2- 172.21.15.28:0/1564114135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73ca30 0x563b3a73ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.169+0000 7f5c93400640 1 -- 172.21.15.28:0/1564114135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a747a20 msgr2=0x563b3a749e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.169+0000 7f5c93400640 1 --2- 172.21.15.28:0/1564114135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a747a20 0x563b3a749e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.169+0000 7f5c93400640 1 -- 172.21.15.28:0/1564114135 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b3a69cf50 con 0x563b3a73d350 2024-04-24T08:18:14.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.169+0000 7f5c8be00640 1 --2- 172.21.15.28:0/3230725264 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5c780a3da0 0x7f5c780a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.169+0000 7f5c93400640 1 --2- 172.21.15.28:0/1564114135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a73d350 0x563b3a7474e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5c940773b0 tx=0x7f5c84005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5053527ddb4dab6a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.169+0000 7f5c8be00640 1 -- 172.21.15.28:0/3230725264 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7f5c8c0d1060 con 0x7f5c9410b010 2024-04-24T08:18:14.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.169+0000 7f5c92a00640 1 --2- 172.21.15.28:0/3230725264 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5c780a3da0 0x7f5c780a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.169+0000 7f5c7fe00640 1 -- 172.21.15.28:0/1564114135 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c84035040 con 0x563b3a73d350 2024-04-24T08:18:14.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.170+0000 7f5c7fe00640 1 -- 172.21.15.28:0/1564114135 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c84005520 con 0x563b3a73d350 2024-04-24T08:18:14.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.170+0000 7f5c7fe00640 1 -- 172.21.15.28:0/1564114135 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c840035f0 con 0x563b3a73d350 2024-04-24T08:18:14.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.170+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/1564114135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a73d350 msgr2=0x563b3a7474e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.170+0000 7f5c92a00640 1 --2- 172.21.15.28:0/3230725264 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5c780a3da0 0x7f5c780a6260 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x563b3a739ee0 tx=0x7f5c80004730 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.170+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/1564114135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a73d350 0x563b3a7474e0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f5c940773b0 tx=0x7f5c84005de0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.170+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/1564114135 shutdown_connections 2024-04-24T08:18:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.170+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/1564114135 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a747a20 0x563b3a749e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.170+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/1564114135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a73d350 0x563b3a7474e0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.170+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/1564114135 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73ca30 0x563b3a73ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.170+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/1564114135 >> 172.21.15.28:0/1564114135 conn(0x563b3a738d50 msgr2=0x563b3a739130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.171+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/1564114135 shutdown_connections 2024-04-24T08:18:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.171+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/1564114135 wait complete. 2024-04-24T08:18:14.180 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.171+0000 7f5c9cdeb740 1 -- start start 2024-04-24T08:18:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.171+0000 7f5c9cdeb740 1 Processor -- start 2024-04-24T08:18:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.172+0000 7f5c9cdeb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73ca30 0x563b3a7449d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.172+0000 7f5c9cdeb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a73d350 0x563b3a744f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.172+0000 7f5c9cdeb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a747a20 0x563b3a745450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.172+0000 7f5c9cdeb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563b3a74a7b0 con 0x563b3a73d350 2024-04-24T08:18:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.172+0000 7f5c9cdeb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563b3a65b690 con 0x563b3a747a20 2024-04-24T08:18:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.172+0000 7f5c9cdeb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563b3a74a930 con 0x563b3a73ca30 2024-04-24T08:18:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.172+0000 7f5c9cdeb740 10 client.0 _ll_register_callbacks cb 0x563b3a697650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:14.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.172+0000 7f5c92a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a73d350 0x563b3a744f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.172+0000 7f5c93400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73ca30 0x563b3a7449d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.172+0000 7f5c92a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a73d350 0x563b3a744f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40454/0 (socket says 172.21.15.28:40454) 2024-04-24T08:18:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.172+0000 7f5c92a00640 1 -- 172.21.15.28:0/2942970603 learned_addr learned my addr 172.21.15.28:0/2942970603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.172+0000 7f5c93e00640 1 --2- 172.21.15.28:0/2942970603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a747a20 0x563b3a745450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.173+0000 7f5c93400640 1 -- 172.21.15.28:0/2942970603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a747a20 msgr2=0x563b3a745450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.173+0000 7f5c93400640 1 --2- 172.21.15.28:0/2942970603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a747a20 0x563b3a745450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.173+0000 7f5c93400640 1 -- 172.21.15.28:0/2942970603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a73d350 msgr2=0x563b3a744f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.173+0000 7f5c93400640 1 --2- 172.21.15.28:0/2942970603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a73d350 0x563b3a744f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.173+0000 7f5c93400640 1 -- 172.21.15.28:0/2942970603 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b3a65a920 con 0x563b3a73ca30 2024-04-24T08:18:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.173+0000 7f5c93400640 1 --2- 172.21.15.28:0/2942970603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73ca30 0x563b3a7449d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5c84031550 tx=0x7f5c84004970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=75effb3be4f01490 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.173+0000 7f5c7cc00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:18:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.173+0000 7f5c7cc00640 1 -- 172.21.15.28:0/2942970603 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c84035040 con 0x563b3a73ca30 2024-04-24T08:18:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.173+0000 7f5c7cc00640 1 -- 172.21.15.28:0/2942970603 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7f5c84004bb0 con 0x563b3a73ca30 2024-04-24T08:18:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.173+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2942970603 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c84005520 con 0x563b3a73ca30 2024-04-24T08:18:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.174+0000 7f5c7cc00640 1 -- 172.21.15.28:0/2942970603 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5c84009520 con 0x563b3a73ca30 2024-04-24T08:18:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.174+0000 7f5c9cdeb740 10 client.26533 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:14.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.174+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2942970603 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563b3a47e7d0 con 0x563b3a73ca30 2024-04-24T08:18:14.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.174+0000 7f5c7cc00640 1 -- 172.21.15.28:0/2942970603 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c84005520 con 0x563b3a73ca30 2024-04-24T08:18:14.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.174+0000 7f5c7cc00640 1 -- 172.21.15.28:0/2942970603 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f5c84055030 con 0x563b3a73ca30 2024-04-24T08:18:14.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.175+0000 7f5c9cdeb740 10 client.26533 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:14.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.175+0000 7f5c9cdeb740 10 client.26533 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:18:14.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.175+0000 7f5c9cdeb740 20 client.26533 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:14.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.175+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/2942970603 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563b3a757c20 0x563b3a75a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.175+0000 7f5c9cdeb740 4 client.26533 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:18:14.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.175+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2942970603 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b3a6afc10 con 0x563b3a757c20 2024-04-24T08:18:14.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.175+0000 7f5c92a00640 1 --2- 172.21.15.28:0/2942970603 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563b3a757c20 0x563b3a75a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.176+0000 7f5c92a00640 1 --2- 172.21.15.28:0/2942970603 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563b3a757c20 0x563b3a75a010 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d08888ff25957d1a server_cookie=435ba803df79a733 in_seq=0 out_seq=0 2024-04-24T08:18:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.176+0000 7f5c7cc00640 10 client.26533 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.176+0000 7f5c7cc00640 1 -- 172.21.15.28:0/2942970603 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563b3a6afc10 con 0x563b3a757c20 2024-04-24T08:18:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.176+0000 7f5c7cc00640 10 client.26533 handle_command_reply: tid=0 2024-04-24T08:18:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.176+0000 7f5c9cdeb740 1 client.26533 shutdown 2024-04-24T08:18:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.177+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2942970603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73ca30 msgr2=0x563b3a7449d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.177+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/2942970603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73ca30 0x563b3a7449d0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f5c84031550 tx=0x7f5c84004970 comp rx=0 tx=0).stop 2024-04-24T08:18:14.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.177+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2942970603 shutdown_connections 2024-04-24T08:18:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.177+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/2942970603 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563b3a757c20 0x563b3a75a010 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.177+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/2942970603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a73d350 0x563b3a744f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.177+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/2942970603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a747a20 0x563b3a745450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.177+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/2942970603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73ca30 0x563b3a7449d0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.177+0000 7f5c7cc00640 0 client.26533 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.177+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2942970603 >> 172.21.15.28:0/2942970603 conn(0x563b3a738d50 msgr2=0x563b3a73eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.177+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2942970603 shutdown_connections 2024-04-24T08:18:14.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.177+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2942970603 wait complete. 2024-04-24T08:18:14.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.177+0000 7f5c9cdeb740 20 client.26533 trim_cache size 0 max 16384 2024-04-24T08:18:14.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.198+0000 7f5c9cdeb740 1 Processor -- start 2024-04-24T08:18:14.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.199+0000 7f5c9cdeb740 1 -- start start 2024-04-24T08:18:14.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.199+0000 7f5c9cdeb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a73ca30 0x563b3a7536a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.199+0000 7f5c9cdeb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73d350 0x563b3a737e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.199+0000 7f5c9cdeb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a747a20 0x563b3a75f610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.199+0000 7f5c9cdeb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c941b27d0 con 0x563b3a747a20 2024-04-24T08:18:14.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.199+0000 7f5c9cdeb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5c941b00a0 con 0x563b3a73ca30 2024-04-24T08:18:14.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.199+0000 7f5c9cdeb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5c941adf10 con 0x563b3a73d350 2024-04-24T08:18:14.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.199+0000 7f5c93e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a747a20 0x563b3a75f610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.199+0000 7f5c93e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a747a20 0x563b3a75f610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40460/0 (socket says 172.21.15.28:40460) 2024-04-24T08:18:14.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.199+0000 7f5c93e00640 1 -- 172.21.15.28:0/2435832725 learned_addr learned my addr 172.21.15.28:0/2435832725 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:14.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.199+0000 7f5c92a00640 1 --2- 172.21.15.28:0/2435832725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73d350 0x563b3a737e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.199+0000 7f5c93400640 1 --2- 172.21.15.28:0/2435832725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a73ca30 0x563b3a7536a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.200+0000 7f5c93e00640 1 -- 172.21.15.28:0/2435832725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73d350 msgr2=0x563b3a737e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.200+0000 7f5c93e00640 1 --2- 172.21.15.28:0/2435832725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73d350 0x563b3a737e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.200+0000 7f5c93e00640 1 -- 172.21.15.28:0/2435832725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a73ca30 msgr2=0x563b3a7536a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.200+0000 7f5c93e00640 1 --2- 172.21.15.28:0/2435832725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a73ca30 0x563b3a7536a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.200+0000 7f5c93e00640 1 -- 172.21.15.28:0/2435832725 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b3a73f7f0 con 0x563b3a747a20 2024-04-24T08:18:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.200+0000 7f5c92a00640 1 --2- 172.21.15.28:0/2435832725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73d350 0x563b3a737e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.200+0000 7f5c93e00640 1 --2- 172.21.15.28:0/2435832725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a747a20 0x563b3a75f610 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00c7d0 tx=0x7f5c8c0056e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e13433c0cfbd2ae server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.200+0000 7f5c7f400640 1 -- 172.21.15.28:0/2435832725 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c09cf20 con 0x563b3a747a20 2024-04-24T08:18:14.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.200+0000 7f5c7f400640 1 -- 172.21.15.28:0/2435832725 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c8c0a12d0 con 0x563b3a747a20 2024-04-24T08:18:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.200+0000 7f5c7f400640 1 -- 172.21.15.28:0/2435832725 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c8c0a5700 con 0x563b3a747a20 2024-04-24T08:18:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.201+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2435832725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a747a20 msgr2=0x563b3a75f610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.201+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/2435832725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a747a20 0x563b3a75f610 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00c7d0 tx=0x7f5c8c0056e0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.201+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2435832725 shutdown_connections 2024-04-24T08:18:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.201+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/2435832725 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a747a20 0x563b3a75f610 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.201+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/2435832725 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73d350 0x563b3a737e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.201+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/2435832725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a73ca30 0x563b3a7536a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.201+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2435832725 >> 172.21.15.28:0/2435832725 conn(0x563b3a738d50 msgr2=0x563b3a737740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.201+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2435832725 shutdown_connections 2024-04-24T08:18:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.201+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2435832725 wait complete. 2024-04-24T08:18:14.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.202+0000 7f5c9cdeb740 1 -- start start 2024-04-24T08:18:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.202+0000 7f5c9cdeb740 1 Processor -- start 2024-04-24T08:18:14.209 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.202+0000 7f5c9cdeb740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a73ca30 0x563b3a74fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.202+0000 7f5c9cdeb740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73d350 0x563b3a7500f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.203+0000 7f5c9cdeb740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a747a20 0x563b3a740e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.203+0000 7f5c9cdeb740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f5c941ae560 con 0x563b3a73ca30 2024-04-24T08:18:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.203+0000 7f5c9cdeb740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f5c941af9d0 con 0x563b3a747a20 2024-04-24T08:18:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.203+0000 7f5c9cdeb740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f5c941aec30 con 0x563b3a73d350 2024-04-24T08:18:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.203+0000 7f5c9cdeb740 10 client.0 _ll_register_callbacks cb 0x7f5c80009dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.203+0000 7f5c93400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a73ca30 0x563b3a74fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.210 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.203+0000 7f5c93400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a73ca30 0x563b3a74fbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40470/0 (socket says 172.21.15.28:40470) 2024-04-24T08:18:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.203+0000 7f5c93400640 1 -- 172.21.15.28:0/2939333307 learned_addr learned my addr 172.21.15.28:0/2939333307 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.203+0000 7f5c93e00640 1 --2- 172.21.15.28:0/2939333307 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a747a20 0x563b3a740e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.203+0000 7f5c92a00640 1 --2- 172.21.15.28:0/2939333307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73d350 0x563b3a7500f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.203+0000 7f5c93400640 1 -- 172.21.15.28:0/2939333307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73d350 msgr2=0x563b3a7500f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.203+0000 7f5c93400640 1 --2- 172.21.15.28:0/2939333307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73d350 0x563b3a7500f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.203+0000 7f5c93400640 1 -- 172.21.15.28:0/2939333307 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a747a20 msgr2=0x563b3a740e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.203+0000 7f5c93400640 1 --2- 172.21.15.28:0/2939333307 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a747a20 0x563b3a740e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.203+0000 7f5c93400640 1 -- 172.21.15.28:0/2939333307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b3a3b43f0 con 0x563b3a73ca30 2024-04-24T08:18:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.204+0000 7f5c93400640 1 --2- 172.21.15.28:0/2939333307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a73ca30 0x563b3a74fbb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5c84001eb0 tx=0x7f5c84005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=386008cb70436a73 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.204+0000 7f5c7cc00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:18:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.204+0000 7f5c7cc00640 1 -- 172.21.15.28:0/2939333307 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c84063060 con 0x563b3a73ca30 2024-04-24T08:18:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.204+0000 7f5c7cc00640 1 -- 172.21.15.28:0/2939333307 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7f5c84003b40 con 0x563b3a73ca30 2024-04-24T08:18:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.204+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2939333307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c8c0a12d0 con 0x563b3a73ca30 2024-04-24T08:18:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.204+0000 7f5c7cc00640 1 -- 172.21.15.28:0/2939333307 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f5c84035040 con 0x563b3a73ca30 2024-04-24T08:18:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.204+0000 7f5c9cdeb740 10 client.16239 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.204+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2939333307 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563b3a3740f0 con 0x563b3a73ca30 2024-04-24T08:18:14.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.205+0000 7f5c7cc00640 1 -- 172.21.15.28:0/2939333307 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c8c0a12d0 con 0x563b3a73ca30 2024-04-24T08:18:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.205+0000 7f5c7cc00640 1 -- 172.21.15.28:0/2939333307 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f5c84055070 con 0x563b3a73ca30 2024-04-24T08:18:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.205+0000 7f5c9cdeb740 10 client.16239 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.205+0000 7f5c9cdeb740 10 client.16239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:18:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.205+0000 7f5c9cdeb740 20 client.16239 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.205+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/2939333307 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563b3a7bdad0 0x563b3a7bfef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.205+0000 7f5c9cdeb740 4 client.16239 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:18:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.205+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2939333307 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563b3a6a9a80 con 0x563b3a7bdad0 2024-04-24T08:18:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.205+0000 7f5c92a00640 1 --2- 172.21.15.28:0/2939333307 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563b3a7bdad0 0x563b3a7bfef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.206+0000 7f5c92a00640 1 --2- 172.21.15.28:0/2939333307 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563b3a7bdad0 0x563b3a7bfef0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40af6b8676bbefae server_cookie=c6904acd3c02f64 in_seq=0 out_seq=0 2024-04-24T08:18:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.206+0000 7f5c7cc00640 10 client.16239 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.206+0000 7f5c7cc00640 1 -- 172.21.15.28:0/2939333307 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x563b3a6a9a80 con 0x563b3a7bdad0 2024-04-24T08:18:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.206+0000 7f5c7cc00640 10 client.16239 handle_command_reply: tid=0 2024-04-24T08:18:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.206+0000 7f5c9cdeb740 1 client.16239 shutdown 2024-04-24T08:18:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.206+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2939333307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a73ca30 msgr2=0x563b3a74fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.206+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/2939333307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a73ca30 0x563b3a74fbb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f5c84001eb0 tx=0x7f5c84005b50 comp rx=0 tx=0).stop 2024-04-24T08:18:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.206+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2939333307 shutdown_connections 2024-04-24T08:18:14.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.206+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/2939333307 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563b3a7bdad0 0x563b3a7bfef0 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.206+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/2939333307 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563b3a747a20 0x563b3a740e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.206+0000 7f5c7cc00640 0 client.16239 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.206+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/2939333307 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563b3a73d350 0x563b3a7500f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.206+0000 7f5c9cdeb740 1 --2- 172.21.15.28:0/2939333307 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563b3a73ca30 0x563b3a74fbb0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.206+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2939333307 >> 172.21.15.28:0/2939333307 conn(0x563b3a738d50 msgr2=0x563b3a737fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.206+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2939333307 shutdown_connections 2024-04-24T08:18:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.207+0000 7f5c9cdeb740 1 -- 172.21.15.28:0/2939333307 wait complete. 2024-04-24T08:18:14.215 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.207+0000 7f5c9cdeb740 20 client.16239 trim_cache size 0 max 16384 2024-04-24T08:18:14.216 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:18:14.216 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-24T08:18:14.216 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-24T08:18:14.216 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:18:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.207+0000 7f5c98a00640 1 -- 172.21.15.28:0/3230725264 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5c780a3da0 msgr2=0x7f5c780a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.207+0000 7f5c98a00640 1 --2- 172.21.15.28:0/3230725264 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5c780a3da0 0x7f5c780a6260 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x563b3a739ee0 tx=0x7f5c80004730 comp rx=0 tx=0).stop 2024-04-24T08:18:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.207+0000 7f5c98a00640 1 -- 172.21.15.28:0/3230725264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c9410b010 msgr2=0x7f5c94109bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.207+0000 7f5c98a00640 1 --2- 172.21.15.28:0/3230725264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c9410b010 0x7f5c94109bf0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f5c8c00eb40 tx=0x7f5c8c00c2d0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.207+0000 7f5c98a00640 1 -- 172.21.15.28:0/3230725264 shutdown_connections 2024-04-24T08:18:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.207+0000 7f5c98a00640 1 --2- 172.21.15.28:0/3230725264 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f5c780a3da0 0x7f5c780a6260 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.207+0000 7f5c98a00640 1 --2- 172.21.15.28:0/3230725264 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f5c9410b010 0x7f5c94109bf0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.207+0000 7f5c98a00640 1 --2- 172.21.15.28:0/3230725264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c94100f40 0x7f5c941a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.207+0000 7f5c98a00640 1 --2- 172.21.15.28:0/3230725264 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f5c940684b0 0x7f5c941a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.208+0000 7f5c98a00640 1 -- 172.21.15.28:0/3230725264 >> 172.21.15.28:0/3230725264 conn(0x7f5c94075d00 msgr2=0x7f5c940ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.208+0000 7f5c98a00640 1 -- 172.21.15.28:0/3230725264 shutdown_connections 2024-04-24T08:18:14.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.208+0000 7f5c98a00640 1 -- 172.21.15.28:0/3230725264 wait complete. 2024-04-24T08:18:14.227 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:18:14.227 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.434+0000 7f7eedc00640 1 -- 172.21.15.28:0/2764592885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ee810d3c0 msgr2=0x7f7ee810d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.434+0000 7f7eedc00640 1 --2- 172.21.15.28:0/2764592885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ee810d3c0 0x7f7ee810d7a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009ca0 tx=0x7f7ed802a940 comp rx=0 tx=0).stop 2024-04-24T08:18:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.434+0000 7f7eedc00640 1 -- 172.21.15.28:0/2764592885 shutdown_connections 2024-04-24T08:18:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.434+0000 7f7eedc00640 1 --2- 172.21.15.28:0/2764592885 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ee8108e40 0x7f7ee81152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.434+0000 7f7eedc00640 1 --2- 172.21.15.28:0/2764592885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ee81084a0 0x7f7ee8108900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.434+0000 7f7eedc00640 1 --2- 172.21.15.28:0/2764592885 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ee810d3c0 0x7f7ee810d7a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.434+0000 7f7eedc00640 1 -- 172.21.15.28:0/2764592885 >> 172.21.15.28:0/2764592885 conn(0x7f7ee8077b80 msgr2=0x7f7ee8079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.434+0000 7f7eedc00640 1 -- 172.21.15.28:0/2764592885 shutdown_connections 2024-04-24T08:18:14.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.434+0000 7f7eedc00640 1 -- 172.21.15.28:0/2764592885 wait complete. 2024-04-24T08:18:14.439 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.435+0000 7f7eedc00640 1 Processor -- start 2024-04-24T08:18:14.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.435+0000 7f7eedc00640 1 -- start start 2024-04-24T08:18:14.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.436+0000 7f7eedc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ee81084a0 0x7f7ee81a4940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.436+0000 7f7eedc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ee8108e40 0x7f7ee81a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.436+0000 7f7eedc00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ee810d3c0 0x7f7ee819eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.436+0000 7f7eedc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee8118830 con 0x7f7ee81084a0 2024-04-24T08:18:14.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.436+0000 7f7eedc00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee81186b0 con 0x7f7ee810d3c0 2024-04-24T08:18:14.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.436+0000 7f7eedc00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7ee81189b0 con 0x7f7ee8108e40 2024-04-24T08:18:14.440 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.436+0000 7f7ee7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ee81084a0 0x7f7ee81a4940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.436+0000 7f7ee7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ee81084a0 0x7f7ee81a4940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40484/0 (socket says 172.21.15.28:40484) 2024-04-24T08:18:14.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.436+0000 7f7ee7e00640 1 -- 172.21.15.28:0/2784991869 learned_addr learned my addr 172.21.15.28:0/2784991869 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:14.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.436+0000 7f7ee7400640 1 --2- 172.21.15.28:0/2784991869 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ee8108e40 0x7f7ee81a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.436+0000 7f7eed200640 1 --2- 172.21.15.28:0/2784991869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ee810d3c0 0x7f7ee819eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.437+0000 7f7ee7400640 1 -- 172.21.15.28:0/2784991869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ee810d3c0 msgr2=0x7f7ee819eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.437+0000 7f7ee7400640 1 --2- 172.21.15.28:0/2784991869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ee810d3c0 0x7f7ee819eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.437+0000 7f7ee7400640 1 -- 172.21.15.28:0/2784991869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ee81084a0 msgr2=0x7f7ee81a4940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.437+0000 7f7ee7400640 1 --2- 172.21.15.28:0/2784991869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ee81084a0 0x7f7ee81a4940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.437+0000 7f7ee7400640 1 -- 172.21.15.28:0/2784991869 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7ee819f270 con 0x7f7ee8108e40 2024-04-24T08:18:14.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.437+0000 7f7ee7e00640 1 --2- 172.21.15.28:0/2784991869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ee81084a0 0x7f7ee81a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:14.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.437+0000 7f7ee7400640 1 --2- 172.21.15.28:0/2784991869 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ee8108e40 0x7f7ee81a4e80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7ed000e9f0 tx=0x7f7ed000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.437+0000 7f7ee4c00640 1 -- 172.21.15.28:0/2784991869 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed000ccc0 con 0x7f7ee8108e40 2024-04-24T08:18:14.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.437+0000 7f7eedc00640 1 -- 172.21.15.28:0/2784991869 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7ee819f560 con 0x7f7ee8108e40 2024-04-24T08:18:14.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.438+0000 7f7ee4c00640 1 -- 172.21.15.28:0/2784991869 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ed0010910 con 0x7f7ee8108e40 2024-04-24T08:18:14.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.438+0000 7f7ee4c00640 1 -- 172.21.15.28:0/2784991869 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ed0018960 con 0x7f7ee8108e40 2024-04-24T08:18:14.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.438+0000 7f7eedc00640 1 -- 172.21.15.28:0/2784991869 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7ee81ab800 con 0x7f7ee8108e40 2024-04-24T08:18:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.439+0000 7f7eedc00640 1 -- 172.21.15.28:0/2784991869 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7ee8069190 con 0x7f7ee8108e40 2024-04-24T08:18:14.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.441+0000 7f7ee4c00640 1 -- 172.21.15.28:0/2784991869 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7ed0010ac0 con 0x7f7ee8108e40 2024-04-24T08:18:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.443+0000 7f7ee4c00640 1 --2- 172.21.15.28:0/2784991869 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7ecc0a3d20 0x7f7ecc0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.443+0000 7f7ee4c00640 1 -- 172.21.15.28:0/2784991869 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7f7ed00d1120 con 0x7f7ee8108e40 2024-04-24T08:18:14.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.443+0000 7f7ee7e00640 1 --2- 172.21.15.28:0/2784991869 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7ecc0a3d20 0x7f7ecc0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.444+0000 7f7ee7e00640 1 --2- 172.21.15.28:0/2784991869 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7ecc0a3d20 0x7f7ecc0a61e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009a60 tx=0x7f7ed8005da0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.445+0000 7f7ee4c00640 1 -- 172.21.15.28:0/2784991869 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f7ed009d090 con 0x7f7ee8108e40 2024-04-24T08:18:14.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.692+0000 7f7eedc00640 1 -- 172.21.15.28:0/2784991869 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7ee8110960 con 0x7f7ee8108e40 2024-04-24T08:18:14.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.694+0000 7f7ee4c00640 1 -- 172.21.15.28:0/2784991869 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11871 (secure 0 0 0) 0x7f7ed00a1390 con 0x7f7ee8108e40 2024-04-24T08:18:14.696 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:18:14.696 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:14:38.663141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:18:14.696 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-24T08:18:14.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.696+0000 7f7eedc00640 1 -- 172.21.15.28:0/2784991869 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7ecc0a3d20 msgr2=0x7f7ecc0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.696+0000 7f7eedc00640 1 --2- 172.21.15.28:0/2784991869 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7ecc0a3d20 0x7f7ecc0a61e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f7ed8009a60 tx=0x7f7ed8005da0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.696+0000 7f7eedc00640 1 -- 172.21.15.28:0/2784991869 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ee8108e40 msgr2=0x7f7ee81a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.696+0000 7f7eedc00640 1 --2- 172.21.15.28:0/2784991869 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ee8108e40 0x7f7ee81a4e80 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7ed000e9f0 tx=0x7f7ed000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.696+0000 7f7eedc00640 1 -- 172.21.15.28:0/2784991869 shutdown_connections 2024-04-24T08:18:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.696+0000 7f7eedc00640 1 --2- 172.21.15.28:0/2784991869 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7ecc0a3d20 0x7f7ecc0a61e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.696+0000 7f7eedc00640 1 --2- 172.21.15.28:0/2784991869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ee810d3c0 0x7f7ee819eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.696+0000 7f7eedc00640 1 --2- 172.21.15.28:0/2784991869 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7ee8108e40 0x7f7ee81a4e80 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.696+0000 7f7eedc00640 1 --2- 172.21.15.28:0/2784991869 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7ee81084a0 0x7f7ee81a4940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.696+0000 7f7eedc00640 1 -- 172.21.15.28:0/2784991869 >> 172.21.15.28:0/2784991869 conn(0x7f7ee8077b80 msgr2=0x7f7ee8078490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.697+0000 7f7eedc00640 1 -- 172.21.15.28:0/2784991869 shutdown_connections 2024-04-24T08:18:14.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.697+0000 7f7eedc00640 1 -- 172.21.15.28:0/2784991869 wait complete. 2024-04-24T08:18:14.718 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:18:14.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.935+0000 7f9867200640 1 -- 172.21.15.28:0/767136014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9868101fc0 msgr2=0x7f98681023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.935+0000 7f9867200640 1 --2- 172.21.15.28:0/767136014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9868101fc0 0x7f98681023a0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f9858009900 tx=0x7f985802a410 comp rx=0 tx=0).stop 2024-04-24T08:18:14.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.935+0000 7f9867200640 1 -- 172.21.15.28:0/767136014 shutdown_connections 2024-04-24T08:18:14.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.935+0000 7f9867200640 1 --2- 172.21.15.28:0/767136014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f986810b430 0x7f986810d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.935+0000 7f9867200640 1 --2- 172.21.15.28:0/767136014 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f98681028e0 0x7f986810adc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.935+0000 7f9867200640 1 --2- 172.21.15.28:0/767136014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9868101fc0 0x7f98681023a0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.935+0000 7f9867200640 1 -- 172.21.15.28:0/767136014 >> 172.21.15.28:0/767136014 conn(0x7f9868075d00 msgr2=0x7f9868076110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.935+0000 7f9867200640 1 -- 172.21.15.28:0/767136014 shutdown_connections 2024-04-24T08:18:14.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.935+0000 7f9867200640 1 -- 172.21.15.28:0/767136014 wait complete. 2024-04-24T08:18:14.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.936+0000 7f9867200640 1 Processor -- start 2024-04-24T08:18:14.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.936+0000 7f9867200640 1 -- start start 2024-04-24T08:18:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.937+0000 7f9867200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9868101fc0 0x7f9868198760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.937+0000 7f9867200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f98681028e0 0x7f9868198ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.937+0000 7f9867200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f986810b430 0x7f986819d020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.937+0000 7f9867200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98681111e0 con 0x7f9868101fc0 2024-04-24T08:18:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.937+0000 7f9867200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9868111060 con 0x7f98681028e0 2024-04-24T08:18:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.937+0000 7f9867200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9868111360 con 0x7f986810b430 2024-04-24T08:18:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.937+0000 7f9865e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9868101fc0 0x7f9868198760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.937+0000 7f9866800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f986810b430 0x7f986819d020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.937+0000 7f9865e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9868101fc0 0x7f9868198760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40508/0 (socket says 172.21.15.28:40508) 2024-04-24T08:18:14.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.937+0000 7f9865e00640 1 -- 172.21.15.28:0/1533786459 learned_addr learned my addr 172.21.15.28:0/1533786459 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:14.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.937+0000 7f9865400640 1 --2- 172.21.15.28:0/1533786459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f98681028e0 0x7f9868198ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.938+0000 7f9866800640 1 -- 172.21.15.28:0/1533786459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f98681028e0 msgr2=0x7f9868198ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.938+0000 7f9866800640 1 --2- 172.21.15.28:0/1533786459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f98681028e0 0x7f9868198ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.938+0000 7f9866800640 1 -- 172.21.15.28:0/1533786459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9868101fc0 msgr2=0x7f9868198760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.938+0000 7f9866800640 1 --2- 172.21.15.28:0/1533786459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9868101fc0 0x7f9868198760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.938+0000 7f9866800640 1 -- 172.21.15.28:0/1533786459 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f986819d750 con 0x7f986810b430 2024-04-24T08:18:14.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.938+0000 7f9865e00640 1 --2- 172.21.15.28:0/1533786459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9868101fc0 0x7f9868198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:14.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.938+0000 7f9866800640 1 --2- 172.21.15.28:0/1533786459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f986810b430 0x7f986819d020 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9860005fd0 tx=0x7f986000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.938+0000 7f985ea00640 1 -- 172.21.15.28:0/1533786459 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f986000cdf0 con 0x7f986810b430 2024-04-24T08:18:14.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.939+0000 7f985ea00640 1 -- 172.21.15.28:0/1533786459 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9860005020 con 0x7f986810b430 2024-04-24T08:18:14.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.939+0000 7f9867200640 1 -- 172.21.15.28:0/1533786459 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f986819da40 con 0x7f986810b430 2024-04-24T08:18:14.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.939+0000 7f985ea00640 1 -- 172.21.15.28:0/1533786459 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98600106d0 con 0x7f986810b430 2024-04-24T08:18:14.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.939+0000 7f9867200640 1 -- 172.21.15.28:0/1533786459 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98681a52d0 con 0x7f986810b430 2024-04-24T08:18:14.944 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.943+0000 7f986fe1a740 1 Processor -- start 2024-04-24T08:18:14.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.944+0000 7f985ea00640 1 -- 172.21.15.28:0/1533786459 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f98600054d0 con 0x7f986810b430 2024-04-24T08:18:14.945 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.944+0000 7f986fe1a740 1 -- start start 2024-04-24T08:18:14.946 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.944+0000 7f986fe1a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f35b40 0x558b17f35f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.944+0000 7f986fe1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 0x558b17f405f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.944+0000 7f986fe1a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f40b30 0x558b17f42f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.944+0000 7f986fe1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558b17e9b1d0 con 0x558b17f36460 2024-04-24T08:18:14.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.944+0000 7f986fe1a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558b17e41450 con 0x558b17f35b40 2024-04-24T08:18:14.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.944+0000 7f986fe1a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558b17e97190 con 0x558b17f40b30 2024-04-24T08:18:14.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.944+0000 7f9865e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 0x558b17f405f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.945+0000 7f9865e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 0x558b17f405f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40520/0 (socket says 172.21.15.28:40520) 2024-04-24T08:18:14.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.945+0000 7f9865e00640 1 -- 172.21.15.28:0/1273082207 learned_addr learned my addr 172.21.15.28:0/1273082207 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:14.947 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.945+0000 7f9865400640 1 --2- 172.21.15.28:0/1273082207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f35b40 0x558b17f35f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.945+0000 7f9865400640 1 --2- 172.21.15.28:0/1273082207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f40b30 0x558b17f42f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.945+0000 7f9865e00640 1 -- 172.21.15.28:0/1273082207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f40b30 msgr2=0x558b17f42f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.945+0000 7f9865e00640 1 --2- 172.21.15.28:0/1273082207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f40b30 0x558b17f42f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.945+0000 7f9865e00640 1 -- 172.21.15.28:0/1273082207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f35b40 msgr2=0x558b17f35f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.945+0000 7f9865e00640 1 --2- 172.21.15.28:0/1273082207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f35b40 0x558b17f35f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.945+0000 7f9865e00640 1 -- 172.21.15.28:0/1273082207 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b17efe310 con 0x558b17f36460 2024-04-24T08:18:14.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.945+0000 7f9865400640 1 --2- 172.21.15.28:0/1273082207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f40b30 0x558b17f42f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:14.948 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.945+0000 7f9865e00640 1 --2- 172.21.15.28:0/1273082207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 0x558b17f405f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9858005c40 tx=0x7f9858005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe26d5d3d6a9221d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.946+0000 7f985ea00640 1 --2- 172.21.15.28:0/1533786459 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f984c0a3d20 0x7f984c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.946+0000 7f9857400640 1 -- 172.21.15.28:0/1273082207 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9858040070 con 0x558b17f36460 2024-04-24T08:18:14.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.946+0000 7f9857400640 1 -- 172.21.15.28:0/1273082207 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98580047a0 con 0x558b17f36460 2024-04-24T08:18:14.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.946+0000 7f9857400640 1 -- 172.21.15.28:0/1273082207 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9858035040 con 0x558b17f36460 2024-04-24T08:18:14.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.946+0000 7f985ea00640 1 -- 172.21.15.28:0/1533786459 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7f98600d0f90 con 0x7f986810b430 2024-04-24T08:18:14.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.946+0000 7f9865400640 1 --2- 172.21.15.28:0/1533786459 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f984c0a3d20 0x7f984c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.947+0000 7f986fe1a740 1 -- 172.21.15.28:0/1273082207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 msgr2=0x558b17f405f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.947+0000 7f986fe1a740 1 --2- 172.21.15.28:0/1273082207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 0x558b17f405f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f9858005c40 tx=0x7f9858005dd0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.947+0000 7f986fe1a740 1 -- 172.21.15.28:0/1273082207 shutdown_connections 2024-04-24T08:18:14.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.947+0000 7f986fe1a740 1 --2- 172.21.15.28:0/1273082207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 0x558b17f405f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.947+0000 7f9865400640 1 --2- 172.21.15.28:0/1533786459 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f984c0a3d20 0x7f984c0a61e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x558b17f32ff0 tx=0x7f98500096c0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.947+0000 7f986fe1a740 1 --2- 172.21.15.28:0/1273082207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f40b30 0x558b17f42f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.947+0000 7f986fe1a740 1 --2- 172.21.15.28:0/1273082207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f35b40 0x558b17f35f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.947+0000 7f986fe1a740 1 -- 172.21.15.28:0/1273082207 >> 172.21.15.28:0/1273082207 conn(0x558b17f31e60 msgr2=0x558b17f32240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.947+0000 7f986fe1a740 1 -- 172.21.15.28:0/1273082207 shutdown_connections 2024-04-24T08:18:14.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.947+0000 7f986fe1a740 1 -- 172.21.15.28:0/1273082207 wait complete. 2024-04-24T08:18:14.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.947+0000 7f986fe1a740 1 -- start start 2024-04-24T08:18:14.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.948+0000 7f986fe1a740 1 Processor -- start 2024-04-24T08:18:14.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.948+0000 7f986fe1a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f35b40 0x558b17f405c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.948+0000 7f986fe1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 0x558b17f3d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.948+0000 7f986fe1a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f40b30 0x558b17f3dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.948+0000 7f986fe1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558b17f43960 con 0x558b17f36460 2024-04-24T08:18:14.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.948+0000 7f986fe1a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558b17e54810 con 0x558b17f40b30 2024-04-24T08:18:14.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.948+0000 7f986fe1a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558b17f43ae0 con 0x558b17f35b40 2024-04-24T08:18:14.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.948+0000 7f986fe1a740 10 client.0 _ll_register_callbacks cb 0x558b17e22140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:14.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.948+0000 7f9865400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 0x558b17f3d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.955 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.948+0000 7f9865e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f35b40 0x558b17f405c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.948+0000 7f9865400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 0x558b17f3d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40524/0 (socket says 172.21.15.28:40524) 2024-04-24T08:18:14.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.948+0000 7f9865400640 1 -- 172.21.15.28:0/2912938917 learned_addr learned my addr 172.21.15.28:0/2912938917 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:14.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.948+0000 7f9866800640 1 --2- 172.21.15.28:0/2912938917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f40b30 0x558b17f3dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.949+0000 7f9865e00640 1 -- 172.21.15.28:0/2912938917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f40b30 msgr2=0x558b17f3dfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.949+0000 7f9865e00640 1 --2- 172.21.15.28:0/2912938917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f40b30 0x558b17f3dfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.949+0000 7f9865e00640 1 -- 172.21.15.28:0/2912938917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 msgr2=0x558b17f3d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.949+0000 7f9865e00640 1 --2- 172.21.15.28:0/2912938917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 0x558b17f3d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.949+0000 7f9865e00640 1 -- 172.21.15.28:0/2912938917 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b17e653f0 con 0x558b17f35b40 2024-04-24T08:18:14.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.949+0000 7f9866800640 1 --2- 172.21.15.28:0/2912938917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f40b30 0x558b17f3dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:14.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.949+0000 7f9865400640 1 --2- 172.21.15.28:0/2912938917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 0x558b17f3d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:14.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.949+0000 7f9865e00640 1 --2- 172.21.15.28:0/2912938917 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f35b40 0x558b17f405c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9858031550 tx=0x7f9858009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=653163ea06694faf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.949+0000 7f984be00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:18:14.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.949+0000 7f984be00640 1 -- 172.21.15.28:0/2912938917 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985804c020 con 0x558b17f35b40 2024-04-24T08:18:14.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.949+0000 7f984be00640 1 -- 172.21.15.28:0/2912938917 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7f98580053b0 con 0x558b17f35b40 2024-04-24T08:18:14.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.949+0000 7f986fe1a740 1 -- 172.21.15.28:0/2912938917 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f98580047a0 con 0x558b17f35b40 2024-04-24T08:18:14.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.950+0000 7f984be00640 1 -- 172.21.15.28:0/2912938917 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9858057050 con 0x558b17f35b40 2024-04-24T08:18:14.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.950+0000 7f986fe1a740 10 client.26557 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:14.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.950+0000 7f986fe1a740 1 -- 172.21.15.28:0/2912938917 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558b17bad3f0 con 0x558b17f35b40 2024-04-24T08:18:14.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.950+0000 7f984be00640 1 -- 172.21.15.28:0/2912938917 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98580047a0 con 0x558b17f35b40 2024-04-24T08:18:14.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.950+0000 7f984be00640 1 -- 172.21.15.28:0/2912938917 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f98580315b0 con 0x558b17f35b40 2024-04-24T08:18:14.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.951+0000 7f986fe1a740 10 client.26557 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:14.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.951+0000 7f986fe1a740 10 client.26557 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:18:14.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.951+0000 7f986fe1a740 20 client.26557 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:14.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.951+0000 7f986fe1a740 1 --2- 172.21.15.28:0/2912938917 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558b17f50b70 0x558b17f52f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.951+0000 7f986fe1a740 4 client.26557 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:18:14.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.951+0000 7f986fe1a740 1 -- 172.21.15.28:0/2912938917 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b17ea8bf0 con 0x558b17f50b70 2024-04-24T08:18:14.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.951+0000 7f9865400640 1 --2- 172.21.15.28:0/2912938917 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558b17f50b70 0x558b17f52f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.952+0000 7f984be00640 10 client.26557 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:18:14.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.952+0000 7f9865400640 1 --2- 172.21.15.28:0/2912938917 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558b17f50b70 0x558b17f52f60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1de0f0d2be41f64b server_cookie=80a18da85a323a0c in_seq=0 out_seq=0 2024-04-24T08:18:14.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.953+0000 7f984be00640 1 -- 172.21.15.28:0/2912938917 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b17ea8bf0 con 0x558b17f50b70 2024-04-24T08:18:14.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.953+0000 7f984be00640 10 client.26557 handle_command_reply: tid=0 2024-04-24T08:18:14.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.953+0000 7f986fe1a740 1 client.26557 shutdown 2024-04-24T08:18:14.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.953+0000 7f986fe1a740 1 -- 172.21.15.28:0/2912938917 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f35b40 msgr2=0x558b17f405c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.953+0000 7f986fe1a740 1 --2- 172.21.15.28:0/2912938917 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f35b40 0x558b17f405c0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f9858031550 tx=0x7f9858009260 comp rx=0 tx=0).stop 2024-04-24T08:18:14.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.953+0000 7f986fe1a740 1 -- 172.21.15.28:0/2912938917 shutdown_connections 2024-04-24T08:18:14.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.953+0000 7f986fe1a740 1 --2- 172.21.15.28:0/2912938917 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558b17f50b70 0x558b17f52f60 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.953+0000 7f986fe1a740 1 --2- 172.21.15.28:0/2912938917 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 0x558b17f3d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.953+0000 7f986fe1a740 1 --2- 172.21.15.28:0/2912938917 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f40b30 0x558b17f3dfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.953+0000 7f986fe1a740 1 --2- 172.21.15.28:0/2912938917 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f35b40 0x558b17f405c0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.953+0000 7f984be00640 0 client.26557 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:18:14.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.953+0000 7f986fe1a740 1 -- 172.21.15.28:0/2912938917 >> 172.21.15.28:0/2912938917 conn(0x558b17f31e60 msgr2=0x558b17f37b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.953+0000 7f986fe1a740 1 -- 172.21.15.28:0/2912938917 shutdown_connections 2024-04-24T08:18:14.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.953+0000 7f986fe1a740 1 -- 172.21.15.28:0/2912938917 wait complete. 2024-04-24T08:18:14.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.953+0000 7f986fe1a740 20 client.26557 trim_cache size 0 max 16384 2024-04-24T08:18:14.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.968+0000 7f986fe1a740 1 Processor -- start 2024-04-24T08:18:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.968+0000 7f986fe1a740 1 -- start start 2024-04-24T08:18:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.968+0000 7f986fe1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f35b40 0x558b17f4c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.969+0000 7f986fe1a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f36460 0x558b17f52f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.969+0000 7f986fe1a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f40b30 0x558b17f49110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.969+0000 7f986fe1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98681b04f0 con 0x558b17f35b40 2024-04-24T08:18:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.969+0000 7f986fe1a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f98681addc0 con 0x558b17f36460 2024-04-24T08:18:14.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.969+0000 7f986fe1a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f98681abc30 con 0x558b17f40b30 2024-04-24T08:18:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.969+0000 7f9865400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f36460 0x558b17f52f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.969+0000 7f9866800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f40b30 0x558b17f49110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.969+0000 7f9865400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f36460 0x558b17f52f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40066/0 (socket says 172.21.15.28:40066) 2024-04-24T08:18:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.969+0000 7f9865400640 1 -- 172.21.15.28:0/1390324266 learned_addr learned my addr 172.21.15.28:0/1390324266 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.969+0000 7f9865e00640 1 --2- 172.21.15.28:0/1390324266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f35b40 0x558b17f4c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.970+0000 7f9866800640 1 -- 172.21.15.28:0/1390324266 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f36460 msgr2=0x558b17f52f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.970+0000 7f9866800640 1 --2- 172.21.15.28:0/1390324266 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f36460 0x558b17f52f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.970+0000 7f9866800640 1 -- 172.21.15.28:0/1390324266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f35b40 msgr2=0x558b17f4c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.970+0000 7f9866800640 1 --2- 172.21.15.28:0/1390324266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f35b40 0x558b17f4c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.970+0000 7f9866800640 1 -- 172.21.15.28:0/1390324266 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b17c777d0 con 0x558b17f40b30 2024-04-24T08:18:14.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.970+0000 7f9866800640 1 --2- 172.21.15.28:0/1390324266 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f40b30 0x558b17f49110 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f98600ce600 tx=0x7f986009cf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb083bcca6feb1d5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.970+0000 7f9856a00640 1 -- 172.21.15.28:0/1390324266 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98600bb070 con 0x558b17f40b30 2024-04-24T08:18:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.970+0000 7f9856a00640 1 -- 172.21.15.28:0/1390324266 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986009f040 con 0x558b17f40b30 2024-04-24T08:18:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.970+0000 7f9856a00640 1 -- 172.21.15.28:0/1390324266 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98600a5890 con 0x558b17f40b30 2024-04-24T08:18:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.971+0000 7f986fe1a740 1 -- 172.21.15.28:0/1390324266 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f40b30 msgr2=0x558b17f49110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.971+0000 7f986fe1a740 1 --2- 172.21.15.28:0/1390324266 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f40b30 0x558b17f49110 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f98600ce600 tx=0x7f986009cf10 comp rx=0 tx=0).stop 2024-04-24T08:18:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.971+0000 7f986fe1a740 1 -- 172.21.15.28:0/1390324266 shutdown_connections 2024-04-24T08:18:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.971+0000 7f986fe1a740 1 --2- 172.21.15.28:0/1390324266 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f40b30 0x558b17f49110 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.971+0000 7f986fe1a740 1 --2- 172.21.15.28:0/1390324266 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f36460 0x558b17f52f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.971+0000 7f986fe1a740 1 --2- 172.21.15.28:0/1390324266 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f35b40 0x558b17f4c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.971+0000 7f986fe1a740 1 -- 172.21.15.28:0/1390324266 >> 172.21.15.28:0/1390324266 conn(0x558b17f31e60 msgr2=0x558b17f3d140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.971+0000 7f986fe1a740 1 -- 172.21.15.28:0/1390324266 shutdown_connections 2024-04-24T08:18:14.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.971+0000 7f986fe1a740 1 -- 172.21.15.28:0/1390324266 wait complete. 2024-04-24T08:18:14.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.971+0000 7f986fe1a740 1 -- start start 2024-04-24T08:18:14.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.972+0000 7f986fe1a740 1 Processor -- start 2024-04-24T08:18:14.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.972+0000 7f986fe1a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f35b40 0x558b17fb3220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.972+0000 7f986fe1a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 0x558b17fb3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.972+0000 7f986fe1a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f40b30 0x558b17f57c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.972+0000 7f986fe1a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f98681ac280 con 0x558b17f36460 2024-04-24T08:18:14.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.972+0000 7f986fe1a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f98681ad6f0 con 0x558b17f35b40 2024-04-24T08:18:14.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.972+0000 7f986fe1a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f98681ac950 con 0x558b17f40b30 2024-04-24T08:18:14.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.972+0000 7f986fe1a740 10 client.0 _ll_register_callbacks cb 0x7f9850009a10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:14.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.972+0000 7f9865400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 0x558b17fb3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.972+0000 7f9865e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f35b40 0x558b17fb3220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.972+0000 7f9865e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f35b40 0x558b17fb3220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40082/0 (socket says 172.21.15.28:40082) 2024-04-24T08:18:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.972+0000 7f9865e00640 1 -- 172.21.15.28:0/929583723 learned_addr learned my addr 172.21.15.28:0/929583723 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.972+0000 7f9866800640 1 --2- 172.21.15.28:0/929583723 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f40b30 0x558b17f57c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.973+0000 7f9865e00640 1 -- 172.21.15.28:0/929583723 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f40b30 msgr2=0x558b17f57c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.973+0000 7f9865e00640 1 --2- 172.21.15.28:0/929583723 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f40b30 0x558b17f57c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.973+0000 7f9865e00640 1 -- 172.21.15.28:0/929583723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 msgr2=0x558b17fb3760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.973+0000 7f9865e00640 1 --2- 172.21.15.28:0/929583723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 0x558b17fb3760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.973+0000 7f9865e00640 1 -- 172.21.15.28:0/929583723 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b17f58360 con 0x558b17f35b40 2024-04-24T08:18:14.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.973+0000 7f9865e00640 1 --2- 172.21.15.28:0/929583723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f35b40 0x558b17fb3220 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9858031980 tx=0x7f98580317c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a960aee16f72ac9d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:14.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.973+0000 7f984be00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:18:14.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.973+0000 7f984be00640 1 -- 172.21.15.28:0/929583723 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f985804c040 con 0x558b17f35b40 2024-04-24T08:18:14.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.973+0000 7f986fe1a740 1 -- 172.21.15.28:0/929583723 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f986009f040 con 0x558b17f35b40 2024-04-24T08:18:14.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.973+0000 7f984be00640 1 -- 172.21.15.28:0/929583723 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7f98580051b0 con 0x558b17f35b40 2024-04-24T08:18:14.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.974+0000 7f984be00640 1 -- 172.21.15.28:0/929583723 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9858040070 con 0x558b17f35b40 2024-04-24T08:18:14.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.974+0000 7f986fe1a740 10 client.26543 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:14.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.974+0000 7f986fe1a740 1 -- 172.21.15.28:0/929583723 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558b17fbaa10 con 0x558b17f35b40 2024-04-24T08:18:14.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.974+0000 7f984be00640 1 -- 172.21.15.28:0/929583723 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f986009f040 con 0x558b17f35b40 2024-04-24T08:18:14.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.974+0000 7f984be00640 1 -- 172.21.15.28:0/929583723 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9858062070 con 0x558b17f35b40 2024-04-24T08:18:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.974+0000 7f986fe1a740 10 client.26543 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.974+0000 7f986fe1a740 10 client.26543 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:18:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.974+0000 7f986fe1a740 20 client.26543 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.974+0000 7f986fe1a740 1 --2- 172.21.15.28:0/929583723 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558b17fbb0e0 0x558b17fbd500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.974+0000 7f986fe1a740 4 client.26543 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:18:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.974+0000 7f986fe1a740 1 -- 172.21.15.28:0/929583723 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558b17ea2a60 con 0x558b17fbb0e0 2024-04-24T08:18:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.975+0000 7f9865400640 1 --2- 172.21.15.28:0/929583723 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558b17fbb0e0 0x558b17fbd500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:14.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.975+0000 7f9865400640 1 --2- 172.21.15.28:0/929583723 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558b17fbb0e0 0x558b17fbd500 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=982441b5881d4691 server_cookie=caea8f462f219cdd in_seq=0 out_seq=0 2024-04-24T08:18:14.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.975+0000 7f984be00640 10 client.26543 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:18:14.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.976+0000 7f984be00640 1 -- 172.21.15.28:0/929583723 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558b17ea2a60 con 0x558b17fbb0e0 2024-04-24T08:18:14.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.976+0000 7f984be00640 10 client.26543 handle_command_reply: tid=0 2024-04-24T08:18:14.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.976+0000 7f986fe1a740 1 client.26543 shutdown 2024-04-24T08:18:14.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.976+0000 7f986fe1a740 1 -- 172.21.15.28:0/929583723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f35b40 msgr2=0x558b17fb3220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.976+0000 7f986fe1a740 1 --2- 172.21.15.28:0/929583723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f35b40 0x558b17fb3220 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f9858031980 tx=0x7f98580317c0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.976+0000 7f986fe1a740 1 -- 172.21.15.28:0/929583723 shutdown_connections 2024-04-24T08:18:14.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.976+0000 7f986fe1a740 1 --2- 172.21.15.28:0/929583723 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x558b17fbb0e0 0x558b17fbd500 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.976+0000 7f986fe1a740 1 --2- 172.21.15.28:0/929583723 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558b17f36460 0x558b17fb3760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.976+0000 7f986fe1a740 1 --2- 172.21.15.28:0/929583723 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558b17f40b30 0x558b17f57c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.976+0000 7f986fe1a740 1 --2- 172.21.15.28:0/929583723 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558b17f35b40 0x558b17fb3220 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.976+0000 7f984be00640 0 client.26543 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:18:14.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.976+0000 7f986fe1a740 1 -- 172.21.15.28:0/929583723 >> 172.21.15.28:0/929583723 conn(0x558b17f31e60 msgr2=0x558b17f32240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.976+0000 7f986fe1a740 1 -- 172.21.15.28:0/929583723 shutdown_connections 2024-04-24T08:18:14.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.976+0000 7f986fe1a740 1 -- 172.21.15.28:0/929583723 wait complete. 2024-04-24T08:18:14.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.976+0000 7f986fe1a740 20 client.26543 trim_cache size 0 max 16384 2024-04-24T08:18:14.984 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:18:14.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.977+0000 7f9867200640 1 -- 172.21.15.28:0/1533786459 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f984c0a3d20 msgr2=0x7f984c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.977+0000 7f9867200640 1 --2- 172.21.15.28:0/1533786459 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f984c0a3d20 0x7f984c0a61e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x558b17f32ff0 tx=0x7f98500096c0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.977+0000 7f9867200640 1 -- 172.21.15.28:0/1533786459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f986810b430 msgr2=0x7f986819d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:14.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.977+0000 7f9867200640 1 --2- 172.21.15.28:0/1533786459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f986810b430 0x7f986819d020 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f9860005fd0 tx=0x7f986000ef10 comp rx=0 tx=0).stop 2024-04-24T08:18:14.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.977+0000 7f9867200640 1 -- 172.21.15.28:0/1533786459 shutdown_connections 2024-04-24T08:18:14.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.977+0000 7f9867200640 1 --2- 172.21.15.28:0/1533786459 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f984c0a3d20 0x7f984c0a61e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.977+0000 7f9867200640 1 --2- 172.21.15.28:0/1533786459 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f986810b430 0x7f986819d020 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.977+0000 7f9867200640 1 --2- 172.21.15.28:0/1533786459 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f98681028e0 0x7f9868198ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.977+0000 7f9867200640 1 --2- 172.21.15.28:0/1533786459 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9868101fc0 0x7f9868198760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:14.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.977+0000 7f9867200640 1 -- 172.21.15.28:0/1533786459 >> 172.21.15.28:0/1533786459 conn(0x7f9868075d00 msgr2=0x7f9868100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:14.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.977+0000 7f9867200640 1 -- 172.21.15.28:0/1533786459 shutdown_connections 2024-04-24T08:18:14.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:14.977+0000 7f9867200640 1 -- 172.21.15.28:0/1533786459 wait complete. 2024-04-24T08:18:14.998 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:18:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.219+0000 7f344be00640 1 -- 172.21.15.28:0/3233320615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f344c104cd0 msgr2=0x7f344c108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.219+0000 7f344be00640 1 --2- 172.21.15.28:0/3233320615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f344c104cd0 0x7f344c108d10 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f3434009900 tx=0x7f343402a410 comp rx=0 tx=0).stop 2024-04-24T08:18:15.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.219+0000 7f344be00640 1 -- 172.21.15.28:0/3233320615 shutdown_connections 2024-04-24T08:18:15.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.219+0000 7f344be00640 1 --2- 172.21.15.28:0/3233320615 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f344c104cd0 0x7f344c108d10 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.219+0000 7f344be00640 1 --2- 172.21.15.28:0/3233320615 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f344c1093f0 0x7f344c110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.219+0000 7f344be00640 1 --2- 172.21.15.28:0/3233320615 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f344c104320 0x7f344c104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.219+0000 7f344be00640 1 -- 172.21.15.28:0/3233320615 >> 172.21.15.28:0/3233320615 conn(0x7f344c0ffb40 msgr2=0x7f344c101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.219+0000 7f344be00640 1 -- 172.21.15.28:0/3233320615 shutdown_connections 2024-04-24T08:18:15.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.220+0000 7f344be00640 1 -- 172.21.15.28:0/3233320615 wait complete. 2024-04-24T08:18:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.221+0000 7f344be00640 1 Processor -- start 2024-04-24T08:18:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.221+0000 7f344be00640 1 -- start start 2024-04-24T08:18:15.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.221+0000 7f344be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f344c104320 0x7f344c1a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.221+0000 7f344be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f344c1093f0 0x7f344c1a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.221+0000 7f344be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f344c19a820 0x7f344c19acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.221+0000 7f344be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f344c114480 con 0x7f344c1093f0 2024-04-24T08:18:15.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.221+0000 7f344be00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f344c114300 con 0x7f344c19a820 2024-04-24T08:18:15.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.221+0000 7f344be00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f344c114600 con 0x7f344c104320 2024-04-24T08:18:15.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.222+0000 7f3442000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f344c1093f0 0x7f344c1a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.222+0000 7f3442000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f344c1093f0 0x7f344c1a0bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40546/0 (socket says 172.21.15.28:40546) 2024-04-24T08:18:15.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.222+0000 7f3442000640 1 -- 172.21.15.28:0/2365987836 learned_addr learned my addr 172.21.15.28:0/2365987836 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:15.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.222+0000 7f344b400640 1 --2- 172.21.15.28:0/2365987836 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f344c19a820 0x7f344c19acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.222+0000 7f344aa00640 1 --2- 172.21.15.28:0/2365987836 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f344c104320 0x7f344c1a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.222+0000 7f3442000640 1 -- 172.21.15.28:0/2365987836 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f344c104320 msgr2=0x7f344c1a06b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.222+0000 7f3442000640 1 --2- 172.21.15.28:0/2365987836 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f344c104320 0x7f344c1a06b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.222+0000 7f3442000640 1 -- 172.21.15.28:0/2365987836 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f344c19a820 msgr2=0x7f344c19acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.222+0000 7f3442000640 1 --2- 172.21.15.28:0/2365987836 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f344c19a820 0x7f344c19acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.222+0000 7f3442000640 1 -- 172.21.15.28:0/2365987836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f344c19b3b0 con 0x7f344c1093f0 2024-04-24T08:18:15.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.222+0000 7f344b400640 1 --2- 172.21.15.28:0/2365987836 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f344c19a820 0x7f344c19acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:15.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.222+0000 7f3442000640 1 --2- 172.21.15.28:0/2365987836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f344c1093f0 0x7f344c1a0bf0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3434005940 tx=0x7f34340040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.223+0000 7f3443e00640 1 -- 172.21.15.28:0/2365987836 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343403f030 con 0x7f344c1093f0 2024-04-24T08:18:15.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.223+0000 7f3443e00640 1 -- 172.21.15.28:0/2365987836 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34340044c0 con 0x7f344c1093f0 2024-04-24T08:18:15.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.223+0000 7f3443e00640 1 -- 172.21.15.28:0/2365987836 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434035070 con 0x7f344c1093f0 2024-04-24T08:18:15.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.223+0000 7f344be00640 1 -- 172.21.15.28:0/2365987836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f344c19b640 con 0x7f344c1093f0 2024-04-24T08:18:15.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.223+0000 7f344be00640 1 -- 172.21.15.28:0/2365987836 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f344c1a77f0 con 0x7f344c1093f0 2024-04-24T08:18:15.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.226+0000 7f3454a12740 1 Processor -- start 2024-04-24T08:18:15.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.226+0000 7f3443e00640 1 -- 172.21.15.28:0/2365987836 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3434042030 con 0x7f344c1093f0 2024-04-24T08:18:15.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.226+0000 7f3454a12740 1 -- start start 2024-04-24T08:18:15.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.227+0000 7f3454a12740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7e3b40 0x55f50b7e3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.227+0000 7f3454a12740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e4460 0x55f50b7ee5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.227+0000 7f3454a12740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 0x55f50b7f0f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.227+0000 7f3454a12740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f50b7491d0 con 0x55f50b7eeb30 2024-04-24T08:18:15.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.227+0000 7f3454a12740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f50b6ef450 con 0x55f50b7e3b40 2024-04-24T08:18:15.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.227+0000 7f3454a12740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f50b745190 con 0x55f50b7e4460 2024-04-24T08:18:15.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.227+0000 7f3442000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 0x55f50b7f0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.227+0000 7f344aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e4460 0x55f50b7ee5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.227+0000 7f344aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e4460 0x55f50b7ee5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:50644/0 (socket says 172.21.15.28:50644) 2024-04-24T08:18:15.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.227+0000 7f344aa00640 1 -- 172.21.15.28:0/130861093 learned_addr learned my addr 172.21.15.28:0/130861093 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:15.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.227+0000 7f344b400640 1 --2- 172.21.15.28:0/130861093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7e3b40 0x55f50b7e3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.228+0000 7f344aa00640 1 -- 172.21.15.28:0/130861093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7e3b40 msgr2=0x55f50b7e3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.228+0000 7f344aa00640 1 --2- 172.21.15.28:0/130861093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7e3b40 0x55f50b7e3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.228+0000 7f344aa00640 1 -- 172.21.15.28:0/130861093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 msgr2=0x55f50b7f0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.228+0000 7f344aa00640 1 --2- 172.21.15.28:0/130861093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 0x55f50b7f0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.228+0000 7f344aa00640 1 -- 172.21.15.28:0/130861093 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f50b7ac310 con 0x55f50b7e4460 2024-04-24T08:18:15.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.228+0000 7f3442000640 1 --2- 172.21.15.28:0/130861093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 0x55f50b7f0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:15.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.228+0000 7f3443e00640 1 --2- 172.21.15.28:0/2365987836 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f34300a3d20 0x7f34300a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.228+0000 7f344aa00640 1 --2- 172.21.15.28:0/130861093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e4460 0x55f50b7ee5f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f343c009d20 tx=0x7f343c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=275c0fe3fcbeef94 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.228+0000 7f3443e00640 1 -- 172.21.15.28:0/2365987836 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7f34340f0c50 con 0x7f344c1093f0 2024-04-24T08:18:15.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.229+0000 7f344b400640 1 --2- 172.21.15.28:0/2365987836 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f34300a3d20 0x7f34300a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.229+0000 7f343be00640 1 -- 172.21.15.28:0/130861093 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343c004220 con 0x55f50b7e4460 2024-04-24T08:18:15.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.229+0000 7f343be00640 1 -- 172.21.15.28:0/130861093 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f343c0043c0 con 0x55f50b7e4460 2024-04-24T08:18:15.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.229+0000 7f343be00640 1 -- 172.21.15.28:0/130861093 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f343c019930 con 0x55f50b7e4460 2024-04-24T08:18:15.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.230+0000 7f3454a12740 1 -- 172.21.15.28:0/130861093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e4460 msgr2=0x55f50b7ee5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.230+0000 7f344b400640 1 --2- 172.21.15.28:0/2365987836 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f34300a3d20 0x7f34300a61e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3444009fd0 tx=0x7f3444009500 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.230+0000 7f3454a12740 1 --2- 172.21.15.28:0/130861093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e4460 0x55f50b7ee5f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f343c009d20 tx=0x7f343c011810 comp rx=0 tx=0).stop 2024-04-24T08:18:15.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.230+0000 7f3454a12740 1 -- 172.21.15.28:0/130861093 shutdown_connections 2024-04-24T08:18:15.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.230+0000 7f3454a12740 1 --2- 172.21.15.28:0/130861093 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 0x55f50b7f0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.230+0000 7f3454a12740 1 --2- 172.21.15.28:0/130861093 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e4460 0x55f50b7ee5f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.230+0000 7f3454a12740 1 --2- 172.21.15.28:0/130861093 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7e3b40 0x55f50b7e3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.230+0000 7f3454a12740 1 -- 172.21.15.28:0/130861093 >> 172.21.15.28:0/130861093 conn(0x55f50b7dfe60 msgr2=0x55f50b7e0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.230+0000 7f3454a12740 1 -- 172.21.15.28:0/130861093 shutdown_connections 2024-04-24T08:18:15.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.230+0000 7f3454a12740 1 -- 172.21.15.28:0/130861093 wait complete. 2024-04-24T08:18:15.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.230+0000 7f3454a12740 1 -- start start 2024-04-24T08:18:15.234 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.231+0000 7f3454a12740 1 Processor -- start 2024-04-24T08:18:15.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.231+0000 7f3454a12740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 0x55f50b7ee5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.231+0000 7f3454a12740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7e4460 0x55f50b7eb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.231+0000 7f3454a12740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 0x55f50b7ebfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.231+0000 7f3454a12740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55f50b7f19c0 con 0x55f50b7eeb30 2024-04-24T08:18:15.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.231+0000 7f3454a12740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55f50b702810 con 0x55f50b7e4460 2024-04-24T08:18:15.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.231+0000 7f3454a12740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55f50b7f1b40 con 0x55f50b7e3b40 2024-04-24T08:18:15.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.231+0000 7f3454a12740 10 client.0 _ll_register_callbacks cb 0x55f50b6d0140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:15.238 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.231+0000 7f344aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 0x55f50b7ee5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.231+0000 7f344b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 0x55f50b7ebfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.232+0000 7f344aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 0x55f50b7ee5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:50654/0 (socket says 172.21.15.28:50654) 2024-04-24T08:18:15.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.232+0000 7f3442000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7e4460 0x55f50b7eb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.232+0000 7f344aa00640 1 -- 172.21.15.28:0/1204808328 learned_addr learned my addr 172.21.15.28:0/1204808328 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:15.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.232+0000 7f3442000640 1 -- 172.21.15.28:0/1204808328 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 msgr2=0x55f50b7ee5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.232+0000 7f3442000640 1 --2- 172.21.15.28:0/1204808328 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 0x55f50b7ee5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.232+0000 7f3442000640 1 -- 172.21.15.28:0/1204808328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 msgr2=0x55f50b7ebfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.232+0000 7f3442000640 1 --2- 172.21.15.28:0/1204808328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 0x55f50b7ebfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.232+0000 7f3442000640 1 -- 172.21.15.28:0/1204808328 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f50b7133f0 con 0x55f50b7e4460 2024-04-24T08:18:15.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.232+0000 7f344aa00640 1 --2- 172.21.15.28:0/1204808328 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 0x55f50b7ee5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:15.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.232+0000 7f344b400640 1 --2- 172.21.15.28:0/1204808328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 0x55f50b7ebfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:15.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.232+0000 7f3442000640 1 --2- 172.21.15.28:0/1204808328 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7e4460 0x55f50b7eb890 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f34340ee2c0 tx=0x7f3434009620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b14a19d62abb40d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.232+0000 7f3438c00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:18:15.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.232+0000 7f3438c00640 1 -- 172.21.15.28:0/1204808328 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434049070 con 0x55f50b7e4460 2024-04-24T08:18:15.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.233+0000 7f3438c00640 1 -- 172.21.15.28:0/1204808328 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7f34340b9230 con 0x55f50b7e4460 2024-04-24T08:18:15.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.233+0000 7f3454a12740 1 -- 172.21.15.28:0/1204808328 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f343c0043c0 con 0x55f50b7e4460 2024-04-24T08:18:15.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.233+0000 7f3438c00640 1 -- 172.21.15.28:0/1204808328 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f34340bc520 con 0x55f50b7e4460 2024-04-24T08:18:15.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.233+0000 7f3454a12740 10 client.26555 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:15.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.233+0000 7f3454a12740 1 -- 172.21.15.28:0/1204808328 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f50b45b3f0 con 0x55f50b7e4460 2024-04-24T08:18:15.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.233+0000 7f3438c00640 1 -- 172.21.15.28:0/1204808328 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f343c0043c0 con 0x55f50b7e4460 2024-04-24T08:18:15.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.233+0000 7f3438c00640 1 -- 172.21.15.28:0/1204808328 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f34340ee540 con 0x55f50b7e4460 2024-04-24T08:18:15.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.233+0000 7f3454a12740 10 client.26555 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:15.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.233+0000 7f3454a12740 10 client.26555 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:18:15.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.233+0000 7f3454a12740 20 client.26555 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:15.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.233+0000 7f3454a12740 1 --2- 172.21.15.28:0/1204808328 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55f50b7fece0 0x55f50b801100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.233+0000 7f3454a12740 4 client.26555 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:18:15.241 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.233+0000 7f3454a12740 1 -- 172.21.15.28:0/1204808328 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f50b756bf0 con 0x55f50b7fece0 2024-04-24T08:18:15.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.234+0000 7f344aa00640 1 --2- 172.21.15.28:0/1204808328 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55f50b7fece0 0x55f50b801100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.234+0000 7f344aa00640 1 --2- 172.21.15.28:0/1204808328 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55f50b7fece0 0x55f50b801100 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4d6e61ec5e05a9e3 server_cookie=967db5bb5457e09c in_seq=0 out_seq=0 2024-04-24T08:18:15.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.234+0000 7f3438c00640 10 client.26555 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:18:15.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.235+0000 7f3438c00640 1 -- 172.21.15.28:0/1204808328 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f50b756bf0 con 0x55f50b7fece0 2024-04-24T08:18:15.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.235+0000 7f3438c00640 10 client.26555 handle_command_reply: tid=0 2024-04-24T08:18:15.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.235+0000 7f3454a12740 1 client.26555 shutdown 2024-04-24T08:18:15.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.235+0000 7f3454a12740 1 -- 172.21.15.28:0/1204808328 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7e4460 msgr2=0x55f50b7eb890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.235+0000 7f3454a12740 1 --2- 172.21.15.28:0/1204808328 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7e4460 0x55f50b7eb890 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f34340ee2c0 tx=0x7f3434009620 comp rx=0 tx=0).stop 2024-04-24T08:18:15.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.236+0000 7f3454a12740 1 -- 172.21.15.28:0/1204808328 shutdown_connections 2024-04-24T08:18:15.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.236+0000 7f3454a12740 1 --2- 172.21.15.28:0/1204808328 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55f50b7fece0 0x55f50b801100 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.236+0000 7f3454a12740 1 --2- 172.21.15.28:0/1204808328 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 0x55f50b7ebfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.236+0000 7f3454a12740 1 --2- 172.21.15.28:0/1204808328 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7e4460 0x55f50b7eb890 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.236+0000 7f3454a12740 1 --2- 172.21.15.28:0/1204808328 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 0x55f50b7ee5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.236+0000 7f3454a12740 1 -- 172.21.15.28:0/1204808328 >> 172.21.15.28:0/1204808328 conn(0x55f50b7dfe60 msgr2=0x55f50b7e5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.236+0000 7f3438c00640 0 client.26555 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:18:15.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.236+0000 7f3454a12740 1 -- 172.21.15.28:0/1204808328 shutdown_connections 2024-04-24T08:18:15.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.236+0000 7f3454a12740 1 -- 172.21.15.28:0/1204808328 wait complete. 2024-04-24T08:18:15.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.236+0000 7f3454a12740 20 client.26555 trim_cache size 0 max 16384 2024-04-24T08:18:15.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.251+0000 7f3454a12740 1 Processor -- start 2024-04-24T08:18:15.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.251+0000 7f3454a12740 1 -- start start 2024-04-24T08:18:15.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.251+0000 7f3454a12740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 0x55f50b7eb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.251+0000 7f3454a12740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7e4460 0x55f50b7fa880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.251+0000 7f3454a12740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 0x55f50b7ee0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.251+0000 7f3454a12740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f344c1b2850 con 0x55f50b7eeb30 2024-04-24T08:18:15.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.251+0000 7f3454a12740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f344c1b0120 con 0x55f50b7e4460 2024-04-24T08:18:15.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.251+0000 7f3454a12740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f344c1adf90 con 0x55f50b7e3b40 2024-04-24T08:18:15.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.251+0000 7f344b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 0x55f50b7ee0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.251+0000 7f344b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 0x55f50b7ee0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40562/0 (socket says 172.21.15.28:40562) 2024-04-24T08:18:15.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.251+0000 7f344b400640 1 -- 172.21.15.28:0/3122871092 learned_addr learned my addr 172.21.15.28:0/3122871092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:15.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.251+0000 7f344aa00640 1 --2- 172.21.15.28:0/3122871092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 0x55f50b7eb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.252+0000 7f3442000640 1 --2- 172.21.15.28:0/3122871092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7e4460 0x55f50b7fa880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.252+0000 7f344b400640 1 -- 172.21.15.28:0/3122871092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 msgr2=0x55f50b7eb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.252+0000 7f344b400640 1 --2- 172.21.15.28:0/3122871092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 0x55f50b7eb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.252+0000 7f344b400640 1 -- 172.21.15.28:0/3122871092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7e4460 msgr2=0x55f50b7fa880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.252+0000 7f344b400640 1 --2- 172.21.15.28:0/3122871092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7e4460 0x55f50b7fa880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.256 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.252+0000 7f344b400640 1 -- 172.21.15.28:0/3122871092 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f50b5257d0 con 0x55f50b7eeb30 2024-04-24T08:18:15.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.252+0000 7f344aa00640 1 --2- 172.21.15.28:0/3122871092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 0x55f50b7eb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:15.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.252+0000 7f3442000640 1 --2- 172.21.15.28:0/3122871092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7e4460 0x55f50b7fa880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:15.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.252+0000 7f344b400640 1 --2- 172.21.15.28:0/3122871092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 0x55f50b7ee0d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f344400fc30 tx=0x7f344401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed98f4aa736dbea3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.253+0000 7f343b400640 1 -- 172.21.15.28:0/3122871092 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344400fc60 con 0x55f50b7eeb30 2024-04-24T08:18:15.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.253+0000 7f343b400640 1 -- 172.21.15.28:0/3122871092 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f344400fe00 con 0x55f50b7eeb30 2024-04-24T08:18:15.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.253+0000 7f343b400640 1 -- 172.21.15.28:0/3122871092 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f344401f9b0 con 0x55f50b7eeb30 2024-04-24T08:18:15.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.253+0000 7f3454a12740 1 -- 172.21.15.28:0/3122871092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 msgr2=0x55f50b7ee0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.253+0000 7f3454a12740 1 --2- 172.21.15.28:0/3122871092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 0x55f50b7ee0d0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f344400fc30 tx=0x7f344401b000 comp rx=0 tx=0).stop 2024-04-24T08:18:15.257 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.253+0000 7f3454a12740 1 -- 172.21.15.28:0/3122871092 shutdown_connections 2024-04-24T08:18:15.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.253+0000 7f3454a12740 1 --2- 172.21.15.28:0/3122871092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7eeb30 0x55f50b7ee0d0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.253+0000 7f3454a12740 1 --2- 172.21.15.28:0/3122871092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7e4460 0x55f50b7fa880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.253+0000 7f3454a12740 1 --2- 172.21.15.28:0/3122871092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 0x55f50b7eb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.253+0000 7f3454a12740 1 -- 172.21.15.28:0/3122871092 >> 172.21.15.28:0/3122871092 conn(0x55f50b7dfe60 msgr2=0x55f50b7eb120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.254+0000 7f3454a12740 1 -- 172.21.15.28:0/3122871092 shutdown_connections 2024-04-24T08:18:15.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.254+0000 7f3454a12740 1 -- 172.21.15.28:0/3122871092 wait complete. 2024-04-24T08:18:15.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.254+0000 7f3454a12740 1 -- start start 2024-04-24T08:18:15.258 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.254+0000 7f3454a12740 1 Processor -- start 2024-04-24T08:18:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.255+0000 7f3454a12740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 0x55f50b7f09c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.255+0000 7f3454a12740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7e4460 0x55f50b7eef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.255+0000 7f3454a12740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7eeb30 0x55f50b7ef650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.255+0000 7f3454a12740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f344c1ae5e0 con 0x55f50b7e4460 2024-04-24T08:18:15.262 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.255+0000 7f3454a12740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f344c1afa50 con 0x55f50b7eeb30 2024-04-24T08:18:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.255+0000 7f3454a12740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f344c1aecb0 con 0x55f50b7e3b40 2024-04-24T08:18:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.255+0000 7f3454a12740 10 client.0 _ll_register_callbacks cb 0x7f34340f43f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.255+0000 7f3442000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7e4460 0x55f50b7eef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.255+0000 7f3442000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7e4460 0x55f50b7eef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40570/0 (socket says 172.21.15.28:40570) 2024-04-24T08:18:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.255+0000 7f3442000640 1 -- 172.21.15.28:0/237475874 learned_addr learned my addr 172.21.15.28:0/237475874 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.255+0000 7f344aa00640 1 --2- 172.21.15.28:0/237475874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 0x55f50b7f09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.255+0000 7f344b400640 1 --2- 172.21.15.28:0/237475874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7eeb30 0x55f50b7ef650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.263 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.255+0000 7f3442000640 1 -- 172.21.15.28:0/237475874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 msgr2=0x55f50b7f09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.255+0000 7f3442000640 1 --2- 172.21.15.28:0/237475874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 0x55f50b7f09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.255+0000 7f3442000640 1 -- 172.21.15.28:0/237475874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7eeb30 msgr2=0x55f50b7ef650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.255+0000 7f3442000640 1 --2- 172.21.15.28:0/237475874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7eeb30 0x55f50b7ef650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.255+0000 7f3442000640 1 -- 172.21.15.28:0/237475874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f50b7f5d90 con 0x55f50b7e4460 2024-04-24T08:18:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.256+0000 7f3442000640 1 --2- 172.21.15.28:0/237475874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7e4460 0x55f50b7eef10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f343403d040 tx=0x7f34340f0ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb8afeb556874e87 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.256+0000 7f3438c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:18:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.256+0000 7f3438c00640 1 -- 172.21.15.28:0/237475874 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3434049070 con 0x55f50b7e4460 2024-04-24T08:18:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.256+0000 7f3438c00640 1 -- 172.21.15.28:0/237475874 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7f34340c52f0 con 0x55f50b7e4460 2024-04-24T08:18:15.264 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.256+0000 7f3454a12740 1 -- 172.21.15.28:0/237475874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f344400fe00 con 0x55f50b7e4460 2024-04-24T08:18:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.256+0000 7f3438c00640 1 -- 172.21.15.28:0/237475874 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f34340bc990 con 0x55f50b7e4460 2024-04-24T08:18:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.256+0000 7f3454a12740 10 client.16281 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.256+0000 7f3454a12740 1 -- 172.21.15.28:0/237475874 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55f50b7f7090 con 0x55f50b7e4460 2024-04-24T08:18:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.256+0000 7f3438c00640 1 -- 172.21.15.28:0/237475874 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f344400fe00 con 0x55f50b7e4460 2024-04-24T08:18:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.257+0000 7f3438c00640 1 -- 172.21.15.28:0/237475874 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f34340d8070 con 0x55f50b7e4460 2024-04-24T08:18:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.257+0000 7f3454a12740 10 client.16281 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.257+0000 7f3454a12740 10 client.16281 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:18:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.257+0000 7f3454a12740 20 client.16281 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:15.265 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.257+0000 7f3454a12740 1 --2- 172.21.15.28:0/237475874 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55f50b865240 0x55f50b867630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.257+0000 7f3454a12740 4 client.16281 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:18:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.257+0000 7f3454a12740 1 -- 172.21.15.28:0/237475874 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55f50b750a60 con 0x55f50b865240 2024-04-24T08:18:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.257+0000 7f344aa00640 1 --2- 172.21.15.28:0/237475874 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55f50b865240 0x55f50b867630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.258+0000 7f344aa00640 1 --2- 172.21.15.28:0/237475874 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55f50b865240 0x55f50b867630 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b260ae5c092e3ec1 server_cookie=43ea9e26b0c0b6e4 in_seq=0 out_seq=0 2024-04-24T08:18:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.258+0000 7f3438c00640 10 client.16281 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:18:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.258+0000 7f3438c00640 1 -- 172.21.15.28:0/237475874 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55f50b750a60 con 0x55f50b865240 2024-04-24T08:18:15.266 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.258+0000 7f3438c00640 10 client.16281 handle_command_reply: tid=0 2024-04-24T08:18:15.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.258+0000 7f3454a12740 1 client.16281 shutdown 2024-04-24T08:18:15.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.258+0000 7f3454a12740 1 -- 172.21.15.28:0/237475874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7e4460 msgr2=0x55f50b7eef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.258+0000 7f3454a12740 1 --2- 172.21.15.28:0/237475874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7e4460 0x55f50b7eef10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f343403d040 tx=0x7f34340f0ed0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.258+0000 7f3454a12740 1 -- 172.21.15.28:0/237475874 shutdown_connections 2024-04-24T08:18:15.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.258+0000 7f3454a12740 1 --2- 172.21.15.28:0/237475874 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55f50b865240 0x55f50b867630 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.258+0000 7f3454a12740 1 --2- 172.21.15.28:0/237475874 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55f50b7e4460 0x55f50b7eef10 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.258+0000 7f3454a12740 1 --2- 172.21.15.28:0/237475874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55f50b7eeb30 0x55f50b7ef650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.258+0000 7f3454a12740 1 --2- 172.21.15.28:0/237475874 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55f50b7e3b40 0x55f50b7f09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.267 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.258+0000 7f3454a12740 1 -- 172.21.15.28:0/237475874 >> 172.21.15.28:0/237475874 conn(0x55f50b7dfe60 msgr2=0x55f50b7ea160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.258+0000 7f3438c00640 0 client.16281 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:18:15.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.259+0000 7f3454a12740 1 -- 172.21.15.28:0/237475874 shutdown_connections 2024-04-24T08:18:15.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.259+0000 7f3454a12740 1 -- 172.21.15.28:0/237475874 wait complete. 2024-04-24T08:18:15.268 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.259+0000 7f3454a12740 20 client.16281 trim_cache size 0 max 16384 2024-04-24T08:18:15.268 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:18:15.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.259+0000 7f344be00640 1 -- 172.21.15.28:0/2365987836 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f34300a3d20 msgr2=0x7f34300a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.259+0000 7f344be00640 1 --2- 172.21.15.28:0/2365987836 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f34300a3d20 0x7f34300a61e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f3444009fd0 tx=0x7f3444009500 comp rx=0 tx=0).stop 2024-04-24T08:18:15.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.259+0000 7f344be00640 1 -- 172.21.15.28:0/2365987836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f344c1093f0 msgr2=0x7f344c1a0bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.269 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.259+0000 7f344be00640 1 --2- 172.21.15.28:0/2365987836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f344c1093f0 0x7f344c1a0bf0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f3434005940 tx=0x7f34340040c0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.259+0000 7f344be00640 1 -- 172.21.15.28:0/2365987836 shutdown_connections 2024-04-24T08:18:15.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.259+0000 7f344be00640 1 --2- 172.21.15.28:0/2365987836 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f34300a3d20 0x7f34300a61e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.259+0000 7f344be00640 1 --2- 172.21.15.28:0/2365987836 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f344c1093f0 0x7f344c1a0bf0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.259+0000 7f344be00640 1 --2- 172.21.15.28:0/2365987836 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f344c19a820 0x7f344c19acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.259+0000 7f344be00640 1 --2- 172.21.15.28:0/2365987836 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f344c104320 0x7f344c1a06b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.259+0000 7f344be00640 1 -- 172.21.15.28:0/2365987836 >> 172.21.15.28:0/2365987836 conn(0x7f344c0ffb40 msgr2=0x7f344c100f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.260+0000 7f344be00640 1 -- 172.21.15.28:0/2365987836 shutdown_connections 2024-04-24T08:18:15.270 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.260+0000 7f344be00640 1 -- 172.21.15.28:0/2365987836 wait complete. 2024-04-24T08:18:15.279 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:18:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.495+0000 7fb2e1200640 1 -- 172.21.15.28:0/902415223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2dc104320 msgr2=0x7fb2dc104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.495+0000 7fb2e1200640 1 --2- 172.21.15.28:0/902415223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2dc104320 0x7fb2dc104700 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc009860 tx=0x7fb2cc02a340 comp rx=0 tx=0).stop 2024-04-24T08:18:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.495+0000 7fb2e1200640 1 -- 172.21.15.28:0/902415223 shutdown_connections 2024-04-24T08:18:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.495+0000 7fb2e1200640 1 --2- 172.21.15.28:0/902415223 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2dc104cd0 0x7fb2dc108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.495+0000 7fb2e1200640 1 --2- 172.21.15.28:0/902415223 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2dc1093f0 0x7fb2dc110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.495+0000 7fb2e1200640 1 --2- 172.21.15.28:0/902415223 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2dc104320 0x7fb2dc104700 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.495+0000 7fb2e1200640 1 -- 172.21.15.28:0/902415223 >> 172.21.15.28:0/902415223 conn(0x7fb2dc0ffb40 msgr2=0x7fb2dc101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.495+0000 7fb2e1200640 1 -- 172.21.15.28:0/902415223 shutdown_connections 2024-04-24T08:18:15.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.495+0000 7fb2e1200640 1 -- 172.21.15.28:0/902415223 wait complete. 2024-04-24T08:18:15.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.496+0000 7fb2e1200640 1 Processor -- start 2024-04-24T08:18:15.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.496+0000 7fb2e1200640 1 -- start start 2024-04-24T08:18:15.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.497+0000 7fb2e1200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2dc104320 0x7fb2dc1a0690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.497+0000 7fb2e1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2dc1093f0 0x7fb2dc1a0bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.497+0000 7fb2e1200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2dc19a800 0x7fb2dc19acb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.497+0000 7fb2e1200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc114480 con 0x7fb2dc1093f0 2024-04-24T08:18:15.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.497+0000 7fb2e1200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc114300 con 0x7fb2dc19a800 2024-04-24T08:18:15.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.497+0000 7fb2e1200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc114600 con 0x7fb2dc104320 2024-04-24T08:18:15.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.497+0000 7fb2daa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2dc1093f0 0x7fb2dc1a0bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.497+0000 7fb2daa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2dc1093f0 0x7fb2dc1a0bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40580/0 (socket says 172.21.15.28:40580) 2024-04-24T08:18:15.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.497+0000 7fb2daa00640 1 -- 172.21.15.28:0/3329869146 learned_addr learned my addr 172.21.15.28:0/3329869146 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:15.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.497+0000 7fb2db400640 1 --2- 172.21.15.28:0/3329869146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2dc104320 0x7fb2dc1a0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.497+0000 7fb2dbe00640 1 --2- 172.21.15.28:0/3329869146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2dc19a800 0x7fb2dc19acb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.497+0000 7fb2daa00640 1 -- 172.21.15.28:0/3329869146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2dc104320 msgr2=0x7fb2dc1a0690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.497+0000 7fb2daa00640 1 --2- 172.21.15.28:0/3329869146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2dc104320 0x7fb2dc1a0690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.497+0000 7fb2daa00640 1 -- 172.21.15.28:0/3329869146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2dc19a800 msgr2=0x7fb2dc19acb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.497+0000 7fb2daa00640 1 --2- 172.21.15.28:0/3329869146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2dc19a800 0x7fb2dc19acb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.497+0000 7fb2daa00640 1 -- 172.21.15.28:0/3329869146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2dc19b390 con 0x7fb2dc1093f0 2024-04-24T08:18:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.498+0000 7fb2daa00640 1 --2- 172.21.15.28:0/3329869146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2dc1093f0 0x7fb2dc1a0bd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800e9a0 tx=0x7fb2c800ee60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.498+0000 7fb2d3e00640 1 -- 172.21.15.28:0/3329869146 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c800cd80 con 0x7fb2dc1093f0 2024-04-24T08:18:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.498+0000 7fb2d3e00640 1 -- 172.21.15.28:0/3329869146 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2c80035f0 con 0x7fb2dc1093f0 2024-04-24T08:18:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.498+0000 7fb2e1200640 1 -- 172.21.15.28:0/3329869146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2dc19b680 con 0x7fb2dc1093f0 2024-04-24T08:18:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.498+0000 7fb2e1200640 1 -- 172.21.15.28:0/3329869146 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2dc1a7860 con 0x7fb2dc1093f0 2024-04-24T08:18:15.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.499+0000 7fb2d3e00640 1 -- 172.21.15.28:0/3329869146 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8010670 con 0x7fb2dc1093f0 2024-04-24T08:18:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.501+0000 7fb2e5578740 1 Processor -- start 2024-04-24T08:18:15.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.502+0000 7fb2e5578740 1 -- start start 2024-04-24T08:18:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.502+0000 7fb2e5578740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea48b40 0x55642ea48f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.502+0000 7fb2e5578740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea535f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.502+0000 7fb2e5578740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea53b30 0x55642ea55f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.502+0000 7fb2e5578740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55642e9ae1d0 con 0x55642ea49460 2024-04-24T08:18:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.502+0000 7fb2e5578740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55642e954450 con 0x55642ea48b40 2024-04-24T08:18:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.502+0000 7fb2e5578740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55642e9aa190 con 0x55642ea53b30 2024-04-24T08:18:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.502+0000 7fb2d3e00640 1 -- 172.21.15.28:0/3329869146 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb2c80037a0 con 0x7fb2dc1093f0 2024-04-24T08:18:15.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.502+0000 7fb2db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea535f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.502+0000 7fb2dbe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea48b40 0x55642ea48f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.502+0000 7fb2dbe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea48b40 0x55642ea48f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40138/0 (socket says 172.21.15.28:40138) 2024-04-24T08:18:15.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.502+0000 7fb2db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea535f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40590/0 (socket says 172.21.15.28:40590) 2024-04-24T08:18:15.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.502+0000 7fb2dbe00640 1 -- 172.21.15.28:0/3523072579 learned_addr learned my addr 172.21.15.28:0/3523072579 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:15.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.503+0000 7fb2daa00640 1 --2- 172.21.15.28:0/3523072579 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea53b30 0x55642ea55f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.503+0000 7fb2dbe00640 1 -- 172.21.15.28:0/3523072579 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea53b30 msgr2=0x55642ea55f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.503+0000 7fb2dbe00640 1 --2- 172.21.15.28:0/3523072579 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea53b30 0x55642ea55f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.503+0000 7fb2dbe00640 1 -- 172.21.15.28:0/3523072579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 msgr2=0x55642ea535f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.503+0000 7fb2dbe00640 1 --2- 172.21.15.28:0/3523072579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea535f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.503+0000 7fb2dbe00640 1 -- 172.21.15.28:0/3523072579 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55642ea11310 con 0x55642ea48b40 2024-04-24T08:18:15.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.503+0000 7fb2db400640 1 --2- 172.21.15.28:0/3523072579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:15.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.503+0000 7fb2dbe00640 1 --2- 172.21.15.28:0/3523072579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea48b40 0x55642ea48f20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009e60 tx=0x7fb2d4010740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=94e6fe1ebed1b63e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.503+0000 7fb2d0c00640 1 -- 172.21.15.28:0/3523072579 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4016070 con 0x55642ea48b40 2024-04-24T08:18:15.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.503+0000 7fb2d0c00640 1 -- 172.21.15.28:0/3523072579 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2d4004d20 con 0x55642ea48b40 2024-04-24T08:18:15.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.504+0000 7fb2d0c00640 1 -- 172.21.15.28:0/3523072579 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d4012870 con 0x55642ea48b40 2024-04-24T08:18:15.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.504+0000 7fb2d3e00640 1 --2- 172.21.15.28:0/3329869146 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb2c00a3d20 0x7fb2c00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.504+0000 7fb2e5578740 1 -- 172.21.15.28:0/3523072579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea48b40 msgr2=0x55642ea48f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.504+0000 7fb2e5578740 1 --2- 172.21.15.28:0/3523072579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea48b40 0x55642ea48f20 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fb2d4009e60 tx=0x7fb2d4010740 comp rx=0 tx=0).stop 2024-04-24T08:18:15.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.504+0000 7fb2e5578740 1 -- 172.21.15.28:0/3523072579 shutdown_connections 2024-04-24T08:18:15.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.504+0000 7fb2e5578740 1 --2- 172.21.15.28:0/3523072579 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea535f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.504+0000 7fb2e5578740 1 --2- 172.21.15.28:0/3523072579 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea53b30 0x55642ea55f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.504+0000 7fb2e5578740 1 --2- 172.21.15.28:0/3523072579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea48b40 0x55642ea48f20 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.504+0000 7fb2e5578740 1 -- 172.21.15.28:0/3523072579 >> 172.21.15.28:0/3523072579 conn(0x55642ea44e60 msgr2=0x55642ea45240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.504+0000 7fb2d3e00640 1 -- 172.21.15.28:0/3329869146 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7fb2c80d0b40 con 0x7fb2dc1093f0 2024-04-24T08:18:15.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.505+0000 7fb2e5578740 1 -- 172.21.15.28:0/3523072579 shutdown_connections 2024-04-24T08:18:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.505+0000 7fb2db400640 1 --2- 172.21.15.28:0/3329869146 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb2c00a3d20 0x7fb2c00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.505+0000 7fb2e5578740 1 -- 172.21.15.28:0/3523072579 wait complete. 2024-04-24T08:18:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.505+0000 7fb2e5578740 1 -- start start 2024-04-24T08:18:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.505+0000 7fb2db400640 1 --2- 172.21.15.28:0/3329869146 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb2c00a3d20 0x7fb2c00a61e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc02a930 tx=0x7fb2cc0056d0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.505+0000 7fb2e5578740 1 Processor -- start 2024-04-24T08:18:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.506+0000 7fb2e5578740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea48b40 0x55642ea50900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.506+0000 7fb2e5578740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea50e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.506+0000 7fb2e5578740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea53b30 0x55642ea51380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.506+0000 7fb2e5578740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55642ea569c0 con 0x55642ea49460 2024-04-24T08:18:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.506+0000 7fb2e5578740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55642e967810 con 0x55642ea53b30 2024-04-24T08:18:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.506+0000 7fb2e5578740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55642ea56b40 con 0x55642ea48b40 2024-04-24T08:18:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.506+0000 7fb2e5578740 10 client.0 _ll_register_callbacks cb 0x55642e935140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.506+0000 7fb2daa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea50e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.506+0000 7fb2dbe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea53b30 0x55642ea51380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.506+0000 7fb2db400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea48b40 0x55642ea50900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.506+0000 7fb2dbe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea53b30 0x55642ea51380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40142/0 (socket says 172.21.15.28:40142) 2024-04-24T08:18:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.506+0000 7fb2dbe00640 1 -- 172.21.15.28:0/1798950970 learned_addr learned my addr 172.21.15.28:0/1798950970 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:15.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.506+0000 7fb2daa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea50e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40606/0 (socket says 172.21.15.28:40606) 2024-04-24T08:18:15.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.507+0000 7fb2daa00640 1 -- 172.21.15.28:0/1798950970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea48b40 msgr2=0x55642ea50900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.507+0000 7fb2daa00640 1 --2- 172.21.15.28:0/1798950970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea48b40 0x55642ea50900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.507+0000 7fb2daa00640 1 -- 172.21.15.28:0/1798950970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea53b30 msgr2=0x55642ea51380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.507+0000 7fb2daa00640 1 --2- 172.21.15.28:0/1798950970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea53b30 0x55642ea51380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.507+0000 7fb2daa00640 1 -- 172.21.15.28:0/1798950970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55642e9783f0 con 0x55642ea49460 2024-04-24T08:18:15.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.507+0000 7fb2dbe00640 1 --2- 172.21.15.28:0/1798950970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea53b30 0x55642ea51380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:15.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.507+0000 7fb2daa00640 1 --2- 172.21.15.28:0/1798950970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea50e40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80ce210 tx=0x7fb2c809c8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5ca98f0f8d6f473 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.507+0000 7fb2c5600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:18:15.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.507+0000 7fb2c5600640 1 -- 172.21.15.28:0/1798950970 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c809ca60 con 0x55642ea49460 2024-04-24T08:18:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.507+0000 7fb2c5600640 1 -- 172.21.15.28:0/1798950970 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7fb2c80990d0 con 0x55642ea49460 2024-04-24T08:18:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.507+0000 7fb2e5578740 1 -- 172.21.15.28:0/1798950970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2d4004d20 con 0x55642ea49460 2024-04-24T08:18:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.508+0000 7fb2c5600640 1 -- 172.21.15.28:0/1798950970 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb2c80a0e10 con 0x55642ea49460 2024-04-24T08:18:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.508+0000 7fb2e5578740 10 client.16293 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.508+0000 7fb2e5578740 1 -- 172.21.15.28:0/1798950970 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55642e6c03f0 con 0x55642ea49460 2024-04-24T08:18:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.508+0000 7fb2c5600640 1 -- 172.21.15.28:0/1798950970 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2d4004d20 con 0x55642ea49460 2024-04-24T08:18:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.508+0000 7fb2c5600640 1 -- 172.21.15.28:0/1798950970 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fb2c809beb0 con 0x55642ea49460 2024-04-24T08:18:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.508+0000 7fb2e5578740 10 client.16293 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.508+0000 7fb2e5578740 10 client.16293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:18:15.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.508+0000 7fb2e5578740 20 client.16293 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:15.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.508+0000 7fb2e5578740 1 --2- 172.21.15.28:0/1798950970 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55642ea63da0 0x55642ea661b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.508+0000 7fb2e5578740 4 client.16293 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:18:15.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.508+0000 7fb2e5578740 1 -- 172.21.15.28:0/1798950970 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55642e9bbbf0 con 0x55642ea63da0 2024-04-24T08:18:15.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.509+0000 7fb2db400640 1 --2- 172.21.15.28:0/1798950970 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55642ea63da0 0x55642ea661b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.509+0000 7fb2db400640 1 --2- 172.21.15.28:0/1798950970 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55642ea63da0 0x55642ea661b0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2de78a5e40bccb8e server_cookie=617e8312eba60718 in_seq=0 out_seq=0 2024-04-24T08:18:15.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.509+0000 7fb2c5600640 10 client.16293 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:15.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.510+0000 7fb2c5600640 1 -- 172.21.15.28:0/1798950970 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55642e9bbbf0 con 0x55642ea63da0 2024-04-24T08:18:15.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.510+0000 7fb2c5600640 10 client.16293 handle_command_reply: tid=0 2024-04-24T08:18:15.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.510+0000 7fb2e5578740 1 client.16293 shutdown 2024-04-24T08:18:15.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.510+0000 7fb2e5578740 1 -- 172.21.15.28:0/1798950970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 msgr2=0x55642ea50e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.510+0000 7fb2e5578740 1 --2- 172.21.15.28:0/1798950970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea50e40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fb2c80ce210 tx=0x7fb2c809c8f0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.510+0000 7fb2e5578740 1 -- 172.21.15.28:0/1798950970 shutdown_connections 2024-04-24T08:18:15.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.510+0000 7fb2e5578740 1 --2- 172.21.15.28:0/1798950970 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55642ea63da0 0x55642ea661b0 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.510+0000 7fb2e5578740 1 --2- 172.21.15.28:0/1798950970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea50e40 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.511+0000 7fb2e5578740 1 --2- 172.21.15.28:0/1798950970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea53b30 0x55642ea51380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.511+0000 7fb2e5578740 1 --2- 172.21.15.28:0/1798950970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea48b40 0x55642ea50900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.511+0000 7fb2c5600640 0 client.16293 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:15.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.511+0000 7fb2e5578740 1 -- 172.21.15.28:0/1798950970 >> 172.21.15.28:0/1798950970 conn(0x55642ea44e60 msgr2=0x55642ea4ac60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.511+0000 7fb2e5578740 1 -- 172.21.15.28:0/1798950970 shutdown_connections 2024-04-24T08:18:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.511+0000 7fb2e5578740 1 -- 172.21.15.28:0/1798950970 wait complete. 2024-04-24T08:18:15.520 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.511+0000 7fb2e5578740 20 client.16293 trim_cache size 0 max 16384 2024-04-24T08:18:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.526+0000 7fb2e5578740 1 Processor -- start 2024-04-24T08:18:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.526+0000 7fb2e5578740 1 -- start start 2024-04-24T08:18:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.526+0000 7fb2e5578740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea48b40 0x55642ea5f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.526+0000 7fb2e5578740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea43dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.526+0000 7fb2e5578740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea53b30 0x55642ea6ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.526+0000 7fb2e5578740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1b2880 con 0x55642ea49460 2024-04-24T08:18:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.526+0000 7fb2e5578740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1b0150 con 0x55642ea48b40 2024-04-24T08:18:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.526+0000 7fb2e5578740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1adfc0 con 0x55642ea53b30 2024-04-24T08:18:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.527+0000 7fb2daa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea43dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.527+0000 7fb2daa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea43dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40608/0 (socket says 172.21.15.28:40608) 2024-04-24T08:18:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.527+0000 7fb2daa00640 1 -- 172.21.15.28:0/2122270072 learned_addr learned my addr 172.21.15.28:0/2122270072 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.527+0000 7fb2db400640 1 --2- 172.21.15.28:0/2122270072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea48b40 0x55642ea5f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.527+0000 7fb2dbe00640 1 --2- 172.21.15.28:0/2122270072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea53b30 0x55642ea6ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.527+0000 7fb2daa00640 1 -- 172.21.15.28:0/2122270072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea53b30 msgr2=0x55642ea6ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.527+0000 7fb2daa00640 1 --2- 172.21.15.28:0/2122270072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea53b30 0x55642ea6ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.527+0000 7fb2daa00640 1 -- 172.21.15.28:0/2122270072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea48b40 msgr2=0x55642ea5f880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.527+0000 7fb2daa00640 1 --2- 172.21.15.28:0/2122270072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea48b40 0x55642ea5f880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.527+0000 7fb2daa00640 1 -- 172.21.15.28:0/2122270072 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55642e78a7d0 con 0x55642ea49460 2024-04-24T08:18:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.527+0000 7fb2db400640 1 --2- 172.21.15.28:0/2122270072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea48b40 0x55642ea5f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.527+0000 7fb2daa00640 1 --2- 172.21.15.28:0/2122270072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea43dd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8000c00 tx=0x7fb2c80d4380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=479515fdfb5aa572 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.528+0000 7fb2c7e00640 1 -- 172.21.15.28:0/2122270072 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c809ca60 con 0x55642ea49460 2024-04-24T08:18:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.528+0000 7fb2c7e00640 1 -- 172.21.15.28:0/2122270072 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2c80a56d0 con 0x55642ea49460 2024-04-24T08:18:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.528+0000 7fb2c7e00640 1 -- 172.21.15.28:0/2122270072 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2c80a5240 con 0x55642ea49460 2024-04-24T08:18:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.528+0000 7fb2e5578740 1 -- 172.21.15.28:0/2122270072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 msgr2=0x55642ea43dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.528+0000 7fb2e5578740 1 --2- 172.21.15.28:0/2122270072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea43dd0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8000c00 tx=0x7fb2c80d4380 comp rx=0 tx=0).stop 2024-04-24T08:18:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.528+0000 7fb2e5578740 1 -- 172.21.15.28:0/2122270072 shutdown_connections 2024-04-24T08:18:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.528+0000 7fb2e5578740 1 --2- 172.21.15.28:0/2122270072 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea49460 0x55642ea43dd0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.528+0000 7fb2e5578740 1 --2- 172.21.15.28:0/2122270072 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea53b30 0x55642ea6ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.528+0000 7fb2e5578740 1 --2- 172.21.15.28:0/2122270072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea48b40 0x55642ea5f880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.528+0000 7fb2e5578740 1 -- 172.21.15.28:0/2122270072 >> 172.21.15.28:0/2122270072 conn(0x55642ea44e60 msgr2=0x55642ea436c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.528+0000 7fb2e5578740 1 -- 172.21.15.28:0/2122270072 shutdown_connections 2024-04-24T08:18:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.528+0000 7fb2e5578740 1 -- 172.21.15.28:0/2122270072 wait complete. 2024-04-24T08:18:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.529+0000 7fb2e5578740 1 -- start start 2024-04-24T08:18:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.529+0000 7fb2e5578740 1 Processor -- start 2024-04-24T08:18:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.529+0000 7fb2e5578740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea48b40 0x55642ea5c070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.529+0000 7fb2e5578740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea49460 0x55642ea4d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.529+0000 7fb2e5578740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea53b30 0x55642ea4d730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.529+0000 7fb2e5578740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1ae610 con 0x55642ea48b40 2024-04-24T08:18:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.529+0000 7fb2e5578740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1afa80 con 0x55642ea49460 2024-04-24T08:18:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.529+0000 7fb2e5578740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb2dc1aece0 con 0x55642ea53b30 2024-04-24T08:18:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.529+0000 7fb2e5578740 10 client.0 _ll_register_callbacks cb 0x55642ea53a80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.530+0000 7fb2db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea48b40 0x55642ea5c070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.530+0000 7fb2db400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea48b40 0x55642ea5c070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40614/0 (socket says 172.21.15.28:40614) 2024-04-24T08:18:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.530+0000 7fb2db400640 1 -- 172.21.15.28:0/2634485320 learned_addr learned my addr 172.21.15.28:0/2634485320 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:15.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.530+0000 7fb2daa00640 1 --2- 172.21.15.28:0/2634485320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea49460 0x55642ea4d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.530+0000 7fb2dbe00640 1 --2- 172.21.15.28:0/2634485320 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea53b30 0x55642ea4d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.530+0000 7fb2db400640 1 -- 172.21.15.28:0/2634485320 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea53b30 msgr2=0x55642ea4d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.530+0000 7fb2db400640 1 --2- 172.21.15.28:0/2634485320 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea53b30 0x55642ea4d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.530+0000 7fb2db400640 1 -- 172.21.15.28:0/2634485320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea49460 msgr2=0x55642ea4d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.530+0000 7fb2db400640 1 --2- 172.21.15.28:0/2634485320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea49460 0x55642ea4d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.530+0000 7fb2db400640 1 -- 172.21.15.28:0/2634485320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55642ea4ddf0 con 0x55642ea48b40 2024-04-24T08:18:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.530+0000 7fb2db400640 1 --2- 172.21.15.28:0/2634485320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea48b40 0x55642ea5c070 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc005e00 tx=0x7fb2cc02a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0674de0bab9161e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.530+0000 7fb2c5600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:18:15.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.531+0000 7fb2c5600640 1 -- 172.21.15.28:0/2634485320 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2cc037040 con 0x55642ea48b40 2024-04-24T08:18:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.531+0000 7fb2c5600640 1 -- 172.21.15.28:0/2634485320 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7fb2cc04f350 con 0x55642ea48b40 2024-04-24T08:18:15.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.531+0000 7fb2e5578740 1 -- 172.21.15.28:0/2634485320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2c80a56d0 con 0x55642ea48b40 2024-04-24T08:18:15.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.531+0000 7fb2c5600640 1 -- 172.21.15.28:0/2634485320 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb2cc05f050 con 0x55642ea48b40 2024-04-24T08:18:15.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.531+0000 7fb2e5578740 10 client.16299 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:15.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.531+0000 7fb2e5578740 1 -- 172.21.15.28:0/2634485320 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55642eaca2f0 con 0x55642ea48b40 2024-04-24T08:18:15.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.531+0000 7fb2c5600640 1 -- 172.21.15.28:0/2634485320 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2c80a56d0 con 0x55642ea48b40 2024-04-24T08:18:15.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.532+0000 7fb2c5600640 1 -- 172.21.15.28:0/2634485320 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fb2cc003c70 con 0x55642ea48b40 2024-04-24T08:18:15.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.532+0000 7fb2e5578740 10 client.16299 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:15.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.532+0000 7fb2e5578740 10 client.16299 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:18:15.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.532+0000 7fb2e5578740 20 client.16299 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.532+0000 7fb2e5578740 1 --2- 172.21.15.28:0/2634485320 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55642eaca8d0 0x55642eacccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.532+0000 7fb2e5578740 4 client.16299 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:18:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.532+0000 7fb2e5578740 1 -- 172.21.15.28:0/2634485320 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55642e9b5a60 con 0x55642eaca8d0 2024-04-24T08:18:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.532+0000 7fb2daa00640 1 --2- 172.21.15.28:0/2634485320 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55642eaca8d0 0x55642eacccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.532+0000 7fb2daa00640 1 --2- 172.21.15.28:0/2634485320 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55642eaca8d0 0x55642eacccf0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34e1a2253f32c8e2 server_cookie=81fd019f8af9677c in_seq=0 out_seq=0 2024-04-24T08:18:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.532+0000 7fb2c5600640 10 client.16299 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.533+0000 7fb2c5600640 1 -- 172.21.15.28:0/2634485320 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55642e9b5a60 con 0x55642eaca8d0 2024-04-24T08:18:15.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.533+0000 7fb2c5600640 10 client.16299 handle_command_reply: tid=0 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.533+0000 7fb2e5578740 1 client.16299 shutdown 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.533+0000 7fb2e5578740 1 -- 172.21.15.28:0/2634485320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea48b40 msgr2=0x55642ea5c070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.533+0000 7fb2e5578740 1 --2- 172.21.15.28:0/2634485320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea48b40 0x55642ea5c070 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc005e00 tx=0x7fb2cc02a340 comp rx=0 tx=0).stop 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.533+0000 7fb2e5578740 1 -- 172.21.15.28:0/2634485320 shutdown_connections 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.533+0000 7fb2e5578740 1 --2- 172.21.15.28:0/2634485320 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55642eaca8d0 0x55642eacccf0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.533+0000 7fb2e5578740 1 --2- 172.21.15.28:0/2634485320 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55642ea53b30 0x55642ea4d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.533+0000 7fb2e5578740 1 --2- 172.21.15.28:0/2634485320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55642ea49460 0x55642ea4d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.533+0000 7fb2c5600640 0 client.16299 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:15.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.533+0000 7fb2e5578740 1 --2- 172.21.15.28:0/2634485320 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55642ea48b40 0x55642ea5c070 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.533+0000 7fb2e5578740 1 -- 172.21.15.28:0/2634485320 >> 172.21.15.28:0/2634485320 conn(0x55642ea44e60 msgr2=0x55642ea43f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.533+0000 7fb2e5578740 1 -- 172.21.15.28:0/2634485320 shutdown_connections 2024-04-24T08:18:15.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.533+0000 7fb2e5578740 1 -- 172.21.15.28:0/2634485320 wait complete. 2024-04-24T08:18:15.542 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.533+0000 7fb2e5578740 20 client.16299 trim_cache size 0 max 16384 2024-04-24T08:18:15.542 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:18:15.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.534+0000 7fb2e1200640 1 -- 172.21.15.28:0/3329869146 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb2c00a3d20 msgr2=0x7fb2c00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.534+0000 7fb2e1200640 1 --2- 172.21.15.28:0/3329869146 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb2c00a3d20 0x7fb2c00a61e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc02a930 tx=0x7fb2cc0056d0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.534+0000 7fb2e1200640 1 -- 172.21.15.28:0/3329869146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2dc1093f0 msgr2=0x7fb2dc1a0bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:15.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.534+0000 7fb2e1200640 1 --2- 172.21.15.28:0/3329869146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2dc1093f0 0x7fb2dc1a0bd0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fb2c800e9a0 tx=0x7fb2c800ee60 comp rx=0 tx=0).stop 2024-04-24T08:18:15.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.534+0000 7fb2e1200640 1 -- 172.21.15.28:0/3329869146 shutdown_connections 2024-04-24T08:18:15.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.534+0000 7fb2e1200640 1 --2- 172.21.15.28:0/3329869146 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb2c00a3d20 0x7fb2c00a61e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.534+0000 7fb2e1200640 1 --2- 172.21.15.28:0/3329869146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb2dc1093f0 0x7fb2dc1a0bd0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.534+0000 7fb2e1200640 1 --2- 172.21.15.28:0/3329869146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb2dc19a800 0x7fb2dc19acb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.534+0000 7fb2e1200640 1 --2- 172.21.15.28:0/3329869146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb2dc104320 0x7fb2dc1a0690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:15.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.534+0000 7fb2e1200640 1 -- 172.21.15.28:0/3329869146 >> 172.21.15.28:0/3329869146 conn(0x7fb2dc0ffb40 msgr2=0x7fb2dc100e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:15.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.534+0000 7fb2e1200640 1 -- 172.21.15.28:0/3329869146 shutdown_connections 2024-04-24T08:18:15.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:15.534+0000 7fb2e1200640 1 -- 172.21.15.28:0/3329869146 wait complete. 2024-04-24T08:18:15.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:15 smithi194 ceph-mon[32836]: pgmap v243: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 67 KiB/s rd, 90 KiB/s wr, 10 op/s 2024-04-24T08:18:15.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:15 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2784991869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:16.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:15 smithi028 ceph-mon[27786]: pgmap v243: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 67 KiB/s rd, 90 KiB/s wr, 10 op/s 2024-04-24T08:18:16.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:15 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2784991869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:16.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:15 smithi117 ceph-mon[32438]: pgmap v243: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 67 KiB/s rd, 90 KiB/s wr, 10 op/s 2024-04-24T08:18:16.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:15 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2784991869' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:16.554 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:18:16.555 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:16.555 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:18:16.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.784+0000 7fe4d3200640 1 -- 172.21.15.28:0/3982826441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d4108580 msgr2=0x7fe4d4108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.784+0000 7fe4d3200640 1 --2- 172.21.15.28:0/3982826441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d4108580 0x7fe4d4108a00 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc009930 tx=0x7fe4bc02a440 comp rx=0 tx=0).stop 2024-04-24T08:18:16.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.784+0000 7fe4d3200640 1 -- 172.21.15.28:0/3982826441 shutdown_connections 2024-04-24T08:18:16.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.784+0000 7fe4d3200640 1 --2- 172.21.15.28:0/3982826441 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d4108580 0x7fe4d4108a00 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.784+0000 7fe4d3200640 1 --2- 172.21.15.28:0/3982826441 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4d4108f40 0x7fe4d41153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.784+0000 7fe4d3200640 1 --2- 172.21.15.28:0/3982826441 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4d410d500 0x7fe4d410d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.784+0000 7fe4d3200640 1 -- 172.21.15.28:0/3982826441 >> 172.21.15.28:0/3982826441 conn(0x7fe4d4077e20 msgr2=0x7fe4d4106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:16.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.784+0000 7fe4d3200640 1 -- 172.21.15.28:0/3982826441 shutdown_connections 2024-04-24T08:18:16.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.784+0000 7fe4d3200640 1 -- 172.21.15.28:0/3982826441 wait complete. 2024-04-24T08:18:16.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.785+0000 7fe4d3200640 1 Processor -- start 2024-04-24T08:18:16.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.785+0000 7fe4d3200640 1 -- start start 2024-04-24T08:18:16.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.786+0000 7fe4d3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d4108580 0x7fe4d419ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.786+0000 7fe4d3200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4d4108f40 0x7fe4d419f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.786+0000 7fe4d3200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4d410d500 0x7fe4d41a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.786+0000 7fe4d3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d41188d0 con 0x7fe4d4108580 2024-04-24T08:18:16.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.786+0000 7fe4d3200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d4118750 con 0x7fe4d4108f40 2024-04-24T08:18:16.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.786+0000 7fe4d3200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d4118a50 con 0x7fe4d410d500 2024-04-24T08:18:16.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.786+0000 7fe4d1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d4108580 0x7fe4d419ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.786+0000 7fe4d1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d4108580 0x7fe4d419ed50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40636/0 (socket says 172.21.15.28:40636) 2024-04-24T08:18:16.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.786+0000 7fe4d1e00640 1 -- 172.21.15.28:0/378561021 learned_addr learned my addr 172.21.15.28:0/378561021 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:16.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.786+0000 7fe4d1400640 1 --2- 172.21.15.28:0/378561021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4d4108f40 0x7fe4d419f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.786+0000 7fe4d2800640 1 --2- 172.21.15.28:0/378561021 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4d410d500 0x7fe4d41a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.787+0000 7fe4d1e00640 1 -- 172.21.15.28:0/378561021 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4d410d500 msgr2=0x7fe4d41a3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.787+0000 7fe4d1e00640 1 --2- 172.21.15.28:0/378561021 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4d410d500 0x7fe4d41a3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.787+0000 7fe4d1e00640 1 -- 172.21.15.28:0/378561021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4d4108f40 msgr2=0x7fe4d419f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.787+0000 7fe4d1e00640 1 --2- 172.21.15.28:0/378561021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4d4108f40 0x7fe4d419f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.787+0000 7fe4d1e00640 1 -- 172.21.15.28:0/378561021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4d41a3d40 con 0x7fe4d4108580 2024-04-24T08:18:16.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.787+0000 7fe4d1400640 1 --2- 172.21.15.28:0/378561021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4d4108f40 0x7fe4d419f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:16.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.787+0000 7fe4d1e00640 1 --2- 172.21.15.28:0/378561021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d4108580 0x7fe4d419ed50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe4c400e9c0 tx=0x7fe4c400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:16.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.787+0000 7fe4caa00640 1 -- 172.21.15.28:0/378561021 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c400cd80 con 0x7fe4d4108580 2024-04-24T08:18:16.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.787+0000 7fe4caa00640 1 -- 172.21.15.28:0/378561021 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4c40035f0 con 0x7fe4d4108580 2024-04-24T08:18:16.792 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.787+0000 7fe4caa00640 1 -- 172.21.15.28:0/378561021 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4c4010690 con 0x7fe4d4108580 2024-04-24T08:18:16.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.787+0000 7fe4d3200640 1 -- 172.21.15.28:0/378561021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4d41a4030 con 0x7fe4d4108580 2024-04-24T08:18:16.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.788+0000 7fe4d3200640 1 -- 172.21.15.28:0/378561021 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4d41ab8c0 con 0x7fe4d4108580 2024-04-24T08:18:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.791+0000 7fe4dbe23740 1 Processor -- start 2024-04-24T08:18:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.791+0000 7fe4caa00640 1 -- 172.21.15.28:0/378561021 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe4c40037a0 con 0x7fe4d4108580 2024-04-24T08:18:16.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.791+0000 7fe4dbe23740 1 -- start start 2024-04-24T08:18:16.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.791+0000 7fe4dbe23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34bb270 0x561ba34bb650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.791+0000 7fe4dbe23740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bbb90 0x561ba34c5d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.791+0000 7fe4dbe23740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34c6260 0x561ba34c8650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.791+0000 7fe4dbe23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561ba3420e70 con 0x561ba34bb270 2024-04-24T08:18:16.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.791+0000 7fe4dbe23740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561ba33c7380 con 0x561ba34bbb90 2024-04-24T08:18:16.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.791+0000 7fe4dbe23740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561ba341ce30 con 0x561ba34c6260 2024-04-24T08:18:16.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.792+0000 7fe4d2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34bb270 0x561ba34bb650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.792+0000 7fe4d2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34bb270 0x561ba34bb650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40650/0 (socket says 172.21.15.28:40650) 2024-04-24T08:18:16.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.792+0000 7fe4d2800640 1 -- 172.21.15.28:0/3932588162 learned_addr learned my addr 172.21.15.28:0/3932588162 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:16.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.792+0000 7fe4d1e00640 1 --2- 172.21.15.28:0/3932588162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bbb90 0x561ba34c5d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.792+0000 7fe4d1400640 1 --2- 172.21.15.28:0/3932588162 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34c6260 0x561ba34c8650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.792+0000 7fe4d2800640 1 -- 172.21.15.28:0/3932588162 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34c6260 msgr2=0x561ba34c8650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:16.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.792+0000 7fe4d2800640 1 --2- 172.21.15.28:0/3932588162 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34c6260 0x561ba34c8650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.792+0000 7fe4d2800640 1 -- 172.21.15.28:0/3932588162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bbb90 msgr2=0x561ba34c5d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:16.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.792+0000 7fe4d2800640 1 --2- 172.21.15.28:0/3932588162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bbb90 0x561ba34c5d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.792+0000 7fe4d2800640 1 -- 172.21.15.28:0/3932588162 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ba31fb7e0 con 0x561ba34bb270 2024-04-24T08:18:16.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.792+0000 7fe4d1e00640 1 --2- 172.21.15.28:0/3932588162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bbb90 0x561ba34c5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:16.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.793+0000 7fe4d2800640 1 --2- 172.21.15.28:0/3932588162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34bb270 0x561ba34bb650 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc009fc0 tx=0x7fe4cc0118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1950051943f95ff server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:16.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.793+0000 7fe4caa00640 1 --2- 172.21.15.28:0/378561021 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4b80a3d20 0x7fe4b80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.793+0000 7fe4caa00640 1 -- 172.21.15.28:0/378561021 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7fe4c40d0c50 con 0x7fe4d4108580 2024-04-24T08:18:16.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.793+0000 7fe4c3400640 1 -- 172.21.15.28:0/3932588162 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc004d20 con 0x561ba34bb270 2024-04-24T08:18:16.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.793+0000 7fe4d1e00640 1 --2- 172.21.15.28:0/378561021 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4b80a3d20 0x7fe4b80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.793+0000 7fe4c3400640 1 -- 172.21.15.28:0/3932588162 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4cc004050 con 0x561ba34bb270 2024-04-24T08:18:16.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.793+0000 7fe4c3400640 1 -- 172.21.15.28:0/3932588162 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4cc005280 con 0x561ba34bb270 2024-04-24T08:18:16.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.794+0000 7fe4d1e00640 1 --2- 172.21.15.28:0/378561021 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4b80a3d20 0x7fe4b80a61e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x561ba34bc830 tx=0x7fe4c40d4700 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:16.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.794+0000 7fe4dbe23740 1 -- 172.21.15.28:0/3932588162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34bb270 msgr2=0x561ba34bb650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.794+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/3932588162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34bb270 0x561ba34bb650 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fe4cc009fc0 tx=0x7fe4cc0118d0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.794+0000 7fe4dbe23740 1 -- 172.21.15.28:0/3932588162 shutdown_connections 2024-04-24T08:18:16.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.794+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/3932588162 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34c6260 0x561ba34c8650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.794+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/3932588162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bbb90 0x561ba34c5d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.794+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/3932588162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34bb270 0x561ba34bb650 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.794+0000 7fe4dbe23740 1 -- 172.21.15.28:0/3932588162 >> 172.21.15.28:0/3932588162 conn(0x561ba34b7590 msgr2=0x561ba34b7970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:16.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.794+0000 7fe4dbe23740 1 -- 172.21.15.28:0/3932588162 shutdown_connections 2024-04-24T08:18:16.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.794+0000 7fe4dbe23740 1 -- 172.21.15.28:0/3932588162 wait complete. 2024-04-24T08:18:16.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.795+0000 7fe4dbe23740 1 -- start start 2024-04-24T08:18:16.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.795+0000 7fe4dbe23740 1 Processor -- start 2024-04-24T08:18:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.795+0000 7fe4dbe23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34bb270 0x561ba34c3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.795+0000 7fe4dbe23740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bbb90 0x561ba34c3690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.795+0000 7fe4dbe23740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34c6260 0x561ba34c3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.806 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.795+0000 7fe4dbe23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x561ba34c9050 con 0x561ba34bb270 2024-04-24T08:18:16.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.795+0000 7fe4dbe23740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x561ba33da660 con 0x561ba34c6260 2024-04-24T08:18:16.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.795+0000 7fe4dbe23740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x561ba34c91d0 con 0x561ba34bbb90 2024-04-24T08:18:16.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.795+0000 7fe4dbe23740 10 client.0 _ll_register_callbacks cb 0x561ba33c8aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:16.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.796+0000 7fe4d2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34bb270 0x561ba34c3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.796+0000 7fe4d2800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34bb270 0x561ba34c3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:40656/0 (socket says 172.21.15.28:40656) 2024-04-24T08:18:16.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.796+0000 7fe4d2800640 1 -- 172.21.15.28:0/1034247622 learned_addr learned my addr 172.21.15.28:0/1034247622 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:16.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.796+0000 7fe4d1400640 1 --2- 172.21.15.28:0/1034247622 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bbb90 0x561ba34c3690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.796+0000 7fe4d2800640 1 --2- 172.21.15.28:0/1034247622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34c6260 0x561ba34c3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.796+0000 7fe4d1400640 1 -- 172.21.15.28:0/1034247622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34c6260 msgr2=0x561ba34c3bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.796+0000 7fe4d1400640 1 --2- 172.21.15.28:0/1034247622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34c6260 0x561ba34c3bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.796+0000 7fe4d1400640 1 -- 172.21.15.28:0/1034247622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34bb270 msgr2=0x561ba34c3150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.796+0000 7fe4d1400640 1 --2- 172.21.15.28:0/1034247622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34bb270 0x561ba34c3150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.796+0000 7fe4d1400640 1 -- 172.21.15.28:0/1034247622 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ba325bf70 con 0x561ba34bbb90 2024-04-24T08:18:16.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.796+0000 7fe4d2800640 1 --2- 172.21.15.28:0/1034247622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34bb270 0x561ba34c3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.797+0000 7fe4d1400640 1 --2- 172.21.15.28:0/1034247622 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bbb90 0x561ba34c3690 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc005940 tx=0x7fe4bc0317e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fa38e801f32fe961 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.797+0000 7fe4b7e00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:18:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.797+0000 7fe4b7e00640 1 -- 172.21.15.28:0/1034247622 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc035040 con 0x561ba34bbb90 2024-04-24T08:18:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.797+0000 7fe4dbe23740 1 -- 172.21.15.28:0/1034247622 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4cc004050 con 0x561ba34bbb90 2024-04-24T08:18:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.797+0000 7fe4b7e00640 1 -- 172.21.15.28:0/1034247622 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7fe4bc004250 con 0x561ba34bbb90 2024-04-24T08:18:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.797+0000 7fe4b7e00640 1 -- 172.21.15.28:0/1034247622 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe4bc004450 con 0x561ba34bbb90 2024-04-24T08:18:16.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.797+0000 7fe4dbe23740 10 client.26611 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:16.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.797+0000 7fe4dbe23740 1 -- 172.21.15.28:0/1034247622 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561ba30f3170 con 0x561ba34bbb90 2024-04-24T08:18:16.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.798+0000 7fe4b7e00640 1 -- 172.21.15.28:0/1034247622 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4cc004050 con 0x561ba34bbb90 2024-04-24T08:18:16.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.798+0000 7fe4b7e00640 1 -- 172.21.15.28:0/1034247622 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fe4bc004c20 con 0x561ba34bbb90 2024-04-24T08:18:16.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.798+0000 7fe4dbe23740 10 client.26611 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:16.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.798+0000 7fe4dbe23740 10 client.26611 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:18:16.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.798+0000 7fe4dbe23740 20 client.26611 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:16.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.798+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/1034247622 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x561ba34d6510 0x561ba34d8920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.798+0000 7fe4dbe23740 4 client.26611 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:18:16.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.798+0000 7fe4dbe23740 1 -- 172.21.15.28:0/1034247622 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561ba342e890 con 0x561ba34d6510 2024-04-24T08:18:16.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.798+0000 7fe4d2800640 1 --2- 172.21.15.28:0/1034247622 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x561ba34d6510 0x561ba34d8920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.799+0000 7fe4d2800640 1 --2- 172.21.15.28:0/1034247622 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x561ba34d6510 0x561ba34d8920 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99a1e1edbf6a81ee server_cookie=5a4d2fa399b214a2 in_seq=0 out_seq=0 2024-04-24T08:18:16.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.799+0000 7fe4b7e00640 10 client.26611 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:16.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.800+0000 7fe4b7e00640 1 -- 172.21.15.28:0/1034247622 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561ba342e890 con 0x561ba34d6510 2024-04-24T08:18:16.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.800+0000 7fe4b7e00640 10 client.26611 handle_command_reply: tid=0 2024-04-24T08:18:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.800+0000 7fe4dbe23740 1 client.26611 shutdown 2024-04-24T08:18:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.800+0000 7fe4dbe23740 1 -- 172.21.15.28:0/1034247622 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bbb90 msgr2=0x561ba34c3690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.800+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/1034247622 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bbb90 0x561ba34c3690 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc005940 tx=0x7fe4bc0317e0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.800+0000 7fe4dbe23740 1 -- 172.21.15.28:0/1034247622 shutdown_connections 2024-04-24T08:18:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.800+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/1034247622 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x561ba34d6510 0x561ba34d8920 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.800+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/1034247622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34c6260 0x561ba34c3bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.800+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/1034247622 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bbb90 0x561ba34c3690 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.800+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/1034247622 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34bb270 0x561ba34c3150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.800+0000 7fe4dbe23740 1 -- 172.21.15.28:0/1034247622 >> 172.21.15.28:0/1034247622 conn(0x561ba34b7590 msgr2=0x561ba34bd270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.800+0000 7fe4b7e00640 0 client.26611 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.800+0000 7fe4dbe23740 1 -- 172.21.15.28:0/1034247622 shutdown_connections 2024-04-24T08:18:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.800+0000 7fe4dbe23740 1 -- 172.21.15.28:0/1034247622 wait complete. 2024-04-24T08:18:16.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.800+0000 7fe4dbe23740 20 client.26611 trim_cache size 0 max 16384 2024-04-24T08:18:16.821 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.820+0000 7fe4dbe23740 1 Processor -- start 2024-04-24T08:18:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.820+0000 7fe4dbe23740 1 -- start start 2024-04-24T08:18:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.820+0000 7fe4dbe23740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bb270 0x561ba34d1f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.820+0000 7fe4dbe23740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bbb90 0x561ba34b6570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.820+0000 7fe4dbe23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34c6260 0x561ba34de1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.820+0000 7fe4dbe23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d41b6c00 con 0x561ba34c6260 2024-04-24T08:18:16.822 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.820+0000 7fe4dbe23740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d41b44d0 con 0x561ba34bb270 2024-04-24T08:18:16.823 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.820+0000 7fe4dbe23740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d41b2340 con 0x561ba34bbb90 2024-04-24T08:18:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.821+0000 7fe4d1400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bbb90 0x561ba34b6570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.821+0000 7fe4d2800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bb270 0x561ba34d1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.821+0000 7fe4d1400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bbb90 0x561ba34b6570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:50798/0 (socket says 172.21.15.28:50798) 2024-04-24T08:18:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.821+0000 7fe4d1400640 1 -- 172.21.15.28:0/299673380 learned_addr learned my addr 172.21.15.28:0/299673380 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.821+0000 7fe4d2800640 1 --2- 172.21.15.28:0/299673380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34c6260 0x561ba34de1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.821+0000 7fe4d1400640 1 -- 172.21.15.28:0/299673380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bb270 msgr2=0x561ba34d1f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:16.827 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.821+0000 7fe4d1400640 1 --2- 172.21.15.28:0/299673380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bb270 0x561ba34d1f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.821+0000 7fe4d1400640 1 -- 172.21.15.28:0/299673380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34c6260 msgr2=0x561ba34de1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.821+0000 7fe4d1400640 1 --2- 172.21.15.28:0/299673380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34c6260 0x561ba34de1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.821+0000 7fe4d1400640 1 -- 172.21.15.28:0/299673380 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561ba34c2120 con 0x561ba34bbb90 2024-04-24T08:18:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.821+0000 7fe4d2800640 1 --2- 172.21.15.28:0/299673380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34c6260 0x561ba34de1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.821+0000 7fe4d1400640 1 --2- 172.21.15.28:0/299673380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bbb90 0x561ba34b6570 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc004650 tx=0x7fe4bc005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb0281f1b73be781 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.822+0000 7fe4c2a00640 1 -- 172.21.15.28:0/299673380 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc035030 con 0x561ba34bbb90 2024-04-24T08:18:16.828 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.822+0000 7fe4c2a00640 1 -- 172.21.15.28:0/299673380 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4bc003980 con 0x561ba34bbb90 2024-04-24T08:18:16.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.822+0000 7fe4c2a00640 1 -- 172.21.15.28:0/299673380 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc0054f0 con 0x561ba34bbb90 2024-04-24T08:18:16.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.822+0000 7fe4dbe23740 1 -- 172.21.15.28:0/299673380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bbb90 msgr2=0x561ba34b6570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.822+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/299673380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bbb90 0x561ba34b6570 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc004650 tx=0x7fe4bc005fd0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.822+0000 7fe4dbe23740 1 -- 172.21.15.28:0/299673380 shutdown_connections 2024-04-24T08:18:16.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.822+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/299673380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34c6260 0x561ba34de1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.822+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/299673380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bbb90 0x561ba34b6570 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.822+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/299673380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bb270 0x561ba34d1f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.829 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.822+0000 7fe4dbe23740 1 -- 172.21.15.28:0/299673380 >> 172.21.15.28:0/299673380 conn(0x561ba34b7590 msgr2=0x561ba34b5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:16.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.822+0000 7fe4dbe23740 1 -- 172.21.15.28:0/299673380 shutdown_connections 2024-04-24T08:18:16.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.822+0000 7fe4dbe23740 1 -- 172.21.15.28:0/299673380 wait complete. 2024-04-24T08:18:16.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.823+0000 7fe4dbe23740 1 -- start start 2024-04-24T08:18:16.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.823+0000 7fe4dbe23740 1 Processor -- start 2024-04-24T08:18:16.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4dbe23740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bb270 0x561ba34ce450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4dbe23740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bbb90 0x561ba34ce990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4dbe23740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34c6260 0x561ba34cb420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.830 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4dbe23740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d41b2990 con 0x561ba34c6260 2024-04-24T08:18:16.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4dbe23740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d41b3e00 con 0x561ba34bbb90 2024-04-24T08:18:16.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4dbe23740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe4d41b3060 con 0x561ba34bb270 2024-04-24T08:18:16.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4dbe23740 10 client.0 _ll_register_callbacks cb 0x7fe4bc009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:16.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4d1400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bbb90 0x561ba34ce990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4d2800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bb270 0x561ba34ce450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4d1400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bbb90 0x561ba34ce990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40206/0 (socket says 172.21.15.28:40206) 2024-04-24T08:18:16.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4d1400640 1 -- 172.21.15.28:0/2871108298 learned_addr learned my addr 172.21.15.28:0/2871108298 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:16.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4d2800640 1 --2- 172.21.15.28:0/2871108298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34c6260 0x561ba34cb420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4d1400640 1 -- 172.21.15.28:0/2871108298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bb270 msgr2=0x561ba34ce450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:16.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4d1400640 1 --2- 172.21.15.28:0/2871108298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bb270 0x561ba34ce450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4d1400640 1 -- 172.21.15.28:0/2871108298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34c6260 msgr2=0x561ba34cb420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:16.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4d1400640 1 --2- 172.21.15.28:0/2871108298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34c6260 0x561ba34cb420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4d1400640 1 -- 172.21.15.28:0/2871108298 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561ba34cb9e0 con 0x561ba34bbb90 2024-04-24T08:18:16.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.824+0000 7fe4d2800640 1 --2- 172.21.15.28:0/2871108298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34c6260 0x561ba34cb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:16.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.825+0000 7fe4d1400640 1 --2- 172.21.15.28:0/2871108298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bbb90 0x561ba34ce990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc0095e0 tx=0x7fe4bc031b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfe1012e1aa7f4f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:16.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.825+0000 7fe4b7e00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:18:16.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.825+0000 7fe4b7e00640 1 -- 172.21.15.28:0/2871108298 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4bc035050 con 0x561ba34bbb90 2024-04-24T08:18:16.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.825+0000 7fe4dbe23740 1 -- 172.21.15.28:0/2871108298 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4bc003980 con 0x561ba34bbb90 2024-04-24T08:18:16.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.825+0000 7fe4b7e00640 1 -- 172.21.15.28:0/2871108298 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7fe4bc004770 con 0x561ba34bbb90 2024-04-24T08:18:16.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.825+0000 7fe4b7e00640 1 -- 172.21.15.28:0/2871108298 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fe4bc065050 con 0x561ba34bbb90 2024-04-24T08:18:16.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.825+0000 7fe4dbe23740 10 client.26597 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:16.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.825+0000 7fe4dbe23740 1 -- 172.21.15.28:0/2871108298 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x561ba353d370 con 0x561ba34bbb90 2024-04-24T08:18:16.833 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.826+0000 7fe4b7e00640 1 -- 172.21.15.28:0/2871108298 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4bc003980 con 0x561ba34bbb90 2024-04-24T08:18:16.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.826+0000 7fe4b7e00640 1 -- 172.21.15.28:0/2871108298 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fe4bc046480 con 0x561ba34bbb90 2024-04-24T08:18:16.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.826+0000 7fe4dbe23740 10 client.26597 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:16.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.826+0000 7fe4dbe23740 10 client.26597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:18:16.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.826+0000 7fe4dbe23740 20 client.26597 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:16.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.826+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/2871108298 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x561ba353d970 0x561ba353fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:16.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.826+0000 7fe4dbe23740 4 client.26597 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:18:16.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.826+0000 7fe4dbe23740 1 -- 172.21.15.28:0/2871108298 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x561ba3428700 con 0x561ba353d970 2024-04-24T08:18:16.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.826+0000 7fe4d2800640 1 --2- 172.21.15.28:0/2871108298 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x561ba353d970 0x561ba353fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:16.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.827+0000 7fe4b7e00640 10 client.26597 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:16.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.827+0000 7fe4d2800640 1 --2- 172.21.15.28:0/2871108298 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x561ba353d970 0x561ba353fd90 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7571a2d9192fde70 server_cookie=3b5663bc5c030a27 in_seq=0 out_seq=0 2024-04-24T08:18:16.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.828+0000 7fe4b7e00640 1 -- 172.21.15.28:0/2871108298 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x561ba3428700 con 0x561ba353d970 2024-04-24T08:18:16.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.828+0000 7fe4b7e00640 10 client.26597 handle_command_reply: tid=0 2024-04-24T08:18:16.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.828+0000 7fe4dbe23740 1 client.26597 shutdown 2024-04-24T08:18:16.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.828+0000 7fe4dbe23740 1 -- 172.21.15.28:0/2871108298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bbb90 msgr2=0x561ba34ce990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.828+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/2871108298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bbb90 0x561ba34ce990 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7fe4bc0095e0 tx=0x7fe4bc031b40 comp rx=0 tx=0).stop 2024-04-24T08:18:16.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.828+0000 7fe4dbe23740 1 -- 172.21.15.28:0/2871108298 shutdown_connections 2024-04-24T08:18:16.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.828+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/2871108298 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x561ba353d970 0x561ba353fd90 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.828+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/2871108298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x561ba34c6260 0x561ba34cb420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.828+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/2871108298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x561ba34bbb90 0x561ba34ce990 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.828+0000 7fe4dbe23740 1 --2- 172.21.15.28:0/2871108298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x561ba34bb270 0x561ba34ce450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.828+0000 7fe4dbe23740 1 -- 172.21.15.28:0/2871108298 >> 172.21.15.28:0/2871108298 conn(0x561ba34b7590 msgr2=0x561ba34b66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:16.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.828+0000 7fe4b7e00640 0 client.26597 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:16.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.829+0000 7fe4dbe23740 1 -- 172.21.15.28:0/2871108298 shutdown_connections 2024-04-24T08:18:16.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.829+0000 7fe4dbe23740 1 -- 172.21.15.28:0/2871108298 wait complete. 2024-04-24T08:18:16.838 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.829+0000 7fe4dbe23740 20 client.26597 trim_cache size 0 max 16384 2024-04-24T08:18:16.839 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:18:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-24T08:18:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "d28a748c-63a7-41c7-9c29-1871fbc308b4", 2024-04-24T08:18:16.839 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-24T08:18:16.839 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:18:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.830+0000 7fe4d3200640 1 -- 172.21.15.28:0/378561021 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4b80a3d20 msgr2=0x7fe4b80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.830+0000 7fe4d3200640 1 --2- 172.21.15.28:0/378561021 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4b80a3d20 0x7fe4b80a61e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x561ba34bc830 tx=0x7fe4c40d4700 comp rx=0 tx=0).stop 2024-04-24T08:18:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.830+0000 7fe4d3200640 1 -- 172.21.15.28:0/378561021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d4108580 msgr2=0x7fe4d419ed50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:16.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.830+0000 7fe4d3200640 1 --2- 172.21.15.28:0/378561021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d4108580 0x7fe4d419ed50 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe4c400e9c0 tx=0x7fe4c400ee80 comp rx=0 tx=0).stop 2024-04-24T08:18:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.830+0000 7fe4d3200640 1 -- 172.21.15.28:0/378561021 shutdown_connections 2024-04-24T08:18:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.830+0000 7fe4d3200640 1 --2- 172.21.15.28:0/378561021 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe4b80a3d20 0x7fe4b80a61e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.830+0000 7fe4d3200640 1 --2- 172.21.15.28:0/378561021 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe4d410d500 0x7fe4d41a3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.830+0000 7fe4d3200640 1 --2- 172.21.15.28:0/378561021 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4d4108f40 0x7fe4d419f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.830+0000 7fe4d3200640 1 --2- 172.21.15.28:0/378561021 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe4d4108580 0x7fe4d419ed50 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.830+0000 7fe4d3200640 1 -- 172.21.15.28:0/378561021 >> 172.21.15.28:0/378561021 conn(0x7fe4d4077e20 msgr2=0x7fe4d4078360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.830+0000 7fe4d3200640 1 -- 172.21.15.28:0/378561021 shutdown_connections 2024-04-24T08:18:16.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:16.831+0000 7fe4d3200640 1 -- 172.21.15.28:0/378561021 wait complete. 2024-04-24T08:18:16.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:18:16.855 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:18:17.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.059+0000 7ff39ac00640 1 -- 172.21.15.28:0/361376707 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff394076e50 msgr2=0x7ff394112800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.059+0000 7ff39ac00640 1 --2- 172.21.15.28:0/361376707 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff394076e50 0x7ff394112800 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff380009890 tx=0x7ff38002a360 comp rx=0 tx=0).stop 2024-04-24T08:18:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.059+0000 7ff39ac00640 1 -- 172.21.15.28:0/361376707 shutdown_connections 2024-04-24T08:18:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.059+0000 7ff39ac00640 1 --2- 172.21.15.28:0/361376707 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff394112d40 0x7ff394115130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.059+0000 7ff39ac00640 1 --2- 172.21.15.28:0/361376707 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff394076e50 0x7ff394112800 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.059+0000 7ff39ac00640 1 --2- 172.21.15.28:0/361376707 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff394076530 0x7ff394076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.059+0000 7ff39ac00640 1 -- 172.21.15.28:0/361376707 >> 172.21.15.28:0/361376707 conn(0x7ff3940ffb70 msgr2=0x7ff394101f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:17.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.059+0000 7ff39ac00640 1 -- 172.21.15.28:0/361376707 shutdown_connections 2024-04-24T08:18:17.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.059+0000 7ff39ac00640 1 -- 172.21.15.28:0/361376707 wait complete. 2024-04-24T08:18:17.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.060+0000 7ff39ac00640 1 Processor -- start 2024-04-24T08:18:17.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.060+0000 7ff39ac00640 1 -- start start 2024-04-24T08:18:17.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.060+0000 7ff39ac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff394076530 0x7ff39419ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.060+0000 7ff39ac00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff394076e50 0x7ff39419f340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.060+0000 7ff39ac00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff394112d40 0x7ff3941a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.060+0000 7ff39ac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941188a0 con 0x7ff394076530 2024-04-24T08:18:17.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.060+0000 7ff39ac00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff394118720 con 0x7ff394076e50 2024-04-24T08:18:17.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.060+0000 7ff39ac00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff394118a20 con 0x7ff394112d40 2024-04-24T08:18:17.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.061+0000 7ff398e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff394076e50 0x7ff39419f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.061+0000 7ff398e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff394076e50 0x7ff39419f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:40216/0 (socket says 172.21.15.28:40216) 2024-04-24T08:18:17.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.061+0000 7ff398e00640 1 -- 172.21.15.28:0/3324391907 learned_addr learned my addr 172.21.15.28:0/3324391907 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:17.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.061+0000 7ff399800640 1 --2- 172.21.15.28:0/3324391907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff394076530 0x7ff39419ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.061+0000 7ff39a200640 1 --2- 172.21.15.28:0/3324391907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff394112d40 0x7ff3941a36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.061+0000 7ff398e00640 1 -- 172.21.15.28:0/3324391907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff394112d40 msgr2=0x7ff3941a36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.061+0000 7ff398e00640 1 --2- 172.21.15.28:0/3324391907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff394112d40 0x7ff3941a36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.061+0000 7ff398e00640 1 -- 172.21.15.28:0/3324391907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff394076530 msgr2=0x7ff39419ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.061+0000 7ff398e00640 1 --2- 172.21.15.28:0/3324391907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff394076530 0x7ff39419ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.061+0000 7ff398e00640 1 -- 172.21.15.28:0/3324391907 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3941a3df0 con 0x7ff394076e50 2024-04-24T08:18:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.062+0000 7ff398e00640 1 --2- 172.21.15.28:0/3324391907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff394076e50 0x7ff39419f340 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff380033ae0 tx=0x7ff380033b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.062+0000 7ff392000640 1 -- 172.21.15.28:0/3324391907 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff380037040 con 0x7ff394076e50 2024-04-24T08:18:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.062+0000 7ff392000640 1 -- 172.21.15.28:0/3324391907 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff380040070 con 0x7ff394076e50 2024-04-24T08:18:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.062+0000 7ff39ac00640 1 -- 172.21.15.28:0/3324391907 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3941a4080 con 0x7ff394076e50 2024-04-24T08:18:17.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.062+0000 7ff392000640 1 -- 172.21.15.28:0/3324391907 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff380003960 con 0x7ff394076e50 2024-04-24T08:18:17.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.062+0000 7ff39ac00640 1 -- 172.21.15.28:0/3324391907 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3941ab970 con 0x7ff394076e50 2024-04-24T08:18:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.065+0000 7ff39f16b740 1 Processor -- start 2024-04-24T08:18:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.065+0000 7ff392000640 1 -- 172.21.15.28:0/3324391907 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff380030580 con 0x7ff394076e50 2024-04-24T08:18:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.065+0000 7ff39f16b740 1 -- start start 2024-04-24T08:18:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.066+0000 7ff39f16b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf4ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.066+0000 7ff39f16b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf4d350 0x563bdcf574e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.066+0000 7ff39f16b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf57a20 0x563bdcf59e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.066+0000 7ff39f16b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563bdceb21f0 con 0x563bdcf57a20 2024-04-24T08:18:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.066+0000 7ff39f16b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563bdce58350 con 0x563bdcf4d350 2024-04-24T08:18:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.066+0000 7ff39f16b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563bdceae1b0 con 0x563bdcf4ca30 2024-04-24T08:18:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.066+0000 7ff39a200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf4ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.066+0000 7ff39a200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf4ce10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:50846/0 (socket says 172.21.15.28:50846) 2024-04-24T08:18:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.066+0000 7ff39a200640 1 -- 172.21.15.28:0/3177492787 learned_addr learned my addr 172.21.15.28:0/3177492787 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:17.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.066+0000 7ff399800640 1 --2- 172.21.15.28:0/3177492787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf4d350 0x563bdcf574e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.066+0000 7ff398e00640 1 --2- 172.21.15.28:0/3177492787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf57a20 0x563bdcf59e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.066+0000 7ff39a200640 1 -- 172.21.15.28:0/3177492787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf4d350 msgr2=0x563bdcf574e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.066+0000 7ff39a200640 1 --2- 172.21.15.28:0/3177492787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf4d350 0x563bdcf574e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.066+0000 7ff39a200640 1 -- 172.21.15.28:0/3177492787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf57a20 msgr2=0x563bdcf59e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.066+0000 7ff39a200640 1 --2- 172.21.15.28:0/3177492787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf57a20 0x563bdcf59e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.066+0000 7ff39a200640 1 -- 172.21.15.28:0/3177492787 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bdceacf50 con 0x563bdcf4ca30 2024-04-24T08:18:17.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.067+0000 7ff39a200640 1 --2- 172.21.15.28:0/3177492787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf4ce10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff38c009f00 tx=0x7ff38c0107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a4ab8d47e2e52344 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.067+0000 7ff392000640 1 --2- 172.21.15.28:0/3324391907 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff37c0a3da0 0x7ff37c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.067+0000 7ff386a00640 1 -- 172.21.15.28:0/3177492787 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c003b70 con 0x563bdcf4ca30 2024-04-24T08:18:17.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.067+0000 7ff386a00640 1 -- 172.21.15.28:0/3177492787 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff38c003d10 con 0x563bdcf4ca30 2024-04-24T08:18:17.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.067+0000 7ff386a00640 1 -- 172.21.15.28:0/3177492787 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c011a40 con 0x563bdcf4ca30 2024-04-24T08:18:17.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.067+0000 7ff392000640 1 -- 172.21.15.28:0/3324391907 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7ff3800f0820 con 0x7ff394076e50 2024-04-24T08:18:17.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.067+0000 7ff399800640 1 --2- 172.21.15.28:0/3324391907 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff37c0a3da0 0x7ff37c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.068+0000 7ff39f16b740 1 -- 172.21.15.28:0/3177492787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 msgr2=0x563bdcf4ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.068+0000 7ff39f16b740 1 --2- 172.21.15.28:0/3177492787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf4ce10 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7ff38c009f00 tx=0x7ff38c0107a0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.068+0000 7ff39f16b740 1 -- 172.21.15.28:0/3177492787 shutdown_connections 2024-04-24T08:18:17.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.068+0000 7ff39f16b740 1 --2- 172.21.15.28:0/3177492787 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf57a20 0x563bdcf59e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.068+0000 7ff39f16b740 1 --2- 172.21.15.28:0/3177492787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf4d350 0x563bdcf574e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.068+0000 7ff39f16b740 1 --2- 172.21.15.28:0/3177492787 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf4ce10 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.068+0000 7ff39f16b740 1 -- 172.21.15.28:0/3177492787 >> 172.21.15.28:0/3177492787 conn(0x563bdcf48d50 msgr2=0x563bdcf49130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:17.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.068+0000 7ff39f16b740 1 -- 172.21.15.28:0/3177492787 shutdown_connections 2024-04-24T08:18:17.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.068+0000 7ff39f16b740 1 -- 172.21.15.28:0/3177492787 wait complete. 2024-04-24T08:18:17.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.068+0000 7ff399800640 1 --2- 172.21.15.28:0/3324391907 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff37c0a3da0 0x7ff37c0a6260 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x563bdcf4dff0 tx=0x7ff388009260 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.069+0000 7ff39f16b740 1 -- start start 2024-04-24T08:18:17.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.069+0000 7ff39f16b740 1 Processor -- start 2024-04-24T08:18:17.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.069+0000 7ff39f16b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf4ca30 0x563bdcf549d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.069+0000 7ff39f16b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4d350 0x563bdcf54f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.069+0000 7ff39f16b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf57a20 0x563bdcf55450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.069+0000 7ff39f16b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563bdcf5a810 con 0x563bdcf4ca30 2024-04-24T08:18:17.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.069+0000 7ff39f16b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563bdce6b690 con 0x563bdcf57a20 2024-04-24T08:18:17.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.069+0000 7ff39f16b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563bdcf5a990 con 0x563bdcf4d350 2024-04-24T08:18:17.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.069+0000 7ff39f16b740 10 client.0 _ll_register_callbacks cb 0x563bdcea7650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:17.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.070+0000 7ff398e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4d350 0x563bdcf54f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.070+0000 7ff39a200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf57a20 0x563bdcf55450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.070+0000 7ff399800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf4ca30 0x563bdcf549d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.070+0000 7ff398e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4d350 0x563bdcf54f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:50856/0 (socket says 172.21.15.28:50856) 2024-04-24T08:18:17.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.070+0000 7ff398e00640 1 -- 172.21.15.28:0/3253554941 learned_addr learned my addr 172.21.15.28:0/3253554941 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:17.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.070+0000 7ff398e00640 1 -- 172.21.15.28:0/3253554941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf57a20 msgr2=0x563bdcf55450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.070+0000 7ff398e00640 1 --2- 172.21.15.28:0/3253554941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf57a20 0x563bdcf55450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.070+0000 7ff398e00640 1 -- 172.21.15.28:0/3253554941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf4ca30 msgr2=0x563bdcf549d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.070+0000 7ff398e00640 1 --2- 172.21.15.28:0/3253554941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf4ca30 0x563bdcf549d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.070+0000 7ff398e00640 1 -- 172.21.15.28:0/3253554941 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bdce6a920 con 0x563bdcf4d350 2024-04-24T08:18:17.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.070+0000 7ff39a200640 1 --2- 172.21.15.28:0/3253554941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf57a20 0x563bdcf55450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:17.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.070+0000 7ff398e00640 1 --2- 172.21.15.28:0/3253554941 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4d350 0x563bdcf54f10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff380030520 tx=0x7ff38002a360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d51d43ff14448fcd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.070+0000 7ff37b400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:18:17.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.070+0000 7ff37b400640 1 -- 172.21.15.28:0/3253554941 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3800bc7f0 con 0x563bdcf4d350 2024-04-24T08:18:17.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.070+0000 7ff37b400640 1 -- 172.21.15.28:0/3253554941 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7ff3800ee110 con 0x563bdcf4d350 2024-04-24T08:18:17.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.070+0000 7ff39f16b740 1 -- 172.21.15.28:0/3253554941 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff38c003d10 con 0x563bdcf4d350 2024-04-24T08:18:17.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.071+0000 7ff37b400640 1 -- 172.21.15.28:0/3253554941 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff3800ee310 con 0x563bdcf4d350 2024-04-24T08:18:17.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.071+0000 7ff39f16b740 10 client.26629 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:17.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.071+0000 7ff39f16b740 1 -- 172.21.15.28:0/3253554941 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563bdcc8e7d0 con 0x563bdcf4d350 2024-04-24T08:18:17.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.071+0000 7ff37b400640 1 -- 172.21.15.28:0/3253554941 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff38c003d10 con 0x563bdcf4d350 2024-04-24T08:18:17.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.071+0000 7ff37b400640 1 -- 172.21.15.28:0/3253554941 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff38003a070 con 0x563bdcf4d350 2024-04-24T08:18:17.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.071+0000 7ff39f16b740 10 client.26629 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:17.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.071+0000 7ff39f16b740 10 client.26629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:18:17.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.071+0000 7ff39f16b740 20 client.26629 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:17.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.071+0000 7ff39f16b740 1 --2- 172.21.15.28:0/3253554941 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563bdcf67dc0 0x563bdcf6a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.071+0000 7ff39f16b740 4 client.26629 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:18:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.071+0000 7ff39f16b740 1 -- 172.21.15.28:0/3253554941 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563bdcebfc10 con 0x563bdcf67dc0 2024-04-24T08:18:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.072+0000 7ff399800640 1 --2- 172.21.15.28:0/3253554941 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563bdcf67dc0 0x563bdcf6a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.072+0000 7ff399800640 1 --2- 172.21.15.28:0/3253554941 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563bdcf67dc0 0x563bdcf6a220 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2032da45549c6f6 server_cookie=e08a4e0c2dee6bed in_seq=0 out_seq=0 2024-04-24T08:18:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.072+0000 7ff37b400640 10 client.26629 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.073+0000 7ff37b400640 1 -- 172.21.15.28:0/3253554941 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563bdcebfc10 con 0x563bdcf67dc0 2024-04-24T08:18:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.073+0000 7ff37b400640 10 client.26629 handle_command_reply: tid=0 2024-04-24T08:18:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.073+0000 7ff39f16b740 1 client.26629 shutdown 2024-04-24T08:18:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.073+0000 7ff39f16b740 1 -- 172.21.15.28:0/3253554941 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4d350 msgr2=0x563bdcf54f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.073+0000 7ff39f16b740 1 --2- 172.21.15.28:0/3253554941 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4d350 0x563bdcf54f10 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7ff380030520 tx=0x7ff38002a360 comp rx=0 tx=0).stop 2024-04-24T08:18:17.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.073+0000 7ff39f16b740 1 -- 172.21.15.28:0/3253554941 shutdown_connections 2024-04-24T08:18:17.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.073+0000 7ff39f16b740 1 --2- 172.21.15.28:0/3253554941 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563bdcf67dc0 0x563bdcf6a220 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.073+0000 7ff39f16b740 1 --2- 172.21.15.28:0/3253554941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf57a20 0x563bdcf55450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.073+0000 7ff39f16b740 1 --2- 172.21.15.28:0/3253554941 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4d350 0x563bdcf54f10 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.073+0000 7ff37b400640 0 client.26629 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:17.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.073+0000 7ff39f16b740 1 --2- 172.21.15.28:0/3253554941 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf4ca30 0x563bdcf549d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.073+0000 7ff39f16b740 1 -- 172.21.15.28:0/3253554941 >> 172.21.15.28:0/3253554941 conn(0x563bdcf48d50 msgr2=0x563bdcf4eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:17.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.074+0000 7ff39f16b740 1 -- 172.21.15.28:0/3253554941 shutdown_connections 2024-04-24T08:18:17.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.074+0000 7ff39f16b740 1 -- 172.21.15.28:0/3253554941 wait complete. 2024-04-24T08:18:17.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.074+0000 7ff39f16b740 20 client.26629 trim_cache size 0 max 16384 2024-04-24T08:18:17.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.093+0000 7ff39f16b740 1 Processor -- start 2024-04-24T08:18:17.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.093+0000 7ff39f16b740 1 -- start start 2024-04-24T08:18:17.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.094+0000 7ff39f16b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf63730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.094+0000 7ff39f16b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf4d350 0x563bdcf6fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.096 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.094+0000 7ff39f16b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf57a20 0x563bdcf70070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.094+0000 7ff39f16b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941b6bf0 con 0x563bdcf57a20 2024-04-24T08:18:17.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.094+0000 7ff39f16b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941b44c0 con 0x563bdcf4d350 2024-04-24T08:18:17.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.094+0000 7ff39f16b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941b2330 con 0x563bdcf4ca30 2024-04-24T08:18:17.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.094+0000 7ff399800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf63730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.094+0000 7ff399800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf63730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:50858/0 (socket says 172.21.15.28:50858) 2024-04-24T08:18:17.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.094+0000 7ff399800640 1 -- 172.21.15.28:0/299504154 learned_addr learned my addr 172.21.15.28:0/299504154 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:17.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.094+0000 7ff39a200640 1 --2- 172.21.15.28:0/299504154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf57a20 0x563bdcf70070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.094+0000 7ff398e00640 1 --2- 172.21.15.28:0/299504154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf4d350 0x563bdcf6fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.095+0000 7ff399800640 1 -- 172.21.15.28:0/299504154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf4d350 msgr2=0x563bdcf6fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.095+0000 7ff399800640 1 --2- 172.21.15.28:0/299504154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf4d350 0x563bdcf6fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.095+0000 7ff399800640 1 -- 172.21.15.28:0/299504154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf57a20 msgr2=0x563bdcf70070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.095+0000 7ff399800640 1 --2- 172.21.15.28:0/299504154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf57a20 0x563bdcf70070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.095+0000 7ff399800640 1 -- 172.21.15.28:0/299504154 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bdcf4d180 con 0x563bdcf4ca30 2024-04-24T08:18:17.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.095+0000 7ff398e00640 1 --2- 172.21.15.28:0/299504154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf4d350 0x563bdcf6fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.095+0000 7ff399800640 1 --2- 172.21.15.28:0/299504154 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf63730 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff388009be0 tx=0x7ff388017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9152f480751acf41 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.095+0000 7ff386000640 1 -- 172.21.15.28:0/299504154 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff388032070 con 0x563bdcf4ca30 2024-04-24T08:18:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.095+0000 7ff386000640 1 -- 172.21.15.28:0/299504154 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3880043f0 con 0x563bdcf4ca30 2024-04-24T08:18:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.095+0000 7ff386000640 1 -- 172.21.15.28:0/299504154 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38802b9b0 con 0x563bdcf4ca30 2024-04-24T08:18:17.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.096+0000 7ff39f16b740 1 -- 172.21.15.28:0/299504154 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 msgr2=0x563bdcf63730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.096+0000 7ff39f16b740 1 --2- 172.21.15.28:0/299504154 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf63730 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7ff388009be0 tx=0x7ff388017000 comp rx=0 tx=0).stop 2024-04-24T08:18:17.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.096+0000 7ff39f16b740 1 -- 172.21.15.28:0/299504154 shutdown_connections 2024-04-24T08:18:17.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.096+0000 7ff39f16b740 1 --2- 172.21.15.28:0/299504154 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf57a20 0x563bdcf70070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.096+0000 7ff39f16b740 1 --2- 172.21.15.28:0/299504154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf4d350 0x563bdcf6fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.096+0000 7ff39f16b740 1 --2- 172.21.15.28:0/299504154 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf63730 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.096+0000 7ff39f16b740 1 -- 172.21.15.28:0/299504154 >> 172.21.15.28:0/299504154 conn(0x563bdcf48d50 msgr2=0x563bdcf49130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:17.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.096+0000 7ff39f16b740 1 -- 172.21.15.28:0/299504154 shutdown_connections 2024-04-24T08:18:17.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.096+0000 7ff39f16b740 1 -- 172.21.15.28:0/299504154 wait complete. 2024-04-24T08:18:17.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.097+0000 7ff39f16b740 1 -- start start 2024-04-24T08:18:17.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.097+0000 7ff39f16b740 1 Processor -- start 2024-04-24T08:18:17.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.097+0000 7ff39f16b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf5fea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.097+0000 7ff39f16b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf4d350 0x563bdcf51280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.097+0000 7ff39f16b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf57a20 0x563bdcf517c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.097+0000 7ff39f16b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941b2980 con 0x563bdcf57a20 2024-04-24T08:18:17.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.097+0000 7ff39f16b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941b3df0 con 0x563bdcf4d350 2024-04-24T08:18:17.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.097+0000 7ff39f16b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff3941b3050 con 0x563bdcf4ca30 2024-04-24T08:18:17.105 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.097+0000 7ff39f16b740 10 client.0 _ll_register_callbacks cb 0x7ff380005ab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:17.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.098+0000 7ff399800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf5fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.098+0000 7ff39a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf57a20 0x563bdcf517c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.098+0000 7ff399800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf5fea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:50862/0 (socket says 172.21.15.28:50862) 2024-04-24T08:18:17.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.098+0000 7ff399800640 1 -- 172.21.15.28:0/581345745 learned_addr learned my addr 172.21.15.28:0/581345745 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:17.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.098+0000 7ff398e00640 1 --2- 172.21.15.28:0/581345745 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf4d350 0x563bdcf51280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.098+0000 7ff39a200640 1 -- 172.21.15.28:0/581345745 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 msgr2=0x563bdcf5fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.098+0000 7ff39a200640 1 --2- 172.21.15.28:0/581345745 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf5fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.098+0000 7ff39a200640 1 -- 172.21.15.28:0/581345745 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf4d350 msgr2=0x563bdcf51280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:17.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.098+0000 7ff39a200640 1 --2- 172.21.15.28:0/581345745 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf4d350 0x563bdcf51280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.098+0000 7ff39a200640 1 -- 172.21.15.28:0/581345745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bdcbc43f0 con 0x563bdcf57a20 2024-04-24T08:18:17.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.098+0000 7ff399800640 1 --2- 172.21.15.28:0/581345745 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf5fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:17.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.098+0000 7ff39a200640 1 --2- 172.21.15.28:0/581345745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf57a20 0x563bdcf517c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff38c009ed0 tx=0x7ff38c012b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c61eb63d7c2581fd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:17.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.098+0000 7ff37b400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:18:17.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.099+0000 7ff37b400640 1 -- 172.21.15.28:0/581345745 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff38c02d060 con 0x563bdcf57a20 2024-04-24T08:18:17.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.099+0000 7ff37b400640 1 -- 172.21.15.28:0/581345745 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 8644+0+0 (secure 0 0 0) 0x7ff38c0052b0 con 0x563bdcf57a20 2024-04-24T08:18:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.099+0000 7ff39f16b740 1 -- 172.21.15.28:0/581345745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff3880043f0 con 0x563bdcf57a20 2024-04-24T08:18:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.099+0000 7ff37b400640 1 -- 172.21.15.28:0/581345745 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff38c0054b0 con 0x563bdcf57a20 2024-04-24T08:18:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.099+0000 7ff39f16b740 10 client.16335 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.099+0000 7ff39f16b740 1 -- 172.21.15.28:0/581345745 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563bdcb840f0 con 0x563bdcf57a20 2024-04-24T08:18:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.099+0000 7ff37b400640 1 -- 172.21.15.28:0/581345745 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3880043f0 con 0x563bdcf57a20 2024-04-24T08:18:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.099+0000 7ff37b400640 1 -- 172.21.15.28:0/581345745 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff38c01bcd0 con 0x563bdcf57a20 2024-04-24T08:18:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.100+0000 7ff39f16b740 10 client.16335 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:17.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.100+0000 7ff39f16b740 10 client.16335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:18:17.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.100+0000 7ff39f16b740 20 client.16335 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:17.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.100+0000 7ff39f16b740 1 --2- 172.21.15.28:0/581345745 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563bdcfce290 0x563bdcfd0680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:17.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.100+0000 7ff39f16b740 4 client.16335 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:18:17.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.100+0000 7ff39f16b740 1 -- 172.21.15.28:0/581345745 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x563bdceb9a80 con 0x563bdcfce290 2024-04-24T08:18:17.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.100+0000 7ff399800640 1 --2- 172.21.15.28:0/581345745 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563bdcfce290 0x563bdcfd0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:17.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.100+0000 7ff399800640 1 --2- 172.21.15.28:0/581345745 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563bdcfce290 0x563bdcfd0680 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37f15ce32e4e00f2 server_cookie=c9dc5e4f9df95107 in_seq=0 out_seq=0 2024-04-24T08:18:17.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.100+0000 7ff37b400640 10 client.16335 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:17.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.101+0000 7ff37b400640 1 -- 172.21.15.28:0/581345745 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x563bdceb9a80 con 0x563bdcfce290 2024-04-24T08:18:17.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.101+0000 7ff37b400640 10 client.16335 handle_command_reply: tid=0 2024-04-24T08:18:17.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.101+0000 7ff39f16b740 1 client.16335 shutdown 2024-04-24T08:18:17.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.101+0000 7ff39f16b740 1 -- 172.21.15.28:0/581345745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf57a20 msgr2=0x563bdcf517c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.101+0000 7ff39f16b740 1 --2- 172.21.15.28:0/581345745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf57a20 0x563bdcf517c0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ff38c009ed0 tx=0x7ff38c012b80 comp rx=0 tx=0).stop 2024-04-24T08:18:17.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.101+0000 7ff39f16b740 1 -- 172.21.15.28:0/581345745 shutdown_connections 2024-04-24T08:18:17.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.101+0000 7ff39f16b740 1 --2- 172.21.15.28:0/581345745 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x563bdcfce290 0x563bdcfd0680 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.101+0000 7ff39f16b740 1 --2- 172.21.15.28:0/581345745 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563bdcf57a20 0x563bdcf517c0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.101+0000 7ff39f16b740 1 --2- 172.21.15.28:0/581345745 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563bdcf4d350 0x563bdcf51280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.101+0000 7ff37b400640 0 client.16335 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:17.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.101+0000 7ff39f16b740 1 --2- 172.21.15.28:0/581345745 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563bdcf4ca30 0x563bdcf5fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.101+0000 7ff39f16b740 1 -- 172.21.15.28:0/581345745 >> 172.21.15.28:0/581345745 conn(0x563bdcf48d50 msgr2=0x563bdcf4e220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:17.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.101+0000 7ff39f16b740 1 -- 172.21.15.28:0/581345745 shutdown_connections 2024-04-24T08:18:17.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.101+0000 7ff39f16b740 1 -- 172.21.15.28:0/581345745 wait complete. 2024-04-24T08:18:17.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.101+0000 7ff39f16b740 20 client.16335 trim_cache size 0 max 16384 2024-04-24T08:18:17.111 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:18:17.112 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:18:17.112 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:18:17.112 INFO:teuthology.orchestra.run.smithi028.stdout: "d28a748c-63a7-41c7-9c29-1871fbc308b4": { 2024-04-24T08:18:17.112 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:18:17.112 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "d28a748c-63a7-41c7-9c29-1871fbc308b4", 2024-04-24T08:18:17.112 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:18:17.112 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:18:17.112 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:18:17.112 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:18:17.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.102+0000 7ff39ac00640 1 -- 172.21.15.28:0/3324391907 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff37c0a3da0 msgr2=0x7ff37c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.102+0000 7ff39ac00640 1 --2- 172.21.15.28:0/3324391907 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff37c0a3da0 0x7ff37c0a6260 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x563bdcf4dff0 tx=0x7ff388009260 comp rx=0 tx=0).stop 2024-04-24T08:18:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.102+0000 7ff39ac00640 1 -- 172.21.15.28:0/3324391907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff394076e50 msgr2=0x7ff39419f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.102+0000 7ff39ac00640 1 --2- 172.21.15.28:0/3324391907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff394076e50 0x7ff39419f340 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7ff380033ae0 tx=0x7ff380033b10 comp rx=0 tx=0).stop 2024-04-24T08:18:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.102+0000 7ff39ac00640 1 -- 172.21.15.28:0/3324391907 shutdown_connections 2024-04-24T08:18:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.102+0000 7ff39ac00640 1 --2- 172.21.15.28:0/3324391907 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff37c0a3da0 0x7ff37c0a6260 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.102+0000 7ff39ac00640 1 --2- 172.21.15.28:0/3324391907 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff394112d40 0x7ff3941a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.102+0000 7ff39ac00640 1 --2- 172.21.15.28:0/3324391907 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff394076e50 0x7ff39419f340 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.102+0000 7ff39ac00640 1 --2- 172.21.15.28:0/3324391907 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff394076530 0x7ff39419ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.102+0000 7ff39ac00640 1 -- 172.21.15.28:0/3324391907 >> 172.21.15.28:0/3324391907 conn(0x7ff3940ffb70 msgr2=0x7ff394101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:17.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.102+0000 7ff39ac00640 1 -- 172.21.15.28:0/3324391907 shutdown_connections 2024-04-24T08:18:17.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:17.102+0000 7ff39ac00640 1 -- 172.21.15.28:0/3324391907 wait complete. 2024-04-24T08:18:17.121 INFO:tasks.cephfs.filesystem:scrub status for tag:d28a748c-63a7-41c7-9c29-1871fbc308b4 - {'path': '/', 'tag': 'd28a748c-63a7-41c7-9c29-1871fbc308b4', 'options': 'recursive,force'} 2024-04-24T08:18:17.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:17 smithi194 ceph-mon[32836]: pgmap v244: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 8 op/s 2024-04-24T08:18:17.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:17 smithi194 ceph-mon[32836]: scrub queued for path: / 2024-04-24T08:18:17.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:17 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:17.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:17 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:18:17.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:17 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:18:17.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:17 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:17.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:17 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:18:17.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:17 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:17.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:17 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:18:17.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:17 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:17.942 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:17 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:18.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:17 smithi028 ceph-mon[27786]: pgmap v244: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 8 op/s 2024-04-24T08:18:18.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:17 smithi028 ceph-mon[27786]: scrub queued for path: / 2024-04-24T08:18:18.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:17 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:18.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:17 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:18:18.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:17 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:18:18.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:17 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:18.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:17 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:18:18.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:17 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:18.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:17 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:18:18.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:17 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:18.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:17 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:18.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:17 smithi117 ceph-mon[32438]: pgmap v244: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 66 KiB/s rd, 64 KiB/s wr, 8 op/s 2024-04-24T08:18:18.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:17 smithi117 ceph-mon[32438]: scrub queued for path: / 2024-04-24T08:18:18.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:17 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:18.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:17 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:18:18.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:17 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:18:18.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:17 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:18.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:17 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:18:18.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:17 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:18.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:17 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:18:18.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:17 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:18.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:17 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:19.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:19 smithi194 ceph-mon[32836]: pgmap v245: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 7.4 KiB/s wr, 7 op/s 2024-04-24T08:18:19.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:19 smithi194 ceph-mon[32836]: osdmap e95: 12 total, 12 up, 12 in 2024-04-24T08:18:20.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:19 smithi028 ceph-mon[27786]: pgmap v245: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 7.4 KiB/s wr, 7 op/s 2024-04-24T08:18:20.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:19 smithi028 ceph-mon[27786]: osdmap e95: 12 total, 12 up, 12 in 2024-04-24T08:18:20.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:19 smithi117 ceph-mon[32438]: pgmap v245: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 7.4 KiB/s wr, 7 op/s 2024-04-24T08:18:20.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:19 smithi117 ceph-mon[32438]: osdmap e95: 12 total, 12 up, 12 in 2024-04-24T08:18:21.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:21 smithi194 ceph-mon[32836]: pgmap v247: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2024-04-24T08:18:22.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:21 smithi028 ceph-mon[27786]: pgmap v247: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2024-04-24T08:18:22.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:21 smithi117 ceph-mon[32438]: pgmap v247: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 10 KiB/s rd, 7.3 KiB/s wr, 7 op/s 2024-04-24T08:18:23.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:23 smithi194 ceph-mon[32836]: pgmap v248: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.8 KiB/s wr, 6 op/s 2024-04-24T08:18:24.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:23 smithi028 ceph-mon[27786]: pgmap v248: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.8 KiB/s wr, 6 op/s 2024-04-24T08:18:24.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:23 smithi117 ceph-mon[32438]: pgmap v248: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.8 KiB/s wr, 6 op/s 2024-04-24T08:18:25.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:24 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:18:25.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:24 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:18:25.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:24 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:18:25.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:24 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:18:25.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:24 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:18:25.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:24 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:18:26.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:25 smithi028 ceph-mon[27786]: pgmap v249: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.8 KiB/s wr, 6 op/s 2024-04-24T08:18:26.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:25 smithi117 ceph-mon[32438]: pgmap v249: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.8 KiB/s wr, 6 op/s 2024-04-24T08:18:26.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:25 smithi194 ceph-mon[32836]: pgmap v249: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.8 KiB/s wr, 6 op/s 2024-04-24T08:18:28.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:27 smithi028 ceph-mon[27786]: pgmap v250: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.8 KiB/s wr, 6 op/s 2024-04-24T08:18:28.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:27 smithi117 ceph-mon[32438]: pgmap v250: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.8 KiB/s wr, 6 op/s 2024-04-24T08:18:28.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:27 smithi194 ceph-mon[32836]: pgmap v250: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 7.9 KiB/s rd, 4.8 KiB/s wr, 6 op/s 2024-04-24T08:18:30.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:29 smithi028 ceph-mon[27786]: pgmap v251: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.5 KiB/s wr, 6 op/s 2024-04-24T08:18:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:29 smithi117 ceph-mon[32438]: pgmap v251: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.5 KiB/s wr, 6 op/s 2024-04-24T08:18:30.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:29 smithi194 ceph-mon[32836]: pgmap v251: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 7.6 KiB/s rd, 4.5 KiB/s wr, 6 op/s 2024-04-24T08:18:31.993 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:31 smithi194 ceph-mon[32836]: pgmap v252: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.0 KiB/s wr, 6 op/s 2024-04-24T08:18:32.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:31 smithi028 ceph-mon[27786]: pgmap v252: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.0 KiB/s wr, 6 op/s 2024-04-24T08:18:32.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:31 smithi117 ceph-mon[32438]: pgmap v252: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.7 KiB/s rd, 4.0 KiB/s wr, 6 op/s 2024-04-24T08:18:34.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:33 smithi028 ceph-mon[27786]: pgmap v253: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 7.1 KiB/s wr, 6 op/s 2024-04-24T08:18:34.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:33 smithi117 ceph-mon[32438]: pgmap v253: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 7.1 KiB/s wr, 6 op/s 2024-04-24T08:18:34.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:33 smithi194 ceph-mon[32836]: pgmap v253: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 9.9 KiB/s rd, 7.1 KiB/s wr, 6 op/s 2024-04-24T08:18:36.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:35 smithi028 ceph-mon[27786]: pgmap v254: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-24T08:18:36.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:35 smithi117 ceph-mon[32438]: pgmap v254: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-24T08:18:36.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:35 smithi194 ceph-mon[32836]: pgmap v254: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-24T08:18:38.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:37 smithi028 ceph-mon[27786]: pgmap v255: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.4 KiB/s wr, 5 op/s 2024-04-24T08:18:38.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:37 smithi117 ceph-mon[32438]: pgmap v255: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.4 KiB/s wr, 5 op/s 2024-04-24T08:18:38.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:37 smithi194 ceph-mon[32836]: pgmap v255: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 5.7 KiB/s rd, 3.4 KiB/s wr, 5 op/s 2024-04-24T08:18:40.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:39 smithi028 ceph-mon[27786]: pgmap v256: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-24T08:18:40.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:39 smithi117 ceph-mon[32438]: pgmap v256: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-24T08:18:40.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:39 smithi194 ceph-mon[32836]: pgmap v256: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.2 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-24T08:18:42.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:41 smithi028 ceph-mon[27786]: pgmap v257: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-24T08:18:42.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:41 smithi117 ceph-mon[32438]: pgmap v257: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-24T08:18:42.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:41 smithi194 ceph-mon[32836]: pgmap v257: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.6 KiB/s rd, 3.4 KiB/s wr, 6 op/s 2024-04-24T08:18:44.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:43 smithi028 ceph-mon[27786]: pgmap v258: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.4 KiB/s wr, 6 op/s 2024-04-24T08:18:44.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:43 smithi117 ceph-mon[32438]: pgmap v258: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.4 KiB/s wr, 6 op/s 2024-04-24T08:18:44.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:43 smithi194 ceph-mon[32836]: pgmap v258: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 9.2 KiB/s rd, 6.4 KiB/s wr, 6 op/s 2024-04-24T08:18:45.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:45 smithi028 ceph-mon[27786]: pgmap v259: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2024-04-24T08:18:45.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:45 smithi117 ceph-mon[32438]: pgmap v259: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2024-04-24T08:18:45.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:45 smithi194 ceph-mon[32836]: pgmap v259: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 3.1 KiB/s wr, 6 op/s 2024-04-24T08:18:47.122 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:18:47.123 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:18:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.345+0000 7f4286c00640 1 -- 172.21.15.28:0/2503178961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4280076530 msgr2=0x7f4280076910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.345+0000 7f4286c00640 1 --2- 172.21.15.28:0/2503178961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4280076530 0x7f4280076910 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f4270009ca0 tx=0x7f427002a8b0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.345+0000 7f4286c00640 1 -- 172.21.15.28:0/2503178961 shutdown_connections 2024-04-24T08:18:47.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.345+0000 7f4286c00640 1 --2- 172.21.15.28:0/2503178961 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4280112ee0 0x7f42801152d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.345+0000 7f4286c00640 1 --2- 172.21.15.28:0/2503178961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4280076e50 0x7f42801129a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.345+0000 7f4286c00640 1 --2- 172.21.15.28:0/2503178961 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4280076530 0x7f4280076910 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.345+0000 7f4286c00640 1 -- 172.21.15.28:0/2503178961 >> 172.21.15.28:0/2503178961 conn(0x7f42800ffd90 msgr2=0x7f42801021b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:47.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.346+0000 7f4286c00640 1 -- 172.21.15.28:0/2503178961 shutdown_connections 2024-04-24T08:18:47.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.346+0000 7f4286c00640 1 -- 172.21.15.28:0/2503178961 wait complete. 2024-04-24T08:18:47.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.347+0000 7f4286c00640 1 Processor -- start 2024-04-24T08:18:47.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.347+0000 7f4286c00640 1 -- start start 2024-04-24T08:18:47.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.348+0000 7f4286c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4280076530 0x7f428019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.348+0000 7f4286c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4280076e50 0x7f428019f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.351 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.348+0000 7f4286c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4280112ee0 0x7f42801a3820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.348+0000 7f4286c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4280118a90 con 0x7f4280112ee0 2024-04-24T08:18:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.348+0000 7f4286c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4280118910 con 0x7f4280076530 2024-04-24T08:18:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.348+0000 7f4286c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4280118c10 con 0x7f4280076e50 2024-04-24T08:18:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.348+0000 7f427fe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4280076530 0x7f428019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.348+0000 7f427fe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4280076530 0x7f428019efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41652/0 (socket says 172.21.15.28:41652) 2024-04-24T08:18:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.348+0000 7f427fe00640 1 -- 172.21.15.28:0/1679719394 learned_addr learned my addr 172.21.15.28:0/1679719394 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.348+0000 7f4284c00640 1 --2- 172.21.15.28:0/1679719394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4280112ee0 0x7f42801a3820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.352 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.348+0000 7f427f400640 1 --2- 172.21.15.28:0/1679719394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4280076e50 0x7f428019f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.348+0000 7f427fe00640 1 -- 172.21.15.28:0/1679719394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4280076e50 msgr2=0x7f428019f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.348+0000 7f427fe00640 1 --2- 172.21.15.28:0/1679719394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4280076e50 0x7f428019f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.348+0000 7f427fe00640 1 -- 172.21.15.28:0/1679719394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4280112ee0 msgr2=0x7f42801a3820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.348+0000 7f427fe00640 1 --2- 172.21.15.28:0/1679719394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4280112ee0 0x7f42801a3820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.348+0000 7f427fe00640 1 -- 172.21.15.28:0/1679719394 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f42801a3f50 con 0x7f4280076530 2024-04-24T08:18:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.348+0000 7f427f400640 1 --2- 172.21.15.28:0/1679719394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4280076e50 0x7f428019f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.349+0000 7f427fe00640 1 --2- 172.21.15.28:0/1679719394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4280076530 0x7f428019efb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4270005e30 tx=0x7f4270030920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:47.353 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.349+0000 7f427cc00640 1 -- 172.21.15.28:0/1679719394 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4270004040 con 0x7f4280076530 2024-04-24T08:18:47.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.349+0000 7f427cc00640 1 -- 172.21.15.28:0/1679719394 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4270040070 con 0x7f4280076530 2024-04-24T08:18:47.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.349+0000 7f4286c00640 1 -- 172.21.15.28:0/1679719394 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f42801a41e0 con 0x7f4280076530 2024-04-24T08:18:47.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.349+0000 7f427cc00640 1 -- 172.21.15.28:0/1679719394 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4270045b80 con 0x7f4280076530 2024-04-24T08:18:47.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.350+0000 7f4286c00640 1 -- 172.21.15.28:0/1679719394 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f42801abad0 con 0x7f4280076530 2024-04-24T08:18:47.355 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.353+0000 7f428997d740 1 Processor -- start 2024-04-24T08:18:47.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.353+0000 7f428997d740 1 -- start start 2024-04-24T08:18:47.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.353+0000 7f428997d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf887860 0x5591cf887c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.353+0000 7f428997d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf888180 0x5591cf892310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.353+0000 7f427cc00640 1 -- 172.21.15.28:0/1679719394 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f42700041e0 con 0x7f4280076530 2024-04-24T08:18:47.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.353+0000 7f428997d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf892850 0x5591cf894c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.353+0000 7f428997d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5591cf7ed1f0 con 0x5591cf887860 2024-04-24T08:18:47.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.353+0000 7f428997d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5591cf793350 con 0x5591cf892850 2024-04-24T08:18:47.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.353+0000 7f428997d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5591cf7e91b0 con 0x5591cf888180 2024-04-24T08:18:47.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.354+0000 7f4284c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf887860 0x5591cf887c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.354+0000 7f4284c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf887860 0x5591cf887c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54640/0 (socket says 172.21.15.28:54640) 2024-04-24T08:18:47.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.354+0000 7f4284c00640 1 -- 172.21.15.28:0/944023808 learned_addr learned my addr 172.21.15.28:0/944023808 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:47.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.354+0000 7f427f400640 1 --2- 172.21.15.28:0/944023808 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf892850 0x5591cf894c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.354+0000 7f427fe00640 1 --2- 172.21.15.28:0/944023808 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf888180 0x5591cf892310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.354+0000 7f4284c00640 1 -- 172.21.15.28:0/944023808 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf888180 msgr2=0x5591cf892310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:47.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.354+0000 7f4284c00640 1 --2- 172.21.15.28:0/944023808 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf888180 0x5591cf892310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.354+0000 7f4284c00640 1 -- 172.21.15.28:0/944023808 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf892850 msgr2=0x5591cf894c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:47.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.354+0000 7f4284c00640 1 --2- 172.21.15.28:0/944023808 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf892850 0x5591cf894c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.354+0000 7f4284c00640 1 -- 172.21.15.28:0/944023808 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591cf7e7f50 con 0x5591cf887860 2024-04-24T08:18:47.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.354+0000 7f427f400640 1 --2- 172.21.15.28:0/944023808 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf892850 0x5591cf894c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.355+0000 7f4284c00640 1 --2- 172.21.15.28:0/944023808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf887860 0x5591cf887c40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4278009ba0 tx=0x7f4278011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=70afdb3510033827 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.355+0000 7f4275200640 1 -- 172.21.15.28:0/944023808 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278004230 con 0x5591cf887860 2024-04-24T08:18:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.355+0000 7f4275200640 1 -- 172.21.15.28:0/944023808 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4278009e10 con 0x5591cf887860 2024-04-24T08:18:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.355+0000 7f4275200640 1 -- 172.21.15.28:0/944023808 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278004d70 con 0x5591cf887860 2024-04-24T08:18:47.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.355+0000 7f427cc00640 1 --2- 172.21.15.28:0/1679719394 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f42640a3da0 0x7f42640a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.355+0000 7f427cc00640 1 -- 172.21.15.28:0/1679719394 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7f42700f18d0 con 0x7f4280076530 2024-04-24T08:18:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.355+0000 7f427fe00640 1 --2- 172.21.15.28:0/1679719394 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f42640a3da0 0x7f42640a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.356+0000 7f428997d740 1 -- 172.21.15.28:0/944023808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf887860 msgr2=0x5591cf887c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.356+0000 7f428997d740 1 --2- 172.21.15.28:0/944023808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf887860 0x5591cf887c40 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f4278009ba0 tx=0x7f4278011810 comp rx=0 tx=0).stop 2024-04-24T08:18:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.356+0000 7f428997d740 1 -- 172.21.15.28:0/944023808 shutdown_connections 2024-04-24T08:18:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.356+0000 7f428997d740 1 --2- 172.21.15.28:0/944023808 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf892850 0x5591cf894c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.356+0000 7f428997d740 1 --2- 172.21.15.28:0/944023808 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf888180 0x5591cf892310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.356+0000 7f428997d740 1 --2- 172.21.15.28:0/944023808 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf887860 0x5591cf887c40 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.356+0000 7f428997d740 1 -- 172.21.15.28:0/944023808 >> 172.21.15.28:0/944023808 conn(0x5591cf883b80 msgr2=0x5591cf883f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.356+0000 7f428997d740 1 -- 172.21.15.28:0/944023808 shutdown_connections 2024-04-24T08:18:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.356+0000 7f428997d740 1 -- 172.21.15.28:0/944023808 wait complete. 2024-04-24T08:18:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.356+0000 7f427fe00640 1 --2- 172.21.15.28:0/1679719394 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f42640a3da0 0x7f42640a6260 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x5591cf888e20 tx=0x7f42700f7530 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.356+0000 7f428997d740 1 -- start start 2024-04-24T08:18:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.357+0000 7f428997d740 1 Processor -- start 2024-04-24T08:18:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.357+0000 7f428997d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf887860 0x5591cf88f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.364 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.357+0000 7f428997d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf888180 0x5591cf88fd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.357+0000 7f428997d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf892850 0x5591cf890260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.357+0000 7f428997d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5591cf895640 con 0x5591cf892850 2024-04-24T08:18:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.357+0000 7f428997d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5591cf7a6690 con 0x5591cf888180 2024-04-24T08:18:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.357+0000 7f428997d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5591cf8957c0 con 0x5591cf887860 2024-04-24T08:18:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.357+0000 7f428997d740 10 client.0 _ll_register_callbacks cb 0x5591cf7e2650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.358+0000 7f427f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf888180 0x5591cf88fd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.358+0000 7f4284c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf892850 0x5591cf890260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.358+0000 7f427f400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf888180 0x5591cf88fd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41670/0 (socket says 172.21.15.28:41670) 2024-04-24T08:18:47.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.358+0000 7f427f400640 1 -- 172.21.15.28:0/1724803648 learned_addr learned my addr 172.21.15.28:0/1724803648 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.358+0000 7f4284c00640 1 --2- 172.21.15.28:0/1724803648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf887860 0x5591cf88f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.358+0000 7f427f400640 1 -- 172.21.15.28:0/1724803648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf887860 msgr2=0x5591cf88f7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.358+0000 7f427f400640 1 --2- 172.21.15.28:0/1724803648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf887860 0x5591cf88f7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.358+0000 7f427f400640 1 -- 172.21.15.28:0/1724803648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf892850 msgr2=0x5591cf890260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.358+0000 7f427f400640 1 --2- 172.21.15.28:0/1724803648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf892850 0x5591cf890260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.358+0000 7f427f400640 1 -- 172.21.15.28:0/1724803648 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591cf7a5920 con 0x5591cf888180 2024-04-24T08:18:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.358+0000 7f4284c00640 1 --2- 172.21.15.28:0/1724803648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf887860 0x5591cf88f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:47.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.358+0000 7f427f400640 1 --2- 172.21.15.28:0/1724803648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf888180 0x5591cf88fd20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f426800bee0 tx=0x7f4268012820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=150855515811b549 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:47.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.358+0000 7f426d600640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:18:47.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.359+0000 7f426d600640 1 -- 172.21.15.28:0/1724803648 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4268004230 con 0x5591cf888180 2024-04-24T08:18:47.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.359+0000 7f426d600640 1 -- 172.21.15.28:0/1724803648 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7f42680043d0 con 0x5591cf888180 2024-04-24T08:18:47.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.359+0000 7f428997d740 1 -- 172.21.15.28:0/1724803648 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4278009e10 con 0x5591cf888180 2024-04-24T08:18:47.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.359+0000 7f426d600640 1 -- 172.21.15.28:0/1724803648 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f42680106a0 con 0x5591cf888180 2024-04-24T08:18:47.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.359+0000 7f428997d740 10 client.26627 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:47.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.359+0000 7f428997d740 1 -- 172.21.15.28:0/1724803648 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5591cf5c97d0 con 0x5591cf888180 2024-04-24T08:18:47.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.359+0000 7f426d600640 1 -- 172.21.15.28:0/1724803648 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4278009e10 con 0x5591cf888180 2024-04-24T08:18:47.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.359+0000 7f426d600640 1 -- 172.21.15.28:0/1724803648 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4268019070 con 0x5591cf888180 2024-04-24T08:18:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.359+0000 7f428997d740 10 client.26627 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.360+0000 7f428997d740 10 client.26627 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:18:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.360+0000 7f428997d740 20 client.26627 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.360+0000 7f428997d740 1 --2- 172.21.15.28:0/1724803648 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5591cf8a2d80 0x5591cf8a51a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.360+0000 7f428997d740 4 client.26627 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:18:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.360+0000 7f428997d740 1 -- 172.21.15.28:0/1724803648 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5591cf7fac10 con 0x5591cf8a2d80 2024-04-24T08:18:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.360+0000 7f4284c00640 1 --2- 172.21.15.28:0/1724803648 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5591cf8a2d80 0x5591cf8a51a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.360+0000 7f4284c00640 1 --2- 172.21.15.28:0/1724803648 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5591cf8a2d80 0x5591cf8a51a0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49b3b097c14fe47d server_cookie=e21b2bff4361eec6 in_seq=0 out_seq=0 2024-04-24T08:18:47.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.360+0000 7f426d600640 10 client.26627 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.361+0000 7f426d600640 1 -- 172.21.15.28:0/1724803648 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5591cf7fac10 con 0x5591cf8a2d80 2024-04-24T08:18:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.361+0000 7f426d600640 10 client.26627 handle_command_reply: tid=0 2024-04-24T08:18:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.361+0000 7f428997d740 1 client.26627 shutdown 2024-04-24T08:18:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.361+0000 7f428997d740 1 -- 172.21.15.28:0/1724803648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf888180 msgr2=0x5591cf88fd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.361+0000 7f428997d740 1 --2- 172.21.15.28:0/1724803648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf888180 0x5591cf88fd20 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f426800bee0 tx=0x7f4268012820 comp rx=0 tx=0).stop 2024-04-24T08:18:47.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.362+0000 7f428997d740 1 -- 172.21.15.28:0/1724803648 shutdown_connections 2024-04-24T08:18:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.362+0000 7f428997d740 1 --2- 172.21.15.28:0/1724803648 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5591cf8a2d80 0x5591cf8a51a0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.362+0000 7f428997d740 1 --2- 172.21.15.28:0/1724803648 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf892850 0x5591cf890260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.362+0000 7f428997d740 1 --2- 172.21.15.28:0/1724803648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf888180 0x5591cf88fd20 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.362+0000 7f428997d740 1 --2- 172.21.15.28:0/1724803648 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf887860 0x5591cf88f7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.362+0000 7f428997d740 1 -- 172.21.15.28:0/1724803648 >> 172.21.15.28:0/1724803648 conn(0x5591cf883b80 msgr2=0x5591cf889880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.362+0000 7f426d600640 0 client.26627 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.362+0000 7f428997d740 1 -- 172.21.15.28:0/1724803648 shutdown_connections 2024-04-24T08:18:47.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.362+0000 7f428997d740 1 -- 172.21.15.28:0/1724803648 wait complete. 2024-04-24T08:18:47.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.362+0000 7f428997d740 20 client.26627 trim_cache size 0 max 16384 2024-04-24T08:18:47.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.382+0000 7f428997d740 1 Processor -- start 2024-04-24T08:18:47.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.382+0000 7f428997d740 1 -- start start 2024-04-24T08:18:47.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.382+0000 7f428997d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf887860 0x5591cf88e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.382+0000 7f428997d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf888180 0x5591cf89e560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.382+0000 7f428997d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf892850 0x5591cf8aaa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.382+0000 7f428997d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42801b6c40 con 0x5591cf887860 2024-04-24T08:18:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.382+0000 7f428997d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f42801b4510 con 0x5591cf892850 2024-04-24T08:18:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.382+0000 7f428997d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f42801b2380 con 0x5591cf888180 2024-04-24T08:18:47.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.382+0000 7f4284c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf887860 0x5591cf88e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.383+0000 7f427f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf888180 0x5591cf89e560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.383+0000 7f4284c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf887860 0x5591cf88e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54660/0 (socket says 172.21.15.28:54660) 2024-04-24T08:18:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.383+0000 7f427f400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf888180 0x5591cf89e560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:46110/0 (socket says 172.21.15.28:46110) 2024-04-24T08:18:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.383+0000 7f4284c00640 1 -- 172.21.15.28:0/4087504724 learned_addr learned my addr 172.21.15.28:0/4087504724 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.383+0000 7f4284c00640 1 --2- 172.21.15.28:0/4087504724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf892850 0x5591cf8aaa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.383+0000 7f427f400640 1 -- 172.21.15.28:0/4087504724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf892850 msgr2=0x5591cf8aaa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.383+0000 7f427f400640 1 --2- 172.21.15.28:0/4087504724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf892850 0x5591cf8aaa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.383+0000 7f427f400640 1 -- 172.21.15.28:0/4087504724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf887860 msgr2=0x5591cf88e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:47.386 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.383+0000 7f427f400640 1 --2- 172.21.15.28:0/4087504724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf887860 0x5591cf88e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.383+0000 7f427f400640 1 -- 172.21.15.28:0/4087504724 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5591cf4ff3f0 con 0x5591cf888180 2024-04-24T08:18:47.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.383+0000 7f427f400640 1 --2- 172.21.15.28:0/4087504724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf888180 0x5591cf89e560 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4268003d80 tx=0x7f426801b6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3b16f10dd1fd1b5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:47.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.384+0000 7f426fe00640 1 -- 172.21.15.28:0/4087504724 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f42680037f0 con 0x5591cf888180 2024-04-24T08:18:47.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.384+0000 7f426fe00640 1 -- 172.21.15.28:0/4087504724 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4268005550 con 0x5591cf888180 2024-04-24T08:18:47.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.384+0000 7f426fe00640 1 -- 172.21.15.28:0/4087504724 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f426801bcc0 con 0x5591cf888180 2024-04-24T08:18:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.384+0000 7f428997d740 1 -- 172.21.15.28:0/4087504724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf888180 msgr2=0x5591cf89e560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.384+0000 7f428997d740 1 --2- 172.21.15.28:0/4087504724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf888180 0x5591cf89e560 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f4268003d80 tx=0x7f426801b6c0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.384+0000 7f428997d740 1 -- 172.21.15.28:0/4087504724 shutdown_connections 2024-04-24T08:18:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.384+0000 7f428997d740 1 --2- 172.21.15.28:0/4087504724 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf892850 0x5591cf8aaa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.384+0000 7f428997d740 1 --2- 172.21.15.28:0/4087504724 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf888180 0x5591cf89e560 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.384+0000 7f428997d740 1 --2- 172.21.15.28:0/4087504724 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf887860 0x5591cf88e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.384+0000 7f428997d740 1 -- 172.21.15.28:0/4087504724 >> 172.21.15.28:0/4087504724 conn(0x5591cf883b80 msgr2=0x5591cf890030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.385+0000 7f428997d740 1 -- 172.21.15.28:0/4087504724 shutdown_connections 2024-04-24T08:18:47.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.385+0000 7f428997d740 1 -- 172.21.15.28:0/4087504724 wait complete. 2024-04-24T08:18:47.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.385+0000 7f428997d740 1 -- start start 2024-04-24T08:18:47.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.385+0000 7f428997d740 1 Processor -- start 2024-04-24T08:18:47.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.386+0000 7f428997d740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf887860 0x5591cf896750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.386+0000 7f428997d740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf888180 0x5591cf896c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.386+0000 7f428997d740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf892850 0x5591cf8971f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.386+0000 7f428997d740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f42801b29d0 con 0x5591cf892850 2024-04-24T08:18:47.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.386+0000 7f428997d740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f42801b3e40 con 0x5591cf888180 2024-04-24T08:18:47.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.386+0000 7f428997d740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f42801b30a0 con 0x5591cf887860 2024-04-24T08:18:47.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.386+0000 7f428997d740 10 client.0 _ll_register_callbacks cb 0x7f426800b790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:47.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.386+0000 7f4284c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf892850 0x5591cf8971f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.386+0000 7f4284c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf887860 0x5591cf896750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.386+0000 7f4284c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf887860 0x5591cf896750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:46114/0 (socket says 172.21.15.28:46114) 2024-04-24T08:18:47.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.386+0000 7f4284c00640 1 -- 172.21.15.28:0/2397385277 learned_addr learned my addr 172.21.15.28:0/2397385277 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:47.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.386+0000 7f427f400640 1 --2- 172.21.15.28:0/2397385277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf888180 0x5591cf896c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.386+0000 7f4284c00640 1 -- 172.21.15.28:0/2397385277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf888180 msgr2=0x5591cf896c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:47.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.386+0000 7f4284c00640 1 --2- 172.21.15.28:0/2397385277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf888180 0x5591cf896c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.386+0000 7f4284c00640 1 -- 172.21.15.28:0/2397385277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf892850 msgr2=0x5591cf8971f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:47.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.386+0000 7f4284c00640 1 --2- 172.21.15.28:0/2397385277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf892850 0x5591cf8971f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.386+0000 7f4284c00640 1 -- 172.21.15.28:0/2397385277 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5591cf4bf0f0 con 0x5591cf887860 2024-04-24T08:18:47.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.387+0000 7f4284c00640 1 --2- 172.21.15.28:0/2397385277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf887860 0x5591cf896750 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f427800a010 tx=0x7f4278005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4becbe0817e2ab2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:47.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.387+0000 7f426d600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:18:47.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.387+0000 7f426d600640 1 -- 172.21.15.28:0/2397385277 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4278019ac0 con 0x5591cf887860 2024-04-24T08:18:47.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.387+0000 7f428997d740 1 -- 172.21.15.28:0/2397385277 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4268005550 con 0x5591cf887860 2024-04-24T08:18:47.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.387+0000 7f426d600640 1 -- 172.21.15.28:0/2397385277 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7f427801e040 con 0x5591cf887860 2024-04-24T08:18:47.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.387+0000 7f426d600640 1 -- 172.21.15.28:0/2397385277 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4278004020 con 0x5591cf887860 2024-04-24T08:18:47.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.387+0000 7f428997d740 10 client.26653 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:47.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.387+0000 7f428997d740 1 -- 172.21.15.28:0/2397385277 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5591cf7b2810 con 0x5591cf887860 2024-04-24T08:18:47.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.388+0000 7f426d600640 1 -- 172.21.15.28:0/2397385277 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4268005550 con 0x5591cf887860 2024-04-24T08:18:47.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.388+0000 7f426d600640 1 -- 172.21.15.28:0/2397385277 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4278024040 con 0x5591cf887860 2024-04-24T08:18:47.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.388+0000 7f428997d740 10 client.26653 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:47.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.388+0000 7f428997d740 10 client.26653 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:18:47.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.388+0000 7f428997d740 20 client.26653 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:47.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.388+0000 7f428997d740 1 --2- 172.21.15.28:0/2397385277 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5591cf909ae0 0x5591cf90bed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.388+0000 7f428997d740 4 client.26653 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:18:47.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.388+0000 7f428997d740 1 -- 172.21.15.28:0/2397385277 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5591cf7f4a80 con 0x5591cf909ae0 2024-04-24T08:18:47.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.388+0000 7f427f400640 1 --2- 172.21.15.28:0/2397385277 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5591cf909ae0 0x5591cf90bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.389+0000 7f427f400640 1 --2- 172.21.15.28:0/2397385277 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5591cf909ae0 0x5591cf90bed0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b9aa9e327a485dbc server_cookie=d59653b4aa4d3ce6 in_seq=0 out_seq=0 2024-04-24T08:18:47.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.389+0000 7f426d600640 10 client.26653 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:47.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.389+0000 7f426d600640 1 -- 172.21.15.28:0/2397385277 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5591cf7f4a80 con 0x5591cf909ae0 2024-04-24T08:18:47.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.389+0000 7f426d600640 10 client.26653 handle_command_reply: tid=0 2024-04-24T08:18:47.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.389+0000 7f428997d740 1 client.26653 shutdown 2024-04-24T08:18:47.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.389+0000 7f428997d740 1 -- 172.21.15.28:0/2397385277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf887860 msgr2=0x5591cf896750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.389+0000 7f428997d740 1 --2- 172.21.15.28:0/2397385277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf887860 0x5591cf896750 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f427800a010 tx=0x7f4278005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:47.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.389+0000 7f428997d740 1 -- 172.21.15.28:0/2397385277 shutdown_connections 2024-04-24T08:18:47.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.389+0000 7f428997d740 1 --2- 172.21.15.28:0/2397385277 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5591cf909ae0 0x5591cf90bed0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.389+0000 7f428997d740 1 --2- 172.21.15.28:0/2397385277 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5591cf892850 0x5591cf8971f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.389+0000 7f426d600640 0 client.26653 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:47.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.389+0000 7f428997d740 1 --2- 172.21.15.28:0/2397385277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5591cf888180 0x5591cf896c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.389+0000 7f428997d740 1 --2- 172.21.15.28:0/2397385277 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5591cf887860 0x5591cf896750 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.389+0000 7f428997d740 1 -- 172.21.15.28:0/2397385277 >> 172.21.15.28:0/2397385277 conn(0x5591cf883b80 msgr2=0x5591cf882390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:47.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.389+0000 7f428997d740 1 -- 172.21.15.28:0/2397385277 shutdown_connections 2024-04-24T08:18:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.389+0000 7f428997d740 1 -- 172.21.15.28:0/2397385277 wait complete. 2024-04-24T08:18:47.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.390+0000 7f428997d740 20 client.26653 trim_cache size 0 max 16384 2024-04-24T08:18:47.399 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:18:47.399 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-24T08:18:47.399 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-24T08:18:47.400 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:18:47.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.390+0000 7f4286c00640 1 -- 172.21.15.28:0/1679719394 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f42640a3da0 msgr2=0x7f42640a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.390+0000 7f4286c00640 1 --2- 172.21.15.28:0/1679719394 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f42640a3da0 0x7f42640a6260 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x5591cf888e20 tx=0x7f42700f7530 comp rx=0 tx=0).stop 2024-04-24T08:18:47.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.390+0000 7f4286c00640 1 -- 172.21.15.28:0/1679719394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4280076530 msgr2=0x7f428019efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.390+0000 7f4286c00640 1 --2- 172.21.15.28:0/1679719394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4280076530 0x7f428019efb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f4270005e30 tx=0x7f4270030920 comp rx=0 tx=0).stop 2024-04-24T08:18:47.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.390+0000 7f4286c00640 1 -- 172.21.15.28:0/1679719394 shutdown_connections 2024-04-24T08:18:47.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.390+0000 7f4286c00640 1 --2- 172.21.15.28:0/1679719394 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f42640a3da0 0x7f42640a6260 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.390+0000 7f4286c00640 1 --2- 172.21.15.28:0/1679719394 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4280112ee0 0x7f42801a3820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.390+0000 7f4286c00640 1 --2- 172.21.15.28:0/1679719394 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4280076e50 0x7f428019f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.390+0000 7f4286c00640 1 --2- 172.21.15.28:0/1679719394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4280076530 0x7f428019efb0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.390+0000 7f4286c00640 1 -- 172.21.15.28:0/1679719394 >> 172.21.15.28:0/1679719394 conn(0x7f42800ffd90 msgr2=0x7f4280114530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.390+0000 7f4286c00640 1 -- 172.21.15.28:0/1679719394 shutdown_connections 2024-04-24T08:18:47.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.391+0000 7f4286c00640 1 -- 172.21.15.28:0/1679719394 wait complete. 2024-04-24T08:18:47.411 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:18:47.411 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:18:47.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.614+0000 7fc3c7200640 1 -- 172.21.15.28:0/1104100462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3c8104890 msgr2=0x7fc3c8104d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.614+0000 7fc3c7200640 1 --2- 172.21.15.28:0/1104100462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3c8104890 0x7fc3c8104d40 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fc3c0009be0 tx=0x7fc3c002a6d0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.614+0000 7fc3c7200640 1 -- 172.21.15.28:0/1104100462 shutdown_connections 2024-04-24T08:18:47.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.614+0000 7fc3c7200640 1 --2- 172.21.15.28:0/1104100462 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3c81071b0 0x7fc3c80696d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.616 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.614+0000 7fc3c7200640 1 --2- 172.21.15.28:0/1104100462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3c8104890 0x7fc3c8104d40 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.614+0000 7fc3c7200640 1 --2- 172.21.15.28:0/1104100462 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3c8106d40 0x7fc3c8069190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.614+0000 7fc3c7200640 1 -- 172.21.15.28:0/1104100462 >> 172.21.15.28:0/1104100462 conn(0x7fc3c8076840 msgr2=0x7fc3c8076c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.614+0000 7fc3c7200640 1 -- 172.21.15.28:0/1104100462 shutdown_connections 2024-04-24T08:18:47.617 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.614+0000 7fc3c7200640 1 -- 172.21.15.28:0/1104100462 wait complete. 2024-04-24T08:18:47.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.615+0000 7fc3c7200640 1 Processor -- start 2024-04-24T08:18:47.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.616+0000 7fc3c7200640 1 -- start start 2024-04-24T08:18:47.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.616+0000 7fc3c7200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3c8104890 0x7fc3c81a4860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.616+0000 7fc3c7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3c8106d40 0x7fc3c81a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.619 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.616+0000 7fc3c7200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3c81071b0 0x7fc3c819ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.616+0000 7fc3c7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c8118730 con 0x7fc3c8106d40 2024-04-24T08:18:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.616+0000 7fc3c7200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c81185b0 con 0x7fc3c81071b0 2024-04-24T08:18:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.616+0000 7fc3c7200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc3c81188b0 con 0x7fc3c8104890 2024-04-24T08:18:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.616+0000 7fc3c5e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3c8104890 0x7fc3c81a4860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.616+0000 7fc3c5400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3c8106d40 0x7fc3c81a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.616+0000 7fc3c5e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3c8104890 0x7fc3c81a4860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:46136/0 (socket says 172.21.15.28:46136) 2024-04-24T08:18:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.616+0000 7fc3c5e00640 1 -- 172.21.15.28:0/3799203140 learned_addr learned my addr 172.21.15.28:0/3799203140 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:47.620 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.617+0000 7fc3c6800640 1 --2- 172.21.15.28:0/3799203140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3c81071b0 0x7fc3c819ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.617+0000 7fc3c5400640 1 -- 172.21.15.28:0/3799203140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3c8104890 msgr2=0x7fc3c81a4860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.617+0000 7fc3c5400640 1 --2- 172.21.15.28:0/3799203140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3c8104890 0x7fc3c81a4860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.617+0000 7fc3c5400640 1 -- 172.21.15.28:0/3799203140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3c81071b0 msgr2=0x7fc3c819ea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.617+0000 7fc3c5400640 1 --2- 172.21.15.28:0/3799203140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3c81071b0 0x7fc3c819ea10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.617+0000 7fc3c5400640 1 -- 172.21.15.28:0/3799203140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc3c819f190 con 0x7fc3c8106d40 2024-04-24T08:18:47.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.617+0000 7fc3c5400640 1 --2- 172.21.15.28:0/3799203140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3c8106d40 0x7fc3c81a4da0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4005fd0 tx=0x7fc3b400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:47.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.618+0000 7fc3bea00640 1 -- 172.21.15.28:0/3799203140 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b400cd00 con 0x7fc3c8106d40 2024-04-24T08:18:47.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.618+0000 7fc3c7200640 1 -- 172.21.15.28:0/3799203140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc3c819f480 con 0x7fc3c8106d40 2024-04-24T08:18:47.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.618+0000 7fc3bea00640 1 -- 172.21.15.28:0/3799203140 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc3b40035f0 con 0x7fc3c8106d40 2024-04-24T08:18:47.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.618+0000 7fc3bea00640 1 -- 172.21.15.28:0/3799203140 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc3b4010670 con 0x7fc3c8106d40 2024-04-24T08:18:47.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.618+0000 7fc3c7200640 1 -- 172.21.15.28:0/3799203140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc3c81ab720 con 0x7fc3c8106d40 2024-04-24T08:18:47.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.619+0000 7fc3c7200640 1 -- 172.21.15.28:0/3799203140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc3c8068d40 con 0x7fc3c8106d40 2024-04-24T08:18:47.623 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.622+0000 7fc3bea00640 1 -- 172.21.15.28:0/3799203140 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fc3b40037a0 con 0x7fc3c8106d40 2024-04-24T08:18:47.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.623+0000 7fc3bea00640 1 --2- 172.21.15.28:0/3799203140 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc3ac0a3d20 0x7fc3ac0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:47.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.623+0000 7fc3bea00640 1 -- 172.21.15.28:0/3799203140 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7fc3b40d09e0 con 0x7fc3c8106d40 2024-04-24T08:18:47.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.624+0000 7fc3c5e00640 1 --2- 172.21.15.28:0/3799203140 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc3ac0a3d20 0x7fc3ac0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:47.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.624+0000 7fc3c5e00640 1 --2- 172.21.15.28:0/3799203140 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc3ac0a3d20 0x7fc3ac0a61e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc3c819fb50 tx=0x7fc3b8005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:47.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.629+0000 7fc3bea00640 1 -- 172.21.15.28:0/3799203140 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc3b4009510 con 0x7fc3c8106d40 2024-04-24T08:18:47.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:47 smithi028 ceph-mon[27786]: pgmap v260: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-04-24T08:18:47.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.873+0000 7fc3c7200640 1 -- 172.21.15.28:0/3799203140 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fc3c81aba80 con 0x7fc3c8106d40 2024-04-24T08:18:47.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.875+0000 7fc3bea00640 1 -- 172.21.15.28:0/3799203140 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11871 (secure 0 0 0) 0x7fc3b409cfd0 con 0x7fc3c8106d40 2024-04-24T08:18:47.877 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:18:47.877 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:14:38.663141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:18:47.877 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-24T08:18:47.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:47 smithi117 ceph-mon[32438]: pgmap v260: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-04-24T08:18:47.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.878+0000 7fc3c7200640 1 -- 172.21.15.28:0/3799203140 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc3ac0a3d20 msgr2=0x7fc3ac0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.878+0000 7fc3c7200640 1 --2- 172.21.15.28:0/3799203140 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc3ac0a3d20 0x7fc3ac0a61e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fc3c819fb50 tx=0x7fc3b8005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.878+0000 7fc3c7200640 1 -- 172.21.15.28:0/3799203140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3c8106d40 msgr2=0x7fc3c81a4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.878+0000 7fc3c7200640 1 --2- 172.21.15.28:0/3799203140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3c8106d40 0x7fc3c81a4da0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fc3b4005fd0 tx=0x7fc3b400ee50 comp rx=0 tx=0).stop 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.878+0000 7fc3c7200640 1 -- 172.21.15.28:0/3799203140 shutdown_connections 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.878+0000 7fc3c7200640 1 --2- 172.21.15.28:0/3799203140 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc3ac0a3d20 0x7fc3ac0a61e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.878+0000 7fc3c7200640 1 --2- 172.21.15.28:0/3799203140 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc3c8106d40 0x7fc3c81a4da0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.878+0000 7fc3c7200640 1 --2- 172.21.15.28:0/3799203140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3c81071b0 0x7fc3c819ea10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.878+0000 7fc3c7200640 1 --2- 172.21.15.28:0/3799203140 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc3c8104890 0x7fc3c81a4860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.878+0000 7fc3c7200640 1 -- 172.21.15.28:0/3799203140 >> 172.21.15.28:0/3799203140 conn(0x7fc3c8076840 msgr2=0x7fc3c8101d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.878+0000 7fc3c7200640 1 -- 172.21.15.28:0/3799203140 shutdown_connections 2024-04-24T08:18:47.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:47.878+0000 7fc3c7200640 1 -- 172.21.15.28:0/3799203140 wait complete. 2024-04-24T08:18:47.899 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:18:47.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:47 smithi194 ceph-mon[32836]: pgmap v260: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 5.5 KiB/s rd, 3.0 KiB/s wr, 5 op/s 2024-04-24T08:18:48.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.112+0000 7f6351e00640 1 -- 172.21.15.28:0/3781879518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c102120 msgr2=0x7f634c102500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.112+0000 7f6351e00640 1 --2- 172.21.15.28:0/3781879518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c102120 0x7f634c102500 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f6340009900 tx=0x7f634002a400 comp rx=0 tx=0).stop 2024-04-24T08:18:48.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.112+0000 7f6351e00640 1 -- 172.21.15.28:0/3781879518 shutdown_connections 2024-04-24T08:18:48.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.112+0000 7f6351e00640 1 --2- 172.21.15.28:0/3781879518 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f634c1071f0 0x7f634c10edc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.112+0000 7f6351e00640 1 --2- 172.21.15.28:0/3781879518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f634c102ad0 0x7f634c106b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.112+0000 7f6351e00640 1 --2- 172.21.15.28:0/3781879518 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c102120 0x7f634c102500 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.112+0000 7f6351e00640 1 -- 172.21.15.28:0/3781879518 >> 172.21.15.28:0/3781879518 conn(0x7f634c0fd9a0 msgr2=0x7f634c0ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.112+0000 7f6351e00640 1 -- 172.21.15.28:0/3781879518 shutdown_connections 2024-04-24T08:18:48.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.112+0000 7f6351e00640 1 -- 172.21.15.28:0/3781879518 wait complete. 2024-04-24T08:18:48.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.113+0000 7f6351e00640 1 Processor -- start 2024-04-24T08:18:48.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.113+0000 7f6351e00640 1 -- start start 2024-04-24T08:18:48.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.113+0000 7f6351e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f634c102120 0x7f634c19e430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.113+0000 7f6351e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c1071f0 0x7f634c19e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.113+0000 7f6351e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f634c198630 0x7f634c198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.113+0000 7f6351e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1122d0 con 0x7f634c1071f0 2024-04-24T08:18:48.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.113+0000 7f6351e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f634c112150 con 0x7f634c102120 2024-04-24T08:18:48.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.113+0000 7f6351e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f634c112450 con 0x7f634c198630 2024-04-24T08:18:48.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.114+0000 7f634be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c1071f0 0x7f634c19e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.114+0000 7f6350a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f634c102120 0x7f634c19e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.114+0000 7f6350a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f634c102120 0x7f634c19e430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41720/0 (socket says 172.21.15.28:41720) 2024-04-24T08:18:48.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.114+0000 7f6350a00640 1 -- 172.21.15.28:0/1409488848 learned_addr learned my addr 172.21.15.28:0/1409488848 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:48.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.114+0000 7f6351400640 1 --2- 172.21.15.28:0/1409488848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f634c198630 0x7f634c198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.114+0000 7f6350a00640 1 -- 172.21.15.28:0/1409488848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f634c198630 msgr2=0x7f634c198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.114+0000 7f6350a00640 1 --2- 172.21.15.28:0/1409488848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f634c198630 0x7f634c198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.114+0000 7f6350a00640 1 -- 172.21.15.28:0/1409488848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c1071f0 msgr2=0x7f634c19e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.114+0000 7f6350a00640 1 --2- 172.21.15.28:0/1409488848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c1071f0 0x7f634c19e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.114+0000 7f6350a00640 1 -- 172.21.15.28:0/1409488848 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f634c199150 con 0x7f634c102120 2024-04-24T08:18:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.115+0000 7f6350a00640 1 --2- 172.21.15.28:0/1409488848 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f634c102120 0x7f634c19e430 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6340004770 tx=0x7f6340003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.115+0000 7f6349600640 1 -- 172.21.15.28:0/1409488848 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f634003f070 con 0x7f634c102120 2024-04-24T08:18:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.115+0000 7f6351e00640 1 -- 172.21.15.28:0/1409488848 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f634c1993e0 con 0x7f634c102120 2024-04-24T08:18:48.119 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.115+0000 7f6349600640 1 -- 172.21.15.28:0/1409488848 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6340031ca0 con 0x7f634c102120 2024-04-24T08:18:48.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.115+0000 7f6349600640 1 -- 172.21.15.28:0/1409488848 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6340035030 con 0x7f634c102120 2024-04-24T08:18:48.120 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.115+0000 7f6351e00640 1 -- 172.21.15.28:0/1409488848 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f634c1a5590 con 0x7f634c102120 2024-04-24T08:18:48.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.118+0000 7f635627e740 1 Processor -- start 2024-04-24T08:18:48.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.118+0000 7f635627e740 1 -- start start 2024-04-24T08:18:48.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.118+0000 7f635627e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fc4b40 0x560478fc4f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.118+0000 7f635627e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc5460 0x560478fcf5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.118+0000 7f635627e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fcfb30 0x560478fd1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.118+0000 7f635627e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560478f2a1d0 con 0x560478fcfb30 2024-04-24T08:18:48.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.118+0000 7f635627e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560478ed0450 con 0x560478fc4b40 2024-04-24T08:18:48.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.118+0000 7f635627e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560478f26190 con 0x560478fc5460 2024-04-24T08:18:48.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.119+0000 7f6349600640 1 -- 172.21.15.28:0/1409488848 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6340042070 con 0x7f634c102120 2024-04-24T08:18:48.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.119+0000 7f634be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fcfb30 0x560478fd1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.119+0000 7f6351400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fc4b40 0x560478fc4f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.119+0000 7f634be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fcfb30 0x560478fd1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54690/0 (socket says 172.21.15.28:54690) 2024-04-24T08:18:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.119+0000 7f634be00640 1 -- 172.21.15.28:0/1818663254 learned_addr learned my addr 172.21.15.28:0/1818663254 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.119+0000 7f6350a00640 1 --2- 172.21.15.28:0/1818663254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc5460 0x560478fcf5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.119+0000 7f6351400640 1 -- 172.21.15.28:0/1818663254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc5460 msgr2=0x560478fcf5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.119+0000 7f6351400640 1 --2- 172.21.15.28:0/1818663254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc5460 0x560478fcf5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.119+0000 7f6351400640 1 -- 172.21.15.28:0/1818663254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fcfb30 msgr2=0x560478fd1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.119+0000 7f6351400640 1 --2- 172.21.15.28:0/1818663254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fcfb30 0x560478fd1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.119+0000 7f6351400640 1 -- 172.21.15.28:0/1818663254 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560478f8d310 con 0x560478fc4b40 2024-04-24T08:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.119+0000 7f634be00640 1 --2- 172.21.15.28:0/1818663254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fcfb30 0x560478fd1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.120+0000 7f6351400640 1 --2- 172.21.15.28:0/1818663254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fc4b40 0x560478fc4f20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6344009f00 tx=0x7f6344011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=169f6f371fffa3c1 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.120+0000 7f6349600640 1 --2- 172.21.15.28:0/1409488848 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f63340a3d20 0x7f63340a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.120+0000 7f633dc00640 1 -- 172.21.15.28:0/1818663254 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6344004d20 con 0x560478fc4b40 2024-04-24T08:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.120+0000 7f633dc00640 1 -- 172.21.15.28:0/1818663254 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6344004050 con 0x560478fc4b40 2024-04-24T08:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.120+0000 7f633dc00640 1 -- 172.21.15.28:0/1818663254 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6344005280 con 0x560478fc4b40 2024-04-24T08:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.120+0000 7f6349600640 1 -- 172.21.15.28:0/1409488848 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7f63400f0e60 con 0x7f634c102120 2024-04-24T08:18:48.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.120+0000 7f6350a00640 1 --2- 172.21.15.28:0/1409488848 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f63340a3d20 0x7f63340a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.121+0000 7f635627e740 1 -- 172.21.15.28:0/1818663254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fc4b40 msgr2=0x560478fc4f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.121+0000 7f635627e740 1 --2- 172.21.15.28:0/1818663254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fc4b40 0x560478fc4f20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6344009f00 tx=0x7f6344011810 comp rx=0 tx=0).stop 2024-04-24T08:18:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.121+0000 7f635627e740 1 -- 172.21.15.28:0/1818663254 shutdown_connections 2024-04-24T08:18:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.121+0000 7f6350a00640 1 --2- 172.21.15.28:0/1409488848 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f63340a3d20 0x7f63340a61e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x560478fc6100 tx=0x7f634003b890 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.121+0000 7f635627e740 1 --2- 172.21.15.28:0/1818663254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fcfb30 0x560478fd1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.121+0000 7f635627e740 1 --2- 172.21.15.28:0/1818663254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc5460 0x560478fcf5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.121+0000 7f635627e740 1 --2- 172.21.15.28:0/1818663254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fc4b40 0x560478fc4f20 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.121+0000 7f635627e740 1 -- 172.21.15.28:0/1818663254 >> 172.21.15.28:0/1818663254 conn(0x560478fc0e60 msgr2=0x560478fc1240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.121+0000 7f635627e740 1 -- 172.21.15.28:0/1818663254 shutdown_connections 2024-04-24T08:18:48.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.121+0000 7f635627e740 1 -- 172.21.15.28:0/1818663254 wait complete. 2024-04-24T08:18:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.121+0000 7f635627e740 1 -- start start 2024-04-24T08:18:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.122+0000 7f635627e740 1 Processor -- start 2024-04-24T08:18:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.122+0000 7f635627e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fc4b40 0x560478fcf5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.122+0000 7f635627e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc5460 0x560478fcc8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.122+0000 7f635627e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fcfb30 0x560478fccfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.122+0000 7f635627e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560478fd29c0 con 0x560478fc4b40 2024-04-24T08:18:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.122+0000 7f635627e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560478ee3810 con 0x560478fcfb30 2024-04-24T08:18:48.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.122+0000 7f635627e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560478fd2b40 con 0x560478fc5460 2024-04-24T08:18:48.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.122+0000 7f635627e740 10 client.0 _ll_register_callbacks cb 0x560478eb1140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:48.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.122+0000 7f6351400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fc4b40 0x560478fcf5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.122+0000 7f6351400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fc4b40 0x560478fcf5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54694/0 (socket says 172.21.15.28:54694) 2024-04-24T08:18:48.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.122+0000 7f6351400640 1 -- 172.21.15.28:0/1580809071 learned_addr learned my addr 172.21.15.28:0/1580809071 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:48.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.123+0000 7f634be00640 1 --2- 172.21.15.28:0/1580809071 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc5460 0x560478fcc8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.123+0000 7f6351400640 1 --2- 172.21.15.28:0/1580809071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fcfb30 0x560478fccfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.123+0000 7f634be00640 1 -- 172.21.15.28:0/1580809071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fcfb30 msgr2=0x560478fccfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.123+0000 7f634be00640 1 --2- 172.21.15.28:0/1580809071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fcfb30 0x560478fccfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.123+0000 7f634be00640 1 -- 172.21.15.28:0/1580809071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fc4b40 msgr2=0x560478fcf5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.123+0000 7f634be00640 1 --2- 172.21.15.28:0/1580809071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fc4b40 0x560478fcf5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.123+0000 7f634be00640 1 -- 172.21.15.28:0/1580809071 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560478ef43f0 con 0x560478fc5460 2024-04-24T08:18:48.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.123+0000 7f634be00640 1 --2- 172.21.15.28:0/1580809071 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc5460 0x560478fcc8a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f63380126d0 tx=0x7f6338012b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c289da534694ed07 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.123+0000 7f6332000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:18:48.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.124+0000 7f6332000640 1 -- 172.21.15.28:0/1580809071 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f633800f920 con 0x560478fc5460 2024-04-24T08:18:48.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.124+0000 7f6332000640 1 -- 172.21.15.28:0/1580809071 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7f633800fac0 con 0x560478fc5460 2024-04-24T08:18:48.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.124+0000 7f635627e740 1 -- 172.21.15.28:0/1580809071 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6344004050 con 0x560478fc5460 2024-04-24T08:18:48.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.124+0000 7f6332000640 1 -- 172.21.15.28:0/1580809071 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f633800fcc0 con 0x560478fc5460 2024-04-24T08:18:48.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.124+0000 7f635627e740 10 client.26671 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:48.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.124+0000 7f635627e740 1 -- 172.21.15.28:0/1580809071 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560478c3c3f0 con 0x560478fc5460 2024-04-24T08:18:48.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.124+0000 7f6332000640 1 -- 172.21.15.28:0/1580809071 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6344004050 con 0x560478fc5460 2024-04-24T08:18:48.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.125+0000 7f6332000640 1 -- 172.21.15.28:0/1580809071 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6338010550 con 0x560478fc5460 2024-04-24T08:18:48.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.125+0000 7f635627e740 10 client.26671 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:48.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.125+0000 7f635627e740 10 client.26671 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:18:48.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.125+0000 7f635627e740 20 client.26671 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:48.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.125+0000 7f635627e740 1 --2- 172.21.15.28:0/1580809071 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x560478fdfda0 0x560478fe2190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.125+0000 7f635627e740 4 client.26671 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:18:48.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.125+0000 7f635627e740 1 -- 172.21.15.28:0/1580809071 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560478f37bf0 con 0x560478fdfda0 2024-04-24T08:18:48.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.125+0000 7f6351400640 1 --2- 172.21.15.28:0/1580809071 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x560478fdfda0 0x560478fe2190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.126+0000 7f6351400640 1 --2- 172.21.15.28:0/1580809071 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x560478fdfda0 0x560478fe2190 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=cfa503dff6b2243b server_cookie=29674f9ad599bfed in_seq=0 out_seq=0 2024-04-24T08:18:48.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.126+0000 7f6332000640 10 client.26671 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:18:48.136 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.127+0000 7f6332000640 1 -- 172.21.15.28:0/1580809071 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560478f37bf0 con 0x560478fdfda0 2024-04-24T08:18:48.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.127+0000 7f6332000640 10 client.26671 handle_command_reply: tid=0 2024-04-24T08:18:48.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.127+0000 7f635627e740 1 client.26671 shutdown 2024-04-24T08:18:48.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.127+0000 7f635627e740 1 -- 172.21.15.28:0/1580809071 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc5460 msgr2=0x560478fcc8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.127+0000 7f635627e740 1 --2- 172.21.15.28:0/1580809071 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc5460 0x560478fcc8a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f63380126d0 tx=0x7f6338012b90 comp rx=0 tx=0).stop 2024-04-24T08:18:48.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.127+0000 7f635627e740 1 -- 172.21.15.28:0/1580809071 shutdown_connections 2024-04-24T08:18:48.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.127+0000 7f635627e740 1 --2- 172.21.15.28:0/1580809071 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x560478fdfda0 0x560478fe2190 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.127+0000 7f635627e740 1 --2- 172.21.15.28:0/1580809071 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fcfb30 0x560478fccfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.127+0000 7f635627e740 1 --2- 172.21.15.28:0/1580809071 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc5460 0x560478fcc8a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.127+0000 7f635627e740 1 --2- 172.21.15.28:0/1580809071 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fc4b40 0x560478fcf5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.127+0000 7f6332000640 0 client.26671 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:18:48.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.127+0000 7f635627e740 1 -- 172.21.15.28:0/1580809071 >> 172.21.15.28:0/1580809071 conn(0x560478fc0e60 msgr2=0x560478fc6c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.127+0000 7f635627e740 1 -- 172.21.15.28:0/1580809071 shutdown_connections 2024-04-24T08:18:48.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.127+0000 7f635627e740 1 -- 172.21.15.28:0/1580809071 wait complete. 2024-04-24T08:18:48.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.127+0000 7f635627e740 20 client.26671 trim_cache size 0 max 16384 2024-04-24T08:18:48.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.142+0000 7f635627e740 1 Processor -- start 2024-04-24T08:18:48.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.142+0000 7f635627e740 1 -- start start 2024-04-24T08:18:48.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.142+0000 7f635627e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc4b40 0x560478fd8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.142+0000 7f635627e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fc5460 0x560478fcba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.142+0000 7f635627e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fcfb30 0x560478fbfdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.142+0000 7f635627e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1b05a0 con 0x560478fc5460 2024-04-24T08:18:48.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.142+0000 7f635627e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1ade70 con 0x560478fcfb30 2024-04-24T08:18:48.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.142+0000 7f635627e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1abce0 con 0x560478fc4b40 2024-04-24T08:18:48.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.143+0000 7f634be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fc5460 0x560478fcba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.143+0000 7f634be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fc5460 0x560478fcba00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54702/0 (socket says 172.21.15.28:54702) 2024-04-24T08:18:48.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.143+0000 7f634be00640 1 -- 172.21.15.28:0/1382463106 learned_addr learned my addr 172.21.15.28:0/1382463106 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:48.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.143+0000 7f6351400640 1 --2- 172.21.15.28:0/1382463106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc4b40 0x560478fd8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.143+0000 7f6351400640 1 --2- 172.21.15.28:0/1382463106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fcfb30 0x560478fbfdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.143+0000 7f634be00640 1 -- 172.21.15.28:0/1382463106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc4b40 msgr2=0x560478fd8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.143+0000 7f634be00640 1 --2- 172.21.15.28:0/1382463106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc4b40 0x560478fd8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.143+0000 7f634be00640 1 -- 172.21.15.28:0/1382463106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fcfb30 msgr2=0x560478fbfdd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.143+0000 7f634be00640 1 --2- 172.21.15.28:0/1382463106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fcfb30 0x560478fbfdd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.143+0000 7f634be00640 1 -- 172.21.15.28:0/1382463106 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560478d067d0 con 0x560478fc5460 2024-04-24T08:18:48.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.143+0000 7f6351400640 1 --2- 172.21.15.28:0/1382463106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fcfb30 0x560478fbfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:18:48.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.143+0000 7f634be00640 1 --2- 172.21.15.28:0/1382463106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fc5460 0x560478fcba00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6338010b60 tx=0x7f6338009ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69157c9cdefc106b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.144+0000 7f633d200640 1 -- 172.21.15.28:0/1382463106 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6338028070 con 0x560478fc5460 2024-04-24T08:18:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.144+0000 7f633d200640 1 -- 172.21.15.28:0/1382463106 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f633800f920 con 0x560478fc5460 2024-04-24T08:18:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.144+0000 7f633d200640 1 -- 172.21.15.28:0/1382463106 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63380054f0 con 0x560478fc5460 2024-04-24T08:18:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.144+0000 7f635627e740 1 -- 172.21.15.28:0/1382463106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fc5460 msgr2=0x560478fcba00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.144+0000 7f635627e740 1 --2- 172.21.15.28:0/1382463106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fc5460 0x560478fcba00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6338010b60 tx=0x7f6338009ea0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.144+0000 7f635627e740 1 -- 172.21.15.28:0/1382463106 shutdown_connections 2024-04-24T08:18:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.144+0000 7f635627e740 1 --2- 172.21.15.28:0/1382463106 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fc5460 0x560478fcba00 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.144+0000 7f635627e740 1 --2- 172.21.15.28:0/1382463106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fcfb30 0x560478fbfdd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.144+0000 7f635627e740 1 --2- 172.21.15.28:0/1382463106 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc4b40 0x560478fd8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.144+0000 7f635627e740 1 -- 172.21.15.28:0/1382463106 >> 172.21.15.28:0/1382463106 conn(0x560478fc0e60 msgr2=0x560478fbf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.145+0000 7f635627e740 1 -- 172.21.15.28:0/1382463106 shutdown_connections 2024-04-24T08:18:48.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.145+0000 7f635627e740 1 -- 172.21.15.28:0/1382463106 wait complete. 2024-04-24T08:18:48.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.145+0000 7f635627e740 1 -- start start 2024-04-24T08:18:48.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.145+0000 7f635627e740 1 Processor -- start 2024-04-24T08:18:48.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.145+0000 7f635627e740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc4b40 0x560478fd0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.145+0000 7f635627e740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fc5460 0x560478fd0640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.145+0000 7f635627e740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fcfb30 0x560478fd0d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.146+0000 7f635627e740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1ac330 con 0x560478fcfb30 2024-04-24T08:18:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.146+0000 7f635627e740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1ad7a0 con 0x560478fc5460 2024-04-24T08:18:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.146+0000 7f635627e740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f634c1aca00 con 0x560478fc4b40 2024-04-24T08:18:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.146+0000 7f635627e740 10 client.0 _ll_register_callbacks cb 0x7f6338009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.146+0000 7f634be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fc5460 0x560478fd0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.146+0000 7f6351400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fcfb30 0x560478fd0d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.146+0000 7f634be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fc5460 0x560478fd0640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41762/0 (socket says 172.21.15.28:41762) 2024-04-24T08:18:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.146+0000 7f634be00640 1 -- 172.21.15.28:0/1785479057 learned_addr learned my addr 172.21.15.28:0/1785479057 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:48.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.146+0000 7f6351400640 1 --2- 172.21.15.28:0/1785479057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc4b40 0x560478fd0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.146+0000 7f634be00640 1 -- 172.21.15.28:0/1785479057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc4b40 msgr2=0x560478fd0100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.146+0000 7f634be00640 1 --2- 172.21.15.28:0/1785479057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc4b40 0x560478fd0100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.146+0000 7f634be00640 1 -- 172.21.15.28:0/1785479057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fcfb30 msgr2=0x560478fd0d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.146+0000 7f634be00640 1 --2- 172.21.15.28:0/1785479057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fcfb30 0x560478fd0d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.146+0000 7f634be00640 1 -- 172.21.15.28:0/1785479057 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560478fd15b0 con 0x560478fc5460 2024-04-24T08:18:48.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.146+0000 7f6351400640 1 --2- 172.21.15.28:0/1785479057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fcfb30 0x560478fd0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:48.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.147+0000 7f634be00640 1 --2- 172.21.15.28:0/1785479057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fc5460 0x560478fd0640 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6338005110 tx=0x7f6338010dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1bb73a20c5b60e9f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.153 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.147+0000 7f6332000640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:18:48.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.147+0000 7f6332000640 1 -- 172.21.15.28:0/1785479057 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6338028030 con 0x560478fc5460 2024-04-24T08:18:48.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.147+0000 7f6332000640 1 -- 172.21.15.28:0/1785479057 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7f6338005480 con 0x560478fc5460 2024-04-24T08:18:48.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.147+0000 7f635627e740 1 -- 172.21.15.28:0/1785479057 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f633800f920 con 0x560478fc5460 2024-04-24T08:18:48.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.147+0000 7f6332000640 1 -- 172.21.15.28:0/1785479057 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6338005680 con 0x560478fc5460 2024-04-24T08:18:48.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.147+0000 7f635627e740 10 client.26654 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:48.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.147+0000 7f635627e740 1 -- 172.21.15.28:0/1785479057 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x56047904acc0 con 0x560478fc5460 2024-04-24T08:18:48.154 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.148+0000 7f6332000640 1 -- 172.21.15.28:0/1785479057 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f633800f920 con 0x560478fc5460 2024-04-24T08:18:48.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.148+0000 7f6332000640 1 -- 172.21.15.28:0/1785479057 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f633801c4e0 con 0x560478fc5460 2024-04-24T08:18:48.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.148+0000 7f635627e740 10 client.26654 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:48.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.148+0000 7f635627e740 10 client.26654 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:18:48.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.148+0000 7f635627e740 20 client.26654 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:48.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.148+0000 7f635627e740 1 --2- 172.21.15.28:0/1785479057 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x56047904b220 0x56047904d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.156 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.148+0000 7f635627e740 4 client.26654 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:18:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.148+0000 7f635627e740 1 -- 172.21.15.28:0/1785479057 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560478f31a60 con 0x56047904b220 2024-04-24T08:18:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.148+0000 7f6351400640 1 --2- 172.21.15.28:0/1785479057 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x56047904b220 0x56047904d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.149+0000 7f6351400640 1 --2- 172.21.15.28:0/1785479057 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x56047904b220 0x56047904d610 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=11536e7e33a8bdd8 server_cookie=d9805a1a8d390ac0 in_seq=0 out_seq=0 2024-04-24T08:18:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.149+0000 7f6332000640 10 client.26654 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:18:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.149+0000 7f6332000640 1 -- 172.21.15.28:0/1785479057 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560478f31a60 con 0x56047904b220 2024-04-24T08:18:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.149+0000 7f6332000640 10 client.26654 handle_command_reply: tid=0 2024-04-24T08:18:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.149+0000 7f635627e740 1 client.26654 shutdown 2024-04-24T08:18:48.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.150+0000 7f635627e740 1 -- 172.21.15.28:0/1785479057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fc5460 msgr2=0x560478fd0640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.150+0000 7f635627e740 1 --2- 172.21.15.28:0/1785479057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fc5460 0x560478fd0640 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6338005110 tx=0x7f6338010dc0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.150+0000 7f635627e740 1 -- 172.21.15.28:0/1785479057 shutdown_connections 2024-04-24T08:18:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.150+0000 7f635627e740 1 --2- 172.21.15.28:0/1785479057 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x56047904b220 0x56047904d610 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.150+0000 7f635627e740 1 --2- 172.21.15.28:0/1785479057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560478fcfb30 0x560478fd0d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.150+0000 7f635627e740 1 --2- 172.21.15.28:0/1785479057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560478fc5460 0x560478fd0640 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.150+0000 7f635627e740 1 --2- 172.21.15.28:0/1785479057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560478fc4b40 0x560478fd0100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.150+0000 7f635627e740 1 -- 172.21.15.28:0/1785479057 >> 172.21.15.28:0/1785479057 conn(0x560478fc0e60 msgr2=0x560478fbf6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.150+0000 7f6332000640 0 client.26654 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:18:48.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.150+0000 7f635627e740 1 -- 172.21.15.28:0/1785479057 shutdown_connections 2024-04-24T08:18:48.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.150+0000 7f635627e740 1 -- 172.21.15.28:0/1785479057 wait complete. 2024-04-24T08:18:48.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.150+0000 7f635627e740 20 client.26654 trim_cache size 0 max 16384 2024-04-24T08:18:48.159 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:18:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.151+0000 7f6351e00640 1 -- 172.21.15.28:0/1409488848 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f63340a3d20 msgr2=0x7f63340a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.151+0000 7f6351e00640 1 --2- 172.21.15.28:0/1409488848 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f63340a3d20 0x7f63340a61e0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x560478fc6100 tx=0x7f634003b890 comp rx=0 tx=0).stop 2024-04-24T08:18:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.151+0000 7f6351e00640 1 -- 172.21.15.28:0/1409488848 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f634c102120 msgr2=0x7f634c19e430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.151+0000 7f6351e00640 1 --2- 172.21.15.28:0/1409488848 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f634c102120 0x7f634c19e430 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f6340004770 tx=0x7f6340003c10 comp rx=0 tx=0).stop 2024-04-24T08:18:48.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.151+0000 7f6351e00640 1 -- 172.21.15.28:0/1409488848 shutdown_connections 2024-04-24T08:18:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.151+0000 7f6351e00640 1 --2- 172.21.15.28:0/1409488848 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f63340a3d20 0x7f63340a61e0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.151+0000 7f6351e00640 1 --2- 172.21.15.28:0/1409488848 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f634c1071f0 0x7f634c19e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.151+0000 7f6351e00640 1 --2- 172.21.15.28:0/1409488848 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f634c198630 0x7f634c198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.151+0000 7f6351e00640 1 --2- 172.21.15.28:0/1409488848 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f634c102120 0x7f634c19e430 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.151+0000 7f6351e00640 1 -- 172.21.15.28:0/1409488848 >> 172.21.15.28:0/1409488848 conn(0x7f634c0fd9a0 msgr2=0x7f634c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.151+0000 7f6351e00640 1 -- 172.21.15.28:0/1409488848 shutdown_connections 2024-04-24T08:18:48.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.151+0000 7f6351e00640 1 -- 172.21.15.28:0/1409488848 wait complete. 2024-04-24T08:18:48.171 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:18:48.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.375+0000 7f934e200640 1 -- 172.21.15.28:0/3076919872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9348100cd0 msgr2=0x7f934810e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.375+0000 7f934e200640 1 --2- 172.21.15.28:0/3076919872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9348100cd0 0x7f934810e3b0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f9330009900 tx=0x7f933002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.375+0000 7f934e200640 1 -- 172.21.15.28:0/3076919872 shutdown_connections 2024-04-24T08:18:48.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.375+0000 7f934e200640 1 --2- 172.21.15.28:0/3076919872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f934810e8f0 0x7f9348110ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.375+0000 7f934e200640 1 --2- 172.21.15.28:0/3076919872 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9348100cd0 0x7f934810e3b0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.375+0000 7f934e200640 1 --2- 172.21.15.28:0/3076919872 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f93481003b0 0x7f9348100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.375+0000 7f934e200640 1 -- 172.21.15.28:0/3076919872 >> 172.21.15.28:0/3076919872 conn(0x7f9348075d00 msgr2=0x7f9348076110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.375+0000 7f934e200640 1 -- 172.21.15.28:0/3076919872 shutdown_connections 2024-04-24T08:18:48.378 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.376+0000 7f934e200640 1 -- 172.21.15.28:0/3076919872 wait complete. 2024-04-24T08:18:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.377+0000 7f934e200640 1 Processor -- start 2024-04-24T08:18:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.377+0000 7f934e200640 1 -- start start 2024-04-24T08:18:48.380 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.378+0000 7f934e200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93481003b0 0x7f93481a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.378+0000 7f934e200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9348100cd0 0x7f93481a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.378+0000 7f934e200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f934810e8f0 0x7f934819a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.378+0000 7f934e200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f93481144f0 con 0x7f934810e8f0 2024-04-24T08:18:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.378+0000 7f934e200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9348114370 con 0x7f93481003b0 2024-04-24T08:18:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.378+0000 7f934e200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9348114670 con 0x7f9348100cd0 2024-04-24T08:18:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.378+0000 7f934d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f934810e8f0 0x7f934819a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.378+0000 7f933fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9348100cd0 0x7f93481a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.378+0000 7f934ce00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93481003b0 0x7f93481a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.378+0000 7f934ce00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93481003b0 0x7f93481a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41768/0 (socket says 172.21.15.28:41768) 2024-04-24T08:18:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.378+0000 7f934d800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f934810e8f0 0x7f934819a7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54728/0 (socket says 172.21.15.28:54728) 2024-04-24T08:18:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.378+0000 7f934d800640 1 -- 172.21.15.28:0/1540236399 learned_addr learned my addr 172.21.15.28:0/1540236399 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.378+0000 7f933fe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9348100cd0 0x7f93481a0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:46210/0 (socket says 172.21.15.28:46210) 2024-04-24T08:18:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.379+0000 7f934d800640 1 -- 172.21.15.28:0/1540236399 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9348100cd0 msgr2=0x7f93481a0be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.379+0000 7f934d800640 1 --2- 172.21.15.28:0/1540236399 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9348100cd0 0x7f93481a0be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.379+0000 7f934d800640 1 -- 172.21.15.28:0/1540236399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93481003b0 msgr2=0x7f93481a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.379+0000 7f934d800640 1 --2- 172.21.15.28:0/1540236399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93481003b0 0x7f93481a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.379+0000 7f934d800640 1 -- 172.21.15.28:0/1540236399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f934819b070 con 0x7f934810e8f0 2024-04-24T08:18:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.379+0000 7f933fe00640 1 --2- 172.21.15.28:0/1540236399 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9348100cd0 0x7f93481a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.379+0000 7f934ce00640 1 --2- 172.21.15.28:0/1540236399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93481003b0 0x7f93481a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:48.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.379+0000 7f934d800640 1 --2- 172.21.15.28:0/1540236399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f934810e8f0 0x7f934819a7b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f934000e970 tx=0x7f934000ee30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.379+0000 7f9346000640 1 -- 172.21.15.28:0/1540236399 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f934000ccc0 con 0x7f934810e8f0 2024-04-24T08:18:48.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.379+0000 7f9346000640 1 -- 172.21.15.28:0/1540236399 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9340005500 con 0x7f934810e8f0 2024-04-24T08:18:48.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.379+0000 7f934e200640 1 -- 172.21.15.28:0/1540236399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f934819b360 con 0x7f934810e8f0 2024-04-24T08:18:48.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.379+0000 7f9346000640 1 -- 172.21.15.28:0/1540236399 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9340010690 con 0x7f934810e8f0 2024-04-24T08:18:48.384 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.380+0000 7f934e200640 1 -- 172.21.15.28:0/1540236399 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f93481a7620 con 0x7f934810e8f0 2024-04-24T08:18:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.383+0000 7f93525c6740 1 Processor -- start 2024-04-24T08:18:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.384+0000 7f9346000640 1 -- 172.21.15.28:0/1540236399 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f93400046c0 con 0x7f934810e8f0 2024-04-24T08:18:48.385 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.384+0000 7f93525c6740 1 -- start start 2024-04-24T08:18:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.384+0000 7f93525c6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a6b40 0x55dfd78a6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.385+0000 7f93525c6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78a7460 0x55dfd78b15f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.385+0000 7f93525c6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78b1b30 0x55dfd78b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.385+0000 7f93525c6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dfd780c1d0 con 0x55dfd78a6b40 2024-04-24T08:18:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.385+0000 7f93525c6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55dfd77b2450 con 0x55dfd78b1b30 2024-04-24T08:18:48.387 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.385+0000 7f93525c6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55dfd7808190 con 0x55dfd78a7460 2024-04-24T08:18:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.385+0000 7f933fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a6b40 0x55dfd78a6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.385+0000 7f933fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a6b40 0x55dfd78a6f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54730/0 (socket says 172.21.15.28:54730) 2024-04-24T08:18:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.385+0000 7f933fe00640 1 -- 172.21.15.28:0/3821860926 learned_addr learned my addr 172.21.15.28:0/3821860926 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.385+0000 7f934ce00640 1 --2- 172.21.15.28:0/3821860926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78a7460 0x55dfd78b15f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.385+0000 7f933fe00640 1 --2- 172.21.15.28:0/3821860926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78b1b30 0x55dfd78b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.385+0000 7f934ce00640 1 -- 172.21.15.28:0/3821860926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78b1b30 msgr2=0x55dfd78b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.385+0000 7f934ce00640 1 --2- 172.21.15.28:0/3821860926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78b1b30 0x55dfd78b3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.388 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.385+0000 7f934ce00640 1 -- 172.21.15.28:0/3821860926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a6b40 msgr2=0x55dfd78a6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.385+0000 7f934ce00640 1 --2- 172.21.15.28:0/3821860926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a6b40 0x55dfd78a6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.389 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.385+0000 7f934ce00640 1 -- 172.21.15.28:0/3821860926 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfd786f310 con 0x55dfd78a7460 2024-04-24T08:18:48.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.386+0000 7f934ce00640 1 --2- 172.21.15.28:0/3821860926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78a7460 0x55dfd78b15f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9338005fd0 tx=0x7f9338011ac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9d277de0a2d6f2c9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.386+0000 7f933e000640 1 -- 172.21.15.28:0/3821860926 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933800f950 con 0x55dfd78a7460 2024-04-24T08:18:48.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.386+0000 7f933e000640 1 -- 172.21.15.28:0/3821860926 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9338012450 con 0x55dfd78a7460 2024-04-24T08:18:48.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.386+0000 7f933e000640 1 -- 172.21.15.28:0/3821860926 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933801a650 con 0x55dfd78a7460 2024-04-24T08:18:48.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.386+0000 7f9346000640 1 --2- 172.21.15.28:0/1540236399 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f932c0a3d20 0x7f932c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.393 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.386+0000 7f9346000640 1 -- 172.21.15.28:0/1540236399 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7f93400d0c10 con 0x7f934810e8f0 2024-04-24T08:18:48.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.387+0000 7f933fe00640 1 --2- 172.21.15.28:0/1540236399 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f932c0a3d20 0x7f932c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.387+0000 7f93525c6740 1 -- 172.21.15.28:0/3821860926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78a7460 msgr2=0x55dfd78b15f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.387+0000 7f93525c6740 1 --2- 172.21.15.28:0/3821860926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78a7460 0x55dfd78b15f0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f9338005fd0 tx=0x7f9338011ac0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.387+0000 7f93525c6740 1 -- 172.21.15.28:0/3821860926 shutdown_connections 2024-04-24T08:18:48.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.387+0000 7f93525c6740 1 --2- 172.21.15.28:0/3821860926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78b1b30 0x55dfd78b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.387+0000 7f93525c6740 1 --2- 172.21.15.28:0/3821860926 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78a7460 0x55dfd78b15f0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.387+0000 7f93525c6740 1 --2- 172.21.15.28:0/3821860926 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a6b40 0x55dfd78a6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.394 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.387+0000 7f93525c6740 1 -- 172.21.15.28:0/3821860926 >> 172.21.15.28:0/3821860926 conn(0x55dfd78a2e60 msgr2=0x55dfd78a3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.387+0000 7f93525c6740 1 -- 172.21.15.28:0/3821860926 shutdown_connections 2024-04-24T08:18:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.387+0000 7f933fe00640 1 --2- 172.21.15.28:0/1540236399 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f932c0a3d20 0x7f932c0a61e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x55dfd78a3ff0 tx=0x7f9330041040 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.387+0000 7f93525c6740 1 -- 172.21.15.28:0/3821860926 wait complete. 2024-04-24T08:18:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.388+0000 7f93525c6740 1 -- start start 2024-04-24T08:18:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.388+0000 7f93525c6740 1 Processor -- start 2024-04-24T08:18:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.388+0000 7f93525c6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78a6b40 0x55dfd78b14e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.388+0000 7f93525c6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a7460 0x55dfd78ae7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.388+0000 7f93525c6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78b1b30 0x55dfd78aed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.388+0000 7f93525c6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55dfd78b49c0 con 0x55dfd78a7460 2024-04-24T08:18:48.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.388+0000 7f93525c6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55dfd77c5810 con 0x55dfd78b1b30 2024-04-24T08:18:48.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.388+0000 7f93525c6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55dfd78b4b40 con 0x55dfd78a6b40 2024-04-24T08:18:48.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.388+0000 7f93525c6740 10 client.0 _ll_register_callbacks cb 0x55dfd7793140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:48.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.389+0000 7f934ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78a6b40 0x55dfd78b14e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.389+0000 7f933fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a7460 0x55dfd78ae7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.389+0000 7f934ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78a6b40 0x55dfd78b14e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:46230/0 (socket says 172.21.15.28:46230) 2024-04-24T08:18:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.389+0000 7f934ce00640 1 -- 172.21.15.28:0/2849767374 learned_addr learned my addr 172.21.15.28:0/2849767374 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.389+0000 7f934d800640 1 --2- 172.21.15.28:0/2849767374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78b1b30 0x55dfd78aed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.389+0000 7f934ce00640 1 -- 172.21.15.28:0/2849767374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78b1b30 msgr2=0x55dfd78aed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.389+0000 7f934ce00640 1 --2- 172.21.15.28:0/2849767374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78b1b30 0x55dfd78aed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.389+0000 7f934ce00640 1 -- 172.21.15.28:0/2849767374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a7460 msgr2=0x55dfd78ae7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.389+0000 7f934ce00640 1 --2- 172.21.15.28:0/2849767374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a7460 0x55dfd78ae7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.389+0000 7f934ce00640 1 -- 172.21.15.28:0/2849767374 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfd77d63f0 con 0x55dfd78a6b40 2024-04-24T08:18:48.397 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.389+0000 7f933fe00640 1 --2- 172.21.15.28:0/2849767374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a7460 0x55dfd78ae7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:48.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.390+0000 7f934ce00640 1 --2- 172.21.15.28:0/2849767374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78a6b40 0x55dfd78b14e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9338000f80 tx=0x7f9338012830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5bb22eb62f6a48e9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.390+0000 7f9336a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:18:48.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.390+0000 7f9336a00640 1 -- 172.21.15.28:0/2849767374 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9338027020 con 0x55dfd78a6b40 2024-04-24T08:18:48.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.390+0000 7f93525c6740 1 -- 172.21.15.28:0/2849767374 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9338012450 con 0x55dfd78a6b40 2024-04-24T08:18:48.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.390+0000 7f9336a00640 1 -- 172.21.15.28:0/2849767374 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7f933800fb60 con 0x55dfd78a6b40 2024-04-24T08:18:48.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.390+0000 7f9336a00640 1 -- 172.21.15.28:0/2849767374 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9338032050 con 0x55dfd78a6b40 2024-04-24T08:18:48.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.390+0000 7f93525c6740 10 client.26689 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.390+0000 7f93525c6740 1 -- 172.21.15.28:0/2849767374 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55dfd751e3f0 con 0x55dfd78a6b40 2024-04-24T08:18:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.390+0000 7f9336a00640 1 -- 172.21.15.28:0/2849767374 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9338012450 con 0x55dfd78a6b40 2024-04-24T08:18:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.391+0000 7f9336a00640 1 -- 172.21.15.28:0/2849767374 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f933800f040 con 0x55dfd78a6b40 2024-04-24T08:18:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.391+0000 7f93525c6740 10 client.26689 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.391+0000 7f93525c6740 10 client.26689 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:18:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.391+0000 7f93525c6740 20 client.26689 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.391+0000 7f93525c6740 1 --2- 172.21.15.28:0/2849767374 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55dfd78c1c00 0x55dfd78c3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.391+0000 7f93525c6740 4 client.26689 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:18:48.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.391+0000 7f93525c6740 1 -- 172.21.15.28:0/2849767374 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dfd7819bf0 con 0x55dfd78c1c00 2024-04-24T08:18:48.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.391+0000 7f933fe00640 1 --2- 172.21.15.28:0/2849767374 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55dfd78c1c00 0x55dfd78c3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.392+0000 7f933fe00640 1 --2- 172.21.15.28:0/2849767374 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55dfd78c1c00 0x55dfd78c3ff0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=4562a1928731f99e server_cookie=a66832126bc0a8bb in_seq=0 out_seq=0 2024-04-24T08:18:48.400 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.392+0000 7f9336a00640 10 client.26689 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:18:48.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.393+0000 7f9336a00640 1 -- 172.21.15.28:0/2849767374 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dfd7819bf0 con 0x55dfd78c1c00 2024-04-24T08:18:48.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.393+0000 7f9336a00640 10 client.26689 handle_command_reply: tid=0 2024-04-24T08:18:48.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.393+0000 7f93525c6740 1 client.26689 shutdown 2024-04-24T08:18:48.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.393+0000 7f93525c6740 1 -- 172.21.15.28:0/2849767374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78a6b40 msgr2=0x55dfd78b14e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.393+0000 7f93525c6740 1 --2- 172.21.15.28:0/2849767374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78a6b40 0x55dfd78b14e0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f9338000f80 tx=0x7f9338012830 comp rx=0 tx=0).stop 2024-04-24T08:18:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.393+0000 7f93525c6740 1 -- 172.21.15.28:0/2849767374 shutdown_connections 2024-04-24T08:18:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.393+0000 7f93525c6740 1 --2- 172.21.15.28:0/2849767374 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55dfd78c1c00 0x55dfd78c3ff0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.393+0000 7f93525c6740 1 --2- 172.21.15.28:0/2849767374 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a7460 0x55dfd78ae7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.393+0000 7f93525c6740 1 --2- 172.21.15.28:0/2849767374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78b1b30 0x55dfd78aed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.393+0000 7f9336a00640 0 client.26689 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:18:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.393+0000 7f93525c6740 1 --2- 172.21.15.28:0/2849767374 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78a6b40 0x55dfd78b14e0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.393+0000 7f93525c6740 1 -- 172.21.15.28:0/2849767374 >> 172.21.15.28:0/2849767374 conn(0x55dfd78a2e60 msgr2=0x55dfd78a8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.393+0000 7f93525c6740 1 -- 172.21.15.28:0/2849767374 shutdown_connections 2024-04-24T08:18:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.393+0000 7f93525c6740 1 -- 172.21.15.28:0/2849767374 wait complete. 2024-04-24T08:18:48.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.393+0000 7f93525c6740 20 client.26689 trim_cache size 0 max 16384 2024-04-24T08:18:48.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.408+0000 7f93525c6740 1 Processor -- start 2024-04-24T08:18:48.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.408+0000 7f93525c6740 1 -- start start 2024-04-24T08:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.408+0000 7f93525c6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78a6b40 0x55dfd78a8de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.408+0000 7f93525c6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a7460 0x55dfd78ad800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.408+0000 7f93525c6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78b1b30 0x55dfd78b1010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.408+0000 7f93525c6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f93481b27a0 con 0x55dfd78a7460 2024-04-24T08:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.408+0000 7f93525c6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f93481b0070 con 0x55dfd78a6b40 2024-04-24T08:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.408+0000 7f93525c6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f93481adee0 con 0x55dfd78b1b30 2024-04-24T08:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.409+0000 7f934ce00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78a6b40 0x55dfd78a8de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.409+0000 7f934ce00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78a6b40 0x55dfd78a8de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:41798/0 (socket says 172.21.15.28:41798) 2024-04-24T08:18:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.409+0000 7f934ce00640 1 -- 172.21.15.28:0/549993196 learned_addr learned my addr 172.21.15.28:0/549993196 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.409+0000 7f933fe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a7460 0x55dfd78ad800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.409+0000 7f934d800640 1 --2- 172.21.15.28:0/549993196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78b1b30 0x55dfd78b1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.409+0000 7f934ce00640 1 -- 172.21.15.28:0/549993196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78b1b30 msgr2=0x55dfd78b1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.409+0000 7f934ce00640 1 --2- 172.21.15.28:0/549993196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78b1b30 0x55dfd78b1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.409+0000 7f934ce00640 1 -- 172.21.15.28:0/549993196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a7460 msgr2=0x55dfd78ad800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.409+0000 7f934ce00640 1 --2- 172.21.15.28:0/549993196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a7460 0x55dfd78ad800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.409+0000 7f934ce00640 1 -- 172.21.15.28:0/549993196 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dfd78af0e0 con 0x55dfd78a6b40 2024-04-24T08:18:48.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.409+0000 7f934d800640 1 --2- 172.21.15.28:0/549993196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78b1b30 0x55dfd78b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:48.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.409+0000 7f934ce00640 1 --2- 172.21.15.28:0/549993196 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78a6b40 0x55dfd78a8de0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9338009960 tx=0x7f9338005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7964806e04c72e8f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.410+0000 7f933d600640 1 -- 172.21.15.28:0/549993196 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933803b070 con 0x55dfd78a6b40 2024-04-24T08:18:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.410+0000 7f933d600640 1 -- 172.21.15.28:0/549993196 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9338012450 con 0x55dfd78a6b40 2024-04-24T08:18:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.410+0000 7f933d600640 1 -- 172.21.15.28:0/549993196 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9338027040 con 0x55dfd78a6b40 2024-04-24T08:18:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.410+0000 7f93525c6740 1 -- 172.21.15.28:0/549993196 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78a6b40 msgr2=0x55dfd78a8de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.410+0000 7f93525c6740 1 --2- 172.21.15.28:0/549993196 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78a6b40 0x55dfd78a8de0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f9338009960 tx=0x7f9338005f10 comp rx=0 tx=0).stop 2024-04-24T08:18:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.410+0000 7f93525c6740 1 -- 172.21.15.28:0/549993196 shutdown_connections 2024-04-24T08:18:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.410+0000 7f93525c6740 1 --2- 172.21.15.28:0/549993196 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a7460 0x55dfd78ad800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.410+0000 7f93525c6740 1 --2- 172.21.15.28:0/549993196 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78b1b30 0x55dfd78b1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.410+0000 7f93525c6740 1 --2- 172.21.15.28:0/549993196 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78a6b40 0x55dfd78a8de0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.410+0000 7f93525c6740 1 -- 172.21.15.28:0/549993196 >> 172.21.15.28:0/549993196 conn(0x55dfd78a2e60 msgr2=0x55dfd78a3240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.411+0000 7f93525c6740 1 -- 172.21.15.28:0/549993196 shutdown_connections 2024-04-24T08:18:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.411+0000 7f93525c6740 1 -- 172.21.15.28:0/549993196 wait complete. 2024-04-24T08:18:48.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.411+0000 7f93525c6740 1 -- start start 2024-04-24T08:18:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.411+0000 7f93525c6740 1 Processor -- start 2024-04-24T08:18:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.412+0000 7f93525c6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a6b40 0x55dfd78ba1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.412+0000 7f93525c6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78a7460 0x55dfd78b24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.412+0000 7f93525c6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78b1b30 0x55dfd78b2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.412+0000 7f93525c6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f93481ae530 con 0x55dfd78a6b40 2024-04-24T08:18:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.412+0000 7f93525c6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f93481af9a0 con 0x55dfd78a7460 2024-04-24T08:18:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.412+0000 7f93525c6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f93481aec00 con 0x55dfd78b1b30 2024-04-24T08:18:48.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.412+0000 7f93525c6740 10 client.0 _ll_register_callbacks cb 0x7f9330009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.412+0000 7f934ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a6b40 0x55dfd78ba1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.412+0000 7f934ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a6b40 0x55dfd78ba1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54766/0 (socket says 172.21.15.28:54766) 2024-04-24T08:18:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.412+0000 7f934ce00640 1 -- 172.21.15.28:0/3655952696 learned_addr learned my addr 172.21.15.28:0/3655952696 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.412+0000 7f933fe00640 1 --2- 172.21.15.28:0/3655952696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78a7460 0x55dfd78b24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.412+0000 7f934d800640 1 --2- 172.21.15.28:0/3655952696 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78b1b30 0x55dfd78b2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.412+0000 7f934ce00640 1 -- 172.21.15.28:0/3655952696 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78b1b30 msgr2=0x55dfd78b2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.412+0000 7f934ce00640 1 --2- 172.21.15.28:0/3655952696 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78b1b30 0x55dfd78b2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.412+0000 7f934ce00640 1 -- 172.21.15.28:0/3655952696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78a7460 msgr2=0x55dfd78b24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.412+0000 7f934ce00640 1 --2- 172.21.15.28:0/3655952696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78a7460 0x55dfd78b24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.412+0000 7f934ce00640 1 -- 172.21.15.28:0/3655952696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dfd75e87d0 con 0x55dfd78a6b40 2024-04-24T08:18:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.413+0000 7f934ce00640 1 --2- 172.21.15.28:0/3655952696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a6b40 0x55dfd78ba1d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9338005b40 tx=0x7f9338005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c46a1af112952f4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.413+0000 7f9336a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:18:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.413+0000 7f9336a00640 1 -- 172.21.15.28:0/3655952696 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f933803b070 con 0x55dfd78a6b40 2024-04-24T08:18:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.413+0000 7f9336a00640 1 -- 172.21.15.28:0/3655952696 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7f933800fbd0 con 0x55dfd78a6b40 2024-04-24T08:18:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.413+0000 7f93525c6740 1 -- 172.21.15.28:0/3655952696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9338012450 con 0x55dfd78a6b40 2024-04-24T08:18:48.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.413+0000 7f9336a00640 1 -- 172.21.15.28:0/3655952696 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9338047050 con 0x55dfd78a6b40 2024-04-24T08:18:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.413+0000 7f93525c6740 10 client.16395 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.413+0000 7f93525c6740 1 -- 172.21.15.28:0/3655952696 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55dfd78b9d10 con 0x55dfd78a6b40 2024-04-24T08:18:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.413+0000 7f9336a00640 1 -- 172.21.15.28:0/3655952696 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9338012450 con 0x55dfd78a6b40 2024-04-24T08:18:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.414+0000 7f9336a00640 1 -- 172.21.15.28:0/3655952696 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f933802ec30 con 0x55dfd78a6b40 2024-04-24T08:18:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.414+0000 7f93525c6740 10 client.16395 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.414+0000 7f93525c6740 10 client.16395 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:18:48.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.414+0000 7f93525c6740 20 client.16395 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.414+0000 7f93525c6740 1 --2- 172.21.15.28:0/3655952696 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55dfd7928260 0x55dfd792a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.414+0000 7f93525c6740 4 client.16395 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:18:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.414+0000 7f93525c6740 1 -- 172.21.15.28:0/3655952696 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55dfd7813a60 con 0x55dfd7928260 2024-04-24T08:18:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.414+0000 7f933fe00640 1 --2- 172.21.15.28:0/3655952696 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55dfd7928260 0x55dfd792a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.415+0000 7f933fe00640 1 --2- 172.21.15.28:0/3655952696 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55dfd7928260 0x55dfd792a680 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f69224c38f127a11 server_cookie=8bcc024dce28add7 in_seq=0 out_seq=0 2024-04-24T08:18:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.415+0000 7f9336a00640 10 client.16395 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:18:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.415+0000 7f9336a00640 1 -- 172.21.15.28:0/3655952696 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55dfd7813a60 con 0x55dfd7928260 2024-04-24T08:18:48.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.415+0000 7f9336a00640 10 client.16395 handle_command_reply: tid=0 2024-04-24T08:18:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.415+0000 7f93525c6740 1 client.16395 shutdown 2024-04-24T08:18:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.415+0000 7f93525c6740 1 -- 172.21.15.28:0/3655952696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a6b40 msgr2=0x55dfd78ba1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.415+0000 7f93525c6740 1 --2- 172.21.15.28:0/3655952696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a6b40 0x55dfd78ba1d0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9338005b40 tx=0x7f9338005f80 comp rx=0 tx=0).stop 2024-04-24T08:18:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.415+0000 7f93525c6740 1 -- 172.21.15.28:0/3655952696 shutdown_connections 2024-04-24T08:18:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.415+0000 7f93525c6740 1 --2- 172.21.15.28:0/3655952696 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55dfd7928260 0x55dfd792a680 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.415+0000 7f93525c6740 1 --2- 172.21.15.28:0/3655952696 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55dfd78b1b30 0x55dfd78b2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.415+0000 7f93525c6740 1 --2- 172.21.15.28:0/3655952696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55dfd78a7460 0x55dfd78b24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.415+0000 7f93525c6740 1 --2- 172.21.15.28:0/3655952696 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55dfd78a6b40 0x55dfd78ba1d0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.415+0000 7f9336a00640 0 client.16395 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:18:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.415+0000 7f93525c6740 1 -- 172.21.15.28:0/3655952696 >> 172.21.15.28:0/3655952696 conn(0x55dfd78a2e60 msgr2=0x55dfd78a1f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.415+0000 7f93525c6740 1 -- 172.21.15.28:0/3655952696 shutdown_connections 2024-04-24T08:18:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.416+0000 7f93525c6740 1 -- 172.21.15.28:0/3655952696 wait complete. 2024-04-24T08:18:48.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.416+0000 7f93525c6740 20 client.16395 trim_cache size 0 max 16384 2024-04-24T08:18:48.425 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:18:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.416+0000 7f934e200640 1 -- 172.21.15.28:0/1540236399 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f932c0a3d20 msgr2=0x7f932c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.416+0000 7f934e200640 1 --2- 172.21.15.28:0/1540236399 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f932c0a3d20 0x7f932c0a61e0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x55dfd78a3ff0 tx=0x7f9330041040 comp rx=0 tx=0).stop 2024-04-24T08:18:48.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.416+0000 7f934e200640 1 -- 172.21.15.28:0/1540236399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f934810e8f0 msgr2=0x7f934819a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.416+0000 7f934e200640 1 --2- 172.21.15.28:0/1540236399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f934810e8f0 0x7f934819a7b0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f934000e970 tx=0x7f934000ee30 comp rx=0 tx=0).stop 2024-04-24T08:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.416+0000 7f934e200640 1 -- 172.21.15.28:0/1540236399 shutdown_connections 2024-04-24T08:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.416+0000 7f934e200640 1 --2- 172.21.15.28:0/1540236399 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f932c0a3d20 0x7f932c0a61e0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.416+0000 7f934e200640 1 --2- 172.21.15.28:0/1540236399 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f934810e8f0 0x7f934819a7b0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.416+0000 7f934e200640 1 --2- 172.21.15.28:0/1540236399 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9348100cd0 0x7f93481a0be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.416+0000 7f934e200640 1 --2- 172.21.15.28:0/1540236399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93481003b0 0x7f93481a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.416+0000 7f934e200640 1 -- 172.21.15.28:0/1540236399 >> 172.21.15.28:0/1540236399 conn(0x7f9348075d00 msgr2=0x7f934810ee10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.417+0000 7f934e200640 1 -- 172.21.15.28:0/1540236399 shutdown_connections 2024-04-24T08:18:48.428 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.417+0000 7f934e200640 1 -- 172.21.15.28:0/1540236399 wait complete. 2024-04-24T08:18:48.435 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:18:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.628+0000 7f4e3be00640 1 -- 172.21.15.28:0/3673655326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e3c10d500 msgr2=0x7f4e3c10d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.630 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.628+0000 7f4e3be00640 1 --2- 172.21.15.28:0/3673655326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e3c10d500 0x7f4e3c10d8e0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c009930 tx=0x7f4e2c02a420 comp rx=0 tx=0).stop 2024-04-24T08:18:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.629+0000 7f4e3be00640 1 -- 172.21.15.28:0/3673655326 shutdown_connections 2024-04-24T08:18:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.629+0000 7f4e3be00640 1 --2- 172.21.15.28:0/3673655326 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e3c108f40 0x7f4e3c1153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.629+0000 7f4e3be00640 1 --2- 172.21.15.28:0/3673655326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e3c108580 0x7f4e3c108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.629+0000 7f4e3be00640 1 --2- 172.21.15.28:0/3673655326 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e3c10d500 0x7f4e3c10d8e0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.629+0000 7f4e3be00640 1 -- 172.21.15.28:0/3673655326 >> 172.21.15.28:0/3673655326 conn(0x7f4e3c077e20 msgr2=0x7f4e3c106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.629+0000 7f4e3be00640 1 -- 172.21.15.28:0/3673655326 shutdown_connections 2024-04-24T08:18:48.631 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.629+0000 7f4e3be00640 1 -- 172.21.15.28:0/3673655326 wait complete. 2024-04-24T08:18:48.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.630+0000 7f4e3be00640 1 Processor -- start 2024-04-24T08:18:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.630+0000 7f4e3be00640 1 -- start start 2024-04-24T08:18:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.630+0000 7f4e3be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e3c108580 0x7f4e3c19ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.630+0000 7f4e3be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e3c108f40 0x7f4e3c19f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.630+0000 7f4e3be00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e3c10d500 0x7f4e3c1a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.630+0000 7f4e3be00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c118900 con 0x7f4e3c108580 2024-04-24T08:18:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.630+0000 7f4e3be00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c118780 con 0x7f4e3c108f40 2024-04-24T08:18:48.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.630+0000 7f4e3be00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c118a80 con 0x7f4e3c10d500 2024-04-24T08:18:48.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.630+0000 7f4e3aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e3c108580 0x7f4e3c19ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.630+0000 7f4e3aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e3c108580 0x7f4e3c19ed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54776/0 (socket says 172.21.15.28:54776) 2024-04-24T08:18:48.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.630+0000 7f4e3aa00640 1 -- 172.21.15.28:0/313628551 learned_addr learned my addr 172.21.15.28:0/313628551 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:48.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.630+0000 7f4e3a000640 1 --2- 172.21.15.28:0/313628551 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e3c108f40 0x7f4e3c19f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.630+0000 7f4e3b400640 1 --2- 172.21.15.28:0/313628551 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e3c10d500 0x7f4e3c1a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.631+0000 7f4e3aa00640 1 -- 172.21.15.28:0/313628551 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e3c10d500 msgr2=0x7f4e3c1a35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.631+0000 7f4e3aa00640 1 --2- 172.21.15.28:0/313628551 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e3c10d500 0x7f4e3c1a35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.631+0000 7f4e3aa00640 1 -- 172.21.15.28:0/313628551 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e3c108f40 msgr2=0x7f4e3c19f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.631+0000 7f4e3aa00640 1 --2- 172.21.15.28:0/313628551 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e3c108f40 0x7f4e3c19f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.631+0000 7f4e3aa00640 1 -- 172.21.15.28:0/313628551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e3c1a3d20 con 0x7f4e3c108580 2024-04-24T08:18:48.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.631+0000 7f4e3aa00640 1 --2- 172.21.15.28:0/313628551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e3c108580 0x7f4e3c19ed80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c02a940 tx=0x7f4e2c003b20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.631+0000 7f4e33400640 1 -- 172.21.15.28:0/313628551 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c03f070 con 0x7f4e3c108580 2024-04-24T08:18:48.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.631+0000 7f4e33400640 1 -- 172.21.15.28:0/313628551 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e2c031c90 con 0x7f4e3c108580 2024-04-24T08:18:48.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.631+0000 7f4e33400640 1 -- 172.21.15.28:0/313628551 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e2c035030 con 0x7f4e3c108580 2024-04-24T08:18:48.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.631+0000 7f4e3be00640 1 -- 172.21.15.28:0/313628551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e3c1a3fb0 con 0x7f4e3c108580 2024-04-24T08:18:48.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.632+0000 7f4e3be00640 1 -- 172.21.15.28:0/313628551 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e3c1ab8a0 con 0x7f4e3c108580 2024-04-24T08:18:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.634+0000 7f4e447e8740 1 Processor -- start 2024-04-24T08:18:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.635+0000 7f4e447e8740 1 -- start start 2024-04-24T08:18:48.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.635+0000 7f4e33400640 1 -- 172.21.15.28:0/313628551 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f4e2c042070 con 0x7f4e3c108580 2024-04-24T08:18:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.635+0000 7f4e447e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd334b40 0x5621cd334f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.635+0000 7f4e447e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd335460 0x5621cd33f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.635+0000 7f4e447e8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd33fb30 0x5621cd341f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.635+0000 7f4e447e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5621cd29a1d0 con 0x5621cd334b40 2024-04-24T08:18:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.635+0000 7f4e447e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5621cd240450 con 0x5621cd335460 2024-04-24T08:18:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.635+0000 7f4e447e8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5621cd296190 con 0x5621cd33fb30 2024-04-24T08:18:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.635+0000 7f4e3b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd334b40 0x5621cd334f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.635+0000 7f4e3b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd334b40 0x5621cd334f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54780/0 (socket says 172.21.15.28:54780) 2024-04-24T08:18:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.635+0000 7f4e3b400640 1 -- 172.21.15.28:0/675526562 learned_addr learned my addr 172.21.15.28:0/675526562 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.635+0000 7f4e3aa00640 1 --2- 172.21.15.28:0/675526562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd335460 0x5621cd33f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.636+0000 7f4e3a000640 1 --2- 172.21.15.28:0/675526562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd33fb30 0x5621cd341f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.636+0000 7f4e3b400640 1 -- 172.21.15.28:0/675526562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd33fb30 msgr2=0x5621cd341f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.636+0000 7f4e3b400640 1 --2- 172.21.15.28:0/675526562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd33fb30 0x5621cd341f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.636+0000 7f4e3b400640 1 -- 172.21.15.28:0/675526562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd335460 msgr2=0x5621cd33f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.636+0000 7f4e3b400640 1 --2- 172.21.15.28:0/675526562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd335460 0x5621cd33f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.636+0000 7f4e3b400640 1 -- 172.21.15.28:0/675526562 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621cd2fd310 con 0x5621cd334b40 2024-04-24T08:18:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.636+0000 7f4e3aa00640 1 --2- 172.21.15.28:0/675526562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd335460 0x5621cd33f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:18:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.636+0000 7f4e3b400640 1 --2- 172.21.15.28:0/675526562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd334b40 0x5621cd334f20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009c90 tx=0x7f4e34011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56db9b41fbef06c5 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.636+0000 7f4e27e00640 1 -- 172.21.15.28:0/675526562 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3401e070 con 0x5621cd334b40 2024-04-24T08:18:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.636+0000 7f4e27e00640 1 -- 172.21.15.28:0/675526562 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e34009e40 con 0x5621cd334b40 2024-04-24T08:18:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.636+0000 7f4e27e00640 1 -- 172.21.15.28:0/675526562 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3400eb30 con 0x5621cd334b40 2024-04-24T08:18:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.637+0000 7f4e33400640 1 --2- 172.21.15.28:0/313628551 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4e200a3da0 0x7f4e200a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.637+0000 7f4e33400640 1 -- 172.21.15.28:0/313628551 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7f4e2c0f1130 con 0x7f4e3c108580 2024-04-24T08:18:48.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.637+0000 7f4e3aa00640 1 --2- 172.21.15.28:0/313628551 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4e200a3da0 0x7f4e200a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.637+0000 7f4e447e8740 1 -- 172.21.15.28:0/675526562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd334b40 msgr2=0x5621cd334f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.637+0000 7f4e447e8740 1 --2- 172.21.15.28:0/675526562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd334b40 0x5621cd334f20 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f4e34009c90 tx=0x7f4e34011810 comp rx=0 tx=0).stop 2024-04-24T08:18:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.637+0000 7f4e447e8740 1 -- 172.21.15.28:0/675526562 shutdown_connections 2024-04-24T08:18:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.637+0000 7f4e447e8740 1 --2- 172.21.15.28:0/675526562 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd33fb30 0x5621cd341f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.637+0000 7f4e447e8740 1 --2- 172.21.15.28:0/675526562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd335460 0x5621cd33f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.637+0000 7f4e447e8740 1 --2- 172.21.15.28:0/675526562 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd334b40 0x5621cd334f20 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.637+0000 7f4e447e8740 1 -- 172.21.15.28:0/675526562 >> 172.21.15.28:0/675526562 conn(0x5621cd330e60 msgr2=0x5621cd331240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.637+0000 7f4e447e8740 1 -- 172.21.15.28:0/675526562 shutdown_connections 2024-04-24T08:18:48.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.637+0000 7f4e447e8740 1 -- 172.21.15.28:0/675526562 wait complete. 2024-04-24T08:18:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.638+0000 7f4e3aa00640 1 --2- 172.21.15.28:0/313628551 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4e200a3da0 0x7f4e200a6260 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c0ee8e0 tx=0x7f4e2c0091c0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.638+0000 7f4e447e8740 1 -- start start 2024-04-24T08:18:48.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.638+0000 7f4e447e8740 1 Processor -- start 2024-04-24T08:18:48.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e447e8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd334b40 0x5621cd33f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e447e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd335460 0x5621cd33c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e447e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd33fb30 0x5621cd33d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e447e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5621cd3429c0 con 0x5621cd33fb30 2024-04-24T08:18:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e447e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5621cd253810 con 0x5621cd335460 2024-04-24T08:18:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e447e8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5621cd342b40 con 0x5621cd334b40 2024-04-24T08:18:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e447e8740 10 client.0 _ll_register_callbacks cb 0x5621cd221140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e3b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd33fb30 0x5621cd33d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e3b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd33fb30 0x5621cd33d010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54792/0 (socket says 172.21.15.28:54792) 2024-04-24T08:18:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e3b400640 1 -- 172.21.15.28:0/2190653322 learned_addr learned my addr 172.21.15.28:0/2190653322 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e3a000640 1 --2- 172.21.15.28:0/2190653322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd335460 0x5621cd33c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e3b400640 1 --2- 172.21.15.28:0/2190653322 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd334b40 0x5621cd33f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e3a000640 1 -- 172.21.15.28:0/2190653322 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd334b40 msgr2=0x5621cd33f5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e3a000640 1 --2- 172.21.15.28:0/2190653322 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd334b40 0x5621cd33f5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e3a000640 1 -- 172.21.15.28:0/2190653322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd33fb30 msgr2=0x5621cd33d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e3a000640 1 --2- 172.21.15.28:0/2190653322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd33fb30 0x5621cd33d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e3a000640 1 -- 172.21.15.28:0/2190653322 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621cd2643f0 con 0x5621cd335460 2024-04-24T08:18:48.649 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.639+0000 7f4e3b400640 1 --2- 172.21.15.28:0/2190653322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd33fb30 0x5621cd33d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:48.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.640+0000 7f4e3a000640 1 --2- 172.21.15.28:0/2190653322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd335460 0x5621cd33c8d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c107f40 tx=0x7f4e2800f790 comp rx=0 tx=0).ready entity=mon.2 client_cookie=43dcfb01b2539f24 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.640+0000 7f4e24c00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:18:48.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.640+0000 7f4e24c00640 1 -- 172.21.15.28:0/2190653322 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e28009e50 con 0x5621cd335460 2024-04-24T08:18:48.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.640+0000 7f4e24c00640 1 -- 172.21.15.28:0/2190653322 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7f4e28004b20 con 0x5621cd335460 2024-04-24T08:18:48.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.640+0000 7f4e447e8740 1 -- 172.21.15.28:0/2190653322 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5621cd2a7bf0 con 0x5621cd335460 2024-04-24T08:18:48.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.640+0000 7f4e24c00640 1 -- 172.21.15.28:0/2190653322 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4e28004d20 con 0x5621cd335460 2024-04-24T08:18:48.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.640+0000 7f4e447e8740 10 client.26684 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:48.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.640+0000 7f4e447e8740 1 -- 172.21.15.28:0/2190653322 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5621ccfac3f0 con 0x5621cd335460 2024-04-24T08:18:48.650 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.641+0000 7f4e24c00640 1 -- 172.21.15.28:0/2190653322 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5621cd2a7bf0 con 0x5621cd335460 2024-04-24T08:18:48.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.641+0000 7f4e24c00640 1 -- 172.21.15.28:0/2190653322 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4e28019030 con 0x5621cd335460 2024-04-24T08:18:48.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.641+0000 7f4e447e8740 10 client.26684 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:48.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.641+0000 7f4e447e8740 10 client.26684 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:18:48.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.641+0000 7f4e447e8740 20 client.26684 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:48.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.641+0000 7f4e447e8740 1 --2- 172.21.15.28:0/2190653322 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5621cd34fe90 0x5621cd3522b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.641+0000 7f4e447e8740 4 client.26684 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:18:48.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.641+0000 7f4e447e8740 1 -- 172.21.15.28:0/2190653322 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5621cd2a1a60 con 0x5621cd34fe90 2024-04-24T08:18:48.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.641+0000 7f4e3b400640 1 --2- 172.21.15.28:0/2190653322 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5621cd34fe90 0x5621cd3522b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.651 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.642+0000 7f4e3b400640 1 --2- 172.21.15.28:0/2190653322 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5621cd34fe90 0x5621cd3522b0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8187b014e055b3d5 server_cookie=88072cc9fe00a777 in_seq=0 out_seq=0 2024-04-24T08:18:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.642+0000 7f4e24c00640 10 client.26684 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.642+0000 7f4e24c00640 1 -- 172.21.15.28:0/2190653322 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5621cd2a1a60 con 0x5621cd34fe90 2024-04-24T08:18:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.643+0000 7f4e24c00640 10 client.26684 handle_command_reply: tid=0 2024-04-24T08:18:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.643+0000 7f4e447e8740 1 client.26684 shutdown 2024-04-24T08:18:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.643+0000 7f4e447e8740 1 -- 172.21.15.28:0/2190653322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd335460 msgr2=0x5621cd33c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.643+0000 7f4e447e8740 1 --2- 172.21.15.28:0/2190653322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd335460 0x5621cd33c8d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f4e3c107f40 tx=0x7f4e2800f790 comp rx=0 tx=0).stop 2024-04-24T08:18:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.643+0000 7f4e447e8740 1 -- 172.21.15.28:0/2190653322 shutdown_connections 2024-04-24T08:18:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.643+0000 7f4e447e8740 1 --2- 172.21.15.28:0/2190653322 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5621cd34fe90 0x5621cd3522b0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.652 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.643+0000 7f4e447e8740 1 --2- 172.21.15.28:0/2190653322 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd33fb30 0x5621cd33d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.643+0000 7f4e447e8740 1 --2- 172.21.15.28:0/2190653322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd335460 0x5621cd33c8d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.643+0000 7f4e447e8740 1 --2- 172.21.15.28:0/2190653322 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd334b40 0x5621cd33f5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.643+0000 7f4e24c00640 0 client.26684 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.643+0000 7f4e447e8740 1 -- 172.21.15.28:0/2190653322 >> 172.21.15.28:0/2190653322 conn(0x5621cd330e60 msgr2=0x5621cd336b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.643+0000 7f4e447e8740 1 -- 172.21.15.28:0/2190653322 shutdown_connections 2024-04-24T08:18:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.643+0000 7f4e447e8740 1 -- 172.21.15.28:0/2190653322 wait complete. 2024-04-24T08:18:48.653 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.643+0000 7f4e447e8740 20 client.26684 trim_cache size 0 max 16384 2024-04-24T08:18:48.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.657+0000 7f4e447e8740 1 Processor -- start 2024-04-24T08:18:48.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.657+0000 7f4e447e8740 1 -- start start 2024-04-24T08:18:48.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.658+0000 7f4e447e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd334b40 0x5621cd34b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.658+0000 7f4e447e8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd335460 0x5621cd33ba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.658+0000 7f4e447e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd33fb30 0x5621cd33f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.658+0000 7f4e447e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c1b6b90 con 0x5621cd33fb30 2024-04-24T08:18:48.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.658+0000 7f4e447e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c1b4460 con 0x5621cd334b40 2024-04-24T08:18:48.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.658+0000 7f4e447e8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c1b22d0 con 0x5621cd335460 2024-04-24T08:18:48.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.658+0000 7f4e3b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd33fb30 0x5621cd33f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.658+0000 7f4e3a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd335460 0x5621cd33ba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.658+0000 7f4e3b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd33fb30 0x5621cd33f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54794/0 (socket says 172.21.15.28:54794) 2024-04-24T08:18:48.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.658+0000 7f4e3b400640 1 -- 172.21.15.28:0/1977003312 learned_addr learned my addr 172.21.15.28:0/1977003312 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:48.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.658+0000 7f4e3a000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd335460 0x5621cd33ba30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:46272/0 (socket says 172.21.15.28:46272) 2024-04-24T08:18:48.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.658+0000 7f4e3b400640 1 --2- 172.21.15.28:0/1977003312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd334b40 0x5621cd34b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.658+0000 7f4e3b400640 1 -- 172.21.15.28:0/1977003312 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd335460 msgr2=0x5621cd33ba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.658+0000 7f4e3b400640 1 --2- 172.21.15.28:0/1977003312 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd335460 0x5621cd33ba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.658+0000 7f4e3b400640 1 -- 172.21.15.28:0/1977003312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd334b40 msgr2=0x5621cd34b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.658+0000 7f4e3b400640 1 --2- 172.21.15.28:0/1977003312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd334b40 0x5621cd34b8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.658+0000 7f4e3b400640 1 -- 172.21.15.28:0/1977003312 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5621cd335290 con 0x5621cd33fb30 2024-04-24T08:18:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.659+0000 7f4e3b400640 1 --2- 172.21.15.28:0/1977003312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd33fb30 0x5621cd33f110 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400e590 tx=0x7f4e34005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24db9d65890718f2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.659+0000 7f4e27400640 1 -- 172.21.15.28:0/1977003312 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3401e040 con 0x5621cd33fb30 2024-04-24T08:18:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.659+0000 7f4e27400640 1 -- 172.21.15.28:0/1977003312 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e3400eca0 con 0x5621cd33fb30 2024-04-24T08:18:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.659+0000 7f4e27400640 1 -- 172.21.15.28:0/1977003312 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e34003d00 con 0x5621cd33fb30 2024-04-24T08:18:48.666 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.659+0000 7f4e447e8740 1 -- 172.21.15.28:0/1977003312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd33fb30 msgr2=0x5621cd33f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.659+0000 7f4e447e8740 1 --2- 172.21.15.28:0/1977003312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd33fb30 0x5621cd33f110 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f4e3400e590 tx=0x7f4e34005f90 comp rx=0 tx=0).stop 2024-04-24T08:18:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.660+0000 7f4e447e8740 1 -- 172.21.15.28:0/1977003312 shutdown_connections 2024-04-24T08:18:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.660+0000 7f4e447e8740 1 --2- 172.21.15.28:0/1977003312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd33fb30 0x5621cd33f110 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.660+0000 7f4e447e8740 1 --2- 172.21.15.28:0/1977003312 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd335460 0x5621cd33ba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.660+0000 7f4e447e8740 1 --2- 172.21.15.28:0/1977003312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd334b40 0x5621cd34b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.660+0000 7f4e447e8740 1 -- 172.21.15.28:0/1977003312 >> 172.21.15.28:0/1977003312 conn(0x5621cd330e60 msgr2=0x5621cd331240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.660+0000 7f4e447e8740 1 -- 172.21.15.28:0/1977003312 shutdown_connections 2024-04-24T08:18:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.660+0000 7f4e447e8740 1 -- 172.21.15.28:0/1977003312 wait complete. 2024-04-24T08:18:48.667 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.660+0000 7f4e447e8740 1 -- start start 2024-04-24T08:18:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.660+0000 7f4e447e8740 1 Processor -- start 2024-04-24T08:18:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.661+0000 7f4e447e8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd334b40 0x5621cd348210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.661+0000 7f4e447e8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd335460 0x5621cd339c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.661+0000 7f4e447e8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd33fb30 0x5621cd33a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.661+0000 7f4e447e8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c1b2920 con 0x5621cd334b40 2024-04-24T08:18:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.661+0000 7f4e447e8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c1b3d90 con 0x5621cd33fb30 2024-04-24T08:18:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.661+0000 7f4e447e8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f4e3c1b2ff0 con 0x5621cd335460 2024-04-24T08:18:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.661+0000 7f4e447e8740 10 client.0 _ll_register_callbacks cb 0x7f4e28009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.661+0000 7f4e3b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd334b40 0x5621cd348210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.668 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.661+0000 7f4e3b400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd334b40 0x5621cd348210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:54806/0 (socket says 172.21.15.28:54806) 2024-04-24T08:18:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.661+0000 7f4e3b400640 1 -- 172.21.15.28:0/547974573 learned_addr learned my addr 172.21.15.28:0/547974573 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.661+0000 7f4e3a000640 1 --2- 172.21.15.28:0/547974573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd335460 0x5621cd339c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.661+0000 7f4e3b400640 1 --2- 172.21.15.28:0/547974573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd33fb30 0x5621cd33a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.661+0000 7f4e3b400640 1 -- 172.21.15.28:0/547974573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd335460 msgr2=0x5621cd339c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.661+0000 7f4e3b400640 1 --2- 172.21.15.28:0/547974573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd335460 0x5621cd339c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.661+0000 7f4e3b400640 1 -- 172.21.15.28:0/547974573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd33fb30 msgr2=0x5621cd33a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.661+0000 7f4e3b400640 1 --2- 172.21.15.28:0/547974573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd33fb30 0x5621cd33a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.661+0000 7f4e3b400640 1 -- 172.21.15.28:0/547974573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5621cd0767d0 con 0x5621cd334b40 2024-04-24T08:18:48.669 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.662+0000 7f4e3b400640 1 --2- 172.21.15.28:0/547974573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd334b40 0x5621cd348210 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4e34011850 tx=0x7f4e340038e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f24c7b752d71054 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.662+0000 7f4e24c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:18:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.662+0000 7f4e24c00640 1 -- 172.21.15.28:0/547974573 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e3401e030 con 0x5621cd334b40 2024-04-24T08:18:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.662+0000 7f4e447e8740 1 -- 172.21.15.28:0/547974573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e3400eca0 con 0x5621cd334b40 2024-04-24T08:18:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.662+0000 7f4e24c00640 1 -- 172.21.15.28:0/547974573 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7f4e3400e6e0 con 0x5621cd334b40 2024-04-24T08:18:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.662+0000 7f4e24c00640 1 -- 172.21.15.28:0/547974573 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f4e3403d050 con 0x5621cd334b40 2024-04-24T08:18:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.662+0000 7f4e447e8740 10 client.16413 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.662+0000 7f4e447e8740 1 -- 172.21.15.28:0/547974573 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5621cd3b6430 con 0x5621cd334b40 2024-04-24T08:18:48.670 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.662+0000 7f4e24c00640 1 -- 172.21.15.28:0/547974573 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e3400eca0 con 0x5621cd334b40 2024-04-24T08:18:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.662+0000 7f4e24c00640 1 -- 172.21.15.28:0/547974573 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f4e34024040 con 0x5621cd334b40 2024-04-24T08:18:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.663+0000 7f4e447e8740 10 client.16413 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.663+0000 7f4e447e8740 10 client.16413 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:18:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.663+0000 7f4e447e8740 20 client.16413 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.663+0000 7f4e447e8740 1 --2- 172.21.15.28:0/547974573 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5621cd3b68c0 0x5621cd3b8cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.663+0000 7f4e447e8740 4 client.16413 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:18:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.663+0000 7f4e447e8740 1 -- 172.21.15.28:0/547974573 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5621cd21d4d0 con 0x5621cd3b68c0 2024-04-24T08:18:48.671 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.663+0000 7f4e3a000640 1 --2- 172.21.15.28:0/547974573 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5621cd3b68c0 0x5621cd3b8cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:48.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.663+0000 7f4e3a000640 1 --2- 172.21.15.28:0/547974573 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5621cd3b68c0 0x5621cd3b8cb0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1812b10d06012f1a server_cookie=1dde1fbdcc73c736 in_seq=0 out_seq=0 2024-04-24T08:18:48.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.663+0000 7f4e24c00640 10 client.16413 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:48.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.664+0000 7f4e24c00640 1 -- 172.21.15.28:0/547974573 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5621cd21d4d0 con 0x5621cd3b68c0 2024-04-24T08:18:48.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.664+0000 7f4e24c00640 10 client.16413 handle_command_reply: tid=0 2024-04-24T08:18:48.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.664+0000 7f4e447e8740 1 client.16413 shutdown 2024-04-24T08:18:48.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.664+0000 7f4e447e8740 1 -- 172.21.15.28:0/547974573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd334b40 msgr2=0x5621cd348210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.664+0000 7f4e447e8740 1 --2- 172.21.15.28:0/547974573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd334b40 0x5621cd348210 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f4e34011850 tx=0x7f4e340038e0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.664+0000 7f4e447e8740 1 -- 172.21.15.28:0/547974573 shutdown_connections 2024-04-24T08:18:48.672 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.664+0000 7f4e447e8740 1 --2- 172.21.15.28:0/547974573 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5621cd3b68c0 0x5621cd3b8cb0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.664+0000 7f4e447e8740 1 --2- 172.21.15.28:0/547974573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5621cd33fb30 0x5621cd33a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.664+0000 7f4e447e8740 1 --2- 172.21.15.28:0/547974573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5621cd335460 0x5621cd339c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.664+0000 7f4e447e8740 1 --2- 172.21.15.28:0/547974573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5621cd334b40 0x5621cd348210 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.664+0000 7f4e24c00640 0 client.16413 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:48.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.664+0000 7f4e447e8740 1 -- 172.21.15.28:0/547974573 >> 172.21.15.28:0/547974573 conn(0x5621cd330e60 msgr2=0x5621cd32f6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.664+0000 7f4e447e8740 1 -- 172.21.15.28:0/547974573 shutdown_connections 2024-04-24T08:18:48.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.664+0000 7f4e447e8740 1 -- 172.21.15.28:0/547974573 wait complete. 2024-04-24T08:18:48.673 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.664+0000 7f4e447e8740 20 client.16413 trim_cache size 0 max 16384 2024-04-24T08:18:48.674 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:18:48.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.665+0000 7f4e3be00640 1 -- 172.21.15.28:0/313628551 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4e200a3da0 msgr2=0x7f4e200a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.674 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.665+0000 7f4e3be00640 1 --2- 172.21.15.28:0/313628551 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4e200a3da0 0x7f4e200a6260 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c0ee8e0 tx=0x7f4e2c0091c0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.665+0000 7f4e3be00640 1 -- 172.21.15.28:0/313628551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e3c108580 msgr2=0x7f4e3c19ed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:48.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.665+0000 7f4e3be00640 1 --2- 172.21.15.28:0/313628551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e3c108580 0x7f4e3c19ed80 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4e2c02a940 tx=0x7f4e2c003b20 comp rx=0 tx=0).stop 2024-04-24T08:18:48.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.665+0000 7f4e3be00640 1 -- 172.21.15.28:0/313628551 shutdown_connections 2024-04-24T08:18:48.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.665+0000 7f4e3be00640 1 --2- 172.21.15.28:0/313628551 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f4e200a3da0 0x7f4e200a6260 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.665+0000 7f4e3be00640 1 --2- 172.21.15.28:0/313628551 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f4e3c10d500 0x7f4e3c1a35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.665+0000 7f4e3be00640 1 --2- 172.21.15.28:0/313628551 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e3c108f40 0x7f4e3c19f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.675 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.665+0000 7f4e3be00640 1 --2- 172.21.15.28:0/313628551 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f4e3c108580 0x7f4e3c19ed80 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:48.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.665+0000 7f4e3be00640 1 -- 172.21.15.28:0/313628551 >> 172.21.15.28:0/313628551 conn(0x7f4e3c077e20 msgr2=0x7f4e3c078340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:48.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.665+0000 7f4e3be00640 1 -- 172.21.15.28:0/313628551 shutdown_connections 2024-04-24T08:18:48.676 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:48.665+0000 7f4e3be00640 1 -- 172.21.15.28:0/313628551 wait complete. 2024-04-24T08:18:48.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:48 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3799203140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:48.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:48 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3799203140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:48.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:48 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3799203140' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:18:49.683 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:18:49.683 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:18:49.683 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:18:49.765 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:49 smithi028 ceph-mon[27786]: pgmap v261: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-04-24T08:18:49.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:49 smithi117 ceph-mon[32438]: pgmap v261: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-04-24T08:18:49.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.912+0000 7f7d07c00640 1 -- 172.21.15.28:0/2583097377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d00074bf0 msgr2=0x7f7d0010e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:49.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.912+0000 7f7d07c00640 1 --2- 172.21.15.28:0/2583097377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d00074bf0 0x7f7d0010e3a0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009860 tx=0x7f7ce802a3d0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.913+0000 7f7d07c00640 1 -- 172.21.15.28:0/2583097377 shutdown_connections 2024-04-24T08:18:49.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.913+0000 7f7d07c00640 1 --2- 172.21.15.28:0/2583097377 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d0010e8e0 0x7f7d00110cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.913+0000 7f7d07c00640 1 --2- 172.21.15.28:0/2583097377 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d00074bf0 0x7f7d0010e3a0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.913+0000 7f7d07c00640 1 --2- 172.21.15.28:0/2583097377 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d000742d0 0x7f7d000746b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.913+0000 7f7d07c00640 1 -- 172.21.15.28:0/2583097377 >> 172.21.15.28:0/2583097377 conn(0x7f7d00079450 msgr2=0x7f7d00079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:49.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.913+0000 7f7d07c00640 1 -- 172.21.15.28:0/2583097377 shutdown_connections 2024-04-24T08:18:49.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.913+0000 7f7d07c00640 1 -- 172.21.15.28:0/2583097377 wait complete. 2024-04-24T08:18:49.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.914+0000 7f7d07c00640 1 Processor -- start 2024-04-24T08:18:49.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.914+0000 7f7d07c00640 1 -- start start 2024-04-24T08:18:49.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.914+0000 7f7d07c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d000742d0 0x7f7d001a07e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.914+0000 7f7d07c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d00074bf0 0x7f7d001a0d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.914+0000 7f7d07c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d0010e8e0 0x7f7d0019a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.914+0000 7f7d07c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d00114620 con 0x7f7d00074bf0 2024-04-24T08:18:49.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.914+0000 7f7d07c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7d001144a0 con 0x7f7d0010e8e0 2024-04-24T08:18:49.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.914+0000 7f7d07c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7d001147a0 con 0x7f7d000742d0 2024-04-24T08:18:49.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.914+0000 7f7d05200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d000742d0 0x7f7d001a07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.914+0000 7f7cffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d00074bf0 0x7f7d001a0d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.914+0000 7f7d05c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d0010e8e0 0x7f7d0019a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.914+0000 7f7d05200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d000742d0 0x7f7d001a07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:39296/0 (socket says 172.21.15.28:39296) 2024-04-24T08:18:49.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.914+0000 7f7d05200640 1 -- 172.21.15.28:0/3692694900 learned_addr learned my addr 172.21.15.28:0/3692694900 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:49.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.915+0000 7f7d05200640 1 -- 172.21.15.28:0/3692694900 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d0010e8e0 msgr2=0x7f7d0019a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:49.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.915+0000 7f7d05200640 1 --2- 172.21.15.28:0/3692694900 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d0010e8e0 0x7f7d0019a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.919 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.915+0000 7f7d05200640 1 -- 172.21.15.28:0/3692694900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d00074bf0 msgr2=0x7f7d001a0d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.915+0000 7f7d05200640 1 --2- 172.21.15.28:0/3692694900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d00074bf0 0x7f7d001a0d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.915+0000 7f7d05200640 1 -- 172.21.15.28:0/3692694900 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d0019b160 con 0x7f7d000742d0 2024-04-24T08:18:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.915+0000 7f7d05200640 1 --2- 172.21.15.28:0/3692694900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d000742d0 0x7f7d001a07e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000ebc0 tx=0x7f7cf000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.915+0000 7f7cfd600640 1 -- 172.21.15.28:0/3692694900 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0010630 con 0x7f7d000742d0 2024-04-24T08:18:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.916+0000 7f7d07c00640 1 -- 172.21.15.28:0/3692694900 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d0019b450 con 0x7f7d000742d0 2024-04-24T08:18:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.916+0000 7f7cfd600640 1 -- 172.21.15.28:0/3692694900 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cf0010cb0 con 0x7f7d000742d0 2024-04-24T08:18:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.916+0000 7f7cfd600640 1 -- 172.21.15.28:0/3692694900 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0018bd0 con 0x7f7d000742d0 2024-04-24T08:18:49.920 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.916+0000 7f7d07c00640 1 -- 172.21.15.28:0/3692694900 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d001a76a0 con 0x7f7d000742d0 2024-04-24T08:18:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.919+0000 7f7d0aa17740 1 Processor -- start 2024-04-24T08:18:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.919+0000 7f7d0aa17740 1 -- start start 2024-04-24T08:18:49.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.919+0000 7f7cfd600640 1 -- 172.21.15.28:0/3692694900 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7cf00107d0 con 0x7f7d000742d0 2024-04-24T08:18:49.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.919+0000 7f7d0aa17740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 0x55ffad328470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.919+0000 7f7d0aa17740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad3289b0 0x55ffad332b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.919+0000 7f7d0aa17740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad333080 0x55ffad335470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.919+0000 7f7d0aa17740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ffad28de70 con 0x55ffad3289b0 2024-04-24T08:18:49.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.919+0000 7f7d0aa17740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ffad234380 con 0x55ffad328090 2024-04-24T08:18:49.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.919+0000 7f7d0aa17740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ffad289e30 con 0x55ffad333080 2024-04-24T08:18:49.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.920+0000 7f7d05200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad3289b0 0x55ffad332b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.920+0000 7f7d05c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 0x55ffad328470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.920+0000 7f7cffe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad333080 0x55ffad335470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.920+0000 7f7cffe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad333080 0x55ffad335470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:39310/0 (socket says 172.21.15.28:39310) 2024-04-24T08:18:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.920+0000 7f7cffe00640 1 -- 172.21.15.28:0/1907828364 learned_addr learned my addr 172.21.15.28:0/1907828364 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.920+0000 7f7d05c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 0x55ffad328470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:57320/0 (socket says 172.21.15.28:57320) 2024-04-24T08:18:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.920+0000 7f7cffe00640 1 -- 172.21.15.28:0/1907828364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 msgr2=0x55ffad328470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.920+0000 7f7cffe00640 1 --2- 172.21.15.28:0/1907828364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 0x55ffad328470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.920+0000 7f7cffe00640 1 -- 172.21.15.28:0/1907828364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad3289b0 msgr2=0x55ffad332b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:49.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.920+0000 7f7cffe00640 1 --2- 172.21.15.28:0/1907828364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad3289b0 0x55ffad332b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.920+0000 7f7cffe00640 1 -- 172.21.15.28:0/1907828364 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffad0687e0 con 0x55ffad333080 2024-04-24T08:18:49.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.920+0000 7f7cffe00640 1 --2- 172.21.15.28:0/1907828364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad333080 0x55ffad335470 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009860 tx=0x7f7ce8030ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6d458082270e6c0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:49.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.921+0000 7f7cf6000640 1 -- 172.21.15.28:0/1907828364 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce8004030 con 0x55ffad333080 2024-04-24T08:18:49.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.921+0000 7f7cf6000640 1 -- 172.21.15.28:0/1907828364 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ce8041030 con 0x55ffad333080 2024-04-24T08:18:49.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.921+0000 7f7cf6000640 1 -- 172.21.15.28:0/1907828364 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7ce80053f0 con 0x55ffad333080 2024-04-24T08:18:49.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.921+0000 7f7cfd600640 1 --2- 172.21.15.28:0/3692694900 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7ce40a3d20 0x7f7ce40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.921+0000 7f7cfd600640 1 -- 172.21.15.28:0/3692694900 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7f7cf00d0ea0 con 0x7f7d000742d0 2024-04-24T08:18:49.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.921+0000 7f7d05c00640 1 --2- 172.21.15.28:0/3692694900 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7ce40a3d20 0x7f7ce40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.921+0000 7f7d0aa17740 1 -- 172.21.15.28:0/1907828364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad333080 msgr2=0x55ffad335470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:49.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.921+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/1907828364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad333080 0x55ffad335470 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f7ce8009860 tx=0x7f7ce8030ab0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.922+0000 7f7d0aa17740 1 -- 172.21.15.28:0/1907828364 shutdown_connections 2024-04-24T08:18:49.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.922+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/1907828364 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad3289b0 0x55ffad332b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.922+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/1907828364 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad333080 0x55ffad335470 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.922+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/1907828364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 0x55ffad328470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.922+0000 7f7d0aa17740 1 -- 172.21.15.28:0/1907828364 >> 172.21.15.28:0/1907828364 conn(0x55ffad3243b0 msgr2=0x55ffad324790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:49.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.922+0000 7f7d0aa17740 1 -- 172.21.15.28:0/1907828364 shutdown_connections 2024-04-24T08:18:49.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.922+0000 7f7d0aa17740 1 -- 172.21.15.28:0/1907828364 wait complete. 2024-04-24T08:18:49.928 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.922+0000 7f7d05c00640 1 --2- 172.21.15.28:0/3692694900 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7ce40a3d20 0x7f7ce40a61e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009530 tx=0x7f7cf8009260 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:49.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.922+0000 7f7d0aa17740 1 -- start start 2024-04-24T08:18:49.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.923+0000 7f7d0aa17740 1 Processor -- start 2024-04-24T08:18:49.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.923+0000 7f7d0aa17740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 0x55ffad32ff40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.923+0000 7f7d0aa17740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad3289b0 0x55ffad330480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.923+0000 7f7d0aa17740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad333080 0x55ffad3309c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.923+0000 7f7d0aa17740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55ffad335e10 con 0x55ffad3289b0 2024-04-24T08:18:49.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.923+0000 7f7d0aa17740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55ffad247660 con 0x55ffad328090 2024-04-24T08:18:49.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.923+0000 7f7d0aa17740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55ffad335f90 con 0x55ffad333080 2024-04-24T08:18:49.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.923+0000 7f7d0aa17740 10 client.0 _ll_register_callbacks cb 0x55ffad235aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:49.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.923+0000 7f7cffe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad3289b0 0x55ffad330480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.923+0000 7f7d05200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 0x55ffad32ff40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.923+0000 7f7d05200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 0x55ffad32ff40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:57324/0 (socket says 172.21.15.28:57324) 2024-04-24T08:18:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.923+0000 7f7d05200640 1 -- 172.21.15.28:0/2398717565 learned_addr learned my addr 172.21.15.28:0/2398717565 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.924+0000 7f7d05c00640 1 --2- 172.21.15.28:0/2398717565 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad333080 0x55ffad3309c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.924+0000 7f7d05200640 1 -- 172.21.15.28:0/2398717565 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad333080 msgr2=0x55ffad3309c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.924+0000 7f7d05200640 1 --2- 172.21.15.28:0/2398717565 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad333080 0x55ffad3309c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.924+0000 7f7d05200640 1 -- 172.21.15.28:0/2398717565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad3289b0 msgr2=0x55ffad330480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.924+0000 7f7d05200640 1 --2- 172.21.15.28:0/2398717565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad3289b0 0x55ffad330480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.924+0000 7f7d05200640 1 -- 172.21.15.28:0/2398717565 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffad0c8f70 con 0x55ffad328090 2024-04-24T08:18:49.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.924+0000 7f7d05200640 1 --2- 172.21.15.28:0/2398717565 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 0x55ffad32ff40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000cba0 tx=0x7f7cf000eec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d5d54563722b6517 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.924+0000 7f7ceea00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:18:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.924+0000 7f7ceea00640 1 -- 172.21.15.28:0/2398717565 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0010e60 con 0x55ffad328090 2024-04-24T08:18:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.924+0000 7f7ceea00640 1 -- 172.21.15.28:0/2398717565 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7f7cf009c290 con 0x55ffad328090 2024-04-24T08:18:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.924+0000 7f7d0aa17740 1 -- 172.21.15.28:0/2398717565 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7ce8041030 con 0x55ffad328090 2024-04-24T08:18:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.924+0000 7f7ceea00640 1 -- 172.21.15.28:0/2398717565 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7cf009c980 con 0x55ffad328090 2024-04-24T08:18:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.924+0000 7f7d0aa17740 10 client.26702 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.925+0000 7f7d0aa17740 1 -- 172.21.15.28:0/2398717565 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ffacf60170 con 0x55ffad328090 2024-04-24T08:18:49.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.925+0000 7f7ceea00640 1 -- 172.21.15.28:0/2398717565 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7ce8041030 con 0x55ffad328090 2024-04-24T08:18:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.925+0000 7f7ceea00640 1 -- 172.21.15.28:0/2398717565 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f7cf0098b90 con 0x55ffad328090 2024-04-24T08:18:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.925+0000 7f7d0aa17740 10 client.26702 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.925+0000 7f7d0aa17740 10 client.26702 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:18:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.925+0000 7f7d0aa17740 20 client.26702 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.925+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/2398717565 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ffad343330 0x55ffad345760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.925+0000 7f7d0aa17740 4 client.26702 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:18:49.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.925+0000 7f7d0aa17740 1 -- 172.21.15.28:0/2398717565 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffad29b890 con 0x55ffad343330 2024-04-24T08:18:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.926+0000 7f7cffe00640 1 --2- 172.21.15.28:0/2398717565 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ffad343330 0x55ffad345760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.926+0000 7f7cffe00640 1 --2- 172.21.15.28:0/2398717565 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ffad343330 0x55ffad345760 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6634475e4bca253 server_cookie=c65bda29d733a873 in_seq=0 out_seq=0 2024-04-24T08:18:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.926+0000 7f7ceea00640 10 client.26702 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.927+0000 7f7ceea00640 1 -- 172.21.15.28:0/2398717565 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ffad29b890 con 0x55ffad343330 2024-04-24T08:18:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.927+0000 7f7ceea00640 10 client.26702 handle_command_reply: tid=0 2024-04-24T08:18:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.927+0000 7f7d0aa17740 1 client.26702 shutdown 2024-04-24T08:18:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.927+0000 7f7d0aa17740 1 -- 172.21.15.28:0/2398717565 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 msgr2=0x55ffad32ff40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.927+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/2398717565 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 0x55ffad32ff40 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000cba0 tx=0x7f7cf000eec0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.927+0000 7f7d0aa17740 1 -- 172.21.15.28:0/2398717565 shutdown_connections 2024-04-24T08:18:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.927+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/2398717565 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ffad343330 0x55ffad345760 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.927+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/2398717565 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad3289b0 0x55ffad330480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.927+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/2398717565 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad333080 0x55ffad3309c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.927+0000 7f7ceea00640 0 client.26702 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.927+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/2398717565 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 0x55ffad32ff40 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.927+0000 7f7d0aa17740 1 -- 172.21.15.28:0/2398717565 >> 172.21.15.28:0/2398717565 conn(0x55ffad3243b0 msgr2=0x55ffad323510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.927+0000 7f7d0aa17740 1 -- 172.21.15.28:0/2398717565 shutdown_connections 2024-04-24T08:18:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.927+0000 7f7d0aa17740 1 -- 172.21.15.28:0/2398717565 wait complete. 2024-04-24T08:18:49.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.928+0000 7f7d0aa17740 20 client.26702 trim_cache size 0 max 16384 2024-04-24T08:18:49.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:49 smithi194 ceph-mon[32836]: pgmap v261: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-04-24T08:18:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.948+0000 7f7d0aa17740 1 Processor -- start 2024-04-24T08:18:49.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.948+0000 7f7d0aa17740 1 -- start start 2024-04-24T08:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.948+0000 7f7d0aa17740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 0x55ffad33ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.948+0000 7f7d0aa17740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad3289b0 0x55ffad34b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.948+0000 7f7d0aa17740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad333080 0x55ffad34b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.948+0000 7f7d0aa17740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d001b2a30 con 0x55ffad333080 2024-04-24T08:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.948+0000 7f7d0aa17740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7d001b0300 con 0x55ffad328090 2024-04-24T08:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.948+0000 7f7d0aa17740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7d001ae170 con 0x55ffad3289b0 2024-04-24T08:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.949+0000 7f7d05c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad333080 0x55ffad34b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.949+0000 7f7d05200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 0x55ffad33ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.951 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.949+0000 7f7cffe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad3289b0 0x55ffad34b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.949+0000 7f7d05200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 0x55ffad33ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:57336/0 (socket says 172.21.15.28:57336) 2024-04-24T08:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.949+0000 7f7d05c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad333080 0x55ffad34b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52940/0 (socket says 172.21.15.28:52940) 2024-04-24T08:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.949+0000 7f7d05200640 1 -- 172.21.15.28:0/2245193970 learned_addr learned my addr 172.21.15.28:0/2245193970 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.949+0000 7f7d05200640 1 -- 172.21.15.28:0/2245193970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad3289b0 msgr2=0x55ffad34b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.949+0000 7f7d05200640 1 --2- 172.21.15.28:0/2245193970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad3289b0 0x55ffad34b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.949+0000 7f7d05200640 1 -- 172.21.15.28:0/2245193970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad333080 msgr2=0x55ffad34b540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.949+0000 7f7d05200640 1 --2- 172.21.15.28:0/2245193970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad333080 0x55ffad34b540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.949+0000 7f7d05200640 1 -- 172.21.15.28:0/2245193970 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffad32e4b0 con 0x55ffad328090 2024-04-24T08:18:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.949+0000 7f7d05200640 1 --2- 172.21.15.28:0/2245193970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 0x55ffad33ecd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7cf009cdd0 tx=0x7f7cf009b350 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28fe514e7d1f598a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:49.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.950+0000 7f7cf5600640 1 -- 172.21.15.28:0/2245193970 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf0010e60 con 0x55ffad328090 2024-04-24T08:18:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.950+0000 7f7cf5600640 1 -- 172.21.15.28:0/2245193970 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cf00a5a00 con 0x55ffad328090 2024-04-24T08:18:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.950+0000 7f7cf5600640 1 -- 172.21.15.28:0/2245193970 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf009c7a0 con 0x55ffad328090 2024-04-24T08:18:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.950+0000 7f7d0aa17740 1 -- 172.21.15.28:0/2245193970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 msgr2=0x55ffad33ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.950+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/2245193970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 0x55ffad33ecd0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f7cf009cdd0 tx=0x7f7cf009b350 comp rx=0 tx=0).stop 2024-04-24T08:18:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.950+0000 7f7d0aa17740 1 -- 172.21.15.28:0/2245193970 shutdown_connections 2024-04-24T08:18:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.950+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/2245193970 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad333080 0x55ffad34b540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.950+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/2245193970 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad3289b0 0x55ffad34b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.950+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/2245193970 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad328090 0x55ffad33ecd0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.951+0000 7f7d0aa17740 1 -- 172.21.15.28:0/2245193970 >> 172.21.15.28:0/2245193970 conn(0x55ffad3243b0 msgr2=0x55ffad323510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.951+0000 7f7d0aa17740 1 -- 172.21.15.28:0/2245193970 shutdown_connections 2024-04-24T08:18:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.951+0000 7f7d0aa17740 1 -- 172.21.15.28:0/2245193970 wait complete. 2024-04-24T08:18:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.951+0000 7f7d0aa17740 1 -- start start 2024-04-24T08:18:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.951+0000 7f7d0aa17740 1 Processor -- start 2024-04-24T08:18:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7d0aa17740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad328090 0x55ffad33b3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7d0aa17740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad3289b0 0x55ffad3381b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.958 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7d0aa17740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad333080 0x55ffad3386f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7d0aa17740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7d001ae7c0 con 0x55ffad333080 2024-04-24T08:18:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7d0aa17740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7d001afc30 con 0x55ffad3289b0 2024-04-24T08:18:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7d0aa17740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7d001aee90 con 0x55ffad328090 2024-04-24T08:18:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7d0aa17740 10 client.0 _ll_register_callbacks cb 0x7f7cf000eb10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7d05c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad333080 0x55ffad3386f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7d05c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad333080 0x55ffad3386f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52956/0 (socket says 172.21.15.28:52956) 2024-04-24T08:18:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7d05c00640 1 -- 172.21.15.28:0/1772489624 learned_addr learned my addr 172.21.15.28:0/1772489624 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:49.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7d05200640 1 --2- 172.21.15.28:0/1772489624 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad328090 0x55ffad33b3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7cffe00640 1 --2- 172.21.15.28:0/1772489624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad3289b0 0x55ffad3381b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7d05c00640 1 -- 172.21.15.28:0/1772489624 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad328090 msgr2=0x55ffad33b3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7d05c00640 1 --2- 172.21.15.28:0/1772489624 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad328090 0x55ffad33b3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7d05c00640 1 -- 172.21.15.28:0/1772489624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad3289b0 msgr2=0x55ffad3381b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7d05c00640 1 --2- 172.21.15.28:0/1772489624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad3289b0 0x55ffad3381b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7d05c00640 1 -- 172.21.15.28:0/1772489624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffad338c60 con 0x55ffad333080 2024-04-24T08:18:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.952+0000 7f7cffe00640 1 --2- 172.21.15.28:0/1772489624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad3289b0 0x55ffad3381b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.953+0000 7f7d05c00640 1 --2- 172.21.15.28:0/1772489624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad333080 0x55ffad3386f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8004640 tx=0x7f7cf8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6fa14f8fd0596a60 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:49.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.953+0000 7f7ceea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:18:49.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.953+0000 7f7ceea00640 1 -- 172.21.15.28:0/1772489624 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cf802e060 con 0x55ffad333080 2024-04-24T08:18:49.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.953+0000 7f7ceea00640 1 -- 172.21.15.28:0/1772489624 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7f7cf80293f0 con 0x55ffad333080 2024-04-24T08:18:49.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.953+0000 7f7d0aa17740 1 -- 172.21.15.28:0/1772489624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cf00a5a00 con 0x55ffad333080 2024-04-24T08:18:49.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.953+0000 7f7ceea00640 1 -- 172.21.15.28:0/1772489624 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7cf80295f0 con 0x55ffad333080 2024-04-24T08:18:49.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.953+0000 7f7d0aa17740 10 client.16431 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:49.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.953+0000 7f7d0aa17740 1 -- 172.21.15.28:0/1772489624 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55ffad3aa6c0 con 0x55ffad333080 2024-04-24T08:18:49.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.954+0000 7f7ceea00640 1 -- 172.21.15.28:0/1772489624 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cf00a5a00 con 0x55ffad333080 2024-04-24T08:18:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.954+0000 7f7ceea00640 1 -- 172.21.15.28:0/1772489624 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f7cf8028070 con 0x55ffad333080 2024-04-24T08:18:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.954+0000 7f7d0aa17740 10 client.16431 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.954+0000 7f7d0aa17740 10 client.16431 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:18:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.954+0000 7f7d0aa17740 20 client.16431 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.954+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/1772489624 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ffad3aadf0 0x55ffad3ad210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.954+0000 7f7d0aa17740 4 client.16431 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:18:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.954+0000 7f7d0aa17740 1 -- 172.21.15.28:0/1772489624 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55ffad295700 con 0x55ffad3aadf0 2024-04-24T08:18:49.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.954+0000 7f7d05200640 1 --2- 172.21.15.28:0/1772489624 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ffad3aadf0 0x55ffad3ad210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:49.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.955+0000 7f7d05200640 1 --2- 172.21.15.28:0/1772489624 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ffad3aadf0 0x55ffad3ad210 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bac40cdccf439ba6 server_cookie=d792363522047994 in_seq=0 out_seq=0 2024-04-24T08:18:49.963 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.955+0000 7f7ceea00640 10 client.16431 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.956+0000 7f7ceea00640 1 -- 172.21.15.28:0/1772489624 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55ffad295700 con 0x55ffad3aadf0 2024-04-24T08:18:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.956+0000 7f7ceea00640 10 client.16431 handle_command_reply: tid=0 2024-04-24T08:18:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.956+0000 7f7d0aa17740 1 client.16431 shutdown 2024-04-24T08:18:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.957+0000 7f7d0aa17740 1 -- 172.21.15.28:0/1772489624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad333080 msgr2=0x55ffad3386f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.957+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/1772489624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad333080 0x55ffad3386f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8004640 tx=0x7f7cf8004730 comp rx=0 tx=0).stop 2024-04-24T08:18:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.957+0000 7f7d0aa17740 1 -- 172.21.15.28:0/1772489624 shutdown_connections 2024-04-24T08:18:49.964 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.957+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/1772489624 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55ffad3aadf0 0x55ffad3ad210 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.957+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/1772489624 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55ffad333080 0x55ffad3386f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.957+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/1772489624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55ffad3289b0 0x55ffad3381b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.957+0000 7f7d0aa17740 1 --2- 172.21.15.28:0/1772489624 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55ffad328090 0x55ffad33b3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.957+0000 7f7ceea00640 0 client.16431 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:49.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.957+0000 7f7d0aa17740 1 -- 172.21.15.28:0/1772489624 >> 172.21.15.28:0/1772489624 conn(0x55ffad3243b0 msgr2=0x55ffad3297e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:49.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.957+0000 7f7d0aa17740 1 -- 172.21.15.28:0/1772489624 shutdown_connections 2024-04-24T08:18:49.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.957+0000 7f7d0aa17740 1 -- 172.21.15.28:0/1772489624 wait complete. 2024-04-24T08:18:49.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.957+0000 7f7d0aa17740 20 client.16431 trim_cache size 0 max 16384 2024-04-24T08:18:49.966 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:18:49.966 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-24T08:18:49.966 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "11fa8315-0ebc-4f6e-9377-d1cec5e3bd3c", 2024-04-24T08:18:49.966 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-24T08:18:49.966 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:18:49.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.958+0000 7f7d07c00640 1 -- 172.21.15.28:0/3692694900 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7ce40a3d20 msgr2=0x7f7ce40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:49.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.958+0000 7f7d07c00640 1 --2- 172.21.15.28:0/3692694900 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7ce40a3d20 0x7f7ce40a61e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f7cf8009530 tx=0x7f7cf8009260 comp rx=0 tx=0).stop 2024-04-24T08:18:49.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.958+0000 7f7d07c00640 1 -- 172.21.15.28:0/3692694900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d000742d0 msgr2=0x7f7d001a07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:49.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.958+0000 7f7d07c00640 1 --2- 172.21.15.28:0/3692694900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d000742d0 0x7f7d001a07e0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f7cf000ebc0 tx=0x7f7cf000c6a0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.958+0000 7f7d07c00640 1 -- 172.21.15.28:0/3692694900 shutdown_connections 2024-04-24T08:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.958+0000 7f7d07c00640 1 --2- 172.21.15.28:0/3692694900 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7ce40a3d20 0x7f7ce40a61e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.958+0000 7f7d07c00640 1 --2- 172.21.15.28:0/3692694900 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7d00074bf0 0x7f7d001a0d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.958+0000 7f7d07c00640 1 --2- 172.21.15.28:0/3692694900 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d0010e8e0 0x7f7d0019a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.958+0000 7f7d07c00640 1 --2- 172.21.15.28:0/3692694900 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7d000742d0 0x7f7d001a07e0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.958+0000 7f7d07c00640 1 -- 172.21.15.28:0/3692694900 >> 172.21.15.28:0/3692694900 conn(0x7f7d00079450 msgr2=0x7f7d000fefb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.959+0000 7f7d07c00640 1 -- 172.21.15.28:0/3692694900 shutdown_connections 2024-04-24T08:18:49.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:49.959+0000 7f7d07c00640 1 -- 172.21.15.28:0/3692694900 wait complete. 2024-04-24T08:18:49.982 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:18:49.982 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:18:50.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.179+0000 7fc892800640 1 -- 172.21.15.28:0/774674233 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc88c107310 msgr2=0x7fc88c10eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:50.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.179+0000 7fc892800640 1 --2- 172.21.15.28:0/774674233 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc88c107310 0x7fc88c10eee0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fc884009900 tx=0x7fc88402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.181 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.179+0000 7fc892800640 1 -- 172.21.15.28:0/774674233 shutdown_connections 2024-04-24T08:18:50.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.179+0000 7fc892800640 1 --2- 172.21.15.28:0/774674233 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc88c107310 0x7fc88c10eee0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.179+0000 7fc892800640 1 --2- 172.21.15.28:0/774674233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc88c102bf0 0x7fc88c106c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.179+0000 7fc892800640 1 --2- 172.21.15.28:0/774674233 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc88c102240 0x7fc88c102620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.179+0000 7fc892800640 1 -- 172.21.15.28:0/774674233 >> 172.21.15.28:0/774674233 conn(0x7fc88c079450 msgr2=0x7fc88c079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:50.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.179+0000 7fc892800640 1 -- 172.21.15.28:0/774674233 shutdown_connections 2024-04-24T08:18:50.183 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.179+0000 7fc892800640 1 -- 172.21.15.28:0/774674233 wait complete. 2024-04-24T08:18:50.184 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.180+0000 7fc892800640 1 Processor -- start 2024-04-24T08:18:50.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.180+0000 7fc892800640 1 -- start start 2024-04-24T08:18:50.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.181+0000 7fc892800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc88c102240 0x7fc88c19e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.181+0000 7fc892800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc88c107310 0x7fc88c19eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.181+0000 7fc892800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc88c1986a0 0x7fc88c198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.181+0000 7fc892800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c112420 con 0x7fc88c1986a0 2024-04-24T08:18:50.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.181+0000 7fc892800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1122a0 con 0x7fc88c107310 2024-04-24T08:18:50.185 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.181+0000 7fc892800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1125a0 con 0x7fc88c102240 2024-04-24T08:18:50.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.181+0000 7fc88b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc88c102240 0x7fc88c19e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.181+0000 7fc88b400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc88c102240 0x7fc88c19e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:39336/0 (socket says 172.21.15.28:39336) 2024-04-24T08:18:50.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.181+0000 7fc88b400640 1 -- 172.21.15.28:0/883688207 learned_addr learned my addr 172.21.15.28:0/883688207 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:50.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.181+0000 7fc88be00640 1 --2- 172.21.15.28:0/883688207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc88c1986a0 0x7fc88c198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.181+0000 7fc88aa00640 1 --2- 172.21.15.28:0/883688207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc88c107310 0x7fc88c19eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.182+0000 7fc88b400640 1 -- 172.21.15.28:0/883688207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc88c107310 msgr2=0x7fc88c19eb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:50.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.182+0000 7fc88b400640 1 --2- 172.21.15.28:0/883688207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc88c107310 0x7fc88c19eb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.186 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.182+0000 7fc88b400640 1 -- 172.21.15.28:0/883688207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc88c1986a0 msgr2=0x7fc88c198b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.182+0000 7fc88b400640 1 --2- 172.21.15.28:0/883688207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc88c1986a0 0x7fc88c198b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.182+0000 7fc88b400640 1 -- 172.21.15.28:0/883688207 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc88c1992a0 con 0x7fc88c102240 2024-04-24T08:18:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.182+0000 7fc88b400640 1 --2- 172.21.15.28:0/883688207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc88c102240 0x7fc88c19e5e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc87c00e9f0 tx=0x7fc87c00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.182+0000 7fc883e00640 1 -- 172.21.15.28:0/883688207 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c00ccc0 con 0x7fc88c102240 2024-04-24T08:18:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.182+0000 7fc892800640 1 -- 172.21.15.28:0/883688207 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc88c1a5380 con 0x7fc88c102240 2024-04-24T08:18:50.187 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.182+0000 7fc883e00640 1 -- 172.21.15.28:0/883688207 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc87c00ce60 con 0x7fc88c102240 2024-04-24T08:18:50.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.182+0000 7fc883e00640 1 -- 172.21.15.28:0/883688207 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c010690 con 0x7fc88c102240 2024-04-24T08:18:50.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.182+0000 7fc892800640 1 -- 172.21.15.28:0/883688207 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc88c1a5730 con 0x7fc88c102240 2024-04-24T08:18:50.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.186+0000 7fc8955b6740 1 Processor -- start 2024-04-24T08:18:50.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.186+0000 7fc883e00640 1 -- 172.21.15.28:0/883688207 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fc87c0046c0 con 0x7fc88c102240 2024-04-24T08:18:50.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.186+0000 7fc8955b6740 1 -- start start 2024-04-24T08:18:50.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.186+0000 7fc8955b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4590 0x5560d41f4970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.186+0000 7fc8955b6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d41f4f40 0x5560d4203720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.186+0000 7fc8955b6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d4203c60 0x5560d4206050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.186+0000 7fc8955b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5560d415a1f0 con 0x5560d41f4590 2024-04-24T08:18:50.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.186+0000 7fc8955b6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5560d4100350 con 0x5560d41f4f40 2024-04-24T08:18:50.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.186+0000 7fc8955b6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5560d41561b0 con 0x5560d4203c60 2024-04-24T08:18:50.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.187+0000 7fc88aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d4203c60 0x5560d4206050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.187+0000 7fc88be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4590 0x5560d41f4970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.187+0000 7fc88aa00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d4203c60 0x5560d4206050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:39346/0 (socket says 172.21.15.28:39346) 2024-04-24T08:18:50.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.187+0000 7fc88aa00640 1 -- 172.21.15.28:0/4269481126 learned_addr learned my addr 172.21.15.28:0/4269481126 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:50.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.187+0000 7fc88b400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d41f4f40 0x5560d4203720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.187+0000 7fc88aa00640 1 -- 172.21.15.28:0/4269481126 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d41f4f40 msgr2=0x5560d4203720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:50.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.187+0000 7fc88aa00640 1 --2- 172.21.15.28:0/4269481126 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d41f4f40 0x5560d4203720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.187+0000 7fc88aa00640 1 -- 172.21.15.28:0/4269481126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4590 msgr2=0x5560d41f4970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:50.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.187+0000 7fc88aa00640 1 --2- 172.21.15.28:0/4269481126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4590 0x5560d41f4970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.187+0000 7fc88aa00640 1 -- 172.21.15.28:0/4269481126 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560d4154f50 con 0x5560d4203c60 2024-04-24T08:18:50.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.187+0000 7fc88be00640 1 --2- 172.21.15.28:0/4269481126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4590 0x5560d41f4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:18:50.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.187+0000 7fc88aa00640 1 --2- 172.21.15.28:0/4269481126 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d4203c60 0x5560d4206050 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc874005c10 tx=0x7fc8740118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d9a452c1ffdca1d0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:50.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.188+0000 7fc880c00640 1 -- 172.21.15.28:0/4269481126 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc874004d20 con 0x5560d4203c60 2024-04-24T08:18:50.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.188+0000 7fc883e00640 1 --2- 172.21.15.28:0/883688207 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc86c0a3d20 0x7fc86c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.188+0000 7fc880c00640 1 -- 172.21.15.28:0/4269481126 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc874004050 con 0x5560d4203c60 2024-04-24T08:18:50.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.188+0000 7fc880c00640 1 -- 172.21.15.28:0/4269481126 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc874005240 con 0x5560d4203c60 2024-04-24T08:18:50.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.188+0000 7fc883e00640 1 -- 172.21.15.28:0/883688207 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7fc87c0d0c90 con 0x7fc88c102240 2024-04-24T08:18:50.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.188+0000 7fc88be00640 1 --2- 172.21.15.28:0/883688207 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc86c0a3d20 0x7fc86c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.188+0000 7fc8955b6740 1 -- 172.21.15.28:0/4269481126 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d4203c60 msgr2=0x5560d4206050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:50.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.188+0000 7fc8955b6740 1 --2- 172.21.15.28:0/4269481126 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d4203c60 0x5560d4206050 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fc874005c10 tx=0x7fc8740118d0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.189+0000 7fc8955b6740 1 -- 172.21.15.28:0/4269481126 shutdown_connections 2024-04-24T08:18:50.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.189+0000 7fc8955b6740 1 --2- 172.21.15.28:0/4269481126 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d4203c60 0x5560d4206050 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.189+0000 7fc8955b6740 1 --2- 172.21.15.28:0/4269481126 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d41f4f40 0x5560d4203720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.189+0000 7fc8955b6740 1 --2- 172.21.15.28:0/4269481126 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4590 0x5560d41f4970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.189+0000 7fc8955b6740 1 -- 172.21.15.28:0/4269481126 >> 172.21.15.28:0/4269481126 conn(0x5560d41f0b80 msgr2=0x5560d41f0f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:50.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.189+0000 7fc8955b6740 1 -- 172.21.15.28:0/4269481126 shutdown_connections 2024-04-24T08:18:50.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.189+0000 7fc88be00640 1 --2- 172.21.15.28:0/883688207 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc86c0a3d20 0x7fc86c0a61e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc884031550 tx=0x7fc8840046c0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:50.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.189+0000 7fc8955b6740 1 -- 172.21.15.28:0/4269481126 wait complete. 2024-04-24T08:18:50.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.189+0000 7fc8955b6740 1 -- start start 2024-04-24T08:18:50.197 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.190+0000 7fc8955b6740 1 Processor -- start 2024-04-24T08:18:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.190+0000 7fc8955b6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d41f4590 0x5560d4200bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.200 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.190+0000 7fc8955b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4f40 0x5560d4201110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.190+0000 7fc8955b6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d4203c60 0x5560d4201650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.190+0000 7fc8955b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5560d4206a20 con 0x5560d41f4f40 2024-04-24T08:18:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.190+0000 7fc8955b6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5560d4113690 con 0x5560d4203c60 2024-04-24T08:18:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.190+0000 7fc8955b6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5560d4206ba0 con 0x5560d41f4590 2024-04-24T08:18:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.190+0000 7fc8955b6740 10 client.0 _ll_register_callbacks cb 0x5560d414f650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.190+0000 7fc88aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4f40 0x5560d4201110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.190+0000 7fc88aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4f40 0x5560d4201110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:52996/0 (socket says 172.21.15.28:52996) 2024-04-24T08:18:50.201 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.190+0000 7fc88aa00640 1 -- 172.21.15.28:0/1657858843 learned_addr learned my addr 172.21.15.28:0/1657858843 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:50.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.190+0000 7fc88be00640 1 --2- 172.21.15.28:0/1657858843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d4203c60 0x5560d4201650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.191+0000 7fc88b400640 1 --2- 172.21.15.28:0/1657858843 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d41f4590 0x5560d4200bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.191+0000 7fc88aa00640 1 -- 172.21.15.28:0/1657858843 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d41f4590 msgr2=0x5560d4200bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:50.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.191+0000 7fc88aa00640 1 --2- 172.21.15.28:0/1657858843 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d41f4590 0x5560d4200bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.191+0000 7fc88aa00640 1 -- 172.21.15.28:0/1657858843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d4203c60 msgr2=0x5560d4201650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:50.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.191+0000 7fc88aa00640 1 --2- 172.21.15.28:0/1657858843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d4203c60 0x5560d4201650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.191+0000 7fc88aa00640 1 -- 172.21.15.28:0/1657858843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560d4112920 con 0x5560d41f4f40 2024-04-24T08:18:50.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.191+0000 7fc88aa00640 1 --2- 172.21.15.28:0/1657858843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4f40 0x5560d4201110 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc87401b040 tx=0x7fc874004cf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b2fbb42c8a1d518 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:50.202 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.191+0000 7fc879600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:18:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.192+0000 7fc879600640 1 -- 172.21.15.28:0/1657858843 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc874027020 con 0x5560d41f4f40 2024-04-24T08:18:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.192+0000 7fc879600640 1 -- 172.21.15.28:0/1657858843 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7fc874004400 con 0x5560d41f4f40 2024-04-24T08:18:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.192+0000 7fc8955b6740 1 -- 172.21.15.28:0/1657858843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc874004050 con 0x5560d41f4f40 2024-04-24T08:18:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.192+0000 7fc879600640 1 -- 172.21.15.28:0/1657858843 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc874032050 con 0x5560d41f4f40 2024-04-24T08:18:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.192+0000 7fc8955b6740 10 client.16443 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.192+0000 7fc8955b6740 1 -- 172.21.15.28:0/1657858843 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5560d3f367d0 con 0x5560d41f4f40 2024-04-24T08:18:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.192+0000 7fc879600640 1 -- 172.21.15.28:0/1657858843 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc874004050 con 0x5560d41f4f40 2024-04-24T08:18:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.192+0000 7fc879600640 1 -- 172.21.15.28:0/1657858843 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc874020030 con 0x5560d41f4f40 2024-04-24T08:18:50.203 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.193+0000 7fc8955b6740 10 client.16443 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.193+0000 7fc8955b6740 10 client.16443 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:18:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.193+0000 7fc8955b6740 20 client.16443 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.193+0000 7fc8955b6740 1 --2- 172.21.15.28:0/1657858843 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5560d4213ec0 0x5560d42162b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.193+0000 7fc8955b6740 4 client.16443 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:18:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.193+0000 7fc8955b6740 1 -- 172.21.15.28:0/1657858843 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5560d4167c10 con 0x5560d4213ec0 2024-04-24T08:18:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.193+0000 7fc88b400640 1 --2- 172.21.15.28:0/1657858843 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5560d4213ec0 0x5560d42162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.194+0000 7fc88b400640 1 --2- 172.21.15.28:0/1657858843 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5560d4213ec0 0x5560d42162b0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=370d319b2a33a538 server_cookie=ce13c460af378944 in_seq=0 out_seq=0 2024-04-24T08:18:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.194+0000 7fc879600640 10 client.16443 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:50.204 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.194+0000 7fc879600640 1 -- 172.21.15.28:0/1657858843 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5560d4167c10 con 0x5560d4213ec0 2024-04-24T08:18:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.194+0000 7fc879600640 10 client.16443 handle_command_reply: tid=0 2024-04-24T08:18:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.194+0000 7fc8955b6740 1 client.16443 shutdown 2024-04-24T08:18:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.195+0000 7fc8955b6740 1 -- 172.21.15.28:0/1657858843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4f40 msgr2=0x5560d4201110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.195+0000 7fc8955b6740 1 --2- 172.21.15.28:0/1657858843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4f40 0x5560d4201110 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fc87401b040 tx=0x7fc874004cf0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.195+0000 7fc8955b6740 1 -- 172.21.15.28:0/1657858843 shutdown_connections 2024-04-24T08:18:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.195+0000 7fc8955b6740 1 --2- 172.21.15.28:0/1657858843 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5560d4213ec0 0x5560d42162b0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.195+0000 7fc8955b6740 1 --2- 172.21.15.28:0/1657858843 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4f40 0x5560d4201110 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.205 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.195+0000 7fc8955b6740 1 --2- 172.21.15.28:0/1657858843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d4203c60 0x5560d4201650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.195+0000 7fc8955b6740 1 --2- 172.21.15.28:0/1657858843 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d41f4590 0x5560d4200bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.195+0000 7fc879600640 0 client.16443 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.195+0000 7fc8955b6740 1 -- 172.21.15.28:0/1657858843 >> 172.21.15.28:0/1657858843 conn(0x5560d41f0b80 msgr2=0x5560d41f69d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.195+0000 7fc8955b6740 1 -- 172.21.15.28:0/1657858843 shutdown_connections 2024-04-24T08:18:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.195+0000 7fc8955b6740 1 -- 172.21.15.28:0/1657858843 wait complete. 2024-04-24T08:18:50.206 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.195+0000 7fc8955b6740 20 client.16443 trim_cache size 0 max 16384 2024-04-24T08:18:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.215+0000 7fc8955b6740 1 Processor -- start 2024-04-24T08:18:50.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.215+0000 7fc8955b6740 1 -- start start 2024-04-24T08:18:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.215+0000 7fc8955b6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d41f4590 0x5560d41ffb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.215+0000 7fc8955b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4f40 0x5560d420f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.215+0000 7fc8955b6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d4203c60 0x5560d421bab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.215+0000 7fc8955b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1ae520 con 0x5560d41f4f40 2024-04-24T08:18:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.215+0000 7fc8955b6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1abdf0 con 0x5560d4203c60 2024-04-24T08:18:50.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.215+0000 7fc8955b6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1a9c40 con 0x5560d41f4590 2024-04-24T08:18:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.215+0000 7fc88aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4f40 0x5560d420f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.215+0000 7fc88be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d4203c60 0x5560d421bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.215+0000 7fc88be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d4203c60 0x5560d421bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:57380/0 (socket says 172.21.15.28:57380) 2024-04-24T08:18:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.215+0000 7fc88aa00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4f40 0x5560d420f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53010/0 (socket says 172.21.15.28:53010) 2024-04-24T08:18:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.216+0000 7fc88be00640 1 -- 172.21.15.28:0/3522786934 learned_addr learned my addr 172.21.15.28:0/3522786934 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.216+0000 7fc88b400640 1 --2- 172.21.15.28:0/3522786934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d41f4590 0x5560d41ffb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.216+0000 7fc88b400640 1 -- 172.21.15.28:0/3522786934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d4203c60 msgr2=0x5560d421bab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.216+0000 7fc88b400640 1 --2- 172.21.15.28:0/3522786934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d4203c60 0x5560d421bab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.220 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.216+0000 7fc88b400640 1 -- 172.21.15.28:0/3522786934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4f40 msgr2=0x5560d420f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:50.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.216+0000 7fc88b400640 1 --2- 172.21.15.28:0/3522786934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4f40 0x5560d420f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.216+0000 7fc88b400640 1 -- 172.21.15.28:0/3522786934 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560d3e6c3f0 con 0x5560d41f4590 2024-04-24T08:18:50.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.216+0000 7fc88b400640 1 --2- 172.21.15.28:0/3522786934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d41f4590 0x5560d41ffb10 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0ce330 tx=0x7fc87c0bd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5246dcd28b7ce757 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:50.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.216+0000 7fc87be00640 1 -- 172.21.15.28:0/3522786934 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c09c4e0 con 0x5560d41f4590 2024-04-24T08:18:50.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.217+0000 7fc87be00640 1 -- 172.21.15.28:0/3522786934 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc87c09d280 con 0x5560d41f4590 2024-04-24T08:18:50.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.217+0000 7fc87be00640 1 -- 172.21.15.28:0/3522786934 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c0991f0 con 0x5560d41f4590 2024-04-24T08:18:50.221 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.217+0000 7fc8955b6740 1 -- 172.21.15.28:0/3522786934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d41f4590 msgr2=0x5560d41ffb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:50.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.217+0000 7fc8955b6740 1 --2- 172.21.15.28:0/3522786934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d41f4590 0x5560d41ffb10 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0ce330 tx=0x7fc87c0bd040 comp rx=0 tx=0).stop 2024-04-24T08:18:50.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.217+0000 7fc8955b6740 1 -- 172.21.15.28:0/3522786934 shutdown_connections 2024-04-24T08:18:50.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.217+0000 7fc8955b6740 1 --2- 172.21.15.28:0/3522786934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d41f4f40 0x5560d420f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.217+0000 7fc8955b6740 1 --2- 172.21.15.28:0/3522786934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d4203c60 0x5560d421bab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.217+0000 7fc8955b6740 1 --2- 172.21.15.28:0/3522786934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d41f4590 0x5560d41ffb10 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.217+0000 7fc8955b6740 1 -- 172.21.15.28:0/3522786934 >> 172.21.15.28:0/3522786934 conn(0x5560d41f0b80 msgr2=0x5560d41efc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:50.222 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.217+0000 7fc8955b6740 1 -- 172.21.15.28:0/3522786934 shutdown_connections 2024-04-24T08:18:50.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.217+0000 7fc8955b6740 1 -- 172.21.15.28:0/3522786934 wait complete. 2024-04-24T08:18:50.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.218+0000 7fc8955b6740 1 -- start start 2024-04-24T08:18:50.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.218+0000 7fc8955b6740 1 Processor -- start 2024-04-24T08:18:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc8955b6740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d41f4590 0x5560d420c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc8955b6740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d41f4f40 0x5560d41fd640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc8955b6740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d4203c60 0x5560d41fdd80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc8955b6740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1aa2b0 con 0x5560d4203c60 2024-04-24T08:18:50.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc8955b6740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1ab720 con 0x5560d41f4590 2024-04-24T08:18:50.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc8955b6740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc88c1aa980 con 0x5560d41f4f40 2024-04-24T08:18:50.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc8955b6740 10 client.0 _ll_register_callbacks cb 0x7fc8740093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:18:50.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc88be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d4203c60 0x5560d41fdd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc88b400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d41f4590 0x5560d420c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc88be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d4203c60 0x5560d41fdd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:53026/0 (socket says 172.21.15.28:53026) 2024-04-24T08:18:50.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc88be00640 1 -- 172.21.15.28:0/525447763 learned_addr learned my addr 172.21.15.28:0/525447763 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:18:50.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc88b400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d41f4590 0x5560d420c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:57384/0 (socket says 172.21.15.28:57384) 2024-04-24T08:18:50.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc88aa00640 1 --2- 172.21.15.28:0/525447763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d41f4f40 0x5560d41fd640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc88b400640 1 -- 172.21.15.28:0/525447763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d41f4f40 msgr2=0x5560d41fd640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:50.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc88b400640 1 --2- 172.21.15.28:0/525447763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d41f4f40 0x5560d41fd640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc88b400640 1 -- 172.21.15.28:0/525447763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d4203c60 msgr2=0x5560d41fdd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:18:50.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc88b400640 1 --2- 172.21.15.28:0/525447763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d4203c60 0x5560d41fdd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.219+0000 7fc88b400640 1 -- 172.21.15.28:0/525447763 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560d3e2c0f0 con 0x5560d41f4590 2024-04-24T08:18:50.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.220+0000 7fc88b400640 1 --2- 172.21.15.28:0/525447763 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d41f4590 0x5560d420c330 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0a10f0 tx=0x7fc87c09c780 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c68634320f9a166 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:18:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.220+0000 7fc879600640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:18:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.220+0000 7fc879600640 1 -- 172.21.15.28:0/525447763 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc87c09c4e0 con 0x5560d41f4590 2024-04-24T08:18:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.220+0000 7fc879600640 1 -- 172.21.15.28:0/525447763 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 8600+0+0 (secure 0 0 0) 0x7fc87c0ce5b0 con 0x5560d41f4590 2024-04-24T08:18:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.220+0000 7fc8955b6740 1 -- 172.21.15.28:0/525447763 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc87c09d280 con 0x5560d41f4590 2024-04-24T08:18:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.220+0000 7fc879600640 1 -- 172.21.15.28:0/525447763 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fc87c0ce7b0 con 0x5560d41f4590 2024-04-24T08:18:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.220+0000 7fc8955b6740 10 client.26726 fetch_fsmap learned FSMap version 26 2024-04-24T08:18:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.220+0000 7fc8955b6740 1 -- 172.21.15.28:0/525447763 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5560d411f810 con 0x5560d41f4590 2024-04-24T08:18:50.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.220+0000 7fc879600640 1 -- 172.21.15.28:0/525447763 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc87c09d280 con 0x5560d41f4590 2024-04-24T08:18:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.221+0000 7fc879600640 1 -- 172.21.15.28:0/525447763 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fc87c09bfd0 con 0x5560d41f4590 2024-04-24T08:18:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.221+0000 7fc8955b6740 10 client.26726 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:18:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.221+0000 7fc8955b6740 10 client.26726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:18:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.221+0000 7fc8955b6740 20 client.26726 populate_metadata read hostname 'smithi028' 2024-04-24T08:18:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.221+0000 7fc8955b6740 1 --2- 172.21.15.28:0/525447763 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5560d427a650 0x5560d427ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:18:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.221+0000 7fc8955b6740 4 client.26726 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:18:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.221+0000 7fc8955b6740 1 -- 172.21.15.28:0/525447763 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5560d4161a80 con 0x5560d427a650 2024-04-24T08:18:50.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.221+0000 7fc88aa00640 1 --2- 172.21.15.28:0/525447763 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5560d427a650 0x5560d427ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:18:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.221+0000 7fc88aa00640 1 --2- 172.21.15.28:0/525447763 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5560d427a650 0x5560d427ca70 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d365cb08325de9b server_cookie=86fab5dd4f8afee1 in_seq=0 out_seq=0 2024-04-24T08:18:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.221+0000 7fc879600640 10 client.26726 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.222+0000 7fc879600640 1 -- 172.21.15.28:0/525447763 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5560d4161a80 con 0x5560d427a650 2024-04-24T08:18:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.222+0000 7fc879600640 10 client.26726 handle_command_reply: tid=0 2024-04-24T08:18:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.222+0000 7fc8955b6740 1 client.26726 shutdown 2024-04-24T08:18:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.222+0000 7fc8955b6740 1 -- 172.21.15.28:0/525447763 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d41f4590 msgr2=0x5560d420c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.222+0000 7fc8955b6740 1 --2- 172.21.15.28:0/525447763 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d41f4590 0x5560d420c330 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7fc87c0a10f0 tx=0x7fc87c09c780 comp rx=0 tx=0).stop 2024-04-24T08:18:50.231 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.222+0000 7fc8955b6740 1 -- 172.21.15.28:0/525447763 shutdown_connections 2024-04-24T08:18:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.222+0000 7fc8955b6740 1 --2- 172.21.15.28:0/525447763 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5560d427a650 0x5560d427ca70 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.222+0000 7fc8955b6740 1 --2- 172.21.15.28:0/525447763 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5560d4203c60 0x5560d41fdd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.222+0000 7fc8955b6740 1 --2- 172.21.15.28:0/525447763 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5560d41f4f40 0x5560d41fd640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.222+0000 7fc879600640 0 client.26726 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:18:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.222+0000 7fc8955b6740 1 --2- 172.21.15.28:0/525447763 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5560d41f4590 0x5560d420c330 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.222+0000 7fc8955b6740 1 -- 172.21.15.28:0/525447763 >> 172.21.15.28:0/525447763 conn(0x5560d41f0b80 msgr2=0x5560d41f6140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:50.232 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.222+0000 7fc8955b6740 1 -- 172.21.15.28:0/525447763 shutdown_connections 2024-04-24T08:18:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.222+0000 7fc8955b6740 1 -- 172.21.15.28:0/525447763 wait complete. 2024-04-24T08:18:50.233 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.222+0000 7fc8955b6740 20 client.26726 trim_cache size 0 max 16384 2024-04-24T08:18:50.233 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:18:50.233 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:18:50.233 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:18:50.234 INFO:teuthology.orchestra.run.smithi028.stdout: "11fa8315-0ebc-4f6e-9377-d1cec5e3bd3c": { 2024-04-24T08:18:50.234 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:18:50.234 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "11fa8315-0ebc-4f6e-9377-d1cec5e3bd3c", 2024-04-24T08:18:50.234 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:18:50.234 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:18:50.234 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:18:50.234 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:18:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.223+0000 7fc892800640 1 -- 172.21.15.28:0/883688207 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc86c0a3d20 msgr2=0x7fc86c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:50.235 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.223+0000 7fc892800640 1 --2- 172.21.15.28:0/883688207 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc86c0a3d20 0x7fc86c0a61e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc884031550 tx=0x7fc8840046c0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.223+0000 7fc892800640 1 -- 172.21.15.28:0/883688207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc88c102240 msgr2=0x7fc88c19e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:18:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.223+0000 7fc892800640 1 --2- 172.21.15.28:0/883688207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc88c102240 0x7fc88c19e5e0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fc87c00e9f0 tx=0x7fc87c00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.223+0000 7fc892800640 1 -- 172.21.15.28:0/883688207 shutdown_connections 2024-04-24T08:18:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.223+0000 7fc892800640 1 --2- 172.21.15.28:0/883688207 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc86c0a3d20 0x7fc86c0a61e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.223+0000 7fc892800640 1 --2- 172.21.15.28:0/883688207 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc88c1986a0 0x7fc88c198b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.223+0000 7fc892800640 1 --2- 172.21.15.28:0/883688207 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc88c107310 0x7fc88c19eb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.223+0000 7fc892800640 1 --2- 172.21.15.28:0/883688207 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc88c102240 0x7fc88c19e5e0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:18:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.223+0000 7fc892800640 1 -- 172.21.15.28:0/883688207 >> 172.21.15.28:0/883688207 conn(0x7fc88c079450 msgr2=0x7fc88c0ffd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:18:50.236 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.223+0000 7fc892800640 1 -- 172.21.15.28:0/883688207 shutdown_connections 2024-04-24T08:18:50.237 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:18:50.223+0000 7fc892800640 1 -- 172.21.15.28:0/883688207 wait complete. 2024-04-24T08:18:50.241 INFO:tasks.cephfs.filesystem:scrub status for tag:11fa8315-0ebc-4f6e-9377-d1cec5e3bd3c - {'path': '/', 'tag': '11fa8315-0ebc-4f6e-9377-d1cec5e3bd3c', 'options': 'recursive,force'} 2024-04-24T08:18:50.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:50 smithi028 ceph-mon[27786]: scrub queued for path: / 2024-04-24T08:18:50.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:50 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:50.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:50 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:18:50.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:50 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:50.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:50 smithi117 ceph-mon[32438]: scrub queued for path: / 2024-04-24T08:18:50.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:50 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:50.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:50 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:18:50.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:50 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:50.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:50 smithi194 ceph-mon[32836]: scrub queued for path: / 2024-04-24T08:18:50.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:50 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:50.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:50 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:18:50.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:50 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:51.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:51 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:18:51.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:51 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:51.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:51 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:18:51.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:51 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:51.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:51 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:18:51.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:51 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:51.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:51 smithi028 ceph-mon[27786]: pgmap v262: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-04-24T08:18:51.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:51 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:18:51.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:51 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:51.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:51 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:18:51.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:51 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:51.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:51 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:18:51.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:51 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:51.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:51 smithi117 ceph-mon[32438]: pgmap v262: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-04-24T08:18:51.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:51 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:18:51.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:51 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:51.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:51 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:18:51.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:51 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:51.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:51 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:18:51.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:51 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:18:51.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:51 smithi194 ceph-mon[32836]: pgmap v262: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.3 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-04-24T08:18:53.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:53 smithi028 ceph-mon[27786]: pgmap v263: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.0 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-04-24T08:18:53.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:53 smithi117 ceph-mon[32438]: pgmap v263: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.0 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-04-24T08:18:53.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:53 smithi194 ceph-mon[32836]: pgmap v263: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 6.0 KiB/s rd, 3.0 KiB/s wr, 6 op/s 2024-04-24T08:18:55.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:55 smithi028 ceph-mon[27786]: pgmap v264: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-24T08:18:55.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:55 smithi117 ceph-mon[32438]: pgmap v264: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-24T08:18:55.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:55 smithi194 ceph-mon[32836]: pgmap v264: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-24T08:18:57.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:57 smithi028 ceph-mon[27786]: pgmap v265: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:18:57.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:57 smithi117 ceph-mon[32438]: pgmap v265: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:18:57.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:57 smithi194 ceph-mon[32836]: pgmap v265: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:18:59.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:59 smithi028 ceph-mon[27786]: pgmap v266: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-24T08:18:59.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:18:59 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:18:59.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:59 smithi117 ceph-mon[32438]: pgmap v266: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-24T08:18:59.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:18:59 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:18:59.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:59 smithi194 ceph-mon[32836]: pgmap v266: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-24T08:18:59.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:18:59 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:19:00.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:00 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:19:00.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:00 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:19:00.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:00 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:19:01.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:01 smithi028 ceph-mon[27786]: pgmap v267: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-24T08:19:01.841 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:01 smithi194 ceph-mon[32836]: pgmap v267: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-24T08:19:01.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:01 smithi117 ceph-mon[32438]: pgmap v267: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-24T08:19:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:03 smithi028 ceph-mon[27786]: pgmap v268: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:19:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:03 smithi117 ceph-mon[32438]: pgmap v268: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:19:03.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:03 smithi194 ceph-mon[32836]: pgmap v268: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:19:05.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:05 smithi028 ceph-mon[27786]: pgmap v269: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-24T08:19:05.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:05 smithi117 ceph-mon[32438]: pgmap v269: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-24T08:19:05.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:05 smithi194 ceph-mon[32836]: pgmap v269: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-24T08:19:07.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:07 smithi028 ceph-mon[27786]: pgmap v270: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-24T08:19:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:07 smithi117 ceph-mon[32438]: pgmap v270: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-24T08:19:07.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:07 smithi194 ceph-mon[32836]: pgmap v270: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-24T08:19:09.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:09 smithi028 ceph-mon[27786]: pgmap v271: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-24T08:19:09.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:09 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:19:09.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:09 smithi117 ceph-mon[32438]: pgmap v271: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-24T08:19:09.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:09 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:19:09.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:09 smithi194 ceph-mon[32836]: pgmap v271: 97 pgs: 97 active+clean; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-24T08:19:09.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:09 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:19:10.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:10 smithi117 ceph-mon[32438]: osdmap e96: 12 total, 12 up, 12 in 2024-04-24T08:19:10.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:10 smithi194 ceph-mon[32836]: osdmap e96: 12 total, 12 up, 12 in 2024-04-24T08:19:11.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:10 smithi028 ceph-mon[27786]: osdmap e96: 12 total, 12 up, 12 in 2024-04-24T08:19:11.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:11 smithi117 ceph-mon[32438]: pgmap v273: 97 pgs: 2 active+clean+snaptrim, 91 active+clean, 4 active+clean+snaptrim_wait; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-24T08:19:11.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:11 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:19:11.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:11 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:19:11.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:11 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:19:11.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:11 smithi194 ceph-mon[32836]: pgmap v273: 97 pgs: 2 active+clean+snaptrim, 91 active+clean, 4 active+clean+snaptrim_wait; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-24T08:19:11.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:11 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:19:11.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:11 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:19:11.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:11 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:19:11.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:11 smithi028 ceph-mon[27786]: pgmap v273: 97 pgs: 2 active+clean+snaptrim, 91 active+clean, 4 active+clean+snaptrim_wait; 24 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-24T08:19:11.941 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:11 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:19:11.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:11 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:19:11.942 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:11 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:19:13.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:13 smithi117 ceph-mon[32438]: pgmap v274: 97 pgs: 2 active+clean+snaptrim, 91 active+clean, 4 active+clean+snaptrim_wait; 22 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:19:13.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:13 smithi194 ceph-mon[32836]: pgmap v274: 97 pgs: 2 active+clean+snaptrim, 91 active+clean, 4 active+clean+snaptrim_wait; 22 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:19:14.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:13 smithi028 ceph-mon[27786]: pgmap v274: 97 pgs: 2 active+clean+snaptrim, 91 active+clean, 4 active+clean+snaptrim_wait; 22 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:19:15.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:15 smithi117 ceph-mon[32438]: pgmap v275: 97 pgs: 2 active+clean+snaptrim, 91 active+clean, 4 active+clean+snaptrim_wait; 21 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:19:15.953 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:15 smithi194 ceph-mon[32836]: pgmap v275: 97 pgs: 2 active+clean+snaptrim, 91 active+clean, 4 active+clean+snaptrim_wait; 21 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:19:16.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:15 smithi028 ceph-mon[27786]: pgmap v275: 97 pgs: 2 active+clean+snaptrim, 91 active+clean, 4 active+clean+snaptrim_wait; 21 GiB data, 54 GiB used, 1019 GiB / 1.0 TiB avail; 4.3 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:19:17.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:17 smithi117 ceph-mon[32438]: pgmap v276: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:19:17.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:17 smithi194 ceph-mon[32836]: pgmap v276: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:19:18.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:17 smithi028 ceph-mon[27786]: pgmap v276: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:19:19.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:19 smithi117 ceph-mon[32438]: pgmap v277: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:19:19.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:19 smithi117 ceph-mon[32438]: osdmap e97: 12 total, 12 up, 12 in 2024-04-24T08:19:19.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:19 smithi194 ceph-mon[32836]: pgmap v277: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:19:19.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:19 smithi194 ceph-mon[32836]: osdmap e97: 12 total, 12 up, 12 in 2024-04-24T08:19:20.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:19 smithi028 ceph-mon[27786]: pgmap v277: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:19:20.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:19 smithi028 ceph-mon[27786]: osdmap e97: 12 total, 12 up, 12 in 2024-04-24T08:19:20.242 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:19:20.242 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:19:20.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.516+0000 7fd58b200640 1 -- 172.21.15.28:0/727803027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd58c10e8f0 msgr2=0x7fd58c110ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:20.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.516+0000 7fd58b200640 1 --2- 172.21.15.28:0/727803027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd58c10e8f0 0x7fd58c110ce0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd584009ca0 tx=0x7fd58402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.516+0000 7fd58b200640 1 -- 172.21.15.28:0/727803027 shutdown_connections 2024-04-24T08:19:20.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.516+0000 7fd58b200640 1 --2- 172.21.15.28:0/727803027 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd58c10e8f0 0x7fd58c110ce0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.516+0000 7fd58b200640 1 --2- 172.21.15.28:0/727803027 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd58c100cd0 0x7fd58c10e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.516+0000 7fd58b200640 1 --2- 172.21.15.28:0/727803027 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd58c1003b0 0x7fd58c100790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.516+0000 7fd58b200640 1 -- 172.21.15.28:0/727803027 >> 172.21.15.28:0/727803027 conn(0x7fd58c075d00 msgr2=0x7fd58c076110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:20.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.516+0000 7fd58b200640 1 -- 172.21.15.28:0/727803027 shutdown_connections 2024-04-24T08:19:20.519 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.516+0000 7fd58b200640 1 -- 172.21.15.28:0/727803027 wait complete. 2024-04-24T08:19:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.517+0000 7fd58b200640 1 Processor -- start 2024-04-24T08:19:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.517+0000 7fd58b200640 1 -- start start 2024-04-24T08:19:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.518+0000 7fd58b200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd58c1003b0 0x7fd58c1a0660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.521 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.518+0000 7fd58b200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd58c100cd0 0x7fd58c1a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.518+0000 7fd58b200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd58c10e8f0 0x7fd58c19a720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.518+0000 7fd58b200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1144a0 con 0x7fd58c100cd0 2024-04-24T08:19:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.518+0000 7fd58b200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c114320 con 0x7fd58c1003b0 2024-04-24T08:19:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.518+0000 7fd58b200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c114620 con 0x7fd58c10e8f0 2024-04-24T08:19:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.518+0000 7fd583e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd58c100cd0 0x7fd58c1a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.518+0000 7fd589e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd58c1003b0 0x7fd58c1a0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.518+0000 7fd589e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd58c1003b0 0x7fd58c1a0660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:52746/0 (socket says 172.21.15.28:52746) 2024-04-24T08:19:20.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.518+0000 7fd589e00640 1 -- 172.21.15.28:0/4102569013 learned_addr learned my addr 172.21.15.28:0/4102569013 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.518+0000 7fd58a800640 1 --2- 172.21.15.28:0/4102569013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd58c10e8f0 0x7fd58c19a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.518+0000 7fd589e00640 1 -- 172.21.15.28:0/4102569013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd58c10e8f0 msgr2=0x7fd58c19a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.518+0000 7fd589e00640 1 --2- 172.21.15.28:0/4102569013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd58c10e8f0 0x7fd58c19a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.518+0000 7fd589e00640 1 -- 172.21.15.28:0/4102569013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd58c100cd0 msgr2=0x7fd58c1a0ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.518+0000 7fd589e00640 1 --2- 172.21.15.28:0/4102569013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd58c100cd0 0x7fd58c1a0ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.518+0000 7fd589e00640 1 -- 172.21.15.28:0/4102569013 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd58c19afe0 con 0x7fd58c1003b0 2024-04-24T08:19:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.518+0000 7fd58a800640 1 --2- 172.21.15.28:0/4102569013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd58c10e8f0 0x7fd58c19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:20.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.519+0000 7fd589e00640 1 --2- 172.21.15.28:0/4102569013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd58c1003b0 0x7fd58c1a0660 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00e9f0 tx=0x7fd57c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.519+0000 7fd582a00640 1 -- 172.21.15.28:0/4102569013 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c00cd80 con 0x7fd58c1003b0 2024-04-24T08:19:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.519+0000 7fd58b200640 1 -- 172.21.15.28:0/4102569013 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd58c19b2d0 con 0x7fd58c1003b0 2024-04-24T08:19:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.519+0000 7fd582a00640 1 -- 172.21.15.28:0/4102569013 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd57c010910 con 0x7fd58c1003b0 2024-04-24T08:19:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.519+0000 7fd582a00640 1 -- 172.21.15.28:0/4102569013 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c0188f0 con 0x7fd58c1003b0 2024-04-24T08:19:20.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.519+0000 7fd58b200640 1 -- 172.21.15.28:0/4102569013 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd58c1a75e0 con 0x7fd58c1003b0 2024-04-24T08:19:20.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.522+0000 7fd593973740 1 Processor -- start 2024-04-24T08:19:20.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.522+0000 7fd593973740 1 -- start start 2024-04-24T08:19:20.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.523+0000 7fd582a00640 1 -- 172.21.15.28:0/4102569013 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fd57c010ac0 con 0x7fd58c1003b0 2024-04-24T08:19:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.523+0000 7fd593973740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476ea30 0x555bf476ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.523+0000 7fd593973740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf476f350 0x555bf47794e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.523+0000 7fd593973740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf4779a20 0x555bf477be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.523+0000 7fd593973740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555bf46d41f0 con 0x555bf476ea30 2024-04-24T08:19:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.523+0000 7fd593973740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555bf467a350 con 0x555bf476f350 2024-04-24T08:19:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.523+0000 7fd593973740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555bf46d01b0 con 0x555bf4779a20 2024-04-24T08:19:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.523+0000 7fd58a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476ea30 0x555bf476ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.523+0000 7fd583e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf4779a20 0x555bf477be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.523+0000 7fd58a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476ea30 0x555bf476ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47596/0 (socket says 172.21.15.28:47596) 2024-04-24T08:19:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.523+0000 7fd58a800640 1 -- 172.21.15.28:0/3304158580 learned_addr learned my addr 172.21.15.28:0/3304158580 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.523+0000 7fd589e00640 1 --2- 172.21.15.28:0/3304158580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf476f350 0x555bf47794e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.524+0000 7fd583e00640 1 -- 172.21.15.28:0/3304158580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf476f350 msgr2=0x555bf47794e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.524+0000 7fd583e00640 1 --2- 172.21.15.28:0/3304158580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf476f350 0x555bf47794e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.524+0000 7fd583e00640 1 -- 172.21.15.28:0/3304158580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476ea30 msgr2=0x555bf476ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.524+0000 7fd583e00640 1 --2- 172.21.15.28:0/3304158580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476ea30 0x555bf476ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.524+0000 7fd583e00640 1 -- 172.21.15.28:0/3304158580 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bf46cef50 con 0x555bf4779a20 2024-04-24T08:19:20.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.524+0000 7fd58a800640 1 --2- 172.21.15.28:0/3304158580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476ea30 0x555bf476ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.524+0000 7fd583e00640 1 --2- 172.21.15.28:0/3304158580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf4779a20 0x555bf477be10 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd574009bf0 tx=0x7fd574011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5db9ff03457b8e0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.524+0000 7fd582a00640 1 --2- 172.21.15.28:0/4102569013 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd5700a3d20 0x7fd5700a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.525+0000 7fd582a00640 1 -- 172.21.15.28:0/4102569013 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7fd57c0d0cf0 con 0x7fd58c1003b0 2024-04-24T08:19:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.525+0000 7fd57b000640 1 -- 172.21.15.28:0/3304158580 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd574004050 con 0x555bf4779a20 2024-04-24T08:19:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.525+0000 7fd57b000640 1 -- 172.21.15.28:0/3304158580 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd574005500 con 0x555bf4779a20 2024-04-24T08:19:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.525+0000 7fd57b000640 1 -- 172.21.15.28:0/3304158580 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd574004d70 con 0x555bf4779a20 2024-04-24T08:19:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.525+0000 7fd58a800640 1 --2- 172.21.15.28:0/4102569013 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd5700a3d20 0x7fd5700a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.525+0000 7fd593973740 1 -- 172.21.15.28:0/3304158580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf4779a20 msgr2=0x555bf477be10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.525+0000 7fd593973740 1 --2- 172.21.15.28:0/3304158580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf4779a20 0x555bf477be10 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fd574009bf0 tx=0x7fd574011810 comp rx=0 tx=0).stop 2024-04-24T08:19:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.525+0000 7fd58a800640 1 --2- 172.21.15.28:0/4102569013 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd5700a3d20 0x7fd5700a61e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd584009430 tx=0x7fd584004a40 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.526+0000 7fd593973740 1 -- 172.21.15.28:0/3304158580 shutdown_connections 2024-04-24T08:19:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.526+0000 7fd593973740 1 --2- 172.21.15.28:0/3304158580 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf4779a20 0x555bf477be10 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.526+0000 7fd593973740 1 --2- 172.21.15.28:0/3304158580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf476f350 0x555bf47794e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.526+0000 7fd593973740 1 --2- 172.21.15.28:0/3304158580 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476ea30 0x555bf476ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.526+0000 7fd593973740 1 -- 172.21.15.28:0/3304158580 >> 172.21.15.28:0/3304158580 conn(0x555bf476ad50 msgr2=0x555bf476b130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:20.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.526+0000 7fd593973740 1 -- 172.21.15.28:0/3304158580 shutdown_connections 2024-04-24T08:19:20.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.526+0000 7fd593973740 1 -- 172.21.15.28:0/3304158580 wait complete. 2024-04-24T08:19:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.526+0000 7fd593973740 1 -- start start 2024-04-24T08:19:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.526+0000 7fd593973740 1 Processor -- start 2024-04-24T08:19:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.527+0000 7fd593973740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf476ea30 0x555bf4776920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.527+0000 7fd593973740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476f350 0x555bf4776e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.527+0000 7fd593973740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf4779a20 0x555bf47773a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.527+0000 7fd593973740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x555bf477c810 con 0x555bf476f350 2024-04-24T08:19:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.527+0000 7fd593973740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x555bf468d690 con 0x555bf476ea30 2024-04-24T08:19:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.527+0000 7fd593973740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x555bf477c990 con 0x555bf4779a20 2024-04-24T08:19:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.527+0000 7fd593973740 10 client.0 _ll_register_callbacks cb 0x555bf46c9650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.527+0000 7fd583e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476f350 0x555bf4776e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.527+0000 7fd589e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf476ea30 0x555bf4776920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.527+0000 7fd583e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476f350 0x555bf4776e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47608/0 (socket says 172.21.15.28:47608) 2024-04-24T08:19:20.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.527+0000 7fd58a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf4779a20 0x555bf47773a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.527+0000 7fd583e00640 1 -- 172.21.15.28:0/718759174 learned_addr learned my addr 172.21.15.28:0/718759174 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.528+0000 7fd583e00640 1 -- 172.21.15.28:0/718759174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf4779a20 msgr2=0x555bf47773a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.528+0000 7fd583e00640 1 --2- 172.21.15.28:0/718759174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf4779a20 0x555bf47773a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.528+0000 7fd583e00640 1 -- 172.21.15.28:0/718759174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf476ea30 msgr2=0x555bf4776920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.528+0000 7fd583e00640 1 --2- 172.21.15.28:0/718759174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf476ea30 0x555bf4776920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.528+0000 7fd583e00640 1 -- 172.21.15.28:0/718759174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bf468c920 con 0x555bf476f350 2024-04-24T08:19:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.528+0000 7fd589e00640 1 --2- 172.21.15.28:0/718759174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf476ea30 0x555bf4776920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.528+0000 7fd583e00640 1 --2- 172.21.15.28:0/718759174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476f350 0x555bf4776e60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd574009730 tx=0x7fd574009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2608598d4465b54 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:20.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.528+0000 7fd56f400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:19:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.529+0000 7fd56f400640 1 -- 172.21.15.28:0/718759174 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd574004f20 con 0x555bf476f350 2024-04-24T08:19:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.529+0000 7fd593973740 1 -- 172.21.15.28:0/718759174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd574005500 con 0x555bf476f350 2024-04-24T08:19:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.529+0000 7fd56f400640 1 -- 172.21.15.28:0/718759174 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7fd5740050c0 con 0x555bf476f350 2024-04-24T08:19:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.529+0000 7fd56f400640 1 -- 172.21.15.28:0/718759174 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd574033050 con 0x555bf476f350 2024-04-24T08:19:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.529+0000 7fd593973740 10 client.16461 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.529+0000 7fd593973740 1 -- 172.21.15.28:0/718759174 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555bf44b07d0 con 0x555bf476f350 2024-04-24T08:19:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.529+0000 7fd56f400640 1 -- 172.21.15.28:0/718759174 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd574005500 con 0x555bf476f350 2024-04-24T08:19:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.530+0000 7fd56f400640 1 -- 172.21.15.28:0/718759174 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd57402e440 con 0x555bf476f350 2024-04-24T08:19:20.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.530+0000 7fd593973740 10 client.16461 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.530+0000 7fd593973740 10 client.16461 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:19:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.530+0000 7fd593973740 20 client.16461 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.530+0000 7fd593973740 1 --2- 172.21.15.28:0/718759174 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x555bf4789c70 0x555bf478c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.530+0000 7fd593973740 4 client.16461 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:19:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.530+0000 7fd593973740 1 -- 172.21.15.28:0/718759174 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555bf46e1c10 con 0x555bf4789c70 2024-04-24T08:19:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.530+0000 7fd589e00640 1 --2- 172.21.15.28:0/718759174 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x555bf4789c70 0x555bf478c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.531+0000 7fd589e00640 1 --2- 172.21.15.28:0/718759174 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x555bf4789c70 0x555bf478c060 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39c4ca7be680ba66 server_cookie=36303a69d7d5db86 in_seq=0 out_seq=0 2024-04-24T08:19:20.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.531+0000 7fd56f400640 10 client.16461 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:20.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.532+0000 7fd56f400640 1 -- 172.21.15.28:0/718759174 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555bf46e1c10 con 0x555bf4789c70 2024-04-24T08:19:20.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.532+0000 7fd56f400640 10 client.16461 handle_command_reply: tid=0 2024-04-24T08:19:20.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.532+0000 7fd593973740 1 client.16461 shutdown 2024-04-24T08:19:20.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.532+0000 7fd593973740 1 -- 172.21.15.28:0/718759174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476f350 msgr2=0x555bf4776e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:20.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.532+0000 7fd593973740 1 --2- 172.21.15.28:0/718759174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476f350 0x555bf4776e60 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd574009730 tx=0x7fd574009fd0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.532+0000 7fd593973740 1 -- 172.21.15.28:0/718759174 shutdown_connections 2024-04-24T08:19:20.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.532+0000 7fd593973740 1 --2- 172.21.15.28:0/718759174 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x555bf4789c70 0x555bf478c060 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.532+0000 7fd593973740 1 --2- 172.21.15.28:0/718759174 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476f350 0x555bf4776e60 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.532+0000 7fd593973740 1 --2- 172.21.15.28:0/718759174 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf4779a20 0x555bf47773a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.532+0000 7fd593973740 1 --2- 172.21.15.28:0/718759174 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf476ea30 0x555bf4776920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.532+0000 7fd593973740 1 -- 172.21.15.28:0/718759174 >> 172.21.15.28:0/718759174 conn(0x555bf476ad50 msgr2=0x555bf4769fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:20.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.532+0000 7fd56f400640 0 client.16461 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:20.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.532+0000 7fd593973740 1 -- 172.21.15.28:0/718759174 shutdown_connections 2024-04-24T08:19:20.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.532+0000 7fd593973740 1 -- 172.21.15.28:0/718759174 wait complete. 2024-04-24T08:19:20.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.533+0000 7fd593973740 20 client.16461 trim_cache size 0 max 16384 2024-04-24T08:19:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.552+0000 7fd593973740 1 Processor -- start 2024-04-24T08:19:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.552+0000 7fd593973740 1 -- start start 2024-04-24T08:19:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.552+0000 7fd593973740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf476ea30 0x555bf4785700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.553+0000 7fd593973740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476f350 0x555bf476a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.554 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.553+0000 7fd593973740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf4779a20 0x555bf4776ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.553+0000 7fd593973740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1b27a0 con 0x555bf476f350 2024-04-24T08:19:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.553+0000 7fd593973740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1b0070 con 0x555bf4779a20 2024-04-24T08:19:20.555 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.553+0000 7fd593973740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1adee0 con 0x555bf476ea30 2024-04-24T08:19:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.553+0000 7fd583e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476f350 0x555bf476a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.553+0000 7fd583e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476f350 0x555bf476a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47624/0 (socket says 172.21.15.28:47624) 2024-04-24T08:19:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.553+0000 7fd583e00640 1 -- 172.21.15.28:0/1444035524 learned_addr learned my addr 172.21.15.28:0/1444035524 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.553+0000 7fd589e00640 1 --2- 172.21.15.28:0/1444035524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf476ea30 0x555bf4785700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.553+0000 7fd58a800640 1 --2- 172.21.15.28:0/1444035524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf4779a20 0x555bf4776ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.553+0000 7fd589e00640 1 -- 172.21.15.28:0/1444035524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf4779a20 msgr2=0x555bf4776ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:20.557 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.553+0000 7fd589e00640 1 --2- 172.21.15.28:0/1444035524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf4779a20 0x555bf4776ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.553+0000 7fd589e00640 1 -- 172.21.15.28:0/1444035524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476f350 msgr2=0x555bf476a750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.553+0000 7fd589e00640 1 --2- 172.21.15.28:0/1444035524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476f350 0x555bf476a750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.553+0000 7fd589e00640 1 -- 172.21.15.28:0/1444035524 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555bf478c3d0 con 0x555bf476ea30 2024-04-24T08:19:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.553+0000 7fd583e00640 1 --2- 172.21.15.28:0/1444035524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476f350 0x555bf476a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.554+0000 7fd589e00640 1 --2- 172.21.15.28:0/1444035524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf476ea30 0x555bf4785700 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0ce390 tx=0x7fd57c0d3140 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6dc0c7e6e006b699 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.554+0000 7fd57a600640 1 -- 172.21.15.28:0/1444035524 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c09d2e0 con 0x555bf476ea30 2024-04-24T08:19:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.554+0000 7fd57a600640 1 -- 172.21.15.28:0/1444035524 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd57c09cbe0 con 0x555bf476ea30 2024-04-24T08:19:20.558 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.554+0000 7fd57a600640 1 -- 172.21.15.28:0/1444035524 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c0bd240 con 0x555bf476ea30 2024-04-24T08:19:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.555+0000 7fd593973740 1 -- 172.21.15.28:0/1444035524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf476ea30 msgr2=0x555bf4785700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.555+0000 7fd593973740 1 --2- 172.21.15.28:0/1444035524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf476ea30 0x555bf4785700 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0ce390 tx=0x7fd57c0d3140 comp rx=0 tx=0).stop 2024-04-24T08:19:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.555+0000 7fd593973740 1 -- 172.21.15.28:0/1444035524 shutdown_connections 2024-04-24T08:19:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.555+0000 7fd593973740 1 --2- 172.21.15.28:0/1444035524 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf476f350 0x555bf476a750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.555+0000 7fd593973740 1 --2- 172.21.15.28:0/1444035524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf4779a20 0x555bf4776ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.555+0000 7fd593973740 1 --2- 172.21.15.28:0/1444035524 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf476ea30 0x555bf4785700 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.555+0000 7fd593973740 1 -- 172.21.15.28:0/1444035524 >> 172.21.15.28:0/1444035524 conn(0x555bf476ad50 msgr2=0x555bf47768e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.555+0000 7fd593973740 1 -- 172.21.15.28:0/1444035524 shutdown_connections 2024-04-24T08:19:20.559 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.555+0000 7fd593973740 1 -- 172.21.15.28:0/1444035524 wait complete. 2024-04-24T08:19:20.560 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.556+0000 7fd593973740 1 -- start start 2024-04-24T08:19:20.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.556+0000 7fd593973740 1 Processor -- start 2024-04-24T08:19:20.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.556+0000 7fd593973740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf476f350 0x555bf4781c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.556+0000 7fd593973740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf4779a20 0x555bf4782150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.556+0000 7fd593973740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf477ebe0 0x555bf477f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.556+0000 7fd593973740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1ae530 con 0x555bf477ebe0 2024-04-24T08:19:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.556+0000 7fd593973740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1af9a0 con 0x555bf476f350 2024-04-24T08:19:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.556+0000 7fd593973740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c1aec00 con 0x555bf4779a20 2024-04-24T08:19:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.556+0000 7fd593973740 10 client.0 _ll_register_callbacks cb 0x7fd57c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.557+0000 7fd58a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf477ebe0 0x555bf477f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.557+0000 7fd58a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf477ebe0 0x555bf477f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47640/0 (socket says 172.21.15.28:47640) 2024-04-24T08:19:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.557+0000 7fd589e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf476f350 0x555bf4781c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.557+0000 7fd58a800640 1 -- 172.21.15.28:0/3950213935 learned_addr learned my addr 172.21.15.28:0/3950213935 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:20.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.557+0000 7fd583e00640 1 --2- 172.21.15.28:0/3950213935 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf4779a20 0x555bf4782150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.557+0000 7fd58a800640 1 -- 172.21.15.28:0/3950213935 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf4779a20 msgr2=0x555bf4782150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.557+0000 7fd58a800640 1 --2- 172.21.15.28:0/3950213935 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf4779a20 0x555bf4782150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.557+0000 7fd58a800640 1 -- 172.21.15.28:0/3950213935 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf476f350 msgr2=0x555bf4781c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.557+0000 7fd58a800640 1 --2- 172.21.15.28:0/3950213935 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf476f350 0x555bf4781c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.557+0000 7fd58a800640 1 -- 172.21.15.28:0/3950213935 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555bf43e63f0 con 0x555bf477ebe0 2024-04-24T08:19:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.557+0000 7fd589e00640 1 --2- 172.21.15.28:0/3950213935 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf476f350 0x555bf4781c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.557+0000 7fd58a800640 1 --2- 172.21.15.28:0/3950213935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf477ebe0 0x555bf477f060 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd584009ca0 tx=0x7fd58402a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1a9d5b5e376fe1a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:20.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.557+0000 7fd56f400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:19:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.558+0000 7fd56f400640 1 -- 172.21.15.28:0/3950213935 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd584059060 con 0x555bf477ebe0 2024-04-24T08:19:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.558+0000 7fd56f400640 1 -- 172.21.15.28:0/3950213935 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7fd584031660 con 0x555bf477ebe0 2024-04-24T08:19:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.558+0000 7fd593973740 1 -- 172.21.15.28:0/3950213935 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd57c09cbe0 con 0x555bf477ebe0 2024-04-24T08:19:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.558+0000 7fd56f400640 1 -- 172.21.15.28:0/3950213935 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd584031860 con 0x555bf477ebe0 2024-04-24T08:19:20.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.558+0000 7fd593973740 10 client.16467 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:20.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.558+0000 7fd593973740 1 -- 172.21.15.28:0/3950213935 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x555bf43a60f0 con 0x555bf477ebe0 2024-04-24T08:19:20.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.558+0000 7fd56f400640 1 -- 172.21.15.28:0/3950213935 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd57c09cbe0 con 0x555bf477ebe0 2024-04-24T08:19:20.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.558+0000 7fd56f400640 1 -- 172.21.15.28:0/3950213935 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd584053030 con 0x555bf477ebe0 2024-04-24T08:19:20.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.558+0000 7fd593973740 10 client.16467 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:20.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.558+0000 7fd593973740 10 client.16467 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:19:20.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.558+0000 7fd593973740 20 client.16467 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:20.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.559+0000 7fd593973740 1 --2- 172.21.15.28:0/3950213935 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x555bf47f1230 0x555bf47f3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.559+0000 7fd593973740 4 client.16467 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:19:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.559+0000 7fd593973740 1 -- 172.21.15.28:0/3950213935 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x555bf46dba80 con 0x555bf47f1230 2024-04-24T08:19:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.559+0000 7fd589e00640 1 --2- 172.21.15.28:0/3950213935 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x555bf47f1230 0x555bf47f3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.559+0000 7fd589e00640 1 --2- 172.21.15.28:0/3950213935 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x555bf47f1230 0x555bf47f3650 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7a5a11c4f522b7b1 server_cookie=3599d283b5312070 in_seq=0 out_seq=0 2024-04-24T08:19:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.559+0000 7fd56f400640 10 client.16467 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.560+0000 7fd56f400640 1 -- 172.21.15.28:0/3950213935 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x555bf46dba80 con 0x555bf47f1230 2024-04-24T08:19:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.560+0000 7fd56f400640 10 client.16467 handle_command_reply: tid=0 2024-04-24T08:19:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.560+0000 7fd593973740 1 client.16467 shutdown 2024-04-24T08:19:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.560+0000 7fd593973740 1 -- 172.21.15.28:0/3950213935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf477ebe0 msgr2=0x555bf477f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:20.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.560+0000 7fd593973740 1 --2- 172.21.15.28:0/3950213935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf477ebe0 0x555bf477f060 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fd584009ca0 tx=0x7fd58402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.560+0000 7fd593973740 1 -- 172.21.15.28:0/3950213935 shutdown_connections 2024-04-24T08:19:20.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.560+0000 7fd593973740 1 --2- 172.21.15.28:0/3950213935 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x555bf47f1230 0x555bf47f3650 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.560+0000 7fd593973740 1 --2- 172.21.15.28:0/3950213935 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x555bf477ebe0 0x555bf477f060 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.560+0000 7fd593973740 1 --2- 172.21.15.28:0/3950213935 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x555bf4779a20 0x555bf4782150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.560+0000 7fd593973740 1 --2- 172.21.15.28:0/3950213935 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x555bf476f350 0x555bf4781c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.560+0000 7fd593973740 1 -- 172.21.15.28:0/3950213935 >> 172.21.15.28:0/3950213935 conn(0x555bf476ad50 msgr2=0x555bf4777150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:20.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.560+0000 7fd56f400640 0 client.16467 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:20.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.560+0000 7fd593973740 1 -- 172.21.15.28:0/3950213935 shutdown_connections 2024-04-24T08:19:20.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.560+0000 7fd593973740 1 -- 172.21.15.28:0/3950213935 wait complete. 2024-04-24T08:19:20.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.560+0000 7fd593973740 20 client.16467 trim_cache size 0 max 16384 2024-04-24T08:19:20.570 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:19:20.570 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-24T08:19:20.570 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-24T08:19:20.571 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:19:20.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.561+0000 7fd58b200640 1 -- 172.21.15.28:0/4102569013 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd5700a3d20 msgr2=0x7fd5700a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:20.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.561+0000 7fd58b200640 1 --2- 172.21.15.28:0/4102569013 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd5700a3d20 0x7fd5700a61e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd584009430 tx=0x7fd584004a40 comp rx=0 tx=0).stop 2024-04-24T08:19:20.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.561+0000 7fd58b200640 1 -- 172.21.15.28:0/4102569013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd58c1003b0 msgr2=0x7fd58c1a0660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:20.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.561+0000 7fd58b200640 1 --2- 172.21.15.28:0/4102569013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd58c1003b0 0x7fd58c1a0660 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fd57c00e9f0 tx=0x7fd57c00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.561+0000 7fd58b200640 1 -- 172.21.15.28:0/4102569013 shutdown_connections 2024-04-24T08:19:20.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.561+0000 7fd58b200640 1 --2- 172.21.15.28:0/4102569013 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd5700a3d20 0x7fd5700a61e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.561+0000 7fd58b200640 1 --2- 172.21.15.28:0/4102569013 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd58c100cd0 0x7fd58c1a0ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.561+0000 7fd58b200640 1 --2- 172.21.15.28:0/4102569013 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd58c10e8f0 0x7fd58c19a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.561+0000 7fd58b200640 1 --2- 172.21.15.28:0/4102569013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd58c1003b0 0x7fd58c1a0660 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.561+0000 7fd58b200640 1 -- 172.21.15.28:0/4102569013 >> 172.21.15.28:0/4102569013 conn(0x7fd58c075d00 msgr2=0x7fd58c10ed90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:20.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.561+0000 7fd58b200640 1 -- 172.21.15.28:0/4102569013 shutdown_connections 2024-04-24T08:19:20.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.561+0000 7fd58b200640 1 -- 172.21.15.28:0/4102569013 wait complete. 2024-04-24T08:19:20.582 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:19:20.583 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:19:20.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.791+0000 7fd831400640 1 -- 172.21.15.28:0/1917865589 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd82c105c50 msgr2=0x7fd82c1060b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:20.793 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.791+0000 7fd831400640 1 --2- 172.21.15.28:0/1917865589 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd82c105c50 0x7fd82c1060b0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fd814009930 tx=0x7fd81402a440 comp rx=0 tx=0).stop 2024-04-24T08:19:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.791+0000 7fd831400640 1 -- 172.21.15.28:0/1917865589 shutdown_connections 2024-04-24T08:19:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.791+0000 7fd831400640 1 --2- 172.21.15.28:0/1917865589 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd82c1065f0 0x7fd82c110ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.791+0000 7fd831400640 1 --2- 172.21.15.28:0/1917865589 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd82c105c50 0x7fd82c1060b0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.791+0000 7fd831400640 1 --2- 172.21.15.28:0/1917865589 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd82c104370 0x7fd82c104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.791+0000 7fd831400640 1 -- 172.21.15.28:0/1917865589 >> 172.21.15.28:0/1917865589 conn(0x7fd82c0ffa90 msgr2=0x7fd82c101eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.791+0000 7fd831400640 1 -- 172.21.15.28:0/1917865589 shutdown_connections 2024-04-24T08:19:20.794 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.792+0000 7fd831400640 1 -- 172.21.15.28:0/1917865589 wait complete. 2024-04-24T08:19:20.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.793+0000 7fd831400640 1 Processor -- start 2024-04-24T08:19:20.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.793+0000 7fd831400640 1 -- start start 2024-04-24T08:19:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.793+0000 7fd831400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd82c104370 0x7fd82c1a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.793+0000 7fd831400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd82c105c50 0x7fd82c1a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.793+0000 7fd831400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd82c1065f0 0x7fd82c19a810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.793+0000 7fd831400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd82c114530 con 0x7fd82c105c50 2024-04-24T08:19:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.793+0000 7fd831400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd82c1143b0 con 0x7fd82c104370 2024-04-24T08:19:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.793+0000 7fd831400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd82c1146b0 con 0x7fd82c1065f0 2024-04-24T08:19:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.794+0000 7fd823400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd82c105c50 0x7fd82c1a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.794+0000 7fd82be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd82c104370 0x7fd82c1a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.794+0000 7fd830a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd82c1065f0 0x7fd82c19a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.794+0000 7fd82be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd82c104370 0x7fd82c1a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:52788/0 (socket says 172.21.15.28:52788) 2024-04-24T08:19:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.794+0000 7fd823400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd82c105c50 0x7fd82c1a0b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47664/0 (socket says 172.21.15.28:47664) 2024-04-24T08:19:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.794+0000 7fd82be00640 1 -- 172.21.15.28:0/1604296888 learned_addr learned my addr 172.21.15.28:0/1604296888 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.794+0000 7fd82be00640 1 -- 172.21.15.28:0/1604296888 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd82c1065f0 msgr2=0x7fd82c19a810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.794+0000 7fd82be00640 1 --2- 172.21.15.28:0/1604296888 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd82c1065f0 0x7fd82c19a810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.794+0000 7fd82be00640 1 -- 172.21.15.28:0/1604296888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd82c105c50 msgr2=0x7fd82c1a0b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:20.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.794+0000 7fd82be00640 1 --2- 172.21.15.28:0/1604296888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd82c105c50 0x7fd82c1a0b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:20.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.794+0000 7fd82be00640 1 -- 172.21.15.28:0/1604296888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd82c19afe0 con 0x7fd82c104370 2024-04-24T08:19:20.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.795+0000 7fd82be00640 1 --2- 172.21.15.28:0/1604296888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd82c104370 0x7fd82c1a0650 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00e9f0 tx=0x7fd81c00eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:20.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.795+0000 7fd829600640 1 -- 172.21.15.28:0/1604296888 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c00cd80 con 0x7fd82c104370 2024-04-24T08:19:20.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.795+0000 7fd831400640 1 -- 172.21.15.28:0/1604296888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd82c19b2d0 con 0x7fd82c104370 2024-04-24T08:19:20.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.795+0000 7fd829600640 1 -- 172.21.15.28:0/1604296888 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd81c010910 con 0x7fd82c104370 2024-04-24T08:19:20.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.795+0000 7fd829600640 1 -- 172.21.15.28:0/1604296888 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd81c0188f0 con 0x7fd82c104370 2024-04-24T08:19:20.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.795+0000 7fd831400640 1 -- 172.21.15.28:0/1604296888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd82c1a74a0 con 0x7fd82c104370 2024-04-24T08:19:20.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.796+0000 7fd831400640 1 -- 172.21.15.28:0/1604296888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd82c105760 con 0x7fd82c104370 2024-04-24T08:19:20.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.798+0000 7fd829600640 1 -- 172.21.15.28:0/1604296888 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fd81c010ac0 con 0x7fd82c104370 2024-04-24T08:19:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.800+0000 7fd829600640 1 --2- 172.21.15.28:0/1604296888 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd8100a3d20 0x7fd8100a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.800+0000 7fd829600640 1 -- 172.21.15.28:0/1604296888 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7fd81c0d0c50 con 0x7fd82c104370 2024-04-24T08:19:20.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.800+0000 7fd823400640 1 --2- 172.21.15.28:0/1604296888 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd8100a3d20 0x7fd8100a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:20.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.801+0000 7fd823400640 1 --2- 172.21.15.28:0/1604296888 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd8100a3d20 0x7fd8100a61e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd814006010 tx=0x7fd814000bc0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:20.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:20.804+0000 7fd829600640 1 -- 172.21.15.28:0/1604296888 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fd81c09cbf0 con 0x7fd82c104370 2024-04-24T08:19:21.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.052+0000 7fd831400640 1 -- 172.21.15.28:0/1604296888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd82c10c640 con 0x7fd82c104370 2024-04-24T08:19:21.055 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.054+0000 7fd829600640 1 -- 172.21.15.28:0/1604296888 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11871 (secure 0 0 0) 0x7fd81c0a0ef0 con 0x7fd82c104370 2024-04-24T08:19:21.056 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:19:21.056 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:14:38.663141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:19:21.057 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-24T08:19:21.058 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.057+0000 7fd831400640 1 -- 172.21.15.28:0/1604296888 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd8100a3d20 msgr2=0x7fd8100a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.057+0000 7fd831400640 1 --2- 172.21.15.28:0/1604296888 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd8100a3d20 0x7fd8100a61e0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fd814006010 tx=0x7fd814000bc0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.057+0000 7fd831400640 1 -- 172.21.15.28:0/1604296888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd82c104370 msgr2=0x7fd82c1a0650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.057+0000 7fd831400640 1 --2- 172.21.15.28:0/1604296888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd82c104370 0x7fd82c1a0650 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fd81c00e9f0 tx=0x7fd81c00eeb0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.057+0000 7fd831400640 1 -- 172.21.15.28:0/1604296888 shutdown_connections 2024-04-24T08:19:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.057+0000 7fd831400640 1 --2- 172.21.15.28:0/1604296888 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd8100a3d20 0x7fd8100a61e0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.057+0000 7fd831400640 1 --2- 172.21.15.28:0/1604296888 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd82c105c50 0x7fd82c1a0b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.057+0000 7fd831400640 1 --2- 172.21.15.28:0/1604296888 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd82c1065f0 0x7fd82c19a810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.057+0000 7fd831400640 1 --2- 172.21.15.28:0/1604296888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd82c104370 0x7fd82c1a0650 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.059 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.057+0000 7fd831400640 1 -- 172.21.15.28:0/1604296888 >> 172.21.15.28:0/1604296888 conn(0x7fd82c0ffa90 msgr2=0x7fd82c100070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.057+0000 7fd831400640 1 -- 172.21.15.28:0/1604296888 shutdown_connections 2024-04-24T08:19:21.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.057+0000 7fd831400640 1 -- 172.21.15.28:0/1604296888 wait complete. 2024-04-24T08:19:21.078 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:19:21.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.294+0000 7ff6d1000640 1 -- 172.21.15.28:0/283142124 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6cc1093f0 msgr2=0x7ff6cc110fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.294+0000 7ff6d1000640 1 --2- 172.21.15.28:0/283142124 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6cc1093f0 0x7ff6cc110fc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4009be0 tx=0x7ff6c402a6c0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.295+0000 7ff6d1000640 1 -- 172.21.15.28:0/283142124 shutdown_connections 2024-04-24T08:19:21.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.295+0000 7ff6d1000640 1 --2- 172.21.15.28:0/283142124 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6cc1093f0 0x7ff6cc110fc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.295+0000 7ff6d1000640 1 --2- 172.21.15.28:0/283142124 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6cc104cd0 0x7ff6cc108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.295+0000 7ff6d1000640 1 --2- 172.21.15.28:0/283142124 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc104320 0x7ff6cc104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.295+0000 7ff6d1000640 1 -- 172.21.15.28:0/283142124 >> 172.21.15.28:0/283142124 conn(0x7ff6cc0ffb40 msgr2=0x7ff6cc101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.295+0000 7ff6d1000640 1 -- 172.21.15.28:0/283142124 shutdown_connections 2024-04-24T08:19:21.298 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.295+0000 7ff6d1000640 1 -- 172.21.15.28:0/283142124 wait complete. 2024-04-24T08:19:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.296+0000 7ff6d1000640 1 Processor -- start 2024-04-24T08:19:21.300 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.296+0000 7ff6d1000640 1 -- start start 2024-04-24T08:19:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.297+0000 7ff6d1000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6cc104320 0x7ff6cc1a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.297+0000 7ff6d1000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6cc1093f0 0x7ff6cc1a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.297+0000 7ff6d1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc19a790 0x7ff6cc19ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.297+0000 7ff6d1000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc114490 con 0x7ff6cc19a790 2024-04-24T08:19:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.297+0000 7ff6d1000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc114310 con 0x7ff6cc1093f0 2024-04-24T08:19:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.297+0000 7ff6d1000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc114610 con 0x7ff6cc104320 2024-04-24T08:19:21.301 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.297+0000 7ff6cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc19a790 0x7ff6cc19ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.297+0000 7ff6cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc19a790 0x7ff6cc19ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47682/0 (socket says 172.21.15.28:47682) 2024-04-24T08:19:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.297+0000 7ff6cbe00640 1 -- 172.21.15.28:0/115919974 learned_addr learned my addr 172.21.15.28:0/115919974 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.297+0000 7ff6c3e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6cc1093f0 0x7ff6cc1a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.297+0000 7ff6cb400640 1 --2- 172.21.15.28:0/115919974 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6cc104320 0x7ff6cc1a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.298+0000 7ff6c3e00640 1 -- 172.21.15.28:0/115919974 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6cc104320 msgr2=0x7ff6cc1a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.298+0000 7ff6c3e00640 1 --2- 172.21.15.28:0/115919974 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6cc104320 0x7ff6cc1a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.298+0000 7ff6c3e00640 1 -- 172.21.15.28:0/115919974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc19a790 msgr2=0x7ff6cc19ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.298+0000 7ff6c3e00640 1 --2- 172.21.15.28:0/115919974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc19a790 0x7ff6cc19ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.298+0000 7ff6c3e00640 1 -- 172.21.15.28:0/115919974 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6cc19b320 con 0x7ff6cc1093f0 2024-04-24T08:19:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.298+0000 7ff6cbe00640 1 --2- 172.21.15.28:0/115919974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc19a790 0x7ff6cc19ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.298+0000 7ff6c3e00640 1 --2- 172.21.15.28:0/115919974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6cc1093f0 0x7ff6cc1a0b60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff6b400e9c0 tx=0x7ff6b400edc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.298+0000 7ff6c8c00640 1 -- 172.21.15.28:0/115919974 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b400ccc0 con 0x7ff6cc1093f0 2024-04-24T08:19:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.298+0000 7ff6c8c00640 1 -- 172.21.15.28:0/115919974 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6b4010910 con 0x7ff6cc1093f0 2024-04-24T08:19:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.298+0000 7ff6d1000640 1 -- 172.21.15.28:0/115919974 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6cc19b610 con 0x7ff6cc1093f0 2024-04-24T08:19:21.303 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.298+0000 7ff6c8c00640 1 -- 172.21.15.28:0/115919974 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b40188e0 con 0x7ff6cc1093f0 2024-04-24T08:19:21.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.299+0000 7ff6d1000640 1 -- 172.21.15.28:0/115919974 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6cc1a77f0 con 0x7ff6cc1093f0 2024-04-24T08:19:21.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.303+0000 7ff6d53e4740 1 Processor -- start 2024-04-24T08:19:21.304 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.303+0000 7ff6d53e4740 1 -- start start 2024-04-24T08:19:21.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.303+0000 7ff6d53e4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573203890 0x559573203c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.303+0000 7ff6d53e4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573204240 0x559573216e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.303+0000 7ff6d53e4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573217340 0x559573219730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.303+0000 7ff6d53e4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5595731691d0 con 0x559573217340 2024-04-24T08:19:21.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.304+0000 7ff6d53e4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55957310f450 con 0x559573203890 2024-04-24T08:19:21.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.304+0000 7ff6d53e4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559573165190 con 0x559573204240 2024-04-24T08:19:21.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.304+0000 7ff6c8c00640 1 -- 172.21.15.28:0/115919974 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff6b4010ac0 con 0x7ff6cc1093f0 2024-04-24T08:19:21.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.304+0000 7ff6c3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573217340 0x559573219730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.304+0000 7ff6cb400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573204240 0x559573216e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.304+0000 7ff6c3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573217340 0x559573219730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47684/0 (socket says 172.21.15.28:47684) 2024-04-24T08:19:21.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.304+0000 7ff6cb400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573204240 0x559573216e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:43942/0 (socket says 172.21.15.28:43942) 2024-04-24T08:19:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.304+0000 7ff6c3e00640 1 -- 172.21.15.28:0/1226200193 learned_addr learned my addr 172.21.15.28:0/1226200193 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.304+0000 7ff6cbe00640 1 --2- 172.21.15.28:0/1226200193 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573203890 0x559573203c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.304+0000 7ff6cb400640 1 -- 172.21.15.28:0/1226200193 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573203890 msgr2=0x559573203c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.304+0000 7ff6cb400640 1 --2- 172.21.15.28:0/1226200193 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573203890 0x559573203c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.304+0000 7ff6cb400640 1 -- 172.21.15.28:0/1226200193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573217340 msgr2=0x559573219730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.304+0000 7ff6cb400640 1 --2- 172.21.15.28:0/1226200193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573217340 0x559573219730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.304+0000 7ff6cb400640 1 -- 172.21.15.28:0/1226200193 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595731cc310 con 0x559573204240 2024-04-24T08:19:21.308 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.305+0000 7ff6cb400640 1 --2- 172.21.15.28:0/1226200193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573204240 0x559573216e00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc009c30 tx=0x7ff6bc011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80ffd26d32a2ea5f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.305+0000 7ff6c0c00640 1 -- 172.21.15.28:0/1226200193 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc004030 con 0x559573204240 2024-04-24T08:19:21.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.305+0000 7ff6c0c00640 1 -- 172.21.15.28:0/1226200193 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6bc0041d0 con 0x559573204240 2024-04-24T08:19:21.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.305+0000 7ff6c0c00640 1 -- 172.21.15.28:0/1226200193 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6bc019760 con 0x559573204240 2024-04-24T08:19:21.314 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.306+0000 7ff6c8c00640 1 --2- 172.21.15.28:0/115919974 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff6b00a3d20 0x7ff6b00a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.306+0000 7ff6c8c00640 1 -- 172.21.15.28:0/115919974 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7ff6b40d0ab0 con 0x7ff6cc1093f0 2024-04-24T08:19:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.306+0000 7ff6cbe00640 1 --2- 172.21.15.28:0/115919974 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff6b00a3d20 0x7ff6b00a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.306+0000 7ff6d53e4740 1 -- 172.21.15.28:0/1226200193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573204240 msgr2=0x559573216e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.306+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/1226200193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573204240 0x559573216e00 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7ff6bc009c30 tx=0x7ff6bc011810 comp rx=0 tx=0).stop 2024-04-24T08:19:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.306+0000 7ff6d53e4740 1 -- 172.21.15.28:0/1226200193 shutdown_connections 2024-04-24T08:19:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.306+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/1226200193 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573217340 0x559573219730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.306+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/1226200193 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573204240 0x559573216e00 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.306+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/1226200193 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573203890 0x559573203c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.306+0000 7ff6d53e4740 1 -- 172.21.15.28:0/1226200193 >> 172.21.15.28:0/1226200193 conn(0x5595731ffe60 msgr2=0x559573200240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.307+0000 7ff6d53e4740 1 -- 172.21.15.28:0/1226200193 shutdown_connections 2024-04-24T08:19:21.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.307+0000 7ff6d53e4740 1 -- 172.21.15.28:0/1226200193 wait complete. 2024-04-24T08:19:21.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.307+0000 7ff6cbe00640 1 --2- 172.21.15.28:0/115919974 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff6b00a3d20 0x7ff6b00a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4004af0 tx=0x7ff6c4009840 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.307+0000 7ff6d53e4740 1 -- start start 2024-04-24T08:19:21.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.307+0000 7ff6d53e4740 1 Processor -- start 2024-04-24T08:19:21.316 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.308+0000 7ff6d53e4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573203890 0x559573216d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.308+0000 7ff6d53e4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573204240 0x559573214030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.308+0000 7ff6d53e4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573217340 0x559573214640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.308+0000 7ff6d53e4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55957321a1a0 con 0x559573204240 2024-04-24T08:19:21.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.308+0000 7ff6d53e4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559573122810 con 0x559573217340 2024-04-24T08:19:21.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.308+0000 7ff6d53e4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55957321a320 con 0x559573203890 2024-04-24T08:19:21.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.308+0000 7ff6d53e4740 10 client.0 _ll_register_callbacks cb 0x5595730f0140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:21.317 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.308+0000 7ff6c3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573204240 0x559573214030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.308+0000 7ff6c3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573204240 0x559573214030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47694/0 (socket says 172.21.15.28:47694) 2024-04-24T08:19:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.308+0000 7ff6c3e00640 1 -- 172.21.15.28:0/2998244275 learned_addr learned my addr 172.21.15.28:0/2998244275 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.308+0000 7ff6cb400640 1 --2- 172.21.15.28:0/2998244275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573203890 0x559573216d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.308+0000 7ff6cbe00640 1 --2- 172.21.15.28:0/2998244275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573217340 0x559573214640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.308+0000 7ff6c3e00640 1 -- 172.21.15.28:0/2998244275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573203890 msgr2=0x559573216d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.308+0000 7ff6c3e00640 1 --2- 172.21.15.28:0/2998244275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573203890 0x559573216d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.308+0000 7ff6c3e00640 1 -- 172.21.15.28:0/2998244275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573217340 msgr2=0x559573214640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.308+0000 7ff6c3e00640 1 --2- 172.21.15.28:0/2998244275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573217340 0x559573214640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.308+0000 7ff6c3e00640 1 -- 172.21.15.28:0/2998244275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5595731333f0 con 0x559573204240 2024-04-24T08:19:21.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.309+0000 7ff6c3e00640 1 --2- 172.21.15.28:0/2998244275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573204240 0x559573214030 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4099370 tx=0x7ff6b40ce0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=529ff76ebddcae70 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.309+0000 7ff6b9600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:19:21.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.309+0000 7ff6b9600640 1 -- 172.21.15.28:0/2998244275 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b409c4e0 con 0x559573204240 2024-04-24T08:19:21.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.309+0000 7ff6b9600640 1 -- 172.21.15.28:0/2998244275 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7ff6b40bbd60 con 0x559573204240 2024-04-24T08:19:21.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.309+0000 7ff6d53e4740 1 -- 172.21.15.28:0/2998244275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6bc0041d0 con 0x559573204240 2024-04-24T08:19:21.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.309+0000 7ff6b9600640 1 -- 172.21.15.28:0/2998244275 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff6b40ce5e0 con 0x559573204240 2024-04-24T08:19:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.309+0000 7ff6d53e4740 10 client.16485 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.309+0000 7ff6d53e4740 1 -- 172.21.15.28:0/2998244275 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559572e7b3f0 con 0x559573204240 2024-04-24T08:19:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.310+0000 7ff6b9600640 1 -- 172.21.15.28:0/2998244275 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6bc0041d0 con 0x559573204240 2024-04-24T08:19:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.310+0000 7ff6b9600640 1 -- 172.21.15.28:0/2998244275 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff6b4087070 con 0x559573204240 2024-04-24T08:19:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.310+0000 7ff6d53e4740 10 client.16485 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.310+0000 7ff6d53e4740 10 client.16485 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:19:21.320 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.310+0000 7ff6d53e4740 20 client.16485 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:21.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.310+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/2998244275 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x559573227410 0x559573229800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.310+0000 7ff6d53e4740 4 client.16485 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:19:21.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.310+0000 7ff6d53e4740 1 -- 172.21.15.28:0/2998244275 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559573176bf0 con 0x559573227410 2024-04-24T08:19:21.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.310+0000 7ff6cb400640 1 --2- 172.21.15.28:0/2998244275 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x559573227410 0x559573229800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.311+0000 7ff6cb400640 1 --2- 172.21.15.28:0/2998244275 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x559573227410 0x559573229800 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=79f213641fe7cde2 server_cookie=3b366d80f7c56a53 in_seq=0 out_seq=0 2024-04-24T08:19:21.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.311+0000 7ff6b9600640 10 client.16485 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:19:21.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.312+0000 7ff6b9600640 1 -- 172.21.15.28:0/2998244275 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559573176bf0 con 0x559573227410 2024-04-24T08:19:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.312+0000 7ff6b9600640 10 client.16485 handle_command_reply: tid=0 2024-04-24T08:19:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.312+0000 7ff6d53e4740 1 client.16485 shutdown 2024-04-24T08:19:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.312+0000 7ff6d53e4740 1 -- 172.21.15.28:0/2998244275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573204240 msgr2=0x559573214030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.312+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/2998244275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573204240 0x559573214030 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4099370 tx=0x7ff6b40ce0f0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.312+0000 7ff6d53e4740 1 -- 172.21.15.28:0/2998244275 shutdown_connections 2024-04-24T08:19:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.312+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/2998244275 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x559573227410 0x559573229800 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.312+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/2998244275 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573204240 0x559573214030 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.312+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/2998244275 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573217340 0x559573214640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.312+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/2998244275 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573203890 0x559573216d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.312+0000 7ff6b9600640 0 client.16485 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:19:21.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.312+0000 7ff6d53e4740 1 -- 172.21.15.28:0/2998244275 >> 172.21.15.28:0/2998244275 conn(0x5595731ffe60 msgr2=0x559573205c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.313+0000 7ff6d53e4740 1 -- 172.21.15.28:0/2998244275 shutdown_connections 2024-04-24T08:19:21.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.313+0000 7ff6d53e4740 1 -- 172.21.15.28:0/2998244275 wait complete. 2024-04-24T08:19:21.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.313+0000 7ff6d53e4740 20 client.16485 trim_cache size 0 max 16384 2024-04-24T08:19:21.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.328+0000 7ff6d53e4740 1 Processor -- start 2024-04-24T08:19:21.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.328+0000 7ff6d53e4740 1 -- start start 2024-04-24T08:19:21.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.328+0000 7ff6d53e4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573203890 0x5595732230a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.328+0000 7ff6d53e4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573204240 0x559573216870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.328+0000 7ff6d53e4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573217340 0x55957322f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.328+0000 7ff6d53e4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1ae4c0 con 0x559573203890 2024-04-24T08:19:21.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.328+0000 7ff6d53e4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1abd90 con 0x559573204240 2024-04-24T08:19:21.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.328+0000 7ff6d53e4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1a9c20 con 0x559573217340 2024-04-24T08:19:21.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.328+0000 7ff6cbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573217340 0x55957322f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.328+0000 7ff6cbe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573217340 0x55957322f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:43948/0 (socket says 172.21.15.28:43948) 2024-04-24T08:19:21.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.328+0000 7ff6cbe00640 1 -- 172.21.15.28:0/207680942 learned_addr learned my addr 172.21.15.28:0/207680942 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:21.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.328+0000 7ff6cb400640 1 --2- 172.21.15.28:0/207680942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573203890 0x5595732230a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.329+0000 7ff6c3e00640 1 --2- 172.21.15.28:0/207680942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573204240 0x559573216870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.329+0000 7ff6cbe00640 1 -- 172.21.15.28:0/207680942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573204240 msgr2=0x559573216870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.329+0000 7ff6cbe00640 1 --2- 172.21.15.28:0/207680942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573204240 0x559573216870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.329+0000 7ff6cbe00640 1 -- 172.21.15.28:0/207680942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573203890 msgr2=0x5595732230a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.329+0000 7ff6cbe00640 1 --2- 172.21.15.28:0/207680942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573203890 0x5595732230a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.329+0000 7ff6cbe00640 1 -- 172.21.15.28:0/207680942 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559573203fe0 con 0x559573217340 2024-04-24T08:19:21.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.329+0000 7ff6cb400640 1 --2- 172.21.15.28:0/207680942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573203890 0x5595732230a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:21.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.329+0000 7ff6cbe00640 1 --2- 172.21.15.28:0/207680942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573217340 0x55957322f0b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4030730 tx=0x7ff6c4040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0d9fe856c4549cf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.329+0000 7ff6bbe00640 1 -- 172.21.15.28:0/207680942 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4036040 con 0x559573217340 2024-04-24T08:19:21.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.329+0000 7ff6bbe00640 1 -- 172.21.15.28:0/207680942 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6c4004690 con 0x559573217340 2024-04-24T08:19:21.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.329+0000 7ff6bbe00640 1 -- 172.21.15.28:0/207680942 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6c4031ce0 con 0x559573217340 2024-04-24T08:19:21.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.330+0000 7ff6d53e4740 1 -- 172.21.15.28:0/207680942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573217340 msgr2=0x55957322f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.330+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/207680942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573217340 0x55957322f0b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4030730 tx=0x7ff6c4040000 comp rx=0 tx=0).stop 2024-04-24T08:19:21.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.330+0000 7ff6d53e4740 1 -- 172.21.15.28:0/207680942 shutdown_connections 2024-04-24T08:19:21.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.330+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/207680942 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573217340 0x55957322f0b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.330+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/207680942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573204240 0x559573216870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.330+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/207680942 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573203890 0x5595732230a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.330+0000 7ff6d53e4740 1 -- 172.21.15.28:0/207680942 >> 172.21.15.28:0/207680942 conn(0x5595731ffe60 msgr2=0x559573200240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.330+0000 7ff6d53e4740 1 -- 172.21.15.28:0/207680942 shutdown_connections 2024-04-24T08:19:21.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.330+0000 7ff6d53e4740 1 -- 172.21.15.28:0/207680942 wait complete. 2024-04-24T08:19:21.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.331+0000 7ff6d53e4740 1 -- start start 2024-04-24T08:19:21.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.331+0000 7ff6d53e4740 1 Processor -- start 2024-04-24T08:19:21.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.331+0000 7ff6d53e4740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573203890 0x55957321f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.331+0000 7ff6d53e4740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573204240 0x55957321fb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.331+0000 7ff6d53e4740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573217340 0x55957321a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.331+0000 7ff6d53e4740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1aa270 con 0x559573217340 2024-04-24T08:19:21.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.331+0000 7ff6d53e4740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1ab6c0 con 0x559573203890 2024-04-24T08:19:21.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.331+0000 7ff6d53e4740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff6cc1aa920 con 0x559573204240 2024-04-24T08:19:21.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.331+0000 7ff6d53e4740 10 client.0 _ll_register_callbacks cb 0x7ff6b400ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:21.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.331+0000 7ff6cb400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573203890 0x55957321f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.332+0000 7ff6cb400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573203890 0x55957321f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:52840/0 (socket says 172.21.15.28:52840) 2024-04-24T08:19:21.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.332+0000 7ff6cb400640 1 -- 172.21.15.28:0/3579774651 learned_addr learned my addr 172.21.15.28:0/3579774651 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:21.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.332+0000 7ff6cbe00640 1 --2- 172.21.15.28:0/3579774651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573217340 0x55957321a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.332+0000 7ff6c3e00640 1 --2- 172.21.15.28:0/3579774651 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573204240 0x55957321fb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.332+0000 7ff6c3e00640 1 -- 172.21.15.28:0/3579774651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573203890 msgr2=0x55957321f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.332+0000 7ff6c3e00640 1 --2- 172.21.15.28:0/3579774651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573203890 0x55957321f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.332+0000 7ff6c3e00640 1 -- 172.21.15.28:0/3579774651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573217340 msgr2=0x55957321a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.332+0000 7ff6c3e00640 1 --2- 172.21.15.28:0/3579774651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573217340 0x55957321a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.332+0000 7ff6c3e00640 1 -- 172.21.15.28:0/3579774651 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559572f457d0 con 0x559573204240 2024-04-24T08:19:21.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.332+0000 7ff6cb400640 1 --2- 172.21.15.28:0/3579774651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573203890 0x55957321f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:21.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.332+0000 7ff6c3e00640 1 --2- 172.21.15.28:0/3579774651 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573204240 0x55957321fb00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4099320 tx=0x7ff6b409c8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e096cdfb1702b410 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.332+0000 7ff6b9600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:19:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.333+0000 7ff6b9600640 1 -- 172.21.15.28:0/3579774651 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6b409c300 con 0x559573204240 2024-04-24T08:19:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.333+0000 7ff6b9600640 1 -- 172.21.15.28:0/3579774651 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7ff6b4010630 con 0x559573204240 2024-04-24T08:19:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.333+0000 7ff6d53e4740 1 -- 172.21.15.28:0/3579774651 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6c4004690 con 0x559573204240 2024-04-24T08:19:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.333+0000 7ff6b9600640 1 -- 172.21.15.28:0/3579774651 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff6b409c4e0 con 0x559573204240 2024-04-24T08:19:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.333+0000 7ff6d53e4740 10 client.26791 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.333+0000 7ff6d53e4740 1 -- 172.21.15.28:0/3579774651 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55957328e200 con 0x559573204240 2024-04-24T08:19:21.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.333+0000 7ff6b9600640 1 -- 172.21.15.28:0/3579774651 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6c4004690 con 0x559573204240 2024-04-24T08:19:21.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.333+0000 7ff6b9600640 1 -- 172.21.15.28:0/3579774651 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff6b409b860 con 0x559573204240 2024-04-24T08:19:21.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.333+0000 7ff6d53e4740 10 client.26791 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:21.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.333+0000 7ff6d53e4740 10 client.26791 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:19:21.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.333+0000 7ff6d53e4740 20 client.26791 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:21.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.333+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/3579774651 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55957328e920 0x559573290d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.333+0000 7ff6d53e4740 4 client.26791 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:19:21.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.333+0000 7ff6d53e4740 1 -- 172.21.15.28:0/3579774651 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559573170a60 con 0x55957328e920 2024-04-24T08:19:21.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.334+0000 7ff6cb400640 1 --2- 172.21.15.28:0/3579774651 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55957328e920 0x559573290d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.334+0000 7ff6cb400640 1 --2- 172.21.15.28:0/3579774651 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55957328e920 0x559573290d40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3cc25a7d047ce0c1 server_cookie=3761f51822bd1eb1 in_seq=0 out_seq=0 2024-04-24T08:19:21.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.334+0000 7ff6b9600640 10 client.26791 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:19:21.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.335+0000 7ff6b9600640 1 -- 172.21.15.28:0/3579774651 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559573170a60 con 0x55957328e920 2024-04-24T08:19:21.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.335+0000 7ff6b9600640 10 client.26791 handle_command_reply: tid=0 2024-04-24T08:19:21.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.335+0000 7ff6d53e4740 1 client.26791 shutdown 2024-04-24T08:19:21.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.335+0000 7ff6d53e4740 1 -- 172.21.15.28:0/3579774651 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573204240 msgr2=0x55957321fb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.335+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/3579774651 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573204240 0x55957321fb00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7ff6b4099320 tx=0x7ff6b409c8a0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.335+0000 7ff6d53e4740 1 -- 172.21.15.28:0/3579774651 shutdown_connections 2024-04-24T08:19:21.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.335+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/3579774651 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55957328e920 0x559573290d40 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.335+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/3579774651 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559573217340 0x55957321a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.335+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/3579774651 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559573204240 0x55957321fb00 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.335+0000 7ff6d53e4740 1 --2- 172.21.15.28:0/3579774651 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559573203890 0x55957321f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.335+0000 7ff6d53e4740 1 -- 172.21.15.28:0/3579774651 >> 172.21.15.28:0/3579774651 conn(0x5595731ffe60 msgr2=0x5595732053d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.335+0000 7ff6b9600640 0 client.26791 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:19:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.335+0000 7ff6d53e4740 1 -- 172.21.15.28:0/3579774651 shutdown_connections 2024-04-24T08:19:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.335+0000 7ff6d53e4740 1 -- 172.21.15.28:0/3579774651 wait complete. 2024-04-24T08:19:21.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.335+0000 7ff6d53e4740 20 client.26791 trim_cache size 0 max 16384 2024-04-24T08:19:21.347 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:19:21.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.336+0000 7ff6d1000640 1 -- 172.21.15.28:0/115919974 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff6b00a3d20 msgr2=0x7ff6b00a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.336+0000 7ff6d1000640 1 --2- 172.21.15.28:0/115919974 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff6b00a3d20 0x7ff6b00a61e0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff6c4004af0 tx=0x7ff6c4009840 comp rx=0 tx=0).stop 2024-04-24T08:19:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.336+0000 7ff6d1000640 1 -- 172.21.15.28:0/115919974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6cc1093f0 msgr2=0x7ff6cc1a0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.336+0000 7ff6d1000640 1 --2- 172.21.15.28:0/115919974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6cc1093f0 0x7ff6cc1a0b60 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7ff6b400e9c0 tx=0x7ff6b400edc0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.336+0000 7ff6d1000640 1 -- 172.21.15.28:0/115919974 shutdown_connections 2024-04-24T08:19:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.336+0000 7ff6d1000640 1 --2- 172.21.15.28:0/115919974 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff6b00a3d20 0x7ff6b00a61e0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.336+0000 7ff6d1000640 1 --2- 172.21.15.28:0/115919974 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6cc19a790 0x7ff6cc19ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.336+0000 7ff6d1000640 1 --2- 172.21.15.28:0/115919974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6cc1093f0 0x7ff6cc1a0b60 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.336+0000 7ff6d1000640 1 --2- 172.21.15.28:0/115919974 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6cc104320 0x7ff6cc1a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.336+0000 7ff6d1000640 1 -- 172.21.15.28:0/115919974 >> 172.21.15.28:0/115919974 conn(0x7ff6cc0ffb40 msgr2=0x7ff6cc100660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.336+0000 7ff6d1000640 1 -- 172.21.15.28:0/115919974 shutdown_connections 2024-04-24T08:19:21.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.336+0000 7ff6d1000640 1 -- 172.21.15.28:0/115919974 wait complete. 2024-04-24T08:19:21.355 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:19:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.561+0000 7ff4b3c00640 1 -- 172.21.15.28:0/1100074573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff4ac102290 msgr2=0x7ff4ac102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.561+0000 7ff4b3c00640 1 --2- 172.21.15.28:0/1100074573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff4ac102290 0x7ff4ac102670 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7ff49c009860 tx=0x7ff49c02a340 comp rx=0 tx=0).stop 2024-04-24T08:19:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.561+0000 7ff4b3c00640 1 -- 172.21.15.28:0/1100074573 shutdown_connections 2024-04-24T08:19:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.561+0000 7ff4b3c00640 1 --2- 172.21.15.28:0/1100074573 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4ac102c40 0x7ff4ac106c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.563 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.561+0000 7ff4b3c00640 1 --2- 172.21.15.28:0/1100074573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4ac107360 0x7ff4ac10ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.561+0000 7ff4b3c00640 1 --2- 172.21.15.28:0/1100074573 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff4ac102290 0x7ff4ac102670 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.561+0000 7ff4b3c00640 1 -- 172.21.15.28:0/1100074573 >> 172.21.15.28:0/1100074573 conn(0x7ff4ac0fdb10 msgr2=0x7ff4ac0fff30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.561+0000 7ff4b3c00640 1 -- 172.21.15.28:0/1100074573 shutdown_connections 2024-04-24T08:19:21.564 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.561+0000 7ff4b3c00640 1 -- 172.21.15.28:0/1100074573 wait complete. 2024-04-24T08:19:21.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.563+0000 7ff4b3c00640 1 Processor -- start 2024-04-24T08:19:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.563+0000 7ff4b3c00640 1 -- start start 2024-04-24T08:19:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.563+0000 7ff4b3c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4ac102290 0x7ff4ac1a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.563+0000 7ff4b3c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff4ac107360 0x7ff4ac1a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.563+0000 7ff4b3c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4ac19a9d0 0x7ff4ac19ae80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.563+0000 7ff4b3c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ac1123f0 con 0x7ff4ac102290 2024-04-24T08:19:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.563+0000 7ff4b3c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ac112270 con 0x7ff4ac19a9d0 2024-04-24T08:19:21.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.563+0000 7ff4b3c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ac112570 con 0x7ff4ac107360 2024-04-24T08:19:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.564+0000 7ff4b1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4ac102290 0x7ff4ac1a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.564+0000 7ff4b1c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4ac19a9d0 0x7ff4ac19ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.564+0000 7ff4abe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff4ac107360 0x7ff4ac1a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.564+0000 7ff4b1c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4ac19a9d0 0x7ff4ac19ae80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:52860/0 (socket says 172.21.15.28:52860) 2024-04-24T08:19:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.564+0000 7ff4b1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4ac102290 0x7ff4ac1a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47722/0 (socket says 172.21.15.28:47722) 2024-04-24T08:19:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.564+0000 7ff4b1c00640 1 -- 172.21.15.28:0/2446085254 learned_addr learned my addr 172.21.15.28:0/2446085254 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.564+0000 7ff4abe00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff4ac107360 0x7ff4ac1a0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:43978/0 (socket says 172.21.15.28:43978) 2024-04-24T08:19:21.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.564+0000 7ff4b1c00640 1 -- 172.21.15.28:0/2446085254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff4ac107360 msgr2=0x7ff4ac1a0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.564+0000 7ff4b1c00640 1 --2- 172.21.15.28:0/2446085254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff4ac107360 0x7ff4ac1a0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.564+0000 7ff4b1c00640 1 -- 172.21.15.28:0/2446085254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4ac102290 msgr2=0x7ff4ac1a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.564+0000 7ff4b1c00640 1 --2- 172.21.15.28:0/2446085254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4ac102290 0x7ff4ac1a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.564+0000 7ff4b1c00640 1 -- 172.21.15.28:0/2446085254 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4ac19b4f0 con 0x7ff4ac19a9d0 2024-04-24T08:19:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.565+0000 7ff4b1c00640 1 --2- 172.21.15.28:0/2446085254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4ac19a9d0 0x7ff4ac19ae80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400efd0 tx=0x7ff4a400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.565+0000 7ff4a9600640 1 -- 172.21.15.28:0/2446085254 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4003b70 con 0x7ff4ac19a9d0 2024-04-24T08:19:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.565+0000 7ff4a9600640 1 -- 172.21.15.28:0/2446085254 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4a4003d10 con 0x7ff4ac19a9d0 2024-04-24T08:19:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.565+0000 7ff4a9600640 1 -- 172.21.15.28:0/2446085254 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a40109f0 con 0x7ff4ac19a9d0 2024-04-24T08:19:21.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.565+0000 7ff4b3c00640 1 -- 172.21.15.28:0/2446085254 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4ac19b7e0 con 0x7ff4ac19a9d0 2024-04-24T08:19:21.570 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.565+0000 7ff4b3c00640 1 -- 172.21.15.28:0/2446085254 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4ac1a79e0 con 0x7ff4ac19a9d0 2024-04-24T08:19:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.568+0000 7ff4b69b3740 1 Processor -- start 2024-04-24T08:19:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.569+0000 7ff4a9600640 1 -- 172.21.15.28:0/2446085254 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff4a4004020 con 0x7ff4ac19a9d0 2024-04-24T08:19:21.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.569+0000 7ff4b69b3740 1 -- start start 2024-04-24T08:19:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.569+0000 7ff4b69b3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6ec950 0x562a2a6ecd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.569+0000 7ff4b69b3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ed270 0x562a2a6f7400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.569+0000 7ff4b69b3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6f7940 0x562a2a6f9d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.569+0000 7ff4b69b3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562a2a6521d0 con 0x562a2a6ec950 2024-04-24T08:19:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.569+0000 7ff4b69b3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562a2a5f8450 con 0x562a2a6f7940 2024-04-24T08:19:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.569+0000 7ff4b69b3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562a2a64e190 con 0x562a2a6ed270 2024-04-24T08:19:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.569+0000 7ff4abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6ec950 0x562a2a6ecd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.570+0000 7ff4abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6ec950 0x562a2a6ecd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47730/0 (socket says 172.21.15.28:47730) 2024-04-24T08:19:21.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.570+0000 7ff4abe00640 1 -- 172.21.15.28:0/1572604574 learned_addr learned my addr 172.21.15.28:0/1572604574 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:21.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.570+0000 7ff4b1200640 1 --2- 172.21.15.28:0/1572604574 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ed270 0x562a2a6f7400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.570+0000 7ff4abe00640 1 --2- 172.21.15.28:0/1572604574 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6f7940 0x562a2a6f9d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.570+0000 7ff4abe00640 1 -- 172.21.15.28:0/1572604574 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ed270 msgr2=0x562a2a6f7400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.570+0000 7ff4abe00640 1 --2- 172.21.15.28:0/1572604574 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ed270 0x562a2a6f7400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.570+0000 7ff4abe00640 1 -- 172.21.15.28:0/1572604574 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6f7940 msgr2=0x562a2a6f9d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.570+0000 7ff4abe00640 1 --2- 172.21.15.28:0/1572604574 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6f7940 0x562a2a6f9d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.570+0000 7ff4abe00640 1 -- 172.21.15.28:0/1572604574 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a2a6b5310 con 0x562a2a6ec950 2024-04-24T08:19:21.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.570+0000 7ff4a9600640 1 --2- 172.21.15.28:0/2446085254 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff48c0a3da0 0x7ff48c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.570+0000 7ff4abe00640 1 --2- 172.21.15.28:0/1572604574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6ec950 0x562a2a6ecd30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff494013780 tx=0x7ff494009640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c4d4d3bbcde6607 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.571+0000 7ff4a9600640 1 -- 172.21.15.28:0/2446085254 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7ff4a40d0c20 con 0x7ff4ac19a9d0 2024-04-24T08:19:21.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.571+0000 7ff4b1200640 1 --2- 172.21.15.28:0/2446085254 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff48c0a3da0 0x7ff48c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.571+0000 7ff4a2000640 1 -- 172.21.15.28:0/1572604574 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff494013d00 con 0x562a2a6ec950 2024-04-24T08:19:21.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.571+0000 7ff4a2000640 1 -- 172.21.15.28:0/1572604574 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff494005020 con 0x562a2a6ec950 2024-04-24T08:19:21.578 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.571+0000 7ff4a2000640 1 -- 172.21.15.28:0/1572604574 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff494004aa0 con 0x562a2a6ec950 2024-04-24T08:19:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.571+0000 7ff4b69b3740 1 -- 172.21.15.28:0/1572604574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6ec950 msgr2=0x562a2a6ecd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.571+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/1572604574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6ec950 0x562a2a6ecd30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7ff494013780 tx=0x7ff494009640 comp rx=0 tx=0).stop 2024-04-24T08:19:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.571+0000 7ff4b1200640 1 --2- 172.21.15.28:0/2446085254 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff48c0a3da0 0x7ff48c0a6260 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff49c009830 tx=0x7ff49c0056d0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.572+0000 7ff4b69b3740 1 -- 172.21.15.28:0/1572604574 shutdown_connections 2024-04-24T08:19:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.572+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/1572604574 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6f7940 0x562a2a6f9d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.572+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/1572604574 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ed270 0x562a2a6f7400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.572+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/1572604574 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6ec950 0x562a2a6ecd30 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.579 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.572+0000 7ff4b69b3740 1 -- 172.21.15.28:0/1572604574 >> 172.21.15.28:0/1572604574 conn(0x562a2a6e8c70 msgr2=0x562a2a6e9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.572+0000 7ff4b69b3740 1 -- 172.21.15.28:0/1572604574 shutdown_connections 2024-04-24T08:19:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.572+0000 7ff4b69b3740 1 -- 172.21.15.28:0/1572604574 wait complete. 2024-04-24T08:19:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.572+0000 7ff4b69b3740 1 -- start start 2024-04-24T08:19:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.572+0000 7ff4b69b3740 1 Processor -- start 2024-04-24T08:19:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.573+0000 7ff4b69b3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 0x562a2a6f7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.573+0000 7ff4b69b3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6ed270 0x562a2a6f4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.573+0000 7ff4b69b3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6f7940 0x562a2a6f4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.573+0000 7ff4b69b3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x562a2a6fa7c0 con 0x562a2a6f7940 2024-04-24T08:19:21.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.573+0000 7ff4b69b3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x562a2a60b810 con 0x562a2a6ed270 2024-04-24T08:19:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.573+0000 7ff4b69b3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x562a2a6fa940 con 0x562a2a6ec950 2024-04-24T08:19:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.573+0000 7ff4b69b3740 10 client.0 _ll_register_callbacks cb 0x562a2a5d9140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.573+0000 7ff4b1200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 0x562a2a6f7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.573+0000 7ff4b1200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 0x562a2a6f7270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:43994/0 (socket says 172.21.15.28:43994) 2024-04-24T08:19:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.573+0000 7ff4b1200640 1 -- 172.21.15.28:0/1049202831 learned_addr learned my addr 172.21.15.28:0/1049202831 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.573+0000 7ff4b1c00640 1 --2- 172.21.15.28:0/1049202831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6f7940 0x562a2a6f4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.573+0000 7ff4abe00640 1 --2- 172.21.15.28:0/1049202831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6ed270 0x562a2a6f4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.581 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.574+0000 7ff4b1200640 1 -- 172.21.15.28:0/1049202831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6ed270 msgr2=0x562a2a6f4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.574+0000 7ff4b1200640 1 --2- 172.21.15.28:0/1049202831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6ed270 0x562a2a6f4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.574+0000 7ff4b1200640 1 -- 172.21.15.28:0/1049202831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6f7940 msgr2=0x562a2a6f4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.574+0000 7ff4b1200640 1 --2- 172.21.15.28:0/1049202831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6f7940 0x562a2a6f4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.574+0000 7ff4b1200640 1 -- 172.21.15.28:0/1049202831 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a2a61c3f0 con 0x562a2a6ec950 2024-04-24T08:19:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.574+0000 7ff4b1c00640 1 --2- 172.21.15.28:0/1049202831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6f7940 0x562a2a6f4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.574+0000 7ff4b1200640 1 --2- 172.21.15.28:0/1049202831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 0x562a2a6f7270 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff49c034000 tx=0x7ff49c02a340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6f0b933a85b1388f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.574+0000 7ff49aa00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:19:21.582 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.574+0000 7ff49aa00640 1 -- 172.21.15.28:0/1049202831 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff49c037040 con 0x562a2a6ec950 2024-04-24T08:19:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.574+0000 7ff49aa00640 1 -- 172.21.15.28:0/1049202831 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7ff49c003a00 con 0x562a2a6ec950 2024-04-24T08:19:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.574+0000 7ff4b69b3740 1 -- 172.21.15.28:0/1049202831 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff494005020 con 0x562a2a6ec950 2024-04-24T08:19:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.575+0000 7ff49aa00640 1 -- 172.21.15.28:0/1049202831 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff49c057050 con 0x562a2a6ec950 2024-04-24T08:19:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.575+0000 7ff4b69b3740 10 client.26803 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.575+0000 7ff4b69b3740 1 -- 172.21.15.28:0/1049202831 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562a2a3643f0 con 0x562a2a6ec950 2024-04-24T08:19:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.575+0000 7ff49aa00640 1 -- 172.21.15.28:0/1049202831 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff494005020 con 0x562a2a6ec950 2024-04-24T08:19:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.575+0000 7ff49aa00640 1 -- 172.21.15.28:0/1049202831 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff49c049ba0 con 0x562a2a6ec950 2024-04-24T08:19:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.575+0000 7ff4b69b3740 10 client.26803 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:21.583 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.575+0000 7ff4b69b3740 10 client.26803 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:19:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.575+0000 7ff4b69b3740 20 client.26803 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.575+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/1049202831 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x562a2a707a50 0x562a2a709ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.575+0000 7ff4b69b3740 4 client.26803 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:19:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.575+0000 7ff4b69b3740 1 -- 172.21.15.28:0/1049202831 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a2a65fbf0 con 0x562a2a707a50 2024-04-24T08:19:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.576+0000 7ff4abe00640 1 --2- 172.21.15.28:0/1049202831 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x562a2a707a50 0x562a2a709ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.577+0000 7ff4abe00640 1 --2- 172.21.15.28:0/1049202831 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x562a2a707a50 0x562a2a709ec0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=23c74927d20b03ea server_cookie=8304fed4eddab8dc in_seq=0 out_seq=0 2024-04-24T08:19:21.584 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.577+0000 7ff49aa00640 10 client.26803 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:19:21.585 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.577+0000 7ff49aa00640 1 -- 172.21.15.28:0/1049202831 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562a2a65fbf0 con 0x562a2a707a50 2024-04-24T08:19:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.577+0000 7ff49aa00640 10 client.26803 handle_command_reply: tid=0 2024-04-24T08:19:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.577+0000 7ff4b69b3740 1 client.26803 shutdown 2024-04-24T08:19:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.578+0000 7ff4b69b3740 1 -- 172.21.15.28:0/1049202831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 msgr2=0x562a2a6f7270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.578+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/1049202831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 0x562a2a6f7270 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7ff49c034000 tx=0x7ff49c02a340 comp rx=0 tx=0).stop 2024-04-24T08:19:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.578+0000 7ff4b69b3740 1 -- 172.21.15.28:0/1049202831 shutdown_connections 2024-04-24T08:19:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.578+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/1049202831 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x562a2a707a50 0x562a2a709ec0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.586 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.578+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/1049202831 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6f7940 0x562a2a6f4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.578+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/1049202831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6ed270 0x562a2a6f4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.578+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/1049202831 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 0x562a2a6f7270 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.578+0000 7ff49aa00640 0 client.26803 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:19:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.578+0000 7ff4b69b3740 1 -- 172.21.15.28:0/1049202831 >> 172.21.15.28:0/1049202831 conn(0x562a2a6e8c70 msgr2=0x562a2a6eea70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.578+0000 7ff4b69b3740 1 -- 172.21.15.28:0/1049202831 shutdown_connections 2024-04-24T08:19:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.578+0000 7ff4b69b3740 1 -- 172.21.15.28:0/1049202831 wait complete. 2024-04-24T08:19:21.587 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.578+0000 7ff4b69b3740 20 client.26803 trim_cache size 0 max 16384 2024-04-24T08:19:21.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.593+0000 7ff4b69b3740 1 Processor -- start 2024-04-24T08:19:21.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.593+0000 7ff4b69b3740 1 -- start start 2024-04-24T08:19:21.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.594+0000 7ff4b69b3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 0x562a2a7036d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.594+0000 7ff4b69b3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6ed270 0x562a2a6f6da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.594+0000 7ff4b69b3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6f7940 0x562a2a70f7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.594+0000 7ff4b69b3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ac1b2930 con 0x562a2a6f7940 2024-04-24T08:19:21.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.594+0000 7ff4b69b3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ac1b0200 con 0x562a2a6ed270 2024-04-24T08:19:21.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.594+0000 7ff4b69b3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ac1ae070 con 0x562a2a6ec950 2024-04-24T08:19:21.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.594+0000 7ff4b1200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 0x562a2a7036d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.594+0000 7ff4b1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6f7940 0x562a2a70f7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.594+0000 7ff4b1200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 0x562a2a7036d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44010/0 (socket says 172.21.15.28:44010) 2024-04-24T08:19:21.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.594+0000 7ff4b1200640 1 -- 172.21.15.28:0/3552980248 learned_addr learned my addr 172.21.15.28:0/3552980248 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:21.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.594+0000 7ff4abe00640 1 --2- 172.21.15.28:0/3552980248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6ed270 0x562a2a6f6da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.594+0000 7ff4b1c00640 1 -- 172.21.15.28:0/3552980248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 msgr2=0x562a2a7036d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.594+0000 7ff4b1c00640 1 --2- 172.21.15.28:0/3552980248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 0x562a2a7036d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.595+0000 7ff4b1c00640 1 -- 172.21.15.28:0/3552980248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6ed270 msgr2=0x562a2a6f6da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.595+0000 7ff4b1c00640 1 --2- 172.21.15.28:0/3552980248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6ed270 0x562a2a6f6da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.595+0000 7ff4b1c00640 1 -- 172.21.15.28:0/3552980248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a2a42e7d0 con 0x562a2a6f7940 2024-04-24T08:19:21.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.595+0000 7ff4b1200640 1 --2- 172.21.15.28:0/3552980248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 0x562a2a7036d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:21.598 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.595+0000 7ff4abe00640 1 --2- 172.21.15.28:0/3552980248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6ed270 0x562a2a6f6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:21.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.595+0000 7ff4b1c00640 1 --2- 172.21.15.28:0/3552980248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6f7940 0x562a2a70f7e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff4a40ce2c0 tx=0x7ff4a4010430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b27f10151df8295a server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.595+0000 7ff4a1600640 1 -- 172.21.15.28:0/3552980248 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a409cb10 con 0x562a2a6f7940 2024-04-24T08:19:21.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.595+0000 7ff4a1600640 1 -- 172.21.15.28:0/3552980248 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4a40a0ec0 con 0x562a2a6f7940 2024-04-24T08:19:21.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.595+0000 7ff4a1600640 1 -- 172.21.15.28:0/3552980248 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a4004430 con 0x562a2a6f7940 2024-04-24T08:19:21.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.596+0000 7ff4b69b3740 1 -- 172.21.15.28:0/3552980248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6f7940 msgr2=0x562a2a70f7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.603 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.596+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/3552980248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6f7940 0x562a2a70f7e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff4a40ce2c0 tx=0x7ff4a4010430 comp rx=0 tx=0).stop 2024-04-24T08:19:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.596+0000 7ff4b69b3740 1 -- 172.21.15.28:0/3552980248 shutdown_connections 2024-04-24T08:19:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.596+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/3552980248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6f7940 0x562a2a70f7e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.596+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/3552980248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6ed270 0x562a2a6f6da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.596+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/3552980248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 0x562a2a7036d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.596+0000 7ff4b69b3740 1 -- 172.21.15.28:0/3552980248 >> 172.21.15.28:0/3552980248 conn(0x562a2a6e8c70 msgr2=0x562a2a6ef800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.596+0000 7ff4b69b3740 1 -- 172.21.15.28:0/3552980248 shutdown_connections 2024-04-24T08:19:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.596+0000 7ff4b69b3740 1 -- 172.21.15.28:0/3552980248 wait complete. 2024-04-24T08:19:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.597+0000 7ff4b69b3740 1 -- start start 2024-04-24T08:19:21.604 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.597+0000 7ff4b69b3740 1 Processor -- start 2024-04-24T08:19:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.597+0000 7ff4b69b3740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 0x562a2a7000c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.597+0000 7ff4b69b3740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6ed270 0x562a2a6f0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.597+0000 7ff4b69b3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6f7940 0x562a2a6f1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.597+0000 7ff4b69b3740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ac1ae6c0 con 0x562a2a6ed270 2024-04-24T08:19:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.597+0000 7ff4b69b3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ac1afb30 con 0x562a2a6f7940 2024-04-24T08:19:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.597+0000 7ff4b69b3740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff4ac1aed90 con 0x562a2a6ec950 2024-04-24T08:19:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.597+0000 7ff4b69b3740 10 client.0 _ll_register_callbacks cb 0x7ff4a40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:21.605 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.598+0000 7ff4abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6ed270 0x562a2a6f0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.598+0000 7ff4b1c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6f7940 0x562a2a6f1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.598+0000 7ff4b1200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 0x562a2a7000c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.598+0000 7ff4abe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6ed270 0x562a2a6f0ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47756/0 (socket says 172.21.15.28:47756) 2024-04-24T08:19:21.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.598+0000 7ff4abe00640 1 -- 172.21.15.28:0/3717230880 learned_addr learned my addr 172.21.15.28:0/3717230880 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:21.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.598+0000 7ff4b1c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6f7940 0x562a2a6f1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:52894/0 (socket says 172.21.15.28:52894) 2024-04-24T08:19:21.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.598+0000 7ff4b1c00640 1 -- 172.21.15.28:0/3717230880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 msgr2=0x562a2a7000c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.598+0000 7ff4b1c00640 1 --2- 172.21.15.28:0/3717230880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 0x562a2a7000c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.598+0000 7ff4b1c00640 1 -- 172.21.15.28:0/3717230880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6ed270 msgr2=0x562a2a6f0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.598+0000 7ff4b1c00640 1 --2- 172.21.15.28:0/3717230880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6ed270 0x562a2a6f0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.598+0000 7ff4b1c00640 1 -- 172.21.15.28:0/3717230880 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a2a6f1dc0 con 0x562a2a6f7940 2024-04-24T08:19:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.598+0000 7ff4abe00640 1 --2- 172.21.15.28:0/3717230880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6ed270 0x562a2a6f0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.598+0000 7ff4b1c00640 1 --2- 172.21.15.28:0/3717230880 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6f7940 0x562a2a6f1590 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4003fb0 tx=0x7ff4a4009920 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6aa538165f776fa4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.598+0000 7ff49aa00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:19:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.599+0000 7ff49aa00640 1 -- 172.21.15.28:0/3717230880 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4a409cb10 con 0x562a2a6f7940 2024-04-24T08:19:21.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.599+0000 7ff49aa00640 1 -- 172.21.15.28:0/3717230880 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7ff4a4004620 con 0x562a2a6f7940 2024-04-24T08:19:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.599+0000 7ff4b69b3740 1 -- 172.21.15.28:0/3717230880 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff4a40a0ec0 con 0x562a2a6f7940 2024-04-24T08:19:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.599+0000 7ff49aa00640 1 -- 172.21.15.28:0/3717230880 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff4a40a5780 con 0x562a2a6f7940 2024-04-24T08:19:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.599+0000 7ff4b69b3740 10 client.26786 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.599+0000 7ff4b69b3740 1 -- 172.21.15.28:0/3717230880 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x562a2a76e0c0 con 0x562a2a6f7940 2024-04-24T08:19:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.599+0000 7ff49aa00640 1 -- 172.21.15.28:0/3717230880 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4a40a0ec0 con 0x562a2a6f7940 2024-04-24T08:19:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.599+0000 7ff49aa00640 1 -- 172.21.15.28:0/3717230880 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff4a401f020 con 0x562a2a6f7940 2024-04-24T08:19:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.599+0000 7ff4b69b3740 10 client.26786 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:21.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.599+0000 7ff4b69b3740 10 client.26786 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:19:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.599+0000 7ff4b69b3740 20 client.26786 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.599+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/3717230880 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x562a2a76e720 0x562a2a770b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.599+0000 7ff4b69b3740 4 client.26786 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:19:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.599+0000 7ff4b69b3740 1 -- 172.21.15.28:0/3717230880 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x562a2a659a60 con 0x562a2a76e720 2024-04-24T08:19:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.600+0000 7ff4b1200640 1 --2- 172.21.15.28:0/3717230880 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x562a2a76e720 0x562a2a770b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.600+0000 7ff4b1200640 1 --2- 172.21.15.28:0/3717230880 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x562a2a76e720 0x562a2a770b10 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7c657bdae3b8a626 server_cookie=79dce64843b36fb4 in_seq=0 out_seq=0 2024-04-24T08:19:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.600+0000 7ff49aa00640 10 client.26786 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:19:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.600+0000 7ff49aa00640 1 -- 172.21.15.28:0/3717230880 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x562a2a659a60 con 0x562a2a76e720 2024-04-24T08:19:21.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.600+0000 7ff49aa00640 10 client.26786 handle_command_reply: tid=0 2024-04-24T08:19:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.600+0000 7ff4b69b3740 1 client.26786 shutdown 2024-04-24T08:19:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.601+0000 7ff4b69b3740 1 -- 172.21.15.28:0/3717230880 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6f7940 msgr2=0x562a2a6f1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.601+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/3717230880 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6f7940 0x562a2a6f1590 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7ff4a4003fb0 tx=0x7ff4a4009920 comp rx=0 tx=0).stop 2024-04-24T08:19:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.601+0000 7ff4b69b3740 1 -- 172.21.15.28:0/3717230880 shutdown_connections 2024-04-24T08:19:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.601+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/3717230880 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x562a2a76e720 0x562a2a770b10 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.601+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/3717230880 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x562a2a6ed270 0x562a2a6f0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.601+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/3717230880 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x562a2a6f7940 0x562a2a6f1590 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.601+0000 7ff4b69b3740 1 --2- 172.21.15.28:0/3717230880 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x562a2a6ec950 0x562a2a7000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.601+0000 7ff49aa00640 0 client.26786 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:19:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.601+0000 7ff4b69b3740 1 -- 172.21.15.28:0/3717230880 >> 172.21.15.28:0/3717230880 conn(0x562a2a6e8c70 msgr2=0x562a2a6e7560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.601+0000 7ff4b69b3740 1 -- 172.21.15.28:0/3717230880 shutdown_connections 2024-04-24T08:19:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.601+0000 7ff4b69b3740 1 -- 172.21.15.28:0/3717230880 wait complete. 2024-04-24T08:19:21.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.601+0000 7ff4b69b3740 20 client.26786 trim_cache size 0 max 16384 2024-04-24T08:19:21.611 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:19:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.602+0000 7ff4b3c00640 1 -- 172.21.15.28:0/2446085254 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff48c0a3da0 msgr2=0x7ff48c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.602+0000 7ff4b3c00640 1 --2- 172.21.15.28:0/2446085254 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff48c0a3da0 0x7ff48c0a6260 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff49c009830 tx=0x7ff49c0056d0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.602+0000 7ff4b3c00640 1 -- 172.21.15.28:0/2446085254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4ac19a9d0 msgr2=0x7ff4ac19ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.602+0000 7ff4b3c00640 1 --2- 172.21.15.28:0/2446085254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4ac19a9d0 0x7ff4ac19ae80 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7ff4a400efd0 tx=0x7ff4a400c6a0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.602+0000 7ff4b3c00640 1 -- 172.21.15.28:0/2446085254 shutdown_connections 2024-04-24T08:19:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.602+0000 7ff4b3c00640 1 --2- 172.21.15.28:0/2446085254 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff48c0a3da0 0x7ff48c0a6260 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.602+0000 7ff4b3c00640 1 --2- 172.21.15.28:0/2446085254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4ac19a9d0 0x7ff4ac19ae80 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.602+0000 7ff4b3c00640 1 --2- 172.21.15.28:0/2446085254 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff4ac107360 0x7ff4ac1a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.602+0000 7ff4b3c00640 1 --2- 172.21.15.28:0/2446085254 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff4ac102290 0x7ff4ac1a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.602+0000 7ff4b3c00640 1 -- 172.21.15.28:0/2446085254 >> 172.21.15.28:0/2446085254 conn(0x7ff4ac0fdb10 msgr2=0x7ff4ac0ffe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.602+0000 7ff4b3c00640 1 -- 172.21.15.28:0/2446085254 shutdown_connections 2024-04-24T08:19:21.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.602+0000 7ff4b3c00640 1 -- 172.21.15.28:0/2446085254 wait complete. 2024-04-24T08:19:21.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:21 smithi028 ceph-mon[27786]: pgmap v279: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:19:21.619 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:21 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1604296888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:19:21.622 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:19:21.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.829+0000 7fd049000640 1 -- 172.21.15.28:0/823057289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0441011a0 msgr2=0x7fd044110f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.831 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.829+0000 7fd049000640 1 --2- 172.21.15.28:0/823057289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0441011a0 0x7fd044110f90 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7fd03c009ca0 tx=0x7fd03c02a8b0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.829+0000 7fd049000640 1 -- 172.21.15.28:0/823057289 shutdown_connections 2024-04-24T08:19:21.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.829+0000 7fd049000640 1 --2- 172.21.15.28:0/823057289 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0441011a0 0x7fd044110f90 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.829+0000 7fd049000640 1 --2- 172.21.15.28:0/823057289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0441007e0 0x7fd044100c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.829+0000 7fd049000640 1 --2- 172.21.15.28:0/823057289 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd044102d30 0x7fd044103110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.829+0000 7fd049000640 1 -- 172.21.15.28:0/823057289 >> 172.21.15.28:0/823057289 conn(0x7fd044076090 msgr2=0x7fd0440764a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.830+0000 7fd049000640 1 -- 172.21.15.28:0/823057289 shutdown_connections 2024-04-24T08:19:21.832 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.830+0000 7fd049000640 1 -- 172.21.15.28:0/823057289 wait complete. 2024-04-24T08:19:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.830+0000 7fd049000640 1 Processor -- start 2024-04-24T08:19:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.831+0000 7fd049000640 1 -- start start 2024-04-24T08:19:21.834 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.831+0000 7fd049000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0441007e0 0x7fd0441a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.831+0000 7fd049000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0441011a0 0x7fd0441a0bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.831+0000 7fd049000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd044102d30 0x7fd04419a7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.831+0000 7fd049000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd044114520 con 0x7fd0441007e0 2024-04-24T08:19:21.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.831+0000 7fd049000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd0441143a0 con 0x7fd0441011a0 2024-04-24T08:19:21.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.831+0000 7fd049000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd0441146a0 con 0x7fd044102d30 2024-04-24T08:19:21.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.831+0000 7fd043400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0441007e0 0x7fd0441a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.831+0000 7fd043400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0441007e0 0x7fd0441a0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47786/0 (socket says 172.21.15.28:47786) 2024-04-24T08:19:21.835 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.831+0000 7fd043400640 1 -- 172.21.15.28:0/498301603 learned_addr learned my addr 172.21.15.28:0/498301603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:21.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.832+0000 7fd043e00640 1 --2- 172.21.15.28:0/498301603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd044102d30 0x7fd04419a7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.832+0000 7fd03be00640 1 --2- 172.21.15.28:0/498301603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0441011a0 0x7fd0441a0bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.832+0000 7fd043400640 1 -- 172.21.15.28:0/498301603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd044102d30 msgr2=0x7fd04419a7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.832+0000 7fd043400640 1 --2- 172.21.15.28:0/498301603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd044102d30 0x7fd04419a7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.832+0000 7fd043400640 1 -- 172.21.15.28:0/498301603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0441011a0 msgr2=0x7fd0441a0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.832+0000 7fd043400640 1 --2- 172.21.15.28:0/498301603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0441011a0 0x7fd0441a0bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.832+0000 7fd043400640 1 -- 172.21.15.28:0/498301603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd04419afb0 con 0x7fd0441007e0 2024-04-24T08:19:21.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.832+0000 7fd043400640 1 --2- 172.21.15.28:0/498301603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0441007e0 0x7fd0441a0670 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd03400e9f0 tx=0x7fd03400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.836 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.832+0000 7fd040c00640 1 -- 172.21.15.28:0/498301603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03400cd80 con 0x7fd0441007e0 2024-04-24T08:19:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.832+0000 7fd040c00640 1 -- 172.21.15.28:0/498301603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd034010910 con 0x7fd0441007e0 2024-04-24T08:19:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.832+0000 7fd049000640 1 -- 172.21.15.28:0/498301603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd04419b2a0 con 0x7fd0441007e0 2024-04-24T08:19:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.832+0000 7fd040c00640 1 -- 172.21.15.28:0/498301603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd034018990 con 0x7fd0441007e0 2024-04-24T08:19:21.837 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.833+0000 7fd049000640 1 -- 172.21.15.28:0/498301603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd0441a74c0 con 0x7fd0441007e0 2024-04-24T08:19:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.836+0000 7fd04d5c8740 1 Processor -- start 2024-04-24T08:19:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.836+0000 7fd040c00640 1 -- 172.21.15.28:0/498301603 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fd034010ac0 con 0x7fd0441007e0 2024-04-24T08:19:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.836+0000 7fd04d5c8740 1 -- start start 2024-04-24T08:19:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.836+0000 7fd04d5c8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd41033b40 0x55cd41033f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.837+0000 7fd04d5c8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 0x55cd4103e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.837+0000 7fd04d5c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4103eb30 0x55cd41040f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.837+0000 7fd04d5c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cd40f991d0 con 0x55cd4103eb30 2024-04-24T08:19:21.839 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.837+0000 7fd04d5c8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cd40f3f450 con 0x55cd41034460 2024-04-24T08:19:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.837+0000 7fd04d5c8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cd40f95190 con 0x55cd41033b40 2024-04-24T08:19:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.837+0000 7fd043400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 0x55cd4103e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.837+0000 7fd043400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 0x55cd4103e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:52914/0 (socket says 172.21.15.28:52914) 2024-04-24T08:19:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.837+0000 7fd043400640 1 -- 172.21.15.28:0/3167709769 learned_addr learned my addr 172.21.15.28:0/3167709769 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.837+0000 7fd03be00640 1 --2- 172.21.15.28:0/3167709769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4103eb30 0x55cd41040f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.837+0000 7fd043e00640 1 --2- 172.21.15.28:0/3167709769 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd41033b40 0x55cd41033f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.837+0000 7fd043400640 1 -- 172.21.15.28:0/3167709769 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd41033b40 msgr2=0x55cd41033f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.840 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.837+0000 7fd043400640 1 --2- 172.21.15.28:0/3167709769 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd41033b40 0x55cd41033f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.837+0000 7fd043400640 1 -- 172.21.15.28:0/3167709769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4103eb30 msgr2=0x55cd41040f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.837+0000 7fd043400640 1 --2- 172.21.15.28:0/3167709769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4103eb30 0x55cd41040f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.841 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.837+0000 7fd043400640 1 -- 172.21.15.28:0/3167709769 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd40ffc310 con 0x55cd41034460 2024-04-24T08:19:21.844 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.838+0000 7fd040c00640 1 --2- 172.21.15.28:0/498301603 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd0280a3d20 0x7fd0280a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.838+0000 7fd043400640 1 --2- 172.21.15.28:0/3167709769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 0x55cd4103e5f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd0340ce270 tx=0x7fd0340d9750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55c614c6a7e4f375 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.838+0000 7fd040c00640 1 -- 172.21.15.28:0/498301603 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7fd0340d0c90 con 0x7fd0441007e0 2024-04-24T08:19:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.838+0000 7fd043e00640 1 --2- 172.21.15.28:0/498301603 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd0280a3d20 0x7fd0280a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.838+0000 7fd038c00640 1 -- 172.21.15.28:0/3167709769 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd0340d9e60 con 0x55cd41034460 2024-04-24T08:19:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.838+0000 7fd038c00640 1 -- 172.21.15.28:0/3167709769 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0340de070 con 0x55cd41034460 2024-04-24T08:19:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.838+0000 7fd038c00640 1 -- 172.21.15.28:0/3167709769 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03409c4e0 con 0x55cd41034460 2024-04-24T08:19:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.839+0000 7fd04d5c8740 1 -- 172.21.15.28:0/3167709769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 msgr2=0x55cd4103e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.845 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.839+0000 7fd043e00640 1 --2- 172.21.15.28:0/498301603 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd0280a3d20 0x7fd0280a61e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd03c005cc0 tx=0x7fd03c005c30 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.839+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/3167709769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 0x55cd4103e5f0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7fd0340ce270 tx=0x7fd0340d9750 comp rx=0 tx=0).stop 2024-04-24T08:19:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.839+0000 7fd04d5c8740 1 -- 172.21.15.28:0/3167709769 shutdown_connections 2024-04-24T08:19:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.839+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/3167709769 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4103eb30 0x55cd41040f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.839+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/3167709769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 0x55cd4103e5f0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.839+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/3167709769 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd41033b40 0x55cd41033f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.839+0000 7fd04d5c8740 1 -- 172.21.15.28:0/3167709769 >> 172.21.15.28:0/3167709769 conn(0x55cd4102fe60 msgr2=0x55cd41030240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.839+0000 7fd04d5c8740 1 -- 172.21.15.28:0/3167709769 shutdown_connections 2024-04-24T08:19:21.846 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.839+0000 7fd04d5c8740 1 -- 172.21.15.28:0/3167709769 wait complete. 2024-04-24T08:19:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.840+0000 7fd04d5c8740 1 -- start start 2024-04-24T08:19:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.840+0000 7fd04d5c8740 1 Processor -- start 2024-04-24T08:19:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.840+0000 7fd04d5c8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd41033b40 0x55cd4103e580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.840+0000 7fd04d5c8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 0x55cd4103b870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.840+0000 7fd04d5c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4103eb30 0x55cd4103bfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.840+0000 7fd04d5c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cd41041960 con 0x55cd4103eb30 2024-04-24T08:19:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.841+0000 7fd04d5c8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cd40f52810 con 0x55cd41034460 2024-04-24T08:19:21.847 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.841+0000 7fd04d5c8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cd41041ae0 con 0x55cd41033b40 2024-04-24T08:19:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.841+0000 7fd04d5c8740 10 client.0 _ll_register_callbacks cb 0x55cd40f20140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.841+0000 7fd043400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd41033b40 0x55cd4103e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.841+0000 7fd043400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd41033b40 0x55cd4103e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44046/0 (socket says 172.21.15.28:44046) 2024-04-24T08:19:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.841+0000 7fd043400640 1 -- 172.21.15.28:0/770481298 learned_addr learned my addr 172.21.15.28:0/770481298 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.841+0000 7fd03be00640 1 --2- 172.21.15.28:0/770481298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 0x55cd4103b870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.841+0000 7fd043e00640 1 --2- 172.21.15.28:0/770481298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4103eb30 0x55cd4103bfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.841+0000 7fd03be00640 1 -- 172.21.15.28:0/770481298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd41033b40 msgr2=0x55cd4103e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.848 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.841+0000 7fd03be00640 1 --2- 172.21.15.28:0/770481298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd41033b40 0x55cd4103e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.841+0000 7fd03be00640 1 -- 172.21.15.28:0/770481298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4103eb30 msgr2=0x55cd4103bfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.841+0000 7fd03be00640 1 --2- 172.21.15.28:0/770481298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4103eb30 0x55cd4103bfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.841+0000 7fd03be00640 1 -- 172.21.15.28:0/770481298 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd40f633f0 con 0x55cd41034460 2024-04-24T08:19:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.842+0000 7fd03be00640 1 --2- 172.21.15.28:0/770481298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 0x55cd4103b870 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd0440fdac0 tx=0x7fd02c0128e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e300079da1eb8702 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.842+0000 7fd031600640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:19:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.842+0000 7fd031600640 1 -- 172.21.15.28:0/770481298 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd02c005020 con 0x55cd41034460 2024-04-24T08:19:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.842+0000 7fd04d5c8740 1 -- 172.21.15.28:0/770481298 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0340de070 con 0x55cd41034460 2024-04-24T08:19:21.849 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.842+0000 7fd031600640 1 -- 172.21.15.28:0/770481298 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7fd02c0051c0 con 0x55cd41034460 2024-04-24T08:19:21.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.842+0000 7fd031600640 1 -- 172.21.15.28:0/770481298 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd02c0053c0 con 0x55cd41034460 2024-04-24T08:19:21.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.842+0000 7fd04d5c8740 10 client.26798 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:21.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.842+0000 7fd04d5c8740 1 -- 172.21.15.28:0/770481298 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cd40cab3f0 con 0x55cd41034460 2024-04-24T08:19:21.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.843+0000 7fd031600640 1 -- 172.21.15.28:0/770481298 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0340de070 con 0x55cd41034460 2024-04-24T08:19:21.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.843+0000 7fd031600640 1 -- 172.21.15.28:0/770481298 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd02c019070 con 0x55cd41034460 2024-04-24T08:19:21.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.843+0000 7fd04d5c8740 10 client.26798 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:21.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.843+0000 7fd04d5c8740 10 client.26798 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:19:21.850 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.843+0000 7fd04d5c8740 20 client.26798 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:21.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.843+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/770481298 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55cd4104ed20 0x55cd41051180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.843+0000 7fd04d5c8740 4 client.26798 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:19:21.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.843+0000 7fd04d5c8740 1 -- 172.21.15.28:0/770481298 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cd40fa6bf0 con 0x55cd4104ed20 2024-04-24T08:19:21.851 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.843+0000 7fd043400640 1 --2- 172.21.15.28:0/770481298 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55cd4104ed20 0x55cd41051180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.844+0000 7fd043400640 1 --2- 172.21.15.28:0/770481298 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55cd4104ed20 0x55cd41051180 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dede6110380eceb server_cookie=8e6816e8edd237f8 in_seq=0 out_seq=0 2024-04-24T08:19:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.844+0000 7fd031600640 10 client.26798 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.845+0000 7fd031600640 1 -- 172.21.15.28:0/770481298 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cd40fa6bf0 con 0x55cd4104ed20 2024-04-24T08:19:21.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.845+0000 7fd031600640 10 client.26798 handle_command_reply: tid=0 2024-04-24T08:19:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.845+0000 7fd04d5c8740 1 client.26798 shutdown 2024-04-24T08:19:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.845+0000 7fd04d5c8740 1 -- 172.21.15.28:0/770481298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 msgr2=0x55cd4103b870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.845+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/770481298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 0x55cd4103b870 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fd0440fdac0 tx=0x7fd02c0128e0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.845+0000 7fd04d5c8740 1 -- 172.21.15.28:0/770481298 shutdown_connections 2024-04-24T08:19:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.845+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/770481298 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55cd4104ed20 0x55cd41051180 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.845+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/770481298 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd4103eb30 0x55cd4103bfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.845+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/770481298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 0x55cd4103b870 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.845+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/770481298 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd41033b40 0x55cd4103e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.845+0000 7fd031600640 0 client.26798 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.845+0000 7fd04d5c8740 1 -- 172.21.15.28:0/770481298 >> 172.21.15.28:0/770481298 conn(0x55cd4102fe60 msgr2=0x55cd41035c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.845+0000 7fd04d5c8740 1 -- 172.21.15.28:0/770481298 shutdown_connections 2024-04-24T08:19:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.845+0000 7fd04d5c8740 1 -- 172.21.15.28:0/770481298 wait complete. 2024-04-24T08:19:21.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.845+0000 7fd04d5c8740 20 client.26798 trim_cache size 0 max 16384 2024-04-24T08:19:21.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.861+0000 7fd04d5c8740 1 Processor -- start 2024-04-24T08:19:21.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.861+0000 7fd04d5c8740 1 -- start start 2024-04-24T08:19:21.862 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.861+0000 7fd04d5c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd41033b40 0x55cd4104a850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.861+0000 7fd04d5c8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd41034460 0x55cd4103e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.861+0000 7fd04d5c8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd4103eb30 0x55cd4103e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.861+0000 7fd04d5c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0441b27c0 con 0x55cd41033b40 2024-04-24T08:19:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.861+0000 7fd04d5c8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd0441b0090 con 0x55cd4103eb30 2024-04-24T08:19:21.863 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.861+0000 7fd04d5c8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd0441adf00 con 0x55cd41034460 2024-04-24T08:19:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.861+0000 7fd043400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd41033b40 0x55cd4104a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.861+0000 7fd043400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd41033b40 0x55cd4104a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47818/0 (socket says 172.21.15.28:47818) 2024-04-24T08:19:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.861+0000 7fd043400640 1 -- 172.21.15.28:0/4052098658 learned_addr learned my addr 172.21.15.28:0/4052098658 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.861+0000 7fd043e00640 1 --2- 172.21.15.28:0/4052098658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd4103eb30 0x55cd4103e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.865 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.861+0000 7fd03be00640 1 --2- 172.21.15.28:0/4052098658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd41034460 0x55cd4103e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.862+0000 7fd043400640 1 -- 172.21.15.28:0/4052098658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd41034460 msgr2=0x55cd4103e0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.862+0000 7fd043400640 1 --2- 172.21.15.28:0/4052098658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd41034460 0x55cd4103e0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.862+0000 7fd043400640 1 -- 172.21.15.28:0/4052098658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd4103eb30 msgr2=0x55cd4103e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.862+0000 7fd043400640 1 --2- 172.21.15.28:0/4052098658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd4103eb30 0x55cd4103e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.866 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.862+0000 7fd043400640 1 -- 172.21.15.28:0/4052098658 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cd40d757d0 con 0x55cd41033b40 2024-04-24T08:19:21.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.862+0000 7fd043400640 1 --2- 172.21.15.28:0/4052098658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd41033b40 0x55cd4104a850 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd03409c880 tx=0x7fd034018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3e48a3dd2d12a2c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.862+0000 7fd033e00640 1 -- 172.21.15.28:0/4052098658 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03409c6c0 con 0x55cd41033b40 2024-04-24T08:19:21.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.862+0000 7fd033e00640 1 -- 172.21.15.28:0/4052098658 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0340d4510 con 0x55cd41033b40 2024-04-24T08:19:21.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.862+0000 7fd033e00640 1 -- 172.21.15.28:0/4052098658 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03409c950 con 0x55cd41033b40 2024-04-24T08:19:21.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.863+0000 7fd04d5c8740 1 -- 172.21.15.28:0/4052098658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd41033b40 msgr2=0x55cd4104a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.863+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/4052098658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd41033b40 0x55cd4104a850 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fd03409c880 tx=0x7fd034018420 comp rx=0 tx=0).stop 2024-04-24T08:19:21.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.863+0000 7fd04d5c8740 1 -- 172.21.15.28:0/4052098658 shutdown_connections 2024-04-24T08:19:21.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.863+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/4052098658 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd4103eb30 0x55cd4103e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.863+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/4052098658 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd41034460 0x55cd4103e0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.863+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/4052098658 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd41033b40 0x55cd4104a850 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.863+0000 7fd04d5c8740 1 -- 172.21.15.28:0/4052098658 >> 172.21.15.28:0/4052098658 conn(0x55cd4102fe60 msgr2=0x55cd41036b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.863+0000 7fd04d5c8740 1 -- 172.21.15.28:0/4052098658 shutdown_connections 2024-04-24T08:19:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.863+0000 7fd04d5c8740 1 -- 172.21.15.28:0/4052098658 wait complete. 2024-04-24T08:19:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.864+0000 7fd04d5c8740 1 -- start start 2024-04-24T08:19:21.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.864+0000 7fd04d5c8740 1 Processor -- start 2024-04-24T08:19:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.864+0000 7fd04d5c8740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd41033b40 0x55cd41046d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.864+0000 7fd04d5c8740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 0x55cd410472a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.864+0000 7fd04d5c8740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd4103eb30 0x55cd4103fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.864+0000 7fd04d5c8740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fd0441ae550 con 0x55cd41033b40 2024-04-24T08:19:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.864+0000 7fd04d5c8740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fd0441af9c0 con 0x55cd41034460 2024-04-24T08:19:21.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.864+0000 7fd04d5c8740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fd0441aec20 con 0x55cd4103eb30 2024-04-24T08:19:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.864+0000 7fd04d5c8740 10 client.0 _ll_register_callbacks cb 0x7fd02c00d8c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.865+0000 7fd043e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd4103eb30 0x55cd4103fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.865+0000 7fd043400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd41033b40 0x55cd41046d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.865+0000 7fd043e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd4103eb30 0x55cd4103fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44066/0 (socket says 172.21.15.28:44066) 2024-04-24T08:19:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.865+0000 7fd043e00640 1 -- 172.21.15.28:0/3816799545 learned_addr learned my addr 172.21.15.28:0/3816799545 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.865+0000 7fd03be00640 1 --2- 172.21.15.28:0/3816799545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 0x55cd410472a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.865+0000 7fd043e00640 1 -- 172.21.15.28:0/3816799545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 msgr2=0x55cd410472a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.865+0000 7fd043e00640 1 --2- 172.21.15.28:0/3816799545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 0x55cd410472a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.865+0000 7fd043e00640 1 -- 172.21.15.28:0/3816799545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd41033b40 msgr2=0x55cd41046d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.865+0000 7fd043e00640 1 --2- 172.21.15.28:0/3816799545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd41033b40 0x55cd41046d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.865+0000 7fd043e00640 1 -- 172.21.15.28:0/3816799545 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cd410403c0 con 0x55cd4103eb30 2024-04-24T08:19:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.865+0000 7fd043400640 1 --2- 172.21.15.28:0/3816799545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd41033b40 0x55cd41046d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.865+0000 7fd043e00640 1 --2- 172.21.15.28:0/3816799545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd4103eb30 0x55cd4103fe00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd03c035000 tx=0x7fd03c0095c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e2916f4dea028f70 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.865+0000 7fd031600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:19:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.866+0000 7fd031600640 1 -- 172.21.15.28:0/3816799545 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd03c059060 con 0x55cd4103eb30 2024-04-24T08:19:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.866+0000 7fd04d5c8740 1 -- 172.21.15.28:0/3816799545 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd0340d4510 con 0x55cd4103eb30 2024-04-24T08:19:21.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.866+0000 7fd031600640 1 -- 172.21.15.28:0/3816799545 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7fd03c0317d0 con 0x55cd4103eb30 2024-04-24T08:19:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.866+0000 7fd031600640 1 -- 172.21.15.28:0/3816799545 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fd03c0319d0 con 0x55cd4103eb30 2024-04-24T08:19:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.866+0000 7fd04d5c8740 10 client.26827 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.866+0000 7fd04d5c8740 1 -- 172.21.15.28:0/3816799545 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cd410b5b90 con 0x55cd4103eb30 2024-04-24T08:19:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.867+0000 7fd031600640 1 -- 172.21.15.28:0/3816799545 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd0340d4510 con 0x55cd4103eb30 2024-04-24T08:19:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.867+0000 7fd031600640 1 -- 172.21.15.28:0/3816799545 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fd03c049420 con 0x55cd4103eb30 2024-04-24T08:19:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.867+0000 7fd04d5c8740 10 client.26827 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.867+0000 7fd04d5c8740 10 client.26827 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:19:21.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.867+0000 7fd04d5c8740 20 client.26827 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:21.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.867+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/3816799545 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55cd410b6370 0x55cd410b8790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:21.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.867+0000 7fd04d5c8740 4 client.26827 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:19:21.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.867+0000 7fd04d5c8740 1 -- 172.21.15.28:0/3816799545 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55cd40fa0a60 con 0x55cd410b6370 2024-04-24T08:19:21.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.867+0000 7fd043400640 1 --2- 172.21.15.28:0/3816799545 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55cd410b6370 0x55cd410b8790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:21.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.868+0000 7fd043400640 1 --2- 172.21.15.28:0/3816799545 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55cd410b6370 0x55cd410b8790 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b041df04a82c86c9 server_cookie=e44c3ec1822ae4bf in_seq=0 out_seq=0 2024-04-24T08:19:21.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.868+0000 7fd031600640 10 client.26827 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:21.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.868+0000 7fd031600640 1 -- 172.21.15.28:0/3816799545 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55cd40fa0a60 con 0x55cd410b6370 2024-04-24T08:19:21.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.868+0000 7fd031600640 10 client.26827 handle_command_reply: tid=0 2024-04-24T08:19:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.868+0000 7fd04d5c8740 1 client.26827 shutdown 2024-04-24T08:19:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.868+0000 7fd04d5c8740 1 -- 172.21.15.28:0/3816799545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd4103eb30 msgr2=0x55cd4103fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.868+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/3816799545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd4103eb30 0x55cd4103fe00 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fd03c035000 tx=0x7fd03c0095c0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.868+0000 7fd04d5c8740 1 -- 172.21.15.28:0/3816799545 shutdown_connections 2024-04-24T08:19:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.868+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/3816799545 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55cd410b6370 0x55cd410b8790 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.868+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/3816799545 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cd4103eb30 0x55cd4103fe00 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.868+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/3816799545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cd41034460 0x55cd410472a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.868+0000 7fd031600640 0 client.26827 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:21.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.868+0000 7fd04d5c8740 1 --2- 172.21.15.28:0/3816799545 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cd41033b40 0x55cd41046d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.868+0000 7fd04d5c8740 1 -- 172.21.15.28:0/3816799545 >> 172.21.15.28:0/3816799545 conn(0x55cd4102fe60 msgr2=0x55cd4102e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.868+0000 7fd04d5c8740 1 -- 172.21.15.28:0/3816799545 shutdown_connections 2024-04-24T08:19:21.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.868+0000 7fd04d5c8740 1 -- 172.21.15.28:0/3816799545 wait complete. 2024-04-24T08:19:21.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.869+0000 7fd04d5c8740 20 client.26827 trim_cache size 0 max 16384 2024-04-24T08:19:21.878 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:19:21.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.869+0000 7fd049000640 1 -- 172.21.15.28:0/498301603 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd0280a3d20 msgr2=0x7fd0280a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.869+0000 7fd049000640 1 --2- 172.21.15.28:0/498301603 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd0280a3d20 0x7fd0280a61e0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd03c005cc0 tx=0x7fd03c005c30 comp rx=0 tx=0).stop 2024-04-24T08:19:21.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.869+0000 7fd049000640 1 -- 172.21.15.28:0/498301603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0441007e0 msgr2=0x7fd0441a0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:21.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.869+0000 7fd049000640 1 --2- 172.21.15.28:0/498301603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0441007e0 0x7fd0441a0670 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fd03400e9f0 tx=0x7fd03400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.869+0000 7fd049000640 1 -- 172.21.15.28:0/498301603 shutdown_connections 2024-04-24T08:19:21.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.869+0000 7fd049000640 1 --2- 172.21.15.28:0/498301603 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fd0280a3d20 0x7fd0280a61e0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.869+0000 7fd049000640 1 --2- 172.21.15.28:0/498301603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fd044102d30 0x7fd04419a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.869+0000 7fd049000640 1 --2- 172.21.15.28:0/498301603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd0441011a0 0x7fd0441a0bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.869+0000 7fd049000640 1 --2- 172.21.15.28:0/498301603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fd0441007e0 0x7fd0441a0670 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:21.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.869+0000 7fd049000640 1 -- 172.21.15.28:0/498301603 >> 172.21.15.28:0/498301603 conn(0x7fd044076090 msgr2=0x7fd0440fe260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:21.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.870+0000 7fd049000640 1 -- 172.21.15.28:0/498301603 shutdown_connections 2024-04-24T08:19:21.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:21.870+0000 7fd049000640 1 -- 172.21.15.28:0/498301603 wait complete. 2024-04-24T08:19:21.881 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:21 smithi117 ceph-mon[32438]: pgmap v279: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:19:21.881 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:21 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1604296888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:19:21.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:21 smithi194 ceph-mon[32836]: pgmap v279: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:19:21.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:21 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1604296888' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:19:22.889 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:19:22.889 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:19:22.889 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:19:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.120+0000 7f7ce2600640 1 -- 172.21.15.28:0/941149943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc100ec0 msgr2=0x7f7cdc1012a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.122 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.120+0000 7f7ce2600640 1 --2- 172.21.15.28:0/941149943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc100ec0 0x7f7cdc1012a0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009900 tx=0x7f7cd002a470 comp rx=0 tx=0).stop 2024-04-24T08:19:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.120+0000 7f7ce2600640 1 -- 172.21.15.28:0/941149943 shutdown_connections 2024-04-24T08:19:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.120+0000 7f7ce2600640 1 --2- 172.21.15.28:0/941149943 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cdc1062c0 0x7f7cdc106750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.120+0000 7f7ce2600640 1 --2- 172.21.15.28:0/941149943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cdc1017e0 0x7f7cdc105ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.120+0000 7f7ce2600640 1 --2- 172.21.15.28:0/941149943 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc100ec0 0x7f7cdc1012a0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.120+0000 7f7ce2600640 1 -- 172.21.15.28:0/941149943 >> 172.21.15.28:0/941149943 conn(0x7f7cdc0fc910 msgr2=0x7f7cdc0fed30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.120+0000 7f7ce2600640 1 -- 172.21.15.28:0/941149943 shutdown_connections 2024-04-24T08:19:23.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.121+0000 7f7ce2600640 1 -- 172.21.15.28:0/941149943 wait complete. 2024-04-24T08:19:23.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.121+0000 7f7ce2600640 1 Processor -- start 2024-04-24T08:19:23.125 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.122+0000 7f7ce2600640 1 -- start start 2024-04-24T08:19:23.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.122+0000 7f7ce2600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cdc100ec0 0x7f7cdc19edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.122+0000 7f7ce2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc1017e0 0x7f7cdc19f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.122+0000 7f7ce2600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cdc1062c0 0x7f7cdc1a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.122+0000 7f7ce2600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc118930 con 0x7f7cdc1017e0 2024-04-24T08:19:23.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.122+0000 7f7ce2600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1187b0 con 0x7f7cdc100ec0 2024-04-24T08:19:23.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.122+0000 7f7ce2600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc118ab0 con 0x7f7cdc1062c0 2024-04-24T08:19:23.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.122+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc1017e0 0x7f7cdc19f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.126 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.122+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc1017e0 0x7f7cdc19f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47840/0 (socket says 172.21.15.28:47840) 2024-04-24T08:19:23.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.122+0000 7f7cdbe00640 1 -- 172.21.15.28:0/3013223456 learned_addr learned my addr 172.21.15.28:0/3013223456 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:23.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.122+0000 7f7ce1c00640 1 --2- 172.21.15.28:0/3013223456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cdc1062c0 0x7f7cdc1a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.122+0000 7f7ce1200640 1 --2- 172.21.15.28:0/3013223456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cdc100ec0 0x7f7cdc19edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.123+0000 7f7ce1c00640 1 -- 172.21.15.28:0/3013223456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cdc100ec0 msgr2=0x7f7cdc19edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.123+0000 7f7ce1c00640 1 --2- 172.21.15.28:0/3013223456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cdc100ec0 0x7f7cdc19edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.123+0000 7f7ce1c00640 1 -- 172.21.15.28:0/3013223456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc1017e0 msgr2=0x7f7cdc19f330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.123+0000 7f7ce1c00640 1 --2- 172.21.15.28:0/3013223456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc1017e0 0x7f7cdc19f330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.123+0000 7f7ce1c00640 1 -- 172.21.15.28:0/3013223456 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7cdc1a3de0 con 0x7f7cdc1062c0 2024-04-24T08:19:23.127 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.123+0000 7f7cdbe00640 1 --2- 172.21.15.28:0/3013223456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc1017e0 0x7f7cdc19f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:23.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.123+0000 7f7ce1c00640 1 --2- 172.21.15.28:0/3013223456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cdc1062c0 0x7f7cdc1a36b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009fd0 tx=0x7f7cd400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:23.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.123+0000 7f7cd9600640 1 -- 172.21.15.28:0/3013223456 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd400ce20 con 0x7f7cdc1062c0 2024-04-24T08:19:23.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.123+0000 7f7ce2600640 1 -- 172.21.15.28:0/3013223456 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7cdc1a40d0 con 0x7f7cdc1062c0 2024-04-24T08:19:23.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.124+0000 7f7cd9600640 1 -- 172.21.15.28:0/3013223456 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cd4005020 con 0x7f7cdc1062c0 2024-04-24T08:19:23.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.124+0000 7f7cd9600640 1 -- 172.21.15.28:0/3013223456 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd4010720 con 0x7f7cdc1062c0 2024-04-24T08:19:23.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.124+0000 7f7ce2600640 1 -- 172.21.15.28:0/3013223456 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7cdc1ab960 con 0x7f7cdc1062c0 2024-04-24T08:19:23.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.127+0000 7f7ce6722740 1 Processor -- start 2024-04-24T08:19:23.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.127+0000 7f7cd9600640 1 -- 172.21.15.28:0/3013223456 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f7cd4005540 con 0x7f7cdc1062c0 2024-04-24T08:19:23.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.127+0000 7f7ce6722740 1 -- start start 2024-04-24T08:19:23.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.128+0000 7f7ce6722740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d628270 0x55848d628650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.128+0000 7f7ce6722740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628b90 0x55848d632d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.128+0000 7f7ce6722740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d633260 0x55848d635650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.128+0000 7f7ce6722740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55848d58de70 con 0x55848d628270 2024-04-24T08:19:23.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.128+0000 7f7ce6722740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55848d534380 con 0x55848d628b90 2024-04-24T08:19:23.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.128+0000 7f7ce6722740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55848d589e30 con 0x55848d633260 2024-04-24T08:19:23.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.128+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d628270 0x55848d628650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.128+0000 7f7cdbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d628270 0x55848d628650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47850/0 (socket says 172.21.15.28:47850) 2024-04-24T08:19:23.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.128+0000 7f7cdbe00640 1 -- 172.21.15.28:0/467022506 learned_addr learned my addr 172.21.15.28:0/467022506 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:23.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.128+0000 7f7ce1200640 1 --2- 172.21.15.28:0/467022506 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628b90 0x55848d632d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.128+0000 7f7cdbe00640 1 --2- 172.21.15.28:0/467022506 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d633260 0x55848d635650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.129+0000 7f7ce1200640 1 -- 172.21.15.28:0/467022506 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d633260 msgr2=0x55848d635650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:23.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.129+0000 7f7ce1200640 1 --2- 172.21.15.28:0/467022506 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d633260 0x55848d635650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.129+0000 7f7ce1200640 1 -- 172.21.15.28:0/467022506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d628270 msgr2=0x55848d628650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:23.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.129+0000 7f7ce1200640 1 --2- 172.21.15.28:0/467022506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d628270 0x55848d628650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.129+0000 7f7ce1200640 1 -- 172.21.15.28:0/467022506 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55848d3687e0 con 0x55848d628b90 2024-04-24T08:19:23.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.129+0000 7f7cdbe00640 1 --2- 172.21.15.28:0/467022506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d628270 0x55848d628650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:23.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.129+0000 7f7cd9600640 1 --2- 172.21.15.28:0/3013223456 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7cc40a3d20 0x7f7cc40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.129+0000 7f7ce1200640 1 --2- 172.21.15.28:0/467022506 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628b90 0x55848d632d20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7cd002aa60 tx=0x7f7cd0005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ca2bc4a9d64cf55c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:23.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.129+0000 7f7cd9600640 1 -- 172.21.15.28:0/3013223456 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7f7cd40d0e70 con 0x7f7cdc1062c0 2024-04-24T08:19:23.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.129+0000 7f7cdbe00640 1 --2- 172.21.15.28:0/3013223456 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7cc40a3d20 0x7f7cc40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.129+0000 7f7cce000640 1 -- 172.21.15.28:0/467022506 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd0035040 con 0x55848d628b90 2024-04-24T08:19:23.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.130+0000 7f7cce000640 1 -- 172.21.15.28:0/467022506 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cd0004400 con 0x55848d628b90 2024-04-24T08:19:23.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.130+0000 7f7cce000640 1 -- 172.21.15.28:0/467022506 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd00035f0 con 0x55848d628b90 2024-04-24T08:19:23.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.130+0000 7f7ce6722740 1 -- 172.21.15.28:0/467022506 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628b90 msgr2=0x55848d632d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.130+0000 7f7cdbe00640 1 --2- 172.21.15.28:0/3013223456 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7cc40a3d20 0x7f7cc40a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x55848d625720 tx=0x7f7cc8009c00 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.130+0000 7f7ce6722740 1 --2- 172.21.15.28:0/467022506 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628b90 0x55848d632d20 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7cd002aa60 tx=0x7f7cd0005de0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.130+0000 7f7ce6722740 1 -- 172.21.15.28:0/467022506 shutdown_connections 2024-04-24T08:19:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.130+0000 7f7ce6722740 1 --2- 172.21.15.28:0/467022506 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d633260 0x55848d635650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.130+0000 7f7ce6722740 1 --2- 172.21.15.28:0/467022506 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628b90 0x55848d632d20 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.130+0000 7f7ce6722740 1 --2- 172.21.15.28:0/467022506 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d628270 0x55848d628650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.130+0000 7f7ce6722740 1 -- 172.21.15.28:0/467022506 >> 172.21.15.28:0/467022506 conn(0x55848d624590 msgr2=0x55848d624970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:23.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.131+0000 7f7ce6722740 1 -- 172.21.15.28:0/467022506 shutdown_connections 2024-04-24T08:19:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.131+0000 7f7ce6722740 1 -- 172.21.15.28:0/467022506 wait complete. 2024-04-24T08:19:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.131+0000 7f7ce6722740 1 -- start start 2024-04-24T08:19:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.131+0000 7f7ce6722740 1 Processor -- start 2024-04-24T08:19:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.132+0000 7f7ce6722740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628270 0x55848d6301c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.132+0000 7f7ce6722740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d628b90 0x55848d630700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.132+0000 7f7ce6722740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d633260 0x55848d630c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.132+0000 7f7ce6722740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55848d635ff0 con 0x55848d633260 2024-04-24T08:19:23.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.132+0000 7f7ce6722740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55848d547660 con 0x55848d628270 2024-04-24T08:19:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.132+0000 7f7ce6722740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55848d636170 con 0x55848d628b90 2024-04-24T08:19:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.132+0000 7f7ce6722740 10 client.0 _ll_register_callbacks cb 0x55848d535aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.132+0000 7f7ce1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d633260 0x55848d630c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.132+0000 7f7ce1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d633260 0x55848d630c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47852/0 (socket says 172.21.15.28:47852) 2024-04-24T08:19:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.132+0000 7f7ce1c00640 1 -- 172.21.15.28:0/1159613959 learned_addr learned my addr 172.21.15.28:0/1159613959 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.132+0000 7f7ce1200640 1 --2- 172.21.15.28:0/1159613959 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628270 0x55848d6301c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.132+0000 7f7cdbe00640 1 --2- 172.21.15.28:0/1159613959 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d628b90 0x55848d630700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.141 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.133+0000 7f7ce1c00640 1 -- 172.21.15.28:0/1159613959 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d628b90 msgr2=0x55848d630700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.133+0000 7f7ce1c00640 1 --2- 172.21.15.28:0/1159613959 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d628b90 0x55848d630700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.133+0000 7f7ce1c00640 1 -- 172.21.15.28:0/1159613959 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628270 msgr2=0x55848d6301c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.133+0000 7f7ce1c00640 1 --2- 172.21.15.28:0/1159613959 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628270 0x55848d6301c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.133+0000 7f7ce1c00640 1 -- 172.21.15.28:0/1159613959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55848d3c8f70 con 0x55848d633260 2024-04-24T08:19:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.133+0000 7f7ce1200640 1 --2- 172.21.15.28:0/1159613959 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628270 0x55848d6301c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.133+0000 7f7ce1c00640 1 --2- 172.21.15.28:0/1159613959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d633260 0x55848d630c40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40a54a0 tx=0x7f7cd409d5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=314f89d5f4870fbd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.133+0000 7f7cc2a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:19:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.133+0000 7f7cc2a00640 1 -- 172.21.15.28:0/1159613959 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd40a5540 con 0x55848d633260 2024-04-24T08:19:23.142 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.133+0000 7f7cc2a00640 1 -- 172.21.15.28:0/1159613959 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7f7cd40a56e0 con 0x55848d633260 2024-04-24T08:19:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.133+0000 7f7ce6722740 1 -- 172.21.15.28:0/1159613959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cd0004400 con 0x55848d633260 2024-04-24T08:19:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.133+0000 7f7cc2a00640 1 -- 172.21.15.28:0/1159613959 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7cd409cb30 con 0x55848d633260 2024-04-24T08:19:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.133+0000 7f7ce6722740 10 client.16539 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.133+0000 7f7ce6722740 1 -- 172.21.15.28:0/1159613959 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55848d260170 con 0x55848d633260 2024-04-24T08:19:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.134+0000 7f7cc2a00640 1 -- 172.21.15.28:0/1159613959 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cd0004400 con 0x55848d633260 2024-04-24T08:19:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.134+0000 7f7cc2a00640 1 -- 172.21.15.28:0/1159613959 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f7cd409c1b0 con 0x55848d633260 2024-04-24T08:19:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.134+0000 7f7ce6722740 10 client.16539 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:23.143 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.134+0000 7f7ce6722740 10 client.16539 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:19:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.134+0000 7f7ce6722740 20 client.16539 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.134+0000 7f7ce6722740 1 --2- 172.21.15.28:0/1159613959 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55848d6434e0 0x55848d645940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.134+0000 7f7ce6722740 4 client.16539 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:19:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.134+0000 7f7ce6722740 1 -- 172.21.15.28:0/1159613959 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55848d59b890 con 0x55848d6434e0 2024-04-24T08:19:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.134+0000 7f7ce1200640 1 --2- 172.21.15.28:0/1159613959 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55848d6434e0 0x55848d645940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.135+0000 7f7ce1200640 1 --2- 172.21.15.28:0/1159613959 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55848d6434e0 0x55848d645940 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9609a7f14fbd85c1 server_cookie=c9504fdf4a199272 in_seq=0 out_seq=0 2024-04-24T08:19:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.135+0000 7f7cc2a00640 10 client.16539 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.136+0000 7f7cc2a00640 1 -- 172.21.15.28:0/1159613959 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55848d59b890 con 0x55848d6434e0 2024-04-24T08:19:23.144 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.136+0000 7f7cc2a00640 10 client.16539 handle_command_reply: tid=0 2024-04-24T08:19:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.136+0000 7f7ce6722740 1 client.16539 shutdown 2024-04-24T08:19:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.136+0000 7f7ce6722740 1 -- 172.21.15.28:0/1159613959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d633260 msgr2=0x55848d630c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.136+0000 7f7ce6722740 1 --2- 172.21.15.28:0/1159613959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d633260 0x55848d630c40 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f7cd40a54a0 tx=0x7f7cd409d5d0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.136+0000 7f7ce6722740 1 -- 172.21.15.28:0/1159613959 shutdown_connections 2024-04-24T08:19:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.136+0000 7f7ce6722740 1 --2- 172.21.15.28:0/1159613959 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55848d6434e0 0x55848d645940 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.136+0000 7f7ce6722740 1 --2- 172.21.15.28:0/1159613959 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d633260 0x55848d630c40 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.136+0000 7f7ce6722740 1 --2- 172.21.15.28:0/1159613959 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d628b90 0x55848d630700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.136+0000 7f7ce6722740 1 --2- 172.21.15.28:0/1159613959 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628270 0x55848d6301c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.136+0000 7f7cc2a00640 0 client.16539 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.136+0000 7f7ce6722740 1 -- 172.21.15.28:0/1159613959 >> 172.21.15.28:0/1159613959 conn(0x55848d624590 msgr2=0x55848d62a290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.136+0000 7f7ce6722740 1 -- 172.21.15.28:0/1159613959 shutdown_connections 2024-04-24T08:19:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.136+0000 7f7ce6722740 1 -- 172.21.15.28:0/1159613959 wait complete. 2024-04-24T08:19:23.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.137+0000 7f7ce6722740 20 client.16539 trim_cache size 0 max 16384 2024-04-24T08:19:23.157 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.156+0000 7f7ce6722740 1 Processor -- start 2024-04-24T08:19:23.158 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.156+0000 7f7ce6722740 1 -- start start 2024-04-24T08:19:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.156+0000 7f7ce6722740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628270 0x55848d63eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.156+0000 7f7ce6722740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d628b90 0x55848d623570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.156+0000 7f7ce6722740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d633260 0x55848d64b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.156+0000 7f7ce6722740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1b6b40 con 0x55848d633260 2024-04-24T08:19:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.156+0000 7f7ce6722740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1b4410 con 0x55848d628270 2024-04-24T08:19:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.156+0000 7f7ce6722740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1b2280 con 0x55848d628b90 2024-04-24T08:19:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.157+0000 7f7ce1200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628270 0x55848d63eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.157+0000 7f7ce1200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628270 0x55848d63eeb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:52992/0 (socket says 172.21.15.28:52992) 2024-04-24T08:19:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.157+0000 7f7ce1200640 1 -- 172.21.15.28:0/3871304758 learned_addr learned my addr 172.21.15.28:0/3871304758 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.157+0000 7f7ce1c00640 1 --2- 172.21.15.28:0/3871304758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d633260 0x55848d64b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.157+0000 7f7ce1200640 1 -- 172.21.15.28:0/3871304758 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d628b90 msgr2=0x55848d623570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.157+0000 7f7cdbe00640 1 --2- 172.21.15.28:0/3871304758 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d628b90 0x55848d623570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.157+0000 7f7ce1200640 1 --2- 172.21.15.28:0/3871304758 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d628b90 0x55848d623570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.157+0000 7f7ce1200640 1 -- 172.21.15.28:0/3871304758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d633260 msgr2=0x55848d64b210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.157+0000 7f7ce1200640 1 --2- 172.21.15.28:0/3871304758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d633260 0x55848d64b210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.157+0000 7f7ce1200640 1 -- 172.21.15.28:0/3871304758 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55848d62f170 con 0x55848d628270 2024-04-24T08:19:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.157+0000 7f7ce1c00640 1 --2- 172.21.15.28:0/3871304758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d633260 0x55848d64b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.157+0000 7f7cdbe00640 1 --2- 172.21.15.28:0/3871304758 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d628b90 0x55848d623570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.158+0000 7f7ce1200640 1 --2- 172.21.15.28:0/3871304758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628270 0x55848d63eeb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009900 tx=0x7f7cd0005a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=268037eb3dae15b8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:23.161 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.158+0000 7f7ccd600640 1 -- 172.21.15.28:0/3871304758 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd0035040 con 0x55848d628270 2024-04-24T08:19:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.158+0000 7f7ccd600640 1 -- 172.21.15.28:0/3871304758 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cd0054070 con 0x55848d628270 2024-04-24T08:19:23.162 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.158+0000 7f7ccd600640 1 -- 172.21.15.28:0/3871304758 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd00046c0 con 0x55848d628270 2024-04-24T08:19:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.158+0000 7f7ce6722740 1 -- 172.21.15.28:0/3871304758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628270 msgr2=0x55848d63eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.158+0000 7f7ce6722740 1 --2- 172.21.15.28:0/3871304758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628270 0x55848d63eeb0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7cd0009900 tx=0x7f7cd0005a40 comp rx=0 tx=0).stop 2024-04-24T08:19:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.159+0000 7f7ce6722740 1 -- 172.21.15.28:0/3871304758 shutdown_connections 2024-04-24T08:19:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.159+0000 7f7ce6722740 1 --2- 172.21.15.28:0/3871304758 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d633260 0x55848d64b210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.159+0000 7f7ce6722740 1 --2- 172.21.15.28:0/3871304758 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d628b90 0x55848d623570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.159+0000 7f7ce6722740 1 --2- 172.21.15.28:0/3871304758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628270 0x55848d63eeb0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.159+0000 7f7ce6722740 1 -- 172.21.15.28:0/3871304758 >> 172.21.15.28:0/3871304758 conn(0x55848d624590 msgr2=0x55848d622e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.159+0000 7f7ce6722740 1 -- 172.21.15.28:0/3871304758 shutdown_connections 2024-04-24T08:19:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.159+0000 7f7ce6722740 1 -- 172.21.15.28:0/3871304758 wait complete. 2024-04-24T08:19:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.159+0000 7f7ce6722740 1 -- start start 2024-04-24T08:19:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.159+0000 7f7ce6722740 1 Processor -- start 2024-04-24T08:19:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.160+0000 7f7ce6722740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d628270 0x55848d637080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.160+0000 7f7ce6722740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628b90 0x55848d6375c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.160+0000 7f7ce6722740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d633260 0x55848d637b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.160+0000 7f7ce6722740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1b28d0 con 0x55848d628270 2024-04-24T08:19:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.160+0000 7f7ce6722740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1b3d40 con 0x55848d628b90 2024-04-24T08:19:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.160+0000 7f7ce6722740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f7cdc1b2fa0 con 0x55848d633260 2024-04-24T08:19:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.160+0000 7f7ce6722740 10 client.0 _ll_register_callbacks cb 0x7f7cd4005830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.160+0000 7f7ce1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d628270 0x55848d637080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.160+0000 7f7ce1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d628270 0x55848d637080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47872/0 (socket says 172.21.15.28:47872) 2024-04-24T08:19:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.160+0000 7f7ce1200640 1 -- 172.21.15.28:0/4115633674 learned_addr learned my addr 172.21.15.28:0/4115633674 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:23.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.160+0000 7f7ce1c00640 1 --2- 172.21.15.28:0/4115633674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d633260 0x55848d637b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.160+0000 7f7cdbe00640 1 --2- 172.21.15.28:0/4115633674 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628b90 0x55848d6375c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.161+0000 7f7ce1200640 1 -- 172.21.15.28:0/4115633674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d633260 msgr2=0x55848d637b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.161+0000 7f7ce1200640 1 --2- 172.21.15.28:0/4115633674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d633260 0x55848d637b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.161+0000 7f7ce1200640 1 -- 172.21.15.28:0/4115633674 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628b90 msgr2=0x55848d6375c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.161+0000 7f7ce1200640 1 --2- 172.21.15.28:0/4115633674 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628b90 0x55848d6375c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.161+0000 7f7ce1200640 1 -- 172.21.15.28:0/4115633674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55848d6a8d20 con 0x55848d628270 2024-04-24T08:19:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.161+0000 7f7ce1200640 1 --2- 172.21.15.28:0/4115633674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d628270 0x55848d637080 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7cd002a990 tx=0x7f7cd0005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebc0c465ca3c79cc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:23.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.161+0000 7f7cc2a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:19:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.161+0000 7f7cc2a00640 1 -- 172.21.15.28:0/4115633674 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7cd0035030 con 0x55848d628270 2024-04-24T08:19:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.161+0000 7f7cc2a00640 1 -- 172.21.15.28:0/4115633674 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7f7cd0005040 con 0x55848d628270 2024-04-24T08:19:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.161+0000 7f7ce6722740 1 -- 172.21.15.28:0/4115633674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7cd0054070 con 0x55848d628270 2024-04-24T08:19:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.161+0000 7f7cc2a00640 1 -- 172.21.15.28:0/4115633674 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f7cd0005240 con 0x55848d628270 2024-04-24T08:19:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.162+0000 7f7ce6722740 10 client.16545 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.162+0000 7f7ce6722740 1 -- 172.21.15.28:0/4115633674 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55848d6a9fd0 con 0x55848d628270 2024-04-24T08:19:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.162+0000 7f7cc2a00640 1 -- 172.21.15.28:0/4115633674 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7cd0054070 con 0x55848d628270 2024-04-24T08:19:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.162+0000 7f7cc2a00640 1 -- 172.21.15.28:0/4115633674 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f7cd00315b0 con 0x55848d628270 2024-04-24T08:19:23.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.162+0000 7f7ce6722740 10 client.16545 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.162+0000 7f7ce6722740 10 client.16545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:19:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.162+0000 7f7ce6722740 20 client.16545 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.162+0000 7f7ce6722740 1 --2- 172.21.15.28:0/4115633674 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55848d6aa2b0 0x55848d6ac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.162+0000 7f7ce6722740 4 client.16545 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:19:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.162+0000 7f7ce6722740 1 -- 172.21.15.28:0/4115633674 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55848d595700 con 0x55848d6aa2b0 2024-04-24T08:19:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.163+0000 7f7cdbe00640 1 --2- 172.21.15.28:0/4115633674 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55848d6aa2b0 0x55848d6ac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.163+0000 7f7cdbe00640 1 --2- 172.21.15.28:0/4115633674 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55848d6aa2b0 0x55848d6ac6a0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fc25f0564d1137 server_cookie=8eda04d6db81c939 in_seq=0 out_seq=0 2024-04-24T08:19:23.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.163+0000 7f7cc2a00640 10 client.16545 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.164+0000 7f7cc2a00640 1 -- 172.21.15.28:0/4115633674 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55848d595700 con 0x55848d6aa2b0 2024-04-24T08:19:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.164+0000 7f7cc2a00640 10 client.16545 handle_command_reply: tid=0 2024-04-24T08:19:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.164+0000 7f7ce6722740 1 client.16545 shutdown 2024-04-24T08:19:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.165+0000 7f7ce6722740 1 -- 172.21.15.28:0/4115633674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d628270 msgr2=0x55848d637080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.165+0000 7f7ce6722740 1 --2- 172.21.15.28:0/4115633674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d628270 0x55848d637080 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f7cd002a990 tx=0x7f7cd0005de0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.165+0000 7f7ce6722740 1 -- 172.21.15.28:0/4115633674 shutdown_connections 2024-04-24T08:19:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.165+0000 7f7ce6722740 1 --2- 172.21.15.28:0/4115633674 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55848d6aa2b0 0x55848d6ac6a0 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.165+0000 7f7ce6722740 1 --2- 172.21.15.28:0/4115633674 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55848d633260 0x55848d637b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.165+0000 7f7ce6722740 1 --2- 172.21.15.28:0/4115633674 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55848d628b90 0x55848d6375c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.165+0000 7f7ce6722740 1 --2- 172.21.15.28:0/4115633674 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55848d628270 0x55848d637080 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.165+0000 7f7ce6722740 1 -- 172.21.15.28:0/4115633674 >> 172.21.15.28:0/4115633674 conn(0x55848d624590 msgr2=0x55848d630a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.165+0000 7f7cc2a00640 0 client.16545 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.165+0000 7f7ce6722740 1 -- 172.21.15.28:0/4115633674 shutdown_connections 2024-04-24T08:19:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.165+0000 7f7ce6722740 1 -- 172.21.15.28:0/4115633674 wait complete. 2024-04-24T08:19:23.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.165+0000 7f7ce6722740 20 client.16545 trim_cache size 0 max 16384 2024-04-24T08:19:23.174 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:19:23.174 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-24T08:19:23.174 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "e972ed18-e5d0-489f-9bf5-ce5dc2ab35b4", 2024-04-24T08:19:23.174 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-24T08:19:23.174 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:19:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.166+0000 7f7ce2600640 1 -- 172.21.15.28:0/3013223456 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7cc40a3d20 msgr2=0x7f7cc40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.166+0000 7f7ce2600640 1 --2- 172.21.15.28:0/3013223456 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7cc40a3d20 0x7f7cc40a61e0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x55848d625720 tx=0x7f7cc8009c00 comp rx=0 tx=0).stop 2024-04-24T08:19:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.166+0000 7f7ce2600640 1 -- 172.21.15.28:0/3013223456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cdc1062c0 msgr2=0x7f7cdc1a36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.166+0000 7f7ce2600640 1 --2- 172.21.15.28:0/3013223456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cdc1062c0 0x7f7cdc1a36b0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f7cd4009fd0 tx=0x7f7cd400ef10 comp rx=0 tx=0).stop 2024-04-24T08:19:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.166+0000 7f7ce2600640 1 -- 172.21.15.28:0/3013223456 shutdown_connections 2024-04-24T08:19:23.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.166+0000 7f7ce2600640 1 --2- 172.21.15.28:0/3013223456 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f7cc40a3d20 0x7f7cc40a61e0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.166+0000 7f7ce2600640 1 --2- 172.21.15.28:0/3013223456 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f7cdc1017e0 0x7f7cdc19f330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.166+0000 7f7ce2600640 1 --2- 172.21.15.28:0/3013223456 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f7cdc1062c0 0x7f7cdc1a36b0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.166+0000 7f7ce2600640 1 --2- 172.21.15.28:0/3013223456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cdc100ec0 0x7f7cdc19edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.166+0000 7f7ce2600640 1 -- 172.21.15.28:0/3013223456 >> 172.21.15.28:0/3013223456 conn(0x7f7cdc0fc910 msgr2=0x7f7cdc0fd970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:23.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.167+0000 7f7ce2600640 1 -- 172.21.15.28:0/3013223456 shutdown_connections 2024-04-24T08:19:23.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.167+0000 7f7ce2600640 1 -- 172.21.15.28:0/3013223456 wait complete. 2024-04-24T08:19:23.197 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:19:23.197 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:19:23.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.396+0000 7fb789800640 1 -- 172.21.15.28:0/4040780343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb784104cd0 msgr2=0x7fb784108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.398 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.396+0000 7fb789800640 1 --2- 172.21.15.28:0/4040780343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb784104cd0 0x7fb784108d10 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fb76c009900 tx=0x7fb76c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.397+0000 7fb789800640 1 -- 172.21.15.28:0/4040780343 shutdown_connections 2024-04-24T08:19:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.397+0000 7fb789800640 1 --2- 172.21.15.28:0/4040780343 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb784104cd0 0x7fb784108d10 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.397+0000 7fb789800640 1 --2- 172.21.15.28:0/4040780343 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb7841093f0 0x7fb784110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.397+0000 7fb789800640 1 --2- 172.21.15.28:0/4040780343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb784104320 0x7fb784104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.397+0000 7fb789800640 1 -- 172.21.15.28:0/4040780343 >> 172.21.15.28:0/4040780343 conn(0x7fb7840ffb40 msgr2=0x7fb784101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.397+0000 7fb789800640 1 -- 172.21.15.28:0/4040780343 shutdown_connections 2024-04-24T08:19:23.399 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.397+0000 7fb789800640 1 -- 172.21.15.28:0/4040780343 wait complete. 2024-04-24T08:19:23.401 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.398+0000 7fb789800640 1 Processor -- start 2024-04-24T08:19:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.398+0000 7fb789800640 1 -- start start 2024-04-24T08:19:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.398+0000 7fb789800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb784104320 0x7fb7841a0650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.398+0000 7fb789800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7841093f0 0x7fb7841a0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.398+0000 7fb789800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78419a810 0x7fb78419acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.398+0000 7fb789800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7841144b0 con 0x7fb78419a810 2024-04-24T08:19:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.398+0000 7fb789800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb784114330 con 0x7fb7841093f0 2024-04-24T08:19:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.398+0000 7fb789800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb784114630 con 0x7fb784104320 2024-04-24T08:19:23.402 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.399+0000 7fb783e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb784104320 0x7fb7841a0650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.399+0000 7fb788e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78419a810 0x7fb78419acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.399+0000 7fb783e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb784104320 0x7fb7841a0650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44136/0 (socket says 172.21.15.28:44136) 2024-04-24T08:19:23.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.399+0000 7fb783e00640 1 -- 172.21.15.28:0/2873331014 learned_addr learned my addr 172.21.15.28:0/2873331014 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:23.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.399+0000 7fb77be00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7841093f0 0x7fb7841a0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.399+0000 7fb77be00640 1 -- 172.21.15.28:0/2873331014 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb784104320 msgr2=0x7fb7841a0650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.399+0000 7fb77be00640 1 --2- 172.21.15.28:0/2873331014 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb784104320 0x7fb7841a0650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.399+0000 7fb77be00640 1 -- 172.21.15.28:0/2873331014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78419a810 msgr2=0x7fb78419acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.399+0000 7fb77be00640 1 --2- 172.21.15.28:0/2873331014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78419a810 0x7fb78419acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.399+0000 7fb77be00640 1 -- 172.21.15.28:0/2873331014 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb78419b3a0 con 0x7fb7841093f0 2024-04-24T08:19:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.399+0000 7fb788e00640 1 --2- 172.21.15.28:0/2873331014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78419a810 0x7fb78419acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.400+0000 7fb77be00640 1 --2- 172.21.15.28:0/2873331014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7841093f0 0x7fb7841a0b90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb76c02a910 tx=0x7fb76c003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.400+0000 7fb781600640 1 -- 172.21.15.28:0/2873331014 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c03f070 con 0x7fb7841093f0 2024-04-24T08:19:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.400+0000 7fb781600640 1 -- 172.21.15.28:0/2873331014 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb76c031c90 con 0x7fb7841093f0 2024-04-24T08:19:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.400+0000 7fb789800640 1 -- 172.21.15.28:0/2873331014 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb78419b630 con 0x7fb7841093f0 2024-04-24T08:19:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.400+0000 7fb781600640 1 -- 172.21.15.28:0/2873331014 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb76c035030 con 0x7fb7841093f0 2024-04-24T08:19:23.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.400+0000 7fb789800640 1 -- 172.21.15.28:0/2873331014 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb7841a7790 con 0x7fb7841093f0 2024-04-24T08:19:23.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.403+0000 7fb78dc53740 1 Processor -- start 2024-04-24T08:19:23.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.403+0000 7fb781600640 1 -- 172.21.15.28:0/2873331014 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fb76c042070 con 0x7fb7841093f0 2024-04-24T08:19:23.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.403+0000 7fb78dc53740 1 -- start start 2024-04-24T08:19:23.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.404+0000 7fb78dc53740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774195a30 0x55e774195e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.404+0000 7fb78dc53740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e774196350 0x55e7741a04e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.404+0000 7fb78dc53740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7741a0a20 0x55e7741a2e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.404+0000 7fb78dc53740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e7740fb1f0 con 0x55e7741a0a20 2024-04-24T08:19:23.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.404+0000 7fb78dc53740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e7740a1350 con 0x55e774195a30 2024-04-24T08:19:23.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.404+0000 7fb78dc53740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e7740f71b0 con 0x55e774196350 2024-04-24T08:19:23.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.404+0000 7fb788e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774195a30 0x55e774195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.404+0000 7fb77be00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7741a0a20 0x55e7741a2e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.404+0000 7fb788e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774195a30 0x55e774195e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:53022/0 (socket says 172.21.15.28:53022) 2024-04-24T08:19:23.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.404+0000 7fb788e00640 1 -- 172.21.15.28:0/1331029405 learned_addr learned my addr 172.21.15.28:0/1331029405 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:23.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.404+0000 7fb783e00640 1 --2- 172.21.15.28:0/1331029405 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e774196350 0x55e7741a04e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.405+0000 7fb788e00640 1 -- 172.21.15.28:0/1331029405 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e774196350 msgr2=0x55e7741a04e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:23.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.405+0000 7fb788e00640 1 --2- 172.21.15.28:0/1331029405 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e774196350 0x55e7741a04e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.405+0000 7fb788e00640 1 -- 172.21.15.28:0/1331029405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7741a0a20 msgr2=0x55e7741a2e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:23.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.405+0000 7fb788e00640 1 --2- 172.21.15.28:0/1331029405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7741a0a20 0x55e7741a2e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.405+0000 7fb788e00640 1 -- 172.21.15.28:0/1331029405 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7740f5f50 con 0x55e774195a30 2024-04-24T08:19:23.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.405+0000 7fb77be00640 1 --2- 172.21.15.28:0/1331029405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7741a0a20 0x55e7741a2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:23.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.405+0000 7fb788e00640 1 --2- 172.21.15.28:0/1331029405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774195a30 0x55e774195e10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb77c011790 tx=0x7fb77c011c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fbdbe856557940fc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:23.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.405+0000 7fb779600640 1 -- 172.21.15.28:0/1331029405 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c003c30 con 0x55e774195a30 2024-04-24T08:19:23.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.405+0000 7fb779600640 1 -- 172.21.15.28:0/1331029405 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb77c00fe80 con 0x55e774195a30 2024-04-24T08:19:23.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.405+0000 7fb779600640 1 -- 172.21.15.28:0/1331029405 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c0193e0 con 0x55e774195a30 2024-04-24T08:19:23.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.406+0000 7fb781600640 1 --2- 172.21.15.28:0/2873331014 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb7680a3d20 0x7fb7680a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.406+0000 7fb781600640 1 -- 172.21.15.28:0/2873331014 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7fb76c0f1150 con 0x7fb7841093f0 2024-04-24T08:19:23.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.406+0000 7fb783e00640 1 --2- 172.21.15.28:0/2873331014 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb7680a3d20 0x7fb7680a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.414 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.406+0000 7fb78dc53740 1 -- 172.21.15.28:0/1331029405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774195a30 msgr2=0x55e774195e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.406+0000 7fb78dc53740 1 --2- 172.21.15.28:0/1331029405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774195a30 0x55e774195e10 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7fb77c011790 tx=0x7fb77c011c50 comp rx=0 tx=0).stop 2024-04-24T08:19:23.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.406+0000 7fb78dc53740 1 -- 172.21.15.28:0/1331029405 shutdown_connections 2024-04-24T08:19:23.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.406+0000 7fb78dc53740 1 --2- 172.21.15.28:0/1331029405 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7741a0a20 0x55e7741a2e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.406+0000 7fb78dc53740 1 --2- 172.21.15.28:0/1331029405 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e774196350 0x55e7741a04e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.406+0000 7fb78dc53740 1 --2- 172.21.15.28:0/1331029405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774195a30 0x55e774195e10 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.406+0000 7fb78dc53740 1 -- 172.21.15.28:0/1331029405 >> 172.21.15.28:0/1331029405 conn(0x55e774191d50 msgr2=0x55e774192130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:23.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.406+0000 7fb78dc53740 1 -- 172.21.15.28:0/1331029405 shutdown_connections 2024-04-24T08:19:23.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.406+0000 7fb78dc53740 1 -- 172.21.15.28:0/1331029405 wait complete. 2024-04-24T08:19:23.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.407+0000 7fb783e00640 1 --2- 172.21.15.28:0/2873331014 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb7680a3d20 0x7fb7680a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x55e774196ff0 tx=0x7fb774005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:23.415 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.407+0000 7fb78dc53740 1 -- start start 2024-04-24T08:19:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.407+0000 7fb78dc53740 1 Processor -- start 2024-04-24T08:19:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.408+0000 7fb78dc53740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e774195a30 0x55e77419d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.408+0000 7fb78dc53740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774196350 0x55e77419df30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.408+0000 7fb78dc53740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7741a0a20 0x55e77419e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.408+0000 7fb78dc53740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55e7741a3810 con 0x55e7741a0a20 2024-04-24T08:19:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.408+0000 7fb78dc53740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55e7740b4690 con 0x55e774196350 2024-04-24T08:19:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.408+0000 7fb78dc53740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55e7741a3990 con 0x55e774195a30 2024-04-24T08:19:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.408+0000 7fb78dc53740 10 client.0 _ll_register_callbacks cb 0x55e7740f0650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:23.416 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.408+0000 7fb788e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7741a0a20 0x55e77419e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.408+0000 7fb788e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7741a0a20 0x55e77419e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47896/0 (socket says 172.21.15.28:47896) 2024-04-24T08:19:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.408+0000 7fb788e00640 1 -- 172.21.15.28:0/3044942042 learned_addr learned my addr 172.21.15.28:0/3044942042 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.408+0000 7fb783e00640 1 --2- 172.21.15.28:0/3044942042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e774195a30 0x55e77419d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.408+0000 7fb77be00640 1 --2- 172.21.15.28:0/3044942042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774196350 0x55e77419df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.408+0000 7fb788e00640 1 -- 172.21.15.28:0/3044942042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e774195a30 msgr2=0x55e77419d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.408+0000 7fb788e00640 1 --2- 172.21.15.28:0/3044942042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e774195a30 0x55e77419d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.408+0000 7fb788e00640 1 -- 172.21.15.28:0/3044942042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774196350 msgr2=0x55e77419df30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.408+0000 7fb788e00640 1 --2- 172.21.15.28:0/3044942042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774196350 0x55e77419df30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.408+0000 7fb788e00640 1 -- 172.21.15.28:0/3044942042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7740b3920 con 0x55e7741a0a20 2024-04-24T08:19:23.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.409+0000 7fb788e00640 1 --2- 172.21.15.28:0/3044942042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7741a0a20 0x55e77419e470 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb77c005f40 tx=0x7fb77c00fe10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9777f7042ca5c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:23.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.409+0000 7fb772000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:19:23.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.409+0000 7fb772000640 1 -- 172.21.15.28:0/3044942042 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77c01f040 con 0x55e7741a0a20 2024-04-24T08:19:23.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.409+0000 7fb772000640 1 -- 172.21.15.28:0/3044942042 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7fb77c011340 con 0x55e7741a0a20 2024-04-24T08:19:23.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.409+0000 7fb78dc53740 1 -- 172.21.15.28:0/3044942042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55e774108c10 con 0x55e7741a0a20 2024-04-24T08:19:23.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.409+0000 7fb772000640 1 -- 172.21.15.28:0/3044942042 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb77c032050 con 0x55e7741a0a20 2024-04-24T08:19:23.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.409+0000 7fb78dc53740 10 client.16557 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:23.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.409+0000 7fb78dc53740 1 -- 172.21.15.28:0/3044942042 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e773ed77d0 con 0x55e7741a0a20 2024-04-24T08:19:23.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.410+0000 7fb772000640 1 -- 172.21.15.28:0/3044942042 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55e774108c10 con 0x55e7741a0a20 2024-04-24T08:19:23.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.410+0000 7fb772000640 1 -- 172.21.15.28:0/3044942042 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fb77c018070 con 0x55e7741a0a20 2024-04-24T08:19:23.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.410+0000 7fb78dc53740 10 client.16557 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:23.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.410+0000 7fb78dc53740 10 client.16557 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:19:23.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.410+0000 7fb78dc53740 20 client.16557 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:23.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.410+0000 7fb78dc53740 1 --2- 172.21.15.28:0/3044942042 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55e7741b0d90 0x55e7741b3180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.410+0000 7fb78dc53740 4 client.16557 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:19:23.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.410+0000 7fb78dc53740 1 -- 172.21.15.28:0/3044942042 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e774102a80 con 0x55e7741b0d90 2024-04-24T08:19:23.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.410+0000 7fb783e00640 1 --2- 172.21.15.28:0/3044942042 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55e7741b0d90 0x55e7741b3180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.411+0000 7fb783e00640 1 --2- 172.21.15.28:0/3044942042 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55e7741b0d90 0x55e7741b3180 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=559e29472ed8e397 server_cookie=e632eb6e1935160f in_seq=0 out_seq=0 2024-04-24T08:19:23.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.411+0000 7fb772000640 10 client.16557 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:23.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.412+0000 7fb772000640 1 -- 172.21.15.28:0/3044942042 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e774102a80 con 0x55e7741b0d90 2024-04-24T08:19:23.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.412+0000 7fb772000640 10 client.16557 handle_command_reply: tid=0 2024-04-24T08:19:23.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.412+0000 7fb78dc53740 1 client.16557 shutdown 2024-04-24T08:19:23.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.412+0000 7fb78dc53740 1 -- 172.21.15.28:0/3044942042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7741a0a20 msgr2=0x55e77419e470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.412+0000 7fb78dc53740 1 --2- 172.21.15.28:0/3044942042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7741a0a20 0x55e77419e470 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7fb77c005f40 tx=0x7fb77c00fe10 comp rx=0 tx=0).stop 2024-04-24T08:19:23.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.412+0000 7fb78dc53740 1 -- 172.21.15.28:0/3044942042 shutdown_connections 2024-04-24T08:19:23.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.412+0000 7fb78dc53740 1 --2- 172.21.15.28:0/3044942042 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55e7741b0d90 0x55e7741b3180 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.412+0000 7fb78dc53740 1 --2- 172.21.15.28:0/3044942042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e7741a0a20 0x55e77419e470 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.412+0000 7fb78dc53740 1 --2- 172.21.15.28:0/3044942042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774196350 0x55e77419df30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.412+0000 7fb772000640 0 client.16557 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.412+0000 7fb78dc53740 1 --2- 172.21.15.28:0/3044942042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e774195a30 0x55e77419d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.412+0000 7fb78dc53740 1 -- 172.21.15.28:0/3044942042 >> 172.21.15.28:0/3044942042 conn(0x55e774191d50 msgr2=0x55e774197b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.412+0000 7fb78dc53740 1 -- 172.21.15.28:0/3044942042 shutdown_connections 2024-04-24T08:19:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.412+0000 7fb78dc53740 1 -- 172.21.15.28:0/3044942042 wait complete. 2024-04-24T08:19:23.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.412+0000 7fb78dc53740 20 client.16557 trim_cache size 0 max 16384 2024-04-24T08:19:23.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.432+0000 7fb78dc53740 1 Processor -- start 2024-04-24T08:19:23.433 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.432+0000 7fb78dc53740 1 -- start start 2024-04-24T08:19:23.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.432+0000 7fb78dc53740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e774195a30 0x55e7741ac6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.432+0000 7fb78dc53740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e774196350 0x55e77419e2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.432+0000 7fb78dc53740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7741a0a20 0x55e7741b88f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.432+0000 7fb78dc53740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7841b27a0 con 0x55e774195a30 2024-04-24T08:19:23.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.432+0000 7fb78dc53740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb7841b0070 con 0x55e7741a0a20 2024-04-24T08:19:23.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.432+0000 7fb78dc53740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb7841adee0 con 0x55e774196350 2024-04-24T08:19:23.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.433+0000 7fb783e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e774195a30 0x55e7741ac6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.433+0000 7fb783e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e774195a30 0x55e7741ac6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47904/0 (socket says 172.21.15.28:47904) 2024-04-24T08:19:23.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.433+0000 7fb783e00640 1 -- 172.21.15.28:0/2431247603 learned_addr learned my addr 172.21.15.28:0/2431247603 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:23.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.433+0000 7fb788e00640 1 --2- 172.21.15.28:0/2431247603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7741a0a20 0x55e7741b88f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.433+0000 7fb77be00640 1 --2- 172.21.15.28:0/2431247603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e774196350 0x55e77419e2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.433+0000 7fb783e00640 1 -- 172.21.15.28:0/2431247603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e774196350 msgr2=0x55e77419e2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.433+0000 7fb783e00640 1 --2- 172.21.15.28:0/2431247603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e774196350 0x55e77419e2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.433+0000 7fb783e00640 1 -- 172.21.15.28:0/2431247603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7741a0a20 msgr2=0x55e7741b88f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.433+0000 7fb783e00640 1 --2- 172.21.15.28:0/2431247603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7741a0a20 0x55e7741b88f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.433+0000 7fb783e00640 1 -- 172.21.15.28:0/2431247603 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e773e0d3f0 con 0x55e774195a30 2024-04-24T08:19:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.433+0000 7fb783e00640 1 --2- 172.21.15.28:0/2431247603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e774195a30 0x55e7741ac6d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x55e7741b3890 tx=0x7fb77401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a731072feadc5c90 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.433+0000 7fb778c00640 1 -- 172.21.15.28:0/2431247603 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77402b470 con 0x55e774195a30 2024-04-24T08:19:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.434+0000 7fb778c00640 1 -- 172.21.15.28:0/2431247603 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb774030070 con 0x55e774195a30 2024-04-24T08:19:23.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.434+0000 7fb778c00640 1 -- 172.21.15.28:0/2431247603 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77401e420 con 0x55e774195a30 2024-04-24T08:19:23.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.434+0000 7fb78dc53740 1 -- 172.21.15.28:0/2431247603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e774195a30 msgr2=0x55e7741ac6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.434+0000 7fb78dc53740 1 --2- 172.21.15.28:0/2431247603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e774195a30 0x55e7741ac6d0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x55e7741b3890 tx=0x7fb77401b000 comp rx=0 tx=0).stop 2024-04-24T08:19:23.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.434+0000 7fb78dc53740 1 -- 172.21.15.28:0/2431247603 shutdown_connections 2024-04-24T08:19:23.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.434+0000 7fb78dc53740 1 --2- 172.21.15.28:0/2431247603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e7741a0a20 0x55e7741b88f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.434+0000 7fb78dc53740 1 --2- 172.21.15.28:0/2431247603 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e774196350 0x55e77419e2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.434+0000 7fb78dc53740 1 --2- 172.21.15.28:0/2431247603 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e774195a30 0x55e7741ac6d0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.434+0000 7fb78dc53740 1 -- 172.21.15.28:0/2431247603 >> 172.21.15.28:0/2431247603 conn(0x55e774191d50 msgr2=0x55e77419f220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:23.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.434+0000 7fb78dc53740 1 -- 172.21.15.28:0/2431247603 shutdown_connections 2024-04-24T08:19:23.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.435+0000 7fb78dc53740 1 -- 172.21.15.28:0/2431247603 wait complete. 2024-04-24T08:19:23.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.435+0000 7fb78dc53740 1 -- start start 2024-04-24T08:19:23.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.435+0000 7fb78dc53740 1 Processor -- start 2024-04-24T08:19:23.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.435+0000 7fb78dc53740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e774195a30 0x55e7741a8be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.435+0000 7fb78dc53740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774196350 0x55e7741a9120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.436+0000 7fb78dc53740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7741a0a20 0x55e77419a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.436+0000 7fb78dc53740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fb7841ae530 con 0x55e774195a30 2024-04-24T08:19:23.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.436+0000 7fb78dc53740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fb7841af9a0 con 0x55e774196350 2024-04-24T08:19:23.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.436+0000 7fb78dc53740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fb7841aec00 con 0x55e7741a0a20 2024-04-24T08:19:23.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.436+0000 7fb78dc53740 10 client.0 _ll_register_callbacks cb 0x7fb774009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:23.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.436+0000 7fb783e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e774195a30 0x55e7741a8be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.436+0000 7fb783e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e774195a30 0x55e7741a8be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:47918/0 (socket says 172.21.15.28:47918) 2024-04-24T08:19:23.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.436+0000 7fb783e00640 1 -- 172.21.15.28:0/4009535158 learned_addr learned my addr 172.21.15.28:0/4009535158 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:23.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.436+0000 7fb77be00640 1 --2- 172.21.15.28:0/4009535158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774196350 0x55e7741a9120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.436+0000 7fb788e00640 1 --2- 172.21.15.28:0/4009535158 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7741a0a20 0x55e77419a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.436+0000 7fb783e00640 1 -- 172.21.15.28:0/4009535158 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7741a0a20 msgr2=0x55e77419a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:23.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.436+0000 7fb783e00640 1 --2- 172.21.15.28:0/4009535158 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7741a0a20 0x55e77419a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.436+0000 7fb783e00640 1 -- 172.21.15.28:0/4009535158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774196350 msgr2=0x55e7741a9120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:23.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.436+0000 7fb783e00640 1 --2- 172.21.15.28:0/4009535158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774196350 0x55e7741a9120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.436+0000 7fb783e00640 1 -- 172.21.15.28:0/4009535158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e773dcd0f0 con 0x55e774195a30 2024-04-24T08:19:23.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.436+0000 7fb77be00640 1 --2- 172.21.15.28:0/4009535158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774196350 0x55e7741a9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:23.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.437+0000 7fb783e00640 1 --2- 172.21.15.28:0/4009535158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e774195a30 0x55e7741a8be0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb774004230 tx=0x7fb774004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6ea152c2883b3c6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:23.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.437+0000 7fb772000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:19:23.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.437+0000 7fb772000640 1 -- 172.21.15.28:0/4009535158 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb77402b470 con 0x55e774195a30 2024-04-24T08:19:23.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.437+0000 7fb772000640 1 -- 172.21.15.28:0/4009535158 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7fb77402b610 con 0x55e774195a30 2024-04-24T08:19:23.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.437+0000 7fb78dc53740 1 -- 172.21.15.28:0/4009535158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb774030070 con 0x55e774195a30 2024-04-24T08:19:23.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.437+0000 7fb772000640 1 -- 172.21.15.28:0/4009535158 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fb77402b810 con 0x55e774195a30 2024-04-24T08:19:23.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.437+0000 7fb78dc53740 10 client.16563 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:23.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.437+0000 7fb78dc53740 1 -- 172.21.15.28:0/4009535158 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55e7740c0810 con 0x55e774195a30 2024-04-24T08:19:23.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.437+0000 7fb772000640 1 -- 172.21.15.28:0/4009535158 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb774030070 con 0x55e774195a30 2024-04-24T08:19:23.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.438+0000 7fb772000640 1 -- 172.21.15.28:0/4009535158 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fb7740046c0 con 0x55e774195a30 2024-04-24T08:19:23.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.438+0000 7fb78dc53740 10 client.16563 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:23.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.438+0000 7fb78dc53740 10 client.16563 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:19:23.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.438+0000 7fb78dc53740 20 client.16563 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:23.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.438+0000 7fb78dc53740 1 --2- 172.21.15.28:0/4009535158 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55e774217010 0x55e774219430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:23.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.438+0000 7fb78dc53740 4 client.16563 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:19:23.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.438+0000 7fb78dc53740 1 -- 172.21.15.28:0/4009535158 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55e77407e360 con 0x55e774217010 2024-04-24T08:19:23.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.438+0000 7fb77be00640 1 --2- 172.21.15.28:0/4009535158 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55e774217010 0x55e774219430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:23.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.439+0000 7fb77be00640 1 --2- 172.21.15.28:0/4009535158 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55e774217010 0x55e774219430 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48ff46dabb0d9c22 server_cookie=e4dc331fd5d3a27e in_seq=0 out_seq=0 2024-04-24T08:19:23.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.439+0000 7fb772000640 10 client.16563 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:23.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.439+0000 7fb772000640 1 -- 172.21.15.28:0/4009535158 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55e77407e360 con 0x55e774217010 2024-04-24T08:19:23.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.439+0000 7fb772000640 10 client.16563 handle_command_reply: tid=0 2024-04-24T08:19:23.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.439+0000 7fb78dc53740 1 client.16563 shutdown 2024-04-24T08:19:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.439+0000 7fb78dc53740 1 -- 172.21.15.28:0/4009535158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e774195a30 msgr2=0x55e7741a8be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.439+0000 7fb78dc53740 1 --2- 172.21.15.28:0/4009535158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e774195a30 0x55e7741a8be0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fb774004230 tx=0x7fb774004360 comp rx=0 tx=0).stop 2024-04-24T08:19:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.439+0000 7fb78dc53740 1 -- 172.21.15.28:0/4009535158 shutdown_connections 2024-04-24T08:19:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.439+0000 7fb78dc53740 1 --2- 172.21.15.28:0/4009535158 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55e774217010 0x55e774219430 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.439+0000 7fb78dc53740 1 --2- 172.21.15.28:0/4009535158 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55e7741a0a20 0x55e77419a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.439+0000 7fb78dc53740 1 --2- 172.21.15.28:0/4009535158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55e774196350 0x55e7741a9120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.439+0000 7fb78dc53740 1 --2- 172.21.15.28:0/4009535158 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55e774195a30 0x55e7741a8be0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.439+0000 7fb772000640 0 client.16563 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:23.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.439+0000 7fb78dc53740 1 -- 172.21.15.28:0/4009535158 >> 172.21.15.28:0/4009535158 conn(0x55e774191d50 msgr2=0x55e77419fab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:23.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.439+0000 7fb78dc53740 1 -- 172.21.15.28:0/4009535158 shutdown_connections 2024-04-24T08:19:23.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.439+0000 7fb78dc53740 1 -- 172.21.15.28:0/4009535158 wait complete. 2024-04-24T08:19:23.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.440+0000 7fb78dc53740 20 client.16563 trim_cache size 0 max 16384 2024-04-24T08:19:23.449 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:19:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:19:23.449 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:19:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "e972ed18-e5d0-489f-9bf5-ce5dc2ab35b4": { 2024-04-24T08:19:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:19:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "e972ed18-e5d0-489f-9bf5-ce5dc2ab35b4", 2024-04-24T08:19:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:19:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:19:23.450 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:19:23.450 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:19:23.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.440+0000 7fb789800640 1 -- 172.21.15.28:0/2873331014 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb7680a3d20 msgr2=0x7fb7680a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.440+0000 7fb789800640 1 --2- 172.21.15.28:0/2873331014 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb7680a3d20 0x7fb7680a61e0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x55e774196ff0 tx=0x7fb774005f90 comp rx=0 tx=0).stop 2024-04-24T08:19:23.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.440+0000 7fb789800640 1 -- 172.21.15.28:0/2873331014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7841093f0 msgr2=0x7fb7841a0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:23.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.440+0000 7fb789800640 1 --2- 172.21.15.28:0/2873331014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7841093f0 0x7fb7841a0b90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fb76c02a910 tx=0x7fb76c003b20 comp rx=0 tx=0).stop 2024-04-24T08:19:23.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.441+0000 7fb789800640 1 -- 172.21.15.28:0/2873331014 shutdown_connections 2024-04-24T08:19:23.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.441+0000 7fb789800640 1 --2- 172.21.15.28:0/2873331014 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fb7680a3d20 0x7fb7680a61e0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.441+0000 7fb789800640 1 --2- 172.21.15.28:0/2873331014 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fb78419a810 0x7fb78419acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.441+0000 7fb789800640 1 --2- 172.21.15.28:0/2873331014 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7841093f0 0x7fb7841a0b90 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.441+0000 7fb789800640 1 --2- 172.21.15.28:0/2873331014 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fb784104320 0x7fb7841a0650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:23.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.441+0000 7fb789800640 1 -- 172.21.15.28:0/2873331014 >> 172.21.15.28:0/2873331014 conn(0x7fb7840ffb40 msgr2=0x7fb784100f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:23.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.441+0000 7fb789800640 1 -- 172.21.15.28:0/2873331014 shutdown_connections 2024-04-24T08:19:23.453 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:23.441+0000 7fb789800640 1 -- 172.21.15.28:0/2873331014 wait complete. 2024-04-24T08:19:23.461 INFO:tasks.cephfs.filesystem:scrub status for tag:e972ed18-e5d0-489f-9bf5-ce5dc2ab35b4 - {'path': '/', 'tag': 'e972ed18-e5d0-489f-9bf5-ce5dc2ab35b4', 'options': 'recursive,force'} 2024-04-24T08:19:23.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:23 smithi117 ceph-mon[32438]: pgmap v280: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:19:23.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:23 smithi117 ceph-mon[32438]: scrub queued for path: / 2024-04-24T08:19:23.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:23 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:23.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:23 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:19:23.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:23 smithi194 ceph-mon[32836]: pgmap v280: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:19:23.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:23 smithi194 ceph-mon[32836]: scrub queued for path: / 2024-04-24T08:19:23.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:23 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:23.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:23 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:19:24.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:23 smithi028 ceph-mon[27786]: pgmap v280: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:19:24.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:23 smithi028 ceph-mon[27786]: scrub queued for path: / 2024-04-24T08:19:24.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:23 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:24.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:23 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:19:24.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:24 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:19:24.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:24 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:24.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:24 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:19:24.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:24 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:24.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:24 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:19:24.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:24 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:24.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:24 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:24.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:24 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:19:24.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:24 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:24.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:24 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:19:24.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:24 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:24.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:24 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:19:24.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:24 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:24.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:24 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:25.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:24 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:19:25.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:24 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:25.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:24 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:19:25.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:24 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:25.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:24 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:19:25.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:24 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:25.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:24 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:25.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:25 smithi117 ceph-mon[32438]: pgmap v281: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:19:25.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:25 smithi194 ceph-mon[32836]: pgmap v281: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:19:26.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:25 smithi028 ceph-mon[27786]: pgmap v281: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:19:27.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:27 smithi117 ceph-mon[32438]: pgmap v282: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:19:27.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:27 smithi194 ceph-mon[32836]: pgmap v282: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:19:28.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:27 smithi028 ceph-mon[27786]: pgmap v282: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 102 B/s wr, 6 op/s 2024-04-24T08:19:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:29 smithi117 ceph-mon[32438]: pgmap v283: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-24T08:19:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:29 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:19:29.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:29 smithi194 ceph-mon[32836]: pgmap v283: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-24T08:19:29.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:29 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:19:30.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:29 smithi028 ceph-mon[27786]: pgmap v283: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-24T08:19:30.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:29 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:19:30.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:30 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:19:30.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:30 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:19:31.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:30 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:19:31.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:31 smithi117 ceph-mon[32438]: pgmap v284: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-24T08:19:31.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:31 smithi194 ceph-mon[32836]: pgmap v284: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-24T08:19:32.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:31 smithi028 ceph-mon[27786]: pgmap v284: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-24T08:19:33.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:33 smithi194 ceph-mon[32836]: pgmap v285: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-24T08:19:34.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:33 smithi028 ceph-mon[27786]: pgmap v285: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-24T08:19:34.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:33 smithi117 ceph-mon[32438]: pgmap v285: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-04-24T08:19:35.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:35 smithi194 ceph-mon[32836]: pgmap v286: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-24T08:19:36.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:35 smithi028 ceph-mon[27786]: pgmap v286: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-24T08:19:36.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:35 smithi117 ceph-mon[32438]: pgmap v286: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-24T08:19:37.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:37 smithi194 ceph-mon[32836]: pgmap v287: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-24T08:19:38.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:37 smithi028 ceph-mon[27786]: pgmap v287: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-24T08:19:38.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:37 smithi117 ceph-mon[32438]: pgmap v287: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-04-24T08:19:39.930 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:39 smithi117 ceph-mon[32438]: pgmap v288: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-04-24T08:19:39.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:39 smithi194 ceph-mon[32836]: pgmap v288: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-04-24T08:19:40.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:39 smithi028 ceph-mon[27786]: pgmap v288: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-04-24T08:19:41.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:41 smithi194 ceph-mon[32836]: pgmap v289: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:19:42.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:41 smithi028 ceph-mon[27786]: pgmap v289: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:19:42.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:41 smithi117 ceph-mon[32438]: pgmap v289: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:19:43.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:43 smithi194 ceph-mon[32836]: pgmap v290: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-24T08:19:44.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:43 smithi028 ceph-mon[27786]: pgmap v290: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-24T08:19:44.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:43 smithi117 ceph-mon[32438]: pgmap v290: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-24T08:19:45.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:45 smithi194 ceph-mon[32836]: pgmap v291: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-24T08:19:46.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:45 smithi028 ceph-mon[27786]: pgmap v291: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-24T08:19:46.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:45 smithi117 ceph-mon[32438]: pgmap v291: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-24T08:19:47.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:47 smithi194 ceph-mon[32836]: pgmap v292: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-24T08:19:48.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:47 smithi028 ceph-mon[27786]: pgmap v292: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-24T08:19:48.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:47 smithi117 ceph-mon[32438]: pgmap v292: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-04-24T08:19:49.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:49 smithi194 ceph-mon[32836]: pgmap v293: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-04-24T08:19:50.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:49 smithi028 ceph-mon[27786]: pgmap v293: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-04-24T08:19:50.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:49 smithi117 ceph-mon[32438]: pgmap v293: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-04-24T08:19:51.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:51 smithi194 ceph-mon[32836]: pgmap v294: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-24T08:19:52.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:51 smithi028 ceph-mon[27786]: pgmap v294: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-24T08:19:52.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:51 smithi117 ceph-mon[32438]: pgmap v294: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-24T08:19:53.463 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:19:53.463 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:19:53.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.690+0000 7fa997e00640 1 -- 172.21.15.28:0/2573894643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa998102110 msgr2=0x7fa9981024f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.690+0000 7fa997e00640 1 --2- 172.21.15.28:0/2573894643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa998102110 0x7fa9981024f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7fa988009900 tx=0x7fa98802a410 comp rx=0 tx=0).stop 2024-04-24T08:19:53.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.690+0000 7fa997e00640 1 -- 172.21.15.28:0/2573894643 shutdown_connections 2024-04-24T08:19:53.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.690+0000 7fa997e00640 1 --2- 172.21.15.28:0/2573894643 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9981071e0 0x7fa99810edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.690+0000 7fa997e00640 1 --2- 172.21.15.28:0/2573894643 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa998102ac0 0x7fa998106b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.690+0000 7fa997e00640 1 --2- 172.21.15.28:0/2573894643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa998102110 0x7fa9981024f0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.690+0000 7fa997e00640 1 -- 172.21.15.28:0/2573894643 >> 172.21.15.28:0/2573894643 conn(0x7fa9980fd9d0 msgr2=0x7fa9980ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:53.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.690+0000 7fa997e00640 1 -- 172.21.15.28:0/2573894643 shutdown_connections 2024-04-24T08:19:53.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.690+0000 7fa997e00640 1 -- 172.21.15.28:0/2573894643 wait complete. 2024-04-24T08:19:53.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.691+0000 7fa997e00640 1 Processor -- start 2024-04-24T08:19:53.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.691+0000 7fa997e00640 1 -- start start 2024-04-24T08:19:53.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.692+0000 7fa997e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa998102110 0x7fa9981a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.692+0000 7fa997e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9981071e0 0x7fa9981a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.692+0000 7fa997e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa99819a780 0x7fa99819ac30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.692+0000 7fa997e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa9981122e0 con 0x7fa9981071e0 2024-04-24T08:19:53.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.692+0000 7fa997e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa998112160 con 0x7fa998102110 2024-04-24T08:19:53.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.692+0000 7fa997e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa998112460 con 0x7fa99819a780 2024-04-24T08:19:53.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.692+0000 7fa98e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9981071e0 0x7fa9981a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.692+0000 7fa98e000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9981071e0 0x7fa9981a0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58210/0 (socket says 172.21.15.28:58210) 2024-04-24T08:19:53.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.692+0000 7fa98e000640 1 -- 172.21.15.28:0/3765116646 learned_addr learned my addr 172.21.15.28:0/3765116646 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:53.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.692+0000 7fa997400640 1 --2- 172.21.15.28:0/3765116646 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa99819a780 0x7fa99819ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.692+0000 7fa996a00640 1 --2- 172.21.15.28:0/3765116646 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa998102110 0x7fa9981a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.692+0000 7fa98e000640 1 -- 172.21.15.28:0/3765116646 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa99819a780 msgr2=0x7fa99819ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.692+0000 7fa98e000640 1 --2- 172.21.15.28:0/3765116646 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa99819a780 0x7fa99819ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.693+0000 7fa98e000640 1 -- 172.21.15.28:0/3765116646 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa998102110 msgr2=0x7fa9981a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.693+0000 7fa98e000640 1 --2- 172.21.15.28:0/3765116646 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa998102110 0x7fa9981a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.693+0000 7fa98e000640 1 -- 172.21.15.28:0/3765116646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa99819b310 con 0x7fa9981071e0 2024-04-24T08:19:53.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.693+0000 7fa996a00640 1 --2- 172.21.15.28:0/3765116646 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa998102110 0x7fa9981a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:53.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.693+0000 7fa98e000640 1 --2- 172.21.15.28:0/3765116646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9981071e0 0x7fa9981a0b50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa98000e9f0 tx=0x7fa98000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.693+0000 7fa98fe00640 1 -- 172.21.15.28:0/3765116646 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa98000ccc0 con 0x7fa9981071e0 2024-04-24T08:19:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.693+0000 7fa98fe00640 1 -- 172.21.15.28:0/3765116646 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa980005500 con 0x7fa9981071e0 2024-04-24T08:19:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.693+0000 7fa98fe00640 1 -- 172.21.15.28:0/3765116646 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa980010690 con 0x7fa9981071e0 2024-04-24T08:19:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.693+0000 7fa997e00640 1 -- 172.21.15.28:0/3765116646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa99819b600 con 0x7fa9981071e0 2024-04-24T08:19:53.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.693+0000 7fa997e00640 1 -- 172.21.15.28:0/3765116646 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9981a7790 con 0x7fa9981071e0 2024-04-24T08:19:53.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.697+0000 7fa9a0574740 1 Processor -- start 2024-04-24T08:19:53.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.697+0000 7fa98fe00640 1 -- 172.21.15.28:0/3765116646 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa9800046c0 con 0x7fa9981071e0 2024-04-24T08:19:53.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.697+0000 7fa9a0574740 1 -- start start 2024-04-24T08:19:53.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.698+0000 7fa9a0574740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabba30 0x55cdceabbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.698+0000 7fa9a0574740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceabc350 0x55cdceac64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.698+0000 7fa9a0574740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceac6a20 0x55cdceac8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.698+0000 7fa9a0574740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cdcea211f0 con 0x55cdceabc350 2024-04-24T08:19:53.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.698+0000 7fa9a0574740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cdce9c7350 con 0x55cdceac6a20 2024-04-24T08:19:53.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.698+0000 7fa9a0574740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cdcea1d1b0 con 0x55cdceabba30 2024-04-24T08:19:53.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.698+0000 7fa996a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceabc350 0x55cdceac64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.698+0000 7fa996a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceabc350 0x55cdceac64e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58224/0 (socket says 172.21.15.28:58224) 2024-04-24T08:19:53.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.698+0000 7fa996a00640 1 -- 172.21.15.28:0/2759904252 learned_addr learned my addr 172.21.15.28:0/2759904252 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:53.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.698+0000 7fa997400640 1 --2- 172.21.15.28:0/2759904252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabba30 0x55cdceabbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.698+0000 7fa98e000640 1 --2- 172.21.15.28:0/2759904252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceac6a20 0x55cdceac8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.698+0000 7fa996a00640 1 -- 172.21.15.28:0/2759904252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabba30 msgr2=0x55cdceabbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:53.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.698+0000 7fa996a00640 1 --2- 172.21.15.28:0/2759904252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabba30 0x55cdceabbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.698+0000 7fa996a00640 1 -- 172.21.15.28:0/2759904252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceac6a20 msgr2=0x55cdceac8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:53.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.699+0000 7fa996a00640 1 --2- 172.21.15.28:0/2759904252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceac6a20 0x55cdceac8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.699+0000 7fa996a00640 1 -- 172.21.15.28:0/2759904252 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdcea1bf50 con 0x55cdceabc350 2024-04-24T08:19:53.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.699+0000 7fa98e000640 1 --2- 172.21.15.28:0/2759904252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceac6a20 0x55cdceac8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:53.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.699+0000 7fa996a00640 1 --2- 172.21.15.28:0/2759904252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceabc350 0x55cdceac64e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa988009900 tx=0x7fa988005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c74305c8e57abda server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:53.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.699+0000 7fa98fe00640 1 --2- 172.21.15.28:0/3765116646 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa97c0a3d20 0x7fa97c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.699+0000 7fa98fe00640 1 -- 172.21.15.28:0/3765116646 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7fa9800d0c10 con 0x7fa9981071e0 2024-04-24T08:19:53.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.699+0000 7fa987400640 1 -- 172.21.15.28:0/2759904252 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa988035040 con 0x55cdceabc350 2024-04-24T08:19:53.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.699+0000 7fa987400640 1 -- 172.21.15.28:0/2759904252 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa988004760 con 0x55cdceabc350 2024-04-24T08:19:53.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.699+0000 7fa987400640 1 -- 172.21.15.28:0/2759904252 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa988003910 con 0x55cdceabc350 2024-04-24T08:19:53.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.699+0000 7fa997400640 1 --2- 172.21.15.28:0/3765116646 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa97c0a3d20 0x7fa97c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.700+0000 7fa9a0574740 1 -- 172.21.15.28:0/2759904252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceabc350 msgr2=0x55cdceac64e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.700+0000 7fa997400640 1 --2- 172.21.15.28:0/3765116646 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa97c0a3d20 0x7fa97c0a61e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa990005fd0 tx=0x7fa990009500 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:53.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.700+0000 7fa9a0574740 1 --2- 172.21.15.28:0/2759904252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceabc350 0x55cdceac64e0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fa988009900 tx=0x7fa988005de0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.700+0000 7fa9a0574740 1 -- 172.21.15.28:0/2759904252 shutdown_connections 2024-04-24T08:19:53.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.700+0000 7fa9a0574740 1 --2- 172.21.15.28:0/2759904252 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceabc350 0x55cdceac64e0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.700+0000 7fa9a0574740 1 --2- 172.21.15.28:0/2759904252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceac6a20 0x55cdceac8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.700+0000 7fa9a0574740 1 --2- 172.21.15.28:0/2759904252 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabba30 0x55cdceabbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.700+0000 7fa9a0574740 1 -- 172.21.15.28:0/2759904252 >> 172.21.15.28:0/2759904252 conn(0x55cdceab7d50 msgr2=0x55cdceab8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:53.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.701+0000 7fa9a0574740 1 -- 172.21.15.28:0/2759904252 shutdown_connections 2024-04-24T08:19:53.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.701+0000 7fa9a0574740 1 -- 172.21.15.28:0/2759904252 wait complete. 2024-04-24T08:19:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.701+0000 7fa9a0574740 1 -- start start 2024-04-24T08:19:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.701+0000 7fa9a0574740 1 Processor -- start 2024-04-24T08:19:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.702+0000 7fa9a0574740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabba30 0x55cdceac3980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.702+0000 7fa9a0574740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceabc350 0x55cdceac3ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.702+0000 7fa9a0574740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceac6a20 0x55cdceac4400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.702+0000 7fa9a0574740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55cdceac97b0 con 0x55cdceac6a20 2024-04-24T08:19:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.702+0000 7fa9a0574740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55cdce9da690 con 0x55cdceabc350 2024-04-24T08:19:53.710 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.702+0000 7fa9a0574740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55cdceac9930 con 0x55cdceabba30 2024-04-24T08:19:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.702+0000 7fa9a0574740 10 client.0 _ll_register_callbacks cb 0x55cdcea16650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.702+0000 7fa997400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceac6a20 0x55cdceac4400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.702+0000 7fa996a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabba30 0x55cdceac3980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.702+0000 7fa996a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabba30 0x55cdceac3980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44598/0 (socket says 172.21.15.28:44598) 2024-04-24T08:19:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.702+0000 7fa997400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceac6a20 0x55cdceac4400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58236/0 (socket says 172.21.15.28:58236) 2024-04-24T08:19:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.702+0000 7fa996a00640 1 -- 172.21.15.28:0/56414925 learned_addr learned my addr 172.21.15.28:0/56414925 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.702+0000 7fa98e000640 1 --2- 172.21.15.28:0/56414925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceabc350 0x55cdceac3ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.711 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.703+0000 7fa996a00640 1 -- 172.21.15.28:0/56414925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceabc350 msgr2=0x55cdceac3ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:53.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.703+0000 7fa996a00640 1 --2- 172.21.15.28:0/56414925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceabc350 0x55cdceac3ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.703+0000 7fa996a00640 1 -- 172.21.15.28:0/56414925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceac6a20 msgr2=0x55cdceac4400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:53.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.703+0000 7fa996a00640 1 --2- 172.21.15.28:0/56414925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceac6a20 0x55cdceac4400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.703+0000 7fa996a00640 1 -- 172.21.15.28:0/56414925 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdce9d9920 con 0x55cdceabba30 2024-04-24T08:19:53.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.703+0000 7fa997400640 1 --2- 172.21.15.28:0/56414925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceac6a20 0x55cdceac4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:53.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.703+0000 7fa996a00640 1 --2- 172.21.15.28:0/56414925 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabba30 0x55cdceac3980 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa988004b40 tx=0x7fa988000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecf87b11a6d14cc4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:53.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.703+0000 7fa97be00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:19:53.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.703+0000 7fa97be00640 1 -- 172.21.15.28:0/56414925 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa988052060 con 0x55cdceabba30 2024-04-24T08:19:53.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.703+0000 7fa9a0574740 1 -- 172.21.15.28:0/56414925 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa988004760 con 0x55cdceabba30 2024-04-24T08:19:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.703+0000 7fa97be00640 1 -- 172.21.15.28:0/56414925 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7fa988005210 con 0x55cdceabba30 2024-04-24T08:19:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.703+0000 7fa97be00640 1 -- 172.21.15.28:0/56414925 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa988035050 con 0x55cdceabba30 2024-04-24T08:19:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.703+0000 7fa9a0574740 10 client.26872 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:53.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.704+0000 7fa9a0574740 1 -- 172.21.15.28:0/56414925 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cdce7fd7d0 con 0x55cdceabba30 2024-04-24T08:19:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.704+0000 7fa97be00640 1 -- 172.21.15.28:0/56414925 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa988004760 con 0x55cdceabba30 2024-04-24T08:19:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.704+0000 7fa97be00640 1 -- 172.21.15.28:0/56414925 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa9880315b0 con 0x55cdceabba30 2024-04-24T08:19:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.704+0000 7fa9a0574740 10 client.26872 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.704+0000 7fa9a0574740 10 client.26872 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:19:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.704+0000 7fa9a0574740 20 client.26872 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.704+0000 7fa9a0574740 1 --2- 172.21.15.28:0/56414925 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55cdcead6b90 0x55cdcead8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.704+0000 7fa9a0574740 4 client.26872 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:19:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.704+0000 7fa9a0574740 1 -- 172.21.15.28:0/56414925 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cdcea2ec10 con 0x55cdcead6b90 2024-04-24T08:19:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.704+0000 7fa98e000640 1 --2- 172.21.15.28:0/56414925 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55cdcead6b90 0x55cdcead8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.705+0000 7fa98e000640 1 --2- 172.21.15.28:0/56414925 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55cdcead6b90 0x55cdcead8f80 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b12226b9adb2e101 server_cookie=d64c4966fba2665 in_seq=0 out_seq=0 2024-04-24T08:19:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.705+0000 7fa97be00640 10 client.26872 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.706+0000 7fa97be00640 1 -- 172.21.15.28:0/56414925 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cdcea2ec10 con 0x55cdcead6b90 2024-04-24T08:19:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.706+0000 7fa97be00640 10 client.26872 handle_command_reply: tid=0 2024-04-24T08:19:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.706+0000 7fa9a0574740 1 client.26872 shutdown 2024-04-24T08:19:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.706+0000 7fa9a0574740 1 -- 172.21.15.28:0/56414925 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabba30 msgr2=0x55cdceac3980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.706+0000 7fa9a0574740 1 --2- 172.21.15.28:0/56414925 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabba30 0x55cdceac3980 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fa988004b40 tx=0x7fa988000bc0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.706+0000 7fa9a0574740 1 -- 172.21.15.28:0/56414925 shutdown_connections 2024-04-24T08:19:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.706+0000 7fa9a0574740 1 --2- 172.21.15.28:0/56414925 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55cdcead6b90 0x55cdcead8f80 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.706+0000 7fa9a0574740 1 --2- 172.21.15.28:0/56414925 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceac6a20 0x55cdceac4400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.706+0000 7fa9a0574740 1 --2- 172.21.15.28:0/56414925 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceabc350 0x55cdceac3ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.706+0000 7fa97be00640 0 client.26872 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.706+0000 7fa9a0574740 1 --2- 172.21.15.28:0/56414925 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabba30 0x55cdceac3980 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.706+0000 7fa9a0574740 1 -- 172.21.15.28:0/56414925 >> 172.21.15.28:0/56414925 conn(0x55cdceab7d50 msgr2=0x55cdceabdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:53.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.706+0000 7fa9a0574740 1 -- 172.21.15.28:0/56414925 shutdown_connections 2024-04-24T08:19:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.706+0000 7fa9a0574740 1 -- 172.21.15.28:0/56414925 wait complete. 2024-04-24T08:19:53.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.706+0000 7fa9a0574740 20 client.26872 trim_cache size 0 max 16384 2024-04-24T08:19:53.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.726+0000 7fa9a0574740 1 Processor -- start 2024-04-24T08:19:53.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.726+0000 7fa9a0574740 1 -- start start 2024-04-24T08:19:53.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.726+0000 7fa9a0574740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceabba30 0x55cdcead2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.726+0000 7fa9a0574740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabc350 0x55cdceac29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.726+0000 7fa9a0574740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceac6a20 0x55cdceabe190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.726+0000 7fa9a0574740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa9981b2770 con 0x55cdceac6a20 2024-04-24T08:19:53.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.726+0000 7fa9a0574740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa9981b0040 con 0x55cdceabba30 2024-04-24T08:19:53.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.726+0000 7fa9a0574740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa9981adeb0 con 0x55cdceabc350 2024-04-24T08:19:53.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.727+0000 7fa997400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceac6a20 0x55cdceabe190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.727+0000 7fa996a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceabba30 0x55cdcead2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.727+0000 7fa997400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceac6a20 0x55cdceabe190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58246/0 (socket says 172.21.15.28:58246) 2024-04-24T08:19:53.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.727+0000 7fa997400640 1 -- 172.21.15.28:0/2249707537 learned_addr learned my addr 172.21.15.28:0/2249707537 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:53.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.727+0000 7fa98e000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabc350 0x55cdceac29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.727+0000 7fa996a00640 1 -- 172.21.15.28:0/2249707537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabc350 msgr2=0x55cdceac29d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.727+0000 7fa996a00640 1 --2- 172.21.15.28:0/2249707537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabc350 0x55cdceac29d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.727+0000 7fa996a00640 1 -- 172.21.15.28:0/2249707537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceac6a20 msgr2=0x55cdceabe190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.727+0000 7fa996a00640 1 --2- 172.21.15.28:0/2249707537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceac6a20 0x55cdceabe190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.727+0000 7fa996a00640 1 -- 172.21.15.28:0/2249707537 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cdce7333f0 con 0x55cdceabba30 2024-04-24T08:19:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.727+0000 7fa997400640 1 --2- 172.21.15.28:0/2249707537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceac6a20 0x55cdceabe190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.727+0000 7fa996a00640 1 --2- 172.21.15.28:0/2249707537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceabba30 0x55cdcead2670 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa98802aa00 tx=0x7fa988005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=955ecb8faf93ec63 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:53.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.728+0000 7fa986a00640 1 -- 172.21.15.28:0/2249707537 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa988052030 con 0x55cdceabba30 2024-04-24T08:19:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.728+0000 7fa986a00640 1 -- 172.21.15.28:0/2249707537 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa988004b70 con 0x55cdceabba30 2024-04-24T08:19:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.728+0000 7fa986a00640 1 -- 172.21.15.28:0/2249707537 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa988003850 con 0x55cdceabba30 2024-04-24T08:19:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.728+0000 7fa9a0574740 1 -- 172.21.15.28:0/2249707537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceabba30 msgr2=0x55cdcead2670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.728+0000 7fa9a0574740 1 --2- 172.21.15.28:0/2249707537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceabba30 0x55cdcead2670 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7fa98802aa00 tx=0x7fa988005de0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.729+0000 7fa9a0574740 1 -- 172.21.15.28:0/2249707537 shutdown_connections 2024-04-24T08:19:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.729+0000 7fa9a0574740 1 --2- 172.21.15.28:0/2249707537 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceac6a20 0x55cdceabe190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.729+0000 7fa9a0574740 1 --2- 172.21.15.28:0/2249707537 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabc350 0x55cdceac29d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.729+0000 7fa9a0574740 1 --2- 172.21.15.28:0/2249707537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceabba30 0x55cdcead2670 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.729+0000 7fa9a0574740 1 -- 172.21.15.28:0/2249707537 >> 172.21.15.28:0/2249707537 conn(0x55cdceab7d50 msgr2=0x55cdceab6740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.729+0000 7fa9a0574740 1 -- 172.21.15.28:0/2249707537 shutdown_connections 2024-04-24T08:19:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.729+0000 7fa9a0574740 1 -- 172.21.15.28:0/2249707537 wait complete. 2024-04-24T08:19:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.729+0000 7fa9a0574740 1 -- start start 2024-04-24T08:19:53.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.729+0000 7fa9a0574740 1 Processor -- start 2024-04-24T08:19:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.730+0000 7fa9a0574740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceabba30 0x55cdceace950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.730+0000 7fa9a0574740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabc350 0x55cdceacee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.730+0000 7fa9a0574740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceac6a20 0x55cdceabfc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.730+0000 7fa9a0574740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa9981ae500 con 0x55cdceabba30 2024-04-24T08:19:53.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.730+0000 7fa9a0574740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa9981af970 con 0x55cdceac6a20 2024-04-24T08:19:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.730+0000 7fa9a0574740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa9981aebd0 con 0x55cdceabc350 2024-04-24T08:19:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.730+0000 7fa9a0574740 10 client.0 _ll_register_callbacks cb 0x7fa990009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.730+0000 7fa996a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceabba30 0x55cdceace950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.730+0000 7fa996a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceabba30 0x55cdceace950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58252/0 (socket says 172.21.15.28:58252) 2024-04-24T08:19:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.730+0000 7fa996a00640 1 -- 172.21.15.28:0/3781729711 learned_addr learned my addr 172.21.15.28:0/3781729711 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.730+0000 7fa98e000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabc350 0x55cdceacee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.730+0000 7fa997400640 1 --2- 172.21.15.28:0/3781729711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceac6a20 0x55cdceabfc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.730+0000 7fa996a00640 1 -- 172.21.15.28:0/3781729711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabc350 msgr2=0x55cdceacee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:53.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.730+0000 7fa996a00640 1 --2- 172.21.15.28:0/3781729711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabc350 0x55cdceacee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.730+0000 7fa996a00640 1 -- 172.21.15.28:0/3781729711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceac6a20 msgr2=0x55cdceabfc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:53.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.730+0000 7fa996a00640 1 --2- 172.21.15.28:0/3781729711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceac6a20 0x55cdceabfc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.730+0000 7fa996a00640 1 -- 172.21.15.28:0/3781729711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cdce6f30f0 con 0x55cdceabba30 2024-04-24T08:19:53.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.731+0000 7fa98e000640 1 --2- 172.21.15.28:0/3781729711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabc350 0x55cdceacee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:53.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.731+0000 7fa996a00640 1 --2- 172.21.15.28:0/3781729711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceabba30 0x55cdceace950 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa988004fd0 tx=0x7fa988000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73946281574e7cbc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:53.739 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.731+0000 7fa97be00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:19:53.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.731+0000 7fa97be00640 1 -- 172.21.15.28:0/3781729711 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa988052070 con 0x55cdceabba30 2024-04-24T08:19:53.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.731+0000 7fa97be00640 1 -- 172.21.15.28:0/3781729711 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7fa988005420 con 0x55cdceabba30 2024-04-24T08:19:53.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.731+0000 7fa9a0574740 1 -- 172.21.15.28:0/3781729711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa988004b70 con 0x55cdceabba30 2024-04-24T08:19:53.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.731+0000 7fa97be00640 1 -- 172.21.15.28:0/3781729711 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa988005620 con 0x55cdceabba30 2024-04-24T08:19:53.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.731+0000 7fa9a0574740 10 client.16581 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:53.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.731+0000 7fa9a0574740 1 -- 172.21.15.28:0/3781729711 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55cdce9e6810 con 0x55cdceabba30 2024-04-24T08:19:53.740 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.732+0000 7fa97be00640 1 -- 172.21.15.28:0/3781729711 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa988004b70 con 0x55cdceabba30 2024-04-24T08:19:53.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.732+0000 7fa97be00640 1 -- 172.21.15.28:0/3781729711 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa98804e690 con 0x55cdceabba30 2024-04-24T08:19:53.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.732+0000 7fa9a0574740 10 client.16581 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:53.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.732+0000 7fa9a0574740 10 client.16581 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:19:53.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.732+0000 7fa9a0574740 20 client.16581 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:53.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.732+0000 7fa9a0574740 1 --2- 172.21.15.28:0/3781729711 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55cdceb3c6a0 0x55cdceb3eac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.732+0000 7fa9a0574740 4 client.16581 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:19:53.741 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.732+0000 7fa9a0574740 1 -- 172.21.15.28:0/3781729711 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55cdcea28a80 con 0x55cdceb3c6a0 2024-04-24T08:19:53.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.732+0000 7fa98e000640 1 --2- 172.21.15.28:0/3781729711 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55cdceb3c6a0 0x55cdceb3eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.733+0000 7fa98e000640 1 --2- 172.21.15.28:0/3781729711 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55cdceb3c6a0 0x55cdceb3eac0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90fedc5dd4722d72 server_cookie=d9bd81de6a3fd5a7 in_seq=0 out_seq=0 2024-04-24T08:19:53.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.733+0000 7fa97be00640 10 client.16581 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:53.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.733+0000 7fa97be00640 1 -- 172.21.15.28:0/3781729711 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55cdcea28a80 con 0x55cdceb3c6a0 2024-04-24T08:19:53.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.733+0000 7fa97be00640 10 client.16581 handle_command_reply: tid=0 2024-04-24T08:19:53.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.733+0000 7fa9a0574740 1 client.16581 shutdown 2024-04-24T08:19:53.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.733+0000 7fa9a0574740 1 -- 172.21.15.28:0/3781729711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceabba30 msgr2=0x55cdceace950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.742 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.733+0000 7fa9a0574740 1 --2- 172.21.15.28:0/3781729711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceabba30 0x55cdceace950 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fa988004fd0 tx=0x7fa988000bc0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.733+0000 7fa9a0574740 1 -- 172.21.15.28:0/3781729711 shutdown_connections 2024-04-24T08:19:53.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.733+0000 7fa9a0574740 1 --2- 172.21.15.28:0/3781729711 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55cdceb3c6a0 0x55cdceb3eac0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.733+0000 7fa9a0574740 1 --2- 172.21.15.28:0/3781729711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55cdceac6a20 0x55cdceabfc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.733+0000 7fa9a0574740 1 --2- 172.21.15.28:0/3781729711 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55cdceabc350 0x55cdceacee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.733+0000 7fa9a0574740 1 --2- 172.21.15.28:0/3781729711 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55cdceabba30 0x55cdceace950 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.733+0000 7fa97be00640 0 client.16581 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:53.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.733+0000 7fa9a0574740 1 -- 172.21.15.28:0/3781729711 >> 172.21.15.28:0/3781729711 conn(0x55cdceab7d50 msgr2=0x55cdceac2850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:53.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.734+0000 7fa9a0574740 1 -- 172.21.15.28:0/3781729711 shutdown_connections 2024-04-24T08:19:53.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.734+0000 7fa9a0574740 1 -- 172.21.15.28:0/3781729711 wait complete. 2024-04-24T08:19:53.744 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.734+0000 7fa9a0574740 20 client.16581 trim_cache size 0 max 16384 2024-04-24T08:19:53.744 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:19:53.744 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-24T08:19:53.744 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-24T08:19:53.745 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:19:53.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.734+0000 7fa997e00640 1 -- 172.21.15.28:0/3765116646 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa97c0a3d20 msgr2=0x7fa97c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.734+0000 7fa997e00640 1 --2- 172.21.15.28:0/3765116646 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa97c0a3d20 0x7fa97c0a61e0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fa990005fd0 tx=0x7fa990009500 comp rx=0 tx=0).stop 2024-04-24T08:19:53.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.734+0000 7fa997e00640 1 -- 172.21.15.28:0/3765116646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9981071e0 msgr2=0x7fa9981a0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.734+0000 7fa997e00640 1 --2- 172.21.15.28:0/3765116646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9981071e0 0x7fa9981a0b50 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fa98000e9f0 tx=0x7fa98000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.735+0000 7fa997e00640 1 -- 172.21.15.28:0/3765116646 shutdown_connections 2024-04-24T08:19:53.746 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.735+0000 7fa997e00640 1 --2- 172.21.15.28:0/3765116646 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa97c0a3d20 0x7fa97c0a61e0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.735+0000 7fa997e00640 1 --2- 172.21.15.28:0/3765116646 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa9981071e0 0x7fa9981a0b50 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.735+0000 7fa997e00640 1 --2- 172.21.15.28:0/3765116646 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa99819a780 0x7fa99819ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.735+0000 7fa997e00640 1 --2- 172.21.15.28:0/3765116646 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa998102110 0x7fa9981a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.735+0000 7fa997e00640 1 -- 172.21.15.28:0/3765116646 >> 172.21.15.28:0/3765116646 conn(0x7fa9980fd9d0 msgr2=0x7fa9980fe870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:53.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.735+0000 7fa997e00640 1 -- 172.21.15.28:0/3765116646 shutdown_connections 2024-04-24T08:19:53.747 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.735+0000 7fa997e00640 1 -- 172.21.15.28:0/3765116646 wait complete. 2024-04-24T08:19:53.753 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:19:53.754 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:19:53.825 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:53 smithi028 ceph-mon[27786]: pgmap v295: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-24T08:19:53.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:53 smithi194 ceph-mon[32836]: pgmap v295: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-24T08:19:53.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.966+0000 7f0204c00640 1 -- 172.21.15.28:0/1616542356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0200069db0 msgr2=0x7f0200106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.966+0000 7f0204c00640 1 --2- 172.21.15.28:0/1616542356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0200069db0 0x7f0200106a70 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009900 tx=0x7f01ec02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.967+0000 7f0204c00640 1 -- 172.21.15.28:0/1616542356 shutdown_connections 2024-04-24T08:19:53.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.967+0000 7f0204c00640 1 --2- 172.21.15.28:0/1616542356 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02001071a0 0x7f020010ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.967+0000 7f0204c00640 1 --2- 172.21.15.28:0/1616542356 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f0200069db0 0x7f0200106a70 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.967+0000 7f0204c00640 1 --2- 172.21.15.28:0/1616542356 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0200069400 0x7f02000697e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.967+0000 7f0204c00640 1 -- 172.21.15.28:0/1616542356 >> 172.21.15.28:0/1616542356 conn(0x7f0200076840 msgr2=0x7f0200076c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:53.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.967+0000 7f0204c00640 1 -- 172.21.15.28:0/1616542356 shutdown_connections 2024-04-24T08:19:53.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.967+0000 7f0204c00640 1 -- 172.21.15.28:0/1616542356 wait complete. 2024-04-24T08:19:53.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.968+0000 7f0204c00640 1 Processor -- start 2024-04-24T08:19:53.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.968+0000 7f0204c00640 1 -- start start 2024-04-24T08:19:53.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.969+0000 7f0204c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0200069400 0x7f02001a06a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.969+0000 7f0204c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02001071a0 0x7f02001a0be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.969+0000 7f0204c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f020019a760 0x7f020019abe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.969+0000 7f0204c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f0200112240 con 0x7f0200069400 2024-04-24T08:19:53.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.969+0000 7f0204c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f02001120c0 con 0x7f020019a760 2024-04-24T08:19:53.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.969+0000 7f0204c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f02001123c0 con 0x7f02001071a0 2024-04-24T08:19:53.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.969+0000 7f01ff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0200069400 0x7f02001a06a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.974 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.969+0000 7f01fea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02001071a0 0x7f02001a0be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.969+0000 7f01fea00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02001071a0 0x7f02001a0be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44638/0 (socket says 172.21.15.28:44638) 2024-04-24T08:19:53.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.969+0000 7f01fea00640 1 -- 172.21.15.28:0/3034863138 learned_addr learned my addr 172.21.15.28:0/3034863138 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:53.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.969+0000 7f01ff400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0200069400 0x7f02001a06a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58276/0 (socket says 172.21.15.28:58276) 2024-04-24T08:19:53.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.969+0000 7f01ffe00640 1 --2- 172.21.15.28:0/3034863138 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f020019a760 0x7f020019abe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.970+0000 7f01fea00640 1 -- 172.21.15.28:0/3034863138 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f020019a760 msgr2=0x7f020019abe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.975 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.970+0000 7f01fea00640 1 --2- 172.21.15.28:0/3034863138 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f020019a760 0x7f020019abe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.970+0000 7f01fea00640 1 -- 172.21.15.28:0/3034863138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0200069400 msgr2=0x7f02001a06a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:53.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.970+0000 7f01fea00640 1 --2- 172.21.15.28:0/3034863138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0200069400 0x7f02001a06a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:53.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.970+0000 7f01fea00640 1 -- 172.21.15.28:0/3034863138 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020019b360 con 0x7f02001071a0 2024-04-24T08:19:53.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.970+0000 7f01ff400640 1 --2- 172.21.15.28:0/3034863138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0200069400 0x7f02001a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:53.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.970+0000 7f01fea00640 1 --2- 172.21.15.28:0/3034863138 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02001071a0 0x7f02001a0be0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f01ec02a910 tx=0x7f01ec003b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:53.976 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.970+0000 7f01f7e00640 1 -- 172.21.15.28:0/3034863138 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec03f070 con 0x7f02001071a0 2024-04-24T08:19:53.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.970+0000 7f01f7e00640 1 -- 172.21.15.28:0/3034863138 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01ec031c90 con 0x7f02001071a0 2024-04-24T08:19:53.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.970+0000 7f01f7e00640 1 -- 172.21.15.28:0/3034863138 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec035030 con 0x7f02001071a0 2024-04-24T08:19:53.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.970+0000 7f0204c00640 1 -- 172.21.15.28:0/3034863138 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f020019b5f0 con 0x7f02001071a0 2024-04-24T08:19:53.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.970+0000 7f0204c00640 1 -- 172.21.15.28:0/3034863138 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02001a76e0 con 0x7f02001071a0 2024-04-24T08:19:53.977 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.972+0000 7f0204c00640 1 -- 172.21.15.28:0/3034863138 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0200103550 con 0x7f02001071a0 2024-04-24T08:19:53.978 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.974+0000 7f01f7e00640 1 -- 172.21.15.28:0/3034863138 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f01ec042070 con 0x7f02001071a0 2024-04-24T08:19:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.975+0000 7f01f7e00640 1 --2- 172.21.15.28:0/3034863138 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f01e40a3da0 0x7f01e40a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.975+0000 7f01f7e00640 1 -- 172.21.15.28:0/3034863138 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7f01ec0f1160 con 0x7f02001071a0 2024-04-24T08:19:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.975+0000 7f01ff400640 1 --2- 172.21.15.28:0/3034863138 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f01e40a3da0 0x7f01e40a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:53.979 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.976+0000 7f01ff400640 1 --2- 172.21.15.28:0/3034863138 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f01e40a3da0 0x7f01e40a6260 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f01f0009fc0 tx=0x7f01f0005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:53.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:53.980+0000 7f01f7e00640 1 -- 172.21.15.28:0/3034863138 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f01ec0bd100 con 0x7f02001071a0 2024-04-24T08:19:54.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:53 smithi117 ceph-mon[32438]: pgmap v295: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-04-24T08:19:54.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.238+0000 7f0204c00640 1 -- 172.21.15.28:0/3034863138 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f0200069db0 con 0x7f02001071a0 2024-04-24T08:19:54.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.240+0000 7f01f7e00640 1 -- 172.21.15.28:0/3034863138 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11871 (secure 0 0 0) 0x7f01ec0c1400 con 0x7f02001071a0 2024-04-24T08:19:54.241 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:19:54.241 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:14:38.663141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:19:54.241 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-24T08:19:54.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.243+0000 7f0204c00640 1 -- 172.21.15.28:0/3034863138 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f01e40a3da0 msgr2=0x7f01e40a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.243+0000 7f0204c00640 1 --2- 172.21.15.28:0/3034863138 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f01e40a3da0 0x7f01e40a6260 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f01f0009fc0 tx=0x7f01f0005f90 comp rx=0 tx=0).stop 2024-04-24T08:19:54.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.243+0000 7f0204c00640 1 -- 172.21.15.28:0/3034863138 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02001071a0 msgr2=0x7f02001a0be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.243+0000 7f0204c00640 1 --2- 172.21.15.28:0/3034863138 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02001071a0 0x7f02001a0be0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f01ec02a910 tx=0x7f01ec003b20 comp rx=0 tx=0).stop 2024-04-24T08:19:54.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.243+0000 7f0204c00640 1 -- 172.21.15.28:0/3034863138 shutdown_connections 2024-04-24T08:19:54.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.243+0000 7f0204c00640 1 --2- 172.21.15.28:0/3034863138 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f01e40a3da0 0x7f01e40a6260 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.243+0000 7f0204c00640 1 --2- 172.21.15.28:0/3034863138 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f020019a760 0x7f020019abe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.243+0000 7f0204c00640 1 --2- 172.21.15.28:0/3034863138 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02001071a0 0x7f02001a0be0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.243+0000 7f0204c00640 1 --2- 172.21.15.28:0/3034863138 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f0200069400 0x7f02001a06a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.243+0000 7f0204c00640 1 -- 172.21.15.28:0/3034863138 >> 172.21.15.28:0/3034863138 conn(0x7f0200076840 msgr2=0x7f020010d230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.243+0000 7f0204c00640 1 -- 172.21.15.28:0/3034863138 shutdown_connections 2024-04-24T08:19:54.245 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.243+0000 7f0204c00640 1 -- 172.21.15.28:0/3034863138 wait complete. 2024-04-24T08:19:54.264 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:19:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.483+0000 7fbebac00640 1 -- 172.21.15.28:0/1115759195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbeb4074cf0 msgr2=0x7fbeb410e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.483+0000 7fbebac00640 1 --2- 172.21.15.28:0/1115759195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbeb4074cf0 0x7fbeb410e3b0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c009860 tx=0x7fbe9c02a3d0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.483+0000 7fbebac00640 1 -- 172.21.15.28:0/1115759195 shutdown_connections 2024-04-24T08:19:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.483+0000 7fbebac00640 1 --2- 172.21.15.28:0/1115759195 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb410e8f0 0x7fbeb4110d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.483+0000 7fbebac00640 1 --2- 172.21.15.28:0/1115759195 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbeb4074cf0 0x7fbeb410e3b0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.484 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.483+0000 7fbebac00640 1 --2- 172.21.15.28:0/1115759195 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeb40743d0 0x7fbeb40747b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.483+0000 7fbebac00640 1 -- 172.21.15.28:0/1115759195 >> 172.21.15.28:0/1115759195 conn(0x7fbeb40fdb60 msgr2=0x7fbeb40fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.483+0000 7fbebac00640 1 -- 172.21.15.28:0/1115759195 shutdown_connections 2024-04-24T08:19:54.485 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.483+0000 7fbebac00640 1 -- 172.21.15.28:0/1115759195 wait complete. 2024-04-24T08:19:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.484+0000 7fbebac00640 1 Processor -- start 2024-04-24T08:19:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.484+0000 7fbebac00640 1 -- start start 2024-04-24T08:19:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.484+0000 7fbebac00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb40743d0 0x7fbeb41a0830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.487 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.484+0000 7fbebac00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbeb4074cf0 0x7fbeb41a0d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.485+0000 7fbebac00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeb410e8f0 0x7fbeb419aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.485+0000 7fbebac00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb4114650 con 0x7fbeb410e8f0 2024-04-24T08:19:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.485+0000 7fbebac00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb41144d0 con 0x7fbeb4074cf0 2024-04-24T08:19:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.485+0000 7fbebac00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb41147d0 con 0x7fbeb40743d0 2024-04-24T08:19:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.485+0000 7fbeb8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeb410e8f0 0x7fbeb419aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.485+0000 7fbeb8c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeb410e8f0 0x7fbeb419aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58290/0 (socket says 172.21.15.28:58290) 2024-04-24T08:19:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.485+0000 7fbeb8c00640 1 -- 172.21.15.28:0/3738358127 learned_addr learned my addr 172.21.15.28:0/3738358127 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.485+0000 7fbeb3400640 1 --2- 172.21.15.28:0/3738358127 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbeb4074cf0 0x7fbeb41a0d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.485+0000 7fbeb3e00640 1 --2- 172.21.15.28:0/3738358127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb40743d0 0x7fbeb41a0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.485+0000 7fbeb8c00640 1 -- 172.21.15.28:0/3738358127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb40743d0 msgr2=0x7fbeb41a0830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.485+0000 7fbeb8c00640 1 --2- 172.21.15.28:0/3738358127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb40743d0 0x7fbeb41a0830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.485+0000 7fbeb8c00640 1 -- 172.21.15.28:0/3738358127 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbeb4074cf0 msgr2=0x7fbeb41a0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.485+0000 7fbeb8c00640 1 --2- 172.21.15.28:0/3738358127 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbeb4074cf0 0x7fbeb41a0d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.485+0000 7fbeb8c00640 1 -- 172.21.15.28:0/3738358127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbeb419b1b0 con 0x7fbeb410e8f0 2024-04-24T08:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.485+0000 7fbeb3400640 1 --2- 172.21.15.28:0/3738358127 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbeb4074cf0 0x7fbeb41a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:54.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.485+0000 7fbeb3e00640 1 --2- 172.21.15.28:0/3738358127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb40743d0 0x7fbeb41a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.486+0000 7fbeb8c00640 1 --2- 172.21.15.28:0/3738358127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeb410e8f0 0x7fbeb419aa30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fbeac00c3a0 tx=0x7fbeac00c860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.486+0000 7fbeb0c00640 1 -- 172.21.15.28:0/3738358127 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeac003970 con 0x7fbeb410e8f0 2024-04-24T08:19:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.486+0000 7fbebac00640 1 -- 172.21.15.28:0/3738358127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbeb419b4a0 con 0x7fbeb410e8f0 2024-04-24T08:19:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.486+0000 7fbeb0c00640 1 -- 172.21.15.28:0/3738358127 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbeac004020 con 0x7fbeb410e8f0 2024-04-24T08:19:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.486+0000 7fbeb0c00640 1 -- 172.21.15.28:0/3738358127 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbeac00bab0 con 0x7fbeb410e8f0 2024-04-24T08:19:54.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.486+0000 7fbebac00640 1 -- 172.21.15.28:0/3738358127 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbeb41a76f0 con 0x7fbeb410e8f0 2024-04-24T08:19:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.489+0000 7fbebd5fd740 1 Processor -- start 2024-04-24T08:19:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.489+0000 7fbeb0c00640 1 -- 172.21.15.28:0/3738358127 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fbeac01d020 con 0x7fbeb410e8f0 2024-04-24T08:19:54.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.490+0000 7fbebd5fd740 1 -- start start 2024-04-24T08:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.490+0000 7fbebd5fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b616950 0x55b95b616d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.490+0000 7fbebd5fd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b617270 0x55b95b621400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.490+0000 7fbebd5fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b621940 0x55b95b623d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.490+0000 7fbebd5fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b95b57c1d0 con 0x55b95b616950 2024-04-24T08:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.490+0000 7fbebd5fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b95b522450 con 0x55b95b621940 2024-04-24T08:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.490+0000 7fbebd5fd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b95b578190 con 0x55b95b617270 2024-04-24T08:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.490+0000 7fbeb3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b616950 0x55b95b616d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.490+0000 7fbeb3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b616950 0x55b95b616d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58296/0 (socket says 172.21.15.28:58296) 2024-04-24T08:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.490+0000 7fbeb3400640 1 -- 172.21.15.28:0/2521684773 learned_addr learned my addr 172.21.15.28:0/2521684773 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.491+0000 7fbeb3e00640 1 --2- 172.21.15.28:0/2521684773 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b617270 0x55b95b621400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.491+0000 7fbeb3400640 1 --2- 172.21.15.28:0/2521684773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b621940 0x55b95b623d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.491+0000 7fbeb3400640 1 -- 172.21.15.28:0/2521684773 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b617270 msgr2=0x55b95b621400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.491+0000 7fbeb3400640 1 --2- 172.21.15.28:0/2521684773 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b617270 0x55b95b621400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.491+0000 7fbeb3400640 1 -- 172.21.15.28:0/2521684773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b621940 msgr2=0x55b95b623d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.491+0000 7fbeb3400640 1 --2- 172.21.15.28:0/2521684773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b621940 0x55b95b623d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.491+0000 7fbeb3400640 1 -- 172.21.15.28:0/2521684773 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b95b5df310 con 0x55b95b616950 2024-04-24T08:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.491+0000 7fbeb3400640 1 --2- 172.21.15.28:0/2521684773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b616950 0x55b95b616d30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c005e00 tx=0x7fbe9c005d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=59f3475cfdd89abf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.491+0000 7fbeb0c00640 1 --2- 172.21.15.28:0/3738358127 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fbe940a3d20 0x7fbe940a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.492+0000 7fbeb0c00640 1 -- 172.21.15.28:0/3738358127 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7fbeac0d07a0 con 0x7fbeb410e8f0 2024-04-24T08:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.492+0000 7fbea9200640 1 -- 172.21.15.28:0/2521684773 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe9c004250 con 0x55b95b616950 2024-04-24T08:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.492+0000 7fbea9200640 1 -- 172.21.15.28:0/2521684773 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe9c047070 con 0x55b95b616950 2024-04-24T08:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.492+0000 7fbea9200640 1 -- 172.21.15.28:0/2521684773 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe9c005640 con 0x55b95b616950 2024-04-24T08:19:54.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.492+0000 7fbeb3e00640 1 --2- 172.21.15.28:0/3738358127 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fbe940a3d20 0x7fbe940a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.493+0000 7fbebd5fd740 1 -- 172.21.15.28:0/2521684773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b616950 msgr2=0x55b95b616d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.493+0000 7fbeb3e00640 1 --2- 172.21.15.28:0/3738358127 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fbe940a3d20 0x7fbe940a61e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x55b95b617f10 tx=0x7fbea4005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.493+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/2521684773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b616950 0x55b95b616d30 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c005e00 tx=0x7fbe9c005d90 comp rx=0 tx=0).stop 2024-04-24T08:19:54.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.493+0000 7fbebd5fd740 1 -- 172.21.15.28:0/2521684773 shutdown_connections 2024-04-24T08:19:54.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.493+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/2521684773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b621940 0x55b95b623d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.493+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/2521684773 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b617270 0x55b95b621400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.493+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/2521684773 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b616950 0x55b95b616d30 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.500 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.493+0000 7fbebd5fd740 1 -- 172.21.15.28:0/2521684773 >> 172.21.15.28:0/2521684773 conn(0x55b95b612c70 msgr2=0x55b95b613050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.493+0000 7fbebd5fd740 1 -- 172.21.15.28:0/2521684773 shutdown_connections 2024-04-24T08:19:54.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.493+0000 7fbebd5fd740 1 -- 172.21.15.28:0/2521684773 wait complete. 2024-04-24T08:19:54.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.493+0000 7fbebd5fd740 1 -- start start 2024-04-24T08:19:54.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.494+0000 7fbebd5fd740 1 Processor -- start 2024-04-24T08:19:54.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.494+0000 7fbebd5fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b616950 0x55b95b6213d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.494+0000 7fbebd5fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b617270 0x55b95b61e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.494+0000 7fbebd5fd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b621940 0x55b95b61edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.494+0000 7fbebd5fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b95b624770 con 0x55b95b617270 2024-04-24T08:19:54.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.494+0000 7fbebd5fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b95b535810 con 0x55b95b616950 2024-04-24T08:19:54.501 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.494+0000 7fbebd5fd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b95b6248f0 con 0x55b95b621940 2024-04-24T08:19:54.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.494+0000 7fbebd5fd740 10 client.0 _ll_register_callbacks cb 0x55b95b503140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:54.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.494+0000 7fbeb3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b617270 0x55b95b61e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.494+0000 7fbeb3e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b616950 0x55b95b6213d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.494+0000 7fbeb3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b617270 0x55b95b61e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58308/0 (socket says 172.21.15.28:58308) 2024-04-24T08:19:54.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.494+0000 7fbeb3400640 1 -- 172.21.15.28:0/140264363 learned_addr learned my addr 172.21.15.28:0/140264363 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:54.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.494+0000 7fbeb8c00640 1 --2- 172.21.15.28:0/140264363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b621940 0x55b95b61edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.495+0000 7fbeb3400640 1 -- 172.21.15.28:0/140264363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b621940 msgr2=0x55b95b61edb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.502 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.495+0000 7fbeb3400640 1 --2- 172.21.15.28:0/140264363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b621940 0x55b95b61edb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.495+0000 7fbeb3400640 1 -- 172.21.15.28:0/140264363 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b616950 msgr2=0x55b95b6213d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.495+0000 7fbeb3400640 1 --2- 172.21.15.28:0/140264363 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b616950 0x55b95b6213d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.495+0000 7fbeb3400640 1 -- 172.21.15.28:0/140264363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b95b5463f0 con 0x55b95b617270 2024-04-24T08:19:54.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.495+0000 7fbeb3e00640 1 --2- 172.21.15.28:0/140264363 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b616950 0x55b95b6213d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:54.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.495+0000 7fbeb3400640 1 --2- 172.21.15.28:0/140264363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b617270 0x55b95b61e6c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c005200 tx=0x7fbe9c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4c2b0f4a0d7aee7 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.495+0000 7fbea1600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:19:54.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.495+0000 7fbea1600640 1 -- 172.21.15.28:0/140264363 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe9c050070 con 0x55b95b617270 2024-04-24T08:19:54.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.496+0000 7fbebd5fd740 1 -- 172.21.15.28:0/140264363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe9c047070 con 0x55b95b617270 2024-04-24T08:19:54.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.496+0000 7fbea1600640 1 -- 172.21.15.28:0/140264363 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7fbe9c0307b0 con 0x55b95b617270 2024-04-24T08:19:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.496+0000 7fbea1600640 1 -- 172.21.15.28:0/140264363 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbe9c05c050 con 0x55b95b617270 2024-04-24T08:19:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.496+0000 7fbebd5fd740 10 client.16599 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.496+0000 7fbebd5fd740 1 -- 172.21.15.28:0/140264363 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b95b28e3f0 con 0x55b95b617270 2024-04-24T08:19:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.496+0000 7fbea1600640 1 -- 172.21.15.28:0/140264363 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe9c047070 con 0x55b95b617270 2024-04-24T08:19:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.496+0000 7fbea1600640 1 -- 172.21.15.28:0/140264363 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fbe9c0046c0 con 0x55b95b617270 2024-04-24T08:19:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.496+0000 7fbebd5fd740 10 client.16599 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.496+0000 7fbebd5fd740 10 client.16599 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:19:54.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.497+0000 7fbebd5fd740 20 client.16599 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.497+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/140264363 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55b95b6319b0 0x55b95b633da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.497+0000 7fbebd5fd740 4 client.16599 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:19:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.497+0000 7fbebd5fd740 1 -- 172.21.15.28:0/140264363 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b95b589bf0 con 0x55b95b6319b0 2024-04-24T08:19:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.497+0000 7fbeb3e00640 1 --2- 172.21.15.28:0/140264363 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55b95b6319b0 0x55b95b633da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.498+0000 7fbeb3e00640 1 --2- 172.21.15.28:0/140264363 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55b95b6319b0 0x55b95b633da0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2dd0e75d24c82a2d server_cookie=71c955436ad5ecf9 in_seq=0 out_seq=0 2024-04-24T08:19:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.498+0000 7fbea1600640 10 client.16599 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:19:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.498+0000 7fbea1600640 1 -- 172.21.15.28:0/140264363 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b95b589bf0 con 0x55b95b6319b0 2024-04-24T08:19:54.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.498+0000 7fbea1600640 10 client.16599 handle_command_reply: tid=0 2024-04-24T08:19:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.499+0000 7fbebd5fd740 1 client.16599 shutdown 2024-04-24T08:19:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.499+0000 7fbebd5fd740 1 -- 172.21.15.28:0/140264363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b617270 msgr2=0x55b95b61e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.499+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/140264363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b617270 0x55b95b61e6c0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c005200 tx=0x7fbe9c0042f0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.499+0000 7fbebd5fd740 1 -- 172.21.15.28:0/140264363 shutdown_connections 2024-04-24T08:19:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.499+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/140264363 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55b95b6319b0 0x55b95b633da0 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.499+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/140264363 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b617270 0x55b95b61e6c0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.499+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/140264363 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b621940 0x55b95b61edb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.499+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/140264363 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b616950 0x55b95b6213d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.499+0000 7fbea1600640 0 client.16599 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:19:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.499+0000 7fbebd5fd740 1 -- 172.21.15.28:0/140264363 >> 172.21.15.28:0/140264363 conn(0x55b95b612c70 msgr2=0x55b95b618970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.499+0000 7fbebd5fd740 1 -- 172.21.15.28:0/140264363 shutdown_connections 2024-04-24T08:19:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.499+0000 7fbebd5fd740 1 -- 172.21.15.28:0/140264363 wait complete. 2024-04-24T08:19:54.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.499+0000 7fbebd5fd740 20 client.16599 trim_cache size 0 max 16384 2024-04-24T08:19:54.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.514+0000 7fbebd5fd740 1 Processor -- start 2024-04-24T08:19:54.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.514+0000 7fbebd5fd740 1 -- start start 2024-04-24T08:19:54.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbebd5fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b616950 0x55b95b612570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbebd5fd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b617270 0x55b95b622d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbebd5fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b621940 0x55b95b623270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbebd5fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb41b2880 con 0x55b95b616950 2024-04-24T08:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbebd5fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb41b0150 con 0x55b95b621940 2024-04-24T08:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbebd5fd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb41adfc0 con 0x55b95b617270 2024-04-24T08:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbeb3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b616950 0x55b95b612570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbeb3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b616950 0x55b95b612570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58314/0 (socket says 172.21.15.28:58314) 2024-04-24T08:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbeb3e00640 1 -- 172.21.15.28:0/3572262098 learned_addr learned my addr 172.21.15.28:0/3572262098 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbeb8c00640 1 --2- 172.21.15.28:0/3572262098 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b621940 0x55b95b623270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbeb3400640 1 --2- 172.21.15.28:0/3572262098 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b617270 0x55b95b622d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbeb3e00640 1 -- 172.21.15.28:0/3572262098 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b617270 msgr2=0x55b95b622d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.517 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbeb3e00640 1 --2- 172.21.15.28:0/3572262098 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b617270 0x55b95b622d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbeb3e00640 1 -- 172.21.15.28:0/3572262098 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b621940 msgr2=0x55b95b623270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbeb3e00640 1 --2- 172.21.15.28:0/3572262098 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b621940 0x55b95b623270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbeb3e00640 1 -- 172.21.15.28:0/3572262098 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b95b60e450 con 0x55b95b616950 2024-04-24T08:19:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbeb8c00640 1 --2- 172.21.15.28:0/3572262098 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b621940 0x55b95b623270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.515+0000 7fbeb3400640 1 --2- 172.21.15.28:0/3572262098 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b617270 0x55b95b622d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:19:54.518 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.516+0000 7fbeb3e00640 1 --2- 172.21.15.28:0/3572262098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b616950 0x55b95b612570 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fbea401d930 tx=0x7fbea4015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee76f41f4d527659 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.516+0000 7fbea3e00640 1 -- 172.21.15.28:0/3572262098 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea402f070 con 0x55b95b616950 2024-04-24T08:19:54.522 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.516+0000 7fbea3e00640 1 -- 172.21.15.28:0/3572262098 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbea401bc40 con 0x55b95b616950 2024-04-24T08:19:54.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.516+0000 7fbea3e00640 1 -- 172.21.15.28:0/3572262098 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbea40287c0 con 0x55b95b616950 2024-04-24T08:19:54.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.516+0000 7fbebd5fd740 1 -- 172.21.15.28:0/3572262098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b616950 msgr2=0x55b95b612570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.516+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/3572262098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b616950 0x55b95b612570 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7fbea401d930 tx=0x7fbea4015000 comp rx=0 tx=0).stop 2024-04-24T08:19:54.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.517+0000 7fbebd5fd740 1 -- 172.21.15.28:0/3572262098 shutdown_connections 2024-04-24T08:19:54.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.517+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/3572262098 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b621940 0x55b95b623270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.517+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/3572262098 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b617270 0x55b95b622d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.517+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/3572262098 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b616950 0x55b95b612570 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.523 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.517+0000 7fbebd5fd740 1 -- 172.21.15.28:0/3572262098 >> 172.21.15.28:0/3572262098 conn(0x55b95b612c70 msgr2=0x55b95b619940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.517+0000 7fbebd5fd740 1 -- 172.21.15.28:0/3572262098 shutdown_connections 2024-04-24T08:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.517+0000 7fbebd5fd740 1 -- 172.21.15.28:0/3572262098 wait complete. 2024-04-24T08:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.517+0000 7fbebd5fd740 1 -- start start 2024-04-24T08:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.517+0000 7fbebd5fd740 1 Processor -- start 2024-04-24T08:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.518+0000 7fbebd5fd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b616950 0x55b95b633390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.518+0000 7fbebd5fd740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b617270 0x55b95b6338d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.518+0000 7fbebd5fd740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b621940 0x55b95b68fb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.524 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.518+0000 7fbebd5fd740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb41ae610 con 0x55b95b617270 2024-04-24T08:19:54.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.518+0000 7fbebd5fd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb41afa80 con 0x55b95b616950 2024-04-24T08:19:54.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.518+0000 7fbebd5fd740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbeb41aece0 con 0x55b95b621940 2024-04-24T08:19:54.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.518+0000 7fbebd5fd740 10 client.0 _ll_register_callbacks cb 0x7fbeac009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:54.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.518+0000 7fbeb3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b617270 0x55b95b6338d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.518+0000 7fbeb3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b617270 0x55b95b6338d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58326/0 (socket says 172.21.15.28:58326) 2024-04-24T08:19:54.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.518+0000 7fbeb3400640 1 -- 172.21.15.28:0/1029831767 learned_addr learned my addr 172.21.15.28:0/1029831767 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:54.525 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.518+0000 7fbeb8c00640 1 --2- 172.21.15.28:0/1029831767 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b621940 0x55b95b68fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.518+0000 7fbeb3e00640 1 --2- 172.21.15.28:0/1029831767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b616950 0x55b95b633390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.518+0000 7fbeb3400640 1 -- 172.21.15.28:0/1029831767 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b621940 msgr2=0x55b95b68fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.518+0000 7fbeb3400640 1 --2- 172.21.15.28:0/1029831767 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b621940 0x55b95b68fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.518+0000 7fbeb3400640 1 -- 172.21.15.28:0/1029831767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b616950 msgr2=0x55b95b633390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.518+0000 7fbeb3400640 1 --2- 172.21.15.28:0/1029831767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b616950 0x55b95b633390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.519+0000 7fbeb3400640 1 -- 172.21.15.28:0/1029831767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b95b3587d0 con 0x55b95b617270 2024-04-24T08:19:54.526 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.519+0000 7fbeb3400640 1 --2- 172.21.15.28:0/1029831767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b617270 0x55b95b6338d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c004c80 tx=0x7fbe9c004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=613db5682fac540d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.519+0000 7fbea1600640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:19:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.519+0000 7fbea1600640 1 -- 172.21.15.28:0/1029831767 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe9c050040 con 0x55b95b617270 2024-04-24T08:19:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.519+0000 7fbea1600640 1 -- 172.21.15.28:0/1029831767 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7fbe9c0307b0 con 0x55b95b617270 2024-04-24T08:19:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.519+0000 7fbebd5fd740 1 -- 172.21.15.28:0/1029831767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbea401bc40 con 0x55b95b617270 2024-04-24T08:19:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.519+0000 7fbea1600640 1 -- 172.21.15.28:0/1029831767 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fbe9c005230 con 0x55b95b617270 2024-04-24T08:19:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.519+0000 7fbebd5fd740 10 client.16605 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.519+0000 7fbebd5fd740 1 -- 172.21.15.28:0/1029831767 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b95b698490 con 0x55b95b617270 2024-04-24T08:19:54.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.520+0000 7fbea1600640 1 -- 172.21.15.28:0/1029831767 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbea401bc40 con 0x55b95b617270 2024-04-24T08:19:54.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.520+0000 7fbea1600640 1 -- 172.21.15.28:0/1029831767 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fbe9c003680 con 0x55b95b617270 2024-04-24T08:19:54.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.520+0000 7fbebd5fd740 10 client.16605 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:54.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.520+0000 7fbebd5fd740 10 client.16605 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:19:54.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.520+0000 7fbebd5fd740 20 client.16605 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:54.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.520+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/1029831767 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55b95b698b80 0x55b95b69afa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.520+0000 7fbebd5fd740 4 client.16605 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:19:54.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.520+0000 7fbebd5fd740 1 -- 172.21.15.28:0/1029831767 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b95b583a60 con 0x55b95b698b80 2024-04-24T08:19:54.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.520+0000 7fbeb3e00640 1 --2- 172.21.15.28:0/1029831767 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55b95b698b80 0x55b95b69afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.521+0000 7fbeb3e00640 1 --2- 172.21.15.28:0/1029831767 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55b95b698b80 0x55b95b69afa0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=bd972928a7c4b084 server_cookie=42e527e599a9440e in_seq=0 out_seq=0 2024-04-24T08:19:54.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.521+0000 7fbea1600640 10 client.16605 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:19:54.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.521+0000 7fbea1600640 1 -- 172.21.15.28:0/1029831767 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b95b583a60 con 0x55b95b698b80 2024-04-24T08:19:54.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.521+0000 7fbea1600640 10 client.16605 handle_command_reply: tid=0 2024-04-24T08:19:54.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.521+0000 7fbebd5fd740 1 client.16605 shutdown 2024-04-24T08:19:54.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.521+0000 7fbebd5fd740 1 -- 172.21.15.28:0/1029831767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b617270 msgr2=0x55b95b6338d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.521+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/1029831767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b617270 0x55b95b6338d0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7fbe9c004c80 tx=0x7fbe9c004610 comp rx=0 tx=0).stop 2024-04-24T08:19:54.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.521+0000 7fbebd5fd740 1 -- 172.21.15.28:0/1029831767 shutdown_connections 2024-04-24T08:19:54.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.521+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/1029831767 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x55b95b698b80 0x55b95b69afa0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.521+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/1029831767 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b95b617270 0x55b95b6338d0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.521+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/1029831767 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b95b621940 0x55b95b68fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.521+0000 7fbea1600640 0 client.16605 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:19:54.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.521+0000 7fbebd5fd740 1 --2- 172.21.15.28:0/1029831767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b95b616950 0x55b95b633390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.521+0000 7fbebd5fd740 1 -- 172.21.15.28:0/1029831767 >> 172.21.15.28:0/1029831767 conn(0x55b95b612c70 msgr2=0x55b95b62cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.522+0000 7fbebd5fd740 1 -- 172.21.15.28:0/1029831767 shutdown_connections 2024-04-24T08:19:54.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.522+0000 7fbebd5fd740 1 -- 172.21.15.28:0/1029831767 wait complete. 2024-04-24T08:19:54.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.522+0000 7fbebd5fd740 20 client.16605 trim_cache size 0 max 16384 2024-04-24T08:19:54.531 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:19:54.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.522+0000 7fbebac00640 1 -- 172.21.15.28:0/3738358127 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fbe940a3d20 msgr2=0x7fbe940a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.522+0000 7fbebac00640 1 --2- 172.21.15.28:0/3738358127 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fbe940a3d20 0x7fbe940a61e0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x55b95b617f10 tx=0x7fbea4005f90 comp rx=0 tx=0).stop 2024-04-24T08:19:54.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.522+0000 7fbebac00640 1 -- 172.21.15.28:0/3738358127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeb410e8f0 msgr2=0x7fbeb419aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.522+0000 7fbebac00640 1 --2- 172.21.15.28:0/3738358127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeb410e8f0 0x7fbeb419aa30 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fbeac00c3a0 tx=0x7fbeac00c860 comp rx=0 tx=0).stop 2024-04-24T08:19:54.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.522+0000 7fbebac00640 1 -- 172.21.15.28:0/3738358127 shutdown_connections 2024-04-24T08:19:54.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.522+0000 7fbebac00640 1 --2- 172.21.15.28:0/3738358127 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fbe940a3d20 0x7fbe940a61e0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.522+0000 7fbebac00640 1 --2- 172.21.15.28:0/3738358127 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbeb410e8f0 0x7fbeb419aa30 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.522+0000 7fbebac00640 1 --2- 172.21.15.28:0/3738358127 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbeb4074cf0 0x7fbeb41a0d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.522+0000 7fbebac00640 1 --2- 172.21.15.28:0/3738358127 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbeb40743d0 0x7fbeb41a0830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.522+0000 7fbebac00640 1 -- 172.21.15.28:0/3738358127 >> 172.21.15.28:0/3738358127 conn(0x7fbeb40fdb60 msgr2=0x7fbeb40fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.523+0000 7fbebac00640 1 -- 172.21.15.28:0/3738358127 shutdown_connections 2024-04-24T08:19:54.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.523+0000 7fbebac00640 1 -- 172.21.15.28:0/3738358127 wait complete. 2024-04-24T08:19:54.541 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:19:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.756+0000 7f12b1a00640 1 -- 172.21.15.28:0/4189329618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12ac107260 msgr2=0x7f12ac10ede0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.756+0000 7f12b1a00640 1 --2- 172.21.15.28:0/4189329618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12ac107260 0x7f12ac10ede0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f12a4009be0 tx=0x7f12a402a6d0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.756+0000 7f12b1a00640 1 -- 172.21.15.28:0/4189329618 shutdown_connections 2024-04-24T08:19:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.756+0000 7f12b1a00640 1 --2- 172.21.15.28:0/4189329618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12ac107260 0x7f12ac10ede0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.756+0000 7f12b1a00640 1 --2- 172.21.15.28:0/4189329618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12ac101400 0x7f12ac106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.757 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.756+0000 7f12b1a00640 1 --2- 172.21.15.28:0/4189329618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ac100a50 0x7f12ac100e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.758 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.756+0000 7f12b1a00640 1 -- 172.21.15.28:0/4189329618 >> 172.21.15.28:0/4189329618 conn(0x7f12ac076090 msgr2=0x7f12ac0764a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.756+0000 7f12b1a00640 1 -- 172.21.15.28:0/4189329618 shutdown_connections 2024-04-24T08:19:54.759 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.756+0000 7f12b1a00640 1 -- 172.21.15.28:0/4189329618 wait complete. 2024-04-24T08:19:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.757+0000 7f12b1a00640 1 Processor -- start 2024-04-24T08:19:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.758+0000 7f12b1a00640 1 -- start start 2024-04-24T08:19:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.758+0000 7f12b1a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12ac100a50 0x7f12ac1a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.758+0000 7f12b1a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12ac107260 0x7f12ac1a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.758+0000 7f12b1a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ac19a840 0x7f12ac19acf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.758+0000 7f12b1a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1122d0 con 0x7f12ac19a840 2024-04-24T08:19:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.758+0000 7f12b1a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac112150 con 0x7f12ac100a50 2024-04-24T08:19:54.760 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.758+0000 7f12b1a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac112450 con 0x7f12ac107260 2024-04-24T08:19:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.758+0000 7f12b1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ac19a840 0x7f12ac19acf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.758+0000 7f12abe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12ac100a50 0x7f12ac1a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.758+0000 7f12ab400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12ac107260 0x7f12ac1a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.758+0000 7f12abe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12ac100a50 0x7f12ac1a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51382/0 (socket says 172.21.15.28:51382) 2024-04-24T08:19:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.758+0000 7f12abe00640 1 -- 172.21.15.28:0/1123712719 learned_addr learned my addr 172.21.15.28:0/1123712719 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.759+0000 7f12abe00640 1 -- 172.21.15.28:0/1123712719 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12ac107260 msgr2=0x7f12ac1a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.759+0000 7f12abe00640 1 --2- 172.21.15.28:0/1123712719 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12ac107260 0x7f12ac1a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.759+0000 7f12abe00640 1 -- 172.21.15.28:0/1123712719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ac19a840 msgr2=0x7f12ac19acf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.761 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.759+0000 7f12abe00640 1 --2- 172.21.15.28:0/1123712719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ac19a840 0x7f12ac19acf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.759+0000 7f12abe00640 1 -- 172.21.15.28:0/1123712719 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12ac19b3d0 con 0x7f12ac100a50 2024-04-24T08:19:54.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.759+0000 7f12b1000640 1 --2- 172.21.15.28:0/1123712719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ac19a840 0x7f12ac19acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:54.762 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.759+0000 7f12abe00640 1 --2- 172.21.15.28:0/1123712719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12ac100a50 0x7f12ac1a0680 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f129c00e9c0 tx=0x7f129c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.759+0000 7f12a8c00640 1 -- 172.21.15.28:0/1123712719 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129c00cde0 con 0x7f12ac100a50 2024-04-24T08:19:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.760+0000 7f12a8c00640 1 -- 172.21.15.28:0/1123712719 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f129c0035f0 con 0x7f12ac100a50 2024-04-24T08:19:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.760+0000 7f12a8c00640 1 -- 172.21.15.28:0/1123712719 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129c010650 con 0x7f12ac100a50 2024-04-24T08:19:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.760+0000 7f12b1a00640 1 -- 172.21.15.28:0/1123712719 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12ac19b6c0 con 0x7f12ac100a50 2024-04-24T08:19:54.763 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.760+0000 7f12b1a00640 1 -- 172.21.15.28:0/1123712719 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12ac1a7850 con 0x7f12ac100a50 2024-04-24T08:19:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.763+0000 7f12a8c00640 1 -- 172.21.15.28:0/1123712719 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f129c0037a0 con 0x7f12ac100a50 2024-04-24T08:19:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.763+0000 7f12b5f97740 1 Processor -- start 2024-04-24T08:19:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.763+0000 7f12b5f97740 1 -- start start 2024-04-24T08:19:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.764+0000 7f12b5f97740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805eb40 0x560e2805ef20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.764+0000 7f12b5f97740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e2805f460 0x560e280695f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.764+0000 7f12b5f97740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e28069b30 0x560e2806bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.764+0000 7f12b5f97740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560e27fc41d0 con 0x560e28069b30 2024-04-24T08:19:54.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.764+0000 7f12b5f97740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560e27f6a450 con 0x560e2805f460 2024-04-24T08:19:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.764+0000 7f12b5f97740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560e27fc0190 con 0x560e2805eb40 2024-04-24T08:19:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.764+0000 7f12ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e28069b30 0x560e2806bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.764+0000 7f12b1000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805eb40 0x560e2805ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.764+0000 7f12b1000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805eb40 0x560e2805ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44732/0 (socket says 172.21.15.28:44732) 2024-04-24T08:19:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.764+0000 7f12b1000640 1 -- 172.21.15.28:0/539755312 learned_addr learned my addr 172.21.15.28:0/539755312 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:54.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.764+0000 7f12abe00640 1 --2- 172.21.15.28:0/539755312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e2805f460 0x560e280695f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.765+0000 7f12b1000640 1 -- 172.21.15.28:0/539755312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e2805f460 msgr2=0x560e280695f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.765+0000 7f12b1000640 1 --2- 172.21.15.28:0/539755312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e2805f460 0x560e280695f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.765+0000 7f12b1000640 1 -- 172.21.15.28:0/539755312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e28069b30 msgr2=0x560e2806bf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.765+0000 7f12b1000640 1 --2- 172.21.15.28:0/539755312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e28069b30 0x560e2806bf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.765+0000 7f12b1000640 1 -- 172.21.15.28:0/539755312 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e28027310 con 0x560e2805eb40 2024-04-24T08:19:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.765+0000 7f12a8c00640 1 --2- 172.21.15.28:0/1123712719 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f12900a3d20 0x7f12900a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.765+0000 7f12b1000640 1 --2- 172.21.15.28:0/539755312 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805eb40 0x560e2805ef20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f12a4009be0 tx=0x7f12a4030de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95fa6d6e0065512e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.765+0000 7f12a8c00640 1 -- 172.21.15.28:0/1123712719 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7f129c0d0bb0 con 0x7f12ac100a50 2024-04-24T08:19:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.765+0000 7f12abe00640 1 --2- 172.21.15.28:0/1123712719 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f12900a3d20 0x7f12900a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.765+0000 7f12a1600640 1 -- 172.21.15.28:0/539755312 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a4041030 con 0x560e2805eb40 2024-04-24T08:19:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.765+0000 7f12a1600640 1 -- 172.21.15.28:0/539755312 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12a4003970 con 0x560e2805eb40 2024-04-24T08:19:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.765+0000 7f12a1600640 1 -- 172.21.15.28:0/539755312 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a4036040 con 0x560e2805eb40 2024-04-24T08:19:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.766+0000 7f12b5f97740 1 -- 172.21.15.28:0/539755312 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805eb40 msgr2=0x560e2805ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.766+0000 7f12b5f97740 1 --2- 172.21.15.28:0/539755312 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805eb40 0x560e2805ef20 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f12a4009be0 tx=0x7f12a4030de0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.766+0000 7f12abe00640 1 --2- 172.21.15.28:0/1123712719 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f12900a3d20 0x7f12900a61e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x560e28060100 tx=0x7f129c0d4660 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.766+0000 7f12b5f97740 1 -- 172.21.15.28:0/539755312 shutdown_connections 2024-04-24T08:19:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.766+0000 7f12b5f97740 1 --2- 172.21.15.28:0/539755312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e28069b30 0x560e2806bf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.766+0000 7f12b5f97740 1 --2- 172.21.15.28:0/539755312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e2805f460 0x560e280695f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.766+0000 7f12b5f97740 1 --2- 172.21.15.28:0/539755312 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805eb40 0x560e2805ef20 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.766+0000 7f12b5f97740 1 -- 172.21.15.28:0/539755312 >> 172.21.15.28:0/539755312 conn(0x560e2805ae60 msgr2=0x560e2805b240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.766+0000 7f12b5f97740 1 -- 172.21.15.28:0/539755312 shutdown_connections 2024-04-24T08:19:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.766+0000 7f12b5f97740 1 -- 172.21.15.28:0/539755312 wait complete. 2024-04-24T08:19:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.767+0000 7f12b5f97740 1 -- start start 2024-04-24T08:19:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.767+0000 7f12b5f97740 1 Processor -- start 2024-04-24T08:19:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.768+0000 7f12b5f97740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805eb40 0x560e280695d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.768+0000 7f12b5f97740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e2805f460 0x560e280668c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.768+0000 7f12b5f97740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e28069b30 0x560e28066fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.768+0000 7f12b5f97740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560e2806c960 con 0x560e28069b30 2024-04-24T08:19:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.768+0000 7f12b5f97740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560e27f7d810 con 0x560e2805f460 2024-04-24T08:19:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.768+0000 7f12b5f97740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560e2806cae0 con 0x560e2805eb40 2024-04-24T08:19:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.768+0000 7f12b5f97740 10 client.0 _ll_register_callbacks cb 0x560e27f4b140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.768+0000 7f12b1000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805eb40 0x560e280695d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.768+0000 7f12b1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e28069b30 0x560e28066fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.768+0000 7f12ab400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e2805f460 0x560e280668c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.768+0000 7f12ab400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e2805f460 0x560e280668c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51394/0 (socket says 172.21.15.28:51394) 2024-04-24T08:19:54.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.768+0000 7f12ab400640 1 -- 172.21.15.28:0/4072936281 learned_addr learned my addr 172.21.15.28:0/4072936281 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.769+0000 7f12ab400640 1 -- 172.21.15.28:0/4072936281 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805eb40 msgr2=0x560e280695d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.769+0000 7f12ab400640 1 --2- 172.21.15.28:0/4072936281 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805eb40 0x560e280695d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.769+0000 7f12ab400640 1 -- 172.21.15.28:0/4072936281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e28069b30 msgr2=0x560e28066fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.769+0000 7f12ab400640 1 --2- 172.21.15.28:0/4072936281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e28069b30 0x560e28066fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.769+0000 7f12ab400640 1 -- 172.21.15.28:0/4072936281 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e27f8e3f0 con 0x560e2805f460 2024-04-24T08:19:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.769+0000 7f12b1000640 1 --2- 172.21.15.28:0/4072936281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e28069b30 0x560e28066fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.769+0000 7f12ab400640 1 --2- 172.21.15.28:0/4072936281 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e2805f460 0x560e280668c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f12980126d0 tx=0x7f1298012b90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95bdbd9ce0e28b86 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.769+0000 7f1296000640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:19:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.769+0000 7f1296000640 1 -- 172.21.15.28:0/4072936281 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129800f920 con 0x560e2805f460 2024-04-24T08:19:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.769+0000 7f12b5f97740 1 -- 172.21.15.28:0/4072936281 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12a4003970 con 0x560e2805f460 2024-04-24T08:19:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.769+0000 7f1296000640 1 -- 172.21.15.28:0/4072936281 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7f129800fac0 con 0x560e2805f460 2024-04-24T08:19:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.770+0000 7f1296000640 1 -- 172.21.15.28:0/4072936281 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f129800fcc0 con 0x560e2805f460 2024-04-24T08:19:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.770+0000 7f12b5f97740 10 client.26894 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.770+0000 7f12b5f97740 1 -- 172.21.15.28:0/4072936281 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560e27cd63f0 con 0x560e2805f460 2024-04-24T08:19:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.770+0000 7f1296000640 1 -- 172.21.15.28:0/4072936281 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12a4003970 con 0x560e2805f460 2024-04-24T08:19:54.777 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.770+0000 7f1296000640 1 -- 172.21.15.28:0/4072936281 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f12980104c0 con 0x560e2805f460 2024-04-24T08:19:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.770+0000 7f12b5f97740 10 client.26894 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.770+0000 7f12b5f97740 10 client.26894 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:19:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.770+0000 7f12b5f97740 20 client.26894 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.770+0000 7f12b5f97740 1 --2- 172.21.15.28:0/4072936281 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560e28079db0 0x560e2807c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.770+0000 7f12b5f97740 4 client.26894 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:19:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.770+0000 7f12b5f97740 1 -- 172.21.15.28:0/4072936281 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560e27fd1bf0 con 0x560e28079db0 2024-04-24T08:19:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.771+0000 7f12b1000640 1 --2- 172.21.15.28:0/4072936281 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560e28079db0 0x560e2807c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.778 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.771+0000 7f12b1000640 1 --2- 172.21.15.28:0/4072936281 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560e28079db0 0x560e2807c1a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=24402d88d1daab70 server_cookie=12ab24abf93cf285 in_seq=0 out_seq=0 2024-04-24T08:19:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.771+0000 7f1296000640 10 client.26894 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:19:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.772+0000 7f1296000640 1 -- 172.21.15.28:0/4072936281 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560e27fd1bf0 con 0x560e28079db0 2024-04-24T08:19:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.772+0000 7f1296000640 10 client.26894 handle_command_reply: tid=0 2024-04-24T08:19:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.772+0000 7f12b5f97740 1 client.26894 shutdown 2024-04-24T08:19:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.773+0000 7f12b5f97740 1 -- 172.21.15.28:0/4072936281 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e2805f460 msgr2=0x560e280668c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.773+0000 7f12b5f97740 1 --2- 172.21.15.28:0/4072936281 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e2805f460 0x560e280668c0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f12980126d0 tx=0x7f1298012b90 comp rx=0 tx=0).stop 2024-04-24T08:19:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.773+0000 7f12b5f97740 1 -- 172.21.15.28:0/4072936281 shutdown_connections 2024-04-24T08:19:54.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.773+0000 7f12b5f97740 1 --2- 172.21.15.28:0/4072936281 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560e28079db0 0x560e2807c1a0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.773+0000 7f12b5f97740 1 --2- 172.21.15.28:0/4072936281 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e28069b30 0x560e28066fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.773+0000 7f12b5f97740 1 --2- 172.21.15.28:0/4072936281 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e2805f460 0x560e280668c0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.773+0000 7f12b5f97740 1 --2- 172.21.15.28:0/4072936281 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805eb40 0x560e280695d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.773+0000 7f1296000640 0 client.26894 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:19:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.773+0000 7f12b5f97740 1 -- 172.21.15.28:0/4072936281 >> 172.21.15.28:0/4072936281 conn(0x560e2805ae60 msgr2=0x560e28060c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.773+0000 7f12b5f97740 1 -- 172.21.15.28:0/4072936281 shutdown_connections 2024-04-24T08:19:54.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.773+0000 7f12b5f97740 1 -- 172.21.15.28:0/4072936281 wait complete. 2024-04-24T08:19:54.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.773+0000 7f12b5f97740 20 client.26894 trim_cache size 0 max 16384 2024-04-24T08:19:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.788+0000 7f12b5f97740 1 Processor -- start 2024-04-24T08:19:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.788+0000 7f12b5f97740 1 -- start start 2024-04-24T08:19:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.788+0000 7f12b5f97740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2805eb40 0x560e28075850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.788+0000 7f12b5f97740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805f460 0x560e28069100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.788+0000 7f12b5f97740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e28069b30 0x560e28081a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.788+0000 7f12b5f97740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1b27c0 con 0x560e2805eb40 2024-04-24T08:19:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.788+0000 7f12b5f97740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1b0090 con 0x560e28069b30 2024-04-24T08:19:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.788+0000 7f12b5f97740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1adf00 con 0x560e2805f460 2024-04-24T08:19:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.788+0000 7f12b1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2805eb40 0x560e28075850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.790 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.789+0000 7f12ab400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805f460 0x560e28069100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.789+0000 7f12b1000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2805eb40 0x560e28075850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58374/0 (socket says 172.21.15.28:58374) 2024-04-24T08:19:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.789+0000 7f12b1000640 1 -- 172.21.15.28:0/1327208092 learned_addr learned my addr 172.21.15.28:0/1327208092 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:54.791 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.789+0000 7f12b1000640 1 --2- 172.21.15.28:0/1327208092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e28069b30 0x560e28081a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.789+0000 7f12ab400640 1 -- 172.21.15.28:0/1327208092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e28069b30 msgr2=0x560e28081a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.789+0000 7f12ab400640 1 --2- 172.21.15.28:0/1327208092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e28069b30 0x560e28081a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.789+0000 7f12ab400640 1 -- 172.21.15.28:0/1327208092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2805eb40 msgr2=0x560e28075850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.789+0000 7f12ab400640 1 --2- 172.21.15.28:0/1327208092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2805eb40 0x560e28075850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.796 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.789+0000 7f12ab400640 1 -- 172.21.15.28:0/1327208092 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560e27da07d0 con 0x560e2805f460 2024-04-24T08:19:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.789+0000 7f12ab400640 1 --2- 172.21.15.28:0/1327208092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805f460 0x560e28069100 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1298010ad0 tx=0x7f1298011f10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f04358ade9dc74e2 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.789+0000 7f12a0c00640 1 -- 172.21.15.28:0/1327208092 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1298028070 con 0x560e2805f460 2024-04-24T08:19:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.789+0000 7f12a0c00640 1 -- 172.21.15.28:0/1327208092 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1298005560 con 0x560e2805f460 2024-04-24T08:19:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.789+0000 7f12a0c00640 1 -- 172.21.15.28:0/1327208092 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1298010b50 con 0x560e2805f460 2024-04-24T08:19:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.790+0000 7f12b5f97740 1 -- 172.21.15.28:0/1327208092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805f460 msgr2=0x560e28069100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.797 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.790+0000 7f12b5f97740 1 --2- 172.21.15.28:0/1327208092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805f460 0x560e28069100 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f1298010ad0 tx=0x7f1298011f10 comp rx=0 tx=0).stop 2024-04-24T08:19:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.790+0000 7f12b5f97740 1 -- 172.21.15.28:0/1327208092 shutdown_connections 2024-04-24T08:19:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.790+0000 7f12b5f97740 1 --2- 172.21.15.28:0/1327208092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e28069b30 0x560e28081a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.790+0000 7f12b5f97740 1 --2- 172.21.15.28:0/1327208092 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805f460 0x560e28069100 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.790+0000 7f12b5f97740 1 --2- 172.21.15.28:0/1327208092 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2805eb40 0x560e28075850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.790+0000 7f12b5f97740 1 -- 172.21.15.28:0/1327208092 >> 172.21.15.28:0/1327208092 conn(0x560e2805ae60 msgr2=0x560e28061bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.790+0000 7f12b5f97740 1 -- 172.21.15.28:0/1327208092 shutdown_connections 2024-04-24T08:19:54.798 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.790+0000 7f12b5f97740 1 -- 172.21.15.28:0/1327208092 wait complete. 2024-04-24T08:19:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.791+0000 7f12b5f97740 1 -- start start 2024-04-24T08:19:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.791+0000 7f12b5f97740 1 Processor -- start 2024-04-24T08:19:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.791+0000 7f12b5f97740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805eb40 0x560e2806ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.791+0000 7f12b5f97740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2805f460 0x560e2806b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.791+0000 7f12b5f97740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e28069b30 0x560e28070250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.799 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.791+0000 7f12b5f97740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1ae550 con 0x560e2805f460 2024-04-24T08:19:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.791+0000 7f12b5f97740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1af9c0 con 0x560e28069b30 2024-04-24T08:19:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.791+0000 7f12b5f97740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1aec20 con 0x560e2805eb40 2024-04-24T08:19:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.791+0000 7f12b5f97740 10 client.0 _ll_register_callbacks cb 0x7f1298009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.791+0000 7f12ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2805f460 0x560e2806b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.792+0000 7f12b1000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e28069b30 0x560e28070250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.792+0000 7f12ab400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2805f460 0x560e2806b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58382/0 (socket says 172.21.15.28:58382) 2024-04-24T08:19:54.800 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.792+0000 7f12ab400640 1 -- 172.21.15.28:0/3000122259 learned_addr learned my addr 172.21.15.28:0/3000122259 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.792+0000 7f12b1000640 1 --2- 172.21.15.28:0/3000122259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805eb40 0x560e2806ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.792+0000 7f12ab400640 1 -- 172.21.15.28:0/3000122259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805eb40 msgr2=0x560e2806ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.792+0000 7f12ab400640 1 --2- 172.21.15.28:0/3000122259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805eb40 0x560e2806ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.792+0000 7f12ab400640 1 -- 172.21.15.28:0/3000122259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e28069b30 msgr2=0x560e28070250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.792+0000 7f12ab400640 1 --2- 172.21.15.28:0/3000122259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e28069b30 0x560e28070250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.792+0000 7f12ab400640 1 -- 172.21.15.28:0/3000122259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560e2806ba10 con 0x560e2805f460 2024-04-24T08:19:54.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.792+0000 7f12ab400640 1 --2- 172.21.15.28:0/3000122259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2805f460 0x560e2806b180 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1298000c00 tx=0x7f1298010660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78f6c0723bcffcec server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.792+0000 7f1296000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:19:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.792+0000 7f1296000640 1 -- 172.21.15.28:0/3000122259 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1298031060 con 0x560e2805f460 2024-04-24T08:19:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.793+0000 7f12b5f97740 1 -- 172.21.15.28:0/3000122259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1298005560 con 0x560e2805f460 2024-04-24T08:19:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.793+0000 7f1296000640 1 -- 172.21.15.28:0/3000122259 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7f1298010d10 con 0x560e2805f460 2024-04-24T08:19:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.793+0000 7f1296000640 1 -- 172.21.15.28:0/3000122259 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f1298028040 con 0x560e2805f460 2024-04-24T08:19:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.793+0000 7f12b5f97740 10 client.16623 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:54.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.793+0000 7f12b5f97740 1 -- 172.21.15.28:0/3000122259 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560e280e08c0 con 0x560e2805f460 2024-04-24T08:19:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.793+0000 7f1296000640 1 -- 172.21.15.28:0/3000122259 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1298005560 con 0x560e2805f460 2024-04-24T08:19:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.793+0000 7f1296000640 1 -- 172.21.15.28:0/3000122259 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f1298003ae0 con 0x560e2805f460 2024-04-24T08:19:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.793+0000 7f12b5f97740 10 client.16623 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.793+0000 7f12b5f97740 10 client.16623 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:19:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.793+0000 7f12b5f97740 20 client.16623 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.793+0000 7f12b5f97740 1 --2- 172.21.15.28:0/3000122259 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560e280e0fe0 0x560e280e3400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.793+0000 7f12b5f97740 4 client.16623 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:19:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.793+0000 7f12b5f97740 1 -- 172.21.15.28:0/3000122259 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560e27fcba60 con 0x560e280e0fe0 2024-04-24T08:19:54.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.794+0000 7f12b1000640 1 --2- 172.21.15.28:0/3000122259 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560e280e0fe0 0x560e280e3400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:54.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.794+0000 7f12b1000640 1 --2- 172.21.15.28:0/3000122259 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560e280e0fe0 0x560e280e3400 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=aa5ca54ca760a98d server_cookie=520a19dae2296f0 in_seq=0 out_seq=0 2024-04-24T08:19:54.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.794+0000 7f1296000640 10 client.16623 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:19:54.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.795+0000 7f1296000640 1 -- 172.21.15.28:0/3000122259 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560e27fcba60 con 0x560e280e0fe0 2024-04-24T08:19:54.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.795+0000 7f1296000640 10 client.16623 handle_command_reply: tid=0 2024-04-24T08:19:54.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.795+0000 7f12b5f97740 1 client.16623 shutdown 2024-04-24T08:19:54.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.795+0000 7f12b5f97740 1 -- 172.21.15.28:0/3000122259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2805f460 msgr2=0x560e2806b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.795+0000 7f12b5f97740 1 --2- 172.21.15.28:0/3000122259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2805f460 0x560e2806b180 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f1298000c00 tx=0x7f1298010660 comp rx=0 tx=0).stop 2024-04-24T08:19:54.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.795+0000 7f12b5f97740 1 -- 172.21.15.28:0/3000122259 shutdown_connections 2024-04-24T08:19:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.795+0000 7f12b5f97740 1 --2- 172.21.15.28:0/3000122259 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x560e280e0fe0 0x560e280e3400 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.795+0000 7f12b5f97740 1 --2- 172.21.15.28:0/3000122259 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560e2805f460 0x560e2806b180 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.795+0000 7f12b5f97740 1 --2- 172.21.15.28:0/3000122259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560e28069b30 0x560e28070250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.795+0000 7f12b5f97740 1 --2- 172.21.15.28:0/3000122259 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560e2805eb40 0x560e2806ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.795+0000 7f12b5f97740 1 -- 172.21.15.28:0/3000122259 >> 172.21.15.28:0/3000122259 conn(0x560e2805ae60 msgr2=0x560e280596c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.795+0000 7f1296000640 0 client.16623 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:19:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.795+0000 7f12b5f97740 1 -- 172.21.15.28:0/3000122259 shutdown_connections 2024-04-24T08:19:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.795+0000 7f12b5f97740 1 -- 172.21.15.28:0/3000122259 wait complete. 2024-04-24T08:19:54.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.795+0000 7f12b5f97740 20 client.16623 trim_cache size 0 max 16384 2024-04-24T08:19:54.806 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:19:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.796+0000 7f12b1a00640 1 -- 172.21.15.28:0/1123712719 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f12900a3d20 msgr2=0x7f12900a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.796+0000 7f12b1a00640 1 --2- 172.21.15.28:0/1123712719 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f12900a3d20 0x7f12900a61e0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x560e28060100 tx=0x7f129c0d4660 comp rx=0 tx=0).stop 2024-04-24T08:19:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.796+0000 7f12b1a00640 1 -- 172.21.15.28:0/1123712719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12ac100a50 msgr2=0x7f12ac1a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.796+0000 7f12b1a00640 1 --2- 172.21.15.28:0/1123712719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12ac100a50 0x7f12ac1a0680 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f129c00e9c0 tx=0x7f129c00ee80 comp rx=0 tx=0).stop 2024-04-24T08:19:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.796+0000 7f12b1a00640 1 -- 172.21.15.28:0/1123712719 shutdown_connections 2024-04-24T08:19:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.796+0000 7f12b1a00640 1 --2- 172.21.15.28:0/1123712719 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f12900a3d20 0x7f12900a61e0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.796+0000 7f12b1a00640 1 --2- 172.21.15.28:0/1123712719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f12ac19a840 0x7f12ac19acf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.807 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.796+0000 7f12b1a00640 1 --2- 172.21.15.28:0/1123712719 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f12ac107260 0x7f12ac1a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.796+0000 7f12b1a00640 1 --2- 172.21.15.28:0/1123712719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12ac100a50 0x7f12ac1a0680 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.796+0000 7f12b1a00640 1 -- 172.21.15.28:0/1123712719 >> 172.21.15.28:0/1123712719 conn(0x7f12ac076090 msgr2=0x7f12ac0fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.796+0000 7f12b1a00640 1 -- 172.21.15.28:0/1123712719 shutdown_connections 2024-04-24T08:19:54.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:54.796+0000 7f12b1a00640 1 -- 172.21.15.28:0/1123712719 wait complete. 2024-04-24T08:19:54.812 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:54 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3034863138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:19:54.816 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:19:55.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.028+0000 7f9d7a800640 1 -- 172.21.15.28:0/752993164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d74104cd0 msgr2=0x7f9d74108d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.028+0000 7f9d7a800640 1 --2- 172.21.15.28:0/752993164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d74104cd0 0x7f9d74108d10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f9d64009ca0 tx=0x7f9d6402a8b0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.029+0000 7f9d7a800640 1 -- 172.21.15.28:0/752993164 shutdown_connections 2024-04-24T08:19:55.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.029+0000 7f9d7a800640 1 --2- 172.21.15.28:0/752993164 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d741093f0 0x7f9d74110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.029+0000 7f9d7a800640 1 --2- 172.21.15.28:0/752993164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d74104cd0 0x7f9d74108d10 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.029+0000 7f9d7a800640 1 --2- 172.21.15.28:0/752993164 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d74104320 0x7f9d74104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.029+0000 7f9d7a800640 1 -- 172.21.15.28:0/752993164 >> 172.21.15.28:0/752993164 conn(0x7f9d740ffb40 msgr2=0x7f9d74101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:55.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.029+0000 7f9d7a800640 1 -- 172.21.15.28:0/752993164 shutdown_connections 2024-04-24T08:19:55.030 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.029+0000 7f9d7a800640 1 -- 172.21.15.28:0/752993164 wait complete. 2024-04-24T08:19:55.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.030+0000 7f9d7a800640 1 Processor -- start 2024-04-24T08:19:55.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.030+0000 7f9d7a800640 1 -- start start 2024-04-24T08:19:55.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.030+0000 7f9d7a800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d74104320 0x7f9d741a0640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.030+0000 7f9d7a800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d741093f0 0x7f9d741a0b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.030+0000 7f9d7a800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d7419a7b0 0x7f9d7419ac60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.030+0000 7f9d7a800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d741144d0 con 0x7f9d74104320 2024-04-24T08:19:55.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.030+0000 7f9d7a800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9d74114350 con 0x7f9d7419a7b0 2024-04-24T08:19:55.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.030+0000 7f9d7a800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9d74114650 con 0x7f9d741093f0 2024-04-24T08:19:55.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.030+0000 7f9d79400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d74104320 0x7f9d741a0640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.030+0000 7f9d79e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d7419a7b0 0x7f9d7419ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.031+0000 7f9d79e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d7419a7b0 0x7f9d7419ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51432/0 (socket says 172.21.15.28:51432) 2024-04-24T08:19:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.031+0000 7f9d79e00640 1 -- 172.21.15.28:0/2063297282 learned_addr learned my addr 172.21.15.28:0/2063297282 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.031+0000 7f9d78a00640 1 --2- 172.21.15.28:0/2063297282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d741093f0 0x7f9d741a0b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.031+0000 7f9d79e00640 1 -- 172.21.15.28:0/2063297282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d741093f0 msgr2=0x7f9d741a0b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.031+0000 7f9d79e00640 1 --2- 172.21.15.28:0/2063297282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d741093f0 0x7f9d741a0b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.031+0000 7f9d79e00640 1 -- 172.21.15.28:0/2063297282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d74104320 msgr2=0x7f9d741a0640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.031+0000 7f9d79e00640 1 --2- 172.21.15.28:0/2063297282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d74104320 0x7f9d741a0640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.031+0000 7f9d79e00640 1 -- 172.21.15.28:0/2063297282 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d7419b340 con 0x7f9d7419a7b0 2024-04-24T08:19:55.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.031+0000 7f9d79400640 1 --2- 172.21.15.28:0/2063297282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d74104320 0x7f9d741a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:55.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.031+0000 7f9d79e00640 1 --2- 172.21.15.28:0/2063297282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d7419a7b0 0x7f9d7419ac60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00e970 tx=0x7f9d6c00ed70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:55.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.032+0000 7f9d72000640 1 -- 172.21.15.28:0/2063297282 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c00ccc0 con 0x7f9d7419a7b0 2024-04-24T08:19:55.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.032+0000 7f9d7a800640 1 -- 172.21.15.28:0/2063297282 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d7419b630 con 0x7f9d7419a7b0 2024-04-24T08:19:55.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.032+0000 7f9d72000640 1 -- 172.21.15.28:0/2063297282 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d6c00ce60 con 0x7f9d7419a7b0 2024-04-24T08:19:55.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.032+0000 7f9d72000640 1 -- 172.21.15.28:0/2063297282 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c010450 con 0x7f9d7419a7b0 2024-04-24T08:19:55.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.032+0000 7f9d7a800640 1 -- 172.21.15.28:0/2063297282 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d741a7830 con 0x7f9d7419a7b0 2024-04-24T08:19:55.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.035+0000 7f9d7edb7740 1 Processor -- start 2024-04-24T08:19:55.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.035+0000 7f9d72000640 1 -- 172.21.15.28:0/2063297282 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9d6c005020 con 0x7f9d7419a7b0 2024-04-24T08:19:55.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.035+0000 7f9d7edb7740 1 -- start start 2024-04-24T08:19:55.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.036+0000 7f9d7edb7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f3b40 0x55630e3f3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.036+0000 7f9d7edb7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f4460 0x55630e3fe5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.036+0000 7f9d7edb7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 0x55630e400f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.036+0000 7f9d7edb7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55630e3591d0 con 0x55630e3feb30 2024-04-24T08:19:55.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.036+0000 7f9d7edb7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55630e2ff450 con 0x55630e3f3b40 2024-04-24T08:19:55.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.036+0000 7f9d7edb7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55630e355190 con 0x55630e3f4460 2024-04-24T08:19:55.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.036+0000 7f9d79400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f4460 0x55630e3fe5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.036+0000 7f9d79400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f4460 0x55630e3fe5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44796/0 (socket says 172.21.15.28:44796) 2024-04-24T08:19:55.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.036+0000 7f9d79400640 1 -- 172.21.15.28:0/3645950995 learned_addr learned my addr 172.21.15.28:0/3645950995 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:55.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.036+0000 7f9d78a00640 1 --2- 172.21.15.28:0/3645950995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f3b40 0x55630e3f3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.036+0000 7f9d78a00640 1 --2- 172.21.15.28:0/3645950995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 0x55630e400f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.037+0000 7f9d79400640 1 -- 172.21.15.28:0/3645950995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f3b40 msgr2=0x55630e3f3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.037+0000 7f9d79400640 1 --2- 172.21.15.28:0/3645950995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f3b40 0x55630e3f3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.037+0000 7f9d79400640 1 -- 172.21.15.28:0/3645950995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 msgr2=0x55630e400f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.037+0000 7f9d79400640 1 --2- 172.21.15.28:0/3645950995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 0x55630e400f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.037+0000 7f9d79400640 1 -- 172.21.15.28:0/3645950995 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55630e3bc310 con 0x55630e3f4460 2024-04-24T08:19:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.037+0000 7f9d78a00640 1 --2- 172.21.15.28:0/3645950995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 0x55630e400f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:55.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.037+0000 7f9d79400640 1 --2- 172.21.15.28:0/3645950995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f4460 0x55630e3fe5f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f9d680115f0 tx=0x7f9d68011ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf1c3bc2b0c163f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:55.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.037+0000 7f9d72000640 1 --2- 172.21.15.28:0/2063297282 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d5c0a3da0 0x7f9d5c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.037+0000 7f9d62600640 1 -- 172.21.15.28:0/3645950995 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6801e070 con 0x55630e3f4460 2024-04-24T08:19:55.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.037+0000 7f9d62600640 1 -- 172.21.15.28:0/3645950995 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d6800fb20 con 0x55630e3f4460 2024-04-24T08:19:55.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.037+0000 7f9d62600640 1 -- 172.21.15.28:0/3645950995 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d68019c40 con 0x55630e3f4460 2024-04-24T08:19:55.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.037+0000 7f9d72000640 1 -- 172.21.15.28:0/2063297282 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7f9d6c0d0be0 con 0x7f9d7419a7b0 2024-04-24T08:19:55.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.038+0000 7f9d78a00640 1 --2- 172.21.15.28:0/2063297282 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d5c0a3da0 0x7f9d5c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.046 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.038+0000 7f9d7edb7740 1 -- 172.21.15.28:0/3645950995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f4460 msgr2=0x55630e3fe5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.038+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/3645950995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f4460 0x55630e3fe5f0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f9d680115f0 tx=0x7f9d68011ab0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.038+0000 7f9d7edb7740 1 -- 172.21.15.28:0/3645950995 shutdown_connections 2024-04-24T08:19:55.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.038+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/3645950995 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 0x55630e400f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.038+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/3645950995 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f4460 0x55630e3fe5f0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.038+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/3645950995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f3b40 0x55630e3f3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.038+0000 7f9d7edb7740 1 -- 172.21.15.28:0/3645950995 >> 172.21.15.28:0/3645950995 conn(0x55630e3efe60 msgr2=0x55630e3f0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:55.047 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.038+0000 7f9d78a00640 1 --2- 172.21.15.28:0/2063297282 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d5c0a3da0 0x7f9d5c0a6260 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9d64004af0 tx=0x7f9d64009820 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:55.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.038+0000 7f9d7edb7740 1 -- 172.21.15.28:0/3645950995 shutdown_connections 2024-04-24T08:19:55.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.039+0000 7f9d7edb7740 1 -- 172.21.15.28:0/3645950995 wait complete. 2024-04-24T08:19:55.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.039+0000 7f9d7edb7740 1 -- start start 2024-04-24T08:19:55.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.039+0000 7f9d7edb7740 1 Processor -- start 2024-04-24T08:19:55.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.040+0000 7f9d7edb7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3f3b40 0x55630e3fe510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.040+0000 7f9d7edb7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 0x55630e3fb800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.040+0000 7f9d7edb7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3feb30 0x55630e3fbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.048 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.040+0000 7f9d7edb7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55630e4019c0 con 0x55630e3f3b40 2024-04-24T08:19:55.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.040+0000 7f9d7edb7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55630e312810 con 0x55630e3f4460 2024-04-24T08:19:55.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.040+0000 7f9d7edb7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55630e401b40 con 0x55630e3feb30 2024-04-24T08:19:55.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.040+0000 7f9d7edb7740 10 client.0 _ll_register_callbacks cb 0x55630e2e0140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:55.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.040+0000 7f9d79400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3f3b40 0x55630e3fe510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.040+0000 7f9d79400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3f3b40 0x55630e3fe510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58420/0 (socket says 172.21.15.28:58420) 2024-04-24T08:19:55.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.040+0000 7f9d79400640 1 -- 172.21.15.28:0/466032121 learned_addr learned my addr 172.21.15.28:0/466032121 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:55.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.040+0000 7f9d79e00640 1 --2- 172.21.15.28:0/466032121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3feb30 0x55630e3fbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.040+0000 7f9d78a00640 1 --2- 172.21.15.28:0/466032121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 0x55630e3fb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.049 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.040+0000 7f9d79400640 1 -- 172.21.15.28:0/466032121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3feb30 msgr2=0x55630e3fbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:55.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.040+0000 7f9d79400640 1 --2- 172.21.15.28:0/466032121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3feb30 0x55630e3fbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.040+0000 7f9d79400640 1 -- 172.21.15.28:0/466032121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 msgr2=0x55630e3fb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:55.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.040+0000 7f9d79400640 1 --2- 172.21.15.28:0/466032121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 0x55630e3fb800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.040+0000 7f9d79400640 1 -- 172.21.15.28:0/466032121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55630e3233f0 con 0x55630e3f3b40 2024-04-24T08:19:55.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.041+0000 7f9d79400640 1 --2- 172.21.15.28:0/466032121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3f3b40 0x55630e3fe510 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9d6800f850 tx=0x7f9d6800f880 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec97f6cda73c44b9 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:55.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.041+0000 7f9d56a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:19:55.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.041+0000 7f9d56a00640 1 -- 172.21.15.28:0/466032121 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d68027060 con 0x55630e3f3b40 2024-04-24T08:19:55.050 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.041+0000 7f9d56a00640 1 -- 172.21.15.28:0/466032121 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7f9d680199c0 con 0x55630e3f3b40 2024-04-24T08:19:55.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.041+0000 7f9d7edb7740 1 -- 172.21.15.28:0/466032121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d6800fb20 con 0x55630e3f3b40 2024-04-24T08:19:55.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.041+0000 7f9d56a00640 1 -- 172.21.15.28:0/466032121 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9d6801e040 con 0x55630e3f3b40 2024-04-24T08:19:55.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.041+0000 7f9d7edb7740 10 client.16635 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:55.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.041+0000 7f9d7edb7740 1 -- 172.21.15.28:0/466032121 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55630e06b3f0 con 0x55630e3f3b40 2024-04-24T08:19:55.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.042+0000 7f9d56a00640 1 -- 172.21.15.28:0/466032121 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d6800fb20 con 0x55630e3f3b40 2024-04-24T08:19:55.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.042+0000 7f9d56a00640 1 -- 172.21.15.28:0/466032121 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9d68021030 con 0x55630e3f3b40 2024-04-24T08:19:55.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.042+0000 7f9d7edb7740 10 client.16635 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:55.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.042+0000 7f9d7edb7740 10 client.16635 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:19:55.051 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.042+0000 7f9d7edb7740 20 client.16635 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:55.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.042+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/466032121 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55630e40eb90 0x55630e410f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.042+0000 7f9d7edb7740 4 client.16635 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:19:55.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.042+0000 7f9d7edb7740 1 -- 172.21.15.28:0/466032121 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55630e366bf0 con 0x55630e40eb90 2024-04-24T08:19:55.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.042+0000 7f9d78a00640 1 --2- 172.21.15.28:0/466032121 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55630e40eb90 0x55630e410f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.043+0000 7f9d78a00640 1 --2- 172.21.15.28:0/466032121 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55630e40eb90 0x55630e410f80 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=302f90c5c4d5dff6 server_cookie=730f0837738e84cf in_seq=0 out_seq=0 2024-04-24T08:19:55.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.043+0000 7f9d56a00640 10 client.16635 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:55.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.044+0000 7f9d56a00640 1 -- 172.21.15.28:0/466032121 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55630e366bf0 con 0x55630e40eb90 2024-04-24T08:19:55.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.044+0000 7f9d56a00640 10 client.16635 handle_command_reply: tid=0 2024-04-24T08:19:55.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.044+0000 7f9d7edb7740 1 client.16635 shutdown 2024-04-24T08:19:55.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.044+0000 7f9d7edb7740 1 -- 172.21.15.28:0/466032121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3f3b40 msgr2=0x55630e3fe510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.044+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/466032121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3f3b40 0x55630e3fe510 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f9d6800f850 tx=0x7f9d6800f880 comp rx=0 tx=0).stop 2024-04-24T08:19:55.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.044+0000 7f9d7edb7740 1 -- 172.21.15.28:0/466032121 shutdown_connections 2024-04-24T08:19:55.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.044+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/466032121 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55630e40eb90 0x55630e410f80 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.044+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/466032121 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3feb30 0x55630e3fbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.044+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/466032121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 0x55630e3fb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.053 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.044+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/466032121 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3f3b40 0x55630e3fe510 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.044+0000 7f9d56a00640 0 client.16635 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:55.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.044+0000 7f9d7edb7740 1 -- 172.21.15.28:0/466032121 >> 172.21.15.28:0/466032121 conn(0x55630e3efe60 msgr2=0x55630e3f5c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:55.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.044+0000 7f9d7edb7740 1 -- 172.21.15.28:0/466032121 shutdown_connections 2024-04-24T08:19:55.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.044+0000 7f9d7edb7740 1 -- 172.21.15.28:0/466032121 wait complete. 2024-04-24T08:19:55.054 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.045+0000 7f9d7edb7740 20 client.16635 trim_cache size 0 max 16384 2024-04-24T08:19:55.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.059+0000 7f9d7edb7740 1 Processor -- start 2024-04-24T08:19:55.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.059+0000 7f9d7edb7740 1 -- start start 2024-04-24T08:19:55.060 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.060+0000 7f9d7edb7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f3b40 0x55630e40a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.060+0000 7f9d7edb7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 0x55630e3fe040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.060+0000 7f9d7edb7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 0x55630e416640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.060+0000 7f9d7edb7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d741b2830 con 0x55630e3feb30 2024-04-24T08:19:55.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.060+0000 7f9d7edb7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9d741b0100 con 0x55630e3f4460 2024-04-24T08:19:55.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.060+0000 7f9d7edb7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9d741adf70 con 0x55630e3f3b40 2024-04-24T08:19:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.060+0000 7f9d79e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 0x55630e416640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.060+0000 7f9d78a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 0x55630e3fe040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.060+0000 7f9d79400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f3b40 0x55630e40a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.060+0000 7f9d78a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 0x55630e3fe040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51452/0 (socket says 172.21.15.28:51452) 2024-04-24T08:19:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.060+0000 7f9d78a00640 1 -- 172.21.15.28:0/1352444770 learned_addr learned my addr 172.21.15.28:0/1352444770 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.060+0000 7f9d79e00640 1 -- 172.21.15.28:0/1352444770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f3b40 msgr2=0x55630e40a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:55.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.060+0000 7f9d79e00640 1 --2- 172.21.15.28:0/1352444770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f3b40 0x55630e40a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.060+0000 7f9d79e00640 1 -- 172.21.15.28:0/1352444770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 msgr2=0x55630e3fe040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.060+0000 7f9d79e00640 1 --2- 172.21.15.28:0/1352444770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 0x55630e3fe040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.060+0000 7f9d79e00640 1 -- 172.21.15.28:0/1352444770 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55630e3fc180 con 0x55630e3feb30 2024-04-24T08:19:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.061+0000 7f9d78a00640 1 --2- 172.21.15.28:0/1352444770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 0x55630e3fe040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.061+0000 7f9d79400640 1 --2- 172.21.15.28:0/1352444770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f3b40 0x55630e40a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.061+0000 7f9d79e00640 1 --2- 172.21.15.28:0/1352444770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 0x55630e416640 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0ce280 tx=0x7f9d6c09c430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29fd7090d3dd29a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:55.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.061+0000 7f9d61c00640 1 -- 172.21.15.28:0/1352444770 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c0a0e80 con 0x55630e3feb30 2024-04-24T08:19:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.061+0000 7f9d61c00640 1 -- 172.21.15.28:0/1352444770 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d6c0a5740 con 0x55630e3feb30 2024-04-24T08:19:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.061+0000 7f9d61c00640 1 -- 172.21.15.28:0/1352444770 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d6c099140 con 0x55630e3feb30 2024-04-24T08:19:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.062+0000 7f9d7edb7740 1 -- 172.21.15.28:0/1352444770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 msgr2=0x55630e416640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.062+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/1352444770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 0x55630e416640 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c0ce280 tx=0x7f9d6c09c430 comp rx=0 tx=0).stop 2024-04-24T08:19:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.062+0000 7f9d7edb7740 1 -- 172.21.15.28:0/1352444770 shutdown_connections 2024-04-24T08:19:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.062+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/1352444770 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 0x55630e416640 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.062+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/1352444770 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 0x55630e3fe040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.062+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/1352444770 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f3b40 0x55630e40a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.062+0000 7f9d7edb7740 1 -- 172.21.15.28:0/1352444770 >> 172.21.15.28:0/1352444770 conn(0x55630e3efe60 msgr2=0x55630e3f0240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:55.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.062+0000 7f9d7edb7740 1 -- 172.21.15.28:0/1352444770 shutdown_connections 2024-04-24T08:19:55.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.062+0000 7f9d7edb7740 1 -- 172.21.15.28:0/1352444770 wait complete. 2024-04-24T08:19:55.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.062+0000 7f9d7edb7740 1 -- start start 2024-04-24T08:19:55.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.062+0000 7f9d7edb7740 1 Processor -- start 2024-04-24T08:19:55.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.063+0000 7f9d7edb7740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f3b40 0x55630e3feff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.063+0000 7f9d7edb7740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 0x55630e3ff530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.063+0000 7f9d7edb7740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 0x55630e3ffab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.063+0000 7f9d7edb7740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d741ae5c0 con 0x55630e3feb30 2024-04-24T08:19:55.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.063+0000 7f9d7edb7740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9d741afa30 con 0x55630e3f4460 2024-04-24T08:19:55.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.063+0000 7f9d7edb7740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9d741aec90 con 0x55630e3f3b40 2024-04-24T08:19:55.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.063+0000 7f9d7edb7740 10 client.0 _ll_register_callbacks cb 0x7f9d6801c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:55.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.063+0000 7f9d78a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 0x55630e3ff530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.063+0000 7f9d79400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f3b40 0x55630e3feff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.063+0000 7f9d78a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 0x55630e3ff530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51466/0 (socket says 172.21.15.28:51466) 2024-04-24T08:19:55.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.063+0000 7f9d78a00640 1 -- 172.21.15.28:0/873371451 learned_addr learned my addr 172.21.15.28:0/873371451 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:55.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.063+0000 7f9d79e00640 1 --2- 172.21.15.28:0/873371451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 0x55630e3ffab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.064+0000 7f9d79400640 1 -- 172.21.15.28:0/873371451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 msgr2=0x55630e3ff530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.064+0000 7f9d79400640 1 --2- 172.21.15.28:0/873371451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 0x55630e3ff530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.064+0000 7f9d79400640 1 -- 172.21.15.28:0/873371451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 msgr2=0x55630e3ffab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.064+0000 7f9d79400640 1 --2- 172.21.15.28:0/873371451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 0x55630e3ffab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.064+0000 7f9d79400640 1 -- 172.21.15.28:0/873371451 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55630e1357d0 con 0x55630e3f3b40 2024-04-24T08:19:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.064+0000 7f9d79e00640 1 --2- 172.21.15.28:0/873371451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 0x55630e3ffab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.064+0000 7f9d79400640 1 --2- 172.21.15.28:0/873371451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f3b40 0x55630e3feff0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9d68005290 tx=0x7f9d680196c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f16efe539663792 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:55.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.064+0000 7f9d56a00640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:19:55.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.064+0000 7f9d56a00640 1 -- 172.21.15.28:0/873371451 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d68027040 con 0x55630e3f3b40 2024-04-24T08:19:55.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.064+0000 7f9d56a00640 1 -- 172.21.15.28:0/873371451 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7f9d68011250 con 0x55630e3f3b40 2024-04-24T08:19:55.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.064+0000 7f9d7edb7740 1 -- 172.21.15.28:0/873371451 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d6c0a5740 con 0x55630e3f3b40 2024-04-24T08:19:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.065+0000 7f9d56a00640 1 -- 172.21.15.28:0/873371451 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9d68039050 con 0x55630e3f3b40 2024-04-24T08:19:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.065+0000 7f9d7edb7740 10 client.26932 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.065+0000 7f9d7edb7740 1 -- 172.21.15.28:0/873371451 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55630e4071b0 con 0x55630e3f3b40 2024-04-24T08:19:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.065+0000 7f9d56a00640 1 -- 172.21.15.28:0/873371451 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d6c0a5740 con 0x55630e3f3b40 2024-04-24T08:19:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.065+0000 7f9d56a00640 1 -- 172.21.15.28:0/873371451 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9d68031020 con 0x55630e3f3b40 2024-04-24T08:19:55.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.065+0000 7f9d7edb7740 10 client.26932 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.065+0000 7f9d7edb7740 10 client.26932 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:19:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.065+0000 7f9d7edb7740 20 client.26932 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.065+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/873371451 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55630e474f00 0x55630e477320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.065+0000 7f9d7edb7740 4 client.26932 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:19:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.065+0000 7f9d7edb7740 1 -- 172.21.15.28:0/873371451 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55630e360a60 con 0x55630e474f00 2024-04-24T08:19:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.066+0000 7f9d78a00640 1 --2- 172.21.15.28:0/873371451 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55630e474f00 0x55630e477320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.066+0000 7f9d56a00640 10 client.26932 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:55.074 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.066+0000 7f9d78a00640 1 --2- 172.21.15.28:0/873371451 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55630e474f00 0x55630e477320 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4c8d52b8726ae4c server_cookie=19ab52329695f614 in_seq=0 out_seq=0 2024-04-24T08:19:55.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.066+0000 7f9d56a00640 1 -- 172.21.15.28:0/873371451 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55630e360a60 con 0x55630e474f00 2024-04-24T08:19:55.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.066+0000 7f9d56a00640 10 client.26932 handle_command_reply: tid=0 2024-04-24T08:19:55.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.066+0000 7f9d7edb7740 1 client.26932 shutdown 2024-04-24T08:19:55.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.067+0000 7f9d7edb7740 1 -- 172.21.15.28:0/873371451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f3b40 msgr2=0x55630e3feff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.067+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/873371451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f3b40 0x55630e3feff0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f9d68005290 tx=0x7f9d680196c0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.067+0000 7f9d7edb7740 1 -- 172.21.15.28:0/873371451 shutdown_connections 2024-04-24T08:19:55.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.067+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/873371451 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55630e474f00 0x55630e477320 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.067+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/873371451 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55630e3feb30 0x55630e3ffab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.075 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.067+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/873371451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55630e3f4460 0x55630e3ff530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.067+0000 7f9d7edb7740 1 --2- 172.21.15.28:0/873371451 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55630e3f3b40 0x55630e3feff0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.067+0000 7f9d56a00640 0 client.26932 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:55.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.067+0000 7f9d7edb7740 1 -- 172.21.15.28:0/873371451 >> 172.21.15.28:0/873371451 conn(0x55630e3efe60 msgr2=0x55630e3fa0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:55.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.067+0000 7f9d7edb7740 1 -- 172.21.15.28:0/873371451 shutdown_connections 2024-04-24T08:19:55.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.067+0000 7f9d7edb7740 1 -- 172.21.15.28:0/873371451 wait complete. 2024-04-24T08:19:55.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.067+0000 7f9d7edb7740 20 client.26932 trim_cache size 0 max 16384 2024-04-24T08:19:55.076 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:19:55.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.068+0000 7f9d7a800640 1 -- 172.21.15.28:0/2063297282 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d5c0a3da0 msgr2=0x7f9d5c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.068+0000 7f9d7a800640 1 --2- 172.21.15.28:0/2063297282 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d5c0a3da0 0x7f9d5c0a6260 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f9d64004af0 tx=0x7f9d64009820 comp rx=0 tx=0).stop 2024-04-24T08:19:55.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.068+0000 7f9d7a800640 1 -- 172.21.15.28:0/2063297282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d7419a7b0 msgr2=0x7f9d7419ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:55.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.068+0000 7f9d7a800640 1 --2- 172.21.15.28:0/2063297282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d7419a7b0 0x7f9d7419ac60 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c00e970 tx=0x7f9d6c00ed70 comp rx=0 tx=0).stop 2024-04-24T08:19:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.068+0000 7f9d7a800640 1 -- 172.21.15.28:0/2063297282 shutdown_connections 2024-04-24T08:19:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.068+0000 7f9d7a800640 1 --2- 172.21.15.28:0/2063297282 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d5c0a3da0 0x7f9d5c0a6260 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.068+0000 7f9d7a800640 1 --2- 172.21.15.28:0/2063297282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d7419a7b0 0x7f9d7419ac60 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.068+0000 7f9d7a800640 1 --2- 172.21.15.28:0/2063297282 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d741093f0 0x7f9d741a0b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.068+0000 7f9d7a800640 1 --2- 172.21.15.28:0/2063297282 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d74104320 0x7f9d741a0640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.068+0000 7f9d7a800640 1 -- 172.21.15.28:0/2063297282 >> 172.21.15.28:0/2063297282 conn(0x7f9d740ffb40 msgr2=0x7f9d741006e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.068+0000 7f9d7a800640 1 -- 172.21.15.28:0/2063297282 shutdown_connections 2024-04-24T08:19:55.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:55.068+0000 7f9d7a800640 1 -- 172.21.15.28:0/2063297282 wait complete. 2024-04-24T08:19:55.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:54 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3034863138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:19:55.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:54 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3034863138' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:19:56.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:55 smithi028 ceph-mon[27786]: pgmap v296: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:19:56.087 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:19:56.087 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:19:56.087 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:19:56.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:55 smithi117 ceph-mon[32438]: pgmap v296: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:19:56.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:55 smithi194 ceph-mon[32836]: pgmap v296: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:19:56.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.316+0000 7f55ca800640 1 -- 172.21.15.28:0/1585254946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55c4074bf0 msgr2=0x7f55c410e3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.318 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.316+0000 7f55ca800640 1 --2- 172.21.15.28:0/1585254946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55c4074bf0 0x7f55c410e3a0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f55b0009900 tx=0x7f55b002a480 comp rx=0 tx=0).stop 2024-04-24T08:19:56.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.316+0000 7f55ca800640 1 -- 172.21.15.28:0/1585254946 shutdown_connections 2024-04-24T08:19:56.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.316+0000 7f55ca800640 1 --2- 172.21.15.28:0/1585254946 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c410e8e0 0x7f55c4110cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.316+0000 7f55ca800640 1 --2- 172.21.15.28:0/1585254946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55c4074bf0 0x7f55c410e3a0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.316+0000 7f55ca800640 1 --2- 172.21.15.28:0/1585254946 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55c40742d0 0x7f55c40746b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.316+0000 7f55ca800640 1 -- 172.21.15.28:0/1585254946 >> 172.21.15.28:0/1585254946 conn(0x7f55c4079450 msgr2=0x7f55c4079860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:56.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.316+0000 7f55ca800640 1 -- 172.21.15.28:0/1585254946 shutdown_connections 2024-04-24T08:19:56.319 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.316+0000 7f55ca800640 1 -- 172.21.15.28:0/1585254946 wait complete. 2024-04-24T08:19:56.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.317+0000 7f55ca800640 1 Processor -- start 2024-04-24T08:19:56.321 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.318+0000 7f55ca800640 1 -- start start 2024-04-24T08:19:56.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.318+0000 7f55ca800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55c40742d0 0x7f55c41a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.318+0000 7f55ca800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55c4074bf0 0x7f55c41a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.318+0000 7f55ca800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c410e8e0 0x7f55c419a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.318+0000 7f55ca800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55c4114640 con 0x7f55c410e8e0 2024-04-24T08:19:56.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.318+0000 7f55ca800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f55c41144c0 con 0x7f55c40742d0 2024-04-24T08:19:56.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.318+0000 7f55ca800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f55c41147c0 con 0x7f55c4074bf0 2024-04-24T08:19:56.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.318+0000 7f55c3400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55c40742d0 0x7f55c41a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.322 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.318+0000 7f55c3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c410e8e0 0x7f55c419a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.318+0000 7f55c3400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55c40742d0 0x7f55c41a0790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51478/0 (socket says 172.21.15.28:51478) 2024-04-24T08:19:56.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.318+0000 7f55c3400640 1 -- 172.21.15.28:0/3650653631 learned_addr learned my addr 172.21.15.28:0/3650653631 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:56.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.318+0000 7f55c2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55c4074bf0 0x7f55c41a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.319+0000 7f55c3e00640 1 -- 172.21.15.28:0/3650653631 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55c4074bf0 msgr2=0x7f55c41a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.319+0000 7f55c3e00640 1 --2- 172.21.15.28:0/3650653631 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55c4074bf0 0x7f55c41a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.319+0000 7f55c3e00640 1 -- 172.21.15.28:0/3650653631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55c40742d0 msgr2=0x7f55c41a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.319+0000 7f55c3e00640 1 --2- 172.21.15.28:0/3650653631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55c40742d0 0x7f55c41a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.319+0000 7f55c3e00640 1 -- 172.21.15.28:0/3650653631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55c419b0c0 con 0x7f55c410e8e0 2024-04-24T08:19:56.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.319+0000 7f55c3400640 1 --2- 172.21.15.28:0/3650653631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55c40742d0 0x7f55c41a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:56.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.319+0000 7f55c3e00640 1 --2- 172.21.15.28:0/3650653631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c410e8e0 0x7f55c419a940 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f55bc005fd0 tx=0x7f55bc00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:56.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.319+0000 7f55bbe00640 1 -- 172.21.15.28:0/3650653631 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55bc00cdf0 con 0x7f55c410e8e0 2024-04-24T08:19:56.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.320+0000 7f55bbe00640 1 -- 172.21.15.28:0/3650653631 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55bc005020 con 0x7f55c410e8e0 2024-04-24T08:19:56.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.320+0000 7f55bbe00640 1 -- 172.21.15.28:0/3650653631 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55bc0106d0 con 0x7f55c410e8e0 2024-04-24T08:19:56.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.320+0000 7f55ca800640 1 -- 172.21.15.28:0/3650653631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55c419b3b0 con 0x7f55c410e8e0 2024-04-24T08:19:56.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.320+0000 7f55ca800640 1 -- 172.21.15.28:0/3650653631 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55c41a7650 con 0x7f55c410e8e0 2024-04-24T08:19:56.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.324+0000 7f55cd5a9740 1 Processor -- start 2024-04-24T08:19:56.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.324+0000 7f55bbe00640 1 -- 172.21.15.28:0/3650653631 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f55bc0054d0 con 0x7f55c410e8e0 2024-04-24T08:19:56.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.325+0000 7f55cd5a9740 1 -- start start 2024-04-24T08:19:56.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.325+0000 7f55cd5a9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a5090 0x5592a34a5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.325+0000 7f55cd5a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a59b0 0x5592a34afb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.325+0000 7f55cd5a9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34b0080 0x5592a34b2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.325+0000 7f55cd5a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5592a340ae70 con 0x5592a34a59b0 2024-04-24T08:19:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.325+0000 7f55cd5a9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5592a33b1380 con 0x5592a34b0080 2024-04-24T08:19:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.325+0000 7f55cd5a9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5592a3406e30 con 0x5592a34a5090 2024-04-24T08:19:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.325+0000 7f55c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a59b0 0x5592a34afb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.325+0000 7f55c2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a5090 0x5592a34a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.325+0000 7f55c2a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a5090 0x5592a34a5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44840/0 (socket says 172.21.15.28:44840) 2024-04-24T08:19:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.325+0000 7f55c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a59b0 0x5592a34afb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58444/0 (socket says 172.21.15.28:58444) 2024-04-24T08:19:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.325+0000 7f55c2a00640 1 -- 172.21.15.28:0/1232670317 learned_addr learned my addr 172.21.15.28:0/1232670317 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:56.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.325+0000 7f55c2a00640 1 --2- 172.21.15.28:0/1232670317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34b0080 0x5592a34b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.326+0000 7f55c3400640 1 -- 172.21.15.28:0/1232670317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a5090 msgr2=0x5592a34a5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.326+0000 7f55c3400640 1 --2- 172.21.15.28:0/1232670317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a5090 0x5592a34a5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.326+0000 7f55c3400640 1 -- 172.21.15.28:0/1232670317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34b0080 msgr2=0x5592a34b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.326+0000 7f55c3400640 1 --2- 172.21.15.28:0/1232670317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34b0080 0x5592a34b2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.326+0000 7f55c3400640 1 -- 172.21.15.28:0/1232670317 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592a31e57e0 con 0x5592a34a59b0 2024-04-24T08:19:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.326+0000 7f55bbe00640 1 --2- 172.21.15.28:0/3650653631 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f55a80a3da0 0x7f55a80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.326+0000 7f55c3400640 1 --2- 172.21.15.28:0/1232670317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a59b0 0x5592a34afb40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f55c419ba80 tx=0x7f55b40119a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=abc8f4c97d5de47 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:56.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.326+0000 7f55bbe00640 1 -- 172.21.15.28:0/3650653631 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7f55bc0d0e70 con 0x7f55c410e8e0 2024-04-24T08:19:56.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.327+0000 7f55c2a00640 1 --2- 172.21.15.28:0/3650653631 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f55a80a3da0 0x7f55a80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.327+0000 7f55b8c00640 1 -- 172.21.15.28:0/1232670317 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b4005020 con 0x5592a34a59b0 2024-04-24T08:19:56.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.327+0000 7f55b8c00640 1 -- 172.21.15.28:0/1232670317 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55b40047a0 con 0x5592a34a59b0 2024-04-24T08:19:56.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.327+0000 7f55b8c00640 1 -- 172.21.15.28:0/1232670317 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b400f9e0 con 0x5592a34a59b0 2024-04-24T08:19:56.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.327+0000 7f55cd5a9740 1 -- 172.21.15.28:0/1232670317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a59b0 msgr2=0x5592a34afb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.335 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.327+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/1232670317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a59b0 0x5592a34afb40 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f55c419ba80 tx=0x7f55b40119a0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.327+0000 7f55c2a00640 1 --2- 172.21.15.28:0/3650653631 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f55a80a3da0 0x7f55a80a6260 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x5592a34a2540 tx=0x7f55b0009260 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:56.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.328+0000 7f55cd5a9740 1 -- 172.21.15.28:0/1232670317 shutdown_connections 2024-04-24T08:19:56.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.328+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/1232670317 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a59b0 0x5592a34afb40 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.328+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/1232670317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34b0080 0x5592a34b2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.328+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/1232670317 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a5090 0x5592a34a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.328+0000 7f55cd5a9740 1 -- 172.21.15.28:0/1232670317 >> 172.21.15.28:0/1232670317 conn(0x5592a34a13b0 msgr2=0x5592a34a1790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:56.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.328+0000 7f55cd5a9740 1 -- 172.21.15.28:0/1232670317 shutdown_connections 2024-04-24T08:19:56.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.328+0000 7f55cd5a9740 1 -- 172.21.15.28:0/1232670317 wait complete. 2024-04-24T08:19:56.336 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.328+0000 7f55cd5a9740 1 -- start start 2024-04-24T08:19:56.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.328+0000 7f55cd5a9740 1 Processor -- start 2024-04-24T08:19:56.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.329+0000 7f55cd5a9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34a5090 0x5592a34acf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.329+0000 7f55cd5a9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a59b0 0x5592a34ad480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.329+0000 7f55cd5a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34b0080 0x5592a34ad9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.329+0000 7f55cd5a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5592a34b2e70 con 0x5592a34b0080 2024-04-24T08:19:56.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.329+0000 7f55cd5a9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5592a33c4660 con 0x5592a34a5090 2024-04-24T08:19:56.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.329+0000 7f55cd5a9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5592a34b2ff0 con 0x5592a34a59b0 2024-04-24T08:19:56.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.329+0000 7f55cd5a9740 10 client.0 _ll_register_callbacks cb 0x5592a33b2aa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:56.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.329+0000 7f55c3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34b0080 0x5592a34ad9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.329+0000 7f55c3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34b0080 0x5592a34ad9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58448/0 (socket says 172.21.15.28:58448) 2024-04-24T08:19:56.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.329+0000 7f55c3e00640 1 -- 172.21.15.28:0/1509592780 learned_addr learned my addr 172.21.15.28:0/1509592780 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:56.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.329+0000 7f55c3400640 1 --2- 172.21.15.28:0/1509592780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34a5090 0x5592a34acf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.329+0000 7f55c2a00640 1 --2- 172.21.15.28:0/1509592780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a59b0 0x5592a34ad480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.330+0000 7f55c3400640 1 -- 172.21.15.28:0/1509592780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a59b0 msgr2=0x5592a34ad480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:56.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.330+0000 7f55c3400640 1 --2- 172.21.15.28:0/1509592780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a59b0 0x5592a34ad480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.330+0000 7f55c3400640 1 -- 172.21.15.28:0/1509592780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34b0080 msgr2=0x5592a34ad9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:56.338 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.330+0000 7f55c3400640 1 --2- 172.21.15.28:0/1509592780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34b0080 0x5592a34ad9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.330+0000 7f55c3400640 1 -- 172.21.15.28:0/1509592780 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592a3245f70 con 0x5592a34a5090 2024-04-24T08:19:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.330+0000 7f55c3e00640 1 --2- 172.21.15.28:0/1509592780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34b0080 0x5592a34ad9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.330+0000 7f55c3400640 1 --2- 172.21.15.28:0/1509592780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34a5090 0x5592a34acf40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f55c419ba80 tx=0x7f55b4005250 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf7f58ab85327757 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.330+0000 7f55ad600640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:19:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.330+0000 7f55ad600640 1 -- 172.21.15.28:0/1509592780 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b4028070 con 0x5592a34a5090 2024-04-24T08:19:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.331+0000 7f55cd5a9740 1 -- 172.21.15.28:0/1509592780 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55b40047a0 con 0x5592a34a5090 2024-04-24T08:19:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.331+0000 7f55ad600640 1 -- 172.21.15.28:0/1509592780 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7f55b400f760 con 0x5592a34a5090 2024-04-24T08:19:56.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.331+0000 7f55ad600640 1 -- 172.21.15.28:0/1509592780 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f55b4034050 con 0x5592a34a5090 2024-04-24T08:19:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.331+0000 7f55cd5a9740 10 client.26930 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.331+0000 7f55cd5a9740 1 -- 172.21.15.28:0/1509592780 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5592a30dd170 con 0x5592a34a5090 2024-04-24T08:19:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.331+0000 7f55ad600640 1 -- 172.21.15.28:0/1509592780 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55b40047a0 con 0x5592a34a5090 2024-04-24T08:19:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.332+0000 7f55ad600640 1 -- 172.21.15.28:0/1509592780 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f55b4024720 con 0x5592a34a5090 2024-04-24T08:19:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.332+0000 7f55cd5a9740 10 client.26930 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.332+0000 7f55cd5a9740 10 client.26930 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:19:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.332+0000 7f55cd5a9740 20 client.26930 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.332+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/1509592780 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5592a34c02d0 0x5592a34c26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.332+0000 7f55cd5a9740 4 client.26930 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:19:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.332+0000 7f55cd5a9740 1 -- 172.21.15.28:0/1509592780 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5592a3418890 con 0x5592a34c02d0 2024-04-24T08:19:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.332+0000 7f55c2a00640 1 --2- 172.21.15.28:0/1509592780 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5592a34c02d0 0x5592a34c26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.333+0000 7f55c2a00640 1 --2- 172.21.15.28:0/1509592780 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5592a34c02d0 0x5592a34c26c0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa5e4a9d9f743ea0 server_cookie=934d44761e608ebf in_seq=0 out_seq=0 2024-04-24T08:19:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.333+0000 7f55ad600640 10 client.26930 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.333+0000 7f55ad600640 1 -- 172.21.15.28:0/1509592780 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5592a3418890 con 0x5592a34c02d0 2024-04-24T08:19:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.333+0000 7f55ad600640 10 client.26930 handle_command_reply: tid=0 2024-04-24T08:19:56.341 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.333+0000 7f55cd5a9740 1 client.26930 shutdown 2024-04-24T08:19:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.334+0000 7f55cd5a9740 1 -- 172.21.15.28:0/1509592780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34a5090 msgr2=0x5592a34acf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.334+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/1509592780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34a5090 0x5592a34acf40 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f55c419ba80 tx=0x7f55b4005250 comp rx=0 tx=0).stop 2024-04-24T08:19:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.334+0000 7f55cd5a9740 1 -- 172.21.15.28:0/1509592780 shutdown_connections 2024-04-24T08:19:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.334+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/1509592780 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5592a34c02d0 0x5592a34c26c0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.334+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/1509592780 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34b0080 0x5592a34ad9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.334+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/1509592780 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a59b0 0x5592a34ad480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.334+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/1509592780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34a5090 0x5592a34acf40 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.334+0000 7f55cd5a9740 1 -- 172.21.15.28:0/1509592780 >> 172.21.15.28:0/1509592780 conn(0x5592a34a13b0 msgr2=0x5592a34a71b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.334+0000 7f55ad600640 0 client.26930 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.334+0000 7f55cd5a9740 1 -- 172.21.15.28:0/1509592780 shutdown_connections 2024-04-24T08:19:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.334+0000 7f55cd5a9740 1 -- 172.21.15.28:0/1509592780 wait complete. 2024-04-24T08:19:56.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.334+0000 7f55cd5a9740 20 client.26930 trim_cache size 0 max 16384 2024-04-24T08:19:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.354+0000 7f55cd5a9740 1 Processor -- start 2024-04-24T08:19:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.354+0000 7f55cd5a9740 1 -- start start 2024-04-24T08:19:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.354+0000 7f55cd5a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a5090 0x5592a34bbd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.356 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.354+0000 7f55cd5a9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34a59b0 0x5592a34a7900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.354+0000 7f55cd5a9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34b0080 0x5592a34a0390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.354+0000 7f55cd5a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55c41b2930 con 0x5592a34a5090 2024-04-24T08:19:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.354+0000 7f55cd5a9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f55c41b0200 con 0x5592a34a59b0 2024-04-24T08:19:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.354+0000 7f55cd5a9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f55c41ae070 con 0x5592a34b0080 2024-04-24T08:19:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.354+0000 7f55c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a5090 0x5592a34bbd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.354+0000 7f55c3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a5090 0x5592a34bbd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58464/0 (socket says 172.21.15.28:58464) 2024-04-24T08:19:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.354+0000 7f55c3400640 1 -- 172.21.15.28:0/2870587355 learned_addr learned my addr 172.21.15.28:0/2870587355 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:56.357 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.354+0000 7f55c2a00640 1 --2- 172.21.15.28:0/2870587355 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34a59b0 0x5592a34a7900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.355+0000 7f55c3e00640 1 --2- 172.21.15.28:0/2870587355 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34b0080 0x5592a34a0390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.355+0000 7f55c3400640 1 -- 172.21.15.28:0/2870587355 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34b0080 msgr2=0x5592a34a0390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.355+0000 7f55c3400640 1 --2- 172.21.15.28:0/2870587355 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34b0080 0x5592a34a0390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.355+0000 7f55c3400640 1 -- 172.21.15.28:0/2870587355 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34a59b0 msgr2=0x5592a34a7900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.355+0000 7f55c3400640 1 --2- 172.21.15.28:0/2870587355 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34a59b0 0x5592a34a7900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.355+0000 7f55c3400640 1 -- 172.21.15.28:0/2870587355 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5592a34a57e0 con 0x5592a34a5090 2024-04-24T08:19:56.358 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.355+0000 7f55c3e00640 1 --2- 172.21.15.28:0/2870587355 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34b0080 0x5592a34a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:19:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.355+0000 7f55c3400640 1 --2- 172.21.15.28:0/2870587355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a5090 0x5592a34bbd30 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f55b4001eb0 tx=0x7f55b4009fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb11462889e861fe server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.355+0000 7f55afe00640 1 -- 172.21.15.28:0/2870587355 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b4031020 con 0x5592a34a5090 2024-04-24T08:19:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.355+0000 7f55afe00640 1 -- 172.21.15.28:0/2870587355 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55b4009cf0 con 0x5592a34a5090 2024-04-24T08:19:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.355+0000 7f55afe00640 1 -- 172.21.15.28:0/2870587355 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b4028040 con 0x5592a34a5090 2024-04-24T08:19:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.356+0000 7f55cd5a9740 1 -- 172.21.15.28:0/2870587355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a5090 msgr2=0x5592a34bbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.362 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.356+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/2870587355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a5090 0x5592a34bbd30 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f55b4001eb0 tx=0x7f55b4009fd0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.363 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.356+0000 7f55cd5a9740 1 -- 172.21.15.28:0/2870587355 shutdown_connections 2024-04-24T08:19:56.365 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.356+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/2870587355 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34b0080 0x5592a34a0390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.356+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/2870587355 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34a59b0 0x5592a34a7900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.356+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/2870587355 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a5090 0x5592a34bbd30 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.356+0000 7f55cd5a9740 1 -- 172.21.15.28:0/2870587355 >> 172.21.15.28:0/2870587355 conn(0x5592a34a13b0 msgr2=0x5592a349fc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.356+0000 7f55cd5a9740 1 -- 172.21.15.28:0/2870587355 shutdown_connections 2024-04-24T08:19:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.356+0000 7f55cd5a9740 1 -- 172.21.15.28:0/2870587355 wait complete. 2024-04-24T08:19:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.357+0000 7f55cd5a9740 1 -- start start 2024-04-24T08:19:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.357+0000 7f55cd5a9740 1 Processor -- start 2024-04-24T08:19:56.366 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55cd5a9740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a5090 0x5592a34b8430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55cd5a9740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a59b0 0x5592a34a9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55cd5a9740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34b0080 0x5592a34a9c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55cd5a9740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f55c41ae6c0 con 0x5592a34a59b0 2024-04-24T08:19:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55cd5a9740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f55c41afb30 con 0x5592a34b0080 2024-04-24T08:19:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55cd5a9740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f55c41aed90 con 0x5592a34a5090 2024-04-24T08:19:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55cd5a9740 10 client.0 _ll_register_callbacks cb 0x7f55b0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55c2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a59b0 0x5592a34a9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.367 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55c3400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a5090 0x5592a34b8430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55c2a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a59b0 0x5592a34a9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58466/0 (socket says 172.21.15.28:58466) 2024-04-24T08:19:56.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55c2a00640 1 -- 172.21.15.28:0/3601442748 learned_addr learned my addr 172.21.15.28:0/3601442748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:56.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55c3400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a5090 0x5592a34b8430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:44868/0 (socket says 172.21.15.28:44868) 2024-04-24T08:19:56.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55c3e00640 1 --2- 172.21.15.28:0/3601442748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34b0080 0x5592a34a9c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55c3400640 1 -- 172.21.15.28:0/3601442748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34b0080 msgr2=0x5592a34a9c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:56.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55c3400640 1 --2- 172.21.15.28:0/3601442748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34b0080 0x5592a34a9c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55c3400640 1 -- 172.21.15.28:0/3601442748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a59b0 msgr2=0x5592a34a9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:56.368 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55c3400640 1 --2- 172.21.15.28:0/3601442748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a59b0 0x5592a34a9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.358+0000 7f55c3400640 1 -- 172.21.15.28:0/3601442748 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5592a34aa1e0 con 0x5592a34a5090 2024-04-24T08:19:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.359+0000 7f55c3400640 1 --2- 172.21.15.28:0/3601442748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a5090 0x5592a34b8430 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f55b4005770 tx=0x7f55b4009400 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6c939c52ebe40cf server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.359+0000 7f55ad600640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:19:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.359+0000 7f55ad600640 1 -- 172.21.15.28:0/3601442748 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f55b4031040 con 0x5592a34a5090 2024-04-24T08:19:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.359+0000 7f55cd5a9740 1 -- 172.21.15.28:0/3601442748 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f55b4009cf0 con 0x5592a34a5090 2024-04-24T08:19:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.359+0000 7f55ad600640 1 -- 172.21.15.28:0/3601442748 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7f55b4024440 con 0x5592a34a5090 2024-04-24T08:19:56.369 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.360+0000 7f55ad600640 1 -- 172.21.15.28:0/3601442748 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f55b4028040 con 0x5592a34a5090 2024-04-24T08:19:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.360+0000 7f55cd5a9740 10 client.26950 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.360+0000 7f55cd5a9740 1 -- 172.21.15.28:0/3601442748 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5592a3526470 con 0x5592a34a5090 2024-04-24T08:19:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.360+0000 7f55ad600640 1 -- 172.21.15.28:0/3601442748 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f55b4009cf0 con 0x5592a34a5090 2024-04-24T08:19:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.360+0000 7f55ad600640 1 -- 172.21.15.28:0/3601442748 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f55b402d440 con 0x5592a34a5090 2024-04-24T08:19:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.360+0000 7f55cd5a9740 10 client.26950 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.360+0000 7f55cd5a9740 10 client.26950 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:19:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.360+0000 7f55cd5a9740 20 client.26950 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:56.370 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.360+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/3601442748 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5592a3526a20 0x5592a3528e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.360+0000 7f55cd5a9740 4 client.26950 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:19:56.371 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.360+0000 7f55cd5a9740 1 -- 172.21.15.28:0/3601442748 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x5592a3412700 con 0x5592a3526a20 2024-04-24T08:19:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.361+0000 7f55c2a00640 1 --2- 172.21.15.28:0/3601442748 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5592a3526a20 0x5592a3528e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.372 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.361+0000 7f55ad600640 10 client.26950 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.361+0000 7f55c2a00640 1 --2- 172.21.15.28:0/3601442748 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5592a3526a20 0x5592a3528e10 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46a712e26af6ac0b server_cookie=78a87fb79a11958d in_seq=0 out_seq=0 2024-04-24T08:19:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.362+0000 7f55ad600640 1 -- 172.21.15.28:0/3601442748 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x5592a3412700 con 0x5592a3526a20 2024-04-24T08:19:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.362+0000 7f55ad600640 10 client.26950 handle_command_reply: tid=0 2024-04-24T08:19:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.362+0000 7f55cd5a9740 1 client.26950 shutdown 2024-04-24T08:19:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.363+0000 7f55cd5a9740 1 -- 172.21.15.28:0/3601442748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a5090 msgr2=0x5592a34b8430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.363+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/3601442748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a5090 0x5592a34b8430 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f55b4005770 tx=0x7f55b4009400 comp rx=0 tx=0).stop 2024-04-24T08:19:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.363+0000 7f55cd5a9740 1 -- 172.21.15.28:0/3601442748 shutdown_connections 2024-04-24T08:19:56.373 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.363+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/3601442748 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5592a3526a20 0x5592a3528e10 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.363+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/3601442748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5592a34a59b0 0x5592a34a9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.363+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/3601442748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5592a34b0080 0x5592a34a9c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.363+0000 7f55cd5a9740 1 --2- 172.21.15.28:0/3601442748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5592a34a5090 0x5592a34b8430 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.363+0000 7f55ad600640 0 client.26950 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.363+0000 7f55cd5a9740 1 -- 172.21.15.28:0/3601442748 >> 172.21.15.28:0/3601442748 conn(0x5592a34a13b0 msgr2=0x5592a34ad770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.363+0000 7f55cd5a9740 1 -- 172.21.15.28:0/3601442748 shutdown_connections 2024-04-24T08:19:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.363+0000 7f55cd5a9740 1 -- 172.21.15.28:0/3601442748 wait complete. 2024-04-24T08:19:56.374 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.363+0000 7f55cd5a9740 20 client.26950 trim_cache size 0 max 16384 2024-04-24T08:19:56.375 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:19:56.375 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-24T08:19:56.375 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "7ba0ff79-0882-4a82-8139-857fd7104488", 2024-04-24T08:19:56.375 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-24T08:19:56.376 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:19:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.364+0000 7f55ca800640 1 -- 172.21.15.28:0/3650653631 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f55a80a3da0 msgr2=0x7f55a80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.364+0000 7f55ca800640 1 --2- 172.21.15.28:0/3650653631 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f55a80a3da0 0x7f55a80a6260 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x5592a34a2540 tx=0x7f55b0009260 comp rx=0 tx=0).stop 2024-04-24T08:19:56.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.364+0000 7f55ca800640 1 -- 172.21.15.28:0/3650653631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c410e8e0 msgr2=0x7f55c419a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.364+0000 7f55ca800640 1 --2- 172.21.15.28:0/3650653631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c410e8e0 0x7f55c419a940 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7f55bc005fd0 tx=0x7f55bc00ef10 comp rx=0 tx=0).stop 2024-04-24T08:19:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.365+0000 7f55ca800640 1 -- 172.21.15.28:0/3650653631 shutdown_connections 2024-04-24T08:19:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.365+0000 7f55ca800640 1 --2- 172.21.15.28:0/3650653631 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f55a80a3da0 0x7f55a80a6260 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.365+0000 7f55ca800640 1 --2- 172.21.15.28:0/3650653631 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f55c410e8e0 0x7f55c419a940 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.365+0000 7f55ca800640 1 --2- 172.21.15.28:0/3650653631 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f55c4074bf0 0x7f55c41a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.365+0000 7f55ca800640 1 --2- 172.21.15.28:0/3650653631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f55c40742d0 0x7f55c41a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.365+0000 7f55ca800640 1 -- 172.21.15.28:0/3650653631 >> 172.21.15.28:0/3650653631 conn(0x7f55c4079450 msgr2=0x7f55c40fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.365+0000 7f55ca800640 1 -- 172.21.15.28:0/3650653631 shutdown_connections 2024-04-24T08:19:56.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.365+0000 7f55ca800640 1 -- 172.21.15.28:0/3650653631 wait complete. 2024-04-24T08:19:56.388 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:19:56.388 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:19:56.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.588+0000 7f2a37000640 1 -- 172.21.15.28:0/2640469643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a30112ec0 msgr2=0x7f2a301152b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.588+0000 7f2a37000640 1 --2- 172.21.15.28:0/2640469643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a30112ec0 0x7f2a301152b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f2a28009900 tx=0x7f2a2802a480 comp rx=0 tx=0).stop 2024-04-24T08:19:56.590 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.588+0000 7f2a37000640 1 -- 172.21.15.28:0/2640469643 shutdown_connections 2024-04-24T08:19:56.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.588+0000 7f2a37000640 1 --2- 172.21.15.28:0/2640469643 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a30112ec0 0x7f2a301152b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.588+0000 7f2a37000640 1 --2- 172.21.15.28:0/2640469643 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a30074cc0 0x7f2a30112980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.588+0000 7f2a37000640 1 --2- 172.21.15.28:0/2640469643 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a300743a0 0x7f2a30074780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.588+0000 7f2a37000640 1 -- 172.21.15.28:0/2640469643 >> 172.21.15.28:0/2640469643 conn(0x7f2a300fd9a0 msgr2=0x7f2a300ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:56.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.588+0000 7f2a37000640 1 -- 172.21.15.28:0/2640469643 shutdown_connections 2024-04-24T08:19:56.591 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.588+0000 7f2a37000640 1 -- 172.21.15.28:0/2640469643 wait complete. 2024-04-24T08:19:56.593 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.589+0000 7f2a37000640 1 Processor -- start 2024-04-24T08:19:56.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.589+0000 7f2a37000640 1 -- start start 2024-04-24T08:19:56.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.590+0000 7f2a37000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a300743a0 0x7f2a3019ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.590+0000 7f2a37000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a30074cc0 0x7f2a3019f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.590+0000 7f2a37000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a30112ec0 0x7f2a301a3640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.590+0000 7f2a37000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a30118860 con 0x7f2a30112ec0 2024-04-24T08:19:56.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.590+0000 7f2a37000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2a301186e0 con 0x7f2a30074cc0 2024-04-24T08:19:56.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.590+0000 7f2a37000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2a301189e0 con 0x7f2a300743a0 2024-04-24T08:19:56.594 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.590+0000 7f2a36600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a30112ec0 0x7f2a301a3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.590+0000 7f2a35c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a300743a0 0x7f2a3019ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.590+0000 7f2a36600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a30112ec0 0x7f2a301a3640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58476/0 (socket says 172.21.15.28:58476) 2024-04-24T08:19:56.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.590+0000 7f2a36600640 1 -- 172.21.15.28:0/120220671 learned_addr learned my addr 172.21.15.28:0/120220671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:56.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.590+0000 7f2a35200640 1 --2- 172.21.15.28:0/120220671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a30074cc0 0x7f2a3019f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.591+0000 7f2a36600640 1 -- 172.21.15.28:0/120220671 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a300743a0 msgr2=0x7f2a3019ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.591+0000 7f2a36600640 1 --2- 172.21.15.28:0/120220671 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a300743a0 0x7f2a3019ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.591+0000 7f2a36600640 1 -- 172.21.15.28:0/120220671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a30074cc0 msgr2=0x7f2a3019f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.595 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.591+0000 7f2a36600640 1 --2- 172.21.15.28:0/120220671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a30074cc0 0x7f2a3019f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.591+0000 7f2a36600640 1 -- 172.21.15.28:0/120220671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a301a3d70 con 0x7f2a30112ec0 2024-04-24T08:19:56.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.591+0000 7f2a36600640 1 --2- 172.21.15.28:0/120220671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a30112ec0 0x7f2a301a3640 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2a28005ab0 tx=0x7f2a280040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:56.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.591+0000 7f2a2e000640 1 -- 172.21.15.28:0/120220671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2803f030 con 0x7f2a30112ec0 2024-04-24T08:19:56.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.591+0000 7f2a37000640 1 -- 172.21.15.28:0/120220671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a301a4000 con 0x7f2a30112ec0 2024-04-24T08:19:56.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.591+0000 7f2a2e000640 1 -- 172.21.15.28:0/120220671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a280044c0 con 0x7f2a30112ec0 2024-04-24T08:19:56.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.591+0000 7f2a2e000640 1 -- 172.21.15.28:0/120220671 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a28035070 con 0x7f2a30112ec0 2024-04-24T08:19:56.596 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.591+0000 7f2a37000640 1 -- 172.21.15.28:0/120220671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a301ab8f0 con 0x7f2a30112ec0 2024-04-24T08:19:56.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.596+0000 7f2a3b3f0740 1 Processor -- start 2024-04-24T08:19:56.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.596+0000 7f2a2e000640 1 -- 172.21.15.28:0/120220671 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f2a28042030 con 0x7f2a30112ec0 2024-04-24T08:19:56.597 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.596+0000 7f2a3b3f0740 1 -- start start 2024-04-24T08:19:56.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.596+0000 7f2a3b3f0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a9a30 0x55be704a9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.599 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.597+0000 7f2a3b3f0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704aa350 0x55be704b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.597+0000 7f2a3b3f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704b4a20 0x55be704b6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.597+0000 7f2a3b3f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55be7040f1f0 con 0x55be704b4a20 2024-04-24T08:19:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.597+0000 7f2a3b3f0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55be703b5350 con 0x55be704aa350 2024-04-24T08:19:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.597+0000 7f2a3b3f0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55be7040b1b0 con 0x55be704a9a30 2024-04-24T08:19:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.597+0000 7f2a35c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704aa350 0x55be704b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.597+0000 7f2a35c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704aa350 0x55be704b44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:51528/0 (socket says 172.21.15.28:51528) 2024-04-24T08:19:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.597+0000 7f2a35c00640 1 -- 172.21.15.28:0/1311823357 learned_addr learned my addr 172.21.15.28:0/1311823357 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.597+0000 7f2a35200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704b4a20 0x55be704b6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.600 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.597+0000 7f2a35200640 1 --2- 172.21.15.28:0/1311823357 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a9a30 0x55be704a9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.597+0000 7f2a35c00640 1 -- 172.21.15.28:0/1311823357 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a9a30 msgr2=0x55be704a9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:56.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.597+0000 7f2a35c00640 1 --2- 172.21.15.28:0/1311823357 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a9a30 0x55be704a9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.597+0000 7f2a35c00640 1 -- 172.21.15.28:0/1311823357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704b4a20 msgr2=0x55be704b6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:56.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.597+0000 7f2a35c00640 1 --2- 172.21.15.28:0/1311823357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704b4a20 0x55be704b6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.597+0000 7f2a35c00640 1 -- 172.21.15.28:0/1311823357 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be70409f50 con 0x55be704aa350 2024-04-24T08:19:56.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.598+0000 7f2a35c00640 1 --2- 172.21.15.28:0/1311823357 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704aa350 0x55be704b44e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2a240114f0 tx=0x7f2a240119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e87e69d8e4b86458 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:56.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.598+0000 7f2a2e000640 1 --2- 172.21.15.28:0/120220671 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2a180a3d20 0x7f2a180a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.601 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.598+0000 7f2a22a00640 1 -- 172.21.15.28:0/1311823357 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a2401e070 con 0x55be704aa350 2024-04-24T08:19:56.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.598+0000 7f2a22a00640 1 -- 172.21.15.28:0/1311823357 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a2400fb00 con 0x55be704aa350 2024-04-24T08:19:56.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.598+0000 7f2a22a00640 1 -- 172.21.15.28:0/1311823357 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a24019bf0 con 0x55be704aa350 2024-04-24T08:19:56.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.598+0000 7f2a2e000640 1 -- 172.21.15.28:0/120220671 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7f2a280f07d0 con 0x7f2a30112ec0 2024-04-24T08:19:56.602 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.598+0000 7f2a35200640 1 --2- 172.21.15.28:0/120220671 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2a180a3d20 0x7f2a180a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.599+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/1311823357 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704aa350 msgr2=0x55be704b44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.599+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/1311823357 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704aa350 0x55be704b44e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f2a240114f0 tx=0x7f2a240119b0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.606 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.599+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/1311823357 shutdown_connections 2024-04-24T08:19:56.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.599+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/1311823357 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704b4a20 0x55be704b6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.599+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/1311823357 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704aa350 0x55be704b44e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.599+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/1311823357 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a9a30 0x55be704a9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.599+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/1311823357 >> 172.21.15.28:0/1311823357 conn(0x55be704a5d50 msgr2=0x55be704a6130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:56.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.599+0000 7f2a35200640 1 --2- 172.21.15.28:0/120220671 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2a180a3d20 0x7f2a180a61e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x55be704a6ee0 tx=0x7f2a1c00b5b0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:56.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.599+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/1311823357 shutdown_connections 2024-04-24T08:19:56.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.599+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/1311823357 wait complete. 2024-04-24T08:19:56.607 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.599+0000 7f2a3b3f0740 1 -- start start 2024-04-24T08:19:56.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.600+0000 7f2a3b3f0740 1 Processor -- start 2024-04-24T08:19:56.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.600+0000 7f2a3b3f0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a9a30 0x55be704b19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.600+0000 7f2a3b3f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704aa350 0x55be704b1ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.600+0000 7f2a3b3f0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704b4a20 0x55be704b2430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.600+0000 7f2a3b3f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55be704b7810 con 0x55be704aa350 2024-04-24T08:19:56.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.600+0000 7f2a3b3f0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55be703c8690 con 0x55be704b4a20 2024-04-24T08:19:56.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.600+0000 7f2a3b3f0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55be704b7990 con 0x55be704a9a30 2024-04-24T08:19:56.608 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.600+0000 7f2a3b3f0740 10 client.0 _ll_register_callbacks cb 0x55be70404650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:56.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.601+0000 7f2a35200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704aa350 0x55be704b1ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.601+0000 7f2a36600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704b4a20 0x55be704b2430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.601+0000 7f2a35200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704aa350 0x55be704b1ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58498/0 (socket says 172.21.15.28:58498) 2024-04-24T08:19:56.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.601+0000 7f2a35200640 1 -- 172.21.15.28:0/3055668437 learned_addr learned my addr 172.21.15.28:0/3055668437 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:56.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.601+0000 7f2a35c00640 1 --2- 172.21.15.28:0/3055668437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a9a30 0x55be704b19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.601+0000 7f2a36600640 1 -- 172.21.15.28:0/3055668437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a9a30 msgr2=0x55be704b19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:56.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.601+0000 7f2a36600640 1 --2- 172.21.15.28:0/3055668437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a9a30 0x55be704b19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.601+0000 7f2a36600640 1 -- 172.21.15.28:0/3055668437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704aa350 msgr2=0x55be704b1ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:56.609 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.601+0000 7f2a36600640 1 --2- 172.21.15.28:0/3055668437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704aa350 0x55be704b1ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.601+0000 7f2a36600640 1 -- 172.21.15.28:0/3055668437 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be703c7920 con 0x55be704b4a20 2024-04-24T08:19:56.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.601+0000 7f2a35200640 1 --2- 172.21.15.28:0/3055668437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704aa350 0x55be704b1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:56.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.601+0000 7f2a35c00640 1 --2- 172.21.15.28:0/3055668437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a9a30 0x55be704b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:56.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.601+0000 7f2a36600640 1 --2- 172.21.15.28:0/3055668437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704b4a20 0x55be704b2430 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2a280eded0 tx=0x7f2a280bc130 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f29dbb13d8571ac8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:56.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.601+0000 7f2a17400640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:19:56.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.601+0000 7f2a17400640 1 -- 172.21.15.28:0/3055668437 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a28049040 con 0x55be704b4a20 2024-04-24T08:19:56.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.602+0000 7f2a17400640 1 -- 172.21.15.28:0/3055668437 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7f2a280ee150 con 0x55be704b4a20 2024-04-24T08:19:56.610 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.602+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/3055668437 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a2400fb00 con 0x55be704b4a20 2024-04-24T08:19:56.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.602+0000 7f2a17400640 1 -- 172.21.15.28:0/3055668437 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2a280c5390 con 0x55be704b4a20 2024-04-24T08:19:56.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.602+0000 7f2a3b3f0740 10 client.26948 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:56.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.602+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/3055668437 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55be701eb7d0 con 0x55be704b4a20 2024-04-24T08:19:56.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.602+0000 7f2a17400640 1 -- 172.21.15.28:0/3055668437 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a2400fb00 con 0x55be704b4a20 2024-04-24T08:19:56.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.602+0000 7f2a17400640 1 -- 172.21.15.28:0/3055668437 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2a280c4f00 con 0x55be704b4a20 2024-04-24T08:19:56.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.602+0000 7f2a3b3f0740 10 client.26948 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:56.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.602+0000 7f2a3b3f0740 10 client.26948 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:19:56.611 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.602+0000 7f2a3b3f0740 20 client.26948 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.602+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/3055668437 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55be704c4d70 0x55be704c71c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.602+0000 7f2a3b3f0740 4 client.26948 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:19:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.602+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/3055668437 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55be7041cc10 con 0x55be704c4d70 2024-04-24T08:19:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.603+0000 7f2a35c00640 1 --2- 172.21.15.28:0/3055668437 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55be704c4d70 0x55be704c71c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.603+0000 7f2a35c00640 1 --2- 172.21.15.28:0/3055668437 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55be704c4d70 0x55be704c71c0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d09494c053162fe server_cookie=93c133c7135a2fa0 in_seq=0 out_seq=0 2024-04-24T08:19:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.603+0000 7f2a17400640 10 client.26948 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.604+0000 7f2a17400640 1 -- 172.21.15.28:0/3055668437 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55be7041cc10 con 0x55be704c4d70 2024-04-24T08:19:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.604+0000 7f2a17400640 10 client.26948 handle_command_reply: tid=0 2024-04-24T08:19:56.612 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.604+0000 7f2a3b3f0740 1 client.26948 shutdown 2024-04-24T08:19:56.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.604+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/3055668437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704b4a20 msgr2=0x55be704b2430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.604+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/3055668437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704b4a20 0x55be704b2430 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f2a280eded0 tx=0x7f2a280bc130 comp rx=0 tx=0).stop 2024-04-24T08:19:56.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.604+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/3055668437 shutdown_connections 2024-04-24T08:19:56.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.604+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/3055668437 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55be704c4d70 0x55be704c71c0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.604+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/3055668437 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704aa350 0x55be704b1ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.604+0000 7f2a17400640 0 client.26948 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:56.613 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.604+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/3055668437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704b4a20 0x55be704b2430 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.604+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/3055668437 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a9a30 0x55be704b19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.604+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/3055668437 >> 172.21.15.28:0/3055668437 conn(0x55be704a5d50 msgr2=0x55be704a4fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.605+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/3055668437 shutdown_connections 2024-04-24T08:19:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.605+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/3055668437 wait complete. 2024-04-24T08:19:56.614 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.605+0000 7f2a3b3f0740 20 client.26948 trim_cache size 0 max 16384 2024-04-24T08:19:56.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.624+0000 7f2a3b3f0740 1 Processor -- start 2024-04-24T08:19:56.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.624+0000 7f2a3b3f0740 1 -- start start 2024-04-24T08:19:56.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.625+0000 7f2a3b3f0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a5d50 0x55be704c0700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.625+0000 7f2a3b3f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704a9a30 0x55be704cca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.625+0000 7f2a3b3f0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704aa350 0x55be704ccfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.625+0000 7f2a3b3f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a301b6b30 con 0x55be704a9a30 2024-04-24T08:19:56.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.625+0000 7f2a3b3f0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2a301b4400 con 0x55be704aa350 2024-04-24T08:19:56.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.625+0000 7f2a3b3f0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2a301b2270 con 0x55be704a5d50 2024-04-24T08:19:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.625+0000 7f2a35200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704a9a30 0x55be704cca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.625+0000 7f2a35200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704a9a30 0x55be704cca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58500/0 (socket says 172.21.15.28:58500) 2024-04-24T08:19:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.625+0000 7f2a35200640 1 -- 172.21.15.28:0/926526728 learned_addr learned my addr 172.21.15.28:0/926526728 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:56.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.625+0000 7f2a35c00640 1 --2- 172.21.15.28:0/926526728 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a5d50 0x55be704c0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.625+0000 7f2a36600640 1 --2- 172.21.15.28:0/926526728 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704aa350 0x55be704ccfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.626+0000 7f2a35200640 1 -- 172.21.15.28:0/926526728 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a5d50 msgr2=0x55be704c0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.626+0000 7f2a35200640 1 --2- 172.21.15.28:0/926526728 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a5d50 0x55be704c0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.626+0000 7f2a35200640 1 -- 172.21.15.28:0/926526728 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704aa350 msgr2=0x55be704ccfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.626+0000 7f2a35200640 1 --2- 172.21.15.28:0/926526728 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704aa350 0x55be704ccfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.626+0000 7f2a35200640 1 -- 172.21.15.28:0/926526728 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55be701213f0 con 0x55be704a9a30 2024-04-24T08:19:56.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.626+0000 7f2a35200640 1 --2- 172.21.15.28:0/926526728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704a9a30 0x55be704cca90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c003c20 tx=0x7f2a1c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17d17bfd9500e9dc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.626+0000 7f2a22000640 1 -- 172.21.15.28:0/926526728 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c023d30 con 0x55be704a9a30 2024-04-24T08:19:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.626+0000 7f2a22000640 1 -- 172.21.15.28:0/926526728 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a1c005500 con 0x55be704a9a30 2024-04-24T08:19:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.626+0000 7f2a22000640 1 -- 172.21.15.28:0/926526728 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a1c0216a0 con 0x55be704a9a30 2024-04-24T08:19:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.627+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/926526728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704a9a30 msgr2=0x55be704cca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.627+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/926526728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704a9a30 0x55be704cca90 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c003c20 tx=0x7f2a1c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.634 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.627+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/926526728 shutdown_connections 2024-04-24T08:19:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.627+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/926526728 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704a9a30 0x55be704cca90 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.627+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/926526728 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704aa350 0x55be704ccfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.627+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/926526728 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a5d50 0x55be704c0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.627+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/926526728 >> 172.21.15.28:0/926526728 conn(0x55be704b05f0 msgr2=0x55be704b2200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.627+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/926526728 shutdown_connections 2024-04-24T08:19:56.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.627+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/926526728 wait complete. 2024-04-24T08:19:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.628+0000 7f2a3b3f0740 1 -- start start 2024-04-24T08:19:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.628+0000 7f2a3b3f0740 1 Processor -- start 2024-04-24T08:19:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.628+0000 7f2a3b3f0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a5d50 0x55be704b8a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.628+0000 7f2a3b3f0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704a9a30 0x55be704b8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.628+0000 7f2a3b3f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704aa350 0x55be7052a300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.628+0000 7f2a3b3f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f2a301b28c0 con 0x55be704aa350 2024-04-24T08:19:56.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.628+0000 7f2a3b3f0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f2a301b3d30 con 0x55be704a9a30 2024-04-24T08:19:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.628+0000 7f2a3b3f0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f2a301b2f90 con 0x55be704a5d50 2024-04-24T08:19:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.628+0000 7f2a3b3f0740 10 client.0 _ll_register_callbacks cb 0x7f2a28005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:19:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.629+0000 7f2a36600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704aa350 0x55be7052a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.629+0000 7f2a35200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704a9a30 0x55be704b8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.629+0000 7f2a36600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704aa350 0x55be7052a300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58514/0 (socket says 172.21.15.28:58514) 2024-04-24T08:19:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.629+0000 7f2a36600640 1 -- 172.21.15.28:0/2693535921 learned_addr learned my addr 172.21.15.28:0/2693535921 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:19:56.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.629+0000 7f2a35c00640 1 --2- 172.21.15.28:0/2693535921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a5d50 0x55be704b8a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.629+0000 7f2a36600640 1 -- 172.21.15.28:0/2693535921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a5d50 msgr2=0x55be704b8a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.629+0000 7f2a36600640 1 --2- 172.21.15.28:0/2693535921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a5d50 0x55be704b8a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.629+0000 7f2a36600640 1 -- 172.21.15.28:0/2693535921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704a9a30 msgr2=0x55be704b8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:19:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.629+0000 7f2a36600640 1 --2- 172.21.15.28:0/2693535921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704a9a30 0x55be704b8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.629+0000 7f2a36600640 1 -- 172.21.15.28:0/2693535921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55be700e10f0 con 0x55be704aa350 2024-04-24T08:19:56.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.629+0000 7f2a35200640 1 --2- 172.21.15.28:0/2693535921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704a9a30 0x55be704b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:19:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.629+0000 7f2a36600640 1 --2- 172.21.15.28:0/2693535921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704aa350 0x55be7052a300 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2a28006010 tx=0x7f2a2803b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43a510e6855e221d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:19:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.629+0000 7f2a17400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:19:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.630+0000 7f2a17400640 1 -- 172.21.15.28:0/2693535921 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2a28049040 con 0x55be704aa350 2024-04-24T08:19:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.630+0000 7f2a17400640 1 -- 172.21.15.28:0/2693535921 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 8600+0+0 (secure 0 0 0) 0x7f2a28031bd0 con 0x55be704aa350 2024-04-24T08:19:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.630+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/2693535921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2a1c005500 con 0x55be704aa350 2024-04-24T08:19:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.630+0000 7f2a17400640 1 -- 172.21.15.28:0/2693535921 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f2a280bbf20 con 0x55be704aa350 2024-04-24T08:19:56.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.630+0000 7f2a3b3f0740 10 client.16677 fetch_fsmap learned FSMap version 26 2024-04-24T08:19:56.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.630+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/2693535921 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55be703d4810 con 0x55be704aa350 2024-04-24T08:19:56.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.630+0000 7f2a17400640 1 -- 172.21.15.28:0/2693535921 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a1c005500 con 0x55be704aa350 2024-04-24T08:19:56.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.630+0000 7f2a17400640 1 -- 172.21.15.28:0/2693535921 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f2a280db330 con 0x55be704aa350 2024-04-24T08:19:56.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.630+0000 7f2a3b3f0740 10 client.16677 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:19:56.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.630+0000 7f2a3b3f0740 10 client.16677 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:19:56.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.630+0000 7f2a3b3f0740 20 client.16677 populate_metadata read hostname 'smithi028' 2024-04-24T08:19:56.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.631+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/2693535921 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55be704b4a20 0x55be7052db30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:19:56.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.631+0000 7f2a3b3f0740 4 client.16677 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:19:56.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.631+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/2693535921 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55be70416a80 con 0x55be704b4a20 2024-04-24T08:19:56.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.631+0000 7f2a35c00640 1 --2- 172.21.15.28:0/2693535921 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55be704b4a20 0x55be7052db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:19:56.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.631+0000 7f2a35c00640 1 --2- 172.21.15.28:0/2693535921 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55be704b4a20 0x55be7052db30 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d5bbc330dd43829 server_cookie=b2f76137d24486f4 in_seq=0 out_seq=0 2024-04-24T08:19:56.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.631+0000 7f2a17400640 10 client.16677 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:56.642 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.632+0000 7f2a17400640 1 -- 172.21.15.28:0/2693535921 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x55be70416a80 con 0x55be704b4a20 2024-04-24T08:19:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.632+0000 7f2a17400640 10 client.16677 handle_command_reply: tid=0 2024-04-24T08:19:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.632+0000 7f2a3b3f0740 1 client.16677 shutdown 2024-04-24T08:19:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.632+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/2693535921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704aa350 msgr2=0x55be7052a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.632+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/2693535921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704aa350 0x55be7052a300 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f2a28006010 tx=0x7f2a2803b4f0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.632+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/2693535921 shutdown_connections 2024-04-24T08:19:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.632+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/2693535921 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55be704b4a20 0x55be7052db30 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.632+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/2693535921 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55be704aa350 0x55be7052a300 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.643 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.632+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/2693535921 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55be704a9a30 0x55be704b8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.632+0000 7f2a3b3f0740 1 --2- 172.21.15.28:0/2693535921 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55be704a5d50 0x55be704b8a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.632+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/2693535921 >> 172.21.15.28:0/2693535921 conn(0x55be704b05f0 msgr2=0x55be704a4740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.632+0000 7f2a17400640 0 client.16677 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:19:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.632+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/2693535921 shutdown_connections 2024-04-24T08:19:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.632+0000 7f2a3b3f0740 1 -- 172.21.15.28:0/2693535921 wait complete. 2024-04-24T08:19:56.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.632+0000 7f2a3b3f0740 20 client.16677 trim_cache size 0 max 16384 2024-04-24T08:19:56.645 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:19:56.645 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:19:56.645 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:19:56.645 INFO:teuthology.orchestra.run.smithi028.stdout: "7ba0ff79-0882-4a82-8139-857fd7104488": { 2024-04-24T08:19:56.645 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:19:56.645 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "7ba0ff79-0882-4a82-8139-857fd7104488", 2024-04-24T08:19:56.645 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:19:56.646 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:19:56.646 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:19:56.646 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:19:56.646 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.633+0000 7f2a37000640 1 -- 172.21.15.28:0/120220671 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2a180a3d20 msgr2=0x7f2a180a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.633+0000 7f2a37000640 1 --2- 172.21.15.28:0/120220671 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2a180a3d20 0x7f2a180a61e0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x55be704a6ee0 tx=0x7f2a1c00b5b0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.633+0000 7f2a37000640 1 -- 172.21.15.28:0/120220671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a30112ec0 msgr2=0x7f2a301a3640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:19:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.633+0000 7f2a37000640 1 --2- 172.21.15.28:0/120220671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a30112ec0 0x7f2a301a3640 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f2a28005ab0 tx=0x7f2a280040c0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.633+0000 7f2a37000640 1 -- 172.21.15.28:0/120220671 shutdown_connections 2024-04-24T08:19:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.633+0000 7f2a37000640 1 --2- 172.21.15.28:0/120220671 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f2a180a3d20 0x7f2a180a61e0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.633+0000 7f2a37000640 1 --2- 172.21.15.28:0/120220671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f2a30112ec0 0x7f2a301a3640 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.633+0000 7f2a37000640 1 --2- 172.21.15.28:0/120220671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a30074cc0 0x7f2a3019f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.647 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.633+0000 7f2a37000640 1 --2- 172.21.15.28:0/120220671 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f2a300743a0 0x7f2a3019ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:19:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.633+0000 7f2a37000640 1 -- 172.21.15.28:0/120220671 >> 172.21.15.28:0/120220671 conn(0x7f2a300fd9a0 msgr2=0x7f2a300ff130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:19:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.633+0000 7f2a37000640 1 -- 172.21.15.28:0/120220671 shutdown_connections 2024-04-24T08:19:56.648 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:19:56.633+0000 7f2a37000640 1 -- 172.21.15.28:0/120220671 wait complete. 2024-04-24T08:19:56.652 INFO:tasks.cephfs.filesystem:scrub status for tag:7ba0ff79-0882-4a82-8139-857fd7104488 - {'path': '/', 'tag': '7ba0ff79-0882-4a82-8139-857fd7104488', 'options': 'recursive,force'} 2024-04-24T08:19:56.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:56 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:19:56.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:56 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:56.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:56 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:19:56.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:56 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:56.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:56 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:19:56.790 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:56 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:57.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:56 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:19:57.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:56 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:57.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:56 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:19:57.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:56 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:57.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:56 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:19:57.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:56 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:57.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:56 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:19:57.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:56 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:57.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:56 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:19:57.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:56 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:57.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:56 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:19:57.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:56 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:58.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:57 smithi028 ceph-mon[27786]: scrub queued for path: / 2024-04-24T08:19:58.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:57 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:58.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:57 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:19:58.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:57 smithi028 ceph-mon[27786]: pgmap v297: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:19:58.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:57 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:58.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:57 smithi117 ceph-mon[32438]: scrub queued for path: / 2024-04-24T08:19:58.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:57 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:58.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:57 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:19:58.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:57 smithi117 ceph-mon[32438]: pgmap v297: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:19:58.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:57 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:58.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:57 smithi194 ceph-mon[32836]: scrub queued for path: / 2024-04-24T08:19:58.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:57 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:19:58.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:57 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:19:58.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:57 smithi194 ceph-mon[32836]: pgmap v297: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:19:58.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:57 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:20:00.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:19:59 smithi028 ceph-mon[27786]: pgmap v298: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-04-24T08:20:00.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:19:59 smithi117 ceph-mon[32438]: pgmap v298: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-04-24T08:20:00.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:19:59 smithi194 ceph-mon[32836]: pgmap v298: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-04-24T08:20:01.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:00 smithi028 ceph-mon[27786]: overall HEALTH_OK 2024-04-24T08:20:01.091 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:00 smithi194 ceph-mon[32836]: overall HEALTH_OK 2024-04-24T08:20:01.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:00 smithi117 ceph-mon[32438]: overall HEALTH_OK 2024-04-24T08:20:02.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:01 smithi028 ceph-mon[27786]: pgmap v299: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-24T08:20:02.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:01 smithi117 ceph-mon[32438]: pgmap v299: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-24T08:20:02.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:01 smithi194 ceph-mon[32836]: pgmap v299: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-04-24T08:20:04.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:03 smithi028 ceph-mon[27786]: pgmap v300: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:20:04.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:03 smithi117 ceph-mon[32438]: pgmap v300: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:20:04.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:03 smithi194 ceph-mon[32836]: pgmap v300: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:20:05.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:04 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:20:05.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:04 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:20:05.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:04 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:20:05.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:04 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:20:05.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:04 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:20:05.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:04 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:20:06.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:05 smithi028 ceph-mon[27786]: pgmap v301: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-24T08:20:06.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:05 smithi117 ceph-mon[32438]: pgmap v301: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-24T08:20:06.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:05 smithi194 ceph-mon[32836]: pgmap v301: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-24T08:20:08.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:07 smithi028 ceph-mon[27786]: pgmap v302: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-24T08:20:08.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:07 smithi117 ceph-mon[32438]: pgmap v302: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-24T08:20:08.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:07 smithi194 ceph-mon[32836]: pgmap v302: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-24T08:20:10.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:09 smithi028 ceph-mon[27786]: pgmap v303: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-24T08:20:10.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:09 smithi028 ceph-mon[27786]: osdmap e98: 12 total, 12 up, 12 in 2024-04-24T08:20:10.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:09 smithi117 ceph-mon[32438]: pgmap v303: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-24T08:20:10.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:09 smithi117 ceph-mon[32438]: osdmap e98: 12 total, 12 up, 12 in 2024-04-24T08:20:10.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:09 smithi194 ceph-mon[32836]: pgmap v303: 97 pgs: 97 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-24T08:20:10.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:09 smithi194 ceph-mon[32836]: osdmap e98: 12 total, 12 up, 12 in 2024-04-24T08:20:12.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:11 smithi028 ceph-mon[27786]: pgmap v305: 97 pgs: 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 79 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-24T08:20:12.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:11 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:20:12.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:11 smithi117 ceph-mon[32438]: pgmap v305: 97 pgs: 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 79 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-24T08:20:12.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:11 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:20:12.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:11 smithi194 ceph-mon[32836]: pgmap v305: 97 pgs: 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 79 active+clean; 21 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-24T08:20:12.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:11 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:20:14.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:13 smithi117 ceph-mon[32438]: pgmap v306: 97 pgs: 31 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 49 active+clean; 20 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:20:14.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:13 smithi194 ceph-mon[32836]: pgmap v306: 97 pgs: 31 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 49 active+clean; 20 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:20:14.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:13 smithi028 ceph-mon[27786]: pgmap v306: 97 pgs: 31 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 49 active+clean; 20 GiB data, 52 GiB used, 1021 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:20:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:14 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:20:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:14 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:20:15.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:14 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:20:15.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:14 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:20:15.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:14 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:20:15.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:14 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:20:15.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:14 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:20:15.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:14 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:20:15.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:14 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:20:16.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:15 smithi117 ceph-mon[32438]: pgmap v307: 97 pgs: 37 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 40 active+clean; 19 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:20:16.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:15 smithi194 ceph-mon[32836]: pgmap v307: 97 pgs: 37 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 40 active+clean; 19 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:20:16.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:15 smithi028 ceph-mon[27786]: pgmap v307: 97 pgs: 37 active+clean+snaptrim_wait, 20 active+clean+snaptrim, 40 active+clean; 19 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 4.4 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:20:18.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:17 smithi117 ceph-mon[32438]: pgmap v308: 97 pgs: 35 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 43 active+clean; 19 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:20:18.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:17 smithi194 ceph-mon[32836]: pgmap v308: 97 pgs: 35 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 43 active+clean; 19 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:20:18.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:17 smithi028 ceph-mon[27786]: pgmap v308: 97 pgs: 35 active+clean+snaptrim_wait, 19 active+clean+snaptrim, 43 active+clean; 19 GiB data, 51 GiB used, 1022 GiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:20:20.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:19 smithi117 ceph-mon[32438]: pgmap v309: 97 pgs: 30 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 50 active+clean; 18 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:20:20.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:19 smithi194 ceph-mon[32836]: pgmap v309: 97 pgs: 30 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 50 active+clean; 18 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:20:20.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:19 smithi028 ceph-mon[27786]: pgmap v309: 97 pgs: 30 active+clean+snaptrim_wait, 17 active+clean+snaptrim, 50 active+clean; 18 GiB data, 49 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:20:22.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:21 smithi117 ceph-mon[32438]: pgmap v310: 97 pgs: 22 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 66 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 175 B/s wr, 5 op/s 2024-04-24T08:20:22.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:21 smithi194 ceph-mon[32836]: pgmap v310: 97 pgs: 22 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 66 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 175 B/s wr, 5 op/s 2024-04-24T08:20:22.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:21 smithi028 ceph-mon[27786]: pgmap v310: 97 pgs: 22 active+clean+snaptrim_wait, 9 active+clean+snaptrim, 66 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 175 B/s wr, 5 op/s 2024-04-24T08:20:24.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:23 smithi117 ceph-mon[32438]: pgmap v311: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-24T08:20:24.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:23 smithi194 ceph-mon[32836]: pgmap v311: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-24T08:20:24.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:23 smithi028 ceph-mon[27786]: pgmap v311: 97 pgs: 6 active+clean+snaptrim_wait, 2 active+clean+snaptrim, 89 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 6 op/s 2024-04-24T08:20:26.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:25 smithi117 ceph-mon[32438]: pgmap v312: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-24T08:20:26.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:25 smithi194 ceph-mon[32836]: pgmap v312: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-24T08:20:26.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:25 smithi028 ceph-mon[27786]: pgmap v312: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-24T08:20:26.653 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:20:26.653 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:20:26.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.886+0000 7efd67200640 1 -- 172.21.15.28:0/3611441533 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd68112ec0 msgr2=0x7efd681152b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.886+0000 7efd67200640 1 --2- 172.21.15.28:0/3611441533 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd68112ec0 0x7efd681152b0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7efd60009be0 tx=0x7efd6002a750 comp rx=0 tx=0).stop 2024-04-24T08:20:26.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.886+0000 7efd67200640 1 -- 172.21.15.28:0/3611441533 shutdown_connections 2024-04-24T08:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.886+0000 7efd67200640 1 --2- 172.21.15.28:0/3611441533 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd68074cc0 0x7efd68112980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.886+0000 7efd67200640 1 --2- 172.21.15.28:0/3611441533 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd68112ec0 0x7efd681152b0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.886+0000 7efd67200640 1 --2- 172.21.15.28:0/3611441533 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd680743a0 0x7efd68074780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.886+0000 7efd67200640 1 -- 172.21.15.28:0/3611441533 >> 172.21.15.28:0/3611441533 conn(0x7efd680fd9a0 msgr2=0x7efd680ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.886+0000 7efd67200640 1 -- 172.21.15.28:0/3611441533 shutdown_connections 2024-04-24T08:20:26.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.886+0000 7efd67200640 1 -- 172.21.15.28:0/3611441533 wait complete. 2024-04-24T08:20:26.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.887+0000 7efd67200640 1 Processor -- start 2024-04-24T08:20:26.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.887+0000 7efd67200640 1 -- start start 2024-04-24T08:20:26.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.888+0000 7efd67200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd680743a0 0x7efd6819edc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.890 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.888+0000 7efd67200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd68074cc0 0x7efd6819f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.888+0000 7efd67200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd68112ec0 0x7efd681a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.888+0000 7efd67200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd68118880 con 0x7efd68112ec0 2024-04-24T08:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.888+0000 7efd67200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efd68118700 con 0x7efd68074cc0 2024-04-24T08:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.888+0000 7efd67200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efd68118a00 con 0x7efd680743a0 2024-04-24T08:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.888+0000 7efd65e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd680743a0 0x7efd6819edc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.888+0000 7efd66800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd68112ec0 0x7efd681a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.888+0000 7efd65e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd680743a0 0x7efd6819edc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:54230/0 (socket says 172.21.15.28:54230) 2024-04-24T08:20:26.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.888+0000 7efd65e00640 1 -- 172.21.15.28:0/1711350967 learned_addr learned my addr 172.21.15.28:0/1711350967 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.888+0000 7efd5d400640 1 --2- 172.21.15.28:0/1711350967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd68074cc0 0x7efd6819f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.888+0000 7efd65e00640 1 -- 172.21.15.28:0/1711350967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd68074cc0 msgr2=0x7efd6819f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.888+0000 7efd65e00640 1 --2- 172.21.15.28:0/1711350967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd68074cc0 0x7efd6819f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.888+0000 7efd65e00640 1 -- 172.21.15.28:0/1711350967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd68112ec0 msgr2=0x7efd681a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.888+0000 7efd65e00640 1 --2- 172.21.15.28:0/1711350967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd68112ec0 0x7efd681a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.888+0000 7efd65e00640 1 -- 172.21.15.28:0/1711350967 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd681a3db0 con 0x7efd680743a0 2024-04-24T08:20:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.889+0000 7efd65e00640 1 --2- 172.21.15.28:0/1711350967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd680743a0 0x7efd6819edc0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7efd5800e9f0 tx=0x7efd5800edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.889+0000 7efd5f400640 1 -- 172.21.15.28:0/1711350967 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5800ccc0 con 0x7efd680743a0 2024-04-24T08:20:26.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.889+0000 7efd67200640 1 -- 172.21.15.28:0/1711350967 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd681a40a0 con 0x7efd680743a0 2024-04-24T08:20:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.889+0000 7efd5f400640 1 -- 172.21.15.28:0/1711350967 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd5800ce60 con 0x7efd680743a0 2024-04-24T08:20:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.889+0000 7efd5f400640 1 -- 172.21.15.28:0/1711350967 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd58010690 con 0x7efd680743a0 2024-04-24T08:20:26.893 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.889+0000 7efd67200640 1 -- 172.21.15.28:0/1711350967 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd681ab930 con 0x7efd680743a0 2024-04-24T08:20:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.892+0000 7efd6fdea740 1 Processor -- start 2024-04-24T08:20:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.892+0000 7efd6fdea740 1 -- start start 2024-04-24T08:20:26.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.893+0000 7efd5f400640 1 -- 172.21.15.28:0/1711350967 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7efd580046c0 con 0x7efd680743a0 2024-04-24T08:20:26.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.893+0000 7efd6fdea740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa765a30 0x55b7aa765e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.893+0000 7efd6fdea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa766350 0x55b7aa7704e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.893+0000 7efd6fdea740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa770a20 0x55b7aa772e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.893+0000 7efd6fdea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b7aa6cb1f0 con 0x55b7aa766350 2024-04-24T08:20:26.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.893+0000 7efd6fdea740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b7aa671350 con 0x55b7aa765a30 2024-04-24T08:20:26.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.893+0000 7efd6fdea740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b7aa6c71b0 con 0x55b7aa770a20 2024-04-24T08:20:26.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.893+0000 7efd65e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa766350 0x55b7aa7704e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.896 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.893+0000 7efd65e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa766350 0x55b7aa7704e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37148/0 (socket says 172.21.15.28:37148) 2024-04-24T08:20:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.893+0000 7efd65e00640 1 -- 172.21.15.28:0/3822747987 learned_addr learned my addr 172.21.15.28:0/3822747987 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.893+0000 7efd66800640 1 --2- 172.21.15.28:0/3822747987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa765a30 0x55b7aa765e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.893+0000 7efd5d400640 1 --2- 172.21.15.28:0/3822747987 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa770a20 0x55b7aa772e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.894+0000 7efd65e00640 1 -- 172.21.15.28:0/3822747987 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa770a20 msgr2=0x55b7aa772e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.894+0000 7efd65e00640 1 --2- 172.21.15.28:0/3822747987 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa770a20 0x55b7aa772e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.894+0000 7efd65e00640 1 -- 172.21.15.28:0/3822747987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa765a30 msgr2=0x55b7aa765e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.894+0000 7efd65e00640 1 --2- 172.21.15.28:0/3822747987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa765a30 0x55b7aa765e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.897 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.894+0000 7efd65e00640 1 -- 172.21.15.28:0/3822747987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7aa6c5f50 con 0x55b7aa766350 2024-04-24T08:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.894+0000 7efd66800640 1 --2- 172.21.15.28:0/3822747987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa765a30 0x55b7aa765e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.894+0000 7efd65e00640 1 --2- 172.21.15.28:0/3822747987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa766350 0x55b7aa7704e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7efd580ce1e0 tx=0x7efd580d4cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=51756864a05cf4b4 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.895+0000 7efd57400640 1 -- 172.21.15.28:0/3822747987 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd580d9d10 con 0x55b7aa766350 2024-04-24T08:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.895+0000 7efd57400640 1 -- 172.21.15.28:0/3822747987 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd580e38e0 con 0x55b7aa766350 2024-04-24T08:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.895+0000 7efd57400640 1 -- 172.21.15.28:0/3822747987 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd580db820 con 0x55b7aa766350 2024-04-24T08:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.895+0000 7efd5f400640 1 --2- 172.21.15.28:0/1711350967 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efd480a3da0 0x7efd480a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.895+0000 7efd5f400640 1 -- 172.21.15.28:0/1711350967 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8644+0+0 (secure 0 0 0) 0x7efd580d0c30 con 0x7efd680743a0 2024-04-24T08:20:26.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.895+0000 7efd66800640 1 --2- 172.21.15.28:0/1711350967 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efd480a3da0 0x7efd480a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.895+0000 7efd6fdea740 1 -- 172.21.15.28:0/3822747987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa766350 msgr2=0x55b7aa7704e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.895+0000 7efd6fdea740 1 --2- 172.21.15.28:0/3822747987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa766350 0x55b7aa7704e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7efd580ce1e0 tx=0x7efd580d4cd0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.896+0000 7efd6fdea740 1 -- 172.21.15.28:0/3822747987 shutdown_connections 2024-04-24T08:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.896+0000 7efd6fdea740 1 --2- 172.21.15.28:0/3822747987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa766350 0x55b7aa7704e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.896+0000 7efd6fdea740 1 --2- 172.21.15.28:0/3822747987 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa770a20 0x55b7aa772e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.896+0000 7efd6fdea740 1 --2- 172.21.15.28:0/3822747987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa765a30 0x55b7aa765e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.896+0000 7efd6fdea740 1 -- 172.21.15.28:0/3822747987 >> 172.21.15.28:0/3822747987 conn(0x55b7aa761d50 msgr2=0x55b7aa762130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.896+0000 7efd6fdea740 1 -- 172.21.15.28:0/3822747987 shutdown_connections 2024-04-24T08:20:26.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.896+0000 7efd66800640 1 --2- 172.21.15.28:0/1711350967 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efd480a3da0 0x7efd480a6260 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efd6002ac70 tx=0x7efd60005fd0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.896+0000 7efd6fdea740 1 -- 172.21.15.28:0/3822747987 wait complete. 2024-04-24T08:20:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.896+0000 7efd6fdea740 1 -- start start 2024-04-24T08:20:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.896+0000 7efd6fdea740 1 Processor -- start 2024-04-24T08:20:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.897+0000 7efd6fdea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 0x55b7aa76da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.897+0000 7efd6fdea740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa766350 0x55b7aa76dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.897+0000 7efd6fdea740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa770a20 0x55b7aa76e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.897+0000 7efd6fdea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b7aa773810 con 0x55b7aa765a30 2024-04-24T08:20:26.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.897+0000 7efd6fdea740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b7aa684690 con 0x55b7aa766350 2024-04-24T08:20:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.897+0000 7efd6fdea740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b7aa773990 con 0x55b7aa770a20 2024-04-24T08:20:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.897+0000 7efd6fdea740 10 client.0 _ll_register_callbacks cb 0x55b7aa6c0650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.897+0000 7efd65e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 0x55b7aa76da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.897+0000 7efd65e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 0x55b7aa76da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37154/0 (socket says 172.21.15.28:37154) 2024-04-24T08:20:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.897+0000 7efd65e00640 1 -- 172.21.15.28:0/4029407424 learned_addr learned my addr 172.21.15.28:0/4029407424 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.897+0000 7efd5d400640 1 --2- 172.21.15.28:0/4029407424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa766350 0x55b7aa76dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.897+0000 7efd66800640 1 --2- 172.21.15.28:0/4029407424 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa770a20 0x55b7aa76e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.898+0000 7efd65e00640 1 -- 172.21.15.28:0/4029407424 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa770a20 msgr2=0x55b7aa76e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:26.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.898+0000 7efd65e00640 1 --2- 172.21.15.28:0/4029407424 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa770a20 0x55b7aa76e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.898+0000 7efd65e00640 1 -- 172.21.15.28:0/4029407424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa766350 msgr2=0x55b7aa76dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:26.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.898+0000 7efd65e00640 1 --2- 172.21.15.28:0/4029407424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa766350 0x55b7aa76dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.898+0000 7efd65e00640 1 -- 172.21.15.28:0/4029407424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7aa683920 con 0x55b7aa765a30 2024-04-24T08:20:26.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.898+0000 7efd65e00640 1 --2- 172.21.15.28:0/4029407424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 0x55b7aa76da80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7efd58098f50 tx=0x7efd580d4f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8dde49fbc256531 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:26.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.898+0000 7efd4fe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:20:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.898+0000 7efd4fe00640 1 -- 172.21.15.28:0/4029407424 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd58098d90 con 0x55b7aa765a30 2024-04-24T08:20:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.898+0000 7efd4fe00640 1 -- 172.21.15.28:0/4029407424 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8644+0+0 (secure 0 0 0) 0x7efd580ce520 con 0x55b7aa765a30 2024-04-24T08:20:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.898+0000 7efd6fdea740 1 -- 172.21.15.28:0/4029407424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd580e38e0 con 0x55b7aa765a30 2024-04-24T08:20:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.899+0000 7efd4fe00640 1 -- 172.21.15.28:0/4029407424 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7efd5809c450 con 0x55b7aa765a30 2024-04-24T08:20:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.899+0000 7efd6fdea740 10 client.16689 fetch_fsmap learned FSMap version 26 2024-04-24T08:20:26.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.899+0000 7efd6fdea740 1 -- 172.21.15.28:0/4029407424 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b7aa4a77d0 con 0x55b7aa765a30 2024-04-24T08:20:26.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.899+0000 7efd4fe00640 1 -- 172.21.15.28:0/4029407424 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd580e38e0 con 0x55b7aa765a30 2024-04-24T08:20:26.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.899+0000 7efd4fe00640 1 -- 172.21.15.28:0/4029407424 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7efd580b8020 con 0x55b7aa765a30 2024-04-24T08:20:26.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.899+0000 7efd6fdea740 10 client.16689 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:20:26.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.899+0000 7efd6fdea740 10 client.16689 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:20:26.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.899+0000 7efd6fdea740 20 client.16689 populate_metadata read hostname 'smithi028' 2024-04-24T08:20:26.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.899+0000 7efd6fdea740 1 --2- 172.21.15.28:0/4029407424 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55b7aa780d10 0x55b7aa783100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.899+0000 7efd6fdea740 4 client.16689 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:20:26.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.899+0000 7efd6fdea740 1 -- 172.21.15.28:0/4029407424 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7aa6d8c10 con 0x55b7aa780d10 2024-04-24T08:20:26.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.900+0000 7efd5d400640 1 --2- 172.21.15.28:0/4029407424 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55b7aa780d10 0x55b7aa783100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.900+0000 7efd5d400640 1 --2- 172.21.15.28:0/4029407424 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55b7aa780d10 0x55b7aa783100 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5c7451ea3f8dde5 server_cookie=f224141b74bbacf6 in_seq=0 out_seq=0 2024-04-24T08:20:26.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.900+0000 7efd4fe00640 10 client.16689 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:20:26.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.901+0000 7efd4fe00640 1 -- 172.21.15.28:0/4029407424 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b7aa6d8c10 con 0x55b7aa780d10 2024-04-24T08:20:26.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.901+0000 7efd4fe00640 10 client.16689 handle_command_reply: tid=0 2024-04-24T08:20:26.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.901+0000 7efd6fdea740 1 client.16689 shutdown 2024-04-24T08:20:26.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.901+0000 7efd6fdea740 1 -- 172.21.15.28:0/4029407424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 msgr2=0x55b7aa76da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.901+0000 7efd6fdea740 1 --2- 172.21.15.28:0/4029407424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 0x55b7aa76da80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7efd58098f50 tx=0x7efd580d4f60 comp rx=0 tx=0).stop 2024-04-24T08:20:26.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.901+0000 7efd6fdea740 1 -- 172.21.15.28:0/4029407424 shutdown_connections 2024-04-24T08:20:26.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.901+0000 7efd6fdea740 1 --2- 172.21.15.28:0/4029407424 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55b7aa780d10 0x55b7aa783100 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.901+0000 7efd6fdea740 1 --2- 172.21.15.28:0/4029407424 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa770a20 0x55b7aa76e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.901+0000 7efd6fdea740 1 --2- 172.21.15.28:0/4029407424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa766350 0x55b7aa76dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.901+0000 7efd6fdea740 1 --2- 172.21.15.28:0/4029407424 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 0x55b7aa76da80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.901+0000 7efd6fdea740 1 -- 172.21.15.28:0/4029407424 >> 172.21.15.28:0/4029407424 conn(0x55b7aa761d50 msgr2=0x55b7aa760fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:26.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.901+0000 7efd4fe00640 0 client.16689 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:20:26.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.901+0000 7efd6fdea740 1 -- 172.21.15.28:0/4029407424 shutdown_connections 2024-04-24T08:20:26.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.902+0000 7efd6fdea740 1 -- 172.21.15.28:0/4029407424 wait complete. 2024-04-24T08:20:26.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.902+0000 7efd6fdea740 20 client.16689 trim_cache size 0 max 16384 2024-04-24T08:20:26.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.921+0000 7efd6fdea740 1 Processor -- start 2024-04-24T08:20:26.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.921+0000 7efd6fdea740 1 -- start start 2024-04-24T08:20:26.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.921+0000 7efd6fdea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 0x55b7aa761750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.921+0000 7efd6fdea740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa766350 0x55b7aa76e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.922+0000 7efd6fdea740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa770a20 0x55b7aa76e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.922+0000 7efd6fdea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd681b6c60 con 0x55b7aa765a30 2024-04-24T08:20:26.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.922+0000 7efd6fdea740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efd681b4530 con 0x55b7aa770a20 2024-04-24T08:20:26.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.922+0000 7efd6fdea740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efd681b23a0 con 0x55b7aa766350 2024-04-24T08:20:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.922+0000 7efd5d400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa766350 0x55b7aa76e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.924 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.922+0000 7efd5d400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa766350 0x55b7aa76e150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:54252/0 (socket says 172.21.15.28:54252) 2024-04-24T08:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.922+0000 7efd5d400640 1 -- 172.21.15.28:0/714998094 learned_addr learned my addr 172.21.15.28:0/714998094 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.922+0000 7efd65e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 0x55b7aa761750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.922+0000 7efd66800640 1 --2- 172.21.15.28:0/714998094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa770a20 0x55b7aa76e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.922+0000 7efd65e00640 1 -- 172.21.15.28:0/714998094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa766350 msgr2=0x55b7aa76e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.922+0000 7efd65e00640 1 --2- 172.21.15.28:0/714998094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa766350 0x55b7aa76e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.922+0000 7efd65e00640 1 -- 172.21.15.28:0/714998094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa770a20 msgr2=0x55b7aa76e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.922+0000 7efd65e00640 1 --2- 172.21.15.28:0/714998094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa770a20 0x55b7aa76e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.925 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.922+0000 7efd65e00640 1 -- 172.21.15.28:0/714998094 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7aa3dd3f0 con 0x55b7aa765a30 2024-04-24T08:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.923+0000 7efd65e00640 1 --2- 172.21.15.28:0/714998094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 0x55b7aa761750 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7efd5809c7f0 tx=0x7efd580d4cd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c204cc3889e74956 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.923+0000 7efd56a00640 1 -- 172.21.15.28:0/714998094 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5809c630 con 0x55b7aa765a30 2024-04-24T08:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.923+0000 7efd56a00640 1 -- 172.21.15.28:0/714998094 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd580d44b0 con 0x55b7aa765a30 2024-04-24T08:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.923+0000 7efd56a00640 1 -- 172.21.15.28:0/714998094 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd5809c8c0 con 0x55b7aa765a30 2024-04-24T08:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.924+0000 7efd6fdea740 1 -- 172.21.15.28:0/714998094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 msgr2=0x55b7aa761750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.924+0000 7efd6fdea740 1 --2- 172.21.15.28:0/714998094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 0x55b7aa761750 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7efd5809c7f0 tx=0x7efd580d4cd0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.924+0000 7efd6fdea740 1 -- 172.21.15.28:0/714998094 shutdown_connections 2024-04-24T08:20:26.926 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.924+0000 7efd6fdea740 1 --2- 172.21.15.28:0/714998094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa770a20 0x55b7aa76e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.924+0000 7efd6fdea740 1 --2- 172.21.15.28:0/714998094 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa766350 0x55b7aa76e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.924+0000 7efd6fdea740 1 --2- 172.21.15.28:0/714998094 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 0x55b7aa761750 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.924+0000 7efd6fdea740 1 -- 172.21.15.28:0/714998094 >> 172.21.15.28:0/714998094 conn(0x55b7aa761d50 msgr2=0x55b7aa76da40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.924+0000 7efd6fdea740 1 -- 172.21.15.28:0/714998094 shutdown_connections 2024-04-24T08:20:26.927 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.924+0000 7efd6fdea740 1 -- 172.21.15.28:0/714998094 wait complete. 2024-04-24T08:20:26.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.924+0000 7efd6fdea740 1 -- start start 2024-04-24T08:20:26.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.925+0000 7efd6fdea740 1 Processor -- start 2024-04-24T08:20:26.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.925+0000 7efd6fdea740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 0x55b7aa783df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.925+0000 7efd6fdea740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa766350 0x55b7aa784330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.931 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.925+0000 7efd6fdea740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa770a20 0x55b7aa784870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.925+0000 7efd6fdea740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd681b29f0 con 0x55b7aa765a30 2024-04-24T08:20:26.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.925+0000 7efd6fdea740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efd681b3e60 con 0x55b7aa770a20 2024-04-24T08:20:26.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.925+0000 7efd6fdea740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efd681b30c0 con 0x55b7aa766350 2024-04-24T08:20:26.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.925+0000 7efd6fdea740 10 client.0 _ll_register_callbacks cb 0x7efd50005ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:26.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.925+0000 7efd65e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 0x55b7aa783df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.926+0000 7efd65e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 0x55b7aa783df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37184/0 (socket says 172.21.15.28:37184) 2024-04-24T08:20:26.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.926+0000 7efd65e00640 1 -- 172.21.15.28:0/2078064332 learned_addr learned my addr 172.21.15.28:0/2078064332 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:26.932 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.926+0000 7efd66800640 1 --2- 172.21.15.28:0/2078064332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa770a20 0x55b7aa784870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.926+0000 7efd5d400640 1 --2- 172.21.15.28:0/2078064332 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa766350 0x55b7aa784330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.926+0000 7efd65e00640 1 -- 172.21.15.28:0/2078064332 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa766350 msgr2=0x55b7aa784330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:26.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.926+0000 7efd65e00640 1 --2- 172.21.15.28:0/2078064332 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa766350 0x55b7aa784330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.926+0000 7efd65e00640 1 -- 172.21.15.28:0/2078064332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa770a20 msgr2=0x55b7aa784870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:26.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.926+0000 7efd65e00640 1 --2- 172.21.15.28:0/2078064332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa770a20 0x55b7aa784870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.926+0000 7efd65e00640 1 -- 172.21.15.28:0/2078064332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7aa39d0f0 con 0x55b7aa765a30 2024-04-24T08:20:26.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.926+0000 7efd65e00640 1 --2- 172.21.15.28:0/2078064332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 0x55b7aa783df0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7efd580d4180 tx=0x7efd580a0f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17985f91caf3b00f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:26.933 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.926+0000 7efd4fe00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:20:26.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.927+0000 7efd4fe00640 1 -- 172.21.15.28:0/2078064332 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd580ab060 con 0x55b7aa765a30 2024-04-24T08:20:26.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.927+0000 7efd4fe00640 1 -- 172.21.15.28:0/2078064332 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8644+0+0 (secure 0 0 0) 0x7efd580a52d0 con 0x55b7aa765a30 2024-04-24T08:20:26.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.927+0000 7efd6fdea740 1 -- 172.21.15.28:0/2078064332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd580d44b0 con 0x55b7aa765a30 2024-04-24T08:20:26.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.927+0000 7efd4fe00640 1 -- 172.21.15.28:0/2078064332 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7efd5809c8c0 con 0x55b7aa765a30 2024-04-24T08:20:26.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.927+0000 7efd6fdea740 10 client.16695 fetch_fsmap learned FSMap version 26 2024-04-24T08:20:26.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.927+0000 7efd6fdea740 1 -- 172.21.15.28:0/2078064332 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b7aa690810 con 0x55b7aa765a30 2024-04-24T08:20:26.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.927+0000 7efd4fe00640 1 -- 172.21.15.28:0/2078064332 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd580d44b0 con 0x55b7aa765a30 2024-04-24T08:20:26.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.927+0000 7efd4fe00640 1 -- 172.21.15.28:0/2078064332 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7efd580b8070 con 0x55b7aa765a30 2024-04-24T08:20:26.934 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.927+0000 7efd6fdea740 10 client.16695 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:20:26.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.927+0000 7efd6fdea740 10 client.16695 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:20:26.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.927+0000 7efd6fdea740 20 client.16695 populate_metadata read hostname 'smithi028' 2024-04-24T08:20:26.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.927+0000 7efd6fdea740 1 --2- 172.21.15.28:0/2078064332 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55b7aa7e73c0 0x55b7aa7e97b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:26.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.927+0000 7efd6fdea740 4 client.16695 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:20:26.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.927+0000 7efd6fdea740 1 -- 172.21.15.28:0/2078064332 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b7aa6d2a80 con 0x55b7aa7e73c0 2024-04-24T08:20:26.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.928+0000 7efd5d400640 1 --2- 172.21.15.28:0/2078064332 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55b7aa7e73c0 0x55b7aa7e97b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:26.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.928+0000 7efd5d400640 1 --2- 172.21.15.28:0/2078064332 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55b7aa7e73c0 0x55b7aa7e97b0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=613cbffe31d02568 server_cookie=b8a8c19d3bde5aa2 in_seq=0 out_seq=0 2024-04-24T08:20:26.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.928+0000 7efd4fe00640 10 client.16695 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:20:26.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.928+0000 7efd4fe00640 1 -- 172.21.15.28:0/2078064332 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b7aa6d2a80 con 0x55b7aa7e73c0 2024-04-24T08:20:26.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.928+0000 7efd4fe00640 10 client.16695 handle_command_reply: tid=0 2024-04-24T08:20:26.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.928+0000 7efd6fdea740 1 client.16695 shutdown 2024-04-24T08:20:26.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.929+0000 7efd6fdea740 1 -- 172.21.15.28:0/2078064332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 msgr2=0x55b7aa783df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.929+0000 7efd6fdea740 1 --2- 172.21.15.28:0/2078064332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 0x55b7aa783df0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7efd580d4180 tx=0x7efd580a0f90 comp rx=0 tx=0).stop 2024-04-24T08:20:26.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.929+0000 7efd6fdea740 1 -- 172.21.15.28:0/2078064332 shutdown_connections 2024-04-24T08:20:26.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.929+0000 7efd6fdea740 1 --2- 172.21.15.28:0/2078064332 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55b7aa7e73c0 0x55b7aa7e97b0 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.936 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.929+0000 7efd6fdea740 1 --2- 172.21.15.28:0/2078064332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b7aa770a20 0x55b7aa784870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.929+0000 7efd6fdea740 1 --2- 172.21.15.28:0/2078064332 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b7aa766350 0x55b7aa784330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.929+0000 7efd6fdea740 1 --2- 172.21.15.28:0/2078064332 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b7aa765a30 0x55b7aa783df0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.929+0000 7efd4fe00640 0 client.16695 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:20:26.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.929+0000 7efd6fdea740 1 -- 172.21.15.28:0/2078064332 >> 172.21.15.28:0/2078064332 conn(0x55b7aa761d50 msgr2=0x55b7aa767700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:26.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.929+0000 7efd6fdea740 1 -- 172.21.15.28:0/2078064332 shutdown_connections 2024-04-24T08:20:26.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.929+0000 7efd6fdea740 1 -- 172.21.15.28:0/2078064332 wait complete. 2024-04-24T08:20:26.937 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.929+0000 7efd6fdea740 20 client.16695 trim_cache size 0 max 16384 2024-04-24T08:20:26.938 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:20:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-24T08:20:26.938 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-24T08:20:26.938 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:20:26.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.930+0000 7efd67200640 1 -- 172.21.15.28:0/1711350967 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efd480a3da0 msgr2=0x7efd480a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.930+0000 7efd67200640 1 --2- 172.21.15.28:0/1711350967 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efd480a3da0 0x7efd480a6260 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7efd6002ac70 tx=0x7efd60005fd0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.930+0000 7efd67200640 1 -- 172.21.15.28:0/1711350967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd680743a0 msgr2=0x7efd6819edc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:26.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.930+0000 7efd67200640 1 --2- 172.21.15.28:0/1711350967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd680743a0 0x7efd6819edc0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7efd5800e9f0 tx=0x7efd5800edf0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.939 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.930+0000 7efd67200640 1 -- 172.21.15.28:0/1711350967 shutdown_connections 2024-04-24T08:20:26.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.930+0000 7efd67200640 1 --2- 172.21.15.28:0/1711350967 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efd480a3da0 0x7efd480a6260 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.930+0000 7efd67200640 1 --2- 172.21.15.28:0/1711350967 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd68112ec0 0x7efd681a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.930+0000 7efd67200640 1 --2- 172.21.15.28:0/1711350967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd68074cc0 0x7efd6819f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.930+0000 7efd67200640 1 --2- 172.21.15.28:0/1711350967 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd680743a0 0x7efd6819edc0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:26.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.930+0000 7efd67200640 1 -- 172.21.15.28:0/1711350967 >> 172.21.15.28:0/1711350967 conn(0x7efd680fd9a0 msgr2=0x7efd680ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:26.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.930+0000 7efd67200640 1 -- 172.21.15.28:0/1711350967 shutdown_connections 2024-04-24T08:20:26.940 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:26.930+0000 7efd67200640 1 -- 172.21.15.28:0/1711350967 wait complete. 2024-04-24T08:20:26.950 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:20:26.951 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:20:27.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.167+0000 7fbda3e00640 1 -- 172.21.15.28:0/3387926058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbda4104260 msgr2=0x7fbda4104640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.167+0000 7fbda3e00640 1 --2- 172.21.15.28:0/3387926058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbda4104260 0x7fbda4104640 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fbd94009860 tx=0x7fbd9402a340 comp rx=0 tx=0).stop 2024-04-24T08:20:27.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.167+0000 7fbda3e00640 1 -- 172.21.15.28:0/3387926058 shutdown_connections 2024-04-24T08:20:27.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.167+0000 7fbda3e00640 1 --2- 172.21.15.28:0/3387926058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbda4109330 0x7fbda4110f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.167+0000 7fbda3e00640 1 --2- 172.21.15.28:0/3387926058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbda4104c10 0x7fbda4108c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.167+0000 7fbda3e00640 1 --2- 172.21.15.28:0/3387926058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbda4104260 0x7fbda4104640 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.167+0000 7fbda3e00640 1 -- 172.21.15.28:0/3387926058 >> 172.21.15.28:0/3387926058 conn(0x7fbda40ffae0 msgr2=0x7fbda4101f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.168+0000 7fbda3e00640 1 -- 172.21.15.28:0/3387926058 shutdown_connections 2024-04-24T08:20:27.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.168+0000 7fbda3e00640 1 -- 172.21.15.28:0/3387926058 wait complete. 2024-04-24T08:20:27.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.169+0000 7fbda3e00640 1 Processor -- start 2024-04-24T08:20:27.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.169+0000 7fbda3e00640 1 -- start start 2024-04-24T08:20:27.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.169+0000 7fbda3e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbda4104260 0x7fbda41a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.169+0000 7fbda3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbda4109330 0x7fbda41a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.169+0000 7fbda3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbda419a7e0 0x7fbda419ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.169+0000 7fbda3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fbda41143e0 con 0x7fbda4109330 2024-04-24T08:20:27.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.169+0000 7fbda3e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fbda4114260 con 0x7fbda4104260 2024-04-24T08:20:27.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.169+0000 7fbda3e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fbda4114560 con 0x7fbda419a7e0 2024-04-24T08:20:27.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.169+0000 7fbda2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbda4109330 0x7fbda41a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.169+0000 7fbda2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbda4109330 0x7fbda41a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37202/0 (socket says 172.21.15.28:37202) 2024-04-24T08:20:27.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.170+0000 7fbda2000640 1 -- 172.21.15.28:0/775720194 learned_addr learned my addr 172.21.15.28:0/775720194 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:27.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.170+0000 7fbda2a00640 1 --2- 172.21.15.28:0/775720194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbda4104260 0x7fbda41a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.170+0000 7fbda3400640 1 --2- 172.21.15.28:0/775720194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbda419a7e0 0x7fbda419ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.170+0000 7fbda2000640 1 -- 172.21.15.28:0/775720194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbda419a7e0 msgr2=0x7fbda419ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.170+0000 7fbda2000640 1 --2- 172.21.15.28:0/775720194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbda419a7e0 0x7fbda419ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.170+0000 7fbda2000640 1 -- 172.21.15.28:0/775720194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbda4104260 msgr2=0x7fbda41a0620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.170+0000 7fbda2000640 1 --2- 172.21.15.28:0/775720194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbda4104260 0x7fbda41a0620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.170+0000 7fbda2000640 1 -- 172.21.15.28:0/775720194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbda419b370 con 0x7fbda4109330 2024-04-24T08:20:27.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.170+0000 7fbda2a00640 1 --2- 172.21.15.28:0/775720194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbda4104260 0x7fbda41a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:27.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.170+0000 7fbda2000640 1 --2- 172.21.15.28:0/775720194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbda4109330 0x7fbda41a0b60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbd9000e9f0 tx=0x7fbd9000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.171+0000 7fbd9b400640 1 -- 172.21.15.28:0/775720194 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd9000ccc0 con 0x7fbda4109330 2024-04-24T08:20:27.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.171+0000 7fbda3e00640 1 -- 172.21.15.28:0/775720194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbda419b660 con 0x7fbda4109330 2024-04-24T08:20:27.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.171+0000 7fbd9b400640 1 -- 172.21.15.28:0/775720194 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbd90005500 con 0x7fbda4109330 2024-04-24T08:20:27.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.171+0000 7fbd9b400640 1 -- 172.21.15.28:0/775720194 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbd90010690 con 0x7fbda4109330 2024-04-24T08:20:27.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.171+0000 7fbda3e00640 1 -- 172.21.15.28:0/775720194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbda41a77f0 con 0x7fbda4109330 2024-04-24T08:20:27.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.172+0000 7fbda3e00640 1 -- 172.21.15.28:0/775720194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbda4105730 con 0x7fbda4109330 2024-04-24T08:20:27.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.175+0000 7fbd9b400640 1 -- 172.21.15.28:0/775720194 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fbd900046c0 con 0x7fbda4109330 2024-04-24T08:20:27.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.176+0000 7fbd9b400640 1 --2- 172.21.15.28:0/775720194 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fbd880a3da0 0x7fbd880a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.176+0000 7fbd9b400640 1 -- 172.21.15.28:0/775720194 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8644+0+0 (secure 0 0 0) 0x7fbd900d0c70 con 0x7fbda4109330 2024-04-24T08:20:27.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.177+0000 7fbda2a00640 1 --2- 172.21.15.28:0/775720194 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fbd880a3da0 0x7fbd880a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.177+0000 7fbda2a00640 1 --2- 172.21.15.28:0/775720194 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fbd880a3da0 0x7fbd880a6260 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbd9402a930 tx=0x7fbd940056d0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.182 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.181+0000 7fbd9b400640 1 -- 172.21.15.28:0/775720194 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fbda419b660 con 0x7fbda4109330 2024-04-24T08:20:27.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.436+0000 7fbda3e00640 1 -- 172.21.15.28:0/775720194 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbda410c560 con 0x7fbda4109330 2024-04-24T08:20:27.438 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.437+0000 7fbd9b400640 1 -- 172.21.15.28:0/775720194 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11871 (secure 0 0 0) 0x7fbd9009d230 con 0x7fbda4109330 2024-04-24T08:20:27.438 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:20:27.438 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:14:38.663141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:20:27.439 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-24T08:20:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.440+0000 7fbda3e00640 1 -- 172.21.15.28:0/775720194 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fbd880a3da0 msgr2=0x7fbd880a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.440+0000 7fbda3e00640 1 --2- 172.21.15.28:0/775720194 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fbd880a3da0 0x7fbd880a6260 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fbd9402a930 tx=0x7fbd940056d0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.440+0000 7fbda3e00640 1 -- 172.21.15.28:0/775720194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbda4109330 msgr2=0x7fbda41a0b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.441 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.440+0000 7fbda3e00640 1 --2- 172.21.15.28:0/775720194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbda4109330 0x7fbda41a0b60 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7fbd9000e9f0 tx=0x7fbd9000eeb0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.440+0000 7fbda3e00640 1 -- 172.21.15.28:0/775720194 shutdown_connections 2024-04-24T08:20:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.440+0000 7fbda3e00640 1 --2- 172.21.15.28:0/775720194 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fbd880a3da0 0x7fbd880a6260 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.440+0000 7fbda3e00640 1 --2- 172.21.15.28:0/775720194 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fbda4109330 0x7fbda41a0b60 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.440+0000 7fbda3e00640 1 --2- 172.21.15.28:0/775720194 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fbda419a7e0 0x7fbda419ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.440+0000 7fbda3e00640 1 --2- 172.21.15.28:0/775720194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbda4104260 0x7fbda41a0620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.440+0000 7fbda3e00640 1 -- 172.21.15.28:0/775720194 >> 172.21.15.28:0/775720194 conn(0x7fbda40ffae0 msgr2=0x7fbda4100ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.441+0000 7fbda3e00640 1 -- 172.21.15.28:0/775720194 shutdown_connections 2024-04-24T08:20:27.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.441+0000 7fbda3e00640 1 -- 172.21.15.28:0/775720194 wait complete. 2024-04-24T08:20:27.462 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:20:27.686 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.686+0000 7fcd4cc00640 1 -- 172.21.15.28:0/3670243488 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd4806b220 msgr2=0x7fcd48106c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.686+0000 7fcd4cc00640 1 --2- 172.21.15.28:0/3670243488 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd4806b220 0x7fcd48106c50 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fcd34009900 tx=0x7fcd3402a410 comp rx=0 tx=0).stop 2024-04-24T08:20:27.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.686+0000 7fcd4cc00640 1 -- 172.21.15.28:0/3670243488 shutdown_connections 2024-04-24T08:20:27.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.686+0000 7fcd4cc00640 1 --2- 172.21.15.28:0/3670243488 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcd48107330 0x7fcd4810eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.686+0000 7fcd4cc00640 1 --2- 172.21.15.28:0/3670243488 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd4806b220 0x7fcd48106c50 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.686+0000 7fcd4cc00640 1 --2- 172.21.15.28:0/3670243488 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd4806a870 0x7fcd4806ac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.686+0000 7fcd4cc00640 1 -- 172.21.15.28:0/3670243488 >> 172.21.15.28:0/3670243488 conn(0x7fcd48076d00 msgr2=0x7fcd48077110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.686+0000 7fcd4cc00640 1 -- 172.21.15.28:0/3670243488 shutdown_connections 2024-04-24T08:20:27.687 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.686+0000 7fcd4cc00640 1 -- 172.21.15.28:0/3670243488 wait complete. 2024-04-24T08:20:27.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.687+0000 7fcd4cc00640 1 Processor -- start 2024-04-24T08:20:27.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.687+0000 7fcd4cc00640 1 -- start start 2024-04-24T08:20:27.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.687+0000 7fcd4cc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcd4806a870 0x7fcd481a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.687+0000 7fcd4cc00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd48107330 0x7fcd481a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.687+0000 7fcd4cc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd4819a8b0 0x7fcd4819ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.689 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.687+0000 7fcd4cc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcd481123f0 con 0x7fcd4819a8b0 2024-04-24T08:20:27.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.688+0000 7fcd4cc00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcd48112270 con 0x7fcd48107330 2024-04-24T08:20:27.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.688+0000 7fcd4cc00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcd48112570 con 0x7fcd4806a870 2024-04-24T08:20:27.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.688+0000 7fcd45e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcd4806a870 0x7fcd481a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.688+0000 7fcd45400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd48107330 0x7fcd481a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.688+0000 7fcd46800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd4819a8b0 0x7fcd4819ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.688+0000 7fcd45400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd48107330 0x7fcd481a0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:36326/0 (socket says 172.21.15.28:36326) 2024-04-24T08:20:27.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.688+0000 7fcd45400640 1 -- 172.21.15.28:0/3711730809 learned_addr learned my addr 172.21.15.28:0/3711730809 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:27.690 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.688+0000 7fcd45400640 1 -- 172.21.15.28:0/3711730809 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcd4806a870 msgr2=0x7fcd481a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.688+0000 7fcd45400640 1 --2- 172.21.15.28:0/3711730809 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcd4806a870 0x7fcd481a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.688+0000 7fcd45400640 1 -- 172.21.15.28:0/3711730809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd4819a8b0 msgr2=0x7fcd4819ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.688+0000 7fcd45400640 1 --2- 172.21.15.28:0/3711730809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd4819a8b0 0x7fcd4819ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.688+0000 7fcd45400640 1 -- 172.21.15.28:0/3711730809 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcd4819b3d0 con 0x7fcd48107330 2024-04-24T08:20:27.691 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.689+0000 7fcd45400640 1 --2- 172.21.15.28:0/3711730809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd48107330 0x7fcd481a0c40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fcd3402a930 tx=0x7fcd34003b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.689+0000 7fcd3ea00640 1 -- 172.21.15.28:0/3711730809 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd3403f070 con 0x7fcd48107330 2024-04-24T08:20:27.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.689+0000 7fcd4cc00640 1 -- 172.21.15.28:0/3711730809 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcd4819b660 con 0x7fcd48107330 2024-04-24T08:20:27.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.689+0000 7fcd3ea00640 1 -- 172.21.15.28:0/3711730809 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd34031c90 con 0x7fcd48107330 2024-04-24T08:20:27.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.689+0000 7fcd3ea00640 1 -- 172.21.15.28:0/3711730809 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd34035030 con 0x7fcd48107330 2024-04-24T08:20:27.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.689+0000 7fcd4cc00640 1 -- 172.21.15.28:0/3711730809 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcd481a78b0 con 0x7fcd48107330 2024-04-24T08:20:27.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.693+0000 7fcd4f963740 1 Processor -- start 2024-04-24T08:20:27.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.693+0000 7fcd4f963740 1 -- start start 2024-04-24T08:20:27.694 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.693+0000 7fcd3ea00640 1 -- 172.21.15.28:0/3711730809 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fcd34042070 con 0x7fcd48107330 2024-04-24T08:20:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.693+0000 7fcd4f963740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a720f950 0x5628a720fd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.693+0000 7fcd4f963740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a7210270 0x5628a721a400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.693+0000 7fcd4f963740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a721a940 0x5628a721cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.693+0000 7fcd4f963740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5628a71751d0 con 0x5628a720f950 2024-04-24T08:20:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.693+0000 7fcd4f963740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5628a711b450 con 0x5628a721a940 2024-04-24T08:20:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.693+0000 7fcd4f963740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5628a7171190 con 0x5628a7210270 2024-04-24T08:20:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.693+0000 7fcd46800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a720f950 0x5628a720fd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.693+0000 7fcd46800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a720f950 0x5628a720fd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37234/0 (socket says 172.21.15.28:37234) 2024-04-24T08:20:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.693+0000 7fcd46800640 1 -- 172.21.15.28:0/2354248248 learned_addr learned my addr 172.21.15.28:0/2354248248 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.693+0000 7fcd45e00640 1 --2- 172.21.15.28:0/2354248248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a7210270 0x5628a721a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.694+0000 7fcd45400640 1 --2- 172.21.15.28:0/2354248248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a721a940 0x5628a721cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.694+0000 7fcd46800640 1 -- 172.21.15.28:0/2354248248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a7210270 msgr2=0x5628a721a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.694+0000 7fcd46800640 1 --2- 172.21.15.28:0/2354248248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a7210270 0x5628a721a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.694+0000 7fcd46800640 1 -- 172.21.15.28:0/2354248248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a721a940 msgr2=0x5628a721cd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.694+0000 7fcd46800640 1 --2- 172.21.15.28:0/2354248248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a721a940 0x5628a721cd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.694+0000 7fcd46800640 1 -- 172.21.15.28:0/2354248248 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628a71d8310 con 0x5628a720f950 2024-04-24T08:20:27.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.694+0000 7fcd46800640 1 --2- 172.21.15.28:0/2354248248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a720f950 0x5628a720fd30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009fd0 tx=0x7fcd400118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83b592587c61e9ba server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.695+0000 7fcd33000640 1 -- 172.21.15.28:0/2354248248 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd40004d20 con 0x5628a720f950 2024-04-24T08:20:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.695+0000 7fcd33000640 1 -- 172.21.15.28:0/2354248248 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd40004050 con 0x5628a720f950 2024-04-24T08:20:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.695+0000 7fcd33000640 1 -- 172.21.15.28:0/2354248248 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd40005260 con 0x5628a720f950 2024-04-24T08:20:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.695+0000 7fcd3ea00640 1 --2- 172.21.15.28:0/3711730809 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fcd2c0a3d20 0x7fcd2c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.695+0000 7fcd3ea00640 1 -- 172.21.15.28:0/3711730809 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8644+0+0 (secure 0 0 0) 0x7fcd340f1200 con 0x7fcd48107330 2024-04-24T08:20:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.695+0000 7fcd45e00640 1 --2- 172.21.15.28:0/3711730809 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fcd2c0a3d20 0x7fcd2c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.695+0000 7fcd4f963740 1 -- 172.21.15.28:0/2354248248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a720f950 msgr2=0x5628a720fd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.695+0000 7fcd4f963740 1 --2- 172.21.15.28:0/2354248248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a720f950 0x5628a720fd30 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009fd0 tx=0x7fcd400118d0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.696+0000 7fcd4f963740 1 -- 172.21.15.28:0/2354248248 shutdown_connections 2024-04-24T08:20:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.696+0000 7fcd4f963740 1 --2- 172.21.15.28:0/2354248248 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a721a940 0x5628a721cd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.696+0000 7fcd4f963740 1 --2- 172.21.15.28:0/2354248248 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a7210270 0x5628a721a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.696+0000 7fcd4f963740 1 --2- 172.21.15.28:0/2354248248 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a720f950 0x5628a720fd30 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.696+0000 7fcd4f963740 1 -- 172.21.15.28:0/2354248248 >> 172.21.15.28:0/2354248248 conn(0x5628a720bc70 msgr2=0x5628a720c050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.696+0000 7fcd4f963740 1 -- 172.21.15.28:0/2354248248 shutdown_connections 2024-04-24T08:20:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.696+0000 7fcd4f963740 1 -- 172.21.15.28:0/2354248248 wait complete. 2024-04-24T08:20:27.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.696+0000 7fcd45e00640 1 --2- 172.21.15.28:0/3711730809 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fcd2c0a3d20 0x7fcd2c0a61e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x5628a7210f10 tx=0x7fcd38005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.696+0000 7fcd4f963740 1 -- start start 2024-04-24T08:20:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.696+0000 7fcd4f963740 1 Processor -- start 2024-04-24T08:20:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.697+0000 7fcd4f963740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a720f950 0x5628a721a3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.697+0000 7fcd4f963740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a7210270 0x5628a72176a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.697+0000 7fcd4f963740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a721a940 0x5628a7217de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.697+0000 7fcd4f963740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5628a721d7d0 con 0x5628a7210270 2024-04-24T08:20:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.697+0000 7fcd4f963740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5628a712e810 con 0x5628a721a940 2024-04-24T08:20:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.697+0000 7fcd4f963740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5628a721d950 con 0x5628a720f950 2024-04-24T08:20:27.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.697+0000 7fcd4f963740 10 client.0 _ll_register_callbacks cb 0x5628a70fc140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.697+0000 7fcd45400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a7210270 0x5628a72176a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.697+0000 7fcd45e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a720f950 0x5628a721a3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.697+0000 7fcd45400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a7210270 0x5628a72176a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37244/0 (socket says 172.21.15.28:37244) 2024-04-24T08:20:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.697+0000 7fcd45e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a720f950 0x5628a721a3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:54332/0 (socket says 172.21.15.28:54332) 2024-04-24T08:20:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.697+0000 7fcd45400640 1 -- 172.21.15.28:0/1434294100 learned_addr learned my addr 172.21.15.28:0/1434294100 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.697+0000 7fcd46800640 1 --2- 172.21.15.28:0/1434294100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a721a940 0x5628a7217de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.698+0000 7fcd45400640 1 -- 172.21.15.28:0/1434294100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a720f950 msgr2=0x5628a721a3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.698+0000 7fcd45400640 1 --2- 172.21.15.28:0/1434294100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a720f950 0x5628a721a3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.703 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.698+0000 7fcd45400640 1 -- 172.21.15.28:0/1434294100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a721a940 msgr2=0x5628a7217de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.698+0000 7fcd45400640 1 --2- 172.21.15.28:0/1434294100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a721a940 0x5628a7217de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.698+0000 7fcd45400640 1 -- 172.21.15.28:0/1434294100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628a713f3f0 con 0x5628a7210270 2024-04-24T08:20:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.698+0000 7fcd45400640 1 --2- 172.21.15.28:0/1434294100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a7210270 0x5628a72176a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcd340b95d0 tx=0x7fcd34005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fab35a163ef5614c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.698+0000 7fcd27400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:20:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.698+0000 7fcd27400640 1 -- 172.21.15.28:0/1434294100 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd34049050 con 0x5628a7210270 2024-04-24T08:20:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.698+0000 7fcd27400640 1 -- 172.21.15.28:0/1434294100 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8644+0+0 (secure 0 0 0) 0x7fcd340eeaf0 con 0x5628a7210270 2024-04-24T08:20:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.698+0000 7fcd4f963740 1 -- 172.21.15.28:0/1434294100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd40004050 con 0x5628a7210270 2024-04-24T08:20:27.704 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.699+0000 7fcd27400640 1 -- 172.21.15.28:0/1434294100 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcd340bccb0 con 0x5628a7210270 2024-04-24T08:20:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.699+0000 7fcd4f963740 10 client.16713 fetch_fsmap learned FSMap version 26 2024-04-24T08:20:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.699+0000 7fcd4f963740 1 -- 172.21.15.28:0/1434294100 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5628a6e873f0 con 0x5628a7210270 2024-04-24T08:20:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.699+0000 7fcd27400640 1 -- 172.21.15.28:0/1434294100 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd40004050 con 0x5628a7210270 2024-04-24T08:20:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.699+0000 7fcd27400640 1 -- 172.21.15.28:0/1434294100 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fcd340b7ce0 con 0x5628a7210270 2024-04-24T08:20:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.699+0000 7fcd4f963740 10 client.16713 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:20:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.699+0000 7fcd4f963740 10 client.16713 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:20:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.699+0000 7fcd4f963740 20 client.16713 populate_metadata read hostname 'smithi028' 2024-04-24T08:20:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.699+0000 7fcd4f963740 1 --2- 172.21.15.28:0/1434294100 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5628a722ab90 0x5628a722cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.705 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.699+0000 7fcd4f963740 4 client.16713 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:20:27.706 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.699+0000 7fcd4f963740 1 -- 172.21.15.28:0/1434294100 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628a7182bf0 con 0x5628a722ab90 2024-04-24T08:20:27.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.700+0000 7fcd45e00640 1 --2- 172.21.15.28:0/1434294100 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5628a722ab90 0x5628a722cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.700+0000 7fcd45e00640 1 --2- 172.21.15.28:0/1434294100 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5628a722ab90 0x5628a722cfc0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=daeaa298590a7aa0 server_cookie=3416b91caf4bd077 in_seq=0 out_seq=0 2024-04-24T08:20:27.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.700+0000 7fcd27400640 10 client.16713 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:20:27.707 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.701+0000 7fcd27400640 1 -- 172.21.15.28:0/1434294100 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5628a7182bf0 con 0x5628a722ab90 2024-04-24T08:20:27.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.701+0000 7fcd27400640 10 client.16713 handle_command_reply: tid=0 2024-04-24T08:20:27.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.701+0000 7fcd4f963740 1 client.16713 shutdown 2024-04-24T08:20:27.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.702+0000 7fcd4f963740 1 -- 172.21.15.28:0/1434294100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a7210270 msgr2=0x5628a72176a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.702+0000 7fcd4f963740 1 --2- 172.21.15.28:0/1434294100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a7210270 0x5628a72176a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7fcd340b95d0 tx=0x7fcd34005c00 comp rx=0 tx=0).stop 2024-04-24T08:20:27.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.702+0000 7fcd4f963740 1 -- 172.21.15.28:0/1434294100 shutdown_connections 2024-04-24T08:20:27.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.702+0000 7fcd4f963740 1 --2- 172.21.15.28:0/1434294100 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5628a722ab90 0x5628a722cfc0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.702+0000 7fcd4f963740 1 --2- 172.21.15.28:0/1434294100 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a7210270 0x5628a72176a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.708 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.702+0000 7fcd4f963740 1 --2- 172.21.15.28:0/1434294100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a721a940 0x5628a7217de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.702+0000 7fcd4f963740 1 --2- 172.21.15.28:0/1434294100 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a720f950 0x5628a721a3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.702+0000 7fcd4f963740 1 -- 172.21.15.28:0/1434294100 >> 172.21.15.28:0/1434294100 conn(0x5628a720bc70 msgr2=0x5628a7211a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.702+0000 7fcd27400640 0 client.16713 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:20:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.702+0000 7fcd4f963740 1 -- 172.21.15.28:0/1434294100 shutdown_connections 2024-04-24T08:20:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.702+0000 7fcd4f963740 1 -- 172.21.15.28:0/1434294100 wait complete. 2024-04-24T08:20:27.709 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.702+0000 7fcd4f963740 20 client.16713 trim_cache size 0 max 16384 2024-04-24T08:20:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.717+0000 7fcd4f963740 1 Processor -- start 2024-04-24T08:20:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.717+0000 7fcd4f963740 1 -- start start 2024-04-24T08:20:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.717+0000 7fcd4f963740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a720f950 0x5628a7226690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.717+0000 7fcd4f963740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a7210270 0x5628a7216800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.717+0000 7fcd4f963740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a721a940 0x5628a7219ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.717+0000 7fcd4f963740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcd481b29f0 con 0x5628a721a940 2024-04-24T08:20:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.717+0000 7fcd4f963740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcd481b02c0 con 0x5628a720f950 2024-04-24T08:20:27.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.717+0000 7fcd4f963740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcd481ae130 con 0x5628a7210270 2024-04-24T08:20:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.718+0000 7fcd46800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a721a940 0x5628a7219ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.718+0000 7fcd45e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a720f950 0x5628a7226690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.718+0000 7fcd45e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a720f950 0x5628a7226690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:36342/0 (socket says 172.21.15.28:36342) 2024-04-24T08:20:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.718+0000 7fcd45e00640 1 -- 172.21.15.28:0/566287042 learned_addr learned my addr 172.21.15.28:0/566287042 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.718+0000 7fcd45400640 1 --2- 172.21.15.28:0/566287042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a7210270 0x5628a7216800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.718+0000 7fcd45e00640 1 -- 172.21.15.28:0/566287042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a7210270 msgr2=0x5628a7216800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.718+0000 7fcd45e00640 1 --2- 172.21.15.28:0/566287042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a7210270 0x5628a7216800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.718+0000 7fcd45e00640 1 -- 172.21.15.28:0/566287042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a721a940 msgr2=0x5628a7219ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.718+0000 7fcd45e00640 1 --2- 172.21.15.28:0/566287042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a721a940 0x5628a7219ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.718+0000 7fcd45e00640 1 -- 172.21.15.28:0/566287042 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628a6f517d0 con 0x5628a720f950 2024-04-24T08:20:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.718+0000 7fcd45400640 1 --2- 172.21.15.28:0/566287042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a7210270 0x5628a7216800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.719+0000 7fcd45e00640 1 --2- 172.21.15.28:0/566287042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a720f950 0x5628a7226690 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fcd38004150 tx=0x7fcd3801b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f6936c94b744acd server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.719+0000 7fcd32600640 1 -- 172.21.15.28:0/566287042 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd380043c0 con 0x5628a720f950 2024-04-24T08:20:27.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.719+0000 7fcd32600640 1 -- 172.21.15.28:0/566287042 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd38005020 con 0x5628a720f950 2024-04-24T08:20:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.719+0000 7fcd32600640 1 -- 172.21.15.28:0/566287042 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd38004900 con 0x5628a720f950 2024-04-24T08:20:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.719+0000 7fcd4f963740 1 -- 172.21.15.28:0/566287042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a720f950 msgr2=0x5628a7226690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.722 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.720+0000 7fcd4f963740 1 --2- 172.21.15.28:0/566287042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a720f950 0x5628a7226690 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fcd38004150 tx=0x7fcd3801b000 comp rx=0 tx=0).stop 2024-04-24T08:20:27.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.720+0000 7fcd4f963740 1 -- 172.21.15.28:0/566287042 shutdown_connections 2024-04-24T08:20:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.720+0000 7fcd4f963740 1 --2- 172.21.15.28:0/566287042 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a721a940 0x5628a7219ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.720+0000 7fcd4f963740 1 --2- 172.21.15.28:0/566287042 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a7210270 0x5628a7216800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.720+0000 7fcd4f963740 1 --2- 172.21.15.28:0/566287042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a720f950 0x5628a7226690 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.720+0000 7fcd4f963740 1 -- 172.21.15.28:0/566287042 >> 172.21.15.28:0/566287042 conn(0x5628a720bc70 msgr2=0x5628a720a560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.720+0000 7fcd4f963740 1 -- 172.21.15.28:0/566287042 shutdown_connections 2024-04-24T08:20:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.720+0000 7fcd4f963740 1 -- 172.21.15.28:0/566287042 wait complete. 2024-04-24T08:20:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.720+0000 7fcd4f963740 1 -- start start 2024-04-24T08:20:27.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.720+0000 7fcd4f963740 1 Processor -- start 2024-04-24T08:20:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.721+0000 7fcd4f963740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a720f950 0x5628a721b860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.721+0000 7fcd4f963740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a7210270 0x5628a721bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.721+0000 7fcd4f963740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a721a940 0x5628a721c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.721+0000 7fcd4f963740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcd481ae780 con 0x5628a7210270 2024-04-24T08:20:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.721+0000 7fcd4f963740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcd481afbf0 con 0x5628a721a940 2024-04-24T08:20:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.721+0000 7fcd4f963740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcd481aee50 con 0x5628a720f950 2024-04-24T08:20:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.721+0000 7fcd4f963740 10 client.0 _ll_register_callbacks cb 0x7fcd340f49a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.721+0000 7fcd45400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a7210270 0x5628a721bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.721+0000 7fcd45e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a720f950 0x5628a721b860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.721+0000 7fcd45e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a720f950 0x5628a721b860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:54356/0 (socket says 172.21.15.28:54356) 2024-04-24T08:20:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.721+0000 7fcd45e00640 1 -- 172.21.15.28:0/752365177 learned_addr learned my addr 172.21.15.28:0/752365177 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.721+0000 7fcd45400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a7210270 0x5628a721bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37252/0 (socket says 172.21.15.28:37252) 2024-04-24T08:20:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.721+0000 7fcd46800640 1 --2- 172.21.15.28:0/752365177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a721a940 0x5628a721c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.722+0000 7fcd45400640 1 -- 172.21.15.28:0/752365177 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a720f950 msgr2=0x5628a721b860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.722+0000 7fcd45400640 1 --2- 172.21.15.28:0/752365177 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a720f950 0x5628a721b860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.722+0000 7fcd45400640 1 -- 172.21.15.28:0/752365177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a721a940 msgr2=0x5628a721c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.722+0000 7fcd45400640 1 --2- 172.21.15.28:0/752365177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a721a940 0x5628a721c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.722+0000 7fcd45400640 1 -- 172.21.15.28:0/752365177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628a7220f60 con 0x5628a7210270 2024-04-24T08:20:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.722+0000 7fcd46800640 1 --2- 172.21.15.28:0/752365177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a721a940 0x5628a721c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:20:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.722+0000 7fcd45e00640 1 --2- 172.21.15.28:0/752365177 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a720f950 0x5628a721b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.722+0000 7fcd45400640 1 --2- 172.21.15.28:0/752365177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a7210270 0x5628a721bda0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fcd34000c00 tx=0x7fcd340f4d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daea108d0e2460cc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.722+0000 7fcd27400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:20:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.722+0000 7fcd27400640 1 -- 172.21.15.28:0/752365177 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcd34049070 con 0x5628a7210270 2024-04-24T08:20:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.722+0000 7fcd27400640 1 -- 172.21.15.28:0/752365177 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8644+0+0 (secure 0 0 0) 0x7fcd340c58a0 con 0x5628a7210270 2024-04-24T08:20:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.722+0000 7fcd4f963740 1 -- 172.21.15.28:0/752365177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcd38005020 con 0x5628a7210270 2024-04-24T08:20:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.723+0000 7fcd27400640 1 -- 172.21.15.28:0/752365177 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fcd340db050 con 0x5628a7210270 2024-04-24T08:20:27.729 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.723+0000 7fcd4f963740 10 client.16719 fetch_fsmap learned FSMap version 26 2024-04-24T08:20:27.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.723+0000 7fcd4f963740 1 -- 172.21.15.28:0/752365177 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5628a7291510 con 0x5628a7210270 2024-04-24T08:20:27.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.723+0000 7fcd27400640 1 -- 172.21.15.28:0/752365177 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcd38005020 con 0x5628a7210270 2024-04-24T08:20:27.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.723+0000 7fcd27400640 1 -- 172.21.15.28:0/752365177 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fcd340b7ce0 con 0x5628a7210270 2024-04-24T08:20:27.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.723+0000 7fcd4f963740 10 client.16719 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:20:27.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.723+0000 7fcd4f963740 10 client.16719 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:20:27.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.723+0000 7fcd4f963740 20 client.16719 populate_metadata read hostname 'smithi028' 2024-04-24T08:20:27.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.723+0000 7fcd4f963740 1 --2- 172.21.15.28:0/752365177 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5628a7291b00 0x5628a7293f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.730 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.723+0000 7fcd4f963740 4 client.16719 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:20:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.723+0000 7fcd4f963740 1 -- 172.21.15.28:0/752365177 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5628a717ca60 con 0x5628a7291b00 2024-04-24T08:20:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.724+0000 7fcd45e00640 1 --2- 172.21.15.28:0/752365177 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5628a7291b00 0x5628a7293f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.724+0000 7fcd45e00640 1 --2- 172.21.15.28:0/752365177 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5628a7291b00 0x5628a7293f20 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=c4983b9a3a8462a9 server_cookie=771505a7adfa5cf5 in_seq=0 out_seq=0 2024-04-24T08:20:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.724+0000 7fcd27400640 10 client.16719 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:20:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.724+0000 7fcd27400640 1 -- 172.21.15.28:0/752365177 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5628a717ca60 con 0x5628a7291b00 2024-04-24T08:20:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.724+0000 7fcd27400640 10 client.16719 handle_command_reply: tid=0 2024-04-24T08:20:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.724+0000 7fcd4f963740 1 client.16719 shutdown 2024-04-24T08:20:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.725+0000 7fcd4f963740 1 -- 172.21.15.28:0/752365177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a7210270 msgr2=0x5628a721bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.731 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.725+0000 7fcd4f963740 1 --2- 172.21.15.28:0/752365177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a7210270 0x5628a721bda0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fcd34000c00 tx=0x7fcd340f4d20 comp rx=0 tx=0).stop 2024-04-24T08:20:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.725+0000 7fcd4f963740 1 -- 172.21.15.28:0/752365177 shutdown_connections 2024-04-24T08:20:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.725+0000 7fcd4f963740 1 --2- 172.21.15.28:0/752365177 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x5628a7291b00 0x5628a7293f20 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.725+0000 7fcd4f963740 1 --2- 172.21.15.28:0/752365177 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5628a7210270 0x5628a721bda0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.725+0000 7fcd4f963740 1 --2- 172.21.15.28:0/752365177 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5628a721a940 0x5628a721c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.725+0000 7fcd4f963740 1 --2- 172.21.15.28:0/752365177 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5628a720f950 0x5628a721b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.725+0000 7fcd4f963740 1 -- 172.21.15.28:0/752365177 >> 172.21.15.28:0/752365177 conn(0x5628a720bc70 msgr2=0x5628a7216680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.725+0000 7fcd27400640 0 client.16719 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:20:27.732 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.725+0000 7fcd4f963740 1 -- 172.21.15.28:0/752365177 shutdown_connections 2024-04-24T08:20:27.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.725+0000 7fcd4f963740 1 -- 172.21.15.28:0/752365177 wait complete. 2024-04-24T08:20:27.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.725+0000 7fcd4f963740 20 client.16719 trim_cache size 0 max 16384 2024-04-24T08:20:27.733 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:20:27.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.726+0000 7fcd4cc00640 1 -- 172.21.15.28:0/3711730809 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fcd2c0a3d20 msgr2=0x7fcd2c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.726+0000 7fcd4cc00640 1 --2- 172.21.15.28:0/3711730809 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fcd2c0a3d20 0x7fcd2c0a61e0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x5628a7210f10 tx=0x7fcd38005f90 comp rx=0 tx=0).stop 2024-04-24T08:20:27.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.726+0000 7fcd4cc00640 1 -- 172.21.15.28:0/3711730809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd48107330 msgr2=0x7fcd481a0c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.726+0000 7fcd4cc00640 1 --2- 172.21.15.28:0/3711730809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd48107330 0x7fcd481a0c40 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fcd3402a930 tx=0x7fcd34003b20 comp rx=0 tx=0).stop 2024-04-24T08:20:27.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.726+0000 7fcd4cc00640 1 -- 172.21.15.28:0/3711730809 shutdown_connections 2024-04-24T08:20:27.734 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.726+0000 7fcd4cc00640 1 --2- 172.21.15.28:0/3711730809 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fcd2c0a3d20 0x7fcd2c0a61e0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.726+0000 7fcd4cc00640 1 --2- 172.21.15.28:0/3711730809 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcd4819a8b0 0x7fcd4819ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.726+0000 7fcd4cc00640 1 --2- 172.21.15.28:0/3711730809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd48107330 0x7fcd481a0c40 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.726+0000 7fcd4cc00640 1 --2- 172.21.15.28:0/3711730809 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcd4806a870 0x7fcd481a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.726+0000 7fcd4cc00640 1 -- 172.21.15.28:0/3711730809 >> 172.21.15.28:0/3711730809 conn(0x7fcd48076d00 msgr2=0x7fcd4810d3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.726+0000 7fcd4cc00640 1 -- 172.21.15.28:0/3711730809 shutdown_connections 2024-04-24T08:20:27.735 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.726+0000 7fcd4cc00640 1 -- 172.21.15.28:0/3711730809 wait complete. 2024-04-24T08:20:27.744 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:20:27.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.948+0000 7fa8c7200640 1 -- 172.21.15.28:0/1100997029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8c8076e50 msgr2=0x7fa8c8112800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.948+0000 7fa8c7200640 1 --2- 172.21.15.28:0/1100997029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8c8076e50 0x7fa8c8112800 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0009890 tx=0x7fa8b002a370 comp rx=0 tx=0).stop 2024-04-24T08:20:27.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.948+0000 7fa8c7200640 1 -- 172.21.15.28:0/1100997029 shutdown_connections 2024-04-24T08:20:27.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.948+0000 7fa8c7200640 1 --2- 172.21.15.28:0/1100997029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8c8112d40 0x7fa8c8115130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.948+0000 7fa8c7200640 1 --2- 172.21.15.28:0/1100997029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8c8076e50 0x7fa8c8112800 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.948+0000 7fa8c7200640 1 --2- 172.21.15.28:0/1100997029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8c8076530 0x7fa8c8076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.949 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.948+0000 7fa8c7200640 1 -- 172.21.15.28:0/1100997029 >> 172.21.15.28:0/1100997029 conn(0x7fa8c80ffb10 msgr2=0x7fa8c8101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.948+0000 7fa8c7200640 1 -- 172.21.15.28:0/1100997029 shutdown_connections 2024-04-24T08:20:27.950 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.948+0000 7fa8c7200640 1 -- 172.21.15.28:0/1100997029 wait complete. 2024-04-24T08:20:27.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.949+0000 7fa8c7200640 1 Processor -- start 2024-04-24T08:20:27.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.949+0000 7fa8c7200640 1 -- start start 2024-04-24T08:20:27.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.950+0000 7fa8c7200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8c8076530 0x7fa8c819edf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.950+0000 7fa8c7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8c8076e50 0x7fa8c819f330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.950+0000 7fa8c7200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8c8112d40 0x7fa8c81a36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.950+0000 7fa8c7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c81188b0 con 0x7fa8c8076e50 2024-04-24T08:20:27.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.950+0000 7fa8c7200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c8118730 con 0x7fa8c8076530 2024-04-24T08:20:27.952 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.950+0000 7fa8c7200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c8118a30 con 0x7fa8c8112d40 2024-04-24T08:20:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.950+0000 7fa8bd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8c8076e50 0x7fa8c819f330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.950+0000 7fa8bd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8c8076e50 0x7fa8c819f330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37272/0 (socket says 172.21.15.28:37272) 2024-04-24T08:20:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.950+0000 7fa8c5e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8c8076530 0x7fa8c819edf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.950+0000 7fa8bd400640 1 -- 172.21.15.28:0/3746136671 learned_addr learned my addr 172.21.15.28:0/3746136671 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.950+0000 7fa8c6800640 1 --2- 172.21.15.28:0/3746136671 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8c8112d40 0x7fa8c81a36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.950+0000 7fa8bd400640 1 -- 172.21.15.28:0/3746136671 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8c8112d40 msgr2=0x7fa8c81a36b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.950+0000 7fa8bd400640 1 --2- 172.21.15.28:0/3746136671 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8c8112d40 0x7fa8c81a36b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.950+0000 7fa8bd400640 1 -- 172.21.15.28:0/3746136671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8c8076530 msgr2=0x7fa8c819edf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.953 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.950+0000 7fa8bd400640 1 --2- 172.21.15.28:0/3746136671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8c8076530 0x7fa8c819edf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.950+0000 7fa8bd400640 1 -- 172.21.15.28:0/3746136671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa8c81a3de0 con 0x7fa8c8076e50 2024-04-24T08:20:27.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.950+0000 7fa8c5e00640 1 --2- 172.21.15.28:0/3746136671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8c8076530 0x7fa8c819edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:27.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.951+0000 7fa8bd400640 1 --2- 172.21.15.28:0/3746136671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8c8076e50 0x7fa8c819f330 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0030510 tx=0x7fa8b0033ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.951+0000 7fa8bf400640 1 -- 172.21.15.28:0/3746136671 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b0037040 con 0x7fa8c8076e50 2024-04-24T08:20:27.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.951+0000 7fa8c7200640 1 -- 172.21.15.28:0/3746136671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa8c81a4070 con 0x7fa8c8076e50 2024-04-24T08:20:27.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.951+0000 7fa8bf400640 1 -- 172.21.15.28:0/3746136671 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8b0040070 con 0x7fa8c8076e50 2024-04-24T08:20:27.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.951+0000 7fa8bf400640 1 -- 172.21.15.28:0/3746136671 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b00038c0 con 0x7fa8c8076e50 2024-04-24T08:20:27.954 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.951+0000 7fa8c7200640 1 -- 172.21.15.28:0/3746136671 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa8c81ab960 con 0x7fa8c8076e50 2024-04-24T08:20:27.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.954+0000 7fa8cf84b740 1 Processor -- start 2024-04-24T08:20:27.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.955+0000 7fa8bf400640 1 -- 172.21.15.28:0/3746136671 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa8b0004050 con 0x7fa8c8076e50 2024-04-24T08:20:27.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.955+0000 7fa8cf84b740 1 -- start start 2024-04-24T08:20:27.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.955+0000 7fa8cf84b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638d9b40 0x55d3638d9f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.955+0000 7fa8cf84b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638da460 0x55d3638e45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.955+0000 7fa8cf84b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638e4b30 0x55d3638e6f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.955+0000 7fa8cf84b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d36383f1d0 con 0x55d3638da460 2024-04-24T08:20:27.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.955+0000 7fa8cf84b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d3637e5450 con 0x55d3638d9b40 2024-04-24T08:20:27.956 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.955+0000 7fa8cf84b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d36383b190 con 0x55d3638e4b30 2024-04-24T08:20:27.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.955+0000 7fa8c5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638da460 0x55d3638e45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.955+0000 7fa8bd400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638e4b30 0x55d3638e6f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.955+0000 7fa8c5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638da460 0x55d3638e45f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37278/0 (socket says 172.21.15.28:37278) 2024-04-24T08:20:27.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.955+0000 7fa8c6800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638d9b40 0x55d3638d9f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.957 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.955+0000 7fa8c5e00640 1 -- 172.21.15.28:0/2772016673 learned_addr learned my addr 172.21.15.28:0/2772016673 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:27.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.956+0000 7fa8c5e00640 1 -- 172.21.15.28:0/2772016673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638e4b30 msgr2=0x55d3638e6f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.956+0000 7fa8c5e00640 1 --2- 172.21.15.28:0/2772016673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638e4b30 0x55d3638e6f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.956+0000 7fa8c5e00640 1 -- 172.21.15.28:0/2772016673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638d9b40 msgr2=0x55d3638d9f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.956+0000 7fa8c5e00640 1 --2- 172.21.15.28:0/2772016673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638d9b40 0x55d3638d9f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.959 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.956+0000 7fa8c5e00640 1 -- 172.21.15.28:0/2772016673 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3638a2310 con 0x55d3638da460 2024-04-24T08:20:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.956+0000 7fa8c6800640 1 --2- 172.21.15.28:0/2772016673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638d9b40 0x55d3638d9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.956+0000 7fa8c5e00640 1 --2- 172.21.15.28:0/2772016673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638da460 0x55d3638e45f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8011860 tx=0x7fa8b8011d20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dc34f37ebffe48d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.957+0000 7fa8bf400640 1 --2- 172.21.15.28:0/3746136671 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa8ac0a3d20 0x7fa8ac0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.957+0000 7fa8b7000640 1 -- 172.21.15.28:0/2772016673 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b80112f0 con 0x55d3638da460 2024-04-24T08:20:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.957+0000 7fa8b7000640 1 -- 172.21.15.28:0/2772016673 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8b8003e50 con 0x55d3638da460 2024-04-24T08:20:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.957+0000 7fa8b7000640 1 -- 172.21.15.28:0/2772016673 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b8004590 con 0x55d3638da460 2024-04-24T08:20:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.957+0000 7fa8bf400640 1 -- 172.21.15.28:0/3746136671 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8644+0+0 (secure 0 0 0) 0x7fa8b00f0740 con 0x7fa8c8076e50 2024-04-24T08:20:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.957+0000 7fa8c6800640 1 --2- 172.21.15.28:0/3746136671 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa8ac0a3d20 0x7fa8ac0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.960 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.958+0000 7fa8cf84b740 1 -- 172.21.15.28:0/2772016673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638da460 msgr2=0x55d3638e45f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.958+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/2772016673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638da460 0x55d3638e45f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8011860 tx=0x7fa8b8011d20 comp rx=0 tx=0).stop 2024-04-24T08:20:27.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.958+0000 7fa8cf84b740 1 -- 172.21.15.28:0/2772016673 shutdown_connections 2024-04-24T08:20:27.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.958+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/2772016673 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638da460 0x55d3638e45f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.958+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/2772016673 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638e4b30 0x55d3638e6f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.958+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/2772016673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638d9b40 0x55d3638d9f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.958+0000 7fa8cf84b740 1 -- 172.21.15.28:0/2772016673 >> 172.21.15.28:0/2772016673 conn(0x55d3638d5e60 msgr2=0x55d3638d6240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.958+0000 7fa8c6800640 1 --2- 172.21.15.28:0/3746136671 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa8ac0a3d20 0x7fa8ac0a61e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa8c806b440 tx=0x7fa8c0009260 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.958+0000 7fa8cf84b740 1 -- 172.21.15.28:0/2772016673 shutdown_connections 2024-04-24T08:20:27.961 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.958+0000 7fa8cf84b740 1 -- 172.21.15.28:0/2772016673 wait complete. 2024-04-24T08:20:27.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.958+0000 7fa8cf84b740 1 -- start start 2024-04-24T08:20:27.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.959+0000 7fa8cf84b740 1 Processor -- start 2024-04-24T08:20:27.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.959+0000 7fa8cf84b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638d9b40 0x55d3638e4590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.959+0000 7fa8cf84b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638da460 0x55d3638e1880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.959+0000 7fa8cf84b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638e4b30 0x55d3638e1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.965 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.959+0000 7fa8cf84b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55d3638e79c0 con 0x55d3638d9b40 2024-04-24T08:20:27.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.959+0000 7fa8cf84b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55d3637f8810 con 0x55d3638e4b30 2024-04-24T08:20:27.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.959+0000 7fa8cf84b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55d3638e7b40 con 0x55d3638da460 2024-04-24T08:20:27.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.959+0000 7fa8cf84b740 10 client.0 _ll_register_callbacks cb 0x55d3637c6140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:27.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.959+0000 7fa8c5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638d9b40 0x55d3638e4590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.959+0000 7fa8bd400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638da460 0x55d3638e1880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.959+0000 7fa8c5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638d9b40 0x55d3638e4590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37290/0 (socket says 172.21.15.28:37290) 2024-04-24T08:20:27.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.959+0000 7fa8c5e00640 1 -- 172.21.15.28:0/2239679010 learned_addr learned my addr 172.21.15.28:0/2239679010 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:27.966 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.959+0000 7fa8c6800640 1 --2- 172.21.15.28:0/2239679010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638e4b30 0x55d3638e1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.960+0000 7fa8bd400640 1 -- 172.21.15.28:0/2239679010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638e4b30 msgr2=0x55d3638e1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.960+0000 7fa8bd400640 1 --2- 172.21.15.28:0/2239679010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638e4b30 0x55d3638e1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.960+0000 7fa8bd400640 1 -- 172.21.15.28:0/2239679010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638d9b40 msgr2=0x55d3638e4590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.960+0000 7fa8bd400640 1 --2- 172.21.15.28:0/2239679010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638d9b40 0x55d3638e4590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.960+0000 7fa8bd400640 1 -- 172.21.15.28:0/2239679010 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d3638093f0 con 0x55d3638da460 2024-04-24T08:20:27.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.960+0000 7fa8c6800640 1 --2- 172.21.15.28:0/2239679010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638e4b30 0x55d3638e1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:20:27.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.960+0000 7fa8bd400640 1 --2- 172.21.15.28:0/2239679010 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638da460 0x55d3638e1880 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa8b00edec0 tx=0x7fa8b0030af0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c656245d6985b776 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.960+0000 7fa8ab400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:20:27.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.960+0000 7fa8ab400640 1 -- 172.21.15.28:0/2239679010 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b00bc4e0 con 0x55d3638da460 2024-04-24T08:20:27.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.960+0000 7fa8cf84b740 1 -- 172.21.15.28:0/2239679010 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8b8003e50 con 0x55d3638da460 2024-04-24T08:20:27.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.960+0000 7fa8ab400640 1 -- 172.21.15.28:0/2239679010 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8644+0+0 (secure 0 0 0) 0x7fa8b00c50f0 con 0x55d3638da460 2024-04-24T08:20:27.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.961+0000 7fa8ab400640 1 -- 172.21.15.28:0/2239679010 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa8b00ee240 con 0x55d3638da460 2024-04-24T08:20:27.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.961+0000 7fa8cf84b740 10 client.27022 fetch_fsmap learned FSMap version 26 2024-04-24T08:20:27.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.961+0000 7fa8cf84b740 1 -- 172.21.15.28:0/2239679010 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d3635513f0 con 0x55d3638da460 2024-04-24T08:20:27.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.961+0000 7fa8ab400640 1 -- 172.21.15.28:0/2239679010 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8b8003e50 con 0x55d3638da460 2024-04-24T08:20:27.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.961+0000 7fa8ab400640 1 -- 172.21.15.28:0/2239679010 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa8b003a070 con 0x55d3638da460 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.961+0000 7fa8cf84b740 10 client.27022 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.961+0000 7fa8cf84b740 10 client.27022 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.961+0000 7fa8cf84b740 20 client.27022 populate_metadata read hostname 'smithi028' 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.961+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/2239679010 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55d3638f4c10 0x55d3638f7090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.961+0000 7fa8cf84b740 4 client.27022 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.961+0000 7fa8cf84b740 1 -- 172.21.15.28:0/2239679010 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d36384cbf0 con 0x55d3638f4c10 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.962+0000 7fa8c5e00640 1 --2- 172.21.15.28:0/2239679010 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55d3638f4c10 0x55d3638f7090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.962+0000 7fa8c5e00640 1 --2- 172.21.15.28:0/2239679010 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55d3638f4c10 0x55d3638f7090 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=56702cb3e34e674b server_cookie=d110fbb19fb7fcbf in_seq=0 out_seq=0 2024-04-24T08:20:27.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.962+0000 7fa8ab400640 10 client.27022 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:20:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.963+0000 7fa8ab400640 1 -- 172.21.15.28:0/2239679010 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d36384cbf0 con 0x55d3638f4c10 2024-04-24T08:20:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.963+0000 7fa8ab400640 10 client.27022 handle_command_reply: tid=0 2024-04-24T08:20:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.963+0000 7fa8cf84b740 1 client.27022 shutdown 2024-04-24T08:20:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.963+0000 7fa8cf84b740 1 -- 172.21.15.28:0/2239679010 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638da460 msgr2=0x55d3638e1880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.963+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/2239679010 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638da460 0x55d3638e1880 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fa8b00edec0 tx=0x7fa8b0030af0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.964+0000 7fa8cf84b740 1 -- 172.21.15.28:0/2239679010 shutdown_connections 2024-04-24T08:20:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.964+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/2239679010 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55d3638f4c10 0x55d3638f7090 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.964+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/2239679010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638e4b30 0x55d3638e1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.964+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/2239679010 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638da460 0x55d3638e1880 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.964+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/2239679010 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638d9b40 0x55d3638e4590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.964+0000 7fa8ab400640 0 client.27022 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:20:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.964+0000 7fa8cf84b740 1 -- 172.21.15.28:0/2239679010 >> 172.21.15.28:0/2239679010 conn(0x55d3638d5e60 msgr2=0x55d3638dbb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.964+0000 7fa8cf84b740 1 -- 172.21.15.28:0/2239679010 shutdown_connections 2024-04-24T08:20:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.964+0000 7fa8cf84b740 1 -- 172.21.15.28:0/2239679010 wait complete. 2024-04-24T08:20:27.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.964+0000 7fa8cf84b740 20 client.27022 trim_cache size 0 max 16384 2024-04-24T08:20:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.979+0000 7fa8cf84b740 1 Processor -- start 2024-04-24T08:20:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.979+0000 7fa8cf84b740 1 -- start start 2024-04-24T08:20:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.979+0000 7fa8cf84b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638d9b40 0x55d3638e09e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.979+0000 7fa8cf84b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638da460 0x55d3638e40c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.979+0000 7fa8cf84b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638e4b30 0x55d3638d4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.979+0000 7fa8cf84b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c81b6c60 con 0x55d3638da460 2024-04-24T08:20:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.979+0000 7fa8cf84b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c81b4530 con 0x55d3638d9b40 2024-04-24T08:20:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.979+0000 7fa8cf84b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c81b23a0 con 0x55d3638e4b30 2024-04-24T08:20:27.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.979+0000 7fa8bd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638da460 0x55d3638e40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.979+0000 7fa8c5e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638d9b40 0x55d3638e09e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.979+0000 7fa8c6800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638e4b30 0x55d3638d4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.979+0000 7fa8bd400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638da460 0x55d3638e40c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37294/0 (socket says 172.21.15.28:37294) 2024-04-24T08:20:27.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.979+0000 7fa8bd400640 1 -- 172.21.15.28:0/4221151719 learned_addr learned my addr 172.21.15.28:0/4221151719 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:27.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.980+0000 7fa8c5e00640 1 -- 172.21.15.28:0/4221151719 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638e4b30 msgr2=0x55d3638d4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.983 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.980+0000 7fa8c5e00640 1 --2- 172.21.15.28:0/4221151719 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638e4b30 0x55d3638d4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.980+0000 7fa8c5e00640 1 -- 172.21.15.28:0/4221151719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638da460 msgr2=0x55d3638e40c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.980+0000 7fa8c5e00640 1 --2- 172.21.15.28:0/4221151719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638da460 0x55d3638e40c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.980+0000 7fa8c5e00640 1 -- 172.21.15.28:0/4221151719 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d3638da290 con 0x55d3638d9b40 2024-04-24T08:20:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.980+0000 7fa8c6800640 1 --2- 172.21.15.28:0/4221151719 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638e4b30 0x55d3638d4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:20:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.980+0000 7fa8c5e00640 1 --2- 172.21.15.28:0/4221151719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638d9b40 0x55d3638e09e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8004110 tx=0x7fa8b8011420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d4e714f14a870d2f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.980+0000 7fa8b6600640 1 -- 172.21.15.28:0/4221151719 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b800fba0 con 0x55d3638d9b40 2024-04-24T08:20:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.980+0000 7fa8b6600640 1 -- 172.21.15.28:0/4221151719 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8b8003e50 con 0x55d3638d9b40 2024-04-24T08:20:27.984 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.980+0000 7fa8b6600640 1 -- 172.21.15.28:0/4221151719 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b80056a0 con 0x55d3638d9b40 2024-04-24T08:20:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.981+0000 7fa8cf84b740 1 -- 172.21.15.28:0/4221151719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638d9b40 msgr2=0x55d3638e09e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.981+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/4221151719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638d9b40 0x55d3638e09e0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8004110 tx=0x7fa8b8011420 comp rx=0 tx=0).stop 2024-04-24T08:20:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.981+0000 7fa8cf84b740 1 -- 172.21.15.28:0/4221151719 shutdown_connections 2024-04-24T08:20:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.981+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/4221151719 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638da460 0x55d3638e40c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.981+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/4221151719 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638e4b30 0x55d3638d4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.981+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/4221151719 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638d9b40 0x55d3638e09e0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.981+0000 7fa8cf84b740 1 -- 172.21.15.28:0/4221151719 >> 172.21.15.28:0/4221151719 conn(0x55d3638d5e60 msgr2=0x55d3638d46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.985 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.981+0000 7fa8cf84b740 1 -- 172.21.15.28:0/4221151719 shutdown_connections 2024-04-24T08:20:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.981+0000 7fa8cf84b740 1 -- 172.21.15.28:0/4221151719 wait complete. 2024-04-24T08:20:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.981+0000 7fa8cf84b740 1 -- start start 2024-04-24T08:20:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.982+0000 7fa8cf84b740 1 Processor -- start 2024-04-24T08:20:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.982+0000 7fa8cf84b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638d9b40 0x55d3638ece00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.982+0000 7fa8cf84b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638da460 0x55d3638ed340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.982+0000 7fa8cf84b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638e4b30 0x55d3638e7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.982+0000 7fa8cf84b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c81b29f0 con 0x55d3638d9b40 2024-04-24T08:20:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.982+0000 7fa8cf84b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c81b3e60 con 0x55d3638e4b30 2024-04-24T08:20:27.986 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.982+0000 7fa8cf84b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa8c81b30c0 con 0x55d3638da460 2024-04-24T08:20:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.982+0000 7fa8cf84b740 10 client.0 _ll_register_callbacks cb 0x7fa8b00f6220 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.982+0000 7fa8c5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638d9b40 0x55d3638ece00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.982+0000 7fa8c5e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638d9b40 0x55d3638ece00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37296/0 (socket says 172.21.15.28:37296) 2024-04-24T08:20:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.982+0000 7fa8c5e00640 1 -- 172.21.15.28:0/719479023 learned_addr learned my addr 172.21.15.28:0/719479023 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.982+0000 7fa8c6800640 1 --2- 172.21.15.28:0/719479023 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638e4b30 0x55d3638e7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.983+0000 7fa8bd400640 1 --2- 172.21.15.28:0/719479023 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638da460 0x55d3638ed340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.983+0000 7fa8c5e00640 1 -- 172.21.15.28:0/719479023 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638da460 msgr2=0x55d3638ed340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.987 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.983+0000 7fa8c5e00640 1 --2- 172.21.15.28:0/719479023 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638da460 0x55d3638ed340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.983+0000 7fa8c5e00640 1 -- 172.21.15.28:0/719479023 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638e4b30 msgr2=0x55d3638e7db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.983+0000 7fa8c5e00640 1 --2- 172.21.15.28:0/719479023 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638e4b30 0x55d3638e7db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.983+0000 7fa8c5e00640 1 -- 172.21.15.28:0/719479023 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d36361b7d0 con 0x55d3638d9b40 2024-04-24T08:20:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.983+0000 7fa8c5e00640 1 --2- 172.21.15.28:0/719479023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638d9b40 0x55d3638ece00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8011d60 tx=0x7fa8b8004080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d0c2f3261ad3e04 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:27.988 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.983+0000 7fa8ab400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:20:27.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.983+0000 7fa8ab400640 1 -- 172.21.15.28:0/719479023 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa8b800f840 con 0x55d3638d9b40 2024-04-24T08:20:27.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.983+0000 7fa8ab400640 1 -- 172.21.15.28:0/719479023 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8644+0+0 (secure 0 0 0) 0x7fa8b800f9e0 con 0x55d3638d9b40 2024-04-24T08:20:27.990 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.984+0000 7fa8cf84b740 1 -- 172.21.15.28:0/719479023 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa8b8003e50 con 0x55d3638d9b40 2024-04-24T08:20:27.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.984+0000 7fa8ab400640 1 -- 172.21.15.28:0/719479023 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fa8b803d050 con 0x55d3638d9b40 2024-04-24T08:20:27.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.984+0000 7fa8cf84b740 10 client.16737 fetch_fsmap learned FSMap version 26 2024-04-24T08:20:27.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.984+0000 7fa8cf84b740 1 -- 172.21.15.28:0/719479023 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55d36395b9b0 con 0x55d3638d9b40 2024-04-24T08:20:27.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.984+0000 7fa8ab400640 1 -- 172.21.15.28:0/719479023 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa8b8003e50 con 0x55d3638d9b40 2024-04-24T08:20:27.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.984+0000 7fa8ab400640 1 -- 172.21.15.28:0/719479023 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fa8b802c0c0 con 0x55d3638d9b40 2024-04-24T08:20:27.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.984+0000 7fa8cf84b740 10 client.16737 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:20:27.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.984+0000 7fa8cf84b740 10 client.16737 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:20:27.991 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.984+0000 7fa8cf84b740 20 client.16737 populate_metadata read hostname 'smithi028' 2024-04-24T08:20:27.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.984+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/719479023 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55d36395be20 0x55d36395e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:27.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.984+0000 7fa8cf84b740 4 client.16737 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:20:27.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.984+0000 7fa8cf84b740 1 -- 172.21.15.28:0/719479023 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55d363846a60 con 0x55d36395be20 2024-04-24T08:20:27.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.985+0000 7fa8bd400640 1 --2- 172.21.15.28:0/719479023 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55d36395be20 0x55d36395e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:27.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.985+0000 7fa8bd400640 1 --2- 172.21.15.28:0/719479023 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55d36395be20 0x55d36395e240 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6fc713a72e9bde2c server_cookie=5df239129b3a9f75 in_seq=0 out_seq=0 2024-04-24T08:20:27.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.985+0000 7fa8ab400640 10 client.16737 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:20:27.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.986+0000 7fa8ab400640 1 -- 172.21.15.28:0/719479023 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55d363846a60 con 0x55d36395be20 2024-04-24T08:20:27.992 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.986+0000 7fa8ab400640 10 client.16737 handle_command_reply: tid=0 2024-04-24T08:20:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.986+0000 7fa8cf84b740 1 client.16737 shutdown 2024-04-24T08:20:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.986+0000 7fa8cf84b740 1 -- 172.21.15.28:0/719479023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638d9b40 msgr2=0x55d3638ece00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.986+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/719479023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638d9b40 0x55d3638ece00 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fa8b8011d60 tx=0x7fa8b8004080 comp rx=0 tx=0).stop 2024-04-24T08:20:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.986+0000 7fa8cf84b740 1 -- 172.21.15.28:0/719479023 shutdown_connections 2024-04-24T08:20:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.986+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/719479023 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x55d36395be20 0x55d36395e240 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.986+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/719479023 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55d3638e4b30 0x55d3638e7db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.986+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/719479023 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55d3638da460 0x55d3638ed340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.986+0000 7fa8cf84b740 1 --2- 172.21.15.28:0/719479023 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55d3638d9b40 0x55d3638ece00 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.993 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.986+0000 7fa8cf84b740 1 -- 172.21.15.28:0/719479023 >> 172.21.15.28:0/719479023 conn(0x55d3638d5e60 msgr2=0x55d3638d46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.986+0000 7fa8ab400640 0 client.16737 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:20:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.986+0000 7fa8cf84b740 1 -- 172.21.15.28:0/719479023 shutdown_connections 2024-04-24T08:20:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.986+0000 7fa8cf84b740 1 -- 172.21.15.28:0/719479023 wait complete. 2024-04-24T08:20:27.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.986+0000 7fa8cf84b740 20 client.16737 trim_cache size 0 max 16384 2024-04-24T08:20:27.994 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:20:27.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.987+0000 7fa8c7200640 1 -- 172.21.15.28:0/3746136671 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa8ac0a3d20 msgr2=0x7fa8ac0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.987+0000 7fa8c7200640 1 --2- 172.21.15.28:0/3746136671 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa8ac0a3d20 0x7fa8ac0a61e0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fa8c806b440 tx=0x7fa8c0009260 comp rx=0 tx=0).stop 2024-04-24T08:20:27.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.987+0000 7fa8c7200640 1 -- 172.21.15.28:0/3746136671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8c8076e50 msgr2=0x7fa8c819f330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:27.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.987+0000 7fa8c7200640 1 --2- 172.21.15.28:0/3746136671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8c8076e50 0x7fa8c819f330 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fa8b0030510 tx=0x7fa8b0033ae0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.987+0000 7fa8c7200640 1 -- 172.21.15.28:0/3746136671 shutdown_connections 2024-04-24T08:20:27.995 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.987+0000 7fa8c7200640 1 --2- 172.21.15.28:0/3746136671 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa8ac0a3d20 0x7fa8ac0a61e0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.987+0000 7fa8c7200640 1 --2- 172.21.15.28:0/3746136671 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa8c8076e50 0x7fa8c819f330 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.987+0000 7fa8c7200640 1 --2- 172.21.15.28:0/3746136671 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa8c8112d40 0x7fa8c81a36b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.987+0000 7fa8c7200640 1 --2- 172.21.15.28:0/3746136671 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8c8076530 0x7fa8c819edf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:27.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.987+0000 7fa8c7200640 1 -- 172.21.15.28:0/3746136671 >> 172.21.15.28:0/3746136671 conn(0x7fa8c80ffb10 msgr2=0x7fa8c8114460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:27.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.987+0000 7fa8c7200640 1 -- 172.21.15.28:0/3746136671 shutdown_connections 2024-04-24T08:20:27.996 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:27.987+0000 7fa8c7200640 1 -- 172.21.15.28:0/3746136671 wait complete. 2024-04-24T08:20:28.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:27 smithi028 ceph-mon[27786]: pgmap v313: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:20:28.004 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:27 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/775720194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:20:28.006 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:20:28.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:27 smithi117 ceph-mon[32438]: pgmap v313: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:20:28.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:27 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/775720194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:20:28.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:27 smithi194 ceph-mon[32836]: pgmap v313: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:20:28.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:27 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/775720194' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:20:28.207 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.206+0000 7ff94a200640 1 -- 172.21.15.28:0/1208255245 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff944076e50 msgr2=0x7ff944112800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.206+0000 7ff94a200640 1 --2- 172.21.15.28:0/1208255245 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff944076e50 0x7ff944112800 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff92c009c10 tx=0x7ff92c02a6e0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.207+0000 7ff94a200640 1 -- 172.21.15.28:0/1208255245 shutdown_connections 2024-04-24T08:20:28.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.207+0000 7ff94a200640 1 --2- 172.21.15.28:0/1208255245 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff944112d40 0x7ff944115130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.207+0000 7ff94a200640 1 --2- 172.21.15.28:0/1208255245 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff944076e50 0x7ff944112800 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.207+0000 7ff94a200640 1 --2- 172.21.15.28:0/1208255245 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff944076530 0x7ff944076910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.207+0000 7ff94a200640 1 -- 172.21.15.28:0/1208255245 >> 172.21.15.28:0/1208255245 conn(0x7ff9440ffb10 msgr2=0x7ff944101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.207+0000 7ff94a200640 1 -- 172.21.15.28:0/1208255245 shutdown_connections 2024-04-24T08:20:28.208 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.207+0000 7ff94a200640 1 -- 172.21.15.28:0/1208255245 wait complete. 2024-04-24T08:20:28.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.208+0000 7ff94a200640 1 Processor -- start 2024-04-24T08:20:28.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.208+0000 7ff94a200640 1 -- start start 2024-04-24T08:20:28.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.208+0000 7ff94a200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff944076530 0x7ff94419ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.208+0000 7ff94a200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff944076e50 0x7ff94419f350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.211 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.208+0000 7ff94a200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff944112d40 0x7ff9441a3680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.208+0000 7ff94a200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff944118890 con 0x7ff944076530 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.208+0000 7ff94a200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff944118710 con 0x7ff944076e50 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.208+0000 7ff94a200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff944118a10 con 0x7ff944112d40 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.209+0000 7ff948e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff944076530 0x7ff94419ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.209+0000 7ff943e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff944076e50 0x7ff94419f350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.209+0000 7ff943e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff944076e50 0x7ff94419f350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:36446/0 (socket says 172.21.15.28:36446) 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.209+0000 7ff948e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff944076530 0x7ff94419ee10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37308/0 (socket says 172.21.15.28:37308) 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.209+0000 7ff943e00640 1 -- 172.21.15.28:0/2535862865 learned_addr learned my addr 172.21.15.28:0/2535862865 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:28.212 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.209+0000 7ff949800640 1 --2- 172.21.15.28:0/2535862865 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff944112d40 0x7ff9441a3680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.209+0000 7ff943e00640 1 -- 172.21.15.28:0/2535862865 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff944112d40 msgr2=0x7ff9441a3680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.209+0000 7ff943e00640 1 --2- 172.21.15.28:0/2535862865 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff944112d40 0x7ff9441a3680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.209+0000 7ff943e00640 1 -- 172.21.15.28:0/2535862865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff944076530 msgr2=0x7ff94419ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.209+0000 7ff943e00640 1 --2- 172.21.15.28:0/2535862865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff944076530 0x7ff94419ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.209+0000 7ff943e00640 1 -- 172.21.15.28:0/2535862865 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff9441a3db0 con 0x7ff944076e50 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.209+0000 7ff943e00640 1 --2- 172.21.15.28:0/2535862865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff944076e50 0x7ff94419f350 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff92c000f80 tx=0x7ff92c0305d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.210+0000 7ff941600640 1 -- 172.21.15.28:0/2535862865 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92c041020 con 0x7ff944076e50 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.210+0000 7ff941600640 1 -- 172.21.15.28:0/2535862865 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff92c003e50 con 0x7ff944076e50 2024-04-24T08:20:28.213 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.210+0000 7ff94a200640 1 -- 172.21.15.28:0/2535862865 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff9441a4040 con 0x7ff944076e50 2024-04-24T08:20:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.210+0000 7ff941600640 1 -- 172.21.15.28:0/2535862865 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92c036040 con 0x7ff944076e50 2024-04-24T08:20:28.214 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.210+0000 7ff94a200640 1 -- 172.21.15.28:0/2535862865 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff9441ab930 con 0x7ff944076e50 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.213+0000 7ff94e7c2740 1 Processor -- start 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.213+0000 7ff941600640 1 -- 172.21.15.28:0/2535862865 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff92c044020 con 0x7ff944076e50 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.213+0000 7ff94e7c2740 1 -- start start 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.214+0000 7ff94e7c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa90b40 0x55b04aa90f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.214+0000 7ff94e7c2740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa91460 0x55b04aa9b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.214+0000 7ff94e7c2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa9bb30 0x55b04aa9df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.214+0000 7ff94e7c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b04a9f61d0 con 0x55b04aa90b40 2024-04-24T08:20:28.216 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.214+0000 7ff94e7c2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b04a99c450 con 0x55b04aa9bb30 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.214+0000 7ff94e7c2740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b04a9f2190 con 0x55b04aa91460 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.214+0000 7ff949800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa90b40 0x55b04aa90f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.214+0000 7ff948e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa91460 0x55b04aa9b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.214+0000 7ff949800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa90b40 0x55b04aa90f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37324/0 (socket says 172.21.15.28:37324) 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.214+0000 7ff948e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa91460 0x55b04aa9b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:54450/0 (socket says 172.21.15.28:54450) 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.214+0000 7ff949800640 1 -- 172.21.15.28:0/3101225400 learned_addr learned my addr 172.21.15.28:0/3101225400 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.214+0000 7ff943e00640 1 --2- 172.21.15.28:0/3101225400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa9bb30 0x55b04aa9df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.215+0000 7ff948e00640 1 -- 172.21.15.28:0/3101225400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa9bb30 msgr2=0x55b04aa9df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.217 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.215+0000 7ff948e00640 1 --2- 172.21.15.28:0/3101225400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa9bb30 0x55b04aa9df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.215+0000 7ff948e00640 1 -- 172.21.15.28:0/3101225400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa90b40 msgr2=0x55b04aa90f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.215+0000 7ff948e00640 1 --2- 172.21.15.28:0/3101225400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa90b40 0x55b04aa90f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.215+0000 7ff948e00640 1 -- 172.21.15.28:0/3101225400 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b04aa59310 con 0x55b04aa91460 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.215+0000 7ff948e00640 1 --2- 172.21.15.28:0/3101225400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa91460 0x55b04aa9b5f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff934005fd0 tx=0x7ff934011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fc5805a9fc5bb7d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.215+0000 7ff941600640 1 --2- 172.21.15.28:0/2535862865 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff9240a3d20 0x7ff9240a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.215+0000 7ff93a000640 1 -- 172.21.15.28:0/3101225400 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff934004020 con 0x55b04aa91460 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.215+0000 7ff93a000640 1 -- 172.21.15.28:0/3101225400 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9340041c0 con 0x55b04aa91460 2024-04-24T08:20:28.218 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.215+0000 7ff93a000640 1 -- 172.21.15.28:0/3101225400 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff934019890 con 0x55b04aa91460 2024-04-24T08:20:28.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.215+0000 7ff941600640 1 -- 172.21.15.28:0/2535862865 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 8644+0+0 (secure 0 0 0) 0x7ff92c0f1600 con 0x7ff944076e50 2024-04-24T08:20:28.219 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.215+0000 7ff949800640 1 --2- 172.21.15.28:0/2535862865 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff9240a3d20 0x7ff9240a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.216+0000 7ff94e7c2740 1 -- 172.21.15.28:0/3101225400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa91460 msgr2=0x55b04aa9b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.216+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/3101225400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa91460 0x55b04aa9b5f0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff934005fd0 tx=0x7ff934011810 comp rx=0 tx=0).stop 2024-04-24T08:20:28.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.216+0000 7ff94e7c2740 1 -- 172.21.15.28:0/3101225400 shutdown_connections 2024-04-24T08:20:28.223 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.216+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/3101225400 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa9bb30 0x55b04aa9df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.216+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/3101225400 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa91460 0x55b04aa9b5f0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.216+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/3101225400 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa90b40 0x55b04aa90f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.216+0000 7ff94e7c2740 1 -- 172.21.15.28:0/3101225400 >> 172.21.15.28:0/3101225400 conn(0x55b04aa8ce60 msgr2=0x55b04aa8d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.216+0000 7ff949800640 1 --2- 172.21.15.28:0/2535862865 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff9240a3d20 0x7ff9240a61e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff93c009de0 tx=0x7ff93c005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.216+0000 7ff94e7c2740 1 -- 172.21.15.28:0/3101225400 shutdown_connections 2024-04-24T08:20:28.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.216+0000 7ff94e7c2740 1 -- 172.21.15.28:0/3101225400 wait complete. 2024-04-24T08:20:28.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.217+0000 7ff94e7c2740 1 -- start start 2024-04-24T08:20:28.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.217+0000 7ff94e7c2740 1 Processor -- start 2024-04-24T08:20:28.224 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.217+0000 7ff94e7c2740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa90b40 0x55b04aa9b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.217+0000 7ff94e7c2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 0x55b04aa988d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.217+0000 7ff94e7c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa9bb30 0x55b04aa99010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.217+0000 7ff94e7c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x55b04aa9e9c0 con 0x55b04aa9bb30 2024-04-24T08:20:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.217+0000 7ff94e7c2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x55b04a9af810 con 0x55b04aa91460 2024-04-24T08:20:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.217+0000 7ff94e7c2740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x55b04aa9eb40 con 0x55b04aa90b40 2024-04-24T08:20:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.217+0000 7ff94e7c2740 10 client.0 _ll_register_callbacks cb 0x55b04a97d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.218+0000 7ff943e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 0x55b04aa988d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.218+0000 7ff943e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 0x55b04aa988d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:36462/0 (socket says 172.21.15.28:36462) 2024-04-24T08:20:28.225 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.218+0000 7ff943e00640 1 -- 172.21.15.28:0/2622246748 learned_addr learned my addr 172.21.15.28:0/2622246748 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:28.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.218+0000 7ff948e00640 1 --2- 172.21.15.28:0/2622246748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa90b40 0x55b04aa9b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.218+0000 7ff949800640 1 --2- 172.21.15.28:0/2622246748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa9bb30 0x55b04aa99010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.218+0000 7ff943e00640 1 -- 172.21.15.28:0/2622246748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa90b40 msgr2=0x55b04aa9b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.218+0000 7ff943e00640 1 --2- 172.21.15.28:0/2622246748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa90b40 0x55b04aa9b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.218+0000 7ff943e00640 1 -- 172.21.15.28:0/2622246748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa9bb30 msgr2=0x55b04aa99010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.218+0000 7ff943e00640 1 --2- 172.21.15.28:0/2622246748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa9bb30 0x55b04aa99010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.218+0000 7ff943e00640 1 -- 172.21.15.28:0/2622246748 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b04a9c03f0 con 0x55b04aa91460 2024-04-24T08:20:28.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.218+0000 7ff948e00640 1 --2- 172.21.15.28:0/2622246748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa90b40 0x55b04aa9b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:28.226 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.219+0000 7ff943e00640 1 --2- 172.21.15.28:0/2622246748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 0x55b04aa988d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff92c0eecd0 tx=0x7ff92c009960 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a32cf41e2bec9f3d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.219+0000 7ff932a00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:20:28.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.219+0000 7ff932a00640 1 -- 172.21.15.28:0/2622246748 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff92c0bce20 con 0x55b04aa91460 2024-04-24T08:20:28.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.219+0000 7ff932a00640 1 -- 172.21.15.28:0/2622246748 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8644+0+0 (secure 0 0 0) 0x7ff92c0eeef0 con 0x55b04aa91460 2024-04-24T08:20:28.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.219+0000 7ff94e7c2740 1 -- 172.21.15.28:0/2622246748 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff9340041c0 con 0x55b04aa91460 2024-04-24T08:20:28.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.219+0000 7ff932a00640 1 -- 172.21.15.28:0/2622246748 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff92c0ef0f0 con 0x55b04aa91460 2024-04-24T08:20:28.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.219+0000 7ff94e7c2740 10 client.27026 fetch_fsmap learned FSMap version 26 2024-04-24T08:20:28.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.219+0000 7ff94e7c2740 1 -- 172.21.15.28:0/2622246748 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b04a7083f0 con 0x55b04aa91460 2024-04-24T08:20:28.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.219+0000 7ff932a00640 1 -- 172.21.15.28:0/2622246748 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff9340041c0 con 0x55b04aa91460 2024-04-24T08:20:28.227 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.219+0000 7ff932a00640 1 -- 172.21.15.28:0/2622246748 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff92c0c5ca0 con 0x55b04aa91460 2024-04-24T08:20:28.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.220+0000 7ff94e7c2740 10 client.27026 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:20:28.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.220+0000 7ff94e7c2740 10 client.27026 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:20:28.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.220+0000 7ff94e7c2740 20 client.27026 populate_metadata read hostname 'smithi028' 2024-04-24T08:20:28.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.220+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/2622246748 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55b04aaabd00 0x55b04aaae1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.220+0000 7ff94e7c2740 4 client.27026 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:20:28.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.220+0000 7ff94e7c2740 1 -- 172.21.15.28:0/2622246748 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b04aa03bf0 con 0x55b04aaabd00 2024-04-24T08:20:28.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.220+0000 7ff948e00640 1 --2- 172.21.15.28:0/2622246748 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55b04aaabd00 0x55b04aaae1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.228 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.220+0000 7ff948e00640 1 --2- 172.21.15.28:0/2622246748 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55b04aaabd00 0x55b04aaae1a0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=594781996600b3f server_cookie=af1eafbad2ce3de0 in_seq=0 out_seq=0 2024-04-24T08:20:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.220+0000 7ff932a00640 10 client.27026 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:20:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.221+0000 7ff932a00640 1 -- 172.21.15.28:0/2622246748 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b04aa03bf0 con 0x55b04aaabd00 2024-04-24T08:20:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.221+0000 7ff932a00640 10 client.27026 handle_command_reply: tid=0 2024-04-24T08:20:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.221+0000 7ff94e7c2740 1 client.27026 shutdown 2024-04-24T08:20:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.222+0000 7ff94e7c2740 1 -- 172.21.15.28:0/2622246748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 msgr2=0x55b04aa988d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.222+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/2622246748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 0x55b04aa988d0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7ff92c0eecd0 tx=0x7ff92c009960 comp rx=0 tx=0).stop 2024-04-24T08:20:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.222+0000 7ff94e7c2740 1 -- 172.21.15.28:0/2622246748 shutdown_connections 2024-04-24T08:20:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.222+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/2622246748 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55b04aaabd00 0x55b04aaae1a0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.229 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.222+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/2622246748 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa9bb30 0x55b04aa99010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.222+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/2622246748 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 0x55b04aa988d0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.222+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/2622246748 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa90b40 0x55b04aa9b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.222+0000 7ff94e7c2740 1 -- 172.21.15.28:0/2622246748 >> 172.21.15.28:0/2622246748 conn(0x55b04aa8ce60 msgr2=0x55b04aa92c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.222+0000 7ff932a00640 0 client.27026 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:20:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.222+0000 7ff94e7c2740 1 -- 172.21.15.28:0/2622246748 shutdown_connections 2024-04-24T08:20:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.222+0000 7ff94e7c2740 1 -- 172.21.15.28:0/2622246748 wait complete. 2024-04-24T08:20:28.230 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.222+0000 7ff94e7c2740 20 client.27026 trim_cache size 0 max 16384 2024-04-24T08:20:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.238+0000 7ff94e7c2740 1 Processor -- start 2024-04-24T08:20:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.238+0000 7ff94e7c2740 1 -- start start 2024-04-24T08:20:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.238+0000 7ff94e7c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa90b40 0x55b04aa8bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.238+0000 7ff94e7c2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 0x55b04aa9b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.238+0000 7ff94e7c2740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa9bb30 0x55b04aab3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.239 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.238+0000 7ff94e7c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff9441b6ad0 con 0x55b04aa90b40 2024-04-24T08:20:28.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.238+0000 7ff94e7c2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff9441b43a0 con 0x55b04aa91460 2024-04-24T08:20:28.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.238+0000 7ff94e7c2740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff9441b2210 con 0x55b04aa9bb30 2024-04-24T08:20:28.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.239+0000 7ff948e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa90b40 0x55b04aa8bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.239+0000 7ff949800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa9bb30 0x55b04aab3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.239+0000 7ff943e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 0x55b04aa9b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.239+0000 7ff948e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa90b40 0x55b04aa8bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37352/0 (socket says 172.21.15.28:37352) 2024-04-24T08:20:28.240 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.239+0000 7ff948e00640 1 -- 172.21.15.28:0/781224850 learned_addr learned my addr 172.21.15.28:0/781224850 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:28.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.239+0000 7ff948e00640 1 -- 172.21.15.28:0/781224850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa9bb30 msgr2=0x55b04aab3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.239+0000 7ff948e00640 1 --2- 172.21.15.28:0/781224850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa9bb30 0x55b04aab3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.239+0000 7ff948e00640 1 -- 172.21.15.28:0/781224850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 msgr2=0x55b04aa9b110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.239+0000 7ff948e00640 1 --2- 172.21.15.28:0/781224850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 0x55b04aa9b110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.239+0000 7ff948e00640 1 -- 172.21.15.28:0/781224850 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b04a7d27d0 con 0x55b04aa90b40 2024-04-24T08:20:28.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.239+0000 7ff943e00640 1 --2- 172.21.15.28:0/781224850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 0x55b04aa9b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:28.242 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.239+0000 7ff948e00640 1 --2- 172.21.15.28:0/781224850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa90b40 0x55b04aa8bdd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff934003d90 tx=0x7ff934003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b418bf0f38eeeee8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.240+0000 7ff939600640 1 -- 172.21.15.28:0/781224850 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff934009db0 con 0x55b04aa90b40 2024-04-24T08:20:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.240+0000 7ff939600640 1 -- 172.21.15.28:0/781224850 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff93402f030 con 0x55b04aa90b40 2024-04-24T08:20:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.240+0000 7ff939600640 1 -- 172.21.15.28:0/781224850 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93402bdd0 con 0x55b04aa90b40 2024-04-24T08:20:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.240+0000 7ff94e7c2740 1 -- 172.21.15.28:0/781224850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa90b40 msgr2=0x55b04aa8bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.240+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/781224850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa90b40 0x55b04aa8bdd0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7ff934003d90 tx=0x7ff934003d00 comp rx=0 tx=0).stop 2024-04-24T08:20:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.241+0000 7ff94e7c2740 1 -- 172.21.15.28:0/781224850 shutdown_connections 2024-04-24T08:20:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.241+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/781224850 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa9bb30 0x55b04aab3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.241+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/781224850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 0x55b04aa9b110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.241+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/781224850 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa90b40 0x55b04aa8bdd0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.243 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.241+0000 7ff94e7c2740 1 -- 172.21.15.28:0/781224850 >> 172.21.15.28:0/781224850 conn(0x55b04aa8ce60 msgr2=0x55b04aa8b6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.241+0000 7ff94e7c2740 1 -- 172.21.15.28:0/781224850 shutdown_connections 2024-04-24T08:20:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.241+0000 7ff94e7c2740 1 -- 172.21.15.28:0/781224850 wait complete. 2024-04-24T08:20:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.241+0000 7ff94e7c2740 1 -- start start 2024-04-24T08:20:28.244 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.241+0000 7ff94e7c2740 1 Processor -- start 2024-04-24T08:20:28.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff94e7c2740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa90b40 0x55b04aa9d390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff94e7c2740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 0x55b04aa9d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff94e7c2740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa9bb30 0x55b04aaa24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.247 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff94e7c2740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff9441b2860 con 0x55b04aa9bb30 2024-04-24T08:20:28.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff94e7c2740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff9441b3cd0 con 0x55b04aa91460 2024-04-24T08:20:28.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff94e7c2740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff9441b2f30 con 0x55b04aa90b40 2024-04-24T08:20:28.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff94e7c2740 10 client.0 _ll_register_callbacks cb 0x7ff92c0f4da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:28.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff949800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa9bb30 0x55b04aaa24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff949800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa9bb30 0x55b04aaa24e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37360/0 (socket says 172.21.15.28:37360) 2024-04-24T08:20:28.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff949800640 1 -- 172.21.15.28:0/2104117075 learned_addr learned my addr 172.21.15.28:0/2104117075 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:28.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff943e00640 1 --2- 172.21.15.28:0/2104117075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 0x55b04aa9d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff948e00640 1 --2- 172.21.15.28:0/2104117075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa90b40 0x55b04aa9d390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff949800640 1 -- 172.21.15.28:0/2104117075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa90b40 msgr2=0x55b04aa9d390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.248 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff949800640 1 --2- 172.21.15.28:0/2104117075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa90b40 0x55b04aa9d390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff949800640 1 -- 172.21.15.28:0/2104117075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 msgr2=0x55b04aa9d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff949800640 1 --2- 172.21.15.28:0/2104117075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 0x55b04aa9d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff949800640 1 -- 172.21.15.28:0/2104117075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b04aaa2b80 con 0x55b04aa9bb30 2024-04-24T08:20:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.242+0000 7ff943e00640 1 --2- 172.21.15.28:0/2104117075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 0x55b04aa9d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.243+0000 7ff949800640 1 --2- 172.21.15.28:0/2104117075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa9bb30 0x55b04aaa24e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff93c0045e0 tx=0x7ff93c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81af666a3e1f718 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.243+0000 7ff932a00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:20:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.243+0000 7ff932a00640 1 -- 172.21.15.28:0/2104117075 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff93c02f060 con 0x55b04aa9bb30 2024-04-24T08:20:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.243+0000 7ff932a00640 1 -- 172.21.15.28:0/2104117075 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 8644+0+0 (secure 0 0 0) 0x7ff93c01f610 con 0x55b04aa9bb30 2024-04-24T08:20:28.249 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.243+0000 7ff94e7c2740 1 -- 172.21.15.28:0/2104117075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff93402f030 con 0x55b04aa9bb30 2024-04-24T08:20:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.243+0000 7ff932a00640 1 -- 172.21.15.28:0/2104117075 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7ff93c01f810 con 0x55b04aa9bb30 2024-04-24T08:20:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.243+0000 7ff94e7c2740 10 client.16755 fetch_fsmap learned FSMap version 26 2024-04-24T08:20:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.243+0000 7ff94e7c2740 1 -- 172.21.15.28:0/2104117075 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x55b04ab12c80 con 0x55b04aa9bb30 2024-04-24T08:20:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.244+0000 7ff932a00640 1 -- 172.21.15.28:0/2104117075 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff93402f030 con 0x55b04aa9bb30 2024-04-24T08:20:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.244+0000 7ff932a00640 1 -- 172.21.15.28:0/2104117075 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7ff93c032b50 con 0x55b04aa9bb30 2024-04-24T08:20:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.244+0000 7ff94e7c2740 10 client.16755 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:20:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.244+0000 7ff94e7c2740 10 client.16755 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:20:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.244+0000 7ff94e7c2740 20 client.16755 populate_metadata read hostname 'smithi028' 2024-04-24T08:20:28.250 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.244+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/2104117075 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55b04ab13400 0x55b04ab15820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.244+0000 7ff94e7c2740 4 client.16755 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:20:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.244+0000 7ff94e7c2740 1 -- 172.21.15.28:0/2104117075 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b04a9fda60 con 0x55b04ab13400 2024-04-24T08:20:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.244+0000 7ff948e00640 1 --2- 172.21.15.28:0/2104117075 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55b04ab13400 0x55b04ab15820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.245+0000 7ff948e00640 1 --2- 172.21.15.28:0/2104117075 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55b04ab13400 0x55b04ab15820 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9eff104a7b144a5 server_cookie=14f29b0fb37a59c4 in_seq=0 out_seq=0 2024-04-24T08:20:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.245+0000 7ff932a00640 10 client.16755 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:20:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.245+0000 7ff932a00640 1 -- 172.21.15.28:0/2104117075 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55b04a9fda60 con 0x55b04ab13400 2024-04-24T08:20:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.245+0000 7ff932a00640 10 client.16755 handle_command_reply: tid=0 2024-04-24T08:20:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.245+0000 7ff94e7c2740 1 client.16755 shutdown 2024-04-24T08:20:28.251 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.245+0000 7ff94e7c2740 1 -- 172.21.15.28:0/2104117075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa9bb30 msgr2=0x55b04aaa24e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.245+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/2104117075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa9bb30 0x55b04aaa24e0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7ff93c0045e0 tx=0x7ff93c0046c0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.245+0000 7ff94e7c2740 1 -- 172.21.15.28:0/2104117075 shutdown_connections 2024-04-24T08:20:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.245+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/2104117075 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x55b04ab13400 0x55b04ab15820 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.245+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/2104117075 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x55b04aa9bb30 0x55b04aaa24e0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.245+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/2104117075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55b04aa91460 0x55b04aa9d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.245+0000 7ff94e7c2740 1 --2- 172.21.15.28:0/2104117075 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x55b04aa90b40 0x55b04aa9d390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.245+0000 7ff932a00640 0 client.16755 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:20:28.252 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.245+0000 7ff94e7c2740 1 -- 172.21.15.28:0/2104117075 >> 172.21.15.28:0/2104117075 conn(0x55b04aa8ce60 msgr2=0x55b04aa8bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.246+0000 7ff94e7c2740 1 -- 172.21.15.28:0/2104117075 shutdown_connections 2024-04-24T08:20:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.246+0000 7ff94e7c2740 1 -- 172.21.15.28:0/2104117075 wait complete. 2024-04-24T08:20:28.253 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.246+0000 7ff94e7c2740 20 client.16755 trim_cache size 0 max 16384 2024-04-24T08:20:28.253 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:20:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.246+0000 7ff94a200640 1 -- 172.21.15.28:0/2535862865 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff9240a3d20 msgr2=0x7ff9240a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.246+0000 7ff94a200640 1 --2- 172.21.15.28:0/2535862865 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff9240a3d20 0x7ff9240a61e0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff93c009de0 tx=0x7ff93c005f90 comp rx=0 tx=0).stop 2024-04-24T08:20:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.246+0000 7ff94a200640 1 -- 172.21.15.28:0/2535862865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff944076e50 msgr2=0x7ff94419f350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.246+0000 7ff94a200640 1 --2- 172.21.15.28:0/2535862865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff944076e50 0x7ff94419f350 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff92c000f80 tx=0x7ff92c0305d0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.254 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.246+0000 7ff94a200640 1 -- 172.21.15.28:0/2535862865 shutdown_connections 2024-04-24T08:20:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.246+0000 7ff94a200640 1 --2- 172.21.15.28:0/2535862865 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff9240a3d20 0x7ff9240a61e0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.246+0000 7ff94a200640 1 --2- 172.21.15.28:0/2535862865 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff944112d40 0x7ff9441a3680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.246+0000 7ff94a200640 1 --2- 172.21.15.28:0/2535862865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff944076e50 0x7ff94419f350 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.246+0000 7ff94a200640 1 --2- 172.21.15.28:0/2535862865 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff944076530 0x7ff94419ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.246+0000 7ff94a200640 1 -- 172.21.15.28:0/2535862865 >> 172.21.15.28:0/2535862865 conn(0x7ff9440ffb10 msgr2=0x7ff944114460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.247+0000 7ff94a200640 1 -- 172.21.15.28:0/2535862865 shutdown_connections 2024-04-24T08:20:28.255 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:28.247+0000 7ff94a200640 1 -- 172.21.15.28:0/2535862865 wait complete. 2024-04-24T08:20:29.266 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-04-24T08:20:29.266 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:20:29.266 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-04-24T08:20:29.488 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.487+0000 7f9d6a600640 1 -- 172.21.15.28:0/993714276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d6410d500 msgr2=0x7f9d6410d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.487+0000 7f9d6a600640 1 --2- 172.21.15.28:0/993714276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d6410d500 0x7f9d6410d8e0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9d58009930 tx=0x7f9d5802a440 comp rx=0 tx=0).stop 2024-04-24T08:20:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.488+0000 7f9d6a600640 1 -- 172.21.15.28:0/993714276 shutdown_connections 2024-04-24T08:20:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.488+0000 7f9d6a600640 1 --2- 172.21.15.28:0/993714276 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d64108f40 0x7f9d641153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.488+0000 7f9d6a600640 1 --2- 172.21.15.28:0/993714276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d64108580 0x7f9d64108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.488+0000 7f9d6a600640 1 --2- 172.21.15.28:0/993714276 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d6410d500 0x7f9d6410d8e0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.489 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.488+0000 7f9d6a600640 1 -- 172.21.15.28:0/993714276 >> 172.21.15.28:0/993714276 conn(0x7f9d64077e20 msgr2=0x7f9d64106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:29.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.488+0000 7f9d6a600640 1 -- 172.21.15.28:0/993714276 shutdown_connections 2024-04-24T08:20:29.490 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.488+0000 7f9d6a600640 1 -- 172.21.15.28:0/993714276 wait complete. 2024-04-24T08:20:29.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.489+0000 7f9d6a600640 1 Processor -- start 2024-04-24T08:20:29.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.489+0000 7f9d6a600640 1 -- start start 2024-04-24T08:20:29.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.489+0000 7f9d6a600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d64108580 0x7f9d6419ed80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.492 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.489+0000 7f9d6a600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d64108f40 0x7f9d6419f2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.489+0000 7f9d6a600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d6410d500 0x7f9d641a35f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.489+0000 7f9d6a600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d64118900 con 0x7f9d64108f40 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.489+0000 7f9d6a600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9d64118780 con 0x7f9d6410d500 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.489+0000 7f9d6a600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9d64118a80 con 0x7f9d64108580 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.489+0000 7f9d63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d64108f40 0x7f9d6419f2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.490+0000 7f9d63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d64108f40 0x7f9d6419f2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37384/0 (socket says 172.21.15.28:37384) 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.490+0000 7f9d63e00640 1 -- 172.21.15.28:0/2428226240 learned_addr learned my addr 172.21.15.28:0/2428226240 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:29.493 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.490+0000 7f9d69c00640 1 --2- 172.21.15.28:0/2428226240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d6410d500 0x7f9d641a35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.490+0000 7f9d69200640 1 --2- 172.21.15.28:0/2428226240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d64108580 0x7f9d6419ed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.490+0000 7f9d69c00640 1 -- 172.21.15.28:0/2428226240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d64108580 msgr2=0x7f9d6419ed80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.490+0000 7f9d69c00640 1 --2- 172.21.15.28:0/2428226240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d64108580 0x7f9d6419ed80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.490+0000 7f9d69c00640 1 -- 172.21.15.28:0/2428226240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d64108f40 msgr2=0x7f9d6419f2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.490+0000 7f9d69c00640 1 --2- 172.21.15.28:0/2428226240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d64108f40 0x7f9d6419f2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.490+0000 7f9d69c00640 1 -- 172.21.15.28:0/2428226240 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d641a3d20 con 0x7f9d6410d500 2024-04-24T08:20:29.494 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.490+0000 7f9d63e00640 1 --2- 172.21.15.28:0/2428226240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d64108f40 0x7f9d6419f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:29.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.490+0000 7f9d69c00640 1 --2- 172.21.15.28:0/2428226240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d6410d500 0x7f9d641a35f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c00ebc0 tx=0x7f9d5c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.491+0000 7f9d61600640 1 -- 172.21.15.28:0/2428226240 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c003b70 con 0x7f9d6410d500 2024-04-24T08:20:29.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.491+0000 7f9d6a600640 1 -- 172.21.15.28:0/2428226240 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d641a4010 con 0x7f9d6410d500 2024-04-24T08:20:29.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.491+0000 7f9d61600640 1 -- 172.21.15.28:0/2428226240 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d5c003d10 con 0x7f9d6410d500 2024-04-24T08:20:29.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.491+0000 7f9d61600640 1 -- 172.21.15.28:0/2428226240 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d5c010720 con 0x7f9d6410d500 2024-04-24T08:20:29.495 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.491+0000 7f9d6a600640 1 -- 172.21.15.28:0/2428226240 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d641ab8a0 con 0x7f9d6410d500 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.495+0000 7f9d6ea06740 1 Processor -- start 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.495+0000 7f9d61600640 1 -- 172.21.15.28:0/2428226240 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9d5c004020 con 0x7f9d6410d500 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.496+0000 7f9d6ea06740 1 -- start start 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.496+0000 7f9d6ea06740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec02fd270 0x560ec02fd650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.497 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.496+0000 7f9d6ea06740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fdb90 0x560ec0307d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.496+0000 7f9d6ea06740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec0308260 0x560ec030a650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.496+0000 7f9d6ea06740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560ec0262e70 con 0x560ec0308260 2024-04-24T08:20:29.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.496+0000 7f9d6ea06740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560ec0209380 con 0x560ec02fdb90 2024-04-24T08:20:29.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.496+0000 7f9d6ea06740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560ec025ee30 con 0x560ec02fd270 2024-04-24T08:20:29.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.496+0000 7f9d63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec0308260 0x560ec030a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.496+0000 7f9d69200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fdb90 0x560ec0307d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.496+0000 7f9d69200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fdb90 0x560ec0307d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:36506/0 (socket says 172.21.15.28:36506) 2024-04-24T08:20:29.498 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.496+0000 7f9d69200640 1 -- 172.21.15.28:0/4137837819 learned_addr learned my addr 172.21.15.28:0/4137837819 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:29.499 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.497+0000 7f9d63e00640 1 --2- 172.21.15.28:0/4137837819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec02fd270 0x560ec02fd650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.503 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.497+0000 7f9d69200640 1 -- 172.21.15.28:0/4137837819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec02fd270 msgr2=0x560ec02fd650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.497+0000 7f9d69200640 1 --2- 172.21.15.28:0/4137837819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec02fd270 0x560ec02fd650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.497+0000 7f9d69200640 1 -- 172.21.15.28:0/4137837819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec0308260 msgr2=0x560ec030a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.497+0000 7f9d69200640 1 --2- 172.21.15.28:0/4137837819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec0308260 0x560ec030a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.497+0000 7f9d69200640 1 -- 172.21.15.28:0/4137837819 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ec003d7e0 con 0x560ec02fdb90 2024-04-24T08:20:29.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.497+0000 7f9d63e00640 1 --2- 172.21.15.28:0/4137837819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec0308260 0x560ec030a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:29.504 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.497+0000 7f9d61600640 1 --2- 172.21.15.28:0/2428226240 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d4c0a3d20 0x7f9d4c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.497+0000 7f9d69200640 1 --2- 172.21.15.28:0/4137837819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fdb90 0x560ec0307d20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9d58009930 tx=0x7f9d58005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52679d12d7557b9e server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.498+0000 7f9d61600640 1 -- 172.21.15.28:0/2428226240 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7f9d5c0d0df0 con 0x7f9d6410d500 2024-04-24T08:20:29.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.498+0000 7f9d63e00640 1 --2- 172.21.15.28:0/2428226240 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d4c0a3d20 0x7f9d4c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.498+0000 7f9d56000640 1 -- 172.21.15.28:0/4137837819 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d58035040 con 0x560ec02fdb90 2024-04-24T08:20:29.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.498+0000 7f9d56000640 1 -- 172.21.15.28:0/4137837819 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d58005520 con 0x560ec02fdb90 2024-04-24T08:20:29.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.498+0000 7f9d56000640 1 -- 172.21.15.28:0/4137837819 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d58004e10 con 0x560ec02fdb90 2024-04-24T08:20:29.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.498+0000 7f9d6ea06740 1 -- 172.21.15.28:0/4137837819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fdb90 msgr2=0x560ec0307d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.505 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.498+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/4137837819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fdb90 0x560ec0307d20 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f9d58009930 tx=0x7f9d58005fd0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.498+0000 7f9d63e00640 1 --2- 172.21.15.28:0/2428226240 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d4c0a3d20 0x7f9d4c0a61e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x560ec02fa720 tx=0x7f9d50009550 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.499+0000 7f9d6ea06740 1 -- 172.21.15.28:0/4137837819 shutdown_connections 2024-04-24T08:20:29.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.499+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/4137837819 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec0308260 0x560ec030a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.499+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/4137837819 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fdb90 0x560ec0307d20 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.499+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/4137837819 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec02fd270 0x560ec02fd650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.506 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.499+0000 7f9d6ea06740 1 -- 172.21.15.28:0/4137837819 >> 172.21.15.28:0/4137837819 conn(0x560ec02f9590 msgr2=0x560ec02f9970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:29.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.499+0000 7f9d6ea06740 1 -- 172.21.15.28:0/4137837819 shutdown_connections 2024-04-24T08:20:29.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.499+0000 7f9d6ea06740 1 -- 172.21.15.28:0/4137837819 wait complete. 2024-04-24T08:20:29.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.499+0000 7f9d6ea06740 1 -- start start 2024-04-24T08:20:29.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.499+0000 7f9d6ea06740 1 Processor -- start 2024-04-24T08:20:29.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.500+0000 7f9d6ea06740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fd270 0x560ec03051b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.500+0000 7f9d6ea06740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fdb90 0x560ec03056f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.507 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.500+0000 7f9d6ea06740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec0308260 0x560ec0305c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.500+0000 7f9d6ea06740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x560ec030aff0 con 0x560ec02fd270 2024-04-24T08:20:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.500+0000 7f9d6ea06740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x560ec021c660 con 0x560ec02fdb90 2024-04-24T08:20:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.500+0000 7f9d6ea06740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x560ec030b170 con 0x560ec0308260 2024-04-24T08:20:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.500+0000 7f9d6ea06740 10 client.0 _ll_register_callbacks cb 0x560ec020aaa0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.500+0000 7f9d69200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fd270 0x560ec03051b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.500+0000 7f9d69200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fd270 0x560ec03051b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37404/0 (socket says 172.21.15.28:37404) 2024-04-24T08:20:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.500+0000 7f9d69200640 1 -- 172.21.15.28:0/749524881 learned_addr learned my addr 172.21.15.28:0/749524881 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:29.508 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.500+0000 7f9d63e00640 1 --2- 172.21.15.28:0/749524881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fdb90 0x560ec03056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.500+0000 7f9d69c00640 1 --2- 172.21.15.28:0/749524881 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec0308260 0x560ec0305c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.501+0000 7f9d69200640 1 -- 172.21.15.28:0/749524881 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec0308260 msgr2=0x560ec0305c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.501+0000 7f9d69200640 1 --2- 172.21.15.28:0/749524881 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec0308260 0x560ec0305c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.501+0000 7f9d69200640 1 -- 172.21.15.28:0/749524881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fdb90 msgr2=0x560ec03056f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.501+0000 7f9d69200640 1 --2- 172.21.15.28:0/749524881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fdb90 0x560ec03056f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.501+0000 7f9d69200640 1 -- 172.21.15.28:0/749524881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ec009df70 con 0x560ec02fd270 2024-04-24T08:20:29.509 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.501+0000 7f9d69200640 1 --2- 172.21.15.28:0/749524881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fd270 0x560ec03051b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9d58004370 tx=0x7f9d580043a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=710da92b2b85a3a3 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.501+0000 7f9d4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:20:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.501+0000 7f9d4aa00640 1 -- 172.21.15.28:0/749524881 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d58052060 con 0x560ec02fd270 2024-04-24T08:20:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.501+0000 7f9d4aa00640 1 -- 172.21.15.28:0/749524881 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7f9d58004da0 con 0x560ec02fd270 2024-04-24T08:20:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.501+0000 7f9d6ea06740 1 -- 172.21.15.28:0/749524881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d58005520 con 0x560ec02fd270 2024-04-24T08:20:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.502+0000 7f9d4aa00640 1 -- 172.21.15.28:0/749524881 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9d58035070 con 0x560ec02fd270 2024-04-24T08:20:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.502+0000 7f9d6ea06740 10 client.16767 fetch_fsmap learned FSMap version 26 2024-04-24T08:20:29.510 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.502+0000 7f9d6ea06740 1 -- 172.21.15.28:0/749524881 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560ebff35170 con 0x560ec02fd270 2024-04-24T08:20:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.502+0000 7f9d4aa00640 1 -- 172.21.15.28:0/749524881 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d58005520 con 0x560ec02fd270 2024-04-24T08:20:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.502+0000 7f9d4aa00640 1 -- 172.21.15.28:0/749524881 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9d580039d0 con 0x560ec02fd270 2024-04-24T08:20:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.502+0000 7f9d6ea06740 10 client.16767 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:20:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.502+0000 7f9d6ea06740 10 client.16767 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:20:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.502+0000 7f9d6ea06740 20 client.16767 populate_metadata read hostname 'smithi028' 2024-04-24T08:20:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.502+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/749524881 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x560ec0318420 0x560ec031a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.511 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.502+0000 7f9d6ea06740 4 client.16767 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:20:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.502+0000 7f9d6ea06740 1 -- 172.21.15.28:0/749524881 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560ec0270890 con 0x560ec0318420 2024-04-24T08:20:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.503+0000 7f9d63e00640 1 --2- 172.21.15.28:0/749524881 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x560ec0318420 0x560ec031a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.503+0000 7f9d63e00640 1 --2- 172.21.15.28:0/749524881 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x560ec0318420 0x560ec031a810 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dceb0a6d224a7f40 server_cookie=9a740a6786e06264 in_seq=0 out_seq=0 2024-04-24T08:20:29.512 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.503+0000 7f9d4aa00640 10 client.16767 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:20:29.513 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.504+0000 7f9d4aa00640 1 -- 172.21.15.28:0/749524881 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560ec0270890 con 0x560ec0318420 2024-04-24T08:20:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.504+0000 7f9d4aa00640 10 client.16767 handle_command_reply: tid=0 2024-04-24T08:20:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.504+0000 7f9d6ea06740 1 client.16767 shutdown 2024-04-24T08:20:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.504+0000 7f9d6ea06740 1 -- 172.21.15.28:0/749524881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fd270 msgr2=0x560ec03051b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.504+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/749524881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fd270 0x560ec03051b0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f9d58004370 tx=0x7f9d580043a0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.505+0000 7f9d6ea06740 1 -- 172.21.15.28:0/749524881 shutdown_connections 2024-04-24T08:20:29.514 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.505+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/749524881 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x560ec0318420 0x560ec031a810 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.505+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/749524881 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec0308260 0x560ec0305c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.505+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/749524881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fdb90 0x560ec03056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.505+0000 7f9d4aa00640 0 client.16767 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:20:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.505+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/749524881 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fd270 0x560ec03051b0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.515 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.505+0000 7f9d6ea06740 1 -- 172.21.15.28:0/749524881 >> 172.21.15.28:0/749524881 conn(0x560ec02f9590 msgr2=0x560ec02ff390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.505+0000 7f9d6ea06740 1 -- 172.21.15.28:0/749524881 shutdown_connections 2024-04-24T08:20:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.505+0000 7f9d6ea06740 1 -- 172.21.15.28:0/749524881 wait complete. 2024-04-24T08:20:29.516 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.505+0000 7f9d6ea06740 20 client.16767 trim_cache size 0 max 16384 2024-04-24T08:20:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.526+0000 7f9d6ea06740 1 Processor -- start 2024-04-24T08:20:29.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.526+0000 7f9d6ea06740 1 -- start start 2024-04-24T08:20:29.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.527+0000 7f9d6ea06740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fd270 0x560ec0313f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.527+0000 7f9d6ea06740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fdb90 0x560ec02ffa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.527+0000 7f9d6ea06740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec0308260 0x560ec02f8570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.527+0000 7f9d6ea06740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d641b6ab0 con 0x560ec02fdb90 2024-04-24T08:20:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.527+0000 7f9d6ea06740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9d641b4380 con 0x560ec02fd270 2024-04-24T08:20:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.527+0000 7f9d6ea06740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9d641b21f0 con 0x560ec0308260 2024-04-24T08:20:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.527+0000 7f9d63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fdb90 0x560ec02ffa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.527+0000 7f9d63e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fdb90 0x560ec02ffa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37418/0 (socket says 172.21.15.28:37418) 2024-04-24T08:20:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.527+0000 7f9d63e00640 1 -- 172.21.15.28:0/2959940604 learned_addr learned my addr 172.21.15.28:0/2959940604 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:29.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.527+0000 7f9d69c00640 1 --2- 172.21.15.28:0/2959940604 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec0308260 0x560ec02f8570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.527+0000 7f9d69200640 1 --2- 172.21.15.28:0/2959940604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fd270 0x560ec0313f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.528+0000 7f9d63e00640 1 -- 172.21.15.28:0/2959940604 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec0308260 msgr2=0x560ec02f8570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.528+0000 7f9d63e00640 1 --2- 172.21.15.28:0/2959940604 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec0308260 0x560ec02f8570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.528+0000 7f9d63e00640 1 -- 172.21.15.28:0/2959940604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fd270 msgr2=0x560ec0313f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.528+0000 7f9d63e00640 1 --2- 172.21.15.28:0/2959940604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fd270 0x560ec0313f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.528+0000 7f9d63e00640 1 -- 172.21.15.28:0/2959940604 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560ec02fd9c0 con 0x560ec02fdb90 2024-04-24T08:20:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.528+0000 7f9d69200640 1 --2- 172.21.15.28:0/2959940604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fd270 0x560ec0313f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:29.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.528+0000 7f9d63e00640 1 --2- 172.21.15.28:0/2959940604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fdb90 0x560ec02ffa10 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9d50009620 tx=0x7f9d500046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7b2eb35a817693b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.528+0000 7f9d55600640 1 -- 172.21.15.28:0/2959940604 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d50030070 con 0x560ec02fdb90 2024-04-24T08:20:29.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.528+0000 7f9d55600640 1 -- 172.21.15.28:0/2959940604 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d5002b910 con 0x560ec02fdb90 2024-04-24T08:20:29.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.528+0000 7f9d55600640 1 -- 172.21.15.28:0/2959940604 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d50003e50 con 0x560ec02fdb90 2024-04-24T08:20:29.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.529+0000 7f9d6ea06740 1 -- 172.21.15.28:0/2959940604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fdb90 msgr2=0x560ec02ffa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.529+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/2959940604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fdb90 0x560ec02ffa10 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9d50009620 tx=0x7f9d500046c0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.529+0000 7f9d6ea06740 1 -- 172.21.15.28:0/2959940604 shutdown_connections 2024-04-24T08:20:29.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.529+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/2959940604 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fdb90 0x560ec02ffa10 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.529+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/2959940604 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec0308260 0x560ec02f8570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.529+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/2959940604 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec02fd270 0x560ec0313f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.535 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.529+0000 7f9d6ea06740 1 -- 172.21.15.28:0/2959940604 >> 172.21.15.28:0/2959940604 conn(0x560ec02f9590 msgr2=0x560ec02f7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:29.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.529+0000 7f9d6ea06740 1 -- 172.21.15.28:0/2959940604 shutdown_connections 2024-04-24T08:20:29.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.529+0000 7f9d6ea06740 1 -- 172.21.15.28:0/2959940604 wait complete. 2024-04-24T08:20:29.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.530+0000 7f9d6ea06740 1 -- start start 2024-04-24T08:20:29.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.530+0000 7f9d6ea06740 1 Processor -- start 2024-04-24T08:20:29.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.530+0000 7f9d6ea06740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fd270 0x560ec030c0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.530+0000 7f9d6ea06740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec02fdb90 0x560ec030c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.530+0000 7f9d6ea06740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec0308260 0x560ec030cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.531+0000 7f9d6ea06740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f9d641b2840 con 0x560ec02fd270 2024-04-24T08:20:29.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.531+0000 7f9d6ea06740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9d641b3cb0 con 0x560ec0308260 2024-04-24T08:20:29.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.531+0000 7f9d6ea06740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9d641b2f10 con 0x560ec02fdb90 2024-04-24T08:20:29.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.531+0000 7f9d6ea06740 10 client.0 _ll_register_callbacks cb 0x7f9d50009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:29.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.531+0000 7f9d69200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fd270 0x560ec030c0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.531+0000 7f9d69200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fd270 0x560ec030c0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:37432/0 (socket says 172.21.15.28:37432) 2024-04-24T08:20:29.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.531+0000 7f9d69200640 1 -- 172.21.15.28:0/493812425 learned_addr learned my addr 172.21.15.28:0/493812425 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:29.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.531+0000 7f9d69c00640 1 --2- 172.21.15.28:0/493812425 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec0308260 0x560ec030cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.531+0000 7f9d63e00640 1 --2- 172.21.15.28:0/493812425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec02fdb90 0x560ec030c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.531+0000 7f9d69200640 1 -- 172.21.15.28:0/493812425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec02fdb90 msgr2=0x560ec030c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.531+0000 7f9d69200640 1 --2- 172.21.15.28:0/493812425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec02fdb90 0x560ec030c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.531+0000 7f9d69200640 1 -- 172.21.15.28:0/493812425 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec0308260 msgr2=0x560ec030cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.531+0000 7f9d69200640 1 --2- 172.21.15.28:0/493812425 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec0308260 0x560ec030cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.531+0000 7f9d69200640 1 -- 172.21.15.28:0/493812425 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560ec037d8c0 con 0x560ec02fd270 2024-04-24T08:20:29.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.531+0000 7f9d63e00640 1 --2- 172.21.15.28:0/493812425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec02fdb90 0x560ec030c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:29.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.531+0000 7f9d69c00640 1 --2- 172.21.15.28:0/493812425 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec0308260 0x560ec030cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:29.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.532+0000 7f9d69200640 1 --2- 172.21.15.28:0/493812425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fd270 0x560ec030c0c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f9d5802aa30 tx=0x7f9d58003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=449f18bebfa7ef8d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.532+0000 7f9d4aa00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:20:29.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.532+0000 7f9d4aa00640 1 -- 172.21.15.28:0/493812425 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d58052040 con 0x560ec02fd270 2024-04-24T08:20:29.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.532+0000 7f9d4aa00640 1 -- 172.21.15.28:0/493812425 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7f9d58031b50 con 0x560ec02fd270 2024-04-24T08:20:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.532+0000 7f9d6ea06740 1 -- 172.21.15.28:0/493812425 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9d5002b910 con 0x560ec02fd270 2024-04-24T08:20:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.532+0000 7f9d4aa00640 1 -- 172.21.15.28:0/493812425 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f9d58049070 con 0x560ec02fd270 2024-04-24T08:20:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.532+0000 7f9d6ea06740 10 client.16773 fetch_fsmap learned FSMap version 26 2024-04-24T08:20:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.532+0000 7f9d6ea06740 1 -- 172.21.15.28:0/493812425 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560ec037ebe0 con 0x560ec02fd270 2024-04-24T08:20:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.532+0000 7f9d4aa00640 1 -- 172.21.15.28:0/493812425 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d5002b910 con 0x560ec02fd270 2024-04-24T08:20:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.533+0000 7f9d4aa00640 1 -- 172.21.15.28:0/493812425 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f9d5805c020 con 0x560ec02fd270 2024-04-24T08:20:29.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.533+0000 7f9d6ea06740 10 client.16773 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:20:29.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.533+0000 7f9d6ea06740 10 client.16773 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:20:29.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.533+0000 7f9d6ea06740 20 client.16773 populate_metadata read hostname 'smithi028' 2024-04-24T08:20:29.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.533+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/493812425 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x560ec037f210 0x560ec0381630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.533+0000 7f9d6ea06740 4 client.16773 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-04-24T08:20:29.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.533+0000 7f9d6ea06740 1 -- 172.21.15.28:0/493812425 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x560ec026a700 con 0x560ec037f210 2024-04-24T08:20:29.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.533+0000 7f9d63e00640 1 --2- 172.21.15.28:0/493812425 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x560ec037f210 0x560ec0381630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.534+0000 7f9d63e00640 1 --2- 172.21.15.28:0/493812425 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x560ec037f210 0x560ec0381630 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df596285cd448253 server_cookie=ea8e62f5a9cffe32 in_seq=0 out_seq=0 2024-04-24T08:20:29.541 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.534+0000 7f9d4aa00640 10 client.16773 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:20:29.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.535+0000 7f9d4aa00640 1 -- 172.21.15.28:0/493812425 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x560ec026a700 con 0x560ec037f210 2024-04-24T08:20:29.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.535+0000 7f9d4aa00640 10 client.16773 handle_command_reply: tid=0 2024-04-24T08:20:29.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.535+0000 7f9d6ea06740 1 client.16773 shutdown 2024-04-24T08:20:29.543 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.535+0000 7f9d6ea06740 1 -- 172.21.15.28:0/493812425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fd270 msgr2=0x560ec030c0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.535+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/493812425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fd270 0x560ec030c0c0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f9d5802aa30 tx=0x7f9d58003f90 comp rx=0 tx=0).stop 2024-04-24T08:20:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.535+0000 7f9d6ea06740 1 -- 172.21.15.28:0/493812425 shutdown_connections 2024-04-24T08:20:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.535+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/493812425 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x560ec037f210 0x560ec0381630 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.535+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/493812425 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x560ec0308260 0x560ec030cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.535+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/493812425 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x560ec02fdb90 0x560ec030c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.535+0000 7f9d6ea06740 1 --2- 172.21.15.28:0/493812425 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x560ec02fd270 0x560ec030c0c0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.536+0000 7f9d6ea06740 1 -- 172.21.15.28:0/493812425 >> 172.21.15.28:0/493812425 conn(0x560ec02f9590 msgr2=0x560ec03059e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:29.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.536+0000 7f9d4aa00640 0 client.16773 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:20:29.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.536+0000 7f9d6ea06740 1 -- 172.21.15.28:0/493812425 shutdown_connections 2024-04-24T08:20:29.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.536+0000 7f9d6ea06740 1 -- 172.21.15.28:0/493812425 wait complete. 2024-04-24T08:20:29.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.536+0000 7f9d6ea06740 20 client.16773 trim_cache size 0 max 16384 2024-04-24T08:20:29.545 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:20:29.545 INFO:teuthology.orchestra.run.smithi028.stdout: "return_code": 0, 2024-04-24T08:20:29.546 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_tag": "21b9f7a4-1da2-4d56-9950-60ec65f23bca", 2024-04-24T08:20:29.546 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": "asynchronous" 2024-04-24T08:20:29.546 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:20:29.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.537+0000 7f9d6a600640 1 -- 172.21.15.28:0/2428226240 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d4c0a3d20 msgr2=0x7f9d4c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.537+0000 7f9d6a600640 1 --2- 172.21.15.28:0/2428226240 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d4c0a3d20 0x7f9d4c0a61e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x560ec02fa720 tx=0x7f9d50009550 comp rx=0 tx=0).stop 2024-04-24T08:20:29.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.537+0000 7f9d6a600640 1 -- 172.21.15.28:0/2428226240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d6410d500 msgr2=0x7f9d641a35f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.537+0000 7f9d6a600640 1 --2- 172.21.15.28:0/2428226240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d6410d500 0x7f9d641a35f0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f9d5c00ebc0 tx=0x7f9d5c00c6a0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.547 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.537+0000 7f9d6a600640 1 -- 172.21.15.28:0/2428226240 shutdown_connections 2024-04-24T08:20:29.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.537+0000 7f9d6a600640 1 --2- 172.21.15.28:0/2428226240 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f9d4c0a3d20 0x7f9d4c0a61e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.537+0000 7f9d6a600640 1 --2- 172.21.15.28:0/2428226240 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9d64108f40 0x7f9d6419f2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.537+0000 7f9d6a600640 1 --2- 172.21.15.28:0/2428226240 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d6410d500 0x7f9d641a35f0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.537+0000 7f9d6a600640 1 --2- 172.21.15.28:0/2428226240 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9d64108580 0x7f9d6419ed80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.537+0000 7f9d6a600640 1 -- 172.21.15.28:0/2428226240 >> 172.21.15.28:0/2428226240 conn(0x7f9d64077e20 msgr2=0x7f9d64078340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:29.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.537+0000 7f9d6a600640 1 -- 172.21.15.28:0/2428226240 shutdown_connections 2024-04-24T08:20:29.548 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.537+0000 7f9d6a600640 1 -- 172.21.15.28:0/2428226240 wait complete. 2024-04-24T08:20:29.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:20:29.561 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:20:29.764 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.764+0000 7fde6f800640 1 -- 172.21.15.28:0/1581228466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde6806a870 msgr2=0x7fde6806ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.764+0000 7fde6f800640 1 --2- 172.21.15.28:0/1581228466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde6806a870 0x7fde6806ac50 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fde5c009900 tx=0x7fde5c02a3f0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.764+0000 7fde6f800640 1 -- 172.21.15.28:0/1581228466 shutdown_connections 2024-04-24T08:20:29.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.764+0000 7fde6f800640 1 --2- 172.21.15.28:0/1581228466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde68107330 0x7fde6810eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.764+0000 7fde6f800640 1 --2- 172.21.15.28:0/1581228466 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde6806b220 0x7fde68106c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.764+0000 7fde6f800640 1 --2- 172.21.15.28:0/1581228466 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde6806a870 0x7fde6806ac50 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.765 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.764+0000 7fde6f800640 1 -- 172.21.15.28:0/1581228466 >> 172.21.15.28:0/1581228466 conn(0x7fde68076d00 msgr2=0x7fde68077110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:29.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.764+0000 7fde6f800640 1 -- 172.21.15.28:0/1581228466 shutdown_connections 2024-04-24T08:20:29.766 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.764+0000 7fde6f800640 1 -- 172.21.15.28:0/1581228466 wait complete. 2024-04-24T08:20:29.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.765+0000 7fde6f800640 1 Processor -- start 2024-04-24T08:20:29.767 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.766+0000 7fde6f800640 1 -- start start 2024-04-24T08:20:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.766+0000 7fde6f800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde6806a870 0x7fde681a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.766+0000 7fde6f800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde68107330 0x7fde681a0c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.766+0000 7fde6f800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde6819a8f0 0x7fde6819ada0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.766+0000 7fde6f800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde68112430 con 0x7fde68107330 2024-04-24T08:20:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.766+0000 7fde6f800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fde681122b0 con 0x7fde6819a8f0 2024-04-24T08:20:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.766+0000 7fde6f800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fde681125b0 con 0x7fde6806a870 2024-04-24T08:20:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.767+0000 7fde6ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde6806a870 0x7fde681a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.768 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.767+0000 7fde6ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde6806a870 0x7fde681a06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:43486/0 (socket says 172.21.15.28:43486) 2024-04-24T08:20:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.767+0000 7fde6ce00640 1 -- 172.21.15.28:0/2682362078 learned_addr learned my addr 172.21.15.28:0/2682362078 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.767+0000 7fde67e00640 1 --2- 172.21.15.28:0/2682362078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde68107330 0x7fde681a0c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.767+0000 7fde6d800640 1 --2- 172.21.15.28:0/2682362078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde6819a8f0 0x7fde6819ada0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.767+0000 7fde6ce00640 1 -- 172.21.15.28:0/2682362078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde6819a8f0 msgr2=0x7fde6819ada0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.767+0000 7fde6ce00640 1 --2- 172.21.15.28:0/2682362078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde6819a8f0 0x7fde6819ada0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.767+0000 7fde6ce00640 1 -- 172.21.15.28:0/2682362078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde68107330 msgr2=0x7fde681a0c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.767+0000 7fde6ce00640 1 --2- 172.21.15.28:0/2682362078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde68107330 0x7fde681a0c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.769 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.767+0000 7fde6ce00640 1 -- 172.21.15.28:0/2682362078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fde6819b410 con 0x7fde6806a870 2024-04-24T08:20:29.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.767+0000 7fde6ce00640 1 --2- 172.21.15.28:0/2682362078 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde6806a870 0x7fde681a06f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fde5c004770 tx=0x7fde5c003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.770 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.768+0000 7fde65600640 1 -- 172.21.15.28:0/2682362078 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c03f070 con 0x7fde6806a870 2024-04-24T08:20:29.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.768+0000 7fde65600640 1 -- 172.21.15.28:0/2682362078 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde5c031ca0 con 0x7fde6806a870 2024-04-24T08:20:29.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.768+0000 7fde65600640 1 -- 172.21.15.28:0/2682362078 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c035030 con 0x7fde6806a870 2024-04-24T08:20:29.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.768+0000 7fde6f800640 1 -- 172.21.15.28:0/2682362078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fde6819b6a0 con 0x7fde6806a870 2024-04-24T08:20:29.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.768+0000 7fde6f800640 1 -- 172.21.15.28:0/2682362078 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fde681a77f0 con 0x7fde6806a870 2024-04-24T08:20:29.771 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.771+0000 7fde7263a740 1 Processor -- start 2024-04-24T08:20:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.771+0000 7fde7263a740 1 -- start start 2024-04-24T08:20:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.771+0000 7fde65600640 1 -- 172.21.15.28:0/2682362078 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fde5c042070 con 0x7fde6806a870 2024-04-24T08:20:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.772+0000 7fde7263a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabb7860 0x5623fabb7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.772+0000 7fde7263a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 0x5623fabc2310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.773 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.772+0000 7fde7263a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabc2850 0x5623fabc4c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.772+0000 7fde7263a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5623fab1d1f0 con 0x5623fabc2850 2024-04-24T08:20:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.772+0000 7fde7263a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5623faac3350 con 0x5623fabb8180 2024-04-24T08:20:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.772+0000 7fde7263a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5623fab191b0 con 0x5623fabb7860 2024-04-24T08:20:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.772+0000 7fde67e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabc2850 0x5623fabc4c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.772+0000 7fde6ce00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 0x5623fabc2310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.772+0000 7fde67e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabc2850 0x5623fabc4c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:57990/0 (socket says 172.21.15.28:57990) 2024-04-24T08:20:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.772+0000 7fde67e00640 1 -- 172.21.15.28:0/3880086873 learned_addr learned my addr 172.21.15.28:0/3880086873 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:29.774 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.772+0000 7fde6d800640 1 --2- 172.21.15.28:0/3880086873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabb7860 0x5623fabb7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.773+0000 7fde67e00640 1 -- 172.21.15.28:0/3880086873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabb7860 msgr2=0x5623fabb7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.773+0000 7fde67e00640 1 --2- 172.21.15.28:0/3880086873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabb7860 0x5623fabb7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.773+0000 7fde67e00640 1 -- 172.21.15.28:0/3880086873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 msgr2=0x5623fabc2310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.773+0000 7fde67e00640 1 --2- 172.21.15.28:0/3880086873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 0x5623fabc2310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.773+0000 7fde67e00640 1 -- 172.21.15.28:0/3880086873 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623fab17f50 con 0x5623fabc2850 2024-04-24T08:20:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.773+0000 7fde6d800640 1 --2- 172.21.15.28:0/3880086873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabb7860 0x5623fabb7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:29.775 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.773+0000 7fde6ce00640 1 --2- 172.21.15.28:0/3880086873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 0x5623fabc2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:29.779 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.773+0000 7fde67e00640 1 --2- 172.21.15.28:0/3880086873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabc2850 0x5623fabc4c40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fde54009c00 tx=0x7fde54011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce443c1e8a89dd6 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.773+0000 7fde5a000640 1 -- 172.21.15.28:0/3880086873 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde54004230 con 0x5623fabc2850 2024-04-24T08:20:29.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.773+0000 7fde5a000640 1 -- 172.21.15.28:0/3880086873 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde540043d0 con 0x5623fabc2850 2024-04-24T08:20:29.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.773+0000 7fde5a000640 1 -- 172.21.15.28:0/3880086873 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde54004d90 con 0x5623fabc2850 2024-04-24T08:20:29.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.774+0000 7fde7263a740 1 -- 172.21.15.28:0/3880086873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabc2850 msgr2=0x5623fabc4c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.774+0000 7fde7263a740 1 --2- 172.21.15.28:0/3880086873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabc2850 0x5623fabc4c40 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fde54009c00 tx=0x7fde54011810 comp rx=0 tx=0).stop 2024-04-24T08:20:29.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.774+0000 7fde65600640 1 --2- 172.21.15.28:0/2682362078 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fde500a3d20 0x7fde500a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.774+0000 7fde7263a740 1 -- 172.21.15.28:0/3880086873 shutdown_connections 2024-04-24T08:20:29.780 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.774+0000 7fde7263a740 1 --2- 172.21.15.28:0/3880086873 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabc2850 0x5623fabc4c40 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.774+0000 7fde7263a740 1 --2- 172.21.15.28:0/3880086873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 0x5623fabc2310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.774+0000 7fde7263a740 1 --2- 172.21.15.28:0/3880086873 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabb7860 0x5623fabb7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.774+0000 7fde65600640 1 -- 172.21.15.28:0/2682362078 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7fde5c0f0e50 con 0x7fde6806a870 2024-04-24T08:20:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.774+0000 7fde7263a740 1 -- 172.21.15.28:0/3880086873 >> 172.21.15.28:0/3880086873 conn(0x5623fabb3b80 msgr2=0x5623fabb3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.774+0000 7fde6d800640 1 --2- 172.21.15.28:0/2682362078 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fde500a3d20 0x7fde500a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.774+0000 7fde7263a740 1 -- 172.21.15.28:0/3880086873 shutdown_connections 2024-04-24T08:20:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.775+0000 7fde7263a740 1 -- 172.21.15.28:0/3880086873 wait complete. 2024-04-24T08:20:29.781 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.775+0000 7fde7263a740 1 -- start start 2024-04-24T08:20:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.775+0000 7fde6d800640 1 --2- 172.21.15.28:0/2682362078 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fde500a3d20 0x7fde500a61e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fde60005fd0 tx=0x7fde60009500 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.775+0000 7fde7263a740 1 Processor -- start 2024-04-24T08:20:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.776+0000 7fde7263a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabb7860 0x5623fabbf750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.776+0000 7fde7263a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 0x5623fabbfc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.776+0000 7fde7263a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabc2850 0x5623fabc01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.776+0000 7fde7263a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5623fabc5640 con 0x5623fabc2850 2024-04-24T08:20:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.776+0000 7fde7263a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5623faad6690 con 0x5623fabb8180 2024-04-24T08:20:29.782 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.776+0000 7fde7263a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5623fabc57c0 con 0x5623fabb7860 2024-04-24T08:20:29.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.776+0000 7fde7263a740 10 client.0 _ll_register_callbacks cb 0x5623fab12650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:29.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.776+0000 7fde67e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 0x5623fabbfc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.776+0000 7fde67e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 0x5623fabbfc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33162/0 (socket says 172.21.15.28:33162) 2024-04-24T08:20:29.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.776+0000 7fde67e00640 1 -- 172.21.15.28:0/549592022 learned_addr learned my addr 172.21.15.28:0/549592022 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:29.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.776+0000 7fde6d800640 1 --2- 172.21.15.28:0/549592022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabc2850 0x5623fabc01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.776+0000 7fde6ce00640 1 --2- 172.21.15.28:0/549592022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabb7860 0x5623fabbf750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.777+0000 7fde67e00640 1 -- 172.21.15.28:0/549592022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabb7860 msgr2=0x5623fabbf750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.783 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.777+0000 7fde67e00640 1 --2- 172.21.15.28:0/549592022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabb7860 0x5623fabbf750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.777+0000 7fde67e00640 1 -- 172.21.15.28:0/549592022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabc2850 msgr2=0x5623fabc01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.777+0000 7fde67e00640 1 --2- 172.21.15.28:0/549592022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabc2850 0x5623fabc01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.777+0000 7fde67e00640 1 -- 172.21.15.28:0/549592022 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623faad5920 con 0x5623fabb8180 2024-04-24T08:20:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.777+0000 7fde6d800640 1 --2- 172.21.15.28:0/549592022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabc2850 0x5623fabc01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.777+0000 7fde67e00640 1 --2- 172.21.15.28:0/549592022 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 0x5623fabbfc90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fde54011d10 tx=0x7fde54019a50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b9a2691dc4bf8847 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.777+0000 7fde4ea00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:20:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.777+0000 7fde4ea00640 1 -- 172.21.15.28:0/549592022 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde54028020 con 0x5623fabb8180 2024-04-24T08:20:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.777+0000 7fde7263a740 1 -- 172.21.15.28:0/549592022 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde540043d0 con 0x5623fabb8180 2024-04-24T08:20:29.784 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.777+0000 7fde4ea00640 1 -- 172.21.15.28:0/549592022 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7fde54024420 con 0x5623fabb8180 2024-04-24T08:20:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.778+0000 7fde4ea00640 1 -- 172.21.15.28:0/549592022 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fde54033050 con 0x5623fabb8180 2024-04-24T08:20:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.778+0000 7fde7263a740 10 client.27062 fetch_fsmap learned FSMap version 26 2024-04-24T08:20:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.778+0000 7fde7263a740 1 -- 172.21.15.28:0/549592022 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5623fa8f97d0 con 0x5623fabb8180 2024-04-24T08:20:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.778+0000 7fde4ea00640 1 -- 172.21.15.28:0/549592022 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde540043d0 con 0x5623fabb8180 2024-04-24T08:20:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.778+0000 7fde4ea00640 1 -- 172.21.15.28:0/549592022 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fde54018070 con 0x5623fabb8180 2024-04-24T08:20:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.778+0000 7fde7263a740 10 client.27062 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:20:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.778+0000 7fde7263a740 10 client.27062 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:20:29.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.778+0000 7fde7263a740 20 client.27062 populate_metadata read hostname 'smithi028' 2024-04-24T08:20:29.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.778+0000 7fde7263a740 1 --2- 172.21.15.28:0/549592022 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5623fabd2a70 0x5623fabd4e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.778+0000 7fde7263a740 4 client.27062 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:20:29.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.778+0000 7fde7263a740 1 -- 172.21.15.28:0/549592022 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623fab2ac10 con 0x5623fabd2a70 2024-04-24T08:20:29.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.779+0000 7fde6ce00640 1 --2- 172.21.15.28:0/549592022 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5623fabd2a70 0x5623fabd4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.779+0000 7fde6ce00640 1 --2- 172.21.15.28:0/549592022 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5623fabd2a70 0x5623fabd4e60 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=355a843ebdd7313d server_cookie=8fdae94a9aaaee23 in_seq=0 out_seq=0 2024-04-24T08:20:29.786 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.779+0000 7fde4ea00640 10 client.27062 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:20:29.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.780+0000 7fde4ea00640 1 -- 172.21.15.28:0/549592022 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5623fab2ac10 con 0x5623fabd2a70 2024-04-24T08:20:29.787 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.780+0000 7fde4ea00640 10 client.27062 handle_command_reply: tid=0 2024-04-24T08:20:29.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.780+0000 7fde7263a740 1 client.27062 shutdown 2024-04-24T08:20:29.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.780+0000 7fde7263a740 1 -- 172.21.15.28:0/549592022 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 msgr2=0x5623fabbfc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.780+0000 7fde7263a740 1 --2- 172.21.15.28:0/549592022 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 0x5623fabbfc90 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fde54011d10 tx=0x7fde54019a50 comp rx=0 tx=0).stop 2024-04-24T08:20:29.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.780+0000 7fde7263a740 1 -- 172.21.15.28:0/549592022 shutdown_connections 2024-04-24T08:20:29.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.780+0000 7fde7263a740 1 --2- 172.21.15.28:0/549592022 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5623fabd2a70 0x5623fabd4e60 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.780+0000 7fde7263a740 1 --2- 172.21.15.28:0/549592022 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabc2850 0x5623fabc01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.780+0000 7fde7263a740 1 --2- 172.21.15.28:0/549592022 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 0x5623fabbfc90 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.780+0000 7fde4ea00640 0 client.27062 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:20:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.780+0000 7fde7263a740 1 --2- 172.21.15.28:0/549592022 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabb7860 0x5623fabbf750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.780+0000 7fde7263a740 1 -- 172.21.15.28:0/549592022 >> 172.21.15.28:0/549592022 conn(0x5623fabb3b80 msgr2=0x5623fabb9980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.781+0000 7fde7263a740 1 -- 172.21.15.28:0/549592022 shutdown_connections 2024-04-24T08:20:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.781+0000 7fde7263a740 1 -- 172.21.15.28:0/549592022 wait complete. 2024-04-24T08:20:29.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.781+0000 7fde7263a740 20 client.27062 trim_cache size 0 max 16384 2024-04-24T08:20:29.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.800+0000 7fde7263a740 1 Processor -- start 2024-04-24T08:20:29.801 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.800+0000 7fde7263a740 1 -- start start 2024-04-24T08:20:29.802 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.801+0000 7fde7263a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabb7860 0x5623fabba0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.801+0000 7fde7263a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabb8180 0x5623fabce500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.801+0000 7fde7263a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabc2850 0x5623fabbe8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.801+0000 7fde7263a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde681b2860 con 0x5623fabb8180 2024-04-24T08:20:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.801+0000 7fde7263a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fde681b0130 con 0x5623fabc2850 2024-04-24T08:20:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.801+0000 7fde7263a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fde681adfa0 con 0x5623fabb7860 2024-04-24T08:20:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.801+0000 7fde67e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabb8180 0x5623fabce500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.801+0000 7fde6d800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabc2850 0x5623fabbe8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.801+0000 7fde6ce00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabb7860 0x5623fabba0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.801+0000 7fde67e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabb8180 0x5623fabce500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58008/0 (socket says 172.21.15.28:58008) 2024-04-24T08:20:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.801+0000 7fde67e00640 1 -- 172.21.15.28:0/3878284185 learned_addr learned my addr 172.21.15.28:0/3878284185 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.802+0000 7fde6ce00640 1 -- 172.21.15.28:0/3878284185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabc2850 msgr2=0x5623fabbe8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.802+0000 7fde6ce00640 1 --2- 172.21.15.28:0/3878284185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabc2850 0x5623fabbe8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.802+0000 7fde6ce00640 1 -- 172.21.15.28:0/3878284185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabb8180 msgr2=0x5623fabce500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.802+0000 7fde6ce00640 1 --2- 172.21.15.28:0/3878284185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabb8180 0x5623fabce500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.802+0000 7fde6ce00640 1 -- 172.21.15.28:0/3878284185 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623fa82f3f0 con 0x5623fabb7860 2024-04-24T08:20:29.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.802+0000 7fde6ce00640 1 --2- 172.21.15.28:0/3878284185 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabb7860 0x5623fabba0a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fde5c0ee4f0 tx=0x7fde5c0f4900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fedc07344361479 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.802+0000 7fde59600640 1 -- 172.21.15.28:0/3878284185 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c049040 con 0x5623fabb7860 2024-04-24T08:20:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.802+0000 7fde59600640 1 -- 172.21.15.28:0/3878284185 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde5c0bceb0 con 0x5623fabb7860 2024-04-24T08:20:29.805 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.802+0000 7fde59600640 1 -- 172.21.15.28:0/3878284185 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c0c5a70 con 0x5623fabb7860 2024-04-24T08:20:29.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.803+0000 7fde7263a740 1 -- 172.21.15.28:0/3878284185 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabb7860 msgr2=0x5623fabba0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.803+0000 7fde7263a740 1 --2- 172.21.15.28:0/3878284185 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabb7860 0x5623fabba0a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fde5c0ee4f0 tx=0x7fde5c0f4900 comp rx=0 tx=0).stop 2024-04-24T08:20:29.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.803+0000 7fde7263a740 1 -- 172.21.15.28:0/3878284185 shutdown_connections 2024-04-24T08:20:29.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.803+0000 7fde7263a740 1 --2- 172.21.15.28:0/3878284185 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabb8180 0x5623fabce500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.803+0000 7fde7263a740 1 --2- 172.21.15.28:0/3878284185 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabc2850 0x5623fabbe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.803+0000 7fde7263a740 1 --2- 172.21.15.28:0/3878284185 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabb7860 0x5623fabba0a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.803+0000 7fde7263a740 1 -- 172.21.15.28:0/3878284185 >> 172.21.15.28:0/3878284185 conn(0x5623fabb3b80 msgr2=0x5623fabb2390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:29.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.803+0000 7fde7263a740 1 -- 172.21.15.28:0/3878284185 shutdown_connections 2024-04-24T08:20:29.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.803+0000 7fde7263a740 1 -- 172.21.15.28:0/3878284185 wait complete. 2024-04-24T08:20:29.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.804+0000 7fde7263a740 1 -- start start 2024-04-24T08:20:29.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.804+0000 7fde7263a740 1 Processor -- start 2024-04-24T08:20:29.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.804+0000 7fde7263a740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabb7860 0x5623fabcab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.804+0000 7fde7263a740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 0x5623fabc5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.804+0000 7fde7263a740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabc2850 0x5623fabc64b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.810 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.804+0000 7fde7263a740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fde681ae5f0 con 0x5623fabb7860 2024-04-24T08:20:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.804+0000 7fde7263a740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fde681afa60 con 0x5623fabb8180 2024-04-24T08:20:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.804+0000 7fde7263a740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fde681aecc0 con 0x5623fabc2850 2024-04-24T08:20:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.804+0000 7fde7263a740 10 client.0 _ll_register_callbacks cb 0x7fde5c0f47d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:20:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.805+0000 7fde6ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabb7860 0x5623fabcab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.805+0000 7fde67e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 0x5623fabc5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.805+0000 7fde6ce00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabb7860 0x5623fabcab00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:58012/0 (socket says 172.21.15.28:58012) 2024-04-24T08:20:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.805+0000 7fde6ce00640 1 -- 172.21.15.28:0/1941348600 learned_addr learned my addr 172.21.15.28:0/1941348600 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:29.811 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.805+0000 7fde6d800640 1 --2- 172.21.15.28:0/1941348600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabc2850 0x5623fabc64b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.805+0000 7fde6ce00640 1 -- 172.21.15.28:0/1941348600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabc2850 msgr2=0x5623fabc64b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.805+0000 7fde6ce00640 1 --2- 172.21.15.28:0/1941348600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabc2850 0x5623fabc64b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.805+0000 7fde6ce00640 1 -- 172.21.15.28:0/1941348600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 msgr2=0x5623fabc5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:20:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.805+0000 7fde6ce00640 1 --2- 172.21.15.28:0/1941348600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 0x5623fabc5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.805+0000 7fde6ce00640 1 -- 172.21.15.28:0/1941348600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623fa7ef0f0 con 0x5623fabb7860 2024-04-24T08:20:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.805+0000 7fde67e00640 1 --2- 172.21.15.28:0/1941348600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 0x5623fabc5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:20:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.805+0000 7fde6ce00640 1 --2- 172.21.15.28:0/1941348600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabb7860 0x5623fabcab00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fde5c0dd010 tx=0x7fde5c031f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15e3b4a7d94488ab server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:29.812 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.805+0000 7fde4ea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:20:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.806+0000 7fde4ea00640 1 -- 172.21.15.28:0/1941348600 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fde5c049040 con 0x5623fabb7860 2024-04-24T08:20:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.806+0000 7fde4ea00640 1 -- 172.21.15.28:0/1941348600 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7fde5c0bb1b0 con 0x5623fabb7860 2024-04-24T08:20:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.806+0000 7fde7263a740 1 -- 172.21.15.28:0/1941348600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fde5c0bceb0 con 0x5623fabb7860 2024-04-24T08:20:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.806+0000 7fde4ea00640 1 -- 172.21.15.28:0/1941348600 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fde5c0bc9f0 con 0x5623fabb7860 2024-04-24T08:20:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.806+0000 7fde7263a740 10 client.16791 fetch_fsmap learned FSMap version 26 2024-04-24T08:20:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.806+0000 7fde7263a740 1 -- 172.21.15.28:0/1941348600 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5623faae2810 con 0x5623fabb7860 2024-04-24T08:20:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.806+0000 7fde4ea00640 1 -- 172.21.15.28:0/1941348600 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fde5c0bceb0 con 0x5623fabb7860 2024-04-24T08:20:29.813 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.806+0000 7fde4ea00640 1 -- 172.21.15.28:0/1941348600 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7fde5c073030 con 0x5623fabb7860 2024-04-24T08:20:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.806+0000 7fde7263a740 10 client.16791 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:20:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.806+0000 7fde7263a740 10 client.16791 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:20:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.806+0000 7fde7263a740 20 client.16791 populate_metadata read hostname 'smithi028' 2024-04-24T08:20:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.806+0000 7fde7263a740 1 --2- 172.21.15.28:0/1941348600 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5623fac415b0 0x5623fac439d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.806+0000 7fde7263a740 4 client.16791 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:20:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.806+0000 7fde7263a740 1 -- 172.21.15.28:0/1941348600 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5623fab24a80 con 0x5623fac415b0 2024-04-24T08:20:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.807+0000 7fde67e00640 1 --2- 172.21.15.28:0/1941348600 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5623fac415b0 0x5623fac439d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:29.814 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.807+0000 7fde67e00640 1 --2- 172.21.15.28:0/1941348600 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5623fac415b0 0x5623fac439d0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9c2e8f6378d7c902 server_cookie=38c4c7539ae2162a in_seq=0 out_seq=0 2024-04-24T08:20:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.807+0000 7fde4ea00640 10 client.16791 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:20:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.807+0000 7fde4ea00640 1 -- 172.21.15.28:0/1941348600 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5623fab24a80 con 0x5623fac415b0 2024-04-24T08:20:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.807+0000 7fde4ea00640 10 client.16791 handle_command_reply: tid=0 2024-04-24T08:20:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.807+0000 7fde7263a740 1 client.16791 shutdown 2024-04-24T08:20:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.808+0000 7fde7263a740 1 -- 172.21.15.28:0/1941348600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabb7860 msgr2=0x5623fabcab00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.808+0000 7fde7263a740 1 --2- 172.21.15.28:0/1941348600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabb7860 0x5623fabcab00 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7fde5c0dd010 tx=0x7fde5c031f30 comp rx=0 tx=0).stop 2024-04-24T08:20:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.808+0000 7fde7263a740 1 -- 172.21.15.28:0/1941348600 shutdown_connections 2024-04-24T08:20:29.815 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.808+0000 7fde7263a740 1 --2- 172.21.15.28:0/1941348600 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5623fac415b0 0x5623fac439d0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.808+0000 7fde7263a740 1 --2- 172.21.15.28:0/1941348600 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5623fabc2850 0x5623fabc64b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.808+0000 7fde7263a740 1 --2- 172.21.15.28:0/1941348600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5623fabb8180 0x5623fabc5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.808+0000 7fde7263a740 1 --2- 172.21.15.28:0/1941348600 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5623fabb7860 0x5623fabcab00 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.808+0000 7fde7263a740 1 -- 172.21.15.28:0/1941348600 >> 172.21.15.28:0/1941348600 conn(0x5623fabb3b80 msgr2=0x5623fabbf6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.808+0000 7fde4ea00640 0 client.16791 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:20:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.808+0000 7fde7263a740 1 -- 172.21.15.28:0/1941348600 shutdown_connections 2024-04-24T08:20:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.808+0000 7fde7263a740 1 -- 172.21.15.28:0/1941348600 wait complete. 2024-04-24T08:20:29.816 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.808+0000 7fde7263a740 20 client.16791 trim_cache size 0 max 16384 2024-04-24T08:20:29.817 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:20:29.817 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "IDLE (waiting for more scrubs)", 2024-04-24T08:20:29.817 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": { 2024-04-24T08:20:29.817 INFO:teuthology.orchestra.run.smithi028.stdout: "21b9f7a4-1da2-4d56-9950-60ec65f23bca": { 2024-04-24T08:20:29.817 INFO:teuthology.orchestra.run.smithi028.stdout: "path": "/", 2024-04-24T08:20:29.817 INFO:teuthology.orchestra.run.smithi028.stdout: "tag": "21b9f7a4-1da2-4d56-9950-60ec65f23bca", 2024-04-24T08:20:29.818 INFO:teuthology.orchestra.run.smithi028.stdout: "options": "recursive,force" 2024-04-24T08:20:29.818 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:20:29.818 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:20:29.818 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:20:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.809+0000 7fde6f800640 1 -- 172.21.15.28:0/2682362078 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fde500a3d20 msgr2=0x7fde500a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.809+0000 7fde6f800640 1 --2- 172.21.15.28:0/2682362078 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fde500a3d20 0x7fde500a61e0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fde60005fd0 tx=0x7fde60009500 comp rx=0 tx=0).stop 2024-04-24T08:20:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.809+0000 7fde6f800640 1 -- 172.21.15.28:0/2682362078 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde6806a870 msgr2=0x7fde681a06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.809+0000 7fde6f800640 1 --2- 172.21.15.28:0/2682362078 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde6806a870 0x7fde681a06f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fde5c004770 tx=0x7fde5c003c10 comp rx=0 tx=0).stop 2024-04-24T08:20:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.809+0000 7fde6f800640 1 -- 172.21.15.28:0/2682362078 shutdown_connections 2024-04-24T08:20:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.809+0000 7fde6f800640 1 --2- 172.21.15.28:0/2682362078 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fde500a3d20 0x7fde500a61e0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.819 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.809+0000 7fde6f800640 1 --2- 172.21.15.28:0/2682362078 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fde68107330 0x7fde681a0c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.809+0000 7fde6f800640 1 --2- 172.21.15.28:0/2682362078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde6819a8f0 0x7fde6819ada0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.809+0000 7fde6f800640 1 --2- 172.21.15.28:0/2682362078 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fde6806a870 0x7fde681a06f0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.809+0000 7fde6f800640 1 -- 172.21.15.28:0/2682362078 >> 172.21.15.28:0/2682362078 conn(0x7fde68076d00 msgr2=0x7fde6810d380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.809+0000 7fde6f800640 1 -- 172.21.15.28:0/2682362078 shutdown_connections 2024-04-24T08:20:29.820 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:29.809+0000 7fde6f800640 1 -- 172.21.15.28:0/2682362078 wait complete. 2024-04-24T08:20:29.829 INFO:tasks.cephfs.filesystem:scrub status for tag:21b9f7a4-1da2-4d56-9950-60ec65f23bca - {'path': '/', 'tag': '21b9f7a4-1da2-4d56-9950-60ec65f23bca', 'options': 'recursive,force'} 2024-04-24T08:20:30.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:29 smithi028 ceph-mon[27786]: pgmap v314: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-24T08:20:30.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:29 smithi028 ceph-mon[27786]: osdmap e99: 12 total, 12 up, 12 in 2024-04-24T08:20:30.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:29 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:20:30.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:29 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:20:30.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:29 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:20:30.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:29 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:20:30.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:29 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:20:30.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:29 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:20:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:29 smithi117 ceph-mon[32438]: pgmap v314: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-24T08:20:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:29 smithi117 ceph-mon[32438]: osdmap e99: 12 total, 12 up, 12 in 2024-04-24T08:20:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:29 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:20:30.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:29 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:20:30.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:29 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:20:30.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:29 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:20:30.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:29 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:20:30.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:29 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:20:30.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:29 smithi194 ceph-mon[32836]: pgmap v314: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-24T08:20:30.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:29 smithi194 ceph-mon[32836]: osdmap e99: 12 total, 12 up, 12 in 2024-04-24T08:20:30.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:29 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:20:30.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:29 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:20:30.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:29 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:20:30.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:29 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:20:30.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:29 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:20:30.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:29 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:20:31.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:30 smithi117 ceph-mon[32438]: scrub queued for path: / 2024-04-24T08:20:31.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:30 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:20:31.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:30 smithi117 ceph-mon[32438]: scrub summary: active paths [/] 2024-04-24T08:20:31.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:30 smithi117 ceph-mon[32438]: scrub summary: idle+waiting paths [/] 2024-04-24T08:20:31.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:30 smithi194 ceph-mon[32836]: scrub queued for path: / 2024-04-24T08:20:31.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:30 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:20:31.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:30 smithi194 ceph-mon[32836]: scrub summary: active paths [/] 2024-04-24T08:20:31.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:30 smithi194 ceph-mon[32836]: scrub summary: idle+waiting paths [/] 2024-04-24T08:20:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:30 smithi028 ceph-mon[27786]: scrub queued for path: / 2024-04-24T08:20:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:30 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:20:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:30 smithi028 ceph-mon[27786]: scrub summary: active paths [/] 2024-04-24T08:20:31.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:30 smithi028 ceph-mon[27786]: scrub summary: idle+waiting paths [/] 2024-04-24T08:20:32.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:31 smithi117 ceph-mon[32438]: pgmap v316: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-24T08:20:32.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:31 smithi194 ceph-mon[32836]: pgmap v316: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-24T08:20:32.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:31 smithi028 ceph-mon[27786]: pgmap v316: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-04-24T08:20:33.818 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-04-24T08:20:34.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.032+0000 7f0301e00640 1 -- 172.21.15.28:0/1584883002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02fc109380 msgr2=0x7f02fc110f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:34.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.032+0000 7f0301e00640 1 --2- 172.21.15.28:0/1584883002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02fc109380 0x7f02fc110f50 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f02f4009ca0 tx=0x7f02f402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.032+0000 7f0301e00640 1 -- 172.21.15.28:0/1584883002 shutdown_connections 2024-04-24T08:20:34.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.032+0000 7f0301e00640 1 --2- 172.21.15.28:0/1584883002 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02fc104c60 0x7f02fc108ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.032+0000 7f0301e00640 1 --2- 172.21.15.28:0/1584883002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02fc109380 0x7f02fc110f50 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.032+0000 7f0301e00640 1 --2- 172.21.15.28:0/1584883002 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02fc1042b0 0x7f02fc104690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.032+0000 7f0301e00640 1 -- 172.21.15.28:0/1584883002 >> 172.21.15.28:0/1584883002 conn(0x7f02fc0ffb10 msgr2=0x7f02fc101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:34.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.032+0000 7f0301e00640 1 -- 172.21.15.28:0/1584883002 shutdown_connections 2024-04-24T08:20:34.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.032+0000 7f0301e00640 1 -- 172.21.15.28:0/1584883002 wait complete. 2024-04-24T08:20:34.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.033+0000 7f0301e00640 1 Processor -- start 2024-04-24T08:20:34.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.033+0000 7f0301e00640 1 -- start start 2024-04-24T08:20:34.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.033+0000 7f0301e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02fc1042b0 0x7f02fc1a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:34.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.033+0000 7f0301e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02fc109380 0x7f02fc1a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:34.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.033+0000 7f0301e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02fc19a770 0x7f02fc19abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:34.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.034+0000 7f0301e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc114490 con 0x7f02fc19a770 2024-04-24T08:20:34.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.034+0000 7f0301e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc114310 con 0x7f02fc1042b0 2024-04-24T08:20:34.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.034+0000 7f0301e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f02fc114610 con 0x7f02fc109380 2024-04-24T08:20:34.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.034+0000 7f0300a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02fc1042b0 0x7f02fc1a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.034+0000 7f0300a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02fc1042b0 0x7f02fc1a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33214/0 (socket says 172.21.15.28:33214) 2024-04-24T08:20:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.034+0000 7f0300a00640 1 -- 172.21.15.28:0/1694723198 learned_addr learned my addr 172.21.15.28:0/1694723198 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.034+0000 7f0301400640 1 --2- 172.21.15.28:0/1694723198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02fc19a770 0x7f02fc19abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.034+0000 7f02f3e00640 1 --2- 172.21.15.28:0/1694723198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02fc109380 0x7f02fc1a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.034+0000 7f0300a00640 1 -- 172.21.15.28:0/1694723198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02fc109380 msgr2=0x7f02fc1a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.034+0000 7f0300a00640 1 --2- 172.21.15.28:0/1694723198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02fc109380 0x7f02fc1a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.034+0000 7f0300a00640 1 -- 172.21.15.28:0/1694723198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02fc19a770 msgr2=0x7f02fc19abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:34.038 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.034+0000 7f0300a00640 1 --2- 172.21.15.28:0/1694723198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02fc19a770 0x7f02fc19abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.034+0000 7f0300a00640 1 -- 172.21.15.28:0/1694723198 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f02fc19b370 con 0x7f02fc1042b0 2024-04-24T08:20:34.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.035+0000 7f0300a00640 1 --2- 172.21.15.28:0/1694723198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02fc1042b0 0x7f02fc1a06b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00e9f0 tx=0x7f02ec00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:34.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.035+0000 7f02fa000640 1 -- 172.21.15.28:0/1694723198 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec00cd80 con 0x7f02fc1042b0 2024-04-24T08:20:34.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.035+0000 7f02fa000640 1 -- 172.21.15.28:0/1694723198 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f02ec005500 con 0x7f02fc1042b0 2024-04-24T08:20:34.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.035+0000 7f0301e00640 1 -- 172.21.15.28:0/1694723198 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02fc1a7450 con 0x7f02fc1042b0 2024-04-24T08:20:34.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.035+0000 7f02fa000640 1 -- 172.21.15.28:0/1694723198 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f02ec010690 con 0x7f02fc1042b0 2024-04-24T08:20:34.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.035+0000 7f0301e00640 1 -- 172.21.15.28:0/1694723198 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f02fc1a7800 con 0x7f02fc1042b0 2024-04-24T08:20:34.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.036+0000 7f0301e00640 1 -- 172.21.15.28:0/1694723198 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f02fc105780 con 0x7f02fc1042b0 2024-04-24T08:20:34.040 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.038+0000 7f02fa000640 1 -- 172.21.15.28:0/1694723198 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f02ec0046c0 con 0x7f02fc1042b0 2024-04-24T08:20:34.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.039+0000 7f02fa000640 1 --2- 172.21.15.28:0/1694723198 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f02e00a3d10 0x7f02e00a61d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:34.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.040+0000 7f02fa000640 1 -- 172.21.15.28:0/1694723198 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7f02ec0d0ca0 con 0x7f02fc1042b0 2024-04-24T08:20:34.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.040+0000 7f02f3e00640 1 --2- 172.21.15.28:0/1694723198 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f02e00a3d10 0x7f02e00a61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:34.042 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.041+0000 7f02f3e00640 1 --2- 172.21.15.28:0/1694723198 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f02e00a3d10 0x7f02e00a61d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f02fc19bfc0 tx=0x7f02e4005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:34.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.044+0000 7f02fa000640 1 -- 172.21.15.28:0/1694723198 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f02ec09d290 con 0x7f02fc1042b0 2024-04-24T08:20:34.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:33 smithi117 ceph-mon[32438]: pgmap v317: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-24T08:20:34.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:33 smithi194 ceph-mon[32836]: pgmap v317: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-24T08:20:34.277 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.275+0000 7f0301e00640 1 -- 172.21.15.28:0/1694723198 --> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) -- 0x7f02fc10c620 con 0x7f02e00a3d10 2024-04-24T08:20:34.288 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:33 smithi028 ceph-mon[27786]: pgmap v317: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-04-24T08:20:34.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.292+0000 7f02fa000640 1 -- 172.21.15.28:0/1694723198 <== mgr.25225 v2:172.21.15.28:6800/353036813 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+350 (secure 0 0 0) 0x7f02e4011bb0 con 0x7f02e00a3d10 2024-04-24T08:20:34.293 INFO:teuthology.orchestra.run.smithi028.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-04-24T00:00:00", "created": "2024-04-24T08:10:30", "first": "2024-04-24T08:11:00", "last": "2024-04-24T08:20:00", "last_pruned": "2024-04-24T08:20:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-04-24T08:20:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.294+0000 7f0301e00640 1 -- 172.21.15.28:0/1694723198 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f02e00a3d10 msgr2=0x7f02e00a61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.294+0000 7f0301e00640 1 --2- 172.21.15.28:0/1694723198 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f02e00a3d10 0x7f02e00a61d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f02fc19bfc0 tx=0x7f02e4005f90 comp rx=0 tx=0).stop 2024-04-24T08:20:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.294+0000 7f0301e00640 1 -- 172.21.15.28:0/1694723198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02fc1042b0 msgr2=0x7f02fc1a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.294+0000 7f0301e00640 1 --2- 172.21.15.28:0/1694723198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02fc1042b0 0x7f02fc1a06b0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00e9f0 tx=0x7f02ec00edf0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.294+0000 7f0301e00640 1 -- 172.21.15.28:0/1694723198 shutdown_connections 2024-04-24T08:20:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.294+0000 7f0301e00640 1 --2- 172.21.15.28:0/1694723198 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f02e00a3d10 0x7f02e00a61d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.296 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.294+0000 7f0301e00640 1 --2- 172.21.15.28:0/1694723198 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f02fc19a770 0x7f02fc19abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.294+0000 7f0301e00640 1 --2- 172.21.15.28:0/1694723198 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f02fc109380 0x7f02fc1a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.294+0000 7f0301e00640 1 --2- 172.21.15.28:0/1694723198 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f02fc1042b0 0x7f02fc1a06b0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.294+0000 7f0301e00640 1 -- 172.21.15.28:0/1694723198 >> 172.21.15.28:0/1694723198 conn(0x7f02fc0ffb10 msgr2=0x7f02fc100600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:34.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.295+0000 7f0301e00640 1 -- 172.21.15.28:0/1694723198 shutdown_connections 2024-04-24T08:20:34.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.295+0000 7f0301e00640 1 -- 172.21.15.28:0/1694723198 wait complete. 2024-04-24T08:20:34.317 DEBUG:teuthology.orchestra.run.smithi028:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-04-24T08:20:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.525+0000 7f1bdec00640 1 -- 172.21.15.28:0/3381841416 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bd8108540 msgr2=0x7f1bd8108980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:34.527 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.525+0000 7f1bdec00640 1 --2- 172.21.15.28:0/3381841416 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bd8108540 0x7f1bd8108980 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4009930 tx=0x7f1bc402a440 comp rx=0 tx=0).stop 2024-04-24T08:20:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.526+0000 7f1bdec00640 1 -- 172.21.15.28:0/3381841416 shutdown_connections 2024-04-24T08:20:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.526+0000 7f1bdec00640 1 --2- 172.21.15.28:0/3381841416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bd8108ec0 0x7f1bd8115300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.526+0000 7f1bdec00640 1 --2- 172.21.15.28:0/3381841416 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bd8108540 0x7f1bd8108980 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.526+0000 7f1bdec00640 1 --2- 172.21.15.28:0/3381841416 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd810e4d0 0x7f1bd810e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.526+0000 7f1bdec00640 1 -- 172.21.15.28:0/3381841416 >> 172.21.15.28:0/3381841416 conn(0x7f1bd8077de0 msgr2=0x7f1bd8106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.526+0000 7f1bdec00640 1 -- 172.21.15.28:0/3381841416 shutdown_connections 2024-04-24T08:20:34.528 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.526+0000 7f1bdec00640 1 -- 172.21.15.28:0/3381841416 wait complete. 2024-04-24T08:20:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.527+0000 7f1bdec00640 1 Processor -- start 2024-04-24T08:20:34.529 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.527+0000 7f1bdec00640 1 -- start start 2024-04-24T08:20:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.528+0000 7f1bdec00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bd8108540 0x7f1bd81a4a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.528+0000 7f1bdec00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bd8108ec0 0x7f1bd81a4fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.528+0000 7f1bdec00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd810e4d0 0x7f1bd819eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.528+0000 7f1bdec00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd81187d0 con 0x7f1bd810e4d0 2024-04-24T08:20:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.528+0000 7f1bdec00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd8118650 con 0x7f1bd8108ec0 2024-04-24T08:20:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.528+0000 7f1bdec00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd8118950 con 0x7f1bd8108540 2024-04-24T08:20:34.530 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.528+0000 7f1bdd800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bd8108540 0x7f1bd81a4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.528+0000 7f1bdce00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bd8108ec0 0x7f1bd81a4fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.528+0000 7f1bdd800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bd8108540 0x7f1bd81a4a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:43550/0 (socket says 172.21.15.28:43550) 2024-04-24T08:20:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.528+0000 7f1bdd800640 1 -- 172.21.15.28:0/366603142 learned_addr learned my addr 172.21.15.28:0/366603142 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:20:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.528+0000 7f1bde200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd810e4d0 0x7f1bd819eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:34.531 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.528+0000 7f1bdce00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bd8108ec0 0x7f1bd81a4fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:33242/0 (socket says 172.21.15.28:33242) 2024-04-24T08:20:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.529+0000 7f1bdd800640 1 -- 172.21.15.28:0/366603142 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bd8108ec0 msgr2=0x7f1bd81a4fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.529+0000 7f1bdd800640 1 --2- 172.21.15.28:0/366603142 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bd8108ec0 0x7f1bd81a4fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.529+0000 7f1bdd800640 1 -- 172.21.15.28:0/366603142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd810e4d0 msgr2=0x7f1bd819eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:34.532 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.529+0000 7f1bdd800640 1 --2- 172.21.15.28:0/366603142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd810e4d0 0x7f1bd819eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.529+0000 7f1bdd800640 1 -- 172.21.15.28:0/366603142 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd819f3f0 con 0x7f1bd8108540 2024-04-24T08:20:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.529+0000 7f1bdd800640 1 --2- 172.21.15.28:0/366603142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bd8108540 0x7f1bd81a4a70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00d950 tx=0x7f1bcc00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.529+0000 7f1bd6000640 1 -- 172.21.15.28:0/366603142 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc017070 con 0x7f1bd8108540 2024-04-24T08:20:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.529+0000 7f1bd6000640 1 -- 172.21.15.28:0/366603142 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bcc009c30 con 0x7f1bd8108540 2024-04-24T08:20:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.529+0000 7f1bdec00640 1 -- 172.21.15.28:0/366603142 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd819f6e0 con 0x7f1bd8108540 2024-04-24T08:20:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.529+0000 7f1bd6000640 1 -- 172.21.15.28:0/366603142 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bcc00f6b0 con 0x7f1bd8108540 2024-04-24T08:20:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.530+0000 7f1bdec00640 1 -- 172.21.15.28:0/366603142 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd81ab9f0 con 0x7f1bd8108540 2024-04-24T08:20:34.533 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.531+0000 7f1bdec00640 1 -- 172.21.15.28:0/366603142 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1bd8109b70 con 0x7f1bd8108540 2024-04-24T08:20:34.534 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.533+0000 7f1bd6000640 1 -- 172.21.15.28:0/366603142 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f1bcc0109d0 con 0x7f1bd8108540 2024-04-24T08:20:34.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.534+0000 7f1bd6000640 1 --2- 172.21.15.28:0/366603142 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1bc00a3d90 0x7f1bc00a6250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:20:34.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.535+0000 7f1bd6000640 1 -- 172.21.15.28:0/366603142 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7f1bcc0d0330 con 0x7f1bd8108540 2024-04-24T08:20:34.536 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.535+0000 7f1bdce00640 1 --2- 172.21.15.28:0/366603142 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1bc00a3d90 0x7f1bc00a6250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:20:34.537 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.536+0000 7f1bdce00640 1 --2- 172.21.15.28:0/366603142 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1bc00a3d90 0x7f1bc00a6250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4009900 tx=0x7f1bc40058d0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:20:34.540 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.539+0000 7f1bd6000640 1 -- 172.21.15.28:0/366603142 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f1bcc009de0 con 0x7f1bd8108540 2024-04-24T08:20:34.772 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.770+0000 7f1bdec00640 1 -- 172.21.15.28:0/366603142 --> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) -- 0x7f1bd8062440 con 0x7f1bc00a3d90 2024-04-24T08:20:34.785 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.783+0000 7f1bd6000640 1 -- 172.21.15.28:0/366603142 <== mgr.25225 v2:172.21.15.28:6800/353036813 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+10 (secure 0 0 0) 0x7f1bd8062440 con 0x7f1bc00a3d90 2024-04-24T08:20:34.785 INFO:teuthology.orchestra.run.smithi028.stdout:/ 1m 6m3h 2024-04-24T08:20:34.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.786+0000 7f1bdec00640 1 -- 172.21.15.28:0/366603142 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1bc00a3d90 msgr2=0x7f1bc00a6250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:34.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.786+0000 7f1bdec00640 1 --2- 172.21.15.28:0/366603142 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1bc00a3d90 0x7f1bc00a6250 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f1bc4009900 tx=0x7f1bc40058d0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.786+0000 7f1bdec00640 1 -- 172.21.15.28:0/366603142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bd8108540 msgr2=0x7f1bd81a4a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:20:34.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.786+0000 7f1bdec00640 1 --2- 172.21.15.28:0/366603142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bd8108540 0x7f1bd81a4a70 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f1bcc00d950 tx=0x7f1bcc00de10 comp rx=0 tx=0).stop 2024-04-24T08:20:34.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.786+0000 7f1bdec00640 1 -- 172.21.15.28:0/366603142 shutdown_connections 2024-04-24T08:20:34.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.786+0000 7f1bdec00640 1 --2- 172.21.15.28:0/366603142 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f1bc00a3d90 0x7f1bc00a6250 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.788 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.786+0000 7f1bdec00640 1 --2- 172.21.15.28:0/366603142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f1bd810e4d0 0x7f1bd819eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.786+0000 7f1bdec00640 1 --2- 172.21.15.28:0/366603142 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bd8108ec0 0x7f1bd81a4fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.786+0000 7f1bdec00640 1 --2- 172.21.15.28:0/366603142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f1bd8108540 0x7f1bd81a4a70 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:20:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.786+0000 7f1bdec00640 1 -- 172.21.15.28:0/366603142 >> 172.21.15.28:0/366603142 conn(0x7f1bd8077de0 msgr2=0x7f1bd8078500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:20:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.786+0000 7f1bdec00640 1 -- 172.21.15.28:0/366603142 shutdown_connections 2024-04-24T08:20:34.789 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:20:34.787+0000 7f1bdec00640 1 -- 172.21.15.28:0/366603142 wait complete. 2024-04-24T08:20:34.809 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-04-24T08:20:34.821 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-04-24T08:20:36.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:35 smithi117 ceph-mon[32438]: from='client.27074 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:36.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:35 smithi117 ceph-mon[32438]: pgmap v318: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-24T08:20:36.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:35 smithi117 ceph-mon[32438]: from='client.27094 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:36.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:35 smithi194 ceph-mon[32836]: from='client.27074 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:36.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:35 smithi194 ceph-mon[32836]: pgmap v318: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-24T08:20:36.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:35 smithi194 ceph-mon[32836]: from='client.27094 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:36.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:35 smithi028 ceph-mon[27786]: from='client.27074 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:36.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:35 smithi028 ceph-mon[27786]: pgmap v318: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-04-24T08:20:36.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:35 smithi028 ceph-mon[27786]: from='client.27094 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-04-24T08:20:38.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:37 smithi117 ceph-mon[32438]: pgmap v319: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:38.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:37 smithi194 ceph-mon[32836]: pgmap v319: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:38.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:37 smithi028 ceph-mon[27786]: pgmap v319: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:40.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:39 smithi117 ceph-mon[32438]: pgmap v320: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-24T08:20:40.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:39 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:20:40.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:39 smithi117 ceph-mon[32438]: scrub summary: idle 2024-04-24T08:20:40.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:39 smithi194 ceph-mon[32836]: pgmap v320: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-24T08:20:40.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:39 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:20:40.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:39 smithi194 ceph-mon[32836]: scrub summary: idle 2024-04-24T08:20:40.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:39 smithi028 ceph-mon[27786]: pgmap v320: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 9 op/s 2024-04-24T08:20:40.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:39 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:20:40.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:39 smithi028 ceph-mon[27786]: scrub summary: idle 2024-04-24T08:20:42.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:41 smithi117 ceph-mon[32438]: pgmap v321: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-24T08:20:42.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:41 smithi194 ceph-mon[32836]: pgmap v321: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-24T08:20:42.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:41 smithi028 ceph-mon[27786]: pgmap v321: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-24T08:20:44.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:43 smithi194 ceph-mon[32836]: pgmap v322: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-24T08:20:44.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:43 smithi028 ceph-mon[27786]: pgmap v322: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-24T08:20:44.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:43 smithi117 ceph-mon[32438]: pgmap v322: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-24T08:20:45.539 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:45 smithi028 ceph-mon[27786]: pgmap v323: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-24T08:20:45.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:45 smithi117 ceph-mon[32438]: pgmap v323: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-24T08:20:45.690 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:45 smithi194 ceph-mon[32836]: pgmap v323: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-24T08:20:47.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:47 smithi028 ceph-mon[27786]: pgmap v324: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-24T08:20:47.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:47 smithi117 ceph-mon[32438]: pgmap v324: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-24T08:20:47.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:47 smithi194 ceph-mon[32836]: pgmap v324: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 0 B/s wr, 7 op/s 2024-04-24T08:20:49.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:49 smithi028 ceph-mon[27786]: pgmap v325: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-24T08:20:49.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:49 smithi117 ceph-mon[32438]: pgmap v325: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-24T08:20:49.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:49 smithi194 ceph-mon[32836]: pgmap v325: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 0 B/s wr, 8 op/s 2024-04-24T08:20:51.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:51 smithi028 ceph-mon[27786]: pgmap v326: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-04-24T08:20:51.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:51 smithi117 ceph-mon[32438]: pgmap v326: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-04-24T08:20:51.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:51 smithi194 ceph-mon[32836]: pgmap v326: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-04-24T08:20:53.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:53 smithi117 ceph-mon[32438]: pgmap v327: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:53.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:53 smithi194 ceph-mon[32836]: pgmap v327: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:54.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:53 smithi028 ceph-mon[27786]: pgmap v327: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:20:55.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:55 smithi117 ceph-mon[32438]: pgmap v328: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:55.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:55 smithi194 ceph-mon[32836]: pgmap v328: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:56.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:55 smithi028 ceph-mon[27786]: pgmap v328: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:20:57.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:57 smithi117 ceph-mon[32438]: pgmap v329: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:57.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:57 smithi194 ceph-mon[32836]: pgmap v329: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:58.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:57 smithi028 ceph-mon[27786]: pgmap v329: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:20:59.831 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-04-24T08:20:59.831 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-04-24T08:20:59.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:20:59 smithi117 ceph-mon[32438]: pgmap v330: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:20:59.913 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:20:59 smithi028 ceph-mon[27786]: pgmap v330: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:20:59.943 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:20:59 smithi194 ceph-mon[32836]: pgmap v330: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:21:00.061 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.060+0000 7efd1ee00640 1 -- 172.21.15.28:0/415050471 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd18104310 msgr2=0x7efd181046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.060+0000 7efd1ee00640 1 --2- 172.21.15.28:0/415050471 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd18104310 0x7efd181046f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7efd0c009900 tx=0x7efd0c02a410 comp rx=0 tx=0).stop 2024-04-24T08:21:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.061+0000 7efd1ee00640 1 -- 172.21.15.28:0/415050471 shutdown_connections 2024-04-24T08:21:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.061+0000 7efd1ee00640 1 --2- 172.21.15.28:0/415050471 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd181093e0 0x7efd18110fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.061+0000 7efd1ee00640 1 --2- 172.21.15.28:0/415050471 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd18104cc0 0x7efd18108d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.061+0000 7efd1ee00640 1 --2- 172.21.15.28:0/415050471 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd18104310 0x7efd181046f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.061+0000 7efd1ee00640 1 -- 172.21.15.28:0/415050471 >> 172.21.15.28:0/415050471 conn(0x7efd180ffb10 msgr2=0x7efd18101f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.062 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.061+0000 7efd1ee00640 1 -- 172.21.15.28:0/415050471 shutdown_connections 2024-04-24T08:21:00.063 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.061+0000 7efd1ee00640 1 -- 172.21.15.28:0/415050471 wait complete. 2024-04-24T08:21:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.062+0000 7efd1ee00640 1 Processor -- start 2024-04-24T08:21:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.062+0000 7efd1ee00640 1 -- start start 2024-04-24T08:21:00.064 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.063+0000 7efd1ee00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd18104310 0x7efd181a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.063+0000 7efd1ee00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd181093e0 0x7efd181a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.063+0000 7efd1ee00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd1819a7c0 0x7efd1819ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.063+0000 7efd1ee00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd181144c0 con 0x7efd18104310 2024-04-24T08:21:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.063+0000 7efd1ee00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efd18114340 con 0x7efd1819a7c0 2024-04-24T08:21:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.063+0000 7efd1ee00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efd18114640 con 0x7efd181093e0 2024-04-24T08:21:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.063+0000 7efd1da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd18104310 0x7efd181a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.063+0000 7efd1da00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd18104310 0x7efd181a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35734/0 (socket says 172.21.15.28:35734) 2024-04-24T08:21:00.065 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.063+0000 7efd1da00640 1 -- 172.21.15.28:0/239551743 learned_addr learned my addr 172.21.15.28:0/239551743 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.063+0000 7efd1d000640 1 --2- 172.21.15.28:0/239551743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd181093e0 0x7efd181a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.063+0000 7efd1e400640 1 --2- 172.21.15.28:0/239551743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd1819a7c0 0x7efd1819ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.063+0000 7efd1da00640 1 -- 172.21.15.28:0/239551743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd181093e0 msgr2=0x7efd181a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.063+0000 7efd1da00640 1 --2- 172.21.15.28:0/239551743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd181093e0 0x7efd181a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.063+0000 7efd1da00640 1 -- 172.21.15.28:0/239551743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd1819a7c0 msgr2=0x7efd1819ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.064+0000 7efd1da00640 1 --2- 172.21.15.28:0/239551743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd1819a7c0 0x7efd1819ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.064+0000 7efd1da00640 1 -- 172.21.15.28:0/239551743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd1819b3c0 con 0x7efd18104310 2024-04-24T08:21:00.066 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.064+0000 7efd1da00640 1 --2- 172.21.15.28:0/239551743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd18104310 0x7efd181a06b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7efd0c004770 tx=0x7efd0c003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.064+0000 7efd16000640 1 -- 172.21.15.28:0/239551743 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd0c03f070 con 0x7efd18104310 2024-04-24T08:21:00.067 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.064+0000 7efd16000640 1 -- 172.21.15.28:0/239551743 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd0c031ca0 con 0x7efd18104310 2024-04-24T08:21:00.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.064+0000 7efd16000640 1 -- 172.21.15.28:0/239551743 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd0c035030 con 0x7efd18104310 2024-04-24T08:21:00.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.064+0000 7efd1ee00640 1 -- 172.21.15.28:0/239551743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd1819b650 con 0x7efd18104310 2024-04-24T08:21:00.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.065+0000 7efd1ee00640 1 -- 172.21.15.28:0/239551743 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd181a76f0 con 0x7efd18104310 2024-04-24T08:21:00.068 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.068+0000 7efd23204740 1 Processor -- start 2024-04-24T08:21:00.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.068+0000 7efd23204740 1 -- start start 2024-04-24T08:21:00.069 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.068+0000 7efd16000640 1 -- 172.21.15.28:0/239551743 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7efd0c042070 con 0x7efd18104310 2024-04-24T08:21:00.070 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.069+0000 7efd23204740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bba30 0x5566d76bbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.069+0000 7efd23204740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76bc350 0x5566d76c64e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.069+0000 7efd23204740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76c6a20 0x5566d76c8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.069+0000 7efd23204740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5566d76211f0 con 0x5566d76c6a20 2024-04-24T08:21:00.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.069+0000 7efd23204740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5566d75c7350 con 0x5566d76bba30 2024-04-24T08:21:00.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.069+0000 7efd23204740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5566d761d1b0 con 0x5566d76bc350 2024-04-24T08:21:00.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.069+0000 7efd1d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76c6a20 0x5566d76c8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.069+0000 7efd1da00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76bc350 0x5566d76c64e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.071 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.069+0000 7efd1d000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76c6a20 0x5566d76c8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35742/0 (socket says 172.21.15.28:35742) 2024-04-24T08:21:00.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.069+0000 7efd1e400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bba30 0x5566d76bbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.069+0000 7efd1d000640 1 -- 172.21.15.28:0/409029614 learned_addr learned my addr 172.21.15.28:0/409029614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:00.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.069+0000 7efd1e400640 1 -- 172.21.15.28:0/409029614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76bc350 msgr2=0x5566d76c64e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.069+0000 7efd1e400640 1 --2- 172.21.15.28:0/409029614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76bc350 0x5566d76c64e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.069+0000 7efd1e400640 1 -- 172.21.15.28:0/409029614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76c6a20 msgr2=0x5566d76c8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.069+0000 7efd1e400640 1 --2- 172.21.15.28:0/409029614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76c6a20 0x5566d76c8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.070+0000 7efd1e400640 1 -- 172.21.15.28:0/409029614 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566d761bf50 con 0x5566d76bba30 2024-04-24T08:21:00.072 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.070+0000 7efd1e400640 1 --2- 172.21.15.28:0/409029614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bba30 0x5566d76bbe10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7efd10009f00 tx=0x7efd10011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=54d9459ab56dcd70 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.070+0000 7efd0aa00640 1 -- 172.21.15.28:0/409029614 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd10004d20 con 0x5566d76bba30 2024-04-24T08:21:00.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.070+0000 7efd0aa00640 1 -- 172.21.15.28:0/409029614 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd10004050 con 0x5566d76bba30 2024-04-24T08:21:00.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.070+0000 7efd16000640 1 --2- 172.21.15.28:0/239551743 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efd000a3d20 0x7efd000a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.073 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.070+0000 7efd0aa00640 1 -- 172.21.15.28:0/409029614 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd10005280 con 0x5566d76bba30 2024-04-24T08:21:00.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.070+0000 7efd16000640 1 -- 172.21.15.28:0/239551743 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7efd0c0f0e30 con 0x7efd18104310 2024-04-24T08:21:00.076 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.070+0000 7efd1da00640 1 --2- 172.21.15.28:0/239551743 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efd000a3d20 0x7efd000a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.071+0000 7efd23204740 1 -- 172.21.15.28:0/409029614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bba30 msgr2=0x5566d76bbe10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.071+0000 7efd23204740 1 --2- 172.21.15.28:0/409029614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bba30 0x5566d76bbe10 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7efd10009f00 tx=0x7efd10011810 comp rx=0 tx=0).stop 2024-04-24T08:21:00.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.071+0000 7efd23204740 1 -- 172.21.15.28:0/409029614 shutdown_connections 2024-04-24T08:21:00.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.071+0000 7efd23204740 1 --2- 172.21.15.28:0/409029614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76c6a20 0x5566d76c8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.071+0000 7efd23204740 1 --2- 172.21.15.28:0/409029614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76bc350 0x5566d76c64e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.071+0000 7efd23204740 1 --2- 172.21.15.28:0/409029614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bba30 0x5566d76bbe10 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.071+0000 7efd23204740 1 -- 172.21.15.28:0/409029614 >> 172.21.15.28:0/409029614 conn(0x5566d76b7d50 msgr2=0x5566d76b8130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.077 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.071+0000 7efd23204740 1 -- 172.21.15.28:0/409029614 shutdown_connections 2024-04-24T08:21:00.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.071+0000 7efd1da00640 1 --2- 172.21.15.28:0/239551743 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efd000a3d20 0x7efd000a61e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x5566d76bcff0 tx=0x7efd0c0f48e0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.071+0000 7efd23204740 1 -- 172.21.15.28:0/409029614 wait complete. 2024-04-24T08:21:00.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.072+0000 7efd23204740 1 -- start start 2024-04-24T08:21:00.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.072+0000 7efd23204740 1 Processor -- start 2024-04-24T08:21:00.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.072+0000 7efd23204740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bba30 0x5566d76c39a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.072+0000 7efd23204740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76bc350 0x5566d76c3ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.072+0000 7efd23204740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76c6a20 0x5566d76c4420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.078 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.072+0000 7efd23204740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x5566d76c9810 con 0x5566d76c6a20 2024-04-24T08:21:00.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.072+0000 7efd23204740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x5566d75da690 con 0x5566d76bba30 2024-04-24T08:21:00.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.072+0000 7efd23204740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x5566d76c9990 con 0x5566d76bc350 2024-04-24T08:21:00.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.072+0000 7efd23204740 10 client.0 _ll_register_callbacks cb 0x5566d7616650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:00.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.073+0000 7efd1e400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bba30 0x5566d76c39a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.073+0000 7efd1e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76c6a20 0x5566d76c4420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.073+0000 7efd1e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76c6a20 0x5566d76c4420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35748/0 (socket says 172.21.15.28:35748) 2024-04-24T08:21:00.079 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.073+0000 7efd1e400640 1 -- 172.21.15.28:0/4161348077 learned_addr learned my addr 172.21.15.28:0/4161348077 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.073+0000 7efd1d000640 1 --2- 172.21.15.28:0/4161348077 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76bc350 0x5566d76c3ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.073+0000 7efd1d000640 1 -- 172.21.15.28:0/4161348077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bba30 msgr2=0x5566d76c39a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.073+0000 7efd1d000640 1 --2- 172.21.15.28:0/4161348077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bba30 0x5566d76c39a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.073+0000 7efd1d000640 1 -- 172.21.15.28:0/4161348077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76c6a20 msgr2=0x5566d76c4420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.073+0000 7efd1d000640 1 --2- 172.21.15.28:0/4161348077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76c6a20 0x5566d76c4420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.073+0000 7efd1d000640 1 -- 172.21.15.28:0/4161348077 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566d75d9920 con 0x5566d76bc350 2024-04-24T08:21:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.073+0000 7efd1e400640 1 --2- 172.21.15.28:0/4161348077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76c6a20 0x5566d76c4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.074+0000 7efd1d000640 1 --2- 172.21.15.28:0/4161348077 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76bc350 0x5566d76c3ee0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7efd040126b0 tx=0x7efd04012ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e3e5f73c0e00e73f server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.080 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.074+0000 7efcff400640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:21:00.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.074+0000 7efcff400640 1 -- 172.21.15.28:0/4161348077 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd04004050 con 0x5566d76bc350 2024-04-24T08:21:00.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.074+0000 7efcff400640 1 -- 172.21.15.28:0/4161348077 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7efd040041f0 con 0x5566d76bc350 2024-04-24T08:21:00.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.074+0000 7efd23204740 1 -- 172.21.15.28:0/4161348077 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd10004050 con 0x5566d76bc350 2024-04-24T08:21:00.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.074+0000 7efcff400640 1 -- 172.21.15.28:0/4161348077 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7efd040043f0 con 0x5566d76bc350 2024-04-24T08:21:00.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.074+0000 7efd23204740 10 client.27106 fetch_fsmap learned FSMap version 26 2024-04-24T08:21:00.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.074+0000 7efd23204740 1 -- 172.21.15.28:0/4161348077 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5566d73fd7d0 con 0x5566d76bc350 2024-04-24T08:21:00.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.075+0000 7efcff400640 1 -- 172.21.15.28:0/4161348077 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd10004050 con 0x5566d76bc350 2024-04-24T08:21:00.081 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.075+0000 7efcff400640 1 -- 172.21.15.28:0/4161348077 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7efd040039c0 con 0x5566d76bc350 2024-04-24T08:21:00.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.075+0000 7efd23204740 10 client.27106 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:21:00.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.075+0000 7efd23204740 10 client.27106 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:21:00.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.075+0000 7efd23204740 20 client.27106 populate_metadata read hostname 'smithi028' 2024-04-24T08:21:00.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.075+0000 7efd23204740 1 --2- 172.21.15.28:0/4161348077 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5566d76d6e30 0x5566d76d9220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.075+0000 7efd23204740 4 client.27106 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:21:00.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.075+0000 7efd23204740 1 -- 172.21.15.28:0/4161348077 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5566d762ec10 con 0x5566d76d6e30 2024-04-24T08:21:00.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.075+0000 7efd1e400640 1 --2- 172.21.15.28:0/4161348077 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5566d76d6e30 0x5566d76d9220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.082 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.076+0000 7efd1e400640 1 --2- 172.21.15.28:0/4161348077 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5566d76d6e30 0x5566d76d9220 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=644f39a8096951b3 server_cookie=aeeb3ea8711316d2 in_seq=0 out_seq=0 2024-04-24T08:21:00.083 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.076+0000 7efcff400640 10 client.27106 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:21:00.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.077+0000 7efcff400640 1 -- 172.21.15.28:0/4161348077 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5566d762ec10 con 0x5566d76d6e30 2024-04-24T08:21:00.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.077+0000 7efcff400640 10 client.27106 handle_command_reply: tid=0 2024-04-24T08:21:00.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.077+0000 7efd23204740 1 client.27106 shutdown 2024-04-24T08:21:00.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.077+0000 7efd23204740 1 -- 172.21.15.28:0/4161348077 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76bc350 msgr2=0x5566d76c3ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.077+0000 7efd23204740 1 --2- 172.21.15.28:0/4161348077 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76bc350 0x5566d76c3ee0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7efd040126b0 tx=0x7efd04012ab0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.077+0000 7efd23204740 1 -- 172.21.15.28:0/4161348077 shutdown_connections 2024-04-24T08:21:00.084 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.077+0000 7efd23204740 1 --2- 172.21.15.28:0/4161348077 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5566d76d6e30 0x5566d76d9220 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.077+0000 7efcff400640 0 client.27106 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:21:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.077+0000 7efd23204740 1 --2- 172.21.15.28:0/4161348077 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76c6a20 0x5566d76c4420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.077+0000 7efd23204740 1 --2- 172.21.15.28:0/4161348077 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76bc350 0x5566d76c3ee0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.077+0000 7efd23204740 1 --2- 172.21.15.28:0/4161348077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bba30 0x5566d76c39a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.077+0000 7efd23204740 1 -- 172.21.15.28:0/4161348077 >> 172.21.15.28:0/4161348077 conn(0x5566d76b7d50 msgr2=0x5566d76bdb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.077+0000 7efd23204740 1 -- 172.21.15.28:0/4161348077 shutdown_connections 2024-04-24T08:21:00.085 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.077+0000 7efd23204740 1 -- 172.21.15.28:0/4161348077 wait complete. 2024-04-24T08:21:00.086 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.077+0000 7efd23204740 20 client.27106 trim_cache size 0 max 16384 2024-04-24T08:21:00.097 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.097+0000 7efd23204740 1 Processor -- start 2024-04-24T08:21:00.098 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.097+0000 7efd23204740 1 -- start start 2024-04-24T08:21:00.099 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.097+0000 7efd23204740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76bba30 0x5566d76c4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.097+0000 7efd23204740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bc350 0x5566d76deb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.098+0000 7efd23204740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76c6a20 0x5566d76df070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.098+0000 7efd23204740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd181b27b0 con 0x5566d76bba30 2024-04-24T08:21:00.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.098+0000 7efd23204740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efd181b0080 con 0x5566d76bc350 2024-04-24T08:21:00.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.098+0000 7efd23204740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efd181adef0 con 0x5566d76c6a20 2024-04-24T08:21:00.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.098+0000 7efd1d000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bc350 0x5566d76deb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.098+0000 7efd1d000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bc350 0x5566d76deb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49432/0 (socket says 172.21.15.28:49432) 2024-04-24T08:21:00.100 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.098+0000 7efd1d000640 1 -- 172.21.15.28:0/901949731 learned_addr learned my addr 172.21.15.28:0/901949731 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.098+0000 7efd1e400640 1 --2- 172.21.15.28:0/901949731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76bba30 0x5566d76c4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.098+0000 7efd1d000640 1 -- 172.21.15.28:0/901949731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76c6a20 msgr2=0x5566d76df070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.098+0000 7efd1e400640 1 --2- 172.21.15.28:0/901949731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76c6a20 0x5566d76df070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.098+0000 7efd1d000640 1 --2- 172.21.15.28:0/901949731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76c6a20 0x5566d76df070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.099+0000 7efd1d000640 1 -- 172.21.15.28:0/901949731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76bba30 msgr2=0x5566d76c4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.099+0000 7efd1d000640 1 --2- 172.21.15.28:0/901949731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76bba30 0x5566d76c4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.099+0000 7efd1d000640 1 -- 172.21.15.28:0/901949731 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5566d73333f0 con 0x5566d76bc350 2024-04-24T08:21:00.101 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.099+0000 7efd1e400640 1 --2- 172.21.15.28:0/901949731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76c6a20 0x5566d76df070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:00.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.099+0000 7efd1d000640 1 --2- 172.21.15.28:0/901949731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bc350 0x5566d76deb30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7efd04009e00 tx=0x7efd040057f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=47a6748f63c47013 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.099+0000 7efd0a000640 1 -- 172.21.15.28:0/901949731 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd040104f0 con 0x5566d76bc350 2024-04-24T08:21:00.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.099+0000 7efd0a000640 1 -- 172.21.15.28:0/901949731 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd04003ad0 con 0x5566d76bc350 2024-04-24T08:21:00.102 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.099+0000 7efd0a000640 1 -- 172.21.15.28:0/901949731 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd04010780 con 0x5566d76bc350 2024-04-24T08:21:00.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.100+0000 7efd23204740 1 -- 172.21.15.28:0/901949731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bc350 msgr2=0x5566d76deb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.100+0000 7efd23204740 1 --2- 172.21.15.28:0/901949731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bc350 0x5566d76deb30 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7efd04009e00 tx=0x7efd040057f0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.100+0000 7efd23204740 1 -- 172.21.15.28:0/901949731 shutdown_connections 2024-04-24T08:21:00.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.100+0000 7efd23204740 1 --2- 172.21.15.28:0/901949731 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76c6a20 0x5566d76df070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.100+0000 7efd23204740 1 --2- 172.21.15.28:0/901949731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76bc350 0x5566d76deb30 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.100+0000 7efd23204740 1 --2- 172.21.15.28:0/901949731 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76bba30 0x5566d76c4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.100+0000 7efd23204740 1 -- 172.21.15.28:0/901949731 >> 172.21.15.28:0/901949731 conn(0x5566d76b7d50 msgr2=0x5566d76b6740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.100+0000 7efd23204740 1 -- 172.21.15.28:0/901949731 shutdown_connections 2024-04-24T08:21:00.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.100+0000 7efd23204740 1 -- 172.21.15.28:0/901949731 wait complete. 2024-04-24T08:21:00.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.100+0000 7efd23204740 1 -- start start 2024-04-24T08:21:00.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.101+0000 7efd23204740 1 Processor -- start 2024-04-24T08:21:00.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.101+0000 7efd23204740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76bba30 0x5566d76c0910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.101+0000 7efd23204740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76bc350 0x5566d76c0e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.101+0000 7efd23204740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76c6a20 0x5566d76c1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.101+0000 7efd23204740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7efd181ae540 con 0x5566d76bba30 2024-04-24T08:21:00.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.101+0000 7efd23204740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7efd181af9b0 con 0x5566d76c6a20 2024-04-24T08:21:00.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.101+0000 7efd23204740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7efd181aec10 con 0x5566d76bc350 2024-04-24T08:21:00.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.101+0000 7efd23204740 10 client.0 _ll_register_callbacks cb 0x7efd04009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:00.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.101+0000 7efd1e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76bba30 0x5566d76c0910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.102+0000 7efd1e400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76bba30 0x5566d76c0910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35766/0 (socket says 172.21.15.28:35766) 2024-04-24T08:21:00.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.102+0000 7efd1e400640 1 -- 172.21.15.28:0/3939135229 learned_addr learned my addr 172.21.15.28:0/3939135229 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:00.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.102+0000 7efd1d000640 1 --2- 172.21.15.28:0/3939135229 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76bc350 0x5566d76c0e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.102+0000 7efd1e400640 1 --2- 172.21.15.28:0/3939135229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76c6a20 0x5566d76c1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.102+0000 7efd1e400640 1 -- 172.21.15.28:0/3939135229 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76bc350 msgr2=0x5566d76c0e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.102+0000 7efd1e400640 1 --2- 172.21.15.28:0/3939135229 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76bc350 0x5566d76c0e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.102+0000 7efd1e400640 1 -- 172.21.15.28:0/3939135229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76c6a20 msgr2=0x5566d76c1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.102+0000 7efd1e400640 1 --2- 172.21.15.28:0/3939135229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76c6a20 0x5566d76c1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.102+0000 7efd1e400640 1 -- 172.21.15.28:0/3939135229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5566d72f30f0 con 0x5566d76bba30 2024-04-24T08:21:00.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.102+0000 7efd1e400640 1 --2- 172.21.15.28:0/3939135229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76bba30 0x5566d76c0910 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7efd10004f00 tx=0x7efd10009ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d44db3492137985c server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.102+0000 7efcff400640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:21:00.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.102+0000 7efcff400640 1 -- 172.21.15.28:0/3939135229 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd10005650 con 0x5566d76bba30 2024-04-24T08:21:00.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.103+0000 7efcff400640 1 -- 172.21.15.28:0/3939135229 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7efd100057f0 con 0x5566d76bba30 2024-04-24T08:21:00.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.103+0000 7efd23204740 1 -- 172.21.15.28:0/3939135229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efd04003ad0 con 0x5566d76bba30 2024-04-24T08:21:00.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.103+0000 7efcff400640 1 -- 172.21.15.28:0/3939135229 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7efd1003e050 con 0x5566d76bba30 2024-04-24T08:21:00.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.103+0000 7efd23204740 10 client.16821 fetch_fsmap learned FSMap version 26 2024-04-24T08:21:00.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.103+0000 7efd23204740 1 -- 172.21.15.28:0/3939135229 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x5566d75e6810 con 0x5566d76bba30 2024-04-24T08:21:00.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.103+0000 7efcff400640 1 -- 172.21.15.28:0/3939135229 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd04003ad0 con 0x5566d76bba30 2024-04-24T08:21:00.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.103+0000 7efcff400640 1 -- 172.21.15.28:0/3939135229 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7efd100285f0 con 0x5566d76bba30 2024-04-24T08:21:00.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.103+0000 7efd23204740 10 client.16821 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:21:00.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.103+0000 7efd23204740 10 client.16821 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:21:00.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.104+0000 7efd23204740 20 client.16821 populate_metadata read hostname 'smithi028' 2024-04-24T08:21:00.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.104+0000 7efd23204740 1 --2- 172.21.15.28:0/3939135229 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5566d76d6e30 0x5566d7748380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.104+0000 7efd23204740 4 client.16821 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-04-24T08:21:00.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.104+0000 7efd23204740 1 -- 172.21.15.28:0/3939135229 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5566d7628a80 con 0x5566d76d6e30 2024-04-24T08:21:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.104+0000 7efd1d000640 1 --2- 172.21.15.28:0/3939135229 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5566d76d6e30 0x5566d7748380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.104+0000 7efd1d000640 1 --2- 172.21.15.28:0/3939135229 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5566d76d6e30 0x5566d7748380 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3991e5cf4cc65ad5 server_cookie=eb788d32411d2a1f in_seq=0 out_seq=0 2024-04-24T08:21:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.104+0000 7efcff400640 10 client.16821 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:21:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.105+0000 7efcff400640 1 -- 172.21.15.28:0/3939135229 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x5566d7628a80 con 0x5566d76d6e30 2024-04-24T08:21:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.105+0000 7efcff400640 10 client.16821 handle_command_reply: tid=0 2024-04-24T08:21:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.105+0000 7efd23204740 1 client.16821 shutdown 2024-04-24T08:21:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.105+0000 7efd23204740 1 -- 172.21.15.28:0/3939135229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76bba30 msgr2=0x5566d76c0910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.105+0000 7efd23204740 1 --2- 172.21.15.28:0/3939135229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76bba30 0x5566d76c0910 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7efd10004f00 tx=0x7efd10009ce0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.105+0000 7efd23204740 1 -- 172.21.15.28:0/3939135229 shutdown_connections 2024-04-24T08:21:00.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.105+0000 7efd23204740 1 --2- 172.21.15.28:0/3939135229 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x5566d76d6e30 0x5566d7748380 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.105+0000 7efd23204740 1 --2- 172.21.15.28:0/3939135229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x5566d76c6a20 0x5566d76c1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.105+0000 7efd23204740 1 --2- 172.21.15.28:0/3939135229 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x5566d76bc350 0x5566d76c0e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.105+0000 7efd23204740 1 --2- 172.21.15.28:0/3939135229 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x5566d76bba30 0x5566d76c0910 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.105+0000 7efcff400640 0 client.16821 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:21:00.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.105+0000 7efd23204740 1 -- 172.21.15.28:0/3939135229 >> 172.21.15.28:0/3939135229 conn(0x5566d76b7d50 msgr2=0x5566d76b6fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.105+0000 7efd23204740 1 -- 172.21.15.28:0/3939135229 shutdown_connections 2024-04-24T08:21:00.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.105+0000 7efd23204740 1 -- 172.21.15.28:0/3939135229 wait complete. 2024-04-24T08:21:00.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.105+0000 7efd23204740 20 client.16821 trim_cache size 0 max 16384 2024-04-24T08:21:00.114 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:21:00.114 INFO:teuthology.orchestra.run.smithi028.stdout: "status": "no active scrubs running", 2024-04-24T08:21:00.114 INFO:teuthology.orchestra.run.smithi028.stdout: "scrubs": {} 2024-04-24T08:21:00.114 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:21:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.106+0000 7efd1ee00640 1 -- 172.21.15.28:0/239551743 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efd000a3d20 msgr2=0x7efd000a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.106+0000 7efd1ee00640 1 --2- 172.21.15.28:0/239551743 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efd000a3d20 0x7efd000a61e0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x5566d76bcff0 tx=0x7efd0c0f48e0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.106+0000 7efd1ee00640 1 -- 172.21.15.28:0/239551743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd18104310 msgr2=0x7efd181a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.106+0000 7efd1ee00640 1 --2- 172.21.15.28:0/239551743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd18104310 0x7efd181a06b0 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7efd0c004770 tx=0x7efd0c003c10 comp rx=0 tx=0).stop 2024-04-24T08:21:00.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.106+0000 7efd1ee00640 1 -- 172.21.15.28:0/239551743 shutdown_connections 2024-04-24T08:21:00.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.106+0000 7efd1ee00640 1 --2- 172.21.15.28:0/239551743 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7efd000a3d20 0x7efd000a61e0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.106+0000 7efd1ee00640 1 --2- 172.21.15.28:0/239551743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd1819a7c0 0x7efd1819ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.106+0000 7efd1ee00640 1 --2- 172.21.15.28:0/239551743 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7efd181093e0 0x7efd181a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.106+0000 7efd1ee00640 1 --2- 172.21.15.28:0/239551743 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7efd18104310 0x7efd181a06b0 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.106+0000 7efd1ee00640 1 -- 172.21.15.28:0/239551743 >> 172.21.15.28:0/239551743 conn(0x7efd180ffb10 msgr2=0x7efd18101e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.106+0000 7efd1ee00640 1 -- 172.21.15.28:0/239551743 shutdown_connections 2024-04-24T08:21:00.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.106+0000 7efd1ee00640 1 -- 172.21.15.28:0/239551743 wait complete. 2024-04-24T08:21:00.126 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-04-24T08:21:00.127 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:21:00.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.338+0000 7f00c3e00640 1 -- 172.21.15.28:0/2581195300 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00c41084a0 msgr2=0x7f00c4108900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.338+0000 7f00c3e00640 1 --2- 172.21.15.28:0/2581195300 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00c41084a0 0x7f00c4108900 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f00ac009900 tx=0x7f00ac02a490 comp rx=0 tx=0).stop 2024-04-24T08:21:00.339 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.338+0000 7f00c3e00640 1 -- 172.21.15.28:0/2581195300 shutdown_connections 2024-04-24T08:21:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.338+0000 7f00c3e00640 1 --2- 172.21.15.28:0/2581195300 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00c4108e40 0x7f00c41152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.338+0000 7f00c3e00640 1 --2- 172.21.15.28:0/2581195300 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00c41084a0 0x7f00c4108900 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.338+0000 7f00c3e00640 1 --2- 172.21.15.28:0/2581195300 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00c410d3c0 0x7f00c410d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.338+0000 7f00c3e00640 1 -- 172.21.15.28:0/2581195300 >> 172.21.15.28:0/2581195300 conn(0x7f00c4077b80 msgr2=0x7f00c4079fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.338+0000 7f00c3e00640 1 -- 172.21.15.28:0/2581195300 shutdown_connections 2024-04-24T08:21:00.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.338+0000 7f00c3e00640 1 -- 172.21.15.28:0/2581195300 wait complete. 2024-04-24T08:21:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.339+0000 7f00c3e00640 1 Processor -- start 2024-04-24T08:21:00.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.339+0000 7f00c3e00640 1 -- start start 2024-04-24T08:21:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.340+0000 7f00c3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00c41084a0 0x7f00c419ed50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.340+0000 7f00c3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00c4108e40 0x7f00c419f290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.340+0000 7f00c3e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00c410d3c0 0x7f00c41a3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.340+0000 7f00c3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f00c4118850 con 0x7f00c4108e40 2024-04-24T08:21:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.340+0000 7f00c3e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f00c41186d0 con 0x7f00c410d3c0 2024-04-24T08:21:00.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.340+0000 7f00c3e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f00c41189d0 con 0x7f00c41084a0 2024-04-24T08:21:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.340+0000 7f00ba000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00c4108e40 0x7f00c419f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.340+0000 7f00ba000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00c4108e40 0x7f00c419f290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35792/0 (socket says 172.21.15.28:35792) 2024-04-24T08:21:00.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.340+0000 7f00ba000640 1 -- 172.21.15.28:0/606415380 learned_addr learned my addr 172.21.15.28:0/606415380 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.340+0000 7f00c2a00640 1 --2- 172.21.15.28:0/606415380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00c41084a0 0x7f00c419ed50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.340+0000 7f00c3400640 1 --2- 172.21.15.28:0/606415380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00c410d3c0 0x7f00c41a3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.341+0000 7f00ba000640 1 -- 172.21.15.28:0/606415380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00c41084a0 msgr2=0x7f00c419ed50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.341+0000 7f00ba000640 1 --2- 172.21.15.28:0/606415380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00c41084a0 0x7f00c419ed50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.341+0000 7f00ba000640 1 -- 172.21.15.28:0/606415380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00c410d3c0 msgr2=0x7f00c41a3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.341+0000 7f00ba000640 1 --2- 172.21.15.28:0/606415380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00c410d3c0 0x7f00c41a3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.341+0000 7f00ba000640 1 -- 172.21.15.28:0/606415380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f00c41a3d40 con 0x7f00c4108e40 2024-04-24T08:21:00.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.341+0000 7f00c2a00640 1 --2- 172.21.15.28:0/606415380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00c41084a0 0x7f00c419ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.341+0000 7f00ba000640 1 --2- 172.21.15.28:0/606415380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00c4108e40 0x7f00c419f290 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f00ac004770 tx=0x7f00ac003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.341+0000 7f00bbe00640 1 -- 172.21.15.28:0/606415380 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac03f070 con 0x7f00c4108e40 2024-04-24T08:21:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.341+0000 7f00c3e00640 1 -- 172.21.15.28:0/606415380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f00c41a3fd0 con 0x7f00c4108e40 2024-04-24T08:21:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.341+0000 7f00bbe00640 1 -- 172.21.15.28:0/606415380 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f00ac031ca0 con 0x7f00c4108e40 2024-04-24T08:21:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.341+0000 7f00bbe00640 1 -- 172.21.15.28:0/606415380 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f00ac035030 con 0x7f00c4108e40 2024-04-24T08:21:00.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.342+0000 7f00c3e00640 1 -- 172.21.15.28:0/606415380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f00c41ab8c0 con 0x7f00c4108e40 2024-04-24T08:21:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.343+0000 7f00c3e00640 1 -- 172.21.15.28:0/606415380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f00c4069190 con 0x7f00c4108e40 2024-04-24T08:21:00.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.345+0000 7f00bbe00640 1 -- 172.21.15.28:0/606415380 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f00ac042070 con 0x7f00c4108e40 2024-04-24T08:21:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.346+0000 7f00bbe00640 1 --2- 172.21.15.28:0/606415380 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f00a80a3d20 0x7f00a80a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.346+0000 7f00bbe00640 1 -- 172.21.15.28:0/606415380 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7f00ac0f1450 con 0x7f00c4108e40 2024-04-24T08:21:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.347+0000 7f00c2a00640 1 --2- 172.21.15.28:0/606415380 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f00a80a3d20 0x7f00a80a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.347+0000 7f00c2a00640 1 --2- 172.21.15.28:0/606415380 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f00a80a3d20 0x7f00a80a61e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f00b4005fd0 tx=0x7f00b4009500 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.349 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.349+0000 7f00bbe00640 1 -- 172.21.15.28:0/606415380 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f00ac0bd4b0 con 0x7f00c4108e40 2024-04-24T08:21:00.621 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.620+0000 7f00c3e00640 1 -- 172.21.15.28:0/606415380 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f00c4110900 con 0x7f00c4108e40 2024-04-24T08:21:00.622 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.621+0000 7f00bbe00640 1 -- 172.21.15.28:0/606415380 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11871 (secure 0 0 0) 0x7f00ac0c17b0 con 0x7f00c4108e40 2024-04-24T08:21:00.622 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:21:00.623 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:14:38.663141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:21:00.623 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-24T08:21:00.625 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.625+0000 7f00c3e00640 1 -- 172.21.15.28:0/606415380 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f00a80a3d20 msgr2=0x7f00a80a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.625+0000 7f00c3e00640 1 --2- 172.21.15.28:0/606415380 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f00a80a3d20 0x7f00a80a61e0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f00b4005fd0 tx=0x7f00b4009500 comp rx=0 tx=0).stop 2024-04-24T08:21:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.625+0000 7f00c3e00640 1 -- 172.21.15.28:0/606415380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00c4108e40 msgr2=0x7f00c419f290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.625+0000 7f00c3e00640 1 --2- 172.21.15.28:0/606415380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00c4108e40 0x7f00c419f290 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f00ac004770 tx=0x7f00ac003c10 comp rx=0 tx=0).stop 2024-04-24T08:21:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.625+0000 7f00c3e00640 1 -- 172.21.15.28:0/606415380 shutdown_connections 2024-04-24T08:21:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.625+0000 7f00c3e00640 1 --2- 172.21.15.28:0/606415380 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f00a80a3d20 0x7f00a80a61e0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.625+0000 7f00c3e00640 1 --2- 172.21.15.28:0/606415380 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f00c4108e40 0x7f00c419f290 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.626 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.625+0000 7f00c3e00640 1 --2- 172.21.15.28:0/606415380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00c410d3c0 0x7f00c41a3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.625+0000 7f00c3e00640 1 --2- 172.21.15.28:0/606415380 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f00c41084a0 0x7f00c419ed50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.625+0000 7f00c3e00640 1 -- 172.21.15.28:0/606415380 >> 172.21.15.28:0/606415380 conn(0x7f00c4077b80 msgr2=0x7f00c4078180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.625+0000 7f00c3e00640 1 -- 172.21.15.28:0/606415380 shutdown_connections 2024-04-24T08:21:00.627 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.625+0000 7f00c3e00640 1 -- 172.21.15.28:0/606415380 wait complete. 2024-04-24T08:21:00.649 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-04-24T08:21:00.852 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.852+0000 7f6ec3e00640 1 -- 172.21.15.28:0/774883270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ec410d500 msgr2=0x7f6ec410d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.852+0000 7f6ec3e00640 1 --2- 172.21.15.28:0/774883270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ec410d500 0x7f6ec410d8e0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f6eb4009890 tx=0x7f6eb402a360 comp rx=0 tx=0).stop 2024-04-24T08:21:00.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.852+0000 7f6ec3e00640 1 -- 172.21.15.28:0/774883270 shutdown_connections 2024-04-24T08:21:00.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.852+0000 7f6ec3e00640 1 --2- 172.21.15.28:0/774883270 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec4108580 0x7f6ec4108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.852+0000 7f6ec3e00640 1 --2- 172.21.15.28:0/774883270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ec4108f40 0x7f6ec41153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.852+0000 7f6ec3e00640 1 --2- 172.21.15.28:0/774883270 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ec410d500 0x7f6ec410d8e0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.852+0000 7f6ec3e00640 1 -- 172.21.15.28:0/774883270 >> 172.21.15.28:0/774883270 conn(0x7f6ec4077e20 msgr2=0x7f6ec4106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.852+0000 7f6ec3e00640 1 -- 172.21.15.28:0/774883270 shutdown_connections 2024-04-24T08:21:00.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.852+0000 7f6ec3e00640 1 -- 172.21.15.28:0/774883270 wait complete. 2024-04-24T08:21:00.856 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.853+0000 7f6ec3e00640 1 Processor -- start 2024-04-24T08:21:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.853+0000 7f6ec3e00640 1 -- start start 2024-04-24T08:21:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.853+0000 7f6ec3e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ec4108580 0x7f6ec419ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.853+0000 7f6ec3e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ec4108f40 0x7f6ec419f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.857 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.854+0000 7f6ec3e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec410d500 0x7f6ec41a35e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.854+0000 7f6ec3e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec41188c0 con 0x7f6ec410d500 2024-04-24T08:21:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.854+0000 7f6ec3e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec4118740 con 0x7f6ec4108580 2024-04-24T08:21:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.854+0000 7f6ec3e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec4118a40 con 0x7f6ec4108f40 2024-04-24T08:21:00.858 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.854+0000 7f6ec2a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ec4108580 0x7f6ec419ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.854+0000 7f6ec2a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ec4108580 0x7f6ec419ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49490/0 (socket says 172.21.15.28:49490) 2024-04-24T08:21:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.854+0000 7f6ec2a00640 1 -- 172.21.15.28:0/817504691 learned_addr learned my addr 172.21.15.28:0/817504691 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.854+0000 7f6ec3400640 1 --2- 172.21.15.28:0/817504691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec410d500 0x7f6ec41a35e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.854+0000 7f6ec2000640 1 --2- 172.21.15.28:0/817504691 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ec4108f40 0x7f6ec419f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.854+0000 7f6ec2000640 1 -- 172.21.15.28:0/817504691 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ec4108580 msgr2=0x7f6ec419ed20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.854+0000 7f6ec2000640 1 --2- 172.21.15.28:0/817504691 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ec4108580 0x7f6ec419ed20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.854+0000 7f6ec2000640 1 -- 172.21.15.28:0/817504691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec410d500 msgr2=0x7f6ec41a35e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.854+0000 7f6ec2000640 1 --2- 172.21.15.28:0/817504691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec410d500 0x7f6ec41a35e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.854+0000 7f6ec2000640 1 -- 172.21.15.28:0/817504691 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6ec41a3d10 con 0x7f6ec4108f40 2024-04-24T08:21:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.854+0000 7f6ec3400640 1 --2- 172.21.15.28:0/817504691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec410d500 0x7f6ec41a35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.855+0000 7f6ec2000640 1 --2- 172.21.15.28:0/817504691 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ec4108f40 0x7f6ec419f260 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009fd0 tx=0x7f6eac00c880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.855+0000 7f6ebb400640 1 -- 172.21.15.28:0/817504691 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eac003970 con 0x7f6ec4108f40 2024-04-24T08:21:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.855+0000 7f6ebb400640 1 -- 172.21.15.28:0/817504691 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eac004020 con 0x7f6ec4108f40 2024-04-24T08:21:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.855+0000 7f6ec3e00640 1 -- 172.21.15.28:0/817504691 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6ec41a4000 con 0x7f6ec4108f40 2024-04-24T08:21:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.855+0000 7f6ebb400640 1 -- 172.21.15.28:0/817504691 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eac00ba60 con 0x7f6ec4108f40 2024-04-24T08:21:00.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.855+0000 7f6ec3e00640 1 -- 172.21.15.28:0/817504691 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6ec41ab890 con 0x7f6ec4108f40 2024-04-24T08:21:00.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.858+0000 7f6ecc58c740 1 Processor -- start 2024-04-24T08:21:00.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.859+0000 7f6ecc58c740 1 -- start start 2024-04-24T08:21:00.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.859+0000 7f6ebb400640 1 -- 172.21.15.28:0/817504691 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6eac0041d0 con 0x7f6ec4108f40 2024-04-24T08:21:00.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.859+0000 7f6ecc58c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21664b40 0x563e21664f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.859+0000 7f6ecc58c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21665460 0x563e2166f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.859+0000 7f6ecc58c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e2166fb30 0x563e21671f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.867 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.859+0000 7f6ecc58c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563e215ca1d0 con 0x563e21664b40 2024-04-24T08:21:00.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.859+0000 7f6ecc58c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563e21570450 con 0x563e21665460 2024-04-24T08:21:00.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.859+0000 7f6ecc58c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563e215c6190 con 0x563e2166fb30 2024-04-24T08:21:00.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.859+0000 7f6ec3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21664b40 0x563e21664f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.859+0000 7f6ec2000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e2166fb30 0x563e21671f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.859+0000 7f6ec2a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21665460 0x563e2166f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.859+0000 7f6ec3400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21664b40 0x563e21664f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35828/0 (socket says 172.21.15.28:35828) 2024-04-24T08:21:00.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.859+0000 7f6ec3400640 1 -- 172.21.15.28:0/199230614 learned_addr learned my addr 172.21.15.28:0/199230614 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.860+0000 7f6ec2a00640 1 -- 172.21.15.28:0/199230614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e2166fb30 msgr2=0x563e21671f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.860+0000 7f6ec2a00640 1 --2- 172.21.15.28:0/199230614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e2166fb30 0x563e21671f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.860+0000 7f6ec2a00640 1 -- 172.21.15.28:0/199230614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21664b40 msgr2=0x563e21664f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.860+0000 7f6ec2a00640 1 --2- 172.21.15.28:0/199230614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21664b40 0x563e21664f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.860+0000 7f6ec2a00640 1 -- 172.21.15.28:0/199230614 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e2162d310 con 0x563e21665460 2024-04-24T08:21:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.860+0000 7f6ec3400640 1 --2- 172.21.15.28:0/199230614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21664b40 0x563e21664f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.860+0000 7f6ec2a00640 1 --2- 172.21.15.28:0/199230614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21665460 0x563e2166f5f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6eb402a950 tx=0x7f6eb4005b20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52b626f51e97fd42 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.861+0000 7f6ebb400640 1 --2- 172.21.15.28:0/817504691 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6ea80a3da0 0x7f6ea80a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.861+0000 7f6eb3e00640 1 -- 172.21.15.28:0/199230614 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4041020 con 0x563e21665460 2024-04-24T08:21:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.861+0000 7f6ebb400640 1 -- 172.21.15.28:0/817504691 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7f6eac0d09c0 con 0x7f6ec4108f40 2024-04-24T08:21:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.861+0000 7f6eb3e00640 1 -- 172.21.15.28:0/199230614 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eb4030e80 con 0x563e21665460 2024-04-24T08:21:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.861+0000 7f6eb3e00640 1 -- 172.21.15.28:0/199230614 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4037040 con 0x563e21665460 2024-04-24T08:21:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.861+0000 7f6ec3400640 1 --2- 172.21.15.28:0/817504691 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6ea80a3da0 0x7f6ea80a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.861+0000 7f6ecc58c740 1 -- 172.21.15.28:0/199230614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21665460 msgr2=0x563e2166f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.861+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/199230614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21665460 0x563e2166f5f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f6eb402a950 tx=0x7f6eb4005b20 comp rx=0 tx=0).stop 2024-04-24T08:21:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.862+0000 7f6ecc58c740 1 -- 172.21.15.28:0/199230614 shutdown_connections 2024-04-24T08:21:00.870 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.862+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/199230614 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e2166fb30 0x563e21671f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.862+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/199230614 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21665460 0x563e2166f5f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.862+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/199230614 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21664b40 0x563e21664f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.862+0000 7f6ecc58c740 1 -- 172.21.15.28:0/199230614 >> 172.21.15.28:0/199230614 conn(0x563e21660e60 msgr2=0x563e21661240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.862+0000 7f6ec3400640 1 --2- 172.21.15.28:0/817504691 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6ea80a3da0 0x7f6ea80a6260 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc004650 tx=0x7f6ebc005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.862+0000 7f6ecc58c740 1 -- 172.21.15.28:0/199230614 shutdown_connections 2024-04-24T08:21:00.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.862+0000 7f6ecc58c740 1 -- 172.21.15.28:0/199230614 wait complete. 2024-04-24T08:21:00.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.862+0000 7f6ecc58c740 1 -- start start 2024-04-24T08:21:00.871 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.862+0000 7f6ecc58c740 1 Processor -- start 2024-04-24T08:21:00.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.863+0000 7f6ecc58c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21664b40 0x563e2166f510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.863+0000 7f6ecc58c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21665460 0x563e2166c800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.863+0000 7f6ecc58c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e2166fb30 0x563e2166ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.863+0000 7f6ecc58c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x563e21672960 con 0x563e21665460 2024-04-24T08:21:00.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.863+0000 7f6ecc58c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x563e21583810 con 0x563e21664b40 2024-04-24T08:21:00.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.863+0000 7f6ecc58c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x563e21672ae0 con 0x563e2166fb30 2024-04-24T08:21:00.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.863+0000 7f6ecc58c740 10 client.0 _ll_register_callbacks cb 0x563e21551140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:00.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.863+0000 7f6ec2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21665460 0x563e2166c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.863+0000 7f6ec2a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21664b40 0x563e2166f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.863+0000 7f6ec2a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21664b40 0x563e2166f510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49494/0 (socket says 172.21.15.28:49494) 2024-04-24T08:21:00.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.863+0000 7f6ec2a00640 1 -- 172.21.15.28:0/3755846618 learned_addr learned my addr 172.21.15.28:0/3755846618 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:00.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.863+0000 7f6ec3400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e2166fb30 0x563e2166ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.864+0000 7f6ec2a00640 1 -- 172.21.15.28:0/3755846618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e2166fb30 msgr2=0x563e2166ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.864+0000 7f6ec2a00640 1 --2- 172.21.15.28:0/3755846618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e2166fb30 0x563e2166ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.864+0000 7f6ec2a00640 1 -- 172.21.15.28:0/3755846618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21665460 msgr2=0x563e2166c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.864+0000 7f6ec2a00640 1 --2- 172.21.15.28:0/3755846618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21665460 0x563e2166c800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.864+0000 7f6ec2a00640 1 -- 172.21.15.28:0/3755846618 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e215943f0 con 0x563e21664b40 2024-04-24T08:21:00.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.864+0000 7f6ec2000640 1 --2- 172.21.15.28:0/3755846618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21665460 0x563e2166c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:00.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.864+0000 7f6ec2a00640 1 --2- 172.21.15.28:0/3755846618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21664b40 0x563e2166f510 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6eb40309f0 tx=0x7f6eb4005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8532ba9276253a05 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.864+0000 7f6eb0c00640 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2024-04-24T08:21:00.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.864+0000 7f6eb0c00640 1 -- 172.21.15.28:0/3755846618 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eb404d070 con 0x563e21664b40 2024-04-24T08:21:00.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.865+0000 7f6eb0c00640 1 -- 172.21.15.28:0/3755846618 <== mon.2 v2:172.21.15.117:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7f6eb40036f0 con 0x563e21664b40 2024-04-24T08:21:00.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.865+0000 7f6ecc58c740 1 -- 172.21.15.28:0/3755846618 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6eb4030e80 con 0x563e21664b40 2024-04-24T08:21:00.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.865+0000 7f6eb0c00640 1 -- 172.21.15.28:0/3755846618 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6eb4059050 con 0x563e21664b40 2024-04-24T08:21:00.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.865+0000 7f6ecc58c740 10 client.27116 fetch_fsmap learned FSMap version 26 2024-04-24T08:21:00.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.865+0000 7f6ecc58c740 1 -- 172.21.15.28:0/3755846618 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563e212dc3f0 con 0x563e21664b40 2024-04-24T08:21:00.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.865+0000 7f6eb0c00640 1 -- 172.21.15.28:0/3755846618 <== mon.2 v2:172.21.15.117:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eb4030e80 con 0x563e21664b40 2024-04-24T08:21:00.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.865+0000 7f6eb0c00640 1 -- 172.21.15.28:0/3755846618 <== mon.2 v2:172.21.15.117:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6eb403d840 con 0x563e21664b40 2024-04-24T08:21:00.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.866+0000 7f6ecc58c740 10 client.27116 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:21:00.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.866+0000 7f6ecc58c740 10 client.27116 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:21:00.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.866+0000 7f6ecc58c740 20 client.27116 populate_metadata read hostname 'smithi028' 2024-04-24T08:21:00.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.866+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/3755846618 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563e2167fad0 0x563e21681ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.875 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.866+0000 7f6ecc58c740 4 client.27116 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:21:00.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.866+0000 7f6ecc58c740 1 -- 172.21.15.28:0/3755846618 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563e215d7bf0 con 0x563e2167fad0 2024-04-24T08:21:00.876 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.866+0000 7f6ec2000640 1 --2- 172.21.15.28:0/3755846618 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563e2167fad0 0x563e21681ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.867+0000 7f6ec2000640 1 --2- 172.21.15.28:0/3755846618 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563e2167fad0 0x563e21681ec0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=16bed0a6b7ee1e42 server_cookie=677c80a01fd0c0a5 in_seq=0 out_seq=0 2024-04-24T08:21:00.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.867+0000 7f6eb0c00640 10 client.27116 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:21:00.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.867+0000 7f6eb0c00640 1 -- 172.21.15.28:0/3755846618 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563e215d7bf0 con 0x563e2167fad0 2024-04-24T08:21:00.877 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.867+0000 7f6eb0c00640 10 client.27116 handle_command_reply: tid=0 2024-04-24T08:21:00.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.868+0000 7f6ecc58c740 1 client.27116 shutdown 2024-04-24T08:21:00.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.868+0000 7f6ecc58c740 1 -- 172.21.15.28:0/3755846618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21664b40 msgr2=0x563e2166f510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.868+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/3755846618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21664b40 0x563e2166f510 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f6eb40309f0 tx=0x7f6eb4005850 comp rx=0 tx=0).stop 2024-04-24T08:21:00.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.868+0000 7f6ecc58c740 1 -- 172.21.15.28:0/3755846618 shutdown_connections 2024-04-24T08:21:00.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.868+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/3755846618 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563e2167fad0 0x563e21681ec0 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.868+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/3755846618 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21665460 0x563e2166c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.868+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/3755846618 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e2166fb30 0x563e2166ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.878 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.868+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/3755846618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21664b40 0x563e2166f510 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.868+0000 7f6eb0c00640 0 client.27116 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:21:00.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.868+0000 7f6ecc58c740 1 -- 172.21.15.28:0/3755846618 >> 172.21.15.28:0/3755846618 conn(0x563e21660e60 msgr2=0x563e21666b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.868+0000 7f6ecc58c740 1 -- 172.21.15.28:0/3755846618 shutdown_connections 2024-04-24T08:21:00.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.868+0000 7f6ecc58c740 1 -- 172.21.15.28:0/3755846618 wait complete. 2024-04-24T08:21:00.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.868+0000 7f6ecc58c740 20 client.27116 trim_cache size 0 max 16384 2024-04-24T08:21:00.884 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.883+0000 7f6ecc58c740 1 Processor -- start 2024-04-24T08:21:00.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.883+0000 7f6ecc58c740 1 -- start start 2024-04-24T08:21:00.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.883+0000 7f6ecc58c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e21664b40 0x563e216672a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.883+0000 7f6ecc58c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21665460 0x563e2166b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.885 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.883+0000 7f6ecc58c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e2166fb30 0x563e2166f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.883+0000 7f6ecc58c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec41b6b90 con 0x563e21665460 2024-04-24T08:21:00.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.883+0000 7f6ecc58c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec41b4460 con 0x563e2166fb30 2024-04-24T08:21:00.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.883+0000 7f6ecc58c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec41b22d0 con 0x563e21664b40 2024-04-24T08:21:00.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.884+0000 7f6ec2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21665460 0x563e2166b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.884+0000 7f6ec2000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21665460 0x563e2166b6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35834/0 (socket says 172.21.15.28:35834) 2024-04-24T08:21:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.884+0000 7f6ec2000640 1 -- 172.21.15.28:0/3906982993 learned_addr learned my addr 172.21.15.28:0/3906982993 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.884+0000 7f6ec3400640 1 --2- 172.21.15.28:0/3906982993 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e2166fb30 0x563e2166f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.884+0000 7f6ec2a00640 1 --2- 172.21.15.28:0/3906982993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e21664b40 0x563e216672a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.884+0000 7f6ec2000640 1 -- 172.21.15.28:0/3906982993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e21664b40 msgr2=0x563e216672a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.884+0000 7f6ec2000640 1 --2- 172.21.15.28:0/3906982993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e21664b40 0x563e216672a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.884+0000 7f6ec2000640 1 -- 172.21.15.28:0/3906982993 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e2166fb30 msgr2=0x563e2166f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.884+0000 7f6ec2000640 1 --2- 172.21.15.28:0/3906982993 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e2166fb30 0x563e2166f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.884+0000 7f6ec2000640 1 -- 172.21.15.28:0/3906982993 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563e2166d180 con 0x563e21665460 2024-04-24T08:21:00.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.884+0000 7f6ec3400640 1 --2- 172.21.15.28:0/3906982993 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e2166fb30 0x563e2166f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:00.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.885+0000 7f6ec2000640 1 --2- 172.21.15.28:0/3906982993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21665460 0x563e2166b6b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6eac099230 tx=0x7f6eac0d2e10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fda99b4b1d825ca8 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.898 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.885+0000 7f6eb3400640 1 -- 172.21.15.28:0/3906982993 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eac09c8b0 con 0x563e21665460 2024-04-24T08:21:00.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.885+0000 7f6eb3400640 1 -- 172.21.15.28:0/3906982993 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eac0a0c60 con 0x563e21665460 2024-04-24T08:21:00.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.885+0000 7f6eb3400640 1 -- 172.21.15.28:0/3906982993 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6eac004680 con 0x563e21665460 2024-04-24T08:21:00.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.886+0000 7f6ecc58c740 1 -- 172.21.15.28:0/3906982993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21665460 msgr2=0x563e2166b6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.899 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.886+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/3906982993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21665460 0x563e2166b6b0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f6eac099230 tx=0x7f6eac0d2e10 comp rx=0 tx=0).stop 2024-04-24T08:21:00.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.886+0000 7f6ecc58c740 1 -- 172.21.15.28:0/3906982993 shutdown_connections 2024-04-24T08:21:00.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.886+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/3906982993 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e21665460 0x563e2166b6b0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.900 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.886+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/3906982993 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e2166fb30 0x563e2166f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.886+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/3906982993 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e21664b40 0x563e216672a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.886+0000 7f6ecc58c740 1 -- 172.21.15.28:0/3906982993 >> 172.21.15.28:0/3906982993 conn(0x563e21660e60 msgr2=0x563e2166afa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.901 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.886+0000 7f6ecc58c740 1 -- 172.21.15.28:0/3906982993 shutdown_connections 2024-04-24T08:21:00.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.886+0000 7f6ecc58c740 1 -- 172.21.15.28:0/3906982993 wait complete. 2024-04-24T08:21:00.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.886+0000 7f6ecc58c740 1 -- start start 2024-04-24T08:21:00.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.886+0000 7f6ecc58c740 1 Processor -- start 2024-04-24T08:21:00.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.887+0000 7f6ecc58c740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21664b40 0x563e21678200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.887+0000 7f6ecc58c740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e21665460 0x563e21669e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.887+0000 7f6ecc58c740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e2166fb30 0x563e2166a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.887+0000 7f6ecc58c740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec41b2920 con 0x563e2166fb30 2024-04-24T08:21:00.903 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.887+0000 7f6ecc58c740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec41b3d90 con 0x563e21664b40 2024-04-24T08:21:00.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.887+0000 7f6ecc58c740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6ec41b2ff0 con 0x563e21665460 2024-04-24T08:21:00.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.887+0000 7f6ecc58c740 10 client.0 _ll_register_callbacks cb 0x7f6ebc009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:00.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.887+0000 7f6ec2a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21664b40 0x563e21678200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.887+0000 7f6ec2a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21664b40 0x563e21678200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49510/0 (socket says 172.21.15.28:49510) 2024-04-24T08:21:00.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.887+0000 7f6ec2a00640 1 -- 172.21.15.28:0/4181242997 learned_addr learned my addr 172.21.15.28:0/4181242997 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:00.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.887+0000 7f6ec3400640 1 --2- 172.21.15.28:0/4181242997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e2166fb30 0x563e2166a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.888+0000 7f6ec2000640 1 --2- 172.21.15.28:0/4181242997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e21665460 0x563e21669e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.888+0000 7f6ec3400640 1 -- 172.21.15.28:0/4181242997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e21665460 msgr2=0x563e21669e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.888+0000 7f6ec3400640 1 --2- 172.21.15.28:0/4181242997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e21665460 0x563e21669e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.888+0000 7f6ec3400640 1 -- 172.21.15.28:0/4181242997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21664b40 msgr2=0x563e21678200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:00.906 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.888+0000 7f6ec3400640 1 --2- 172.21.15.28:0/4181242997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21664b40 0x563e21678200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.888+0000 7f6ec3400640 1 -- 172.21.15.28:0/4181242997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563e213a67d0 con 0x563e2166fb30 2024-04-24T08:21:00.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.888+0000 7f6ec2a00640 1 --2- 172.21.15.28:0/4181242997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21664b40 0x563e21678200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:00.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.888+0000 7f6ec3400640 1 --2- 172.21.15.28:0/4181242997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e2166fb30 0x563e2166a470 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc00f790 tx=0x7f6ebc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed24f6fe771973b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:00.907 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.888+0000 7f6eb0c00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:21:00.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.889+0000 7f6eb0c00640 1 -- 172.21.15.28:0/4181242997 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ebc01db20 con 0x563e2166fb30 2024-04-24T08:21:00.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.889+0000 7f6eb0c00640 1 -- 172.21.15.28:0/4181242997 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7f6ebc01dcc0 con 0x563e2166fb30 2024-04-24T08:21:00.908 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.889+0000 7f6ecc58c740 1 -- 172.21.15.28:0/4181242997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6eac0a0c60 con 0x563e2166fb30 2024-04-24T08:21:00.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.889+0000 7f6eb0c00640 1 -- 172.21.15.28:0/4181242997 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f6ebc037050 con 0x563e2166fb30 2024-04-24T08:21:00.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.889+0000 7f6ecc58c740 10 client.16845 fetch_fsmap learned FSMap version 26 2024-04-24T08:21:00.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.889+0000 7f6ecc58c740 1 -- 172.21.15.28:0/4181242997 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x563e216e6050 con 0x563e2166fb30 2024-04-24T08:21:00.909 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.889+0000 7f6eb0c00640 1 -- 172.21.15.28:0/4181242997 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6eac0a0c60 con 0x563e2166fb30 2024-04-24T08:21:00.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.890+0000 7f6eb0c00640 1 -- 172.21.15.28:0/4181242997 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f6ebc004b70 con 0x563e2166fb30 2024-04-24T08:21:00.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.890+0000 7f6ecc58c740 10 client.16845 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:21:00.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.890+0000 7f6ecc58c740 10 client.16845 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.i 2024-04-24T08:21:00.910 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.890+0000 7f6ecc58c740 20 client.16845 populate_metadata read hostname 'smithi028' 2024-04-24T08:21:00.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.890+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/4181242997 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563e216e6850 0x563e216e8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:00.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.890+0000 7f6ecc58c740 4 client.16845 mds_command: new command op to 24556 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:21:00.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.890+0000 7f6ecc58c740 1 -- 172.21.15.28:0/4181242997 --> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] -- command(tid 0: {"prefix": "damage ls"}) -- 0x563e215d1a60 con 0x563e216e6850 2024-04-24T08:21:00.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.890+0000 7f6ec2a00640 1 --2- 172.21.15.28:0/4181242997 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563e216e6850 0x563e216e8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:00.911 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.890+0000 7f6ec2a00640 1 --2- 172.21.15.28:0/4181242997 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563e216e6850 0x563e216e8c70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=19c44e0e66b057c6 server_cookie=915197f2c259a5d9 in_seq=0 out_seq=0 2024-04-24T08:21:00.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.890+0000 7f6eb0c00640 10 client.16845 ms_handle_connect on v2:172.21.15.194:6838/2785495353 2024-04-24T08:21:00.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.891+0000 7f6eb0c00640 1 -- 172.21.15.28:0/4181242997 <== mds.2 v2:172.21.15.194:6838/2785495353 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x563e215d1a60 con 0x563e216e6850 2024-04-24T08:21:00.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.891+0000 7f6eb0c00640 10 client.16845 handle_command_reply: tid=0 2024-04-24T08:21:00.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.891+0000 7f6ecc58c740 1 client.16845 shutdown 2024-04-24T08:21:00.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.891+0000 7f6ecc58c740 1 -- 172.21.15.28:0/4181242997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e2166fb30 msgr2=0x563e2166a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.891+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/4181242997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e2166fb30 0x563e2166a470 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc00f790 tx=0x7f6ebc004730 comp rx=0 tx=0).stop 2024-04-24T08:21:00.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.891+0000 7f6ecc58c740 1 -- 172.21.15.28:0/4181242997 shutdown_connections 2024-04-24T08:21:00.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.891+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/4181242997 >> [v2:172.21.15.194:6838/2785495353,v1:172.21.15.194:6839/2785495353] conn(0x563e216e6850 0x563e216e8c70 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.912 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.891+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/4181242997 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x563e2166fb30 0x563e2166a470 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.891+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/4181242997 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x563e21665460 0x563e21669e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.891+0000 7f6ecc58c740 1 --2- 172.21.15.28:0/4181242997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x563e21664b40 0x563e21678200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.891+0000 7f6eb0c00640 0 client.16845 ms_handle_reset on v2:172.21.15.194:6838/2785495353 2024-04-24T08:21:00.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.891+0000 7f6ecc58c740 1 -- 172.21.15.28:0/4181242997 >> 172.21.15.28:0/4181242997 conn(0x563e21660e60 msgr2=0x563e2166b830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.891+0000 7f6ecc58c740 1 -- 172.21.15.28:0/4181242997 shutdown_connections 2024-04-24T08:21:00.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.891+0000 7f6ecc58c740 1 -- 172.21.15.28:0/4181242997 wait complete. 2024-04-24T08:21:00.913 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.891+0000 7f6ecc58c740 20 client.16845 trim_cache size 0 max 16384 2024-04-24T08:21:00.914 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:21:00.915 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.892+0000 7f6ec3e00640 1 -- 172.21.15.28:0/817504691 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6ea80a3da0 msgr2=0x7f6ea80a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.892+0000 7f6ec3e00640 1 --2- 172.21.15.28:0/817504691 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6ea80a3da0 0x7f6ea80a6260 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6ebc004650 tx=0x7f6ebc005f90 comp rx=0 tx=0).stop 2024-04-24T08:21:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.892+0000 7f6ec3e00640 1 -- 172.21.15.28:0/817504691 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ec4108f40 msgr2=0x7f6ec419f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.892+0000 7f6ec3e00640 1 --2- 172.21.15.28:0/817504691 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ec4108f40 0x7f6ec419f260 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f6eac009fd0 tx=0x7f6eac00c880 comp rx=0 tx=0).stop 2024-04-24T08:21:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.892+0000 7f6ec3e00640 1 -- 172.21.15.28:0/817504691 shutdown_connections 2024-04-24T08:21:00.916 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.892+0000 7f6ec3e00640 1 --2- 172.21.15.28:0/817504691 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6ea80a3da0 0x7f6ea80a6260 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.892+0000 7f6ec3e00640 1 --2- 172.21.15.28:0/817504691 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6ec410d500 0x7f6ec41a35e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.892+0000 7f6ec3e00640 1 --2- 172.21.15.28:0/817504691 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6ec4108f40 0x7f6ec419f260 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.892+0000 7f6ec3e00640 1 --2- 172.21.15.28:0/817504691 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ec4108580 0x7f6ec419ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.892+0000 7f6ec3e00640 1 -- 172.21.15.28:0/817504691 >> 172.21.15.28:0/817504691 conn(0x7f6ec4077e20 msgr2=0x7f6ec4078340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.892+0000 7f6ec3e00640 1 -- 172.21.15.28:0/817504691 shutdown_connections 2024-04-24T08:21:00.917 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:00.892+0000 7f6ec3e00640 1 -- 172.21.15.28:0/817504691 wait complete. 2024-04-24T08:21:00.917 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-04-24T08:21:01.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.128+0000 7f54d2600640 1 -- 172.21.15.28:0/1122883709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54cc108f40 msgr2=0x7f54cc1153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.128+0000 7f54d2600640 1 --2- 172.21.15.28:0/1122883709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54cc108f40 0x7f54cc1153b0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f54c4009ca0 tx=0x7f54c402a8c0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.128+0000 7f54d2600640 1 -- 172.21.15.28:0/1122883709 shutdown_connections 2024-04-24T08:21:01.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.128+0000 7f54d2600640 1 --2- 172.21.15.28:0/1122883709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54cc108f40 0x7f54cc1153b0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.128+0000 7f54d2600640 1 --2- 172.21.15.28:0/1122883709 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54cc108580 0x7f54cc108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.128+0000 7f54d2600640 1 --2- 172.21.15.28:0/1122883709 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc10d500 0x7f54cc10d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.128+0000 7f54d2600640 1 -- 172.21.15.28:0/1122883709 >> 172.21.15.28:0/1122883709 conn(0x7f54cc077e20 msgr2=0x7f54cc106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.128+0000 7f54d2600640 1 -- 172.21.15.28:0/1122883709 shutdown_connections 2024-04-24T08:21:01.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.128+0000 7f54d2600640 1 -- 172.21.15.28:0/1122883709 wait complete. 2024-04-24T08:21:01.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.129+0000 7f54d2600640 1 Processor -- start 2024-04-24T08:21:01.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.129+0000 7f54d2600640 1 -- start start 2024-04-24T08:21:01.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.129+0000 7f54d2600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc108580 0x7f54cc1a4900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.132 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.129+0000 7f54d2600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54cc108f40 0x7f54cc1a4e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.129+0000 7f54d2600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54cc10d500 0x7f54cc19eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.129+0000 7f54d2600640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc118920 con 0x7f54cc108580 2024-04-24T08:21:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.129+0000 7f54d2600640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc1187a0 con 0x7f54cc108f40 2024-04-24T08:21:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.129+0000 7f54d2600640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc118aa0 con 0x7f54cc10d500 2024-04-24T08:21:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.130+0000 7f54d1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc108580 0x7f54cc1a4900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.130+0000 7f54d1200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc108580 0x7f54cc1a4900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35852/0 (socket says 172.21.15.28:35852) 2024-04-24T08:21:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.130+0000 7f54d1200640 1 -- 172.21.15.28:0/4253646339 learned_addr learned my addr 172.21.15.28:0/4253646339 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:01.133 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.130+0000 7f54cbe00640 1 --2- 172.21.15.28:0/4253646339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54cc108f40 0x7f54cc1a4e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.130+0000 7f54d1200640 1 -- 172.21.15.28:0/4253646339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54cc10d500 msgr2=0x7f54cc19eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.130+0000 7f54d1200640 1 --2- 172.21.15.28:0/4253646339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54cc10d500 0x7f54cc19eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.130+0000 7f54d1200640 1 -- 172.21.15.28:0/4253646339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54cc108f40 msgr2=0x7f54cc1a4e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.130+0000 7f54d1200640 1 --2- 172.21.15.28:0/4253646339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54cc108f40 0x7f54cc1a4e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.130+0000 7f54d1200640 1 -- 172.21.15.28:0/4253646339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54cc19f280 con 0x7f54cc108580 2024-04-24T08:21:01.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.130+0000 7f54cbe00640 1 --2- 172.21.15.28:0/4253646339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54cc108f40 0x7f54cc1a4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:21:01.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.130+0000 7f54d1200640 1 --2- 172.21.15.28:0/4253646339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc108580 0x7f54cc1a4900 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f54c000d950 tx=0x7f54c000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.131+0000 7f54c9600640 1 -- 172.21.15.28:0/4253646339 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c0017020 con 0x7f54cc108580 2024-04-24T08:21:01.134 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.131+0000 7f54c9600640 1 -- 172.21.15.28:0/4253646339 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54c0009c30 con 0x7f54cc108580 2024-04-24T08:21:01.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.131+0000 7f54d2600640 1 -- 172.21.15.28:0/4253646339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54cc19f570 con 0x7f54cc108580 2024-04-24T08:21:01.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.131+0000 7f54c9600640 1 -- 172.21.15.28:0/4253646339 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c000f720 con 0x7f54cc108580 2024-04-24T08:21:01.135 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.131+0000 7f54d2600640 1 -- 172.21.15.28:0/4253646339 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54cc1ab7c0 con 0x7f54cc108580 2024-04-24T08:21:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.134+0000 7f54d6a5b740 1 Processor -- start 2024-04-24T08:21:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.134+0000 7f54d6a5b740 1 -- start start 2024-04-24T08:21:01.137 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.135+0000 7f54c9600640 1 -- 172.21.15.28:0/4253646339 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f54c001a020 con 0x7f54cc108580 2024-04-24T08:21:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.135+0000 7f54d6a5b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d90b40 0x558af5d90f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.135+0000 7f54d6a5b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d91460 0x558af5d9b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.135+0000 7f54d6a5b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d9bb30 0x558af5d9df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.135+0000 7f54d6a5b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558af5cf61d0 con 0x558af5d9bb30 2024-04-24T08:21:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.135+0000 7f54d6a5b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558af5c9c450 con 0x558af5d91460 2024-04-24T08:21:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.135+0000 7f54d6a5b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558af5cf2190 con 0x558af5d90b40 2024-04-24T08:21:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.135+0000 7f54d1c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d90b40 0x558af5d90f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.138 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.135+0000 7f54cbe00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d9bb30 0x558af5d9df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.135+0000 7f54d1c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d90b40 0x558af5d90f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:55654/0 (socket says 172.21.15.28:55654) 2024-04-24T08:21:01.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.135+0000 7f54d1c00640 1 -- 172.21.15.28:0/880991898 learned_addr learned my addr 172.21.15.28:0/880991898 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:01.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.135+0000 7f54d1200640 1 --2- 172.21.15.28:0/880991898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d91460 0x558af5d9b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.136+0000 7f54d1c00640 1 -- 172.21.15.28:0/880991898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d91460 msgr2=0x558af5d9b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.136+0000 7f54d1c00640 1 --2- 172.21.15.28:0/880991898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d91460 0x558af5d9b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.136+0000 7f54d1c00640 1 -- 172.21.15.28:0/880991898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d9bb30 msgr2=0x558af5d9df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.136+0000 7f54d1c00640 1 --2- 172.21.15.28:0/880991898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d9bb30 0x558af5d9df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.136+0000 7f54d1c00640 1 -- 172.21.15.28:0/880991898 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558af5d59310 con 0x558af5d90b40 2024-04-24T08:21:01.139 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.136+0000 7f54cbe00640 1 --2- 172.21.15.28:0/880991898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d9bb30 0x558af5d9df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:01.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.136+0000 7f54d1c00640 1 --2- 172.21.15.28:0/880991898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d90b40 0x558af5d90f20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f54c402ade0 tx=0x7f54c4009800 comp rx=0 tx=0).ready entity=mon.1 client_cookie=165ad801f2cbcf29 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.136+0000 7f54ba000640 1 -- 172.21.15.28:0/880991898 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c4031da0 con 0x558af5d90b40 2024-04-24T08:21:01.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.136+0000 7f54ba000640 1 -- 172.21.15.28:0/880991898 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54c403f070 con 0x558af5d90b40 2024-04-24T08:21:01.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.136+0000 7f54ba000640 1 -- 172.21.15.28:0/880991898 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c40042a0 con 0x558af5d90b40 2024-04-24T08:21:01.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.136+0000 7f54c9600640 1 --2- 172.21.15.28:0/4253646339 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f54b40a3d20 0x7f54b40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.137+0000 7f54c9600640 1 -- 172.21.15.28:0/4253646339 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7f54c00d07b0 con 0x7f54cc108580 2024-04-24T08:21:01.140 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.137+0000 7f54d1200640 1 --2- 172.21.15.28:0/4253646339 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f54b40a3d20 0x7f54b40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.137+0000 7f54d6a5b740 1 -- 172.21.15.28:0/880991898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d90b40 msgr2=0x558af5d90f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.137+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/880991898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d90b40 0x558af5d90f20 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f54c402ade0 tx=0x7f54c4009800 comp rx=0 tx=0).stop 2024-04-24T08:21:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.137+0000 7f54d6a5b740 1 -- 172.21.15.28:0/880991898 shutdown_connections 2024-04-24T08:21:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.137+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/880991898 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d9bb30 0x558af5d9df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.137+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/880991898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d91460 0x558af5d9b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.137+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/880991898 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d90b40 0x558af5d90f20 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.145 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.137+0000 7f54d6a5b740 1 -- 172.21.15.28:0/880991898 >> 172.21.15.28:0/880991898 conn(0x558af5d8ce60 msgr2=0x558af5d8d240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.138+0000 7f54d6a5b740 1 -- 172.21.15.28:0/880991898 shutdown_connections 2024-04-24T08:21:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.138+0000 7f54d1200640 1 --2- 172.21.15.28:0/4253646339 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f54b40a3d20 0x7f54b40a61e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f54c00cdf60 tx=0x7f54c00d6550 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.138+0000 7f54d6a5b740 1 -- 172.21.15.28:0/880991898 wait complete. 2024-04-24T08:21:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.138+0000 7f54d6a5b740 1 -- start start 2024-04-24T08:21:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.138+0000 7f54d6a5b740 1 Processor -- start 2024-04-24T08:21:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.139+0000 7f54d6a5b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d90b40 0x558af5d9b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.139+0000 7f54d6a5b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d91460 0x558af5d98890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.146 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.139+0000 7f54d6a5b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d9bb30 0x558af5d98fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.139+0000 7f54d6a5b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x558af5d9e960 con 0x558af5d90b40 2024-04-24T08:21:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.139+0000 7f54d6a5b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x558af5caf810 con 0x558af5d9bb30 2024-04-24T08:21:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.139+0000 7f54d6a5b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x558af5d9eae0 con 0x558af5d91460 2024-04-24T08:21:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.139+0000 7f54d6a5b740 10 client.0 _ll_register_callbacks cb 0x558af5c7d140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.139+0000 7f54d1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d90b40 0x558af5d9b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.139+0000 7f54d1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d90b40 0x558af5d9b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35864/0 (socket says 172.21.15.28:35864) 2024-04-24T08:21:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.139+0000 7f54d1c00640 1 -- 172.21.15.28:0/3495771175 learned_addr learned my addr 172.21.15.28:0/3495771175 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.139+0000 7f54cbe00640 1 --2- 172.21.15.28:0/3495771175 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d91460 0x558af5d98890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.147 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.139+0000 7f54d1c00640 1 --2- 172.21.15.28:0/3495771175 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d9bb30 0x558af5d98fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.140+0000 7f54d1c00640 1 -- 172.21.15.28:0/3495771175 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d91460 msgr2=0x558af5d98890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.140+0000 7f54d1c00640 1 --2- 172.21.15.28:0/3495771175 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d91460 0x558af5d98890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.140+0000 7f54d1c00640 1 -- 172.21.15.28:0/3495771175 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d9bb30 msgr2=0x558af5d98fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.140+0000 7f54d1c00640 1 --2- 172.21.15.28:0/3495771175 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d9bb30 0x558af5d98fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.140+0000 7f54d1c00640 1 -- 172.21.15.28:0/3495771175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558af5cc03f0 con 0x558af5d90b40 2024-04-24T08:21:01.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.140+0000 7f54d1c00640 1 --2- 172.21.15.28:0/3495771175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d90b40 0x558af5d9b5a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f54c4031a70 tx=0x7f54c4005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ddef4a8d9a6eccc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.140+0000 7f54aea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:21:01.148 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.140+0000 7f54aea00640 1 -- 172.21.15.28:0/3495771175 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c4051060 con 0x558af5d90b40 2024-04-24T08:21:01.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.140+0000 7f54aea00640 1 -- 172.21.15.28:0/3495771175 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7f54c4004240 con 0x558af5d90b40 2024-04-24T08:21:01.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.140+0000 7f54d6a5b740 1 -- 172.21.15.28:0/3495771175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54c403f070 con 0x558af5d90b40 2024-04-24T08:21:01.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.140+0000 7f54aea00640 1 -- 172.21.15.28:0/3495771175 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f54c4031450 con 0x558af5d90b40 2024-04-24T08:21:01.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.140+0000 7f54d6a5b740 10 client.16857 fetch_fsmap learned FSMap version 26 2024-04-24T08:21:01.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.140+0000 7f54d6a5b740 1 -- 172.21.15.28:0/3495771175 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558af5a083f0 con 0x558af5d90b40 2024-04-24T08:21:01.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.141+0000 7f54aea00640 1 -- 172.21.15.28:0/3495771175 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54c403f070 con 0x558af5d90b40 2024-04-24T08:21:01.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.141+0000 7f54aea00640 1 -- 172.21.15.28:0/3495771175 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f54c402fa80 con 0x558af5d90b40 2024-04-24T08:21:01.149 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.141+0000 7f54d6a5b740 10 client.16857 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:21:01.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.141+0000 7f54d6a5b740 10 client.16857 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:21:01.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.141+0000 7f54d6a5b740 20 client.16857 populate_metadata read hostname 'smithi028' 2024-04-24T08:21:01.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.141+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/3495771175 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x558af5dabbd0 0x558af5dadfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.141+0000 7f54d6a5b740 4 client.16857 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:21:01.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.141+0000 7f54d6a5b740 1 -- 172.21.15.28:0/3495771175 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558af5d03bf0 con 0x558af5dabbd0 2024-04-24T08:21:01.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.142+0000 7f54cbe00640 1 --2- 172.21.15.28:0/3495771175 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x558af5dabbd0 0x558af5dadfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.142+0000 7f54cbe00640 1 --2- 172.21.15.28:0/3495771175 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x558af5dabbd0 0x558af5dadfc0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=290b1e658dc5673d server_cookie=95abf8c2c4041670 in_seq=0 out_seq=0 2024-04-24T08:21:01.150 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.142+0000 7f54aea00640 10 client.16857 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:21:01.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.143+0000 7f54aea00640 1 -- 172.21.15.28:0/3495771175 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558af5d03bf0 con 0x558af5dabbd0 2024-04-24T08:21:01.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.143+0000 7f54aea00640 10 client.16857 handle_command_reply: tid=0 2024-04-24T08:21:01.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.143+0000 7f54d6a5b740 1 client.16857 shutdown 2024-04-24T08:21:01.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.143+0000 7f54d6a5b740 1 -- 172.21.15.28:0/3495771175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d90b40 msgr2=0x558af5d9b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.143+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/3495771175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d90b40 0x558af5d9b5a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f54c4031a70 tx=0x7f54c4005bd0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.143+0000 7f54d6a5b740 1 -- 172.21.15.28:0/3495771175 shutdown_connections 2024-04-24T08:21:01.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.143+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/3495771175 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x558af5dabbd0 0x558af5dadfc0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.151 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.143+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/3495771175 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d9bb30 0x558af5d98fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.143+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/3495771175 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d91460 0x558af5d98890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.143+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/3495771175 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d90b40 0x558af5d9b5a0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.143+0000 7f54aea00640 0 client.16857 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:21:01.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.143+0000 7f54d6a5b740 1 -- 172.21.15.28:0/3495771175 >> 172.21.15.28:0/3495771175 conn(0x558af5d8ce60 msgr2=0x558af5d92b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.144+0000 7f54d6a5b740 1 -- 172.21.15.28:0/3495771175 shutdown_connections 2024-04-24T08:21:01.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.144+0000 7f54d6a5b740 1 -- 172.21.15.28:0/3495771175 wait complete. 2024-04-24T08:21:01.152 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.144+0000 7f54d6a5b740 20 client.16857 trim_cache size 0 max 16384 2024-04-24T08:21:01.159 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.159+0000 7f54d6a5b740 1 Processor -- start 2024-04-24T08:21:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.159+0000 7f54d6a5b740 1 -- start start 2024-04-24T08:21:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.159+0000 7f54d6a5b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d90b40 0x558af5da74a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.159+0000 7f54d6a5b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d91460 0x558af5d9b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.159+0000 7f54d6a5b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d9bb30 0x558af5db0fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.159+0000 7f54d6a5b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc1b6ba0 con 0x558af5d9bb30 2024-04-24T08:21:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.159+0000 7f54d6a5b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc1b4470 con 0x558af5d91460 2024-04-24T08:21:01.160 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.159+0000 7f54d6a5b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc1b22e0 con 0x558af5d90b40 2024-04-24T08:21:01.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.160+0000 7f54cbe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d91460 0x558af5d9b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.160+0000 7f54cbe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d91460 0x558af5d9b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49554/0 (socket says 172.21.15.28:49554) 2024-04-24T08:21:01.164 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.160+0000 7f54cbe00640 1 -- 172.21.15.28:0/3734836824 learned_addr learned my addr 172.21.15.28:0/3734836824 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:01.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.160+0000 7f54d1c00640 1 --2- 172.21.15.28:0/3734836824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d90b40 0x558af5da74a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.160+0000 7f54d1c00640 1 --2- 172.21.15.28:0/3734836824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d9bb30 0x558af5db0fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.160+0000 7f54cbe00640 1 -- 172.21.15.28:0/3734836824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d90b40 msgr2=0x558af5da74a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.160+0000 7f54cbe00640 1 --2- 172.21.15.28:0/3734836824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d90b40 0x558af5da74a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.160+0000 7f54cbe00640 1 -- 172.21.15.28:0/3734836824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d9bb30 msgr2=0x558af5db0fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.160+0000 7f54cbe00640 1 --2- 172.21.15.28:0/3734836824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d9bb30 0x558af5db0fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.160+0000 7f54cbe00640 1 -- 172.21.15.28:0/3734836824 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558af5ad27d0 con 0x558af5d91460 2024-04-24T08:21:01.165 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.160+0000 7f54d1c00640 1 --2- 172.21.15.28:0/3734836824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d9bb30 0x558af5db0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.160+0000 7f54cbe00640 1 --2- 172.21.15.28:0/3734836824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d91460 0x558af5d9b0d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f54cc107ed0 tx=0x7f54bc0138f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=59a72db5f6b8bb2b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.161+0000 7f54b9600640 1 -- 172.21.15.28:0/3734836824 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc023070 con 0x558af5d91460 2024-04-24T08:21:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.161+0000 7f54b9600640 1 -- 172.21.15.28:0/3734836824 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54bc01d550 con 0x558af5d91460 2024-04-24T08:21:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.161+0000 7f54b9600640 1 -- 172.21.15.28:0/3734836824 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54bc010ba0 con 0x558af5d91460 2024-04-24T08:21:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.161+0000 7f54d6a5b740 1 -- 172.21.15.28:0/3734836824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d91460 msgr2=0x558af5d9b0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.161+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/3734836824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d91460 0x558af5d9b0d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f54cc107ed0 tx=0x7f54bc0138f0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.161+0000 7f54d6a5b740 1 -- 172.21.15.28:0/3734836824 shutdown_connections 2024-04-24T08:21:01.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.161+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/3734836824 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d9bb30 0x558af5db0fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.161+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/3734836824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d91460 0x558af5d9b0d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.161+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/3734836824 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d90b40 0x558af5da74a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.161+0000 7f54d6a5b740 1 -- 172.21.15.28:0/3734836824 >> 172.21.15.28:0/3734836824 conn(0x558af5d8ce60 msgr2=0x558af5da6d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.161+0000 7f54d6a5b740 1 -- 172.21.15.28:0/3734836824 shutdown_connections 2024-04-24T08:21:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.161+0000 7f54d6a5b740 1 -- 172.21.15.28:0/3734836824 wait complete. 2024-04-24T08:21:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.162+0000 7f54d6a5b740 1 -- start start 2024-04-24T08:21:01.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.162+0000 7f54d6a5b740 1 Processor -- start 2024-04-24T08:21:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.162+0000 7f54d6a5b740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d90b40 0x558af5dad270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.162+0000 7f54d6a5b740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d91460 0x558af5dad7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.162+0000 7f54d6a5b740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d9bb30 0x558af5e0ade0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.162+0000 7f54d6a5b740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc1b2930 con 0x558af5d90b40 2024-04-24T08:21:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.162+0000 7f54d6a5b740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc1b3da0 con 0x558af5d91460 2024-04-24T08:21:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.162+0000 7f54d6a5b740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f54cc1b3000 con 0x558af5d9bb30 2024-04-24T08:21:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.162+0000 7f54d6a5b740 10 client.0 _ll_register_callbacks cb 0x7f54bc00b440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.163+0000 7f54d1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d90b40 0x558af5dad270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.168 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.163+0000 7f54d1c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d90b40 0x558af5dad270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35886/0 (socket says 172.21.15.28:35886) 2024-04-24T08:21:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.163+0000 7f54d1c00640 1 -- 172.21.15.28:0/1967549244 learned_addr learned my addr 172.21.15.28:0/1967549244 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.163+0000 7f54cbe00640 1 --2- 172.21.15.28:0/1967549244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d91460 0x558af5dad7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.163+0000 7f54d1c00640 1 --2- 172.21.15.28:0/1967549244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d9bb30 0x558af5e0ade0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.163+0000 7f54d1c00640 1 -- 172.21.15.28:0/1967549244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d9bb30 msgr2=0x558af5e0ade0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.163+0000 7f54d1c00640 1 --2- 172.21.15.28:0/1967549244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d9bb30 0x558af5e0ade0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.163+0000 7f54d1c00640 1 -- 172.21.15.28:0/1967549244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d91460 msgr2=0x558af5dad7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.163+0000 7f54d1c00640 1 --2- 172.21.15.28:0/1967549244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d91460 0x558af5dad7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.169 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.163+0000 7f54d1c00640 1 -- 172.21.15.28:0/1967549244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558af5e0b3d0 con 0x558af5d90b40 2024-04-24T08:21:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.163+0000 7f54d1c00640 1 --2- 172.21.15.28:0/1967549244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d90b40 0x558af5dad270 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f54c4031610 tx=0x7f54c4031ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89cd8c0404131fc server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.163+0000 7f54aea00640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:21:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.164+0000 7f54aea00640 1 -- 172.21.15.28:0/1967549244 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54c4048070 con 0x558af5d90b40 2024-04-24T08:21:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.164+0000 7f54aea00640 1 -- 172.21.15.28:0/1967549244 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7f54c4031700 con 0x558af5d90b40 2024-04-24T08:21:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.164+0000 7f54d6a5b740 1 -- 172.21.15.28:0/1967549244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f54bc01d550 con 0x558af5d90b40 2024-04-24T08:21:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.164+0000 7f54aea00640 1 -- 172.21.15.28:0/1967549244 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f54c4051040 con 0x558af5d90b40 2024-04-24T08:21:01.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.164+0000 7f54d6a5b740 10 client.16860 fetch_fsmap learned FSMap version 26 2024-04-24T08:21:01.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.164+0000 7f54d6a5b740 1 -- 172.21.15.28:0/1967549244 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x558af5e126e0 con 0x558af5d90b40 2024-04-24T08:21:01.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.164+0000 7f54aea00640 1 -- 172.21.15.28:0/1967549244 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54bc01d550 con 0x558af5d90b40 2024-04-24T08:21:01.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.164+0000 7f54aea00640 1 -- 172.21.15.28:0/1967549244 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f54c4004b40 con 0x558af5d90b40 2024-04-24T08:21:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.165+0000 7f54d6a5b740 10 client.16860 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:21:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.165+0000 7f54d6a5b740 10 client.16860 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.b 2024-04-24T08:21:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.165+0000 7f54d6a5b740 20 client.16860 populate_metadata read hostname 'smithi028' 2024-04-24T08:21:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.165+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/1967549244 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x558af5e12dd0 0x558af5e151f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.165+0000 7f54d6a5b740 4 client.16860 mds_command: new command op to 24572 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:21:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.165+0000 7f54d6a5b740 1 -- 172.21.15.28:0/1967549244 --> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] -- command(tid 0: {"prefix": "damage ls"}) -- 0x558af5cfda60 con 0x558af5e12dd0 2024-04-24T08:21:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.165+0000 7f54cbe00640 1 --2- 172.21.15.28:0/1967549244 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x558af5e12dd0 0x558af5e151f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.165+0000 7f54cbe00640 1 --2- 172.21.15.28:0/1967549244 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x558af5e12dd0 0x558af5e151f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6d7bc6cffc85a0ac server_cookie=60e87ecffc1a8220 in_seq=0 out_seq=0 2024-04-24T08:21:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.165+0000 7f54aea00640 10 client.16860 ms_handle_connect on v2:172.21.15.117:6838/3606021351 2024-04-24T08:21:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.166+0000 7f54aea00640 1 -- 172.21.15.28:0/1967549244 <== mds.1 v2:172.21.15.117:6838/3606021351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x558af5cfda60 con 0x558af5e12dd0 2024-04-24T08:21:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.166+0000 7f54aea00640 10 client.16860 handle_command_reply: tid=0 2024-04-24T08:21:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.166+0000 7f54d6a5b740 1 client.16860 shutdown 2024-04-24T08:21:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.166+0000 7f54d6a5b740 1 -- 172.21.15.28:0/1967549244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d90b40 msgr2=0x558af5dad270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.166+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/1967549244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d90b40 0x558af5dad270 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f54c4031610 tx=0x7f54c4031ac0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.166+0000 7f54d6a5b740 1 -- 172.21.15.28:0/1967549244 shutdown_connections 2024-04-24T08:21:01.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.166+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/1967549244 >> [v2:172.21.15.117:6838/3606021351,v1:172.21.15.117:6839/3606021351] conn(0x558af5e12dd0 0x558af5e151f0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.166+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/1967549244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x558af5d9bb30 0x558af5e0ade0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.166+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/1967549244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x558af5d91460 0x558af5dad7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.166+0000 7f54d6a5b740 1 --2- 172.21.15.28:0/1967549244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x558af5d90b40 0x558af5dad270 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.166+0000 7f54d6a5b740 1 -- 172.21.15.28:0/1967549244 >> 172.21.15.28:0/1967549244 conn(0x558af5d8ce60 msgr2=0x558af5da7620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.166+0000 7f54aea00640 0 client.16860 ms_handle_reset on v2:172.21.15.117:6838/3606021351 2024-04-24T08:21:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.166+0000 7f54d6a5b740 1 -- 172.21.15.28:0/1967549244 shutdown_connections 2024-04-24T08:21:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.166+0000 7f54d6a5b740 1 -- 172.21.15.28:0/1967549244 wait complete. 2024-04-24T08:21:01.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.166+0000 7f54d6a5b740 20 client.16860 trim_cache size 0 max 16384 2024-04-24T08:21:01.176 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:21:01.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.167+0000 7f54d2600640 1 -- 172.21.15.28:0/4253646339 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f54b40a3d20 msgr2=0x7f54b40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.167+0000 7f54d2600640 1 --2- 172.21.15.28:0/4253646339 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f54b40a3d20 0x7f54b40a61e0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f54c00cdf60 tx=0x7f54c00d6550 comp rx=0 tx=0).stop 2024-04-24T08:21:01.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.167+0000 7f54d2600640 1 -- 172.21.15.28:0/4253646339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc108580 msgr2=0x7f54cc1a4900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.167+0000 7f54d2600640 1 --2- 172.21.15.28:0/4253646339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc108580 0x7f54cc1a4900 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f54c000d950 tx=0x7f54c000de10 comp rx=0 tx=0).stop 2024-04-24T08:21:01.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.167+0000 7f54d2600640 1 -- 172.21.15.28:0/4253646339 shutdown_connections 2024-04-24T08:21:01.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.167+0000 7f54d2600640 1 --2- 172.21.15.28:0/4253646339 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f54b40a3d20 0x7f54b40a61e0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.167+0000 7f54d2600640 1 --2- 172.21.15.28:0/4253646339 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f54cc10d500 0x7f54cc19eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.167+0000 7f54d2600640 1 --2- 172.21.15.28:0/4253646339 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54cc108f40 0x7f54cc1a4e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.167+0000 7f54d2600640 1 --2- 172.21.15.28:0/4253646339 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f54cc108580 0x7f54cc1a4900 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.167+0000 7f54d2600640 1 -- 172.21.15.28:0/4253646339 >> 172.21.15.28:0/4253646339 conn(0x7f54cc077e20 msgr2=0x7f54cc0782a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.167+0000 7f54d2600640 1 -- 172.21.15.28:0/4253646339 shutdown_connections 2024-04-24T08:21:01.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.167+0000 7f54d2600640 1 -- 172.21.15.28:0/4253646339 wait complete. 2024-04-24T08:21:01.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-04-24T08:21:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.402+0000 7f72be000640 1 -- 172.21.15.28:0/559256161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b81093f0 msgr2=0x7f72b8110fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.402+0000 7f72be000640 1 --2- 172.21.15.28:0/559256161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b81093f0 0x7f72b8110fc0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f72b0009900 tx=0x7f72b002a410 comp rx=0 tx=0).stop 2024-04-24T08:21:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.402+0000 7f72be000640 1 -- 172.21.15.28:0/559256161 shutdown_connections 2024-04-24T08:21:01.403 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.402+0000 7f72be000640 1 --2- 172.21.15.28:0/559256161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b81093f0 0x7f72b8110fc0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.402+0000 7f72be000640 1 --2- 172.21.15.28:0/559256161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72b8104cd0 0x7f72b8108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.402+0000 7f72be000640 1 --2- 172.21.15.28:0/559256161 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72b8104320 0x7f72b8104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.402+0000 7f72be000640 1 -- 172.21.15.28:0/559256161 >> 172.21.15.28:0/559256161 conn(0x7f72b80ffb40 msgr2=0x7f72b8101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.402+0000 7f72be000640 1 -- 172.21.15.28:0/559256161 shutdown_connections 2024-04-24T08:21:01.404 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.402+0000 7f72be000640 1 -- 172.21.15.28:0/559256161 wait complete. 2024-04-24T08:21:01.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.403+0000 7f72be000640 1 Processor -- start 2024-04-24T08:21:01.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.404+0000 7f72be000640 1 -- start start 2024-04-24T08:21:01.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.404+0000 7f72be000640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72b8104320 0x7f72b81a0620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.404+0000 7f72be000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b81093f0 0x7f72b81a0b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.404+0000 7f72be000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72b819a790 0x7f72b819ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.404+0000 7f72be000640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72b8114490 con 0x7f72b81093f0 2024-04-24T08:21:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.404+0000 7f72be000640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f72b8114310 con 0x7f72b819a790 2024-04-24T08:21:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.404+0000 7f72be000640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f72b8114610 con 0x7f72b8104320 2024-04-24T08:21:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.404+0000 7f72b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b81093f0 0x7f72b81a0b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.404+0000 7f72b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b81093f0 0x7f72b81a0b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35900/0 (socket says 172.21.15.28:35900) 2024-04-24T08:21:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.404+0000 7f72b7e00640 1 -- 172.21.15.28:0/4078271568 learned_addr learned my addr 172.21.15.28:0/4078271568 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:01.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.404+0000 7f72bcc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72b8104320 0x7f72b81a0620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.404+0000 7f72bd600640 1 --2- 172.21.15.28:0/4078271568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72b819a790 0x7f72b819ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.405+0000 7f72bcc00640 1 -- 172.21.15.28:0/4078271568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72b819a790 msgr2=0x7f72b819ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.405+0000 7f72bcc00640 1 --2- 172.21.15.28:0/4078271568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72b819a790 0x7f72b819ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.405+0000 7f72bcc00640 1 -- 172.21.15.28:0/4078271568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b81093f0 msgr2=0x7f72b81a0b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.405+0000 7f72bcc00640 1 --2- 172.21.15.28:0/4078271568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b81093f0 0x7f72b81a0b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.405+0000 7f72bcc00640 1 -- 172.21.15.28:0/4078271568 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72b819b320 con 0x7f72b8104320 2024-04-24T08:21:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.405+0000 7f72bd600640 1 --2- 172.21.15.28:0/4078271568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72b819a790 0x7f72b819ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:01.408 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.405+0000 7f72b7e00640 1 --2- 172.21.15.28:0/4078271568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b81093f0 0x7f72b81a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.405+0000 7f72bcc00640 1 --2- 172.21.15.28:0/4078271568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72b8104320 0x7f72b81a0620 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f72a800eb00 tx=0x7f72a800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.405+0000 7f72b5600640 1 -- 172.21.15.28:0/4078271568 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a8010430 con 0x7f72b8104320 2024-04-24T08:21:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.406+0000 7f72be000640 1 -- 172.21.15.28:0/4078271568 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72b819b610 con 0x7f72b8104320 2024-04-24T08:21:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.406+0000 7f72b5600640 1 -- 172.21.15.28:0/4078271568 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72a8010ab0 con 0x7f72b8104320 2024-04-24T08:21:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.406+0000 7f72b5600640 1 -- 172.21.15.28:0/4078271568 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a8017bb0 con 0x7f72b8104320 2024-04-24T08:21:01.409 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.406+0000 7f72be000640 1 -- 172.21.15.28:0/4078271568 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72b81a7840 con 0x7f72b8104320 2024-04-24T08:21:01.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.409+0000 7f72c23f0740 1 Processor -- start 2024-04-24T08:21:01.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.409+0000 7f72c23f0740 1 -- start start 2024-04-24T08:21:01.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.409+0000 7f72c23f0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e53b40 0x559b24e53f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.409+0000 7f72c23f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e54460 0x559b24e5e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.409+0000 7f72c23f0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e5eb30 0x559b24e60f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.409+0000 7f72c23f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559b24db91d0 con 0x559b24e54460 2024-04-24T08:21:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.409+0000 7f72c23f0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559b24d5f450 con 0x559b24e5eb30 2024-04-24T08:21:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.409+0000 7f72c23f0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559b24db5190 con 0x559b24e53b40 2024-04-24T08:21:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.409+0000 7f72b5600640 1 -- 172.21.15.28:0/4078271568 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f72a8010c60 con 0x7f72b8104320 2024-04-24T08:21:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.410+0000 7f72bd600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e53b40 0x559b24e53f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.410+0000 7f72b7e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e5eb30 0x559b24e60f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.410+0000 7f72bd600640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e53b40 0x559b24e53f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:55710/0 (socket says 172.21.15.28:55710) 2024-04-24T08:21:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.410+0000 7f72bd600640 1 -- 172.21.15.28:0/3032854015 learned_addr learned my addr 172.21.15.28:0/3032854015 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.410+0000 7f72bcc00640 1 --2- 172.21.15.28:0/3032854015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e54460 0x559b24e5e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.410+0000 7f72bd600640 1 -- 172.21.15.28:0/3032854015 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e5eb30 msgr2=0x559b24e60f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.410+0000 7f72bd600640 1 --2- 172.21.15.28:0/3032854015 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e5eb30 0x559b24e60f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.410+0000 7f72bd600640 1 -- 172.21.15.28:0/3032854015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e54460 msgr2=0x559b24e5e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.410+0000 7f72bd600640 1 --2- 172.21.15.28:0/3032854015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e54460 0x559b24e5e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.410+0000 7f72bd600640 1 -- 172.21.15.28:0/3032854015 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b24e1c310 con 0x559b24e53b40 2024-04-24T08:21:01.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.410+0000 7f72b7e00640 1 --2- 172.21.15.28:0/3032854015 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e5eb30 0x559b24e60f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:01.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.410+0000 7f72bcc00640 1 --2- 172.21.15.28:0/3032854015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e54460 0x559b24e5e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:01.413 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.411+0000 7f72bd600640 1 --2- 172.21.15.28:0/3032854015 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e53b40 0x559b24e53f20 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f72b0033000 tx=0x7f72b0005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2fc79adaf3f1889d server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.411+0000 7f72adc00640 1 -- 172.21.15.28:0/3032854015 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b0035040 con 0x559b24e53b40 2024-04-24T08:21:01.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.411+0000 7f72adc00640 1 -- 172.21.15.28:0/3032854015 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72b00317b0 con 0x559b24e53b40 2024-04-24T08:21:01.417 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.411+0000 7f72adc00640 1 -- 172.21.15.28:0/3032854015 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b003c440 con 0x559b24e53b40 2024-04-24T08:21:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.411+0000 7f72b5600640 1 --2- 172.21.15.28:0/4078271568 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f729c0a3da0 0x7f729c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.411+0000 7f72b5600640 1 -- 172.21.15.28:0/4078271568 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7f72a80d1190 con 0x7f72b8104320 2024-04-24T08:21:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.412+0000 7f72bcc00640 1 --2- 172.21.15.28:0/4078271568 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f729c0a3da0 0x7f729c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.412+0000 7f72c23f0740 1 -- 172.21.15.28:0/3032854015 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e53b40 msgr2=0x559b24e53f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.412+0000 7f72c23f0740 1 --2- 172.21.15.28:0/3032854015 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e53b40 0x559b24e53f20 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f72b0033000 tx=0x7f72b0005d50 comp rx=0 tx=0).stop 2024-04-24T08:21:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.412+0000 7f72c23f0740 1 -- 172.21.15.28:0/3032854015 shutdown_connections 2024-04-24T08:21:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.412+0000 7f72c23f0740 1 --2- 172.21.15.28:0/3032854015 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e54460 0x559b24e5e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.418 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.412+0000 7f72c23f0740 1 --2- 172.21.15.28:0/3032854015 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e5eb30 0x559b24e60f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.412+0000 7f72c23f0740 1 --2- 172.21.15.28:0/3032854015 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e53b40 0x559b24e53f20 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.412+0000 7f72c23f0740 1 -- 172.21.15.28:0/3032854015 >> 172.21.15.28:0/3032854015 conn(0x559b24e4fe60 msgr2=0x559b24e50240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.412+0000 7f72c23f0740 1 -- 172.21.15.28:0/3032854015 shutdown_connections 2024-04-24T08:21:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.412+0000 7f72c23f0740 1 -- 172.21.15.28:0/3032854015 wait complete. 2024-04-24T08:21:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.412+0000 7f72bcc00640 1 --2- 172.21.15.28:0/4078271568 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f729c0a3da0 0x7f729c0a6260 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x559b24e55100 tx=0x7f72a8000f40 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.412+0000 7f72c23f0740 1 -- start start 2024-04-24T08:21:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.413+0000 7f72c23f0740 1 Processor -- start 2024-04-24T08:21:01.419 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.413+0000 7f72c23f0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e53b40 0x559b24e5e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.413+0000 7f72c23f0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e54460 0x559b24e5b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.413+0000 7f72c23f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e5eb30 0x559b24e5bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.413+0000 7f72c23f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x559b24e61960 con 0x559b24e5eb30 2024-04-24T08:21:01.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.413+0000 7f72c23f0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x559b24d72810 con 0x559b24e53b40 2024-04-24T08:21:01.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.413+0000 7f72c23f0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x559b24e61ae0 con 0x559b24e54460 2024-04-24T08:21:01.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.413+0000 7f72c23f0740 10 client.0 _ll_register_callbacks cb 0x559b24d40140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:01.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.413+0000 7f72bd600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e53b40 0x559b24e5e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.420 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.414+0000 7f72bd600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e5eb30 0x559b24e5bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.414+0000 7f72b7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e54460 0x559b24e5b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.414+0000 7f72bd600640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e53b40 0x559b24e5e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49582/0 (socket says 172.21.15.28:49582) 2024-04-24T08:21:01.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.414+0000 7f72bd600640 1 -- 172.21.15.28:0/1804949933 learned_addr learned my addr 172.21.15.28:0/1804949933 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:01.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.414+0000 7f72b7e00640 1 -- 172.21.15.28:0/1804949933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e53b40 msgr2=0x559b24e5e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.414+0000 7f72b7e00640 1 --2- 172.21.15.28:0/1804949933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e53b40 0x559b24e5e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.414+0000 7f72b7e00640 1 -- 172.21.15.28:0/1804949933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e5eb30 msgr2=0x559b24e5bfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.414+0000 7f72b7e00640 1 --2- 172.21.15.28:0/1804949933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e5eb30 0x559b24e5bfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.414+0000 7f72b7e00640 1 -- 172.21.15.28:0/1804949933 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b24d833f0 con 0x559b24e54460 2024-04-24T08:21:01.421 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.414+0000 7f72b7e00640 1 --2- 172.21.15.28:0/1804949933 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e54460 0x559b24e5b8b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f72b810aa10 tx=0x7f72a000f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77d4df1491b1bc96 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.414+0000 7f72a6000640 10 client.0 ms_handle_connect on v2:172.21.15.194:3300/0 2024-04-24T08:21:01.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.415+0000 7f72a6000640 1 -- 172.21.15.28:0/1804949933 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a00120d0 con 0x559b24e54460 2024-04-24T08:21:01.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.415+0000 7f72a6000640 1 -- 172.21.15.28:0/1804949933 <== mon.1 v2:172.21.15.194:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7f72a00047a0 con 0x559b24e54460 2024-04-24T08:21:01.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.415+0000 7f72c23f0740 1 -- 172.21.15.28:0/1804949933 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72b00317b0 con 0x559b24e54460 2024-04-24T08:21:01.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.415+0000 7f72a6000640 1 -- 172.21.15.28:0/1804949933 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f72a0012270 con 0x559b24e54460 2024-04-24T08:21:01.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.415+0000 7f72c23f0740 10 client.27160 fetch_fsmap learned FSMap version 26 2024-04-24T08:21:01.422 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.415+0000 7f72c23f0740 1 -- 172.21.15.28:0/1804949933 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559b24acb3f0 con 0x559b24e54460 2024-04-24T08:21:01.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.415+0000 7f72a6000640 1 -- 172.21.15.28:0/1804949933 <== mon.1 v2:172.21.15.194:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72b00317b0 con 0x559b24e54460 2024-04-24T08:21:01.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.416+0000 7f72a6000640 1 -- 172.21.15.28:0/1804949933 <== mon.1 v2:172.21.15.194:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f72a0010630 con 0x559b24e54460 2024-04-24T08:21:01.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.416+0000 7f72c23f0740 10 client.27160 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:21:01.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.416+0000 7f72c23f0740 10 client.27160 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:21:01.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.416+0000 7f72c23f0740 20 client.27160 populate_metadata read hostname 'smithi028' 2024-04-24T08:21:01.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.416+0000 7f72c23f0740 1 --2- 172.21.15.28:0/1804949933 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x559b24e6ed00 0x559b24e71120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.416+0000 7f72c23f0740 4 client.27160 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-04-24T08:21:01.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.416+0000 7f72c23f0740 1 -- 172.21.15.28:0/1804949933 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b24dc6bf0 con 0x559b24e6ed00 2024-04-24T08:21:01.423 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.416+0000 7f72bd600640 1 --2- 172.21.15.28:0/1804949933 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x559b24e6ed00 0x559b24e71120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.417+0000 7f72bd600640 1 --2- 172.21.15.28:0/1804949933 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x559b24e6ed00 0x559b24e71120 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9fb16b27f320e0e9 server_cookie=8ca9cc450c9eb6c6 in_seq=0 out_seq=0 2024-04-24T08:21:01.424 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.417+0000 7f72a6000640 10 client.27160 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:21:01.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.417+0000 7f72a6000640 1 -- 172.21.15.28:0/1804949933 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559b24dc6bf0 con 0x559b24e6ed00 2024-04-24T08:21:01.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.417+0000 7f72a6000640 10 client.27160 handle_command_reply: tid=0 2024-04-24T08:21:01.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.418+0000 7f72c23f0740 1 client.27160 shutdown 2024-04-24T08:21:01.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.418+0000 7f72c23f0740 1 -- 172.21.15.28:0/1804949933 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e54460 msgr2=0x559b24e5b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.425 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.418+0000 7f72c23f0740 1 --2- 172.21.15.28:0/1804949933 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e54460 0x559b24e5b8b0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f72b810aa10 tx=0x7f72a000f790 comp rx=0 tx=0).stop 2024-04-24T08:21:01.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.418+0000 7f72c23f0740 1 -- 172.21.15.28:0/1804949933 shutdown_connections 2024-04-24T08:21:01.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.418+0000 7f72c23f0740 1 --2- 172.21.15.28:0/1804949933 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x559b24e6ed00 0x559b24e71120 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.418+0000 7f72c23f0740 1 --2- 172.21.15.28:0/1804949933 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e5eb30 0x559b24e5bfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.418+0000 7f72c23f0740 1 --2- 172.21.15.28:0/1804949933 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e54460 0x559b24e5b8b0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.418+0000 7f72a6000640 0 client.27160 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:21:01.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.418+0000 7f72c23f0740 1 --2- 172.21.15.28:0/1804949933 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e53b40 0x559b24e5e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.418+0000 7f72c23f0740 1 -- 172.21.15.28:0/1804949933 >> 172.21.15.28:0/1804949933 conn(0x559b24e4fe60 msgr2=0x559b24e55c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.426 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.418+0000 7f72c23f0740 1 -- 172.21.15.28:0/1804949933 shutdown_connections 2024-04-24T08:21:01.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.418+0000 7f72c23f0740 1 -- 172.21.15.28:0/1804949933 wait complete. 2024-04-24T08:21:01.427 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.418+0000 7f72c23f0740 20 client.27160 trim_cache size 0 max 16384 2024-04-24T08:21:01.434 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.433+0000 7f72c23f0740 1 Processor -- start 2024-04-24T08:21:01.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.433+0000 7f72c23f0740 1 -- start start 2024-04-24T08:21:01.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.433+0000 7f72c23f0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e53b40 0x559b24e5bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.433+0000 7f72c23f0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e54460 0x559b24e4edd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.433+0000 7f72c23f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e5eb30 0x559b24e5e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.433+0000 7f72c23f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72b81b27e0 con 0x559b24e5eb30 2024-04-24T08:21:01.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.433+0000 7f72c23f0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f72b81b00b0 con 0x559b24e53b40 2024-04-24T08:21:01.435 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.433+0000 7f72c23f0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f72b81adf20 con 0x559b24e54460 2024-04-24T08:21:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.434+0000 7f72bd600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e5eb30 0x559b24e5e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.434+0000 7f72b7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e54460 0x559b24e4edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.434+0000 7f72bd600640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e5eb30 0x559b24e5e0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35940/0 (socket says 172.21.15.28:35940) 2024-04-24T08:21:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.434+0000 7f72bd600640 1 -- 172.21.15.28:0/197341146 learned_addr learned my addr 172.21.15.28:0/197341146 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.434+0000 7f72bd600640 1 --2- 172.21.15.28:0/197341146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e53b40 0x559b24e5bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.434+0000 7f72b7e00640 1 -- 172.21.15.28:0/197341146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e53b40 msgr2=0x559b24e5bf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.434+0000 7f72b7e00640 1 --2- 172.21.15.28:0/197341146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e53b40 0x559b24e5bf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.436 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.434+0000 7f72b7e00640 1 -- 172.21.15.28:0/197341146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e5eb30 msgr2=0x559b24e5e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.434+0000 7f72b7e00640 1 --2- 172.21.15.28:0/197341146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e5eb30 0x559b24e5e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.437 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.434+0000 7f72b7e00640 1 -- 172.21.15.28:0/197341146 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b24b957d0 con 0x559b24e54460 2024-04-24T08:21:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.434+0000 7f72b7e00640 1 --2- 172.21.15.28:0/197341146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e54460 0x559b24e4edd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f72a000ffd0 tx=0x7f72a0009520 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7d0edb8dc0a6e992 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.435+0000 7f72ad200640 1 -- 172.21.15.28:0/197341146 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a0010430 con 0x559b24e54460 2024-04-24T08:21:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.435+0000 7f72ad200640 1 -- 172.21.15.28:0/197341146 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72a0005550 con 0x559b24e54460 2024-04-24T08:21:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.435+0000 7f72ad200640 1 -- 172.21.15.28:0/197341146 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a0004970 con 0x559b24e54460 2024-04-24T08:21:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.435+0000 7f72c23f0740 1 -- 172.21.15.28:0/197341146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e54460 msgr2=0x559b24e4edd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.435+0000 7f72c23f0740 1 --2- 172.21.15.28:0/197341146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e54460 0x559b24e4edd0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f72a000ffd0 tx=0x7f72a0009520 comp rx=0 tx=0).stop 2024-04-24T08:21:01.442 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.436+0000 7f72c23f0740 1 -- 172.21.15.28:0/197341146 shutdown_connections 2024-04-24T08:21:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.436+0000 7f72c23f0740 1 --2- 172.21.15.28:0/197341146 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e5eb30 0x559b24e5e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.436+0000 7f72c23f0740 1 --2- 172.21.15.28:0/197341146 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e54460 0x559b24e4edd0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.436+0000 7f72c23f0740 1 --2- 172.21.15.28:0/197341146 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e53b40 0x559b24e5bf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.436+0000 7f72c23f0740 1 -- 172.21.15.28:0/197341146 >> 172.21.15.28:0/197341146 conn(0x559b24e4fe60 msgr2=0x559b24e4e6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.436+0000 7f72c23f0740 1 -- 172.21.15.28:0/197341146 shutdown_connections 2024-04-24T08:21:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.436+0000 7f72c23f0740 1 -- 172.21.15.28:0/197341146 wait complete. 2024-04-24T08:21:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.436+0000 7f72c23f0740 1 -- start start 2024-04-24T08:21:01.443 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.436+0000 7f72c23f0740 1 Processor -- start 2024-04-24T08:21:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.437+0000 7f72c23f0740 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e53b40 0x559b24e5ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.437+0000 7f72c23f0740 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e54460 0x559b24e604c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.437+0000 7f72c23f0740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e5eb30 0x559b24e653c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.437+0000 7f72c23f0740 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f72b81ae570 con 0x559b24e54460 2024-04-24T08:21:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.437+0000 7f72c23f0740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f72b81af9e0 con 0x559b24e5eb30 2024-04-24T08:21:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.437+0000 7f72c23f0740 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f72b81aec40 con 0x559b24e53b40 2024-04-24T08:21:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.437+0000 7f72c23f0740 10 client.0 _ll_register_callbacks cb 0x7f72a0009ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-04-24T08:21:01.444 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.437+0000 7f72b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e54460 0x559b24e604c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.437+0000 7f72b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e54460 0x559b24e604c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35944/0 (socket says 172.21.15.28:35944) 2024-04-24T08:21:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.437+0000 7f72b7e00640 1 -- 172.21.15.28:0/3494369475 learned_addr learned my addr 172.21.15.28:0/3494369475 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.437+0000 7f72bd600640 1 --2- 172.21.15.28:0/3494369475 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e53b40 0x559b24e5ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.437+0000 7f72bd600640 1 --2- 172.21.15.28:0/3494369475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e5eb30 0x559b24e653c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.437+0000 7f72b7e00640 1 -- 172.21.15.28:0/3494369475 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e53b40 msgr2=0x559b24e5ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.437+0000 7f72b7e00640 1 --2- 172.21.15.28:0/3494369475 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e53b40 0x559b24e5ff80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.445 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.437+0000 7f72b7e00640 1 -- 172.21.15.28:0/3494369475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e5eb30 msgr2=0x559b24e653c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-04-24T08:21:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.437+0000 7f72b7e00640 1 --2- 172.21.15.28:0/3494369475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e5eb30 0x559b24e653c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.437+0000 7f72b7e00640 1 -- 172.21.15.28:0/3494369475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b24e65900 con 0x559b24e54460 2024-04-24T08:21:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.438+0000 7f72bd600640 1 --2- 172.21.15.28:0/3494369475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e5eb30 0x559b24e653c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:21:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.438+0000 7f72b7e00640 1 --2- 172.21.15.28:0/3494369475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e54460 0x559b24e604c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f72a0010c40 tx=0x7f72a0012910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d57cf876a94b36b server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.438+0000 7f72a6000640 10 client.0 ms_handle_connect on v2:172.21.15.28:3300/0 2024-04-24T08:21:01.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.438+0000 7f72a6000640 1 -- 172.21.15.28:0/3494369475 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a0004220 con 0x559b24e54460 2024-04-24T08:21:01.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.438+0000 7f72c23f0740 1 -- 172.21.15.28:0/3494369475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72a0005550 con 0x559b24e54460 2024-04-24T08:21:01.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.438+0000 7f72a6000640 1 -- 172.21.15.28:0/3494369475 <== mon.0 v2:172.21.15.28:3300/0 2 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7f72a00043c0 con 0x559b24e54460 2024-04-24T08:21:01.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.438+0000 7f72a6000640 1 -- 172.21.15.28:0/3494369475 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7f72a003c050 con 0x559b24e54460 2024-04-24T08:21:01.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.439+0000 7f72c23f0740 10 client.16875 fetch_fsmap learned FSMap version 26 2024-04-24T08:21:01.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.439+0000 7f72c23f0740 1 -- 172.21.15.28:0/3494369475 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x559b24ed5a00 con 0x559b24e54460 2024-04-24T08:21:01.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.439+0000 7f72a6000640 1 -- 172.21.15.28:0/3494369475 <== mon.0 v2:172.21.15.28:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72a0005550 con 0x559b24e54460 2024-04-24T08:21:01.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.439+0000 7f72a6000640 1 -- 172.21.15.28:0/3494369475 <== mon.0 v2:172.21.15.28:3300/0 5 ==== fsmap(e 26) ==== 8338+0+0 (secure 0 0 0) 0x7f72a0034020 con 0x559b24e54460 2024-04-24T08:21:01.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.439+0000 7f72c23f0740 10 client.16875 fetch_fsmap finished waiting for FSMap version 26 2024-04-24T08:21:01.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.439+0000 7f72c23f0740 10 client.16875 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.g 2024-04-24T08:21:01.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.439+0000 7f72c23f0740 20 client.16875 populate_metadata read hostname 'smithi028' 2024-04-24T08:21:01.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.439+0000 7f72c23f0740 1 --2- 172.21.15.28:0/3494369475 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x559b24ed60b0 0x559b24ed84a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:01.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.439+0000 7f72c23f0740 4 client.16875 mds_command: new command op to 24574 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-04-24T08:21:01.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.439+0000 7f72c23f0740 1 -- 172.21.15.28:0/3494369475 --> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559b24dc0a60 con 0x559b24ed60b0 2024-04-24T08:21:01.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.440+0000 7f72bd600640 1 --2- 172.21.15.28:0/3494369475 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x559b24ed60b0 0x559b24ed84a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:01.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.440+0000 7f72bd600640 1 --2- 172.21.15.28:0/3494369475 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x559b24ed60b0 0x559b24ed84a0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce0b41d2d1f363d5 server_cookie=334b915eff3c5a60 in_seq=0 out_seq=0 2024-04-24T08:21:01.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.440+0000 7f72a6000640 10 client.16875 ms_handle_connect on v2:172.21.15.28:6838/1247092633 2024-04-24T08:21:01.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.440+0000 7f72a6000640 1 -- 172.21.15.28:0/3494369475 <== mds.0 v2:172.21.15.28:6838/1247092633 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559b24dc0a60 con 0x559b24ed60b0 2024-04-24T08:21:01.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.440+0000 7f72a6000640 10 client.16875 handle_command_reply: tid=0 2024-04-24T08:21:01.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.440+0000 7f72c23f0740 1 client.16875 shutdown 2024-04-24T08:21:01.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.440+0000 7f72c23f0740 1 -- 172.21.15.28:0/3494369475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e54460 msgr2=0x559b24e604c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.440+0000 7f72c23f0740 1 --2- 172.21.15.28:0/3494369475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e54460 0x559b24e604c0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f72a0010c40 tx=0x7f72a0012910 comp rx=0 tx=0).stop 2024-04-24T08:21:01.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.441+0000 7f72c23f0740 1 -- 172.21.15.28:0/3494369475 shutdown_connections 2024-04-24T08:21:01.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.441+0000 7f72c23f0740 1 --2- 172.21.15.28:0/3494369475 >> [v2:172.21.15.28:6838/1247092633,v1:172.21.15.28:6839/1247092633] conn(0x559b24ed60b0 0x559b24ed84a0 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.441+0000 7f72c23f0740 1 --2- 172.21.15.28:0/3494369475 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x559b24e54460 0x559b24e604c0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.441+0000 7f72c23f0740 1 --2- 172.21.15.28:0/3494369475 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x559b24e5eb30 0x559b24e653c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.441+0000 7f72c23f0740 1 --2- 172.21.15.28:0/3494369475 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x559b24e53b40 0x559b24e5ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.441+0000 7f72c23f0740 1 -- 172.21.15.28:0/3494369475 >> 172.21.15.28:0/3494369475 conn(0x559b24e4fe60 msgr2=0x559b24e4ef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.441+0000 7f72a6000640 0 client.16875 ms_handle_reset on v2:172.21.15.28:6838/1247092633 2024-04-24T08:21:01.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.441+0000 7f72c23f0740 1 -- 172.21.15.28:0/3494369475 shutdown_connections 2024-04-24T08:21:01.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.441+0000 7f72c23f0740 1 -- 172.21.15.28:0/3494369475 wait complete. 2024-04-24T08:21:01.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.441+0000 7f72c23f0740 20 client.16875 trim_cache size 0 max 16384 2024-04-24T08:21:01.450 INFO:teuthology.orchestra.run.smithi028.stdout:[] 2024-04-24T08:21:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.442+0000 7f72be000640 1 -- 172.21.15.28:0/4078271568 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f729c0a3da0 msgr2=0x7f729c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.442+0000 7f72be000640 1 --2- 172.21.15.28:0/4078271568 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f729c0a3da0 0x7f729c0a6260 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x559b24e55100 tx=0x7f72a8000f40 comp rx=0 tx=0).stop 2024-04-24T08:21:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.442+0000 7f72be000640 1 -- 172.21.15.28:0/4078271568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72b8104320 msgr2=0x7f72b81a0620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.442+0000 7f72be000640 1 --2- 172.21.15.28:0/4078271568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72b8104320 0x7f72b81a0620 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f72a800eb00 tx=0x7f72a800c2d0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.442+0000 7f72be000640 1 -- 172.21.15.28:0/4078271568 shutdown_connections 2024-04-24T08:21:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.442+0000 7f72be000640 1 --2- 172.21.15.28:0/4078271568 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f729c0a3da0 0x7f729c0a6260 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.442+0000 7f72be000640 1 --2- 172.21.15.28:0/4078271568 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f72b81093f0 0x7f72b81a0b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.442+0000 7f72be000640 1 --2- 172.21.15.28:0/4078271568 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72b819a790 0x7f72b819ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.442+0000 7f72be000640 1 --2- 172.21.15.28:0/4078271568 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f72b8104320 0x7f72b81a0620 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.442+0000 7f72be000640 1 -- 172.21.15.28:0/4078271568 >> 172.21.15.28:0/4078271568 conn(0x7f72b80ffb40 msgr2=0x7f72b8101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.442+0000 7f72be000640 1 -- 172.21.15.28:0/4078271568 shutdown_connections 2024-04-24T08:21:01.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:01.442+0000 7f72be000640 1 -- 172.21.15.28:0/4078271568 wait complete. 2024-04-24T08:21:01.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:01 smithi028 ceph-mon[27786]: pgmap v331: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:21:01.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:01 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/606415380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:21:01.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:01 smithi117 ceph-mon[32438]: pgmap v331: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:21:01.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:01 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/606415380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:21:01.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:01 smithi194 ceph-mon[32836]: pgmap v331: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:21:01.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:01 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/606415380' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:21:02.461 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-04-24T08:21:02.461 INFO:tasks.fwd_scrub:done joining 2024-04-24T08:21:02.462 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-04-24T08:21:02.474 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:21:02.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.691+0000 7ff6a9400640 1 -- 172.21.15.28:0/3275828833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a410c630 msgr2=0x7ff6a410ea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.691+0000 7ff6a9400640 1 --2- 172.21.15.28:0/3275828833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a410c630 0x7ff6a410ea60 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7ff69c009900 tx=0x7ff69c02a480 comp rx=0 tx=0).stop 2024-04-24T08:21:02.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.691+0000 7ff6a9400640 1 -- 172.21.15.28:0/3275828833 shutdown_connections 2024-04-24T08:21:02.692 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.691+0000 7ff6a9400640 1 --2- 172.21.15.28:0/3275828833 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a410c630 0x7ff6a410ea60 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.691+0000 7ff6a9400640 1 --2- 172.21.15.28:0/3275828833 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6a4103b80 0x7ff6a410c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.691+0000 7ff6a9400640 1 --2- 172.21.15.28:0/3275828833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6a4103260 0x7ff6a4103640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.691+0000 7ff6a9400640 1 -- 172.21.15.28:0/3275828833 >> 172.21.15.28:0/3275828833 conn(0x7ff6a4077280 msgr2=0x7ff6a4077690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:02.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.691+0000 7ff6a9400640 1 -- 172.21.15.28:0/3275828833 shutdown_connections 2024-04-24T08:21:02.693 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.691+0000 7ff6a9400640 1 -- 172.21.15.28:0/3275828833 wait complete. 2024-04-24T08:21:02.695 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.692+0000 7ff6a9400640 1 Processor -- start 2024-04-24T08:21:02.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.692+0000 7ff6a9400640 1 -- start start 2024-04-24T08:21:02.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.692+0000 7ff6a9400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6a4103260 0x7ff6a41a0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.692+0000 7ff6a9400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6a4103b80 0x7ff6a41a0c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.692+0000 7ff6a9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a410c630 0x7ff6a419a8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.692+0000 7ff6a9400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a41123f0 con 0x7ff6a410c630 2024-04-24T08:21:02.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.692+0000 7ff6a9400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a4112270 con 0x7ff6a4103260 2024-04-24T08:21:02.696 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.692+0000 7ff6a9400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff6a4112570 con 0x7ff6a4103b80 2024-04-24T08:21:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.693+0000 7ff6a3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a410c630 0x7ff6a419a8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.693+0000 7ff6a1e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6a4103b80 0x7ff6a41a0c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.693+0000 7ff6a3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a410c630 0x7ff6a419a8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35960/0 (socket says 172.21.15.28:35960) 2024-04-24T08:21:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.693+0000 7ff6a3200640 1 -- 172.21.15.28:0/1128518642 learned_addr learned my addr 172.21.15.28:0/1128518642 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.693+0000 7ff6a1e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6a4103b80 0x7ff6a41a0c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:55756/0 (socket says 172.21.15.28:55756) 2024-04-24T08:21:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.693+0000 7ff6a2800640 1 --2- 172.21.15.28:0/1128518642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6a4103260 0x7ff6a41a0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.693+0000 7ff6a3200640 1 -- 172.21.15.28:0/1128518642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6a4103b80 msgr2=0x7ff6a41a0c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.697 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.693+0000 7ff6a3200640 1 --2- 172.21.15.28:0/1128518642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6a4103b80 0x7ff6a41a0c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.693+0000 7ff6a3200640 1 -- 172.21.15.28:0/1128518642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6a4103260 msgr2=0x7ff6a41a0700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.693+0000 7ff6a3200640 1 --2- 172.21.15.28:0/1128518642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6a4103260 0x7ff6a41a0700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.693+0000 7ff6a3200640 1 -- 172.21.15.28:0/1128518642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6a419b030 con 0x7ff6a410c630 2024-04-24T08:21:02.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.693+0000 7ff6a2800640 1 --2- 172.21.15.28:0/1128518642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6a4103260 0x7ff6a41a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:02.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.693+0000 7ff6a3200640 1 --2- 172.21.15.28:0/1128518642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a410c630 0x7ff6a419a8b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff69c004770 tx=0x7ff69c003b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:02.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.694+0000 7ff69b400640 1 -- 172.21.15.28:0/1128518642 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c03f070 con 0x7ff6a410c630 2024-04-24T08:21:02.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.694+0000 7ff69b400640 1 -- 172.21.15.28:0/1128518642 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff69c031c90 con 0x7ff6a410c630 2024-04-24T08:21:02.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.694+0000 7ff6a9400640 1 -- 172.21.15.28:0/1128518642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6a419b2c0 con 0x7ff6a410c630 2024-04-24T08:21:02.698 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.694+0000 7ff69b400640 1 -- 172.21.15.28:0/1128518642 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff69c035030 con 0x7ff6a410c630 2024-04-24T08:21:02.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.694+0000 7ff6a9400640 1 -- 172.21.15.28:0/1128518642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6a41a75c0 con 0x7ff6a410c630 2024-04-24T08:21:02.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.695+0000 7ff6a9400640 1 -- 172.21.15.28:0/1128518642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff6a406ae80 con 0x7ff6a410c630 2024-04-24T08:21:02.699 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.698+0000 7ff69b400640 1 -- 172.21.15.28:0/1128518642 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff69c042070 con 0x7ff6a410c630 2024-04-24T08:21:02.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.699+0000 7ff69b400640 1 --2- 172.21.15.28:0/1128518642 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff6880a3d20 0x7ff6880a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:02.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.699+0000 7ff69b400640 1 -- 172.21.15.28:0/1128518642 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7ff69c0f1500 con 0x7ff6a410c630 2024-04-24T08:21:02.700 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.699+0000 7ff6a2800640 1 --2- 172.21.15.28:0/1128518642 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff6880a3d20 0x7ff6880a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:02.701 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.700+0000 7ff6a2800640 1 --2- 172.21.15.28:0/1128518642 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff6880a3d20 0x7ff6880a61e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff6940099a0 tx=0x7ff694009500 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:02.702 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.702+0000 7ff69b400640 1 -- 172.21.15.28:0/1128518642 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff69c0bdbb0 con 0x7ff6a410c630 2024-04-24T08:21:02.914 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.913+0000 7ff6a9400640 1 -- 172.21.15.28:0/1128518642 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff6a4069270 con 0x7ff6a410c630 2024-04-24T08:21:02.918 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.918+0000 7ff69b400640 1 -- 172.21.15.28:0/1128518642 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v99) ==== 74+0+23189 (secure 0 0 0) 0x7ff6a4069270 con 0x7ff6a410c630 2024-04-24T08:21:02.919 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:21:02.919 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":99,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:20:29.214864+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"98","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":15,"snap_epoch":98,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1819419617":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6801/361125225":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/3758227512":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/874707228":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/2660639233":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/361125225":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":5,"length":1},{"begin":15,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:21:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.921+0000 7ff6a9400640 1 -- 172.21.15.28:0/1128518642 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff6880a3d20 msgr2=0x7ff6880a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.921+0000 7ff6a9400640 1 --2- 172.21.15.28:0/1128518642 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff6880a3d20 0x7ff6880a61e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7ff6940099a0 tx=0x7ff694009500 comp rx=0 tx=0).stop 2024-04-24T08:21:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.921+0000 7ff6a9400640 1 -- 172.21.15.28:0/1128518642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a410c630 msgr2=0x7ff6a419a8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.921+0000 7ff6a9400640 1 --2- 172.21.15.28:0/1128518642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a410c630 0x7ff6a419a8b0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7ff69c004770 tx=0x7ff69c003b90 comp rx=0 tx=0).stop 2024-04-24T08:21:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.921+0000 7ff6a9400640 1 -- 172.21.15.28:0/1128518642 shutdown_connections 2024-04-24T08:21:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.921+0000 7ff6a9400640 1 --2- 172.21.15.28:0/1128518642 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff6880a3d20 0x7ff6880a61e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.921+0000 7ff6a9400640 1 --2- 172.21.15.28:0/1128518642 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff6a410c630 0x7ff6a419a8b0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.921+0000 7ff6a9400640 1 --2- 172.21.15.28:0/1128518642 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff6a4103b80 0x7ff6a41a0c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.921+0000 7ff6a9400640 1 --2- 172.21.15.28:0/1128518642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6a4103260 0x7ff6a41a0700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.921+0000 7ff6a9400640 1 -- 172.21.15.28:0/1128518642 >> 172.21.15.28:0/1128518642 conn(0x7ff6a4077280 msgr2=0x7ff6a4101fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.921+0000 7ff6a9400640 1 -- 172.21.15.28:0/1128518642 shutdown_connections 2024-04-24T08:21:02.923 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:02.921+0000 7ff6a9400640 1 -- 172.21.15.28:0/1128518642 wait complete. 2024-04-24T08:21:02.942 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:02:34.086371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:07:49.819536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:07:50.493486+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '98', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 15, 'snap_epoch': 98, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:21:02.943 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:21:03.166 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.165+0000 7f8512800640 1 -- 172.21.15.28:0/31434421 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f850c100620 msgr2=0x7f850c100a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.165+0000 7f8512800640 1 --2- 172.21.15.28:0/31434421 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f850c100620 0x7f850c100a00 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8500009ca0 tx=0x7f850002a8b0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.166+0000 7f8512800640 1 -- 172.21.15.28:0/31434421 shutdown_connections 2024-04-24T08:21:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.166+0000 7f8512800640 1 --2- 172.21.15.28:0/31434421 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f850c1052b0 0x7f850c105760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.166+0000 7f8512800640 1 --2- 172.21.15.28:0/31434421 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f850c100f40 0x7f850c104d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.166+0000 7f8512800640 1 --2- 172.21.15.28:0/31434421 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f850c100620 0x7f850c100a00 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.166+0000 7f8512800640 1 -- 172.21.15.28:0/31434421 >> 172.21.15.28:0/31434421 conn(0x7f850c075d00 msgr2=0x7f850c076110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.166+0000 7f8512800640 1 -- 172.21.15.28:0/31434421 shutdown_connections 2024-04-24T08:21:03.167 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.166+0000 7f8512800640 1 -- 172.21.15.28:0/31434421 wait complete. 2024-04-24T08:21:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.167+0000 7f8512800640 1 Processor -- start 2024-04-24T08:21:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.167+0000 7f8512800640 1 -- start start 2024-04-24T08:21:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.168+0000 7f8512800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f850c100620 0x7f850c19e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.168+0000 7f8512800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f850c100f40 0x7f850c19eac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.168+0000 7f8512800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f850c1052b0 0x7f850c198640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.168+0000 7f8512800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f850c1111c0 con 0x7f850c100f40 2024-04-24T08:21:03.170 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.168+0000 7f8512800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f850c111040 con 0x7f850c100620 2024-04-24T08:21:03.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.168+0000 7f8512800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f850c111340 con 0x7f850c1052b0 2024-04-24T08:21:03.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.168+0000 7f8511400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f850c100620 0x7f850c19e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.168+0000 7f8511400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f850c100620 0x7f850c19e580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49648/0 (socket says 172.21.15.28:49648) 2024-04-24T08:21:03.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.168+0000 7f8511400640 1 -- 172.21.15.28:0/540448840 learned_addr learned my addr 172.21.15.28:0/540448840 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:03.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.168+0000 7f8510a00640 1 --2- 172.21.15.28:0/540448840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f850c100f40 0x7f850c19eac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.168+0000 7f8511e00640 1 --2- 172.21.15.28:0/540448840 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f850c1052b0 0x7f850c198640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.168+0000 7f8511400640 1 -- 172.21.15.28:0/540448840 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f850c1052b0 msgr2=0x7f850c198640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.168+0000 7f8511400640 1 --2- 172.21.15.28:0/540448840 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f850c1052b0 0x7f850c198640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.171 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.168+0000 7f8511400640 1 -- 172.21.15.28:0/540448840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f850c100f40 msgr2=0x7f850c19eac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.168+0000 7f8511400640 1 --2- 172.21.15.28:0/540448840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f850c100f40 0x7f850c19eac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.168+0000 7f8511400640 1 -- 172.21.15.28:0/540448840 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f850c198f00 con 0x7f850c100620 2024-04-24T08:21:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.168+0000 7f8510a00640 1 --2- 172.21.15.28:0/540448840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f850c100f40 0x7f850c19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.169+0000 7f8511400640 1 --2- 172.21.15.28:0/540448840 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f850c100620 0x7f850c19e580 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f850002add0 tx=0x7f8500030ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.169+0000 7f850a000640 1 -- 172.21.15.28:0/540448840 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8500003e40 con 0x7f850c100620 2024-04-24T08:21:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.169+0000 7f850a000640 1 -- 172.21.15.28:0/540448840 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8500040070 con 0x7f850c100620 2024-04-24T08:21:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.169+0000 7f8512800640 1 -- 172.21.15.28:0/540448840 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f850c199190 con 0x7f850c100620 2024-04-24T08:21:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.169+0000 7f850a000640 1 -- 172.21.15.28:0/540448840 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8500045b20 con 0x7f850c100620 2024-04-24T08:21:03.172 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.169+0000 7f8512800640 1 -- 172.21.15.28:0/540448840 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f850c1a5500 con 0x7f850c100620 2024-04-24T08:21:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.171+0000 7f8512800640 1 -- 172.21.15.28:0/540448840 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f850c199500 con 0x7f850c100620 2024-04-24T08:21:03.173 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.173+0000 7f850a000640 1 -- 172.21.15.28:0/540448840 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f8500045cc0 con 0x7f850c100620 2024-04-24T08:21:03.174 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.174+0000 7f850a000640 1 --2- 172.21.15.28:0/540448840 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f84f40a3d20 0x7f84f40a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.174+0000 7f850a000640 1 -- 172.21.15.28:0/540448840 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7f85000f1760 con 0x7f850c100620 2024-04-24T08:21:03.175 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.174+0000 7f8510a00640 1 --2- 172.21.15.28:0/540448840 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f84f40a3d20 0x7f84f40a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.176 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.175+0000 7f8510a00640 1 --2- 172.21.15.28:0/540448840 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f84f40a3d20 0x7f84f40a61e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f850c199e90 tx=0x7f84f8005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:03.179 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.178+0000 7f850a000640 1 -- 172.21.15.28:0/540448840 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f85000bdef0 con 0x7f850c100620 2024-04-24T08:21:03.391 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.390+0000 7f8512800640 1 -- 172.21.15.28:0/540448840 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f850c199b70 con 0x7f850c100620 2024-04-24T08:21:03.392 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.391+0000 7f850a000640 1 -- 172.21.15.28:0/540448840 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v99) ==== 93+0+10 (secure 0 0 0) 0x7f85000bd7f0 con 0x7f850c100620 2024-04-24T08:21:03.392 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-24T08:21:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.394+0000 7f8512800640 1 -- 172.21.15.28:0/540448840 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f84f40a3d20 msgr2=0x7f84f40a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.394+0000 7f8512800640 1 --2- 172.21.15.28:0/540448840 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f84f40a3d20 0x7f84f40a61e0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f850c199e90 tx=0x7f84f8005f90 comp rx=0 tx=0).stop 2024-04-24T08:21:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.394+0000 7f8512800640 1 -- 172.21.15.28:0/540448840 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f850c100620 msgr2=0x7f850c19e580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.394+0000 7f8512800640 1 --2- 172.21.15.28:0/540448840 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f850c100620 0x7f850c19e580 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f850002add0 tx=0x7f8500030ed0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.394+0000 7f8512800640 1 -- 172.21.15.28:0/540448840 shutdown_connections 2024-04-24T08:21:03.395 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.394+0000 7f8512800640 1 --2- 172.21.15.28:0/540448840 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f84f40a3d20 0x7f84f40a61e0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.394+0000 7f8512800640 1 --2- 172.21.15.28:0/540448840 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f850c100f40 0x7f850c19eac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.394+0000 7f8512800640 1 --2- 172.21.15.28:0/540448840 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f850c1052b0 0x7f850c198640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.394+0000 7f8512800640 1 --2- 172.21.15.28:0/540448840 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f850c100620 0x7f850c19e580 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.394+0000 7f8512800640 1 -- 172.21.15.28:0/540448840 >> 172.21.15.28:0/540448840 conn(0x7f850c075d00 msgr2=0x7f850c101cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.395+0000 7f8512800640 1 -- 172.21.15.28:0/540448840 shutdown_connections 2024-04-24T08:21:03.396 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.395+0000 7f8512800640 1 -- 172.21.15.28:0/540448840 wait complete. 2024-04-24T08:21:03.415 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:21:03.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.631+0000 7fcab3200640 1 -- 172.21.15.28:0/26805150 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcab4108f40 msgr2=0x7fcab41153b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.631+0000 7fcab3200640 1 --2- 172.21.15.28:0/26805150 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcab4108f40 0x7fcab41153b0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7fcaac009c10 tx=0x7fcaac02a6e0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.632 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.631+0000 7fcab3200640 1 -- 172.21.15.28:0/26805150 shutdown_connections 2024-04-24T08:21:03.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.631+0000 7fcab3200640 1 --2- 172.21.15.28:0/26805150 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcab4108f40 0x7fcab41153b0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.631+0000 7fcab3200640 1 --2- 172.21.15.28:0/26805150 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcab4108580 0x7fcab4108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.631+0000 7fcab3200640 1 --2- 172.21.15.28:0/26805150 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab410d500 0x7fcab410d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.631+0000 7fcab3200640 1 -- 172.21.15.28:0/26805150 >> 172.21.15.28:0/26805150 conn(0x7fcab4077e20 msgr2=0x7fcab4106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:03.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.631+0000 7fcab3200640 1 -- 172.21.15.28:0/26805150 shutdown_connections 2024-04-24T08:21:03.633 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.631+0000 7fcab3200640 1 -- 172.21.15.28:0/26805150 wait complete. 2024-04-24T08:21:03.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.632+0000 7fcab3200640 1 Processor -- start 2024-04-24T08:21:03.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.632+0000 7fcab3200640 1 -- start start 2024-04-24T08:21:03.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.633+0000 7fcab3200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcab4108580 0x7fcab41a4a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.635 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.633+0000 7fcab3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4108f40 0x7fcab41a4f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.633+0000 7fcab3200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcab410d500 0x7fcab419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.633+0000 7fcab3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fcab4118860 con 0x7fcab4108f40 2024-04-24T08:21:03.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.633+0000 7fcab3200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fcab41186e0 con 0x7fcab4108580 2024-04-24T08:21:03.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.633+0000 7fcab3200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fcab41189e0 con 0x7fcab410d500 2024-04-24T08:21:03.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.633+0000 7fcaa9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4108f40 0x7fcab41a4f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.633+0000 7fcaa9400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4108f40 0x7fcab41a4f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:35996/0 (socket says 172.21.15.28:35996) 2024-04-24T08:21:03.636 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.633+0000 7fcaa9400640 1 -- 172.21.15.28:0/3258409987 learned_addr learned my addr 172.21.15.28:0/3258409987 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.633+0000 7fcab1e00640 1 --2- 172.21.15.28:0/3258409987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcab4108580 0x7fcab41a4a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.633+0000 7fcab2800640 1 --2- 172.21.15.28:0/3258409987 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcab410d500 0x7fcab419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.634+0000 7fcaa9400640 1 -- 172.21.15.28:0/3258409987 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcab410d500 msgr2=0x7fcab419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.634+0000 7fcaa9400640 1 --2- 172.21.15.28:0/3258409987 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcab410d500 0x7fcab419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.634+0000 7fcaa9400640 1 -- 172.21.15.28:0/3258409987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcab4108580 msgr2=0x7fcab41a4a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.634+0000 7fcaa9400640 1 --2- 172.21.15.28:0/3258409987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcab4108580 0x7fcab41a4a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.634+0000 7fcaa9400640 1 -- 172.21.15.28:0/3258409987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcab419f420 con 0x7fcab4108f40 2024-04-24T08:21:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.634+0000 7fcaa9400640 1 --2- 172.21.15.28:0/3258409987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4108f40 0x7fcab41a4f90 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fca9c00ea60 tx=0x7fca9c00ef20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:03.637 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.634+0000 7fcaab400640 1 -- 172.21.15.28:0/3258409987 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca9c00cde0 con 0x7fcab4108f40 2024-04-24T08:21:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.634+0000 7fcaab400640 1 -- 172.21.15.28:0/3258409987 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fca9c0035f0 con 0x7fcab4108f40 2024-04-24T08:21:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.634+0000 7fcab3200640 1 -- 172.21.15.28:0/3258409987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcab419f710 con 0x7fcab4108f40 2024-04-24T08:21:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.634+0000 7fcaab400640 1 -- 172.21.15.28:0/3258409987 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fca9c010670 con 0x7fcab4108f40 2024-04-24T08:21:03.638 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.635+0000 7fcab3200640 1 -- 172.21.15.28:0/3258409987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcab41ab9d0 con 0x7fcab4108f40 2024-04-24T08:21:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.636+0000 7fcab3200640 1 -- 172.21.15.28:0/3258409987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcab4109bf0 con 0x7fcab4108f40 2024-04-24T08:21:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.638+0000 7fcaab400640 1 -- 172.21.15.28:0/3258409987 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fca9c0037a0 con 0x7fcab4108f40 2024-04-24T08:21:03.639 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.639+0000 7fcaab400640 1 --2- 172.21.15.28:0/3258409987 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fca980a3da0 0x7fca980a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:03.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.639+0000 7fcaab400640 1 -- 172.21.15.28:0/3258409987 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7fca9c0d0b10 con 0x7fcab4108f40 2024-04-24T08:21:03.640 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.640+0000 7fcab1e00640 1 --2- 172.21.15.28:0/3258409987 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fca980a3da0 0x7fca980a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:03.641 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.640+0000 7fcab1e00640 1 --2- 172.21.15.28:0/3258409987 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fca980a3da0 0x7fca980a6260 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4009530 tx=0x7fcaa4009260 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:03.644 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.644+0000 7fcaab400640 1 -- 172.21.15.28:0/3258409987 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fca9c09d100 con 0x7fcab4108f40 2024-04-24T08:21:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:03 smithi028 ceph-mon[27786]: pgmap v332: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:21:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:03 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1128518642' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:21:03.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:03 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/540448840' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:21:03.853 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.853+0000 7fcab3200640 1 -- 172.21.15.28:0/3258409987 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fcab4069190 con 0x7fcab4108f40 2024-04-24T08:21:03.854 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.854+0000 7fcaab400640 1 -- 172.21.15.28:0/3258409987 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v99) ==== 104+0+11 (secure 0 0 0) 0x7fca9c09ca00 con 0x7fcab4108f40 2024-04-24T08:21:03.855 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-24T08:21:03.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.858+0000 7fcab3200640 1 -- 172.21.15.28:0/3258409987 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fca980a3da0 msgr2=0x7fca980a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.858+0000 7fcab3200640 1 --2- 172.21.15.28:0/3258409987 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fca980a3da0 0x7fca980a6260 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcaa4009530 tx=0x7fcaa4009260 comp rx=0 tx=0).stop 2024-04-24T08:21:03.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.858+0000 7fcab3200640 1 -- 172.21.15.28:0/3258409987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4108f40 msgr2=0x7fcab41a4f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:03.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.858+0000 7fcab3200640 1 --2- 172.21.15.28:0/3258409987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4108f40 0x7fcab41a4f90 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7fca9c00ea60 tx=0x7fca9c00ef20 comp rx=0 tx=0).stop 2024-04-24T08:21:03.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.858+0000 7fcab3200640 1 -- 172.21.15.28:0/3258409987 shutdown_connections 2024-04-24T08:21:03.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.858+0000 7fcab3200640 1 --2- 172.21.15.28:0/3258409987 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fca980a3da0 0x7fca980a6260 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.859 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.858+0000 7fcab3200640 1 --2- 172.21.15.28:0/3258409987 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fcab4108f40 0x7fcab41a4f90 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.858+0000 7fcab3200640 1 --2- 172.21.15.28:0/3258409987 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fcab410d500 0x7fcab419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.858+0000 7fcab3200640 1 --2- 172.21.15.28:0/3258409987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcab4108580 0x7fcab41a4a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:03.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.858+0000 7fcab3200640 1 -- 172.21.15.28:0/3258409987 >> 172.21.15.28:0/3258409987 conn(0x7fcab4077e20 msgr2=0x7fcab40783e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:03.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.858+0000 7fcab3200640 1 -- 172.21.15.28:0/3258409987 shutdown_connections 2024-04-24T08:21:03.860 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:03.858+0000 7fcab3200640 1 -- 172.21.15.28:0/3258409987 wait complete. 2024-04-24T08:21:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:03 smithi117 ceph-mon[32438]: pgmap v332: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:21:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:03 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1128518642' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:21:03.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:03 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/540448840' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:21:03.880 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:21:03.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:03 smithi194 ceph-mon[32836]: pgmap v332: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:21:03.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:03 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1128518642' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:21:03.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:03 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/540448840' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:21:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.102+0000 7f48b7e00640 1 -- 172.21.15.28:0/4006609141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48b80743a0 msgr2=0x7f48b8074780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.102+0000 7f48b7e00640 1 --2- 172.21.15.28:0/4006609141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48b80743a0 0x7f48b8074780 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f48a8009900 tx=0x7f48a802a480 comp rx=0 tx=0).stop 2024-04-24T08:21:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.102+0000 7f48b7e00640 1 -- 172.21.15.28:0/4006609141 shutdown_connections 2024-04-24T08:21:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.102+0000 7f48b7e00640 1 --2- 172.21.15.28:0/4006609141 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48b8112ec0 0x7f48b81152b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.103 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.102+0000 7f48b7e00640 1 --2- 172.21.15.28:0/4006609141 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48b8074cc0 0x7f48b8112980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.102+0000 7f48b7e00640 1 --2- 172.21.15.28:0/4006609141 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48b80743a0 0x7f48b8074780 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.102+0000 7f48b7e00640 1 -- 172.21.15.28:0/4006609141 >> 172.21.15.28:0/4006609141 conn(0x7f48b80fd9a0 msgr2=0x7f48b80ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.102+0000 7f48b7e00640 1 -- 172.21.15.28:0/4006609141 shutdown_connections 2024-04-24T08:21:04.104 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.102+0000 7f48b7e00640 1 -- 172.21.15.28:0/4006609141 wait complete. 2024-04-24T08:21:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.103+0000 7f48b7e00640 1 Processor -- start 2024-04-24T08:21:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.103+0000 7f48b7e00640 1 -- start start 2024-04-24T08:21:04.106 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.104+0000 7f48b7e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48b80743a0 0x7f48b819ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.104+0000 7f48b7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48b8074cc0 0x7f48b819f390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.104+0000 7f48b7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48b8112ec0 0x7f48b81a36c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.104+0000 7f48b7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f48b8118890 con 0x7f48b8074cc0 2024-04-24T08:21:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.104+0000 7f48b7e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f48b8118710 con 0x7f48b80743a0 2024-04-24T08:21:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.104+0000 7f48b7e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f48b8118a10 con 0x7f48b8112ec0 2024-04-24T08:21:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.104+0000 7f48b6000640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48b8074cc0 0x7f48b819f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.104+0000 7f48b6a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48b80743a0 0x7f48b819ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:04.107 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.104+0000 7f48b6a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48b80743a0 0x7f48b819ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49692/0 (socket says 172.21.15.28:49692) 2024-04-24T08:21:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.104+0000 7f48b6a00640 1 -- 172.21.15.28:0/775753515 learned_addr learned my addr 172.21.15.28:0/775753515 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.104+0000 7f48b7400640 1 --2- 172.21.15.28:0/775753515 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48b8112ec0 0x7f48b81a36c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.105+0000 7f48b6a00640 1 -- 172.21.15.28:0/775753515 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48b8112ec0 msgr2=0x7f48b81a36c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.105+0000 7f48b6a00640 1 --2- 172.21.15.28:0/775753515 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48b8112ec0 0x7f48b81a36c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.105+0000 7f48b6a00640 1 -- 172.21.15.28:0/775753515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48b8074cc0 msgr2=0x7f48b819f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.105+0000 7f48b6a00640 1 --2- 172.21.15.28:0/775753515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48b8074cc0 0x7f48b819f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.105+0000 7f48b6a00640 1 -- 172.21.15.28:0/775753515 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f48b81a3df0 con 0x7f48b80743a0 2024-04-24T08:21:04.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.105+0000 7f48b6000640 1 --2- 172.21.15.28:0/775753515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48b8074cc0 0x7f48b819f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:21:04.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.105+0000 7f48b6a00640 1 --2- 172.21.15.28:0/775753515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48b80743a0 0x7f48b819ee50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f48a8006010 tx=0x7f48a8004040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:04.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.105+0000 7f48af400640 1 -- 172.21.15.28:0/775753515 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a803f030 con 0x7f48b80743a0 2024-04-24T08:21:04.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.106+0000 7f48b7e00640 1 -- 172.21.15.28:0/775753515 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f48b81a4080 con 0x7f48b80743a0 2024-04-24T08:21:04.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.106+0000 7f48af400640 1 -- 172.21.15.28:0/775753515 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f48a8004440 con 0x7f48b80743a0 2024-04-24T08:21:04.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.106+0000 7f48af400640 1 -- 172.21.15.28:0/775753515 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f48a8035070 con 0x7f48b80743a0 2024-04-24T08:21:04.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.106+0000 7f48b7e00640 1 -- 172.21.15.28:0/775753515 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f48b81ab970 con 0x7f48b80743a0 2024-04-24T08:21:04.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.107+0000 7f48b7e00640 1 -- 172.21.15.28:0/775753515 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f48b8069190 con 0x7f48b80743a0 2024-04-24T08:21:04.110 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.109+0000 7f48af400640 1 -- 172.21.15.28:0/775753515 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f48a8042030 con 0x7f48b80743a0 2024-04-24T08:21:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.110+0000 7f48af400640 1 --2- 172.21.15.28:0/775753515 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f489c0a3da0 0x7f489c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.110+0000 7f48af400640 1 -- 172.21.15.28:0/775753515 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7f48a80f0dd0 con 0x7f48b80743a0 2024-04-24T08:21:04.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.111+0000 7f48b6000640 1 --2- 172.21.15.28:0/775753515 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f489c0a3da0 0x7f489c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:04.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.111+0000 7f48b6000640 1 --2- 172.21.15.28:0/775753515 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f489c0a3da0 0x7f489c0a6260 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f48b8100100 tx=0x7f48a4005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:04.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.113+0000 7f48af400640 1 -- 172.21.15.28:0/775753515 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f48a80bcd70 con 0x7f48b80743a0 2024-04-24T08:21:04.323 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.322+0000 7f48b7e00640 1 -- 172.21.15.28:0/775753515 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f48b8105310 con 0x7f48b80743a0 2024-04-24T08:21:04.324 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.323+0000 7f48af400640 1 -- 172.21.15.28:0/775753515 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v99) ==== 100+0+11 (secure 0 0 0) 0x7f48a80c1070 con 0x7f48b80743a0 2024-04-24T08:21:04.325 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-24T08:21:04.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.326+0000 7f48b7e00640 1 -- 172.21.15.28:0/775753515 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f489c0a3da0 msgr2=0x7f489c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:04.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.326+0000 7f48b7e00640 1 --2- 172.21.15.28:0/775753515 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f489c0a3da0 0x7f489c0a6260 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f48b8100100 tx=0x7f48a4005f90 comp rx=0 tx=0).stop 2024-04-24T08:21:04.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.326+0000 7f48b7e00640 1 -- 172.21.15.28:0/775753515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48b80743a0 msgr2=0x7f48b819ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:04.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.326+0000 7f48b7e00640 1 --2- 172.21.15.28:0/775753515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48b80743a0 0x7f48b819ee50 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f48a8006010 tx=0x7f48a8004040 comp rx=0 tx=0).stop 2024-04-24T08:21:04.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.326+0000 7f48b7e00640 1 -- 172.21.15.28:0/775753515 shutdown_connections 2024-04-24T08:21:04.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.326+0000 7f48b7e00640 1 --2- 172.21.15.28:0/775753515 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f489c0a3da0 0x7f489c0a6260 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.326+0000 7f48b7e00640 1 --2- 172.21.15.28:0/775753515 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f48b8074cc0 0x7f48b819f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.326+0000 7f48b7e00640 1 --2- 172.21.15.28:0/775753515 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f48b8112ec0 0x7f48b81a36c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.326+0000 7f48b7e00640 1 --2- 172.21.15.28:0/775753515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48b80743a0 0x7f48b819ee50 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.326+0000 7f48b7e00640 1 -- 172.21.15.28:0/775753515 >> 172.21.15.28:0/775753515 conn(0x7f48b80fd9a0 msgr2=0x7f48b80ff2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:04.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.327+0000 7f48b7e00640 1 -- 172.21.15.28:0/775753515 shutdown_connections 2024-04-24T08:21:04.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.327+0000 7f48b7e00640 1 -- 172.21.15.28:0/775753515 wait complete. 2024-04-24T08:21:04.347 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-04-24T08:21:04.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.560+0000 7ffa77200640 1 -- 172.21.15.28:0/1295753934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa78109340 msgr2=0x7ffa78110f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:04.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.560+0000 7ffa77200640 1 --2- 172.21.15.28:0/1295753934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa78109340 0x7ffa78110f10 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7ffa70009ca0 tx=0x7ffa7002a8c0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.561 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.560+0000 7ffa77200640 1 -- 172.21.15.28:0/1295753934 shutdown_connections 2024-04-24T08:21:04.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.560+0000 7ffa77200640 1 --2- 172.21.15.28:0/1295753934 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa78109340 0x7ffa78110f10 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.560+0000 7ffa77200640 1 --2- 172.21.15.28:0/1295753934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa78104c20 0x7ffa78108c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.560+0000 7ffa77200640 1 --2- 172.21.15.28:0/1295753934 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa78104270 0x7ffa78104650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.560+0000 7ffa77200640 1 -- 172.21.15.28:0/1295753934 >> 172.21.15.28:0/1295753934 conn(0x7ffa780ffad0 msgr2=0x7ffa78101ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:04.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.560+0000 7ffa77200640 1 -- 172.21.15.28:0/1295753934 shutdown_connections 2024-04-24T08:21:04.562 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.560+0000 7ffa77200640 1 -- 172.21.15.28:0/1295753934 wait complete. 2024-04-24T08:21:04.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.561+0000 7ffa77200640 1 Processor -- start 2024-04-24T08:21:04.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.562+0000 7ffa77200640 1 -- start start 2024-04-24T08:21:04.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.562+0000 7ffa77200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa78104270 0x7ffa781a0610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:04.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.562+0000 7ffa77200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa78109340 0x7ffa781a0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:04.565 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.562+0000 7ffa77200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa7819a6d0 0x7ffa7819ab50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.562+0000 7ffa77200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ffa781143d0 con 0x7ffa78109340 2024-04-24T08:21:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.562+0000 7ffa77200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ffa78114250 con 0x7ffa78104270 2024-04-24T08:21:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.562+0000 7ffa77200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ffa78114550 con 0x7ffa7819a6d0 2024-04-24T08:21:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.563+0000 7ffa75e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa78104270 0x7ffa781a0610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.563+0000 7ffa75400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa78109340 0x7ffa781a0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.563+0000 7ffa75e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa78104270 0x7ffa781a0610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49718/0 (socket says 172.21.15.28:49718) 2024-04-24T08:21:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.563+0000 7ffa75e00640 1 -- 172.21.15.28:0/969528721 learned_addr learned my addr 172.21.15.28:0/969528721 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:21:04.566 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.563+0000 7ffa76800640 1 --2- 172.21.15.28:0/969528721 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa7819a6d0 0x7ffa7819ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:04.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.563+0000 7ffa75400640 1 -- 172.21.15.28:0/969528721 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa7819a6d0 msgr2=0x7ffa7819ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:04.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.563+0000 7ffa75400640 1 --2- 172.21.15.28:0/969528721 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa7819a6d0 0x7ffa7819ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.563+0000 7ffa75400640 1 -- 172.21.15.28:0/969528721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa78104270 msgr2=0x7ffa781a0610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:04.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.563+0000 7ffa75400640 1 --2- 172.21.15.28:0/969528721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa78104270 0x7ffa781a0610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:04.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.563+0000 7ffa75400640 1 -- 172.21.15.28:0/969528721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa7819b2d0 con 0x7ffa78109340 2024-04-24T08:21:04.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.563+0000 7ffa76800640 1 --2- 172.21.15.28:0/969528721 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa7819a6d0 0x7ffa7819ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:21:04.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.563+0000 7ffa75400640 1 --2- 172.21.15.28:0/969528721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa78109340 0x7ffa781a0b50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ffa6400e9c0 tx=0x7ffa6400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:04.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.564+0000 7ffa6ea00640 1 -- 172.21.15.28:0/969528721 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa6400ccc0 con 0x7ffa78109340 2024-04-24T08:21:04.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.564+0000 7ffa6ea00640 1 -- 172.21.15.28:0/969528721 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa64010910 con 0x7ffa78109340 2024-04-24T08:21:04.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.564+0000 7ffa77200640 1 -- 172.21.15.28:0/969528721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa781a73b0 con 0x7ffa78109340 2024-04-24T08:21:04.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.564+0000 7ffa6ea00640 1 -- 172.21.15.28:0/969528721 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa640188c0 con 0x7ffa78109340 2024-04-24T08:21:04.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.564+0000 7ffa77200640 1 -- 172.21.15.28:0/969528721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa781a7810 con 0x7ffa78109340 2024-04-24T08:21:04.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.565+0000 7ffa77200640 1 -- 172.21.15.28:0/969528721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa78105740 con 0x7ffa78109340 2024-04-24T08:21:04.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.568+0000 7ffa6ea00640 1 -- 172.21.15.28:0/969528721 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ffa64010ac0 con 0x7ffa78109340 2024-04-24T08:21:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.570+0000 7ffa6ea00640 1 --2- 172.21.15.28:0/969528721 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ffa5c0a3da0 0x7ffa5c0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:21:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.571+0000 7ffa6ea00640 1 -- 172.21.15.28:0/969528721 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 8600+0+0 (secure 0 0 0) 0x7ffa640d0be0 con 0x7ffa78109340 2024-04-24T08:21:04.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.571+0000 7ffa75e00640 1 --2- 172.21.15.28:0/969528721 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ffa5c0a3da0 0x7ffa5c0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:21:04.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.572+0000 7ffa75e00640 1 --2- 172.21.15.28:0/969528721 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ffa5c0a3da0 0x7ffa5c0a6260 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ffa68009c90 tx=0x7ffa68005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:21:04.577 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.577+0000 7ffa6ea00640 1 -- 172.21.15.28:0/969528721 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ffa6409d1d0 con 0x7ffa78109340 2024-04-24T08:21:04.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:04 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3258409987' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:21:04.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:04 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/775753515' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:21:04.843 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.843+0000 7ffa77200640 1 -- 172.21.15.28:0/969528721 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7ffa78069190 con 0x7ffa78109340 2024-04-24T08:21:04.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:04 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3258409987' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:21:04.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:04 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/775753515' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:21:04.879 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.878+0000 7ffa6ea00640 1 -- 172.21.15.28:0/969528721 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v27) ==== 81+0+746057 (secure 0 0 0) 0x7ffa6409cad0 con 0x7ffa78109340 2024-04-24T08:21:04.891 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:21:04.891 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:21:04.891 INFO:teuthology.orchestra.run.smithi028.stdout: "epoch": 27, 2024-04-24T08:21:04.891 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: "active_gid": 25225, 2024-04-24T08:21:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: "active_name": "x", 2024-04-24T08:21:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addrs": { 2024-04-24T08:21:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-24T08:21:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-24T08:21:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6800", 2024-04-24T08:21:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 353036813 2024-04-24T08:21:04.892 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v1", 2024-04-24T08:21:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:6801", 2024-04-24T08:21:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 353036813 2024-04-24T08:21:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-24T08:21:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.893 INFO:teuthology.orchestra.run.smithi028.stdout: "active_addr": "172.21.15.28:6801/353036813", 2024-04-24T08:21:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: "active_change": "2024-04-24T08:10:20.116363+0000", 2024-04-24T08:21:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: "active_mgr_features": 4540701547738038271, 2024-04-24T08:21:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: "available": true, 2024-04-24T08:21:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: "standbys": [ 2024-04-24T08:21:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 25193, 2024-04-24T08:21:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "y", 2024-04-24T08:21:04.894 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-04-24T08:21:04.895 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-04-24T08:21:04.895 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:04.895 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-04-24T08:21:04.895 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:04.895 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:04.895 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:04.895 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-24T08:21:04.895 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-24T08:21:04.895 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:04.896 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.896 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.896 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-24T08:21:04.896 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.896 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.896 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.896 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-04-24T08:21:04.896 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.897 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.897 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.897 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:04.897 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:04.897 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.897 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.897 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.898 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:04.898 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.898 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.898 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:04.898 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:04.898 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:04.898 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:04.898 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:04.899 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:04.899 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:04.899 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:04.899 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:04.899 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.899 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.899 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.900 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:04.900 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:04.900 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:04.900 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.900 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.900 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:04.900 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.900 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.901 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.901 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:04.901 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.901 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.901 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.901 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.901 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:04.901 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:04.902 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.902 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.902 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.902 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:04.902 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.902 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.902 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:04.902 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:04.903 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:04.903 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:04.903 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:04.903 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:04.903 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:04.903 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:04.903 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:04.903 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.903 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.904 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.904 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:04.904 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:04.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:04.904 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.904 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.904 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:04.904 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-04-24T08:21:04.905 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-04-24T08:21:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.906 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-04-24T08:21:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-04-24T08:21:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-04-24T08:21:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.907 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.908 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.908 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-04-24T08:21:04.908 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.908 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.908 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.908 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-04-24T08:21:04.908 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.908 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.908 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.909 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-04-24T08:21:04.909 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-04-24T08:21:04.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.909 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.909 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:04.909 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.910 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.910 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.910 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-04-24T08:21:04.910 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.910 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.910 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.910 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-04-24T08:21:04.911 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-04-24T08:21:04.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.911 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.911 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.911 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:04.911 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.911 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.911 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.912 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-04-24T08:21:04.912 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.912 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.912 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.912 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-04-24T08:21:04.912 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-04-24T08:21:04.912 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:04.913 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.913 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.913 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-04-24T08:21:04.913 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.913 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.913 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.913 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-04-24T08:21:04.913 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.914 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.914 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.914 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-04-24T08:21:04.914 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-04-24T08:21:04.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.914 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.915 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.915 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:04.915 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.915 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.915 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.915 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-04-24T08:21:04.915 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.915 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.916 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.916 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-04-24T08:21:04.916 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-04-24T08:21:04.916 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:04.916 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.916 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.917 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:04.917 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.917 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.917 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.917 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-24T08:21:04.917 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.917 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.917 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.918 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-04-24T08:21:04.918 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-04-24T08:21:04.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.918 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.918 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.918 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:04.918 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.919 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.919 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.919 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-04-24T08:21:04.919 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.919 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.919 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.919 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:04.919 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:04.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.920 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:04.920 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-04-24T08:21:04.920 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:04.920 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:04.920 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:04.920 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-04-24T08:21:04.920 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-04-24T08:21:04.921 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:04.921 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.921 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.921 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:04.921 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.921 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.921 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.921 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-04-24T08:21:04.922 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.922 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.922 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.922 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-04-24T08:21:04.922 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-04-24T08:21:04.922 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.923 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.923 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.923 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-04-24T08:21:04.923 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.923 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.923 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.923 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-04-24T08:21:04.923 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T08:21:04.923 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.924 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.924 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-04-24T08:21:04.924 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-04-24T08:21:04.924 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-24T08:21:04.924 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.924 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.924 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-24T08:21:04.925 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-24T08:21:04.925 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-24T08:21:04.925 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.925 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-24T08:21:04.925 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T08:21:04.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.925 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.926 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-04-24T08:21:04.926 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-04-24T08:21:04.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-24T08:21:04.926 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.926 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.926 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-04-24T08:21:04.926 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-24T08:21:04.926 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-04-24T08:21:04.927 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.927 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-24T08:21:04.927 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.927 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.927 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.927 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-04-24T08:21:04.927 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-04-24T08:21:04.928 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.928 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.928 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.928 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-04-24T08:21:04.928 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.928 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.928 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.928 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-24T08:21:04.928 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-24T08:21:04.929 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.929 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.929 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-04-24T08:21:04.929 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-04-24T08:21:04.929 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:04.929 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.929 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.930 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-04-24T08:21:04.930 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-04-24T08:21:04.930 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-04-24T08:21:04.930 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.930 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-04-24T08:21:04.930 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-24T08:21:04.930 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.930 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.931 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-04-24T08:21:04.931 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-04-24T08:21:04.931 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.931 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.931 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.931 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-04-24T08:21:04.931 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.931 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.932 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.932 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-04-24T08:21:04.932 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T08:21:04.932 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.932 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.932 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-04-24T08:21:04.932 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-04-24T08:21:04.933 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-24T08:21:04.933 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.933 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.933 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-24T08:21:04.933 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-24T08:21:04.933 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-24T08:21:04.933 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.933 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-24T08:21:04.934 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T08:21:04.934 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.934 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.934 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.934 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:04.934 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:04.934 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.934 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.935 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.935 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:04.935 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.935 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.935 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:04.935 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:04.935 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:04.935 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:04.935 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:04.936 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:04.936 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:04.936 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:04.936 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:04.936 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.936 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.936 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.937 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:04.937 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:04.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:04.937 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.937 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:04.937 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.937 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.938 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.938 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:04.938 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.938 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.938 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.938 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:04.938 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:04.939 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:04.940 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.941 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-04-24T08:21:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-04-24T08:21:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-24T08:21:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.943 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-24T08:21:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.944 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-24T08:21:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-24T08:21:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-04-24T08:21:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.945 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-04-24T08:21:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-04-24T08:21:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-04-24T08:21:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-04-24T08:21:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-04-24T08:21:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-04-24T08:21:04.946 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-04-24T08:21:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-04-24T08:21:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-24T08:21:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.948 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-24T08:21:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-24T08:21:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.949 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-24T08:21:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-24T08:21:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.950 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-04-24T08:21:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-04-24T08:21:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-24T08:21:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-24T08:21:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.951 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-24T08:21:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-24T08:21:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-04-24T08:21:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-04-24T08:21:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-24T08:21:04.952 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-24T08:21:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.953 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-04-24T08:21:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:04.954 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-04-24T08:21:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-04-24T08:21:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-24T08:21:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.955 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-24T08:21:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.956 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-04-24T08:21:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-04-24T08:21:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-04-24T08:21:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.957 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-24T08:21:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-04-24T08:21:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-04-24T08:21:04.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-04-24T08:21:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-24T08:21:04.959 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.960 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.960 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.960 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-04-24T08:21:04.960 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-04-24T08:21:04.960 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:04.960 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.960 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-24T08:21:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-24T08:21:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.961 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-04-24T08:21:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-04-24T08:21:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-24T08:21:04.962 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-24T08:21:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.963 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-04-24T08:21:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-04-24T08:21:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-04-24T08:21:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.964 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-24T08:21:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-04-24T08:21:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-04-24T08:21:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.965 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-04-24T08:21:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-04-24T08:21:04.966 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-04-24T08:21:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-24T08:21:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-04-24T08:21:04.967 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-04-24T08:21:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-24T08:21:04.968 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-04-24T08:21:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-04-24T08:21:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.969 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-24T08:21:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.970 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-04-24T08:21:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-04-24T08:21:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.971 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-24T08:21:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-04-24T08:21:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-04-24T08:21:04.972 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-24T08:21:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-24T08:21:04.973 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-04-24T08:21:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-04-24T08:21:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:04.974 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-24T08:21:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-04-24T08:21:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-04-24T08:21:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-04-24T08:21:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-24T08:21:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.976 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-04-24T08:21:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-04-24T08:21:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-04-24T08:21:04.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-24T08:21:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-24T08:21:04.978 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-04-24T08:21:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-04-24T08:21:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-24T08:21:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-04-24T08:21:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.980 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-04-24T08:21:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-04-24T08:21:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-24T08:21:04.981 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-04-24T08:21:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.982 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-04-24T08:21:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-04-24T08:21:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-24T08:21:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.983 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-04-24T08:21:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-04-24T08:21:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-04-24T08:21:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.984 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-24T08:21:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-04-24T08:21:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.985 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-04-24T08:21:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-04-24T08:21:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.986 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-24T08:21:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-04-24T08:21:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-04-24T08:21:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-04-24T08:21:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-24T08:21:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.988 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-04-24T08:21:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-04-24T08:21:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-04-24T08:21:04.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-24T08:21:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-24T08:21:04.990 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-04-24T08:21:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-04-24T08:21:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.991 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-24T08:21:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-04-24T08:21:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-04-24T08:21:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-04-24T08:21:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-24T08:21:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.993 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-24T08:21:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-04-24T08:21:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-04-24T08:21:04.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-24T08:21:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-04-24T08:21:04.995 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-04-24T08:21:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-04-24T08:21:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.996 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-24T08:21:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-04-24T08:21:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.997 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-04-24T08:21:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-04-24T08:21:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-24T08:21:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:04.998 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-04-24T08:21:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-04-24T08:21:04.999 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-04-24T08:21:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.000 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-04-24T08:21:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-04-24T08:21:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-04-24T08:21:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-24T08:21:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-24T08:21:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.002 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-04-24T08:21:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-04-24T08:21:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-04-24T08:21:05.003 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-24T08:21:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.004 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-04-24T08:21:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-04-24T08:21:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-04-24T08:21:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.005 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-24T08:21:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-04-24T08:21:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-04-24T08:21:05.006 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-04-24T08:21:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-04-24T08:21:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.007 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-04-24T08:21:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-04-24T08:21:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.008 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-24T08:21:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.009 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-04-24T08:21:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-04-24T08:21:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-24T08:21:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.010 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-04-24T08:21:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-04-24T08:21:05.011 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-04-24T08:21:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-24T08:21:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-24T08:21:05.012 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-04-24T08:21:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-04-24T08:21:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.013 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-24T08:21:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-04-24T08:21:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.014 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-04-24T08:21:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-04-24T08:21:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.015 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-24T08:21:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-04-24T08:21:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-04-24T08:21:05.016 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-24T08:21:05.017 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.018 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.019 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-04-24T08:21:05.020 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-04-24T08:21:05.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.021 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.021 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.021 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.021 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.021 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.021 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-24T08:21:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.022 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-24T08:21:05.023 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.024 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.025 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.026 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.027 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.027 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.027 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.027 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.027 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.027 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.027 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.027 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.027 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.028 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.028 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.028 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.028 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-24T08:21:05.028 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-24T08:21:05.028 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.028 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.028 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.029 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.029 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.029 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.029 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.029 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-24T08:21:05.029 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.029 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.029 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.030 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-24T08:21:05.030 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-24T08:21:05.030 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.030 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.030 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.030 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-04-24T08:21:05.030 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.031 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.031 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.031 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-24T08:21:05.031 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.031 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.031 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.031 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-04-24T08:21:05.031 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-04-24T08:21:05.032 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.032 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.032 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.032 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:05.032 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.032 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.033 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-04-24T08:21:05.033 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.033 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.033 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.033 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-04-24T08:21:05.033 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-04-24T08:21:05.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.033 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.034 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:05.034 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.034 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.034 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.034 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-24T08:21:05.034 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.034 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.035 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.035 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-04-24T08:21:05.035 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-04-24T08:21:05.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.035 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.035 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.035 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.036 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.036 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.036 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.036 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-04-24T08:21:05.036 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.036 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.036 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.037 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-24T08:21:05.037 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-24T08:21:05.037 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.037 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.037 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.037 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-04-24T08:21:05.037 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.037 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.037 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.038 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-04-24T08:21:05.038 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-04-24T08:21:05.038 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.038 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-04-24T08:21:05.038 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.038 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.038 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.039 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-04-24T08:21:05.039 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-04-24T08:21:05.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.039 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.039 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.039 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-04-24T08:21:05.039 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.039 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.039 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.040 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-24T08:21:05.040 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.040 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.040 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-04-24T08:21:05.040 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-04-24T08:21:05.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.041 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.041 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.041 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-24T08:21:05.041 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.041 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.041 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.041 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-24T08:21:05.041 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.042 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.042 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.042 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-04-24T08:21:05.042 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-04-24T08:21:05.042 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.042 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.043 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.043 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.043 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.043 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.043 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.043 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-24T08:21:05.043 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.043 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.043 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.044 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-04-24T08:21:05.044 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-04-24T08:21:05.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.044 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.044 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.044 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.045 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.045 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.045 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.045 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-24T08:21:05.045 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.045 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.045 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.045 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.046 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-04-24T08:21:05.046 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-04-24T08:21:05.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.046 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.046 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.046 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.046 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.046 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.046 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.047 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-24T08:21:05.050 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.050 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.050 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.050 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-04-24T08:21:05.050 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-04-24T08:21:05.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.050 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.051 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.051 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.051 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.051 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.051 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.051 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-24T08:21:05.051 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.051 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.052 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.052 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-04-24T08:21:05.052 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-04-24T08:21:05.052 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.052 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.052 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.052 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.053 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.053 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.053 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.053 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-24T08:21:05.053 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.053 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.053 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.053 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-04-24T08:21:05.054 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-04-24T08:21:05.054 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.054 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.054 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.054 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-04-24T08:21:05.054 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.054 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.054 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.055 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-04-24T08:21:05.055 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.055 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.055 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.055 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-04-24T08:21:05.055 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-04-24T08:21:05.055 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.055 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.056 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.056 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.056 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.056 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.056 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.056 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-24T08:21:05.056 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.056 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.056 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.057 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-04-24T08:21:05.057 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-04-24T08:21:05.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.057 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.057 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.057 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.057 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.058 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.058 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.058 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-24T08:21:05.058 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.058 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.058 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.058 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-04-24T08:21:05.059 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-04-24T08:21:05.059 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.059 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.059 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.059 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.059 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.059 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.059 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.059 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-24T08:21:05.060 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.060 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.060 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.060 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-04-24T08:21:05.060 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-04-24T08:21:05.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.060 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.060 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.061 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.061 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.061 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.061 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.061 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-04-24T08:21:05.061 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.061 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.061 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.062 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-04-24T08:21:05.062 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-04-24T08:21:05.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.062 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.062 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.062 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.062 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.063 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.063 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-24T08:21:05.063 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.063 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.063 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.063 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.063 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-04-24T08:21:05.063 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-04-24T08:21:05.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.064 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.064 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.064 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.064 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.064 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.064 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.064 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-24T08:21:05.064 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.065 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.065 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.065 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.065 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.065 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.065 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.065 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-04-24T08:21:05.065 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.066 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.066 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.066 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.066 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.066 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.066 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.067 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.067 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.067 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.067 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.067 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.067 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.067 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.067 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.067 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.068 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.068 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.068 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.068 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.068 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.068 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.068 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.068 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.068 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.068 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.069 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.069 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.069 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.069 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.069 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.069 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.069 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.069 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.069 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.070 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.070 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.070 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.070 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.070 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.070 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.070 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.071 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.071 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.071 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.071 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.071 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.071 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.071 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.071 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.071 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.072 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.072 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.072 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.072 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.072 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.072 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.072 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.072 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.073 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.073 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.073 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.073 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.073 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.073 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.073 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.074 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.074 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.074 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-04-24T08:21:05.074 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-04-24T08:21:05.074 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.074 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.074 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.075 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-04-24T08:21:05.075 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.075 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.075 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.075 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-24T08:21:05.075 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.075 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.075 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.075 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.076 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-04-24T08:21:05.076 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-04-24T08:21:05.076 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.076 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.076 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-04-24T08:21:05.076 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.077 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.077 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.077 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-24T08:21:05.077 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.077 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.077 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.077 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.077 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.078 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.078 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-04-24T08:21:05.078 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.078 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.078 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.078 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-24T08:21:05.078 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-24T08:21:05.078 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.079 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.079 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.079 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:05.079 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.079 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.079 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.079 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.079 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.080 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.080 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.080 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-04-24T08:21:05.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-24T08:21:05.080 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.080 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.080 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.080 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.081 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.081 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.081 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.081 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.081 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.081 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.081 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.082 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-24T08:21:05.082 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-24T08:21:05.082 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.082 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.082 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.082 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.082 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.083 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.083 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.083 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.083 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.083 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.083 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-04-24T08:21:05.083 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-04-24T08:21:05.084 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.084 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.084 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.084 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.084 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.084 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.084 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.084 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.084 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.085 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.085 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.085 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-24T08:21:05.085 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-24T08:21:05.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.085 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.085 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.086 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.086 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.086 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.086 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.086 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.086 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.086 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.086 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.087 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-04-24T08:21:05.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-24T08:21:05.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.087 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.087 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.087 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.088 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.088 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.088 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.088 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.088 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.088 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.088 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.088 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-24T08:21:05.089 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-24T08:21:05.089 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.089 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.089 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.089 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.089 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.089 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.089 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.090 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.090 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.090 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.090 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.090 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.090 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-24T08:21:05.090 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-24T08:21:05.090 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.091 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.091 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.091 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.091 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.091 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.091 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.091 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.091 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.092 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.092 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.092 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-24T08:21:05.092 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-24T08:21:05.092 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.092 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.092 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.093 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.093 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.093 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.093 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.093 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.093 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.093 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.093 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.094 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-24T08:21:05.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-24T08:21:05.094 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.094 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.094 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.094 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.094 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.094 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.095 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.095 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.095 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.095 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.095 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.095 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-24T08:21:05.095 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-24T08:21:05.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.096 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.096 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.096 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.096 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.096 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.097 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.097 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.097 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.097 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.097 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-24T08:21:05.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-24T08:21:05.097 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.098 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.098 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.098 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.098 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.098 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.098 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.098 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.098 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.099 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.099 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.099 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-24T08:21:05.099 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-24T08:21:05.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.099 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.099 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.100 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.100 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.100 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.100 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.100 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.100 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.100 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.100 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-24T08:21:05.101 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-24T08:21:05.101 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.101 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.101 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.101 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.101 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.101 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.101 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.102 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.102 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.102 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.102 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.102 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-04-24T08:21:05.102 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-24T08:21:05.102 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.103 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.103 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.103 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-24T08:21:05.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.103 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.103 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.103 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.103 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.104 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.104 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.104 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-24T08:21:05.104 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-24T08:21:05.104 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.104 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.104 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.104 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.105 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.105 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.105 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.105 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.105 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.105 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.105 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.106 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-04-24T08:21:05.106 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-04-24T08:21:05.106 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.106 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.106 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.106 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.106 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.106 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.107 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.107 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.107 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.107 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.107 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.107 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-04-24T08:21:05.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-24T08:21:05.108 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.108 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.108 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.108 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-24T08:21:05.108 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.108 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.108 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.108 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.109 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.109 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.109 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.109 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-24T08:21:05.109 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-24T08:21:05.109 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.109 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.110 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.110 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.110 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.110 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.110 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.110 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.110 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.110 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.111 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.111 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-24T08:21:05.111 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-24T08:21:05.111 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.111 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.112 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.112 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.112 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.112 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.112 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.112 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.112 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.112 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.113 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-24T08:21:05.113 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-24T08:21:05.113 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.113 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.113 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.113 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.113 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.113 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.114 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.114 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.114 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.114 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.114 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.114 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-24T08:21:05.114 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-24T08:21:05.115 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.115 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.115 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.115 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.115 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.115 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.115 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.115 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.116 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.116 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.116 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.116 INFO:teuthology.orchestra.run.smithi028.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-24T08:21:05.116 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-24T08:21:05.116 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.116 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.116 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.117 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "[]", 2024-04-24T08:21:05.117 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.117 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.117 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.117 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.117 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.117 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.117 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.117 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.118 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-04-24T08:21:05.118 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-24T08:21:05.118 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.118 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.118 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.118 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-04-24T08:21:05.118 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.118 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.119 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.119 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.119 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.119 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.119 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.119 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.119 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-04-24T08:21:05.119 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-24T08:21:05.120 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.120 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.120 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.120 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.120 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.120 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.120 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.120 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.120 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.121 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.121 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.121 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.121 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-24T08:21:05.121 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-24T08:21:05.121 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.121 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.121 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.121 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.122 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.122 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.122 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.122 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.122 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.122 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.122 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.122 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.123 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-24T08:21:05.123 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-24T08:21:05.123 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.123 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.123 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.123 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.123 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.123 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.123 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.123 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.124 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.124 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.124 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.124 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-24T08:21:05.124 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-24T08:21:05.124 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.124 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.124 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.125 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.125 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.125 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.125 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.125 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.125 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.125 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.125 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.126 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-24T08:21:05.126 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-24T08:21:05.126 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.126 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.126 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.126 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.126 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.126 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.127 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.127 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.127 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.127 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.127 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-24T08:21:05.127 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-24T08:21:05.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.128 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.128 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.128 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.128 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.128 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.128 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.128 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.129 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.129 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.129 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-24T08:21:05.129 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-24T08:21:05.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.129 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.129 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.129 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.130 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.130 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.130 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.130 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.130 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.130 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.130 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.131 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-24T08:21:05.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-24T08:21:05.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.131 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.131 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.131 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.131 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.132 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.132 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.132 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.132 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.132 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.132 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.132 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-24T08:21:05.132 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-24T08:21:05.133 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.133 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.133 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.133 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.133 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.133 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.133 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.133 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.134 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.134 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.134 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.134 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-04-24T08:21:05.134 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-24T08:21:05.134 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.134 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.135 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.135 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.135 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.135 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.135 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.135 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.135 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.135 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.135 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.136 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-24T08:21:05.136 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-24T08:21:05.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.136 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.136 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.136 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-24T08:21:05.136 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.137 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.137 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.137 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.137 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.137 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.137 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-24T08:21:05.138 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-24T08:21:05.138 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.138 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.138 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.138 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:05.138 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.138 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.138 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.139 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.139 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.139 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.139 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.139 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-24T08:21:05.139 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-24T08:21:05.139 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.139 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.140 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-04-24T08:21:05.140 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.140 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.140 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.140 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.140 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.140 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.141 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.141 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-24T08:21:05.141 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-24T08:21:05.141 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.141 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.141 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.141 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-04-24T08:21:05.141 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.142 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-04-24T08:21:05.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-24T08:21:05.143 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.143 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.143 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.143 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.143 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.143 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.143 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.143 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.144 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.144 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.144 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.144 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-24T08:21:05.144 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-24T08:21:05.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.144 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.145 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-24T08:21:05.145 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.145 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.145 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.145 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.145 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.145 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.145 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.146 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.146 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-04-24T08:21:05.146 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-24T08:21:05.146 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.146 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.146 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.146 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.147 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.147 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.147 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.147 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.147 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.147 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.147 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.148 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-04-24T08:21:05.148 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-24T08:21:05.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.148 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.148 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.148 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.148 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.148 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.148 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.149 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.149 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-04-24T08:21:05.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-24T08:21:05.149 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.149 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.150 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.150 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.150 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.150 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.150 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.150 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.150 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.151 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.151 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.151 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-24T08:21:05.151 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-24T08:21:05.151 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.151 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.151 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.151 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-24T08:21:05.151 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.152 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.152 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.152 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.152 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.152 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.152 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.152 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-24T08:21:05.152 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-24T08:21:05.153 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.153 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.153 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.153 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:05.153 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.153 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.153 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.153 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.154 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.154 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.154 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.154 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.154 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-24T08:21:05.154 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-24T08:21:05.154 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.155 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.155 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.155 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-24T08:21:05.155 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.155 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.155 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.155 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.155 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.155 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.156 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.156 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-04-24T08:21:05.156 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-04-24T08:21:05.156 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.156 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.156 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.157 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.157 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.157 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.157 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.157 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.157 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.157 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-04-24T08:21:05.158 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-04-24T08:21:05.158 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.158 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.158 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.158 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.158 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.158 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.158 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.159 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.159 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.159 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.159 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.159 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-04-24T08:21:05.159 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-04-24T08:21:05.159 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.159 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.160 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.160 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.160 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.160 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.160 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.160 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-04-24T08:21:05.160 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.160 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.161 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.161 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-04-24T08:21:05.161 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-04-24T08:21:05.161 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.161 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.161 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.161 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-04-24T08:21:05.161 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.162 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.162 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.162 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.162 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.162 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.162 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.162 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.162 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-24T08:21:05.163 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-24T08:21:05.163 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.163 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.163 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.163 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.163 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.163 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.163 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.163 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.164 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.164 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.164 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.164 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.164 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.164 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.165 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.165 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.165 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.165 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.165 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.165 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.165 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.165 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.166 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.166 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.166 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.166 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.166 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.166 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.166 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.167 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.167 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.167 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.167 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.167 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.167 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.167 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.167 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.167 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.168 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.168 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.168 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.168 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.168 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.168 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.168 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.169 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.169 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.169 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.169 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.169 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.169 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.169 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.169 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.170 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.170 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.170 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.170 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.170 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.170 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.170 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.170 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.171 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.171 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.171 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.171 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.171 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.171 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.171 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.171 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.172 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.172 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.172 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.172 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.172 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.172 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.172 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-04-24T08:21:05.173 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-04-24T08:21:05.173 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.173 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.173 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.173 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.173 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.173 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.173 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.173 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-04-24T08:21:05.174 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.174 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.174 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.174 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.174 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-04-24T08:21:05.174 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-04-24T08:21:05.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.174 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.175 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.175 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.175 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.175 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.175 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.175 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.175 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.175 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.176 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.176 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-24T08:21:05.176 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-24T08:21:05.176 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.176 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.176 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.176 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-24T08:21:05.176 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.177 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.177 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.177 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.177 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.177 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.177 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.177 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.177 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-24T08:21:05.178 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-24T08:21:05.178 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.178 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.178 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.178 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-04-24T08:21:05.178 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.178 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.178 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.179 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.179 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.179 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.179 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.179 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-24T08:21:05.179 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-24T08:21:05.179 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.180 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.180 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.180 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.180 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.180 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.180 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.180 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.180 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.181 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.181 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-04-24T08:21:05.181 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-04-24T08:21:05.181 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.181 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.181 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.181 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-04-24T08:21:05.182 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.182 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.182 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.182 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.182 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.182 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.182 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.182 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.182 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-24T08:21:05.183 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-24T08:21:05.183 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.183 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.183 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.183 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-04-24T08:21:05.183 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.183 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.183 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.184 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.184 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-04-24T08:21:05.184 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.184 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.184 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.184 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.184 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.184 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.184 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-24T08:21:05.185 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-24T08:21:05.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.185 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.185 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.185 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-24T08:21:05.185 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-24T08:21:05.185 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-24T08:21:05.185 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.185 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.186 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.186 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.186 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.186 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.186 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-04-24T08:21:05.186 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-04-24T08:21:05.186 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.186 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.187 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-24T08:21:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.188 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-04-24T08:21:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-04-24T08:21:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.189 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.190 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.190 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.190 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-04-24T08:21:05.190 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.190 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.190 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.190 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.190 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.191 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.191 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.191 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.191 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.191 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.191 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.192 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.192 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.192 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.192 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.192 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.192 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.192 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.192 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.193 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.193 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.193 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.193 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.193 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.193 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.193 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.193 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.194 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.194 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.194 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.194 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.194 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.194 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.194 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.194 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.195 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.195 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.195 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.195 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.195 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.195 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.196 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.196 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.196 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.196 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.196 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.196 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.196 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.196 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.196 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.197 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.197 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.197 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.197 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.197 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.197 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.198 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.198 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.198 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.198 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.198 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.198 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.198 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.198 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.198 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.199 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.199 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-04-24T08:21:05.199 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-04-24T08:21:05.199 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.199 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.199 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.199 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-04-24T08:21:05.200 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.200 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.200 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.200 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-24T08:21:05.200 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.200 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.200 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.201 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-04-24T08:21:05.201 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-04-24T08:21:05.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.201 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.201 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.201 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-04-24T08:21:05.201 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.202 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.202 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.202 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-24T08:21:05.202 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.202 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.202 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.202 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.202 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-04-24T08:21:05.203 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-04-24T08:21:05.203 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.203 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.203 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.203 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-04-24T08:21:05.203 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.203 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.203 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.204 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-04-24T08:21:05.204 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.204 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.204 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.204 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.204 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-04-24T08:21:05.204 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-04-24T08:21:05.204 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.205 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.205 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.205 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-24T08:21:05.205 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.205 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.205 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.205 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-24T08:21:05.205 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.205 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.206 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.206 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.206 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-04-24T08:21:05.206 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-04-24T08:21:05.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.206 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.206 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.207 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.207 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.207 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.207 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-24T08:21:05.207 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.208 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-24T08:21:05.208 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-24T08:21:05.208 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.208 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.208 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.208 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-24T08:21:05.208 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.208 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.209 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.209 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-04-24T08:21:05.209 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.209 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.209 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.209 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.209 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-04-24T08:21:05.209 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-04-24T08:21:05.209 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.210 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.210 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-04-24T08:21:05.210 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.210 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.210 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.210 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-24T08:21:05.210 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.211 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.211 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.211 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.211 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.211 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.211 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-04-24T08:21:05.211 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.211 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.212 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.212 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.212 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.212 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.212 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.213 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.213 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.213 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.213 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.213 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.213 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.213 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.213 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.214 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.214 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.214 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.214 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.214 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.214 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.214 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.214 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.215 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.215 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.215 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.215 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.215 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.215 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.216 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.216 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.216 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.216 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.216 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.216 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.216 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.216 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.217 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.217 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.217 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.217 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.217 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.217 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.217 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.217 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.218 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.218 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.218 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.218 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.218 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.218 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.218 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.219 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.219 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.219 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.219 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.219 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.219 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.219 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.219 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.219 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.220 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.220 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.220 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.220 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.220 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-04-24T08:21:05.220 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-04-24T08:21:05.220 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.221 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.221 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.221 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-24T08:21:05.221 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.221 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.221 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.221 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.222 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.222 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.222 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.222 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-04-24T08:21:05.222 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-04-24T08:21:05.222 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.222 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.222 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.222 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-04-24T08:21:05.223 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.223 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.223 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.223 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.223 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.223 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.223 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.224 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.224 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-24T08:21:05.224 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-24T08:21:05.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.224 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.224 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.224 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-24T08:21:05.224 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.224 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.225 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.225 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.225 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.225 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.225 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.225 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.225 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.226 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.226 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.226 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-04-24T08:21:05.226 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-04-24T08:21:05.226 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-04-24T08:21:05.226 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.226 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-04-24T08:21:05.226 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-04-24T08:21:05.227 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.227 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.227 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.227 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-04-24T08:21:05.227 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.227 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.227 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.227 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-24T08:21:05.228 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.228 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.228 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.228 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-04-24T08:21:05.228 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-04-24T08:21:05.228 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.228 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.228 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.229 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-04-24T08:21:05.229 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.229 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.229 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.229 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-24T08:21:05.229 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.229 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.229 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.230 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-04-24T08:21:05.230 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-04-24T08:21:05.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.230 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.230 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.230 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.230 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.231 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.231 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.231 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-04-24T08:21:05.231 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.231 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.231 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.231 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.231 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-24T08:21:05.232 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-24T08:21:05.232 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.232 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.232 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.232 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-04-24T08:21:05.232 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-04-24T08:21:05.232 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.232 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.232 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-24T08:21:05.233 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.233 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.233 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.233 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.233 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.233 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.234 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.234 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.234 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.234 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.234 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.234 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.234 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.234 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.235 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.235 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.235 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.235 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.235 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.235 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.235 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.235 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.235 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.236 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.236 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.236 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.236 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.236 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.236 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.237 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.237 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.237 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.237 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.237 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.237 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.237 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.237 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.238 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.238 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.238 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.238 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.238 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.238 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.238 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.239 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.239 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.239 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.239 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.239 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.239 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.239 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.240 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.241 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.241 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.241 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.241 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.241 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.241 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.241 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.241 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-04-24T08:21:05.241 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-04-24T08:21:05.242 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.242 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.242 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.242 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.242 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.242 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.243 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-04-24T08:21:05.243 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.243 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.243 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.243 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-04-24T08:21:05.243 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-04-24T08:21:05.243 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.243 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.244 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.244 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-04-24T08:21:05.244 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.244 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.244 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.244 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-04-24T08:21:05.244 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.244 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.245 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.245 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.245 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-24T08:21:05.245 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-24T08:21:05.245 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.245 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.245 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.245 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-04-24T08:21:05.246 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.246 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.246 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.246 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T08:21:05.246 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.246 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.246 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.247 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-04-24T08:21:05.247 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-04-24T08:21:05.247 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.247 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.247 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.247 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-24T08:21:05.247 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-24T08:21:05.247 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-04-24T08:21:05.248 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.248 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-24T08:21:05.248 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.248 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.248 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.248 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.248 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-04-24T08:21:05.248 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-04-24T08:21:05.248 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.249 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.249 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.249 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.249 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.249 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.249 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.249 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-04-24T08:21:05.249 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.250 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.250 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.250 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-04-24T08:21:05.250 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-04-24T08:21:05.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.250 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.250 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.253 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-04-24T08:21:05.253 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.253 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.254 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.254 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T08:21:05.254 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.254 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.254 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.254 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-04-24T08:21:05.255 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.255 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.256 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.256 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.256 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.256 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.256 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.256 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.257 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.257 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.257 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.257 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.257 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.257 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.257 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.258 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.258 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.258 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.258 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.258 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.258 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.258 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.259 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.259 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.259 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.259 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.259 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.260 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.260 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.260 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.260 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.260 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.260 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.260 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.261 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.261 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.261 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.261 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.261 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.262 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.262 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.262 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.262 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.262 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.262 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.262 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.262 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.263 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.263 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.263 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.263 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.263 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.263 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.263 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.264 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.264 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.264 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.264 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.264 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.264 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.264 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.265 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.265 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.265 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.265 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.265 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.265 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.265 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.266 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.266 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.266 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.266 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-04-24T08:21:05.266 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.266 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.266 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.267 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.267 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.267 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.267 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.267 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.267 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.267 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.268 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.268 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.268 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.268 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.268 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.268 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.268 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.269 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.269 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.269 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.269 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.269 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.269 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.270 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.270 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.270 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.270 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.270 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.271 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.271 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.271 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.271 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.271 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.271 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.271 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.272 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.272 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.272 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.272 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.272 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.272 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.272 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.273 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.273 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.273 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.273 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.273 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.273 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.273 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.273 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.274 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.274 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.274 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.274 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.274 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.275 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.275 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.275 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.275 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.275 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.275 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.275 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.276 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.277 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.277 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-04-24T08:21:05.277 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.277 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.278 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.278 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-04-24T08:21:05.278 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-04-24T08:21:05.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.278 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.278 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.278 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-04-24T08:21:05.279 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.279 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.279 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.279 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-24T08:21:05.279 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.279 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.280 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.280 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-04-24T08:21:05.280 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-04-24T08:21:05.280 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.280 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.280 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.281 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:05.281 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-04-24T08:21:05.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.281 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-24T08:21:05.281 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.281 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.282 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.282 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.282 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.282 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.283 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.283 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.283 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.283 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.283 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.283 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.284 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.284 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.284 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.284 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.284 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.284 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.284 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.285 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.285 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.285 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.285 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.285 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.286 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.286 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.286 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.286 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.286 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.286 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.286 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.287 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.287 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.287 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.287 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.287 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.287 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.288 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.288 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.288 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.288 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.288 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.288 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.288 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.289 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.289 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.289 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.289 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.289 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.289 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.289 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.290 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.290 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.290 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.290 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.290 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.290 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.291 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.291 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.291 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.291 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.291 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.291 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.291 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.292 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.292 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.292 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.292 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.292 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.292 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.292 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.292 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.293 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-04-24T08:21:05.293 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.293 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.293 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.293 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-04-24T08:21:05.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-04-24T08:21:05.293 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.294 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.294 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-04-24T08:21:05.294 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.294 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.294 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.294 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-04-24T08:21:05.294 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-24T08:21:05.295 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.295 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.295 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.295 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.295 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.295 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.295 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.296 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.296 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.296 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.296 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.296 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.296 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.296 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.297 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.298 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.298 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.298 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.298 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.298 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.299 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.300 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.301 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.302 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.302 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.302 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.302 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.302 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.302 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.302 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.303 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-04-24T08:21:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-04-24T08:21:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.305 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-04-24T08:21:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-24T08:21:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.306 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-04-24T08:21:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-04-24T08:21:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-04-24T08:21:05.307 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-04-24T08:21:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.308 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-04-24T08:21:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-04-24T08:21:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-04-24T08:21:05.309 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-04-24T08:21:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.310 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-04-24T08:21:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-04-24T08:21:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.311 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.312 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.312 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.312 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-04-24T08:21:05.312 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.312 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.312 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.312 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.312 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-04-24T08:21:05.313 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-04-24T08:21:05.313 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.313 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.313 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.313 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-04-24T08:21:05.313 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.313 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.314 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.314 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-24T08:21:05.314 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-24T08:21:05.314 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.314 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.314 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.314 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.315 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-04-24T08:21:05.315 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.315 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.315 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.315 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.316 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.316 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.316 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.316 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.316 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.316 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.316 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.317 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.317 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.317 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.317 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.317 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.317 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.317 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.317 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.318 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.318 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.318 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.318 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.318 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.319 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.319 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.319 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.319 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.319 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.319 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.319 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.319 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.320 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.320 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.320 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.320 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.320 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.321 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.321 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.321 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.321 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.321 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.322 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.322 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.322 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.322 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.322 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.322 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.322 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.323 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.323 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.323 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.323 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.323 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.323 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.323 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.324 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.324 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.324 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.324 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.324 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.324 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.325 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-04-24T08:21:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.326 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.327 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.328 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.329 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.330 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.331 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.332 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.333 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.335 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-04-24T08:21:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.338 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.339 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.340 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.341 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.342 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.342 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.342 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.342 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.342 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.342 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.343 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.343 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.343 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.343 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.343 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.343 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.343 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.344 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.344 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.344 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.344 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.344 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.344 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.344 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.345 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.345 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.345 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.345 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.345 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.345 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.345 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.346 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.346 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.346 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.346 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.346 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.346 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.347 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.347 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.347 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.347 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.347 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-24T08:21:05.347 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.347 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.348 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.348 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-04-24T08:21:05.348 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-04-24T08:21:05.348 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.348 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.348 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.348 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.348 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.349 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.349 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.349 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-24T08:21:05.349 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.349 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.350 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.350 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.350 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.350 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.350 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.350 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.351 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.351 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.351 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.351 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.351 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.351 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.351 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.352 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.352 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.352 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.352 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.352 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.352 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.352 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.353 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.353 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.353 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.353 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.353 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.353 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.353 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.354 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.354 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.354 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.354 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.354 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.354 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.354 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.355 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.355 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.355 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.355 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.355 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.355 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.355 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.356 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.356 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.356 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.356 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.356 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.356 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.356 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.356 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.357 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.357 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.357 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.357 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.357 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.357 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.357 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.358 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.358 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.358 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.358 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.358 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.358 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.358 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.358 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.359 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.359 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.359 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.359 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-04-24T08:21:05.359 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-24T08:21:05.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.359 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.360 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.360 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.360 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.360 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.360 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.360 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-24T08:21:05.360 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-04-24T08:21:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-04-24T08:21:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-04-24T08:21:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.361 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.362 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-04-24T08:21:05.362 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.362 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.362 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.362 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.363 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.364 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.364 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.364 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.364 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.364 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.364 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.364 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.364 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.364 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.365 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.365 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.365 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.365 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.365 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.366 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.366 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.366 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.366 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.366 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.366 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.366 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.367 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.367 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.367 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.367 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.367 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.368 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.368 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.368 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.368 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.368 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.368 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.368 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.369 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.369 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.369 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.369 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.369 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.369 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.369 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.370 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.370 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.370 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.370 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.370 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.370 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.371 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.371 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.371 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.371 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.371 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.371 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.371 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.372 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.372 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.372 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.372 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.372 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-04-24T08:21:05.372 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.372 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.373 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.373 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.373 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.373 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.373 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.373 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.373 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.373 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.374 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.374 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.374 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.374 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.374 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.374 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.374 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.375 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.375 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.375 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.375 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.375 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.375 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.375 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.376 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.376 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.376 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.376 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.376 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.376 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.377 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.377 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.377 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.377 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.377 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.377 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.377 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.378 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.378 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.378 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.378 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.379 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.379 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.379 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.379 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.379 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.379 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.379 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.379 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.380 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.380 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.380 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.380 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.380 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.380 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.381 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.381 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.381 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.381 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.381 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.382 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.382 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.382 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.382 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.382 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.382 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.382 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.383 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.383 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.383 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.383 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-04-24T08:21:05.383 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.383 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.383 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.384 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.384 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.384 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.384 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.384 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.384 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.384 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.385 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.385 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.385 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.385 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.385 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.385 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.385 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.386 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.386 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.386 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.386 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.386 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.386 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.386 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.387 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.389 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.390 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.392 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-24T08:21:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-24T08:21:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.393 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-24T08:21:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.394 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-04-24T08:21:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-04-24T08:21:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:05.395 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-24T08:21:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-04-24T08:21:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-04-24T08:21:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-24T08:21:05.396 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.397 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.397 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.397 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.397 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.397 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-04-24T08:21:05.397 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.397 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.397 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.398 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-04-24T08:21:05.398 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-04-24T08:21:05.398 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.398 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.398 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.398 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.398 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.399 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.399 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-24T08:21:05.399 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.399 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.399 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.400 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-24T08:21:05.400 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-24T08:21:05.400 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.401 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.401 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.401 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.401 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.401 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.402 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.402 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.402 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.402 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.402 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.402 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.402 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.403 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.403 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.403 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.403 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.403 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.403 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.403 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.403 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.404 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.404 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.404 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.404 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.404 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.404 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.404 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.405 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.405 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.405 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.405 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.405 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.405 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.405 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.406 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.406 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.406 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.406 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.406 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.406 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.406 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.406 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.407 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.407 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.407 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.407 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.407 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.407 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.407 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.407 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.408 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.408 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.408 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.408 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.408 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.408 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.408 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.408 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.409 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.409 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.409 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.409 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.409 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.409 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.409 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.410 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.410 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.410 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.410 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.410 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.410 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.410 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.411 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.411 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-04-24T08:21:05.411 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-04-24T08:21:05.411 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.411 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.411 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.411 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-04-24T08:21:05.412 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.412 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.412 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.412 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-04-24T08:21:05.412 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.412 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.412 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.413 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.413 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-24T08:21:05.413 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-24T08:21:05.413 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.413 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.413 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.413 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-24T08:21:05.414 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.414 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.414 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.414 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-04-24T08:21:05.414 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.414 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.414 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.414 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.415 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.415 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.415 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.415 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-04-24T08:21:05.415 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.415 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.416 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.416 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-04-24T08:21:05.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-04-24T08:21:05.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.416 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.416 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.416 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.416 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.417 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.417 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.417 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.417 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.417 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.417 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.418 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-04-24T08:21:05.418 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-04-24T08:21:05.418 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.418 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.418 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.418 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.419 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.419 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.419 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.419 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-24T08:21:05.419 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-24T08:21:05.419 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.419 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.420 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.420 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.420 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.420 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.420 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.420 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.421 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.421 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.421 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.421 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.421 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.421 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.421 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.422 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.422 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.422 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.422 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.422 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.422 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.422 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.423 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.423 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.423 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.423 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.423 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.423 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.423 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.424 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.425 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.426 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.427 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.428 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.429 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-04-24T08:21:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-04-24T08:21:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.430 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.431 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-24T08:21:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-24T08:21:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-04-24T08:21:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.432 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.433 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-04-24T08:21:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-04-24T08:21:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-04-24T08:21:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.434 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.435 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-24T08:21:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-24T08:21:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-24T08:21:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.436 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-24T08:21:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.437 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-24T08:21:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-24T08:21:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-04-24T08:21:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.438 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-24T08:21:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-04-24T08:21:05.439 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-04-24T08:21:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-04-24T08:21:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.440 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-24T08:21:05.441 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-24T08:21:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-04-24T08:21:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.442 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-04-24T08:21:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-04-24T08:21:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.443 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-24T08:21:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-24T08:21:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.444 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-24T08:21:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-24T08:21:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-24T08:21:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.445 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-24T08:21:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.446 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.447 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.448 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.449 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.450 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.451 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.452 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.453 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.454 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.455 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-04-24T08:21:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-04-24T08:21:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.456 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.457 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-04-24T08:21:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-04-24T08:21:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.458 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.459 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-04-24T08:21:05.460 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-04-24T08:21:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.461 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-04-24T08:21:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-04-24T08:21:05.463 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-04-24T08:21:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.464 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-24T08:21:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-24T08:21:05.465 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.466 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.467 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.468 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.469 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.470 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.471 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.473 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.474 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.475 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.476 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-24T08:21:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-24T08:21:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.477 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.478 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-24T08:21:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-24T08:21:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.479 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.480 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-04-24T08:21:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.481 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.482 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.483 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.484 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.485 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.486 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.487 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.488 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.490 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.491 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-04-24T08:21:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.492 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.493 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.494 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.495 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.496 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.497 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.498 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.500 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.501 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-04-24T08:21:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-04-24T08:21:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-04-24T08:21:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-24T08:21:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.503 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-04-24T08:21:05.504 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.505 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.506 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.507 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.508 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.509 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.510 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.511 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.512 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.513 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.514 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-04-24T08:21:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-04-24T08:21:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.515 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.516 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-04-24T08:21:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-04-24T08:21:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-04-24T08:21:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.518 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-04-24T08:21:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-04-24T08:21:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.519 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-04-24T08:21:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-04-24T08:21:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.521 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-04-24T08:21:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-04-24T08:21:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.523 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-04-24T08:21:05.524 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-04-24T08:21:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.525 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-04-24T08:21:05.526 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-04-24T08:21:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.527 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-04-24T08:21:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-04-24T08:21:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.528 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.529 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-04-24T08:21:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-04-24T08:21:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.530 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-24T08:21:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-04-24T08:21:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.531 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-04-24T08:21:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-04-24T08:21:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.532 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-04-24T08:21:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-04-24T08:21:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.534 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-04-24T08:21:05.535 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-04-24T08:21:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.536 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.537 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-24T08:21:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-04-24T08:21:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-04-24T08:21:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.538 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-04-24T08:21:05.539 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-04-24T08:21:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-04-24T08:21:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.540 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-04-24T08:21:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.541 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.542 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.543 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.544 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.545 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.546 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.547 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.548 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.548 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.548 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.548 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.548 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.548 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.548 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.549 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.550 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-04-24T08:21:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.552 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.553 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.554 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.555 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.556 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.557 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.558 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.559 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.560 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.561 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.562 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-04-24T08:21:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.563 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.564 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.565 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.566 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.567 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.568 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.569 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.570 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.571 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.572 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-04-24T08:21:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.573 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-04-24T08:21:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-04-24T08:21:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-24T08:21:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.574 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.575 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-24T08:21:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-24T08:21:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-04-24T08:21:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.576 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.577 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.578 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.579 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.580 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.581 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.582 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.583 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.584 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.585 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.586 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-04-24T08:21:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.587 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-04-24T08:21:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-04-24T08:21:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.588 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-24T08:21:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.589 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-04-24T08:21:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-04-24T08:21:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.590 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-24T08:21:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.591 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-04-24T08:21:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-04-24T08:21:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.592 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-24T08:21:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.593 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-04-24T08:21:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-04-24T08:21:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.594 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-24T08:21:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-04-24T08:21:05.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-04-24T08:21:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.596 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-24T08:21:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-04-24T08:21:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-04-24T08:21:05.597 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.598 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-04-24T08:21:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-04-24T08:21:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.599 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.600 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-04-24T08:21:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-04-24T08:21:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.601 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-24T08:21:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.602 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-24T08:21:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-24T08:21:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.603 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.604 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-24T08:21:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-24T08:21:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-04-24T08:21:05.605 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-04-24T08:21:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.606 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-04-24T08:21:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-04-24T08:21:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-04-24T08:21:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.607 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.608 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-04-24T08:21:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-04-24T08:21:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.609 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-04-24T08:21:05.610 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-04-24T08:21:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.611 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.612 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.613 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.614 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.615 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.616 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.618 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.619 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.620 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.621 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-04-24T08:21:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-04-24T08:21:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.622 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.623 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-04-24T08:21:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-04-24T08:21:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.624 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-04-24T08:21:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-04-24T08:21:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-24T08:21:05.626 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.627 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-04-24T08:21:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.628 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.629 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.630 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.632 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.633 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.634 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.635 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.636 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.637 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-24T08:21:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.638 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.639 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.640 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.641 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.642 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.643 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.644 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.645 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.647 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-04-24T08:21:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-04-24T08:21:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.648 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-04-24T08:21:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-24T08:21:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.649 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-04-24T08:21:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-04-24T08:21:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.650 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-04-24T08:21:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.651 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-04-24T08:21:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-04-24T08:21:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.652 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-24T08:21:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-24T08:21:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.653 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-04-24T08:21:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-04-24T08:21:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.654 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-24T08:21:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.655 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-04-24T08:21:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.656 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-04-24T08:21:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-04-24T08:21:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-24T08:21:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-04-24T08:21:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.657 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-04-24T08:21:05.658 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-04-24T08:21:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.659 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-24T08:21:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-24T08:21:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-24T08:21:05.662 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.663 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.664 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.665 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.666 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.667 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.668 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.669 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.670 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.671 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-04-24T08:21:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-04-24T08:21:05.672 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.673 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-04-24T08:21:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-04-24T08:21:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.674 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-04-24T08:21:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.675 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-04-24T08:21:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-04-24T08:21:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-24T08:21:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.676 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.677 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-24T08:21:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: "gid": 25222, 2024-04-24T08:21:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "z", 2024-04-24T08:21:05.678 INFO:teuthology.orchestra.run.smithi028.stdout: "mgr_features": 4540701547738038271, 2024-04-24T08:21:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-04-24T08:21:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-04-24T08:21:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-24T08:21:05.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-24T08:21:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-24T08:21:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.680 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-04-24T08:21:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.681 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.682 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.683 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.684 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.685 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.686 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.687 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.688 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.689 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-04-24T08:21:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-04-24T08:21:05.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-04-24T08:21:05.691 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-04-24T08:21:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-04-24T08:21:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.692 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-04-24T08:21:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-04-24T08:21:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.693 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-04-24T08:21:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-04-24T08:21:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.694 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-04-24T08:21:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-04-24T08:21:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-04-24T08:21:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.696 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-04-24T08:21:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.697 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-04-24T08:21:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-04-24T08:21:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-04-24T08:21:05.698 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-04-24T08:21:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-04-24T08:21:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-04-24T08:21:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.700 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-04-24T08:21:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-04-24T08:21:05.701 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-04-24T08:21:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.702 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-24T08:21:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-04-24T08:21:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-04-24T08:21:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.703 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-04-24T08:21:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.704 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-04-24T08:21:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.705 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-04-24T08:21:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-04-24T08:21:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.706 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-04-24T08:21:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.707 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-04-24T08:21:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-04-24T08:21:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-04-24T08:21:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.708 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-04-24T08:21:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T08:21:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-04-24T08:21:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-04-24T08:21:05.709 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-24T08:21:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-24T08:21:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-24T08:21:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-24T08:21:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-24T08:21:05.710 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T08:21:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-04-24T08:21:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-04-24T08:21:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-24T08:21:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.711 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-04-24T08:21:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-24T08:21:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-04-24T08:21:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-24T08:21:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.712 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-04-24T08:21:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-04-24T08:21:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-04-24T08:21:05.713 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-24T08:21:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-24T08:21:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.714 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-04-24T08:21:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-04-24T08:21:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-04-24T08:21:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-04-24T08:21:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-04-24T08:21:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.715 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-04-24T08:21:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-24T08:21:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-04-24T08:21:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-04-24T08:21:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.716 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-04-24T08:21:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-04-24T08:21:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T08:21:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.717 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-04-24T08:21:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-04-24T08:21:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-24T08:21:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-24T08:21:05.718 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-24T08:21:05.719 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-24T08:21:05.719 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.719 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-24T08:21:05.719 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T08:21:05.719 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.719 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.720 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.720 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.720 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.720 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.720 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.720 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.720 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.721 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.721 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.721 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.721 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.721 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.721 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.721 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.721 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.722 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.722 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.722 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.723 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.724 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.725 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.726 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.727 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.728 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.729 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-04-24T08:21:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-04-24T08:21:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-24T08:21:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.731 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-24T08:21:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.732 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-24T08:21:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-24T08:21:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.733 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-04-24T08:21:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-04-24T08:21:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-04-24T08:21:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-04-24T08:21:05.734 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-04-24T08:21:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-04-24T08:21:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-04-24T08:21:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-04-24T08:21:05.735 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-04-24T08:21:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-24T08:21:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.736 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-24T08:21:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-24T08:21:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.737 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-24T08:21:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-24T08:21:05.738 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-04-24T08:21:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-04-24T08:21:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.739 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-24T08:21:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-24T08:21:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-24T08:21:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-24T08:21:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.740 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-04-24T08:21:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-04-24T08:21:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-24T08:21:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.741 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-24T08:21:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.742 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-04-24T08:21:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-04-24T08:21:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-04-24T08:21:05.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-24T08:21:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-24T08:21:05.744 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-04-24T08:21:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-04-24T08:21:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.745 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-04-24T08:21:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-24T08:21:05.746 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-04-24T08:21:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-04-24T08:21:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.747 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-04-24T08:21:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-24T08:21:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.748 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-04-24T08:21:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-04-24T08:21:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.749 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-24T08:21:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-24T08:21:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.750 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-04-24T08:21:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-04-24T08:21:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-24T08:21:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.751 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-24T08:21:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-04-24T08:21:05.752 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-04-24T08:21:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-04-24T08:21:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.753 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-24T08:21:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-04-24T08:21:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-04-24T08:21:05.754 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-04-24T08:21:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-04-24T08:21:05.755 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-04-24T08:21:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-24T08:21:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-04-24T08:21:05.756 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-04-24T08:21:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.757 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-24T08:21:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.758 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-04-24T08:21:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-04-24T08:21:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.759 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.760 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-24T08:21:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-04-24T08:21:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-04-24T08:21:05.761 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-24T08:21:05.762 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-04-24T08:21:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-04-24T08:21:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.763 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-24T08:21:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.764 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-24T08:21:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-04-24T08:21:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-04-24T08:21:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.765 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-24T08:21:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.766 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-04-24T08:21:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.767 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-04-24T08:21:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-04-24T08:21:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-24T08:21:05.768 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-04-24T08:21:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.769 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-04-24T08:21:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-04-24T08:21:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.770 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-24T08:21:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-24T08:21:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.771 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-04-24T08:21:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-04-24T08:21:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-24T08:21:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.772 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-04-24T08:21:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-04-24T08:21:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-04-24T08:21:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.773 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-24T08:21:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-04-24T08:21:05.774 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-04-24T08:21:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-04-24T08:21:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.775 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-24T08:21:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-04-24T08:21:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.776 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-04-24T08:21:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-04-24T08:21:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.777 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-24T08:21:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-04-24T08:21:05.778 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-04-24T08:21:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-04-24T08:21:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.779 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-24T08:21:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-04-24T08:21:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.780 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-04-24T08:21:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-04-24T08:21:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.781 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-24T08:21:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-04-24T08:21:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.782 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-04-24T08:21:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-04-24T08:21:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.783 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-24T08:21:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-24T08:21:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.784 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-04-24T08:21:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-04-24T08:21:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.785 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-24T08:21:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-04-24T08:21:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.786 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-04-24T08:21:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-04-24T08:21:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-24T08:21:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.787 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-24T08:21:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-04-24T08:21:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-04-24T08:21:05.788 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-24T08:21:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.789 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-04-24T08:21:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-04-24T08:21:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-04-24T08:21:05.790 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-24T08:21:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.791 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-04-24T08:21:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-04-24T08:21:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-04-24T08:21:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.792 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-24T08:21:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-04-24T08:21:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-04-24T08:21:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-04-24T08:21:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.794 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-04-24T08:21:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.795 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-04-24T08:21:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-04-24T08:21:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.796 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-24T08:21:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-24T08:21:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-04-24T08:21:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-04-24T08:21:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-04-24T08:21:05.798 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-24T08:21:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.799 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-04-24T08:21:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-04-24T08:21:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-04-24T08:21:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.800 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-24T08:21:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-04-24T08:21:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-04-24T08:21:05.801 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-04-24T08:21:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.802 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-04-24T08:21:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-04-24T08:21:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-04-24T08:21:05.803 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-24T08:21:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.804 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-04-24T08:21:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-04-24T08:21:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.805 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-24T08:21:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-04-24T08:21:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.806 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-04-24T08:21:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-04-24T08:21:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-24T08:21:05.807 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-24T08:21:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.808 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-04-24T08:21:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-04-24T08:21:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-24T08:21:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.809 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-04-24T08:21:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-04-24T08:21:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-04-24T08:21:05.810 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.811 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-24T08:21:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-04-24T08:21:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-04-24T08:21:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.812 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-24T08:21:05.813 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.814 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.815 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.816 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-04-24T08:21:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-04-24T08:21:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.817 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-24T08:21:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.818 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.819 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-24T08:21:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.820 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.821 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.822 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.823 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.824 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-24T08:21:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-24T08:21:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.825 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-24T08:21:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.826 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-24T08:21:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-24T08:21:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.827 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-04-24T08:21:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-24T08:21:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-04-24T08:21:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-04-24T08:21:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.829 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-04-24T08:21:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-04-24T08:21:05.830 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-04-24T08:21:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-24T08:21:05.831 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-04-24T08:21:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-04-24T08:21:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.832 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-04-24T08:21:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.833 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-24T08:21:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-24T08:21:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-04-24T08:21:05.834 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-04-24T08:21:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-04-24T08:21:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-04-24T08:21:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.835 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-04-24T08:21:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-04-24T08:21:05.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-04-24T08:21:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.837 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-24T08:21:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.838 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-04-24T08:21:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-04-24T08:21:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.839 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-24T08:21:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-24T08:21:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.840 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-04-24T08:21:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-04-24T08:21:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.841 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-24T08:21:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.842 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-04-24T08:21:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-04-24T08:21:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.843 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-24T08:21:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.844 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-04-24T08:21:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-04-24T08:21:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.845 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-24T08:21:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.846 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-04-24T08:21:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-04-24T08:21:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.847 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-24T08:21:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.848 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-04-24T08:21:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-04-24T08:21:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.849 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-24T08:21:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.850 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-04-24T08:21:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-04-24T08:21:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-04-24T08:21:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.851 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-04-24T08:21:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-04-24T08:21:05.852 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-04-24T08:21:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.853 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-24T08:21:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-04-24T08:21:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-04-24T08:21:05.854 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-24T08:21:05.855 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.856 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-04-24T08:21:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-04-24T08:21:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.857 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-24T08:21:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.858 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-04-24T08:21:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-04-24T08:21:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.859 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-04-24T08:21:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-04-24T08:21:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-04-24T08:21:05.860 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-24T08:21:05.861 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-04-24T08:21:05.862 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-04-24T08:21:05.863 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.865 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-24T08:21:05.866 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-04-24T08:21:05.867 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.868 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.869 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.870 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.871 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.872 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.873 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.874 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.875 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.876 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-04-24T08:21:05.877 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-04-24T08:21:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-04-24T08:21:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.878 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-24T08:21:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-04-24T08:21:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-04-24T08:21:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-04-24T08:21:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.880 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-24T08:21:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.881 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-04-24T08:21:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-24T08:21:05.882 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-24T08:21:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.883 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-04-24T08:21:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-24T08:21:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.884 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.885 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-24T08:21:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-24T08:21:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.886 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.887 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-04-24T08:21:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-04-24T08:21:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.888 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.889 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-24T08:21:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-24T08:21:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.890 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.891 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-04-24T08:21:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-24T08:21:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.892 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.893 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-24T08:21:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-24T08:21:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.894 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.895 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-24T08:21:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-24T08:21:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.896 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-24T08:21:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-24T08:21:05.897 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.898 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-24T08:21:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-24T08:21:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.899 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-24T08:21:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-24T08:21:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.901 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-24T08:21:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-24T08:21:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.902 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.903 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-24T08:21:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-24T08:21:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.904 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-24T08:21:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-24T08:21:05.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.906 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-04-24T08:21:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-24T08:21:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.907 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-24T08:21:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.908 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-24T08:21:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-24T08:21:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.909 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.910 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-04-24T08:21:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-04-24T08:21:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.911 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-04-24T08:21:05.912 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-24T08:21:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-24T08:21:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.913 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-24T08:21:05.914 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-24T08:21:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.915 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-24T08:21:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-24T08:21:05.916 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.917 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-24T08:21:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-24T08:21:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.918 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.919 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-24T08:21:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-24T08:21:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.920 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.921 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-24T08:21:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-24T08:21:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.922 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "[]", 2024-04-24T08:21:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.923 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-04-24T08:21:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-24T08:21:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-04-24T08:21:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.924 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-04-24T08:21:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-24T08:21:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.925 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.926 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-24T08:21:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-24T08:21:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.927 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-24T08:21:05.928 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-24T08:21:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.929 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-24T08:21:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-24T08:21:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.930 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.931 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-24T08:21:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-24T08:21:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.932 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.933 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-24T08:21:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-24T08:21:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.934 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-24T08:21:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-24T08:21:05.935 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.936 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-24T08:21:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-24T08:21:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.937 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.938 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-24T08:21:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-24T08:21:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.939 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.940 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-04-24T08:21:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-24T08:21:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.941 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-24T08:21:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-24T08:21:05.942 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-24T08:21:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.943 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-24T08:21:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-24T08:21:05.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.945 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-24T08:21:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-24T08:21:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.946 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-04-24T08:21:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-24T08:21:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-24T08:21:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-04-24T08:21:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.948 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-04-24T08:21:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-24T08:21:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.949 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.950 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-24T08:21:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-24T08:21:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-24T08:21:05.951 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-04-24T08:21:05.952 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-24T08:21:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.953 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.954 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-04-24T08:21:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-24T08:21:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.955 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-04-24T08:21:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-24T08:21:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.956 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.957 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-24T08:21:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-24T08:21:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-24T08:21:05.958 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-24T08:21:05.959 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-24T08:21:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.960 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-24T08:21:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-24T08:21:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-24T08:21:05.961 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.962 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-04-24T08:21:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-04-24T08:21:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.963 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-04-24T08:21:05.964 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-04-24T08:21:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.965 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-04-24T08:21:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-04-24T08:21:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.966 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-04-24T08:21:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.967 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-04-24T08:21:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-04-24T08:21:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.968 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-04-24T08:21:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.969 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-24T08:21:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-24T08:21:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.970 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.971 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.972 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.973 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.974 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.975 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.976 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:05.977 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.978 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.979 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-04-24T08:21:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-04-24T08:21:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.980 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-04-24T08:21:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.981 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-04-24T08:21:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-04-24T08:21:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.982 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-24T08:21:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-24T08:21:05.983 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-24T08:21:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.984 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-24T08:21:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-24T08:21:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.985 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-04-24T08:21:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.986 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-24T08:21:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-24T08:21:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.987 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-04-24T08:21:05.988 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-04-24T08:21:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-04-24T08:21:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.989 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-24T08:21:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-24T08:21:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-04-24T08:21:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-04-24T08:21:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:05.991 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-24T08:21:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-24T08:21:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:05.992 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-24T08:21:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-24T08:21:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-24T08:21:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.993 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-04-24T08:21:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-04-24T08:21:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.994 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.995 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-24T08:21:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-04-24T08:21:05.996 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-04-24T08:21:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-04-24T08:21:05.997 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:05.998 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:05.999 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.001 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.002 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.003 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.005 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.006 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-04-24T08:21:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-04-24T08:21:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-04-24T08:21:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.007 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-24T08:21:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-04-24T08:21:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-04-24T08:21:06.008 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-04-24T08:21:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-24T08:21:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-04-24T08:21:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-04-24T08:21:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-04-24T08:21:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-04-24T08:21:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-04-24T08:21:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-04-24T08:21:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-24T08:21:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.012 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-24T08:21:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-04-24T08:21:06.013 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-04-24T08:21:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.014 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-24T08:21:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-24T08:21:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-24T08:21:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.015 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-24T08:21:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-04-24T08:21:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.016 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-04-24T08:21:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-04-24T08:21:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-04-24T08:21:06.017 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-24T08:21:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.018 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-04-24T08:21:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.019 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.020 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.021 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.022 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.023 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.024 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.025 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.026 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.027 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-04-24T08:21:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-04-24T08:21:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.028 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-24T08:21:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.029 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-04-24T08:21:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-04-24T08:21:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-04-24T08:21:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.030 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-24T08:21:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-24T08:21:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.031 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-24T08:21:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.032 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-04-24T08:21:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-04-24T08:21:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-04-24T08:21:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.033 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-04-24T08:21:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-04-24T08:21:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-04-24T08:21:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.034 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-24T08:21:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-04-24T08:21:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-04-24T08:21:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.035 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-04-24T08:21:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-24T08:21:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.036 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-04-24T08:21:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-04-24T08:21:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.037 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-04-24T08:21:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.038 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-24T08:21:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-24T08:21:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-04-24T08:21:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-04-24T08:21:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.039 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-24T08:21:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.041 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.042 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.043 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.045 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.046 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.047 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.048 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-04-24T08:21:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-04-24T08:21:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.049 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-04-24T08:21:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-04-24T08:21:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-04-24T08:21:06.050 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-04-24T08:21:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-04-24T08:21:06.051 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-24T08:21:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-24T08:21:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.052 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-04-24T08:21:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T08:21:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.053 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-04-24T08:21:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-04-24T08:21:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-24T08:21:06.054 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-24T08:21:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-04-24T08:21:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-24T08:21:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.055 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-04-24T08:21:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-04-24T08:21:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.056 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-04-24T08:21:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-04-24T08:21:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-04-24T08:21:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.057 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-04-24T08:21:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T08:21:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.058 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-04-24T08:21:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.059 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.060 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.061 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.062 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.063 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.064 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.065 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.066 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.067 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.068 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-04-24T08:21:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.070 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.071 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.072 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.073 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.074 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.075 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.076 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.077 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.078 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-04-24T08:21:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.079 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-04-24T08:21:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-04-24T08:21:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-04-24T08:21:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.080 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-24T08:21:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-04-24T08:21:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-04-24T08:21:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.081 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-04-24T08:21:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-24T08:21:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.082 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.083 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.084 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.085 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.086 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.087 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.088 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.089 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.090 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-04-24T08:21:06.091 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-04-24T08:21:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-04-24T08:21:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.092 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-04-24T08:21:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-04-24T08:21:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-24T08:21:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.093 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.094 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.095 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.097 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.098 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.099 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.101 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-04-24T08:21:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-04-24T08:21:06.102 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-04-24T08:21:06.103 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-24T08:21:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-04-24T08:21:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-04-24T08:21:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.104 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-04-24T08:21:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-04-24T08:21:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.105 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-04-24T08:21:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-04-24T08:21:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-04-24T08:21:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.106 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-04-24T08:21:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-04-24T08:21:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-04-24T08:21:06.107 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-04-24T08:21:06.108 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-04-24T08:21:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-04-24T08:21:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.109 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-04-24T08:21:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-24T08:21:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-24T08:21:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.110 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-04-24T08:21:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.111 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.112 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.113 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.116 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.117 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.118 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.119 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.120 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-04-24T08:21:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.121 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.122 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.123 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.124 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.126 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.127 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.127 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.127 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.127 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.127 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.127 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.127 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.128 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.128 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.128 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.128 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.128 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.128 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.128 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.128 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.129 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.129 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.129 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.129 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.129 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.129 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.129 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.129 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-04-24T08:21:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.130 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.131 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.132 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.133 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.135 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.136 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.137 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.138 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-24T08:21:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.139 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-04-24T08:21:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-04-24T08:21:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.140 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-24T08:21:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.141 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.142 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.143 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.144 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.145 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.146 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.147 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.148 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-04-24T08:21:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-24T08:21:06.149 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-24T08:21:06.150 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-04-24T08:21:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-04-24T08:21:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-04-24T08:21:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.151 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-04-24T08:21:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.152 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.153 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.154 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.155 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.156 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.157 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.158 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.159 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.160 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-04-24T08:21:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.161 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.162 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.163 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.164 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.165 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.166 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.167 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.168 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.169 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-04-24T08:21:06.170 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.171 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.172 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.173 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.174 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.175 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.176 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.177 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.178 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-24T08:21:06.179 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-24T08:21:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-24T08:21:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.180 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-04-24T08:21:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-04-24T08:21:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-24T08:21:06.181 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-04-24T08:21:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-04-24T08:21:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-24T08:21:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.182 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-04-24T08:21:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-04-24T08:21:06.183 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-04-24T08:21:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.184 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-24T08:21:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-24T08:21:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-24T08:21:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.185 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.186 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.187 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.188 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.189 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.190 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.191 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.192 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.193 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.194 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-04-24T08:21:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-04-24T08:21:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.195 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-04-24T08:21:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-04-24T08:21:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.196 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-24T08:21:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-24T08:21:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-24T08:21:06.197 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-04-24T08:21:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.198 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-04-24T08:21:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-04-24T08:21:06.199 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-04-24T08:21:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.200 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-04-24T08:21:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-04-24T08:21:06.201 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-24T08:21:06.202 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-24T08:21:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.203 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.204 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.205 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.206 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.207 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.208 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.209 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.210 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.211 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-04-24T08:21:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-04-24T08:21:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.212 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-24T08:21:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-24T08:21:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.213 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-04-24T08:21:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.214 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-04-24T08:21:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-04-24T08:21:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.215 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-04-24T08:21:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.216 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-24T08:21:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-24T08:21:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-24T08:21:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.217 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-24T08:21:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-24T08:21:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-24T08:21:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.218 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-04-24T08:21:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-24T08:21:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.219 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-04-24T08:21:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-04-24T08:21:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-04-24T08:21:06.220 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.221 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-24T08:21:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-24T08:21:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-04-24T08:21:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.222 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-04-24T08:21:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-04-24T08:21:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.223 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-24T08:21:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-24T08:21:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-24T08:21:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-24T08:21:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-24T08:21:06.224 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.225 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-24T08:21:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.226 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.227 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.228 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.229 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.230 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.231 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.232 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.233 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-04-24T08:21:06.234 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-04-24T08:21:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.235 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-04-24T08:21:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-04-24T08:21:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.236 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.237 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-04-24T08:21:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-04-24T08:21:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.238 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.239 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-04-24T08:21:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-04-24T08:21:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-04-24T08:21:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.240 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.241 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-24T08:21:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-24T08:21:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.242 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.243 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.244 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.245 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.246 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.247 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.248 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.249 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.250 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.251 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-24T08:21:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-24T08:21:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.252 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-24T08:21:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-24T08:21:06.253 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.254 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-04-24T08:21:06.255 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.256 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.257 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.258 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.259 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.260 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.261 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.262 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.263 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.264 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-04-24T08:21:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.265 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.266 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.267 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.268 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.269 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.270 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.271 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.273 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.273 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.273 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.273 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.274 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.275 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-04-24T08:21:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-04-24T08:21:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-04-24T08:21:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.276 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-24T08:21:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.277 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-04-24T08:21:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.278 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.279 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.280 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.281 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.282 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.283 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.284 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.285 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.286 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-04-24T08:21:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-04-24T08:21:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.287 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-04-24T08:21:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-04-24T08:21:06.288 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-04-24T08:21:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.289 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-04-24T08:21:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-04-24T08:21:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.290 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.291 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-04-24T08:21:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-04-24T08:21:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.292 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-04-24T08:21:06.293 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-04-24T08:21:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.294 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-04-24T08:21:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-04-24T08:21:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.295 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.296 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-04-24T08:21:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-04-24T08:21:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.297 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-04-24T08:21:06.298 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-04-24T08:21:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.299 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-04-24T08:21:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-04-24T08:21:06.300 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-24T08:21:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-04-24T08:21:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.301 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-04-24T08:21:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-04-24T08:21:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.302 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.303 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-04-24T08:21:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-04-24T08:21:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.304 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-04-24T08:21:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-04-24T08:21:06.305 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.306 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-24T08:21:06.307 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-04-24T08:21:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-04-24T08:21:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.308 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-04-24T08:21:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.309 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-04-24T08:21:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-04-24T08:21:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.310 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-04-24T08:21:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.311 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.312 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.313 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.314 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.315 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.316 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.317 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.318 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.319 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-04-24T08:21:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.320 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.321 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.322 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.323 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.325 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.326 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.327 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.328 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-04-24T08:21:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.329 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.330 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.331 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.332 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.332 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.332 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.332 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.332 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.332 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.332 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.332 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.332 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.333 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.333 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.333 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.333 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.333 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.333 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.333 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.333 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.334 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.335 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.336 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.337 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.338 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-04-24T08:21:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-04-24T08:21:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-04-24T08:21:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.339 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-24T08:21:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.340 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-24T08:21:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-24T08:21:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-04-24T08:21:06.341 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.342 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.343 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.344 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.345 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.346 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.347 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.348 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.349 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.350 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-04-24T08:21:06.351 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-04-24T08:21:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-04-24T08:21:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.352 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-24T08:21:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.353 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-04-24T08:21:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-04-24T08:21:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.354 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-24T08:21:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-04-24T08:21:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-04-24T08:21:06.355 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-24T08:21:06.356 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-04-24T08:21:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-04-24T08:21:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.357 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.358 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-04-24T08:21:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-04-24T08:21:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.359 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-24T08:21:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-04-24T08:21:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-04-24T08:21:06.360 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.361 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-04-24T08:21:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-04-24T08:21:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.362 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.363 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-04-24T08:21:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-04-24T08:21:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-24T08:21:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.364 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-24T08:21:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-24T08:21:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.365 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.366 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-24T08:21:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-24T08:21:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.367 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-04-24T08:21:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-04-24T08:21:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.368 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-04-24T08:21:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-04-24T08:21:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-04-24T08:21:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.369 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-04-24T08:21:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-04-24T08:21:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.370 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.371 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-04-24T08:21:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-04-24T08:21:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.372 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.373 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.374 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.375 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.376 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.377 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.378 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.379 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.380 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.381 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-04-24T08:21:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-04-24T08:21:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.382 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.383 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-04-24T08:21:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-04-24T08:21:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.384 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-04-24T08:21:06.385 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-04-24T08:21:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-24T08:21:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.386 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-04-24T08:21:06.387 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.388 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.389 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.390 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.391 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.392 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.393 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.394 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.395 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-24T08:21:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.396 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.397 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.398 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.399 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.400 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.401 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.402 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.403 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.404 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-04-24T08:21:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-04-24T08:21:06.405 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-04-24T08:21:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-24T08:21:06.406 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-04-24T08:21:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-04-24T08:21:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.407 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-04-24T08:21:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.408 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-04-24T08:21:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-04-24T08:21:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-24T08:21:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.409 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-24T08:21:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-04-24T08:21:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-04-24T08:21:06.410 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-24T08:21:06.411 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-04-24T08:21:06.412 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-04-24T08:21:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-04-24T08:21:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-24T08:21:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-04-24T08:21:06.413 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.414 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-04-24T08:21:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-04-24T08:21:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.415 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-24T08:21:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-24T08:21:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.416 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-24T08:21:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.417 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.418 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.419 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.420 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.421 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.422 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.423 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.424 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.425 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.426 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-04-24T08:21:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-04-24T08:21:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.427 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-04-24T08:21:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-04-24T08:21:06.428 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-04-24T08:21:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.429 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-04-24T08:21:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-04-24T08:21:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.430 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-24T08:21:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.431 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-24T08:21:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "modules": [ 2024-04-24T08:21:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-24T08:21:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard", 2024-04-24T08:21:06.432 INFO:teuthology.orchestra.run.smithi028.stdout: "iostat", 2024-04-24T08:21:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "nfs", 2024-04-24T08:21:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "restful", 2024-04-24T08:21:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "snap_schedule" 2024-04-24T08:21:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "available_modules": [ 2024-04-24T08:21:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "alerts", 2024-04-24T08:21:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.433 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-24T08:21:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-24T08:21:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-24T08:21:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.434 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How frequently to reexamine health status", 2024-04-24T08:21:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.435 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.436 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.437 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.438 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.439 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.440 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.441 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.442 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.443 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_destination": { 2024-04-24T08:21:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_destination", 2024-04-24T08:21:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.444 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email address to send alerts to", 2024-04-24T08:21:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_from_name": { 2024-04-24T08:21:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_from_name", 2024-04-24T08:21:06.445 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "Ceph", 2024-04-24T08:21:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Email From: name", 2024-04-24T08:21:06.446 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_host": { 2024-04-24T08:21:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_host", 2024-04-24T08:21:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.447 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.448 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.451 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP server", 2024-04-24T08:21:06.451 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.451 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_password": { 2024-04-24T08:21:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_password", 2024-04-24T08:21:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.452 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Password to authenticate with", 2024-04-24T08:21:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.453 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_port": { 2024-04-24T08:21:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_port", 2024-04-24T08:21:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "465", 2024-04-24T08:21:06.454 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP port", 2024-04-24T08:21:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_sender": { 2024-04-24T08:21:06.455 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_sender", 2024-04-24T08:21:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SMTP envelope sender", 2024-04-24T08:21:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.456 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_ssl": { 2024-04-24T08:21:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_ssl", 2024-04-24T08:21:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.457 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use SSL to connect to SMTP server", 2024-04-24T08:21:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.458 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "smtp_user": { 2024-04-24T08:21:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "smtp_user", 2024-04-24T08:21:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.459 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "User to authenticate as", 2024-04-24T08:21:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.460 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "balancer", 2024-04-24T08:21:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "active": { 2024-04-24T08:21:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "active", 2024-04-24T08:21:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.461 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically balance PGs across cluster", 2024-04-24T08:21:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.462 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_time": { 2024-04-24T08:21:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_time", 2024-04-24T08:21:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0000", 2024-04-24T08:21:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.463 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "beginning time of day to automatically balance", 2024-04-24T08:21:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T08:21:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "begin_weekday": { 2024-04-24T08:21:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "begin_weekday", 2024-04-24T08:21:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-24T08:21:06.464 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-24T08:21:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-24T08:21:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-24T08:21:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-04-24T08:21:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T08:21:06.465 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_max_iterations": { 2024-04-24T08:21:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_max_iterations", 2024-04-24T08:21:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-24T08:21:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.466 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "25", 2024-04-24T08:21:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-24T08:21:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "250", 2024-04-24T08:21:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-04-24T08:21:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.467 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_metrics": { 2024-04-24T08:21:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_metrics", 2024-04-24T08:21:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "pgs,objects,bytes", 2024-04-24T08:21:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.468 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-04-24T08:21:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-04-24T08:21:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "crush_compat_step": { 2024-04-24T08:21:06.469 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crush_compat_step", 2024-04-24T08:21:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.5", 2024-04-24T08:21:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0.001", 2024-04-24T08:21:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "0.999", 2024-04-24T08:21:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.470 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "aggressiveness of optimization", 2024-04-24T08:21:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-04-24T08:21:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "end_time": { 2024-04-24T08:21:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_time", 2024-04-24T08:21:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.471 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2359", 2024-04-24T08:21:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ending time of day to automatically balance", 2024-04-24T08:21:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-04-24T08:21:06.472 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: "end_weekday": { 2024-04-24T08:21:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "end_weekday", 2024-04-24T08:21:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-24T08:21:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.473 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-24T08:21:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "0", 2024-04-24T08:21:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "6", 2024-04-24T08:21:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-04-24T08:21:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-04-24T08:21:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.474 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.475 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.476 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.477 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.478 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.479 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.480 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.481 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.482 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "min_score": { 2024-04-24T08:21:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_score", 2024-04-24T08:21:06.483 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-24T08:21:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-04-24T08:21:06.484 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-24T08:21:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-24T08:21:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.485 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "upmap", 2024-04-24T08:21:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "crush-compat", 2024-04-24T08:21:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "none", 2024-04-24T08:21:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "read", 2024-04-24T08:21:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap", 2024-04-24T08:21:06.486 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap-read" 2024-04-24T08:21:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Balancer mode", 2024-04-24T08:21:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_ids": { 2024-04-24T08:21:06.487 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_ids", 2024-04-24T08:21:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.488 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-04-24T08:21:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-24T08:21:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-24T08:21:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.489 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-24T08:21:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-04-24T08:21:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.490 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_deviation": { 2024-04-24T08:21:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_deviation", 2024-04-24T08:21:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-24T08:21:06.491 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-24T08:21:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "deviation below which no optimization is attempted", 2024-04-24T08:21:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-04-24T08:21:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.492 INFO:teuthology.orchestra.run.smithi028.stdout: "upmap_max_optimizations": { 2024-04-24T08:21:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "upmap_max_optimizations", 2024-04-24T08:21:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-24T08:21:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.493 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-04-24T08:21:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.494 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm", 2024-04-24T08:21:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_down_multiplier": { 2024-04-24T08:21:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_down_multiplier", 2024-04-24T08:21:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.495 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-24T08:21:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-04-24T08:21:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.496 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_refresh_rate": { 2024-04-24T08:21:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_refresh_rate", 2024-04-24T08:21:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "20", 2024-04-24T08:21:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.497 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-04-24T08:21:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "agent_starting_port": { 2024-04-24T08:21:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "agent_starting_port", 2024-04-24T08:21:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.498 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4721", 2024-04-24T08:21:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-04-24T08:21:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.499 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_ptrace": { 2024-04-24T08:21:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_ptrace", 2024-04-24T08:21:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.500 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-04-24T08:21:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-04-24T08:21:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_interval": { 2024-04-24T08:21:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_interval", 2024-04-24T08:21:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-24T08:21:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.502 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to autotune daemon memory", 2024-04-24T08:21:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "autotune_memory_target_ratio": { 2024-04-24T08:21:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "autotune_memory_target_ratio", 2024-04-24T08:21:06.503 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0.7", 2024-04-24T08:21:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-04-24T08:21:06.504 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm_log_destination": { 2024-04-24T08:21:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cephadm_log_destination", 2024-04-24T08:21:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.505 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: "file", 2024-04-24T08:21:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: "file,syslog", 2024-04-24T08:21:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: "syslog" 2024-04-24T08:21:06.506 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-04-24T08:21:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "cgroups_split": { 2024-04-24T08:21:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cgroups_split", 2024-04-24T08:21:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.507 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-04-24T08:21:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.508 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "config_checks_enabled": { 2024-04-24T08:21:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_checks_enabled", 2024-04-24T08:21:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.509 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-04-24T08:21:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "config_dashboard": { 2024-04-24T08:21:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_dashboard", 2024-04-24T08:21:06.510 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-04-24T08:21:06.511 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_alertmanager": { 2024-04-24T08:21:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_alertmanager", 2024-04-24T08:21:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.512 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-04-24T08:21:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-24T08:21:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.513 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_base": { 2024-04-24T08:21:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_base", 2024-04-24T08:21:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/ceph", 2024-04-24T08:21:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.514 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Container image name, without the tag", 2024-04-24T08:21:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_elasticsearch": { 2024-04-24T08:21:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_elasticsearch", 2024-04-24T08:21:06.515 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-04-24T08:21:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "elasticsearch container image", 2024-04-24T08:21:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.516 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_grafana": { 2024-04-24T08:21:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_grafana", 2024-04-24T08:21:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-04-24T08:21:06.517 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-24T08:21:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.518 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_haproxy": { 2024-04-24T08:21:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_haproxy", 2024-04-24T08:21:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-04-24T08:21:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.519 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "HAproxy container image", 2024-04-24T08:21:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_agent": { 2024-04-24T08:21:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_agent", 2024-04-24T08:21:06.520 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-04-24T08:21:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger agent container image", 2024-04-24T08:21:06.521 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_collector": { 2024-04-24T08:21:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_collector", 2024-04-24T08:21:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.522 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-04-24T08:21:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger collector container image", 2024-04-24T08:21:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.523 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_jaeger_query": { 2024-04-24T08:21:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_jaeger_query", 2024-04-24T08:21:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-04-24T08:21:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.524 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Jaeger query container image", 2024-04-24T08:21:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_keepalived": { 2024-04-24T08:21:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_keepalived", 2024-04-24T08:21:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.525 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-04-24T08:21:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Keepalived container image", 2024-04-24T08:21:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.526 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_loki": { 2024-04-24T08:21:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_loki", 2024-04-24T08:21:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.527 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-04-24T08:21:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Loki container image", 2024-04-24T08:21:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_node_exporter": { 2024-04-24T08:21:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_node_exporter", 2024-04-24T08:21:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-04-24T08:21:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.529 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-24T08:21:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_nvmeof": { 2024-04-24T08:21:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_nvmeof", 2024-04-24T08:21:06.530 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-04-24T08:21:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Nvme-of container image", 2024-04-24T08:21:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.531 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_prometheus": { 2024-04-24T08:21:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_prometheus", 2024-04-24T08:21:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-04-24T08:21:06.532 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Prometheus container image", 2024-04-24T08:21:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_promtail": { 2024-04-24T08:21:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_promtail", 2024-04-24T08:21:06.533 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-04-24T08:21:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Promtail container image", 2024-04-24T08:21:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.534 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_samba": { 2024-04-24T08:21:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_samba", 2024-04-24T08:21:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.535 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-04-24T08:21:06.536 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.536 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.536 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.536 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Samba/SMB container image", 2024-04-24T08:21:06.536 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.536 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.536 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.536 INFO:teuthology.orchestra.run.smithi028.stdout: "container_image_snmp_gateway": { 2024-04-24T08:21:06.537 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_image_snmp_gateway", 2024-04-24T08:21:06.537 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.537 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.537 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.537 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-04-24T08:21:06.537 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.537 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.537 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "SNMP Gateway container image", 2024-04-24T08:21:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: "container_init": { 2024-04-24T08:21:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "container_init", 2024-04-24T08:21:06.538 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Run podman/docker with `--init`", 2024-04-24T08:21:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.539 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "daemon_cache_timeout": { 2024-04-24T08:21:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "daemon_cache_timeout", 2024-04-24T08:21:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.540 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-24T08:21:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-04-24T08:21:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.541 INFO:teuthology.orchestra.run.smithi028.stdout: "default_cephadm_command_timeout": { 2024-04-24T08:21:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_cephadm_command_timeout", 2024-04-24T08:21:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "900", 2024-04-24T08:21:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.542 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-04-24T08:21:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "default_registry": { 2024-04-24T08:21:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "default_registry", 2024-04-24T08:21:06.543 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "docker.io", 2024-04-24T08:21:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-04-24T08:21:06.544 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: "device_cache_timeout": { 2024-04-24T08:21:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_cache_timeout", 2024-04-24T08:21:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.545 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1800", 2024-04-24T08:21:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache device inventory", 2024-04-24T08:21:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.546 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "device_enhanced_scan": { 2024-04-24T08:21:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_enhanced_scan", 2024-04-24T08:21:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.547 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use libstoragemgmt during device scans", 2024-04-24T08:21:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "facts_cache_timeout": { 2024-04-24T08:21:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "facts_cache_timeout", 2024-04-24T08:21:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.548 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-24T08:21:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "seconds to cache host facts data", 2024-04-24T08:21:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.549 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "grafana_dashboards_path": { 2024-04-24T08:21:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "grafana_dashboards_path", 2024-04-24T08:21:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-04-24T08:21:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.550 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-04-24T08:21:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: "host_check_interval": { 2024-04-24T08:21:06.551 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "host_check_interval", 2024-04-24T08:21:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-24T08:21:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to perform a host check", 2024-04-24T08:21:06.552 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "hw_monitoring": { 2024-04-24T08:21:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hw_monitoring", 2024-04-24T08:21:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.553 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-04-24T08:21:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.554 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "inventory_list_all": { 2024-04-24T08:21:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "inventory_list_all", 2024-04-24T08:21:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.555 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-04-24T08:21:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.556 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.557 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.558 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "log_refresh_metadata": { 2024-04-24T08:21:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_refresh_metadata", 2024-04-24T08:21:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.559 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-04-24T08:21:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.560 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.561 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-04-24T08:21:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.562 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.563 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.564 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.565 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf": { 2024-04-24T08:21:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-04-24T08:21:06.566 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-04-24T08:21:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.567 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-04-24T08:21:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-04-24T08:21:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "*", 2024-04-24T08:21:06.568 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-04-24T08:21:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "max_count_per_host": { 2024-04-24T08:21:06.569 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_count_per_host", 2024-04-24T08:21:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.570 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of daemons per service per host", 2024-04-24T08:21:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "max_osd_draining_count": { 2024-04-24T08:21:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_osd_draining_count", 2024-04-24T08:21:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.571 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-04-24T08:21:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.572 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "migration_current": { 2024-04-24T08:21:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "migration_current", 2024-04-24T08:21:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.573 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "internal - do not modify", 2024-04-24T08:21:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.574 INFO:teuthology.orchestra.run.smithi028.stdout: "mode": { 2024-04-24T08:21:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mode", 2024-04-24T08:21:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "root", 2024-04-24T08:21:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.575 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm-package", 2024-04-24T08:21:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "root" 2024-04-24T08:21:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "mode for remote execution of cephadm", 2024-04-24T08:21:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.576 INFO:teuthology.orchestra.run.smithi028.stdout: "oob_default_addr": { 2024-04-24T08:21:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "oob_default_addr", 2024-04-24T08:21:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "169.254.1.1", 2024-04-24T08:21:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.577 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Default address for RedFish API (oob management).", 2024-04-24T08:21:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: "prometheus_alerts_path": { 2024-04-24T08:21:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus_alerts_path", 2024-04-24T08:21:06.578 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-04-24T08:21:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-04-24T08:21:06.579 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_insecure": { 2024-04-24T08:21:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_insecure", 2024-04-24T08:21:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.580 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-04-24T08:21:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.581 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_password": { 2024-04-24T08:21:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_password", 2024-04-24T08:21:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.582 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-04-24T08:21:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_url": { 2024-04-24T08:21:06.583 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_url", 2024-04-24T08:21:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.584 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-04-24T08:21:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "registry_username": { 2024-04-24T08:21:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "registry_username", 2024-04-24T08:21:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.585 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-04-24T08:21:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.586 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "secure_monitoring_stack": { 2024-04-24T08:21:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "secure_monitoring_stack", 2024-04-24T08:21:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.587 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-04-24T08:21:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.588 INFO:teuthology.orchestra.run.smithi028.stdout: "service_discovery_port": { 2024-04-24T08:21:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "service_discovery_port", 2024-04-24T08:21:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8765", 2024-04-24T08:21:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.589 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "cephadm service discovery port", 2024-04-24T08:21:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "ssh_config_file": { 2024-04-24T08:21:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssh_config_file", 2024-04-24T08:21:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.590 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-04-24T08:21:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.591 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: "use_agent": { 2024-04-24T08:21:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_agent", 2024-04-24T08:21:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.592 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-04-24T08:21:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.593 INFO:teuthology.orchestra.run.smithi028.stdout: "use_repo_digest": { 2024-04-24T08:21:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "use_repo_digest", 2024-04-24T08:21:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.594 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-04-24T08:21:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_failed_host_check": { 2024-04-24T08:21:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_failed_host_check", 2024-04-24T08:21:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.595 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if the host check fails", 2024-04-24T08:21:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.596 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_daemons": { 2024-04-24T08:21:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_daemons", 2024-04-24T08:21:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.597 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-04-24T08:21:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_on_stray_hosts": { 2024-04-24T08:21:06.598 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_on_stray_hosts", 2024-04-24T08:21:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.599 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-04-24T08:21:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.600 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crash", 2024-04-24T08:21:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.601 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.602 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.603 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.604 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.605 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.606 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.607 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.608 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: "retain_interval": { 2024-04-24T08:21:06.609 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retain_interval", 2024-04-24T08:21:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "31536000", 2024-04-24T08:21:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.610 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain crashes before pruning them", 2024-04-24T08:21:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_recent_interval": { 2024-04-24T08:21:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_recent_interval", 2024-04-24T08:21:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.611 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1209600", 2024-04-24T08:21:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "time interval in which to warn about recent crashes", 2024-04-24T08:21:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.612 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dashboard", 2024-04-24T08:21:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.613 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-04-24T08:21:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-04-24T08:21:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.614 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_HOST": { 2024-04-24T08:21:06.615 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_HOST", 2024-04-24T08:21:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.616 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-04-24T08:21:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-04-24T08:21:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.617 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.618 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_ENABLED": { 2024-04-24T08:21:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_ENABLED", 2024-04-24T08:21:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.619 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.620 INFO:teuthology.orchestra.run.smithi028.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-04-24T08:21:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-04-24T08:21:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.621 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: "ENABLE_BROWSABLE_API": { 2024-04-24T08:21:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ENABLE_BROWSABLE_API", 2024-04-24T08:21:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.622 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.623 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-04-24T08:21:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-04-24T08:21:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.624 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.625 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-04-24T08:21:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-04-24T08:21:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.626 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-04-24T08:21:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-04-24T08:21:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.627 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.628 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-04-24T08:21:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-04-24T08:21:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.629 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.630 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_NFS": { 2024-04-24T08:21:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-04-24T08:21:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.631 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RBD": { 2024-04-24T08:21:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-04-24T08:21:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.632 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.633 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "FEATURE_TOGGLE_RGW": { 2024-04-24T08:21:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-04-24T08:21:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.634 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.635 INFO:teuthology.orchestra.run.smithi028.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-04-24T08:21:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-04-24T08:21:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.636 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_PASSWORD": { 2024-04-24T08:21:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_PASSWORD", 2024-04-24T08:21:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.637 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-24T08:21:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.638 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-04-24T08:21:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-04-24T08:21:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.639 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.640 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_URL": { 2024-04-24T08:21:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_URL", 2024-04-24T08:21:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.641 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_API_USERNAME": { 2024-04-24T08:21:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_API_USERNAME", 2024-04-24T08:21:06.642 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-24T08:21:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.643 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-04-24T08:21:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-04-24T08:21:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.644 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.645 INFO:teuthology.orchestra.run.smithi028.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-04-24T08:21:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-04-24T08:21:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.646 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-04-24T08:21:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-04-24T08:21:06.647 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.648 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "ISSUE_TRACKER_API_KEY": { 2024-04-24T08:21:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-04-24T08:21:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.649 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.650 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "MANAGED_BY_CLUSTERS": { 2024-04-24T08:21:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MANAGED_BY_CLUSTERS", 2024-04-24T08:21:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "[]", 2024-04-24T08:21:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.651 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "MULTICLUSTER_CONFIG": { 2024-04-24T08:21:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "MULTICLUSTER_CONFIG", 2024-04-24T08:21:06.652 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "{}", 2024-04-24T08:21:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.653 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_HOST": { 2024-04-24T08:21:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_HOST", 2024-04-24T08:21:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.654 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.655 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-04-24T08:21:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-04-24T08:21:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.656 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.657 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-04-24T08:21:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-04-24T08:21:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.658 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-04-24T08:21:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-04-24T08:21:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.659 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.660 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-04-24T08:21:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-04-24T08:21:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.661 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.662 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-04-24T08:21:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-04-24T08:21:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.663 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-04-24T08:21:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-04-24T08:21:06.664 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.665 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-04-24T08:21:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-04-24T08:21:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.666 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.667 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-04-24T08:21:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-04-24T08:21:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.668 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_ENABLED": { 2024-04-24T08:21:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_ENABLED", 2024-04-24T08:21:06.669 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.670 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-04-24T08:21:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-04-24T08:21:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.671 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-04-24T08:21:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.672 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-04-24T08:21:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-04-24T08:21:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.673 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-04-24T08:21:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-04-24T08:21:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.674 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8", 2024-04-24T08:21:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.675 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "REST_REQUESTS_TIMEOUT": { 2024-04-24T08:21:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-04-24T08:21:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "45", 2024-04-24T08:21:06.676 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ACCESS_KEY": { 2024-04-24T08:21:06.677 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ACCESS_KEY", 2024-04-24T08:21:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.678 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-04-24T08:21:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-04-24T08:21:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.679 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.681 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "admin", 2024-04-24T08:21:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.682 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SECRET_KEY": { 2024-04-24T08:21:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SECRET_KEY", 2024-04-24T08:21:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.683 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.684 INFO:teuthology.orchestra.run.smithi028.stdout: "RGW_API_SSL_VERIFY": { 2024-04-24T08:21:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "RGW_API_SSL_VERIFY", 2024-04-24T08:21:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.685 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "UNSAFE_TLS_v1_2": { 2024-04-24T08:21:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "UNSAFE_TLS_v1_2", 2024-04-24T08:21:06.686 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.687 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-04-24T08:21:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-04-24T08:21:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.688 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-24T08:21:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.689 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-04-24T08:21:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-04-24T08:21:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.690 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-04-24T08:21:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-04-24T08:21:06.691 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-24T08:21:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.692 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "cross_origin_url": { 2024-04-24T08:21:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cross_origin_url", 2024-04-24T08:21:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.693 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.694 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "crt_file": { 2024-04-24T08:21:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "crt_file", 2024-04-24T08:21:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.695 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "debug": { 2024-04-24T08:21:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "debug", 2024-04-24T08:21:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.696 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Enable/disable debug options", 2024-04-24T08:21:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.697 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "jwt_token_ttl": { 2024-04-24T08:21:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "jwt_token_ttl", 2024-04-24T08:21:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "28800", 2024-04-24T08:21:06.698 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.699 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-24T08:21:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-24T08:21:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.700 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.701 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.702 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.703 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.704 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.705 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.706 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.707 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.708 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "motd": { 2024-04-24T08:21:06.709 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "motd", 2024-04-24T08:21:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.710 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "The message of the day", 2024-04-24T08:21:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect_resolve_ip_addr": { 2024-04-24T08:21:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "redirect_resolve_ip_addr", 2024-04-24T08:21:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.711 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.712 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-24T08:21:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-24T08:21:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-24T08:21:06.713 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.714 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-24T08:21:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-24T08:21:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8080", 2024-04-24T08:21:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.715 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-24T08:21:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-24T08:21:06.716 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.717 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl_server_port": { 2024-04-24T08:21:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl_server_port", 2024-04-24T08:21:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.718 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8443", 2024-04-24T08:21:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.719 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-24T08:21:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-24T08:21:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "redirect", 2024-04-24T08:21:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.720 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "redirect" 2024-04-24T08:21:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.721 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-24T08:21:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-24T08:21:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-24T08:21:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-24T08:21:06.722 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-24T08:21:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "url_prefix": { 2024-04-24T08:21:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url_prefix", 2024-04-24T08:21:06.723 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.724 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-24T08:21:06.725 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_monitoring": { 2024-04-24T08:21:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_monitoring", 2024-04-24T08:21:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.726 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "monitor device health metrics", 2024-04-24T08:21:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.727 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.728 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.729 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.730 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.731 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.732 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.733 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.734 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.735 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "mark_out_threshold": { 2024-04-24T08:21:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mark_out_threshold", 2024-04-24T08:21:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.736 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "2419200", 2024-04-24T08:21:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-04-24T08:21:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.737 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: "pool_name": { 2024-04-24T08:21:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pool_name", 2024-04-24T08:21:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "device_health_metrics", 2024-04-24T08:21:06.738 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name of pool in which to store device health metrics", 2024-04-24T08:21:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "retention_period": { 2024-04-24T08:21:06.739 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "retention_period", 2024-04-24T08:21:06.740 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.740 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.740 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.740 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15552000", 2024-04-24T08:21:06.740 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.740 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.741 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.741 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long to retain device health metrics", 2024-04-24T08:21:06.741 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.741 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.741 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.741 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.741 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_frequency": { 2024-04-24T08:21:06.741 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_frequency", 2024-04-24T08:21:06.742 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.742 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.742 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.742 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-24T08:21:06.742 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.742 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.742 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.742 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to scrape device health metrics", 2024-04-24T08:21:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: "self_heal": { 2024-04-24T08:21:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "self_heal", 2024-04-24T08:21:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.743 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-04-24T08:21:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.744 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-24T08:21:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-24T08:21:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.745 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-24T08:21:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how frequently to wake up and check device health", 2024-04-24T08:21:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.746 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "warn_threshold": { 2024-04-24T08:21:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "warn_threshold", 2024-04-24T08:21:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7257600", 2024-04-24T08:21:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.747 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-04-24T08:21:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.748 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "diskprediction_local", 2024-04-24T08:21:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.749 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.750 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.751 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.752 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.753 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.754 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.755 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.756 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "predict_interval": { 2024-04-24T08:21:06.757 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predict_interval", 2024-04-24T08:21:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "86400", 2024-04-24T08:21:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.758 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: "predictor_model": { 2024-04-24T08:21:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "predictor_model", 2024-04-24T08:21:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.759 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "prophetstor", 2024-04-24T08:21:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.760 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-24T08:21:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-24T08:21:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "600", 2024-04-24T08:21:06.761 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.762 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "influx", 2024-04-24T08:21:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": false, 2024-04-24T08:21:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "influxdb python module not found", 2024-04-24T08:21:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "batch_size": { 2024-04-24T08:21:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "batch_size", 2024-04-24T08:21:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.763 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5000", 2024-04-24T08:21:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-04-24T08:21:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.764 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "database": { 2024-04-24T08:21:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "database", 2024-04-24T08:21:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "ceph", 2024-04-24T08:21:06.765 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-04-24T08:21:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.766 INFO:teuthology.orchestra.run.smithi028.stdout: "hostname": { 2024-04-24T08:21:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "hostname", 2024-04-24T08:21:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.767 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server hostname", 2024-04-24T08:21:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-24T08:21:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-24T08:21:06.768 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "30", 2024-04-24T08:21:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "5", 2024-04-24T08:21:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-04-24T08:21:06.769 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.770 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.771 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.772 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.773 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.774 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.775 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.776 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.777 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: "password": { 2024-04-24T08:21:06.778 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "password", 2024-04-24T08:21:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "password of InfluxDB server user", 2024-04-24T08:21:06.779 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "port": { 2024-04-24T08:21:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "port", 2024-04-24T08:21:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.780 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "8086", 2024-04-24T08:21:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "InfluxDB server port", 2024-04-24T08:21:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.781 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "ssl": { 2024-04-24T08:21:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ssl", 2024-04-24T08:21:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "false", 2024-04-24T08:21:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.782 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T08:21:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "threads": { 2024-04-24T08:21:06.783 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threads", 2024-04-24T08:21:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-24T08:21:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-24T08:21:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "32", 2024-04-24T08:21:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-04-24T08:21:06.784 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "username": { 2024-04-24T08:21:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "username", 2024-04-24T08:21:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.785 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "username of InfluxDB server user", 2024-04-24T08:21:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.786 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: "verify_ssl": { 2024-04-24T08:21:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "verify_ssl", 2024-04-24T08:21:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "true", 2024-04-24T08:21:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.787 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-04-24T08:21:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.788 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "insights", 2024-04-24T08:21:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.789 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.790 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.791 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.792 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.793 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.794 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.795 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.796 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "iostat", 2024-04-24T08:21:06.797 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.798 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.799 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.800 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.801 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.802 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.803 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.804 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.805 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.806 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "k8sevents", 2024-04-24T08:21:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_event_retention_days": { 2024-04-24T08:21:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "ceph_event_retention_days", 2024-04-24T08:21:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.807 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "7", 2024-04-24T08:21:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Days to hold ceph event information within local cache", 2024-04-24T08:21:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.808 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: "config_check_secs": { 2024-04-24T08:21:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "config_check_secs", 2024-04-24T08:21:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "10", 2024-04-24T08:21:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.809 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.810 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-04-24T08:21:06.810 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.810 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.810 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.810 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.810 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.810 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.811 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.811 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.811 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.811 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.811 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.811 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.811 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.811 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.811 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.812 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.812 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.812 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.812 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.812 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.812 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.812 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.812 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.813 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.813 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.813 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.813 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.813 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.813 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.813 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.814 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.814 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.814 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.814 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.814 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.814 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.814 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.815 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.815 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.815 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.815 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.815 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.815 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.815 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.815 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.816 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.816 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.816 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.816 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.816 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.816 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.816 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.816 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.817 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.817 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.817 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.817 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.817 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.817 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.817 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.818 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.818 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.818 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.818 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.818 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.818 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.818 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.819 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.819 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.819 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.819 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.819 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.819 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.819 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "localpool", 2024-04-24T08:21:06.819 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.819 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.820 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.820 INFO:teuthology.orchestra.run.smithi028.stdout: "failure_domain": { 2024-04-24T08:21:06.820 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "failure_domain", 2024-04-24T08:21:06.820 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.820 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.820 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.820 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "host", 2024-04-24T08:21:06.820 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.821 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.821 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.821 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "failure domain for any created local pool", 2024-04-24T08:21:06.821 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-04-24T08:21:06.821 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.821 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.821 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.821 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.821 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.822 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.822 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.822 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.822 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.822 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.822 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.822 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.822 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.823 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.823 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.823 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.823 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.823 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.823 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.823 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.823 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.823 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.824 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.824 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.824 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.824 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.824 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.824 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.824 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.825 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.825 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.825 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.825 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.825 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.825 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.825 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.825 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.826 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.826 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.826 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.826 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.826 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.826 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.826 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.826 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.827 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.827 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.827 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.827 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.827 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.827 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.827 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.827 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.827 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.828 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.828 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.828 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.828 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.828 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.828 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.828 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.828 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.829 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.829 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.829 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.829 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.829 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.829 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.829 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.829 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.829 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.830 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.830 INFO:teuthology.orchestra.run.smithi028.stdout: "min_size": { 2024-04-24T08:21:06.830 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "min_size", 2024-04-24T08:21:06.830 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.830 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.830 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.830 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.831 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.831 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.831 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.831 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default min_size for any created local pool", 2024-04-24T08:21:06.831 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-04-24T08:21:06.831 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.831 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.831 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.831 INFO:teuthology.orchestra.run.smithi028.stdout: "num_rep": { 2024-04-24T08:21:06.832 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "num_rep", 2024-04-24T08:21:06.832 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.832 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.832 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.832 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3", 2024-04-24T08:21:06.832 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.832 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.832 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.832 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default replica count for any created local pool", 2024-04-24T08:21:06.833 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.833 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.833 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.833 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.833 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_num": { 2024-04-24T08:21:06.833 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_num", 2024-04-24T08:21:06.833 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.833 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.833 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.834 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "128", 2024-04-24T08:21:06.834 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.834 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.834 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.834 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "default pg_num for any created local pool", 2024-04-24T08:21:06.834 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.834 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.834 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.834 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.835 INFO:teuthology.orchestra.run.smithi028.stdout: "prefix": { 2024-04-24T08:21:06.835 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prefix", 2024-04-24T08:21:06.835 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.835 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.835 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.835 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.835 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.835 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.835 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.836 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "name prefix for any created local pool", 2024-04-24T08:21:06.836 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.836 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.836 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.836 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.836 INFO:teuthology.orchestra.run.smithi028.stdout: "subtree": { 2024-04-24T08:21:06.836 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "subtree", 2024-04-24T08:21:06.836 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.837 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.837 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.837 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "rack", 2024-04-24T08:21:06.837 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.837 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.837 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.837 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "CRUSH level for which to create a local pool", 2024-04-24T08:21:06.837 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-04-24T08:21:06.837 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mds_autoscaler", 2024-04-24T08:21:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.838 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.839 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.839 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.839 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.839 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.839 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.839 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.839 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.839 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.840 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.840 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.840 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.840 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.840 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.840 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.840 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.840 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.840 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.841 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.841 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.841 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.841 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.841 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.841 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.841 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.841 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.841 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.842 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.842 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.842 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.842 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.842 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.842 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.842 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.842 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.843 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.843 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.843 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.843 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.843 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.843 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.843 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.843 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.844 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.844 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.844 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.844 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.844 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.844 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.844 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.844 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.844 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.845 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.845 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.845 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.845 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.845 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.845 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.845 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.845 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.846 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.846 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.846 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.846 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.846 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.846 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.846 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.847 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.847 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.847 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.847 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.847 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.847 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.847 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.847 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.847 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirroring", 2024-04-24T08:21:06.848 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.848 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.848 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.848 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.848 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.848 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.848 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.848 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.849 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.849 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.849 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.849 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.849 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.849 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.849 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.849 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.849 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.850 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.850 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.850 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.850 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.850 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.850 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.850 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.851 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.851 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.851 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.851 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.851 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.851 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.851 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.851 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.851 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.852 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.852 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.852 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.852 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.852 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.852 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.852 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.852 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.853 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.853 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.853 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.853 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.853 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.853 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.853 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.853 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.853 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.854 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.854 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.854 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.854 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.854 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.854 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.854 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.854 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.855 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.855 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.855 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.855 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.855 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.855 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.855 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.855 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.856 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.856 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.856 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.856 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.856 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.856 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.856 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.856 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.857 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.857 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.857 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "nfs", 2024-04-24T08:21:06.857 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.857 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.857 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.857 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.857 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.857 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.858 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.858 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.858 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.858 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.858 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.858 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.858 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.858 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.858 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.859 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.859 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.859 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.859 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.859 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.859 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.859 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.859 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.860 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.860 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.860 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.860 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.860 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.860 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.860 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.860 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.861 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.861 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.861 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.861 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.861 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.861 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.861 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.861 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.862 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.862 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.862 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.862 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.862 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.862 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.862 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.862 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.863 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.863 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.863 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.863 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.863 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.863 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.863 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.863 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.864 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.864 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.864 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.864 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.864 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.864 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.864 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.864 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.865 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.865 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.865 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.865 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.865 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.865 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.865 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.865 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.865 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.866 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.866 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.866 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.866 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.866 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-24T08:21:06.866 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.866 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.866 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.867 INFO:teuthology.orchestra.run.smithi028.stdout: "fail_fs": { 2024-04-24T08:21:06.867 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "fail_fs", 2024-04-24T08:21:06.867 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.867 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.867 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.867 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.867 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.867 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.867 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.868 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-04-24T08:21:06.868 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.868 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.868 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.868 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.868 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.868 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.868 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.869 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.869 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.869 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.869 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.869 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.869 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.869 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.869 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.870 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.870 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.870 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.870 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.870 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.870 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.870 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.871 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.871 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.871 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.871 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.871 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.871 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.871 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.871 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.871 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.872 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.872 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.872 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.872 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.872 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.872 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.872 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.872 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.873 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.873 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.873 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.873 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.873 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.873 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.873 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.873 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.873 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.874 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.874 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.874 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.874 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.874 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.874 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.874 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.874 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.875 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.875 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.875 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.875 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.875 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.875 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.875 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.876 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.876 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.876 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.876 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.876 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.876 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.876 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.876 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.876 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.877 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.877 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.877 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator": { 2024-04-24T08:21:06.877 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "orchestrator", 2024-04-24T08:21:06.877 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.877 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.877 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.877 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.878 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.878 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.878 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.878 INFO:teuthology.orchestra.run.smithi028.stdout: "cephadm", 2024-04-24T08:21:06.878 INFO:teuthology.orchestra.run.smithi028.stdout: "rook", 2024-04-24T08:21:06.878 INFO:teuthology.orchestra.run.smithi028.stdout: "test_orchestrator" 2024-04-24T08:21:06.878 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.878 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Orchestrator backend", 2024-04-24T08:21:06.879 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.879 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.879 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.879 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.879 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.879 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.879 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.879 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_perf_query", 2024-04-24T08:21:06.880 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.880 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.880 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.880 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.880 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.880 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.880 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.880 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.880 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.881 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.881 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.881 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.881 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.881 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.881 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.881 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.881 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.882 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.882 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.882 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.882 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.882 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.882 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.882 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.882 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.883 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.883 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.883 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.883 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.883 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.883 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.883 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.883 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.883 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.884 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.884 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.886 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.886 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.886 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.886 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.886 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.886 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.886 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.886 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.887 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.887 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.887 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.887 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.887 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.887 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.887 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.887 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.887 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.888 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.888 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.888 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.888 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.888 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.888 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.888 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.888 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.889 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.889 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.889 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.889 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.889 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.889 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.889 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.889 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.889 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.890 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.890 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.890 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.890 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.890 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.890 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.890 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "osd_support", 2024-04-24T08:21:06.890 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.891 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.891 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.891 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.891 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.891 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.891 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.891 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.891 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.891 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.892 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.892 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.892 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.892 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.892 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.892 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.892 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.892 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.892 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.893 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.893 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.893 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.893 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.893 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.893 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.893 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.893 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.894 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.894 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.894 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.894 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.894 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.895 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.895 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.895 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.896 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.896 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.896 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.896 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.896 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.896 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.896 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.896 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.897 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.897 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.897 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.897 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.897 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.897 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.897 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.897 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.898 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.898 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.898 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.898 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.898 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.898 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.898 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.898 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.899 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.899 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.899 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.899 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.899 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.899 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.899 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.899 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.899 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.900 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.900 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.900 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.900 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.900 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.900 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.900 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.900 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.901 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "pg_autoscaler", 2024-04-24T08:21:06.901 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.901 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.901 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.901 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.901 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.901 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.901 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.901 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.902 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.902 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.902 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.902 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.902 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.902 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.902 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.902 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.903 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.903 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.903 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.903 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.903 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.903 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.903 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.903 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.904 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.904 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.904 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.904 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.904 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.904 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.904 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.904 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.904 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.905 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.905 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.905 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.905 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.905 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.905 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.905 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.905 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.906 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.906 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.906 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.906 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.906 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.906 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.906 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.906 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.907 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.907 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.907 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.907 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.907 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.907 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.907 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.907 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.908 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.908 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.908 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.908 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.908 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.908 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.908 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.908 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.909 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.909 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.909 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.909 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.909 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.909 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.909 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.909 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.910 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-24T08:21:06.910 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-24T08:21:06.910 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.910 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.910 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.910 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-24T08:21:06.910 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.910 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: "threshold": { 2024-04-24T08:21:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "threshold", 2024-04-24T08:21:06.911 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "3.0", 2024-04-24T08:21:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1.0", 2024-04-24T08:21:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.912 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "scaling threshold", 2024-04-24T08:21:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-04-24T08:21:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.913 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "progress", 2024-04-24T08:21:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_pg_recovery_event": { 2024-04-24T08:21:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_pg_recovery_event", 2024-04-24T08:21:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.914 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow the module to show pg recovery progress", 2024-04-24T08:21:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.915 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-24T08:21:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-24T08:21:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.916 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.917 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.918 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.919 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.920 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.921 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.922 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.923 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.924 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.925 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "max_completed_events": { 2024-04-24T08:21:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_completed_events", 2024-04-24T08:21:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.926 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "50", 2024-04-24T08:21:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "number of past completed events to remember", 2024-04-24T08:21:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.927 INFO:teuthology.orchestra.run.smithi028.stdout: "sleep_interval": { 2024-04-24T08:21:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "sleep_interval", 2024-04-24T08:21:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "5", 2024-04-24T08:21:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.928 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "how long the module is going to sleep", 2024-04-24T08:21:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.929 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "prometheus", 2024-04-24T08:21:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "cache": { 2024-04-24T08:21:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "cache", 2024-04-24T08:21:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.930 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.931 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "exclude_perf_counters": { 2024-04-24T08:21:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "exclude_perf_counters", 2024-04-24T08:21:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.932 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-04-24T08:21:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-04-24T08:21:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.933 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.934 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.935 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.936 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.937 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.938 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.939 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.940 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.941 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools": { 2024-04-24T08:21:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools", 2024-04-24T08:21:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.942 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.943 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_stats_pools_refresh_interval": { 2024-04-24T08:21:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-04-24T08:21:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "300", 2024-04-24T08:21:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.944 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "scrape_interval": { 2024-04-24T08:21:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "scrape_interval", 2024-04-24T08:21:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:06.945 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15.0", 2024-04-24T08:21:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.946 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-24T08:21:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-24T08:21:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "::", 2024-04-24T08:21:06.947 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-04-24T08:21:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.948 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-24T08:21:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-24T08:21:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "9283", 2024-04-24T08:21:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.949 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-04-24T08:21:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "stale_cache_strategy": { 2024-04-24T08:21:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stale_cache_strategy", 2024-04-24T08:21:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.950 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "log", 2024-04-24T08:21:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.951 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_behaviour": { 2024-04-24T08:21:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_behaviour", 2024-04-24T08:21:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.952 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "default", 2024-04-24T08:21:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "default", 2024-04-24T08:21:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "error" 2024-04-24T08:21:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.953 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "standby_error_status_code": { 2024-04-24T08:21:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "standby_error_status_code", 2024-04-24T08:21:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.954 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "500", 2024-04-24T08:21:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "400", 2024-04-24T08:21:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "599", 2024-04-24T08:21:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.955 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-24T08:21:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.956 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.957 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.958 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.959 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.960 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.961 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.962 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.963 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.964 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_snap_create": { 2024-04-24T08:21:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_snap_create", 2024-04-24T08:21:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.965 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10", 2024-04-24T08:21:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.966 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "mirror_snapshot_schedule": { 2024-04-24T08:21:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "mirror_snapshot_schedule", 2024-04-24T08:21:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.967 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "trash_purge_schedule": { 2024-04-24T08:21:06.968 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "trash_purge_schedule", 2024-04-24T08:21:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.969 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.970 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "restful", 2024-04-24T08:21:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "enable_auth": { 2024-04-24T08:21:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enable_auth", 2024-04-24T08:21:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.971 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.972 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "key_file": { 2024-04-24T08:21:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "key_file", 2024-04-24T08:21:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.973 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.974 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.975 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.976 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.977 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.978 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.979 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.980 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.981 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.982 INFO:teuthology.orchestra.run.smithi028.stdout: "server_addr": { 2024-04-24T08:21:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_addr", 2024-04-24T08:21:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.983 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "server_port": { 2024-04-24T08:21:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "server_port", 2024-04-24T08:21:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.984 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.985 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rgw", 2024-04-24T08:21:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.986 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.987 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.988 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.989 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.990 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.991 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.992 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:06.993 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.994 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rook", 2024-04-24T08:21:06.995 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.996 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:06.997 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:06.998 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:06.999 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.000 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.001 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.002 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.003 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "storage_class": { 2024-04-24T08:21:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "storage_class", 2024-04-24T08:21:07.004 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "local", 2024-04-24T08:21:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-04-24T08:21:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.005 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "selftest", 2024-04-24T08:21:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:07.006 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.007 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.008 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.009 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.010 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.011 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.012 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:07.013 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.014 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "roption1": { 2024-04-24T08:21:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption1", 2024-04-24T08:21:07.015 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.016 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "roption2": { 2024-04-24T08:21:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "roption2", 2024-04-24T08:21:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.017 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "xyz", 2024-04-24T08:21:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.018 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption1": { 2024-04-24T08:21:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption1", 2024-04-24T08:21:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.019 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.020 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption2": { 2024-04-24T08:21:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption2", 2024-04-24T08:21:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.021 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption3": { 2024-04-24T08:21:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption3", 2024-04-24T08:21:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "float", 2024-04-24T08:21:07.022 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.023 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption4": { 2024-04-24T08:21:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption4", 2024-04-24T08:21:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.024 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption5": { 2024-04-24T08:21:07.025 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption5", 2024-04-24T08:21:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.026 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption6": { 2024-04-24T08:21:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption6", 2024-04-24T08:21:07.027 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.028 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "rwoption7": { 2024-04-24T08:21:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rwoption7", 2024-04-24T08:21:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.029 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "1", 2024-04-24T08:21:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "42", 2024-04-24T08:21:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.030 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "testkey": { 2024-04-24T08:21:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testkey", 2024-04-24T08:21:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.031 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: "testlkey": { 2024-04-24T08:21:07.032 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testlkey", 2024-04-24T08:21:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.033 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "testnewline": { 2024-04-24T08:21:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "testnewline", 2024-04-24T08:21:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.034 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.035 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-24T08:21:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:07.036 INFO:teuthology.orchestra.run.smithi028.stdout: "allow_m_granularity": { 2024-04-24T08:21:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "allow_m_granularity", 2024-04-24T08:21:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.037 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "allow minute scheduled snapshots", 2024-04-24T08:21:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "dump_on_update": { 2024-04-24T08:21:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "dump_on_update", 2024-04-24T08:21:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.038 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "dump database to debug log on update", 2024-04-24T08:21:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.039 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.040 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.041 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.042 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.043 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:07.044 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.045 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.046 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.047 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "stats", 2024-04-24T08:21:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:07.048 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.049 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.050 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:07.051 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.052 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.053 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.054 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.055 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.056 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.057 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "status", 2024-04-24T08:21:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.058 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.059 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:07.060 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.061 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.062 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.063 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.064 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.065 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.066 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telegraf", 2024-04-24T08:21:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "address": { 2024-04-24T08:21:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "address", 2024-04-24T08:21:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.067 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-04-24T08:21:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.068 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-24T08:21:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-24T08:21:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.069 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "15", 2024-04-24T08:21:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.070 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.071 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.073 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.074 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.075 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.076 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.077 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:07.078 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.079 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.080 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "telemetry", 2024-04-24T08:21:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_basic": { 2024-04-24T08:21:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_basic", 2024-04-24T08:21:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.081 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share basic cluster information (size, version)", 2024-04-24T08:21:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.082 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.083 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_crash": { 2024-04-24T08:21:07.083 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_crash", 2024-04-24T08:21:07.083 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.083 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.083 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.083 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:07.083 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.083 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.083 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.084 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-04-24T08:21:07.084 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.084 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.084 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.084 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.084 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_device": { 2024-04-24T08:21:07.084 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_device", 2024-04-24T08:21:07.084 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.084 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.085 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.085 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:07.085 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.085 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.085 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.085 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-04-24T08:21:07.085 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.085 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.086 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.086 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.086 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_ident": { 2024-04-24T08:21:07.086 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_ident", 2024-04-24T08:21:07.086 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.086 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.086 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.086 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.087 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.087 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.087 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.087 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-04-24T08:21:07.087 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.087 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.087 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.087 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.088 INFO:teuthology.orchestra.run.smithi028.stdout: "channel_perf": { 2024-04-24T08:21:07.088 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "channel_perf", 2024-04-24T08:21:07.088 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.088 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.088 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.088 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.088 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.088 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.088 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.089 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Share various performance metrics of a cluster", 2024-04-24T08:21:07.089 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.089 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.089 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.089 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.089 INFO:teuthology.orchestra.run.smithi028.stdout: "contact": { 2024-04-24T08:21:07.089 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "contact", 2024-04-24T08:21:07.089 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.090 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "description": { 2024-04-24T08:21:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "description", 2024-04-24T08:21:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.091 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.092 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "device_url": { 2024-04-24T08:21:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "device_url", 2024-04-24T08:21:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-04-24T08:21:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.093 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "enabled": { 2024-04-24T08:21:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "enabled", 2024-04-24T08:21:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.094 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.095 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-24T08:21:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-24T08:21:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "24", 2024-04-24T08:21:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "8", 2024-04-24T08:21:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.096 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "last_opt_revision": { 2024-04-24T08:21:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "last_opt_revision", 2024-04-24T08:21:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:07.097 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "1", 2024-04-24T08:21:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.098 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard": { 2024-04-24T08:21:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard", 2024-04-24T08:21:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.099 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "leaderboard_description": { 2024-04-24T08:21:07.100 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "leaderboard_description", 2024-04-24T08:21:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.101 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.102 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.103 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:07.104 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.105 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.106 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.107 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:07.108 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.109 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "organization": { 2024-04-24T08:21:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "organization", 2024-04-24T08:21:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.110 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.111 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "proxy": { 2024-04-24T08:21:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "proxy", 2024-04-24T08:21:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.112 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.113 INFO:teuthology.orchestra.run.smithi028.stdout: "url": { 2024-04-24T08:21:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "url", 2024-04-24T08:21:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-04-24T08:21:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.114 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.115 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "test_orchestrator", 2024-04-24T08:21:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.116 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.117 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.118 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.119 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.120 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.121 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.122 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.123 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.124 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-24T08:21:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.125 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.126 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:07.127 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.128 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:07.129 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.130 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:07.131 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.132 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "max_concurrent_clones": { 2024-04-24T08:21:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "max_concurrent_clones", 2024-04-24T08:21:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.133 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "4", 2024-04-24T08:21:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Number of asynchronous cloner threads", 2024-04-24T08:21:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.134 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "periodic_async_work": { 2024-04-24T08:21:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "periodic_async_work", 2024-04-24T08:21:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.135 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Periodically check for async work", 2024-04-24T08:21:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_delay": { 2024-04-24T08:21:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_delay", 2024-04-24T08:21:07.136 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "0", 2024-04-24T08:21:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-04-24T08:21:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.137 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "snapshot_clone_no_wait": { 2024-04-24T08:21:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snapshot_clone_no_wait", 2024-04-24T08:21:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.138 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "True", 2024-04-24T08:21:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-04-24T08:21:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.139 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix", 2024-04-24T08:21:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "can_run": true, 2024-04-24T08:21:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "error_string": "", 2024-04-24T08:21:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "module_options": { 2024-04-24T08:21:07.140 INFO:teuthology.orchestra.run.smithi028.stdout: "discovery_interval": { 2024-04-24T08:21:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "discovery_interval", 2024-04-24T08:21:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "uint", 2024-04-24T08:21:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "100", 2024-04-24T08:21:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.141 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "identifier": { 2024-04-24T08:21:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "identifier", 2024-04-24T08:21:07.142 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.143 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "interval": { 2024-04-24T08:21:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "interval", 2024-04-24T08:21:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "secs", 2024-04-24T08:21:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "60", 2024-04-24T08:21:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.144 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "log_level": { 2024-04-24T08:21:07.145 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_level", 2024-04-24T08:21:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.146 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.147 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster": { 2024-04-24T08:21:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster", 2024-04-24T08:21:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.148 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_cluster_level": { 2024-04-24T08:21:07.149 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_cluster_level", 2024-04-24T08:21:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "info", 2024-04-24T08:21:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [ 2024-04-24T08:21:07.150 INFO:teuthology.orchestra.run.smithi028.stdout: "", 2024-04-24T08:21:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "critical", 2024-04-24T08:21:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "debug", 2024-04-24T08:21:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "error", 2024-04-24T08:21:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "info", 2024-04-24T08:21:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "warning" 2024-04-24T08:21:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.151 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "log_to_file": { 2024-04-24T08:21:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "log_to_file", 2024-04-24T08:21:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "bool", 2024-04-24T08:21:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 1, 2024-04-24T08:21:07.152 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "False", 2024-04-24T08:21:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.153 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_host": { 2024-04-24T08:21:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_host", 2024-04-24T08:21:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "", 2024-04-24T08:21:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.154 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_port": { 2024-04-24T08:21:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_port", 2024-04-24T08:21:07.155 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "int", 2024-04-24T08:21:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "10051", 2024-04-24T08:21:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.156 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "zabbix_sender": { 2024-04-24T08:21:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "zabbix_sender", 2024-04-24T08:21:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "str", 2024-04-24T08:21:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "level": "advanced", 2024-04-24T08:21:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "flags": 0, 2024-04-24T08:21:07.157 INFO:teuthology.orchestra.run.smithi028.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-04-24T08:21:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "min": "", 2024-04-24T08:21:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "max": "", 2024-04-24T08:21:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "enum_allowed": [], 2024-04-24T08:21:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "desc": "", 2024-04-24T08:21:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "long_desc": "", 2024-04-24T08:21:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "tags": [], 2024-04-24T08:21:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: "see_also": [] 2024-04-24T08:21:07.158 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: "services": { 2024-04-24T08:21:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: "dashboard": "https://172.21.15.28:8443/" 2024-04-24T08:21:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: "always_on_modules": { 2024-04-24T08:21:07.159 INFO:teuthology.orchestra.run.smithi028.stdout: "octopus": [ 2024-04-24T08:21:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-24T08:21:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-24T08:21:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-24T08:21:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-24T08:21:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-24T08:21:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-24T08:21:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-24T08:21:07.160 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-24T08:21:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-24T08:21:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-24T08:21:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "pacific": [ 2024-04-24T08:21:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-24T08:21:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-24T08:21:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-24T08:21:07.161 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-24T08:21:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-24T08:21:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-24T08:21:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-24T08:21:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-24T08:21:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-24T08:21:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-24T08:21:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.162 INFO:teuthology.orchestra.run.smithi028.stdout: "quincy": [ 2024-04-24T08:21:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-24T08:21:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-24T08:21:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-24T08:21:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-24T08:21:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-24T08:21:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-24T08:21:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-24T08:21:07.163 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-24T08:21:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-24T08:21:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-24T08:21:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "reef": [ 2024-04-24T08:21:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-24T08:21:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-24T08:21:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-24T08:21:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-24T08:21:07.164 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-24T08:21:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-24T08:21:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-24T08:21:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-24T08:21:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-24T08:21:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-24T08:21:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: ], 2024-04-24T08:21:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "squid": [ 2024-04-24T08:21:07.165 INFO:teuthology.orchestra.run.smithi028.stdout: "balancer", 2024-04-24T08:21:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "crash", 2024-04-24T08:21:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "devicehealth", 2024-04-24T08:21:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "orchestrator", 2024-04-24T08:21:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "pg_autoscaler", 2024-04-24T08:21:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "progress", 2024-04-24T08:21:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "rbd_support", 2024-04-24T08:21:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "status", 2024-04-24T08:21:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "telemetry", 2024-04-24T08:21:07.166 INFO:teuthology.orchestra.run.smithi028.stdout: "volumes" 2024-04-24T08:21:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-24T08:21:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: "last_failure_osd_epoch": 91, 2024-04-24T08:21:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: "active_clients": [ 2024-04-24T08:21:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "devicehealth", 2024-04-24T08:21:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-24T08:21:07.167 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-24T08:21:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-24T08:21:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 108567609 2024-04-24T08:21:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-24T08:21:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "libcephsqlite", 2024-04-24T08:21:07.168 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-24T08:21:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-24T08:21:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-24T08:21:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 903509040 2024-04-24T08:21:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-24T08:21:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.169 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "rbd_support", 2024-04-24T08:21:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-24T08:21:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-24T08:21:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-24T08:21:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 104973679 2024-04-24T08:21:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-24T08:21:07.170 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-24T08:21:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-24T08:21:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-24T08:21:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-24T08:21:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 1059217802 2024-04-24T08:21:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.171 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-24T08:21:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "snap_schedule", 2024-04-24T08:21:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-24T08:21:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-24T08:21:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-24T08:21:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3834997220 2024-04-24T08:21:07.172 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-24T08:21:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: "name": "volumes", 2024-04-24T08:21:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: "addrvec": [ 2024-04-24T08:21:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: { 2024-04-24T08:21:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: "type": "v2", 2024-04-24T08:21:07.173 INFO:teuthology.orchestra.run.smithi028.stdout: "addr": "172.21.15.28:0", 2024-04-24T08:21:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: "nonce": 3771825660 2024-04-24T08:21:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-24T08:21:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:07.174 INFO:teuthology.orchestra.run.smithi028.stdout: ] 2024-04-24T08:21:07.174 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:21:07.176 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:05 smithi117 ceph-mon[32438]: pgmap v333: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-24T08:21:07.176 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:05 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/969528721' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-24T08:21:07.176 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:04 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3258409987' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:21:07.176 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:04 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/775753515' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:21:07.177 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:05 smithi194 ceph-mon[32836]: pgmap v333: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-24T08:21:07.177 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:05 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/969528721' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-24T08:21:07.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.887+0000 7ffa77200640 1 -- 172.21.15.28:0/969528721 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ffa5c0a3da0 msgr2=0x7ffa5c0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:07.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.887+0000 7ffa77200640 1 --2- 172.21.15.28:0/969528721 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ffa5c0a3da0 0x7ffa5c0a6260 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7ffa68009c90 tx=0x7ffa68005f90 comp rx=0 tx=0).stop 2024-04-24T08:21:07.177 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.887+0000 7ffa77200640 1 -- 172.21.15.28:0/969528721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa78109340 msgr2=0x7ffa781a0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:21:07.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.887+0000 7ffa77200640 1 --2- 172.21.15.28:0/969528721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa78109340 0x7ffa781a0b50 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7ffa6400e9c0 tx=0x7ffa6400ee80 comp rx=0 tx=0).stop 2024-04-24T08:21:07.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.887+0000 7ffa77200640 1 -- 172.21.15.28:0/969528721 shutdown_connections 2024-04-24T08:21:07.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.887+0000 7ffa77200640 1 --2- 172.21.15.28:0/969528721 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ffa5c0a3da0 0x7ffa5c0a6260 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:07.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.887+0000 7ffa77200640 1 --2- 172.21.15.28:0/969528721 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ffa78109340 0x7ffa781a0b50 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:07.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.887+0000 7ffa77200640 1 --2- 172.21.15.28:0/969528721 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ffa7819a6d0 0x7ffa7819ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:07.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.887+0000 7ffa77200640 1 --2- 172.21.15.28:0/969528721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa78104270 0x7ffa781a0610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:21:07.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.887+0000 7ffa77200640 1 -- 172.21.15.28:0/969528721 >> 172.21.15.28:0/969528721 conn(0x7ffa780ffad0 msgr2=0x7ffa78101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:21:07.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.887+0000 7ffa77200640 1 -- 172.21.15.28:0/969528721 shutdown_connections 2024-04-24T08:21:07.178 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:21:04.887+0000 7ffa77200640 1 -- 172.21.15.28:0/969528721 wait complete. 2024-04-24T08:21:07.185 DEBUG:tasks.check_counter:Getting stats from a 2024-04-24T08:21:07.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph daemon mds.a perf dump 2024-04-24T08:21:07.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:05 smithi028 ceph-mon[27786]: pgmap v333: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-24T08:21:07.186 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:05 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/969528721' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-04-24T08:21:07.702 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:21:07.852 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:07 smithi028 ceph-mon[27786]: pgmap v334: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-24T08:21:07.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:07 smithi117 ceph-mon[32438]: pgmap v334: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-24T08:21:07.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:07 smithi194 ceph-mon[32836]: pgmap v334: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 45 KiB/s wr, 22 op/s 2024-04-24T08:21:09.603 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:09 smithi028 ceph-mon[27786]: pgmap v335: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-24T08:21:09.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:09 smithi117 ceph-mon[32438]: pgmap v335: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-24T08:21:09.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:09 smithi194 ceph-mon[32836]: pgmap v335: 97 pgs: 97 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 45 KiB/s wr, 23 op/s 2024-04-24T08:21:10.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:10 smithi028 ceph-mon[27786]: osdmap e100: 12 total, 12 up, 12 in 2024-04-24T08:21:10.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:10 smithi117 ceph-mon[32438]: osdmap e100: 12 total, 12 up, 12 in 2024-04-24T08:21:10.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:10 smithi194 ceph-mon[32836]: osdmap e100: 12 total, 12 up, 12 in 2024-04-24T08:21:11.870 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:11 smithi028 ceph-mon[27786]: pgmap v337: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-24T08:21:11.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:11 smithi117 ceph-mon[32438]: pgmap v337: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-24T08:21:11.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:11 smithi194 ceph-mon[32836]: pgmap v337: 97 pgs: 2 active+clean+snaptrim, 4 active+clean+snaptrim_wait, 91 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 26 op/s 2024-04-24T08:21:13.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:13 smithi117 ceph-mon[32438]: pgmap v338: 97 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 67 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:21:13.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:13 smithi194 ceph-mon[32836]: pgmap v338: 97 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 67 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:21:14.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:13 smithi028 ceph-mon[27786]: pgmap v338: 97 pgs: 10 active+clean+snaptrim, 20 active+clean+snaptrim_wait, 67 active+clean; 17 GiB data, 48 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:21:15.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:15 smithi194 ceph-mon[32836]: pgmap v339: 97 pgs: 20 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 39 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:21:15.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:15 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:21:15.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:15 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:15.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:15 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:16.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:15 smithi028 ceph-mon[27786]: pgmap v339: 97 pgs: 20 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 39 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:21:16.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:15 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:21:16.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:15 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:16.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:15 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:16.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:15 smithi117 ceph-mon[32438]: pgmap v339: 97 pgs: 20 active+clean+snaptrim, 38 active+clean+snaptrim_wait, 39 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 27 KiB/s wr, 9 op/s 2024-04-24T08:21:16.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:15 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-04-24T08:21:16.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:15 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:16.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:15 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:17.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:17 smithi194 ceph-mon[32836]: pgmap v340: 97 pgs: 22 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 35 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:21:18.038 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:17 smithi028 ceph-mon[27786]: pgmap v340: 97 pgs: 22 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 35 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:21:18.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:17 smithi117 ceph-mon[32438]: pgmap v340: 97 pgs: 22 active+clean+snaptrim, 40 active+clean+snaptrim_wait, 35 active+clean; 16 GiB data, 47 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:21:19.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:18 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:19.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:18 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:19.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:18 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:18 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:18 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:18 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:18 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:18 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.130 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:18 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.130 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:18 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:18 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:19.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:18 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:19.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:18 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:18 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:18 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:18 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:18 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:18 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:18 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:18 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:18 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:19.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:18 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:19.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:18 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:18 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:18 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:18 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:18 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:18 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:18 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:19.292 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:18 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:20.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:19 smithi117 ceph-mon[32438]: pgmap v341: 97 pgs: 19 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 44 active+clean; 16 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:21:20.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:19 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-24T08:21:20.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:19 smithi117 ceph-mon[32438]: Unable to set osd_memory_target on smithi194 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-24T08:21:20.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:19 smithi028 ceph-mon[27786]: pgmap v341: 97 pgs: 19 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 44 active+clean; 16 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:21:20.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:19 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-24T08:21:20.140 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:20 smithi028 ceph-mon[27786]: Unable to set osd_memory_target on smithi194 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-24T08:21:20.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:19 smithi194 ceph-mon[32836]: pgmap v341: 97 pgs: 19 active+clean+snaptrim, 34 active+clean+snaptrim_wait, 44 active+clean; 16 GiB data, 46 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 204 B/s wr, 6 op/s 2024-04-24T08:21:20.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:19 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi194 to 176.2M 2024-04-24T08:21:20.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:19 smithi194 ceph-mon[32836]: Unable to set osd_memory_target on smithi194 to 184792473: error parsing value: Value '184792473' is below minimum 939524096 2024-04-24T08:21:21.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:21 smithi117 ceph-mon[32438]: pgmap v342: 97 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 66 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 188 B/s wr, 5 op/s 2024-04-24T08:21:21.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:21 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:21.879 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:21 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:21.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:21 smithi194 ceph-mon[32836]: pgmap v342: 97 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 66 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 188 B/s wr, 5 op/s 2024-04-24T08:21:21.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:21 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:21.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:21 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:22.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:21 smithi028 ceph-mon[27786]: pgmap v342: 97 pgs: 9 active+clean+snaptrim, 22 active+clean+snaptrim_wait, 66 active+clean; 15 GiB data, 45 GiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 188 B/s wr, 5 op/s 2024-04-24T08:21:22.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:21 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:22.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:21 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:23.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:23 smithi117 ceph-mon[32438]: pgmap v343: 97 pgs: 6 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 75 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:21:23.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:23 smithi194 ceph-mon[32836]: pgmap v343: 97 pgs: 6 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 75 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:21:24.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:23 smithi028 ceph-mon[27786]: pgmap v343: 97 pgs: 6 active+clean+snaptrim, 16 active+clean+snaptrim_wait, 75 active+clean; 15 GiB data, 44 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 170 B/s wr, 5 op/s 2024-04-24T08:21:25.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:25 smithi117 ceph-mon[32438]: pgmap v344: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-24T08:21:25.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:25 smithi194 ceph-mon[32836]: pgmap v344: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-24T08:21:26.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:25 smithi028 ceph-mon[27786]: pgmap v344: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-24T08:21:27.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:26 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:27.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:26 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:27.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:26 smithi028 ceph-mon[27786]: pgmap v345: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:21:27.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:26 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:26 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:26 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:26 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:26 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:26 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:26 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:26 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:26 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi117 to 176.2M 2024-04-24T08:21:27.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:26 smithi028 ceph-mon[27786]: Unable to set osd_memory_target on smithi117 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-24T08:21:27.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:26 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:27.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:26 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:27.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:26 smithi117 ceph-mon[32438]: pgmap v345: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:21:27.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:26 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:26 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:26 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:26 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:26 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:26 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:26 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.379 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:26 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.380 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:26 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi117 to 176.2M 2024-04-24T08:21:27.380 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:26 smithi117 ceph-mon[32438]: Unable to set osd_memory_target on smithi117 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-24T08:21:27.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:26 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:27.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:26 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:27.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:26 smithi194 ceph-mon[32836]: pgmap v345: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 85 B/s wr, 5 op/s 2024-04-24T08:21:27.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:26 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:26 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:26 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:26 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:26 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:26 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:26 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:26 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:27.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:26 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi117 to 176.2M 2024-04-24T08:21:27.442 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:26 smithi194 ceph-mon[32836]: Unable to set osd_memory_target on smithi117 to 184785305: error parsing value: Value '184785305' is below minimum 939524096 2024-04-24T08:21:29.004 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:21:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:21:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4, 2024-04-24T08:21:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 4, 2024-04-24T08:21:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 3024, 2024-04-24T08:21:29.051 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 448, 2024-04-24T08:21:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-04-24T08:21:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-04-24T08:21:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.005016887, 2024-04-24T08:21:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000087353, 2024-04-24T08:21:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001343692, 2024-04-24T08:21:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:21:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:29.052 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 3, 2024-04-24T08:21:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.001436298, 2024-04-24T08:21:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000478766 2024-04-24T08:21:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-24T08:21:29.053 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 448 2024-04-24T08:21:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:21:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2, 2024-04-24T08:21:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 169, 2024-04-24T08:21:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 320, 2024-04-24T08:21:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 186048, 2024-04-24T08:21:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-04-24T08:21:29.054 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-24T08:21:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.032093338, 2024-04-24T08:21:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.021945551, 2024-04-24T08:21:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.005268526, 2024-04-24T08:21:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:21:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 166, 2024-04-24T08:21:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.038352649, 2024-04-24T08:21:29.055 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000231040 2024-04-24T08:21:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-24T08:21:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 186048 2024-04-24T08:21:29.056 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:21:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 233, 2024-04-24T08:21:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 221, 2024-04-24T08:21:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1731424, 2024-04-24T08:21:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 136224, 2024-04-24T08:21:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-04-24T08:21:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-04-24T08:21:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.101418482, 2024-04-24T08:21:29.057 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.034785920, 2024-04-24T08:21:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.075499494, 2024-04-24T08:21:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.012255358, 2024-04-24T08:21:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 220, 2024-04-24T08:21:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.042866814, 2024-04-24T08:21:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000194849 2024-04-24T08:21:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:29.058 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1731424, 2024-04-24T08:21:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 136224 2024-04-24T08:21:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:29.059 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-24T08:21:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-04-24T08:21:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-24T08:21:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-24T08:21:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:21:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-24T08:21:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:21:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:21:29.060 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:21:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:21:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:21:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:21:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:21:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:21:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:21:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:21:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:21:29.061 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:21:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:21:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-24T08:21:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:21:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-24T08:21:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:21:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:21:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:21:29.062 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:21:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:21:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:21:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:21:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-24T08:21:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:21:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:21:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:21:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-24T08:21:29.063 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-24T08:21:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-24T08:21:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:21:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:21:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:21:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:21:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 148994, 2024-04-24T08:21:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 31, 2024-04-24T08:21:29.064 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:21:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-24T08:21:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-24T08:21:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-24T08:21:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:21:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-24T08:21:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:21:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-24T08:21:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-04-24T08:21:29.065 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-04-24T08:21:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:21:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:21:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-24T08:21:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-24T08:21:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-04-24T08:21:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-04-24T08:21:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-24T08:21:29.066 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-24T08:21:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-24T08:21:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-24T08:21:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:21:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-24T08:21:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-24T08:21:29.067 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-24T08:21:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 235, 2024-04-24T08:21:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1698562, 2024-04-24T08:21:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 235, 2024-04-24T08:21:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-24T08:21:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-24T08:21:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 235, 2024-04-24T08:21:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1698562, 2024-04-24T08:21:29.068 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-24T08:21:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:29.069 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:29.069 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:21:29.755 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:29 smithi028 ceph-mon[27786]: pgmap v346: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-24T08:21:29.756 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:29 smithi028 ceph-mon[27786]: osdmap e101: 12 total, 12 up, 12 in 2024-04-24T08:21:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:29 smithi117 ceph-mon[32438]: pgmap v346: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-24T08:21:29.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:29 smithi117 ceph-mon[32438]: osdmap e101: 12 total, 12 up, 12 in 2024-04-24T08:21:29.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:29 smithi194 ceph-mon[32836]: pgmap v346: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 85 B/s wr, 6 op/s 2024-04-24T08:21:29.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:29 smithi194 ceph-mon[32836]: osdmap e101: 12 total, 12 up, 12 in 2024-04-24T08:21:30.252 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-04-24T08:21:30.252 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-04-24T08:21:30.252 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-04-24T08:21:30.252 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-04-24T08:21:30.252 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-04-24T08:21:30.252 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-04-24T08:21:30.252 DEBUG:tasks.check_counter:Getting stats from d 2024-04-24T08:21:30.253 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph daemon mds.d perf dump 2024-04-24T08:21:30.734 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:21:31.822 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:31 smithi028 ceph-mon[27786]: pgmap v348: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:21:31.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:31 smithi117 ceph-mon[32438]: pgmap v348: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:21:31.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:31 smithi194 ceph-mon[32836]: pgmap v348: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:21:33.808 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:21:33.808 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:21:33.808 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 0, 2024-04-24T08:21:33.808 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2, 2024-04-24T08:21:33.808 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 0, 2024-04-24T08:21:33.808 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 0, 2024-04-24T08:21:33.809 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-04-24T08:21:33.809 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-04-24T08:21:33.809 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.004084403, 2024-04-24T08:21:33.809 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000000000, 2024-04-24T08:21:33.809 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.000835139, 2024-04-24T08:21:33.809 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:21:33.809 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:33.809 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:33.810 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:33.810 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:33.810 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:33.810 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:33.810 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:33.810 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:33.810 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:33.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:33.811 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T08:21:33.811 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T08:21:33.811 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:33.811 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:21:33.811 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 6, 2024-04-24T08:21:33.811 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 177, 2024-04-24T08:21:33.811 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 3344, 2024-04-24T08:21:33.811 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 187552, 2024-04-24T08:21:33.812 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-04-24T08:21:33.812 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-24T08:21:33.812 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.037697895, 2024-04-24T08:21:33.812 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.027191551, 2024-04-24T08:21:33.812 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.005565853, 2024-04-24T08:21:33.812 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:21:33.812 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:33.812 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 175, 2024-04-24T08:21:33.812 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.031011115, 2024-04-24T08:21:33.813 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000177206 2024-04-24T08:21:33.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:33.813 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:33.813 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:33.813 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:33.813 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:33.813 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:33.813 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-24T08:21:33.814 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 187552 2024-04-24T08:21:33.814 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:33.814 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:21:33.814 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 240, 2024-04-24T08:21:33.814 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 228, 2024-04-24T08:21:33.814 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1733328, 2024-04-24T08:21:33.814 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 140704, 2024-04-24T08:21:33.814 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-04-24T08:21:33.815 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-04-24T08:21:33.815 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.098720272, 2024-04-24T08:21:33.815 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.028241135, 2024-04-24T08:21:33.815 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.086320222, 2024-04-24T08:21:33.815 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.010095626, 2024-04-24T08:21:33.815 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:33.815 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 227, 2024-04-24T08:21:33.815 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.032788830, 2024-04-24T08:21:33.816 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000144444 2024-04-24T08:21:33.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:33.816 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:33.816 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:33.816 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:33.816 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:33.816 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:33.816 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1733328, 2024-04-24T08:21:33.816 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 140704 2024-04-24T08:21:33.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:33.817 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-24T08:21:33.817 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-04-24T08:21:33.817 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-24T08:21:33.817 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:33.817 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-24T08:21:33.817 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:21:33.817 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-24T08:21:33.818 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:21:33.818 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:21:33.818 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:21:33.818 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:21:33.818 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:21:33.818 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:21:33.818 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:21:33.818 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:21:33.819 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:21:33.819 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:21:33.819 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:21:33.819 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:21:33.819 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:21:33.819 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-24T08:21:33.819 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:21:33.819 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-24T08:21:33.819 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:21:33.820 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:21:33.820 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:21:33.820 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:21:33.820 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:21:33.820 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:21:33.820 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:21:33.820 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-24T08:21:33.820 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:21:33.821 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:21:33.821 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:21:33.821 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-24T08:21:33.821 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-24T08:21:33.821 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-24T08:21:33.821 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:21:33.821 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:21:33.821 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:21:33.821 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:21:33.822 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 149745, 2024-04-24T08:21:33.822 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 31, 2024-04-24T08:21:33.822 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:21:33.822 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-24T08:21:33.822 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-24T08:21:33.822 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-24T08:21:33.822 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:21:33.822 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-24T08:21:33.823 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:21:33.823 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-24T08:21:33.823 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-04-24T08:21:33.823 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-04-24T08:21:33.823 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:21:33.823 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:21:33.823 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-24T08:21:33.823 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-24T08:21:33.823 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-04-24T08:21:33.824 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-04-24T08:21:33.824 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-24T08:21:33.824 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-24T08:21:33.824 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-24T08:21:33.824 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-24T08:21:33.824 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:33.824 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:21:33.824 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-24T08:21:33.824 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-24T08:21:33.825 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-24T08:21:33.825 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 242, 2024-04-24T08:21:33.825 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1699472, 2024-04-24T08:21:33.825 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:33.825 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 242, 2024-04-24T08:21:33.825 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-24T08:21:33.825 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-24T08:21:33.825 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 242, 2024-04-24T08:21:33.825 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1699472, 2024-04-24T08:21:33.826 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-24T08:21:33.826 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:33.826 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:33.826 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:33.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:33.826 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:33.826 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:21:34.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:33 smithi028 ceph-mon[27786]: pgmap v349: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:21:34.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:33 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:34.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:33 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:34.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:33 smithi117 ceph-mon[32438]: pgmap v349: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:21:34.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:33 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:34.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:33 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:34.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:33 smithi194 ceph-mon[32836]: pgmap v349: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:21:34.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:33 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:34.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:33 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:34.679 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-04-24T08:21:34.679 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-04-24T08:21:34.679 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-04-24T08:21:34.680 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-04-24T08:21:34.680 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-04-24T08:21:34.680 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-04-24T08:21:34.680 DEBUG:tasks.check_counter:Getting stats from g 2024-04-24T08:21:34.680 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph daemon mds.g perf dump 2024-04-24T08:21:35.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:34 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:34 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:34 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:34 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:34 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:34 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:34 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:34 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:34 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:21:35.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:34 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:21:35.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:34 smithi028 ceph-mon[27786]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:35.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:34 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:34 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:34 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:34 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:34 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:34 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:34 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:34 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:34 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:21:35.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:34 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:21:35.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:34 smithi117 ceph-mon[32438]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:35.160 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:21:35.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:34 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:34 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:34 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:34 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:34 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:34 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:34 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:34 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-04-24T08:21:35.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:34 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-04-24T08:21:35.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:34 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-04-24T08:21:35.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:34 smithi194 ceph-mon[32836]: from='mgr.25225 ' entity='mgr.x' 2024-04-24T08:21:36.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:35 smithi028 ceph-mon[27786]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-24T08:21:36.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:35 smithi028 ceph-mon[27786]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-24T08:21:36.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:35 smithi028 ceph-mon[27786]: pgmap v350: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:21:36.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:35 smithi117 ceph-mon[32438]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-24T08:21:36.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:35 smithi117 ceph-mon[32438]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-24T08:21:36.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:35 smithi117 ceph-mon[32438]: pgmap v350: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:21:36.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:35 smithi194 ceph-mon[32836]: Adjusting osd_memory_target on smithi028 to 177.5M 2024-04-24T08:21:36.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:35 smithi194 ceph-mon[32836]: Unable to set osd_memory_target on smithi028 to 186173030: error parsing value: Value '186173030' is below minimum 939524096 2024-04-24T08:21:36.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:35 smithi194 ceph-mon[32836]: pgmap v350: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:21:37.492 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:21:37.492 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:21:37.492 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 3938, 2024-04-24T08:21:37.492 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 4393, 2024-04-24T08:21:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1174373, 2024-04-24T08:21:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 17221092, 2024-04-24T08:21:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 25, 2024-04-24T08:21:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-24T08:21:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 1.153886409, 2024-04-24T08:21:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.278502567, 2024-04-24T08:21:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.219225059, 2024-04-24T08:21:37.493 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.581583396, 2024-04-24T08:21:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4394, 2024-04-24T08:21:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1.487100823, 2024-04-24T08:21:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000338438 2024-04-24T08:21:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2049, 2024-04-24T08:21:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.001121288, 2024-04-24T08:21:37.494 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000547 2024-04-24T08:21:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 160, 2024-04-24T08:21:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 335888 2024-04-24T08:21:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:21:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 4364, 2024-04-24T08:21:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 4117, 2024-04-24T08:21:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 2903988, 2024-04-24T08:21:37.495 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 18016442, 2024-04-24T08:21:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 126, 2024-04-24T08:21:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 5, 2024-04-24T08:21:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 1.206270394, 2024-04-24T08:21:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.346279852, 2024-04-24T08:21:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.358193542, 2024-04-24T08:21:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.474533791, 2024-04-24T08:21:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4116, 2024-04-24T08:21:37.496 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1.337871335, 2024-04-24T08:21:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000325041 2024-04-24T08:21:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 503, 2024-04-24T08:21:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000103661, 2024-04-24T08:21:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000206 2024-04-24T08:21:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1938032, 2024-04-24T08:21:37.497 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 202288 2024-04-24T08:21:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:21:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 15531, 2024-04-24T08:21:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 18264, 2024-04-24T08:21:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 6116573, 2024-04-24T08:21:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 14212197, 2024-04-24T08:21:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 17, 2024-04-24T08:21:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 8, 2024-04-24T08:21:37.498 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 7.082352710, 2024-04-24T08:21:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 4.735471146, 2024-04-24T08:21:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 2.143160465, 2024-04-24T08:21:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.650588863, 2024-04-24T08:21:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 18259, 2024-04-24T08:21:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 9.882142203, 2024-04-24T08:21:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000541220 2024-04-24T08:21:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.499 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 26135, 2024-04-24T08:21:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.021745935, 2024-04-24T08:21:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000832 2024-04-24T08:21:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 5200304, 2024-04-24T08:21:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 8801888 2024-04-24T08:21:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-24T08:21:37.500 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 1, 2024-04-24T08:21:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-24T08:21:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-MDSRank": { 2024-04-24T08:21:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-04-24T08:21:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-04-24T08:21:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 7308, 2024-04-24T08:21:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 12.857650560, 2024-04-24T08:21:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.001759393 2024-04-24T08:21:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:37.501 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "finisher-PurgeQueue": { 2024-04-24T08:21:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "queue_len": 0, 2024-04-24T08:21:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "complete_latency": { 2024-04-24T08:21:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 656, 2024-04-24T08:21:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.123931611, 2024-04-24T08:21:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000188920 2024-04-24T08:21:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:37.502 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "mds": { 2024-04-24T08:21:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "request": 2043, 2024-04-24T08:21:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "reply": 2040, 2024-04-24T08:21:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "reply_latency": { 2024-04-24T08:21:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2040, 2024-04-24T08:21:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 40.183904977, 2024-04-24T08:21:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.019697992 2024-04-24T08:21:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.503 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_reply": 0, 2024-04-24T08:21:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "forward": 3, 2024-04-24T08:21:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_complete": 0, 2024-04-24T08:21:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_fetch_keys": 0, 2024-04-24T08:21:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_commit": 37, 2024-04-24T08:21:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_split": 3, 2024-04-24T08:21:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_merge": 0, 2024-04-24T08:21:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes": 1161, 2024-04-24T08:21:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_top": 147, 2024-04-24T08:21:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_bottom": 986, 2024-04-24T08:21:37.504 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pin_tail": 28, 2024-04-24T08:21:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_pinned": 952, 2024-04-24T08:21:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_expired": 13, 2024-04-24T08:21:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "inodes_with_caps": 401, 2024-04-24T08:21:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "caps": 401, 2024-04-24T08:21:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "subtrees": 9, 2024-04-24T08:21:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse": 2641, 2024-04-24T08:21:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_hit": 2522, 2024-04-24T08:21:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_forward": 0, 2024-04-24T08:21:37.505 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_discover": 0, 2024-04-24T08:21:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_dir_fetch": 0, 2024-04-24T08:21:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_remote_ino": 0, 2024-04-24T08:21:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: "traverse_lock": 51, 2024-04-24T08:21:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: "load_cent": 0, 2024-04-24T08:21:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: "q": 0, 2024-04-24T08:21:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: "exported": 3, 2024-04-24T08:21:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: "exported_inodes": 9, 2024-04-24T08:21:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: "imported": 0, 2024-04-24T08:21:37.506 INFO:teuthology.orchestra.run.smithi028.stdout: "imported_inodes": 0, 2024-04-24T08:21:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_dir_fetch": 0, 2024-04-24T08:21:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_backtrace_fetch": 4819, 2024-04-24T08:21:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "openino_peer_discover": 0, 2024-04-24T08:21:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rfiles": 325, 2024-04-24T08:21:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rbytes": 13424230511, 2024-04-24T08:21:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "root_rsnaps": 6, 2024-04-24T08:21:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_fetch": 4819, 2024-04-24T08:21:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_set_tag": 0, 2024-04-24T08:21:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T08:21:37.507 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_inotable_repaired": 0, 2024-04-24T08:21:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_inodes": 1099, 2024-04-24T08:21:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dir_base_inodes": 16, 2024-04-24T08:21:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_dirfrag_rstats": 1083, 2024-04-24T08:21:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "scrub_file_inodes": 3720, 2024-04-24T08:21:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_inode_file_caps": 4, 2024-04-24T08:21:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_revoke": 530, 2024-04-24T08:21:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_grant": 3594, 2024-04-24T08:21:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_trunc": 6, 2024-04-24T08:21:37.508 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T08:21:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T08:21:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps": 9516, 2024-04-24T08:21:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_caps_dirty": 7668, 2024-04-24T08:21:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_cap_release": 22, 2024-04-24T08:21:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: "process_request_cap_release": 913 2024-04-24T08:21:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_cache": { 2024-04-24T08:21:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update": 1, 2024-04-24T08:21:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_update_receipt": 0, 2024-04-24T08:21:37.509 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_try_discover": 0, 2024-04-24T08:21:37.510 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_send_discover": 0, 2024-04-24T08:21:37.510 INFO:teuthology.orchestra.run.smithi028.stdout: "dir_handle_discover": 10, 2024-04-24T08:21:37.510 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays": 0, 2024-04-24T08:21:37.510 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_delayed": 0, 2024-04-24T08:21:37.510 INFO:teuthology.orchestra.run.smithi028.stdout: "num_strays_enqueuing": 0, 2024-04-24T08:21:37.510 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_created": 530, 2024-04-24T08:21:37.510 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_enqueued": 469, 2024-04-24T08:21:37.510 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_reintegrated": 61, 2024-04-24T08:21:37.510 INFO:teuthology.orchestra.run.smithi028.stdout: "strays_migrated": 0, 2024-04-24T08:21:37.511 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_processing": 0, 2024-04-24T08:21:37.511 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_enqueued": 0, 2024-04-24T08:21:37.511 INFO:teuthology.orchestra.run.smithi028.stdout: "num_recovering_prioritized": 0, 2024-04-24T08:21:37.511 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_started": 0, 2024-04-24T08:21:37.511 INFO:teuthology.orchestra.run.smithi028.stdout: "recovery_completed": 0, 2024-04-24T08:21:37.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_path": 0, 2024-04-24T08:21:37.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_quiesce_inode": 0, 2024-04-24T08:21:37.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_enqueue_scrub": 16, 2024-04-24T08:21:37.511 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_exportdir": 3, 2024-04-24T08:21:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_flush": 0, 2024-04-24T08:21:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragmentdir": 3, 2024-04-24T08:21:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_fragstats": 0, 2024-04-24T08:21:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: "ireq_inodestats": 0 2024-04-24T08:21:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_log": { 2024-04-24T08:21:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: "evlrg": 0, 2024-04-24T08:21:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: "evadd": 11070, 2024-04-24T08:21:37.512 INFO:teuthology.orchestra.run.smithi028.stdout: "evex": 0, 2024-04-24T08:21:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "evtrm": 0, 2024-04-24T08:21:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "ev": 11070, 2024-04-24T08:21:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "evexg": 0, 2024-04-24T08:21:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "evexd": 0, 2024-04-24T08:21:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "segadd": 16, 2024-04-24T08:21:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "segex": 0, 2024-04-24T08:21:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "segtrm": 0, 2024-04-24T08:21:37.513 INFO:teuthology.orchestra.run.smithi028.stdout: "seg": 16, 2024-04-24T08:21:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "segmjr": 2, 2024-04-24T08:21:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "segexg": 0, 2024-04-24T08:21:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "segexd": 0, 2024-04-24T08:21:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "expos": 4194304, 2024-04-24T08:21:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "wrpos": 41592929, 2024-04-24T08:21:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "rdpos": 0, 2024-04-24T08:21:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "jlat": { 2024-04-24T08:21:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 2272, 2024-04-24T08:21:37.514 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 323.902295482, 2024-04-24T08:21:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.142562630 2024-04-24T08:21:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "replayed": 0 2024-04-24T08:21:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_mem": { 2024-04-24T08:21:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "ino": 944, 2024-04-24T08:21:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "ino+": 1434, 2024-04-24T08:21:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "ino-": 490, 2024-04-24T08:21:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "dir": 121, 2024-04-24T08:21:37.515 INFO:teuthology.orchestra.run.smithi028.stdout: "dir+": 141, 2024-04-24T08:21:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "dir-": 20, 2024-04-24T08:21:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "dn": 1161, 2024-04-24T08:21:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "dn+": 1701, 2024-04-24T08:21:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "dn-": 540, 2024-04-24T08:21:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "cap": 401, 2024-04-24T08:21:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "cap+": 887, 2024-04-24T08:21:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "cap-": 486, 2024-04-24T08:21:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "rss": 69428, 2024-04-24T08:21:37.516 INFO:teuthology.orchestra.run.smithi028.stdout: "heap": 231800 2024-04-24T08:21:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_server": { 2024-04-24T08:21:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_client_request": 2426, 2024-04-24T08:21:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "dispatch_server_request": 1, 2024-04-24T08:21:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_request": 2043, 2024-04-24T08:21:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_client_session": 35, 2024-04-24T08:21:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "handle_peer_request": 2, 2024-04-24T08:21:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "req_create_latency": { 2024-04-24T08:21:37.517 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 765, 2024-04-24T08:21:37.518 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1.878115751, 2024-04-24T08:21:37.518 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002455053 2024-04-24T08:21:37.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.518 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getattr_latency": { 2024-04-24T08:21:37.518 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 177, 2024-04-24T08:21:37.518 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.083579643, 2024-04-24T08:21:37.518 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000472201 2024-04-24T08:21:37.518 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.518 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getfilelock_latency": { 2024-04-24T08:21:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "req_link_latency": { 2024-04-24T08:21:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 61, 2024-04-24T08:21:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.143216797, 2024-04-24T08:21:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002347816 2024-04-24T08:21:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.519 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookup_latency": { 2024-04-24T08:21:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 58, 2024-04-24T08:21:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.051545096, 2024-04-24T08:21:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000888708 2024-04-24T08:21:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookuphash_latency": { 2024-04-24T08:21:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.520 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupino_latency": { 2024-04-24T08:21:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupname_latency": { 2024-04-24T08:21:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.521 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupparent_latency": { 2024-04-24T08:21:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lookupsnap_latency": { 2024-04-24T08:21:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.522 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: "req_lssnap_latency": { 2024-04-24T08:21:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 11, 2024-04-24T08:21:37.523 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.002945372, 2024-04-24T08:21:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000267761 2024-04-24T08:21:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mkdir_latency": { 2024-04-24T08:21:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 105, 2024-04-24T08:21:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.255085488, 2024-04-24T08:21:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002429385 2024-04-24T08:21:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mknod_latency": { 2024-04-24T08:21:37.524 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "req_mksnap_latency": { 2024-04-24T08:21:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 11, 2024-04-24T08:21:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.603284494, 2024-04-24T08:21:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.054844044 2024-04-24T08:21:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "req_open_latency": { 2024-04-24T08:21:37.525 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 49, 2024-04-24T08:21:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.028639279, 2024-04-24T08:21:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000584475 2024-04-24T08:21:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "req_readdir_latency": { 2024-04-24T08:21:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 156, 2024-04-24T08:21:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1.809782830, 2024-04-24T08:21:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.011601171 2024-04-24T08:21:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.526 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rename_latency": { 2024-04-24T08:21:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 113, 2024-04-24T08:21:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 33.429448920, 2024-04-24T08:21:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.295835831 2024-04-24T08:21:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: "req_renamesnap_latency": { 2024-04-24T08:21:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.527 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "req_snapdiff_latency": { 2024-04-24T08:21:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmdir_latency": { 2024-04-24T08:21:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 16, 2024-04-24T08:21:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.401135091, 2024-04-24T08:21:37.528 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.025070943 2024-04-24T08:21:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmsnap_latency": { 2024-04-24T08:21:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 5, 2024-04-24T08:21:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.036817823, 2024-04-24T08:21:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.007363564 2024-04-24T08:21:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: "req_rmxattr_latency": { 2024-04-24T08:21:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.529 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setattr_latency": { 2024-04-24T08:21:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 7, 2024-04-24T08:21:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.062638978, 2024-04-24T08:21:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.008948425 2024-04-24T08:21:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setdirlayout_latency": { 2024-04-24T08:21:37.530 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setfilelock_latency": { 2024-04-24T08:21:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 4, 2024-04-24T08:21:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.001390814, 2024-04-24T08:21:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000347703 2024-04-24T08:21:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.531 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setlayout_latency": { 2024-04-24T08:21:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: "req_setxattr_latency": { 2024-04-24T08:21:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 8, 2024-04-24T08:21:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.027926036, 2024-04-24T08:21:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.003490754 2024-04-24T08:21:37.532 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "req_symlink_latency": { 2024-04-24T08:21:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 13, 2024-04-24T08:21:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.014659655, 2024-04-24T08:21:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.001127665 2024-04-24T08:21:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "req_unlink_latency": { 2024-04-24T08:21:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 481, 2024-04-24T08:21:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 1.353692910, 2024-04-24T08:21:37.533 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.002814330 2024-04-24T08:21:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_revoke_eviction": 0, 2024-04-24T08:21:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "cap_acquisition_throttle": 0, 2024-04-24T08:21:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "req_getvxattr_latency": { 2024-04-24T08:21:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:37.534 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_sessions": { 2024-04-24T08:21:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "session_count": 2, 2024-04-24T08:21:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "session_add": 5, 2024-04-24T08:21:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "session_remove": 3, 2024-04-24T08:21:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_open": 2, 2024-04-24T08:21:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "sessions_stale": 0, 2024-04-24T08:21:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "total_load": 32, 2024-04-24T08:21:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "average_load": 16, 2024-04-24T08:21:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "avg_session_uptime": 653, 2024-04-24T08:21:37.535 INFO:teuthology.orchestra.run.smithi028.stdout: "mdthresh_evicted": 0 2024-04-24T08:21:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-24T08:21:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:21:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-24T08:21:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:21:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:21:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:21:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:21:37.536 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:21:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:21:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:21:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:21:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:21:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:21:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:21:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:21:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:21:37.537 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-24T08:21:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:21:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-24T08:21:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:21:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:21:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:21:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:21:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:21:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:21:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:21:37.538 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-24T08:21:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:21:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:21:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:21:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-24T08:21:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-24T08:21:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-24T08:21:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:21:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:21:37.539 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:21:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:21:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 338482, 2024-04-24T08:21:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 532, 2024-04-24T08:21:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 968, 2024-04-24T08:21:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 11, 2024-04-24T08:21:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-24T08:21:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-24T08:21:37.540 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:21:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-24T08:21:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:21:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-24T08:21:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 8320, 2024-04-24T08:21:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 173, 2024-04-24T08:21:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:21:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:21:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-24T08:21:37.541 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-24T08:21:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 3393602, 2024-04-24T08:21:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 36827, 2024-04-24T08:21:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-24T08:21:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-24T08:21:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-24T08:21:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-24T08:21:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "objecter": { 2024-04-24T08:21:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "op_active": 0, 2024-04-24T08:21:37.542 INFO:teuthology.orchestra.run.smithi028.stdout: "op_laggy": 0, 2024-04-24T08:21:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send": 8531, 2024-04-24T08:21:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "op_send_bytes": 37904575, 2024-04-24T08:21:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "op_resend": 37, 2024-04-24T08:21:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "op_reply": 8531, 2024-04-24T08:21:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "op_latency": { 2024-04-24T08:21:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 8531, 2024-04-24T08:21:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 908.655233030, 2024-04-24T08:21:37.543 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.106512159 2024-04-24T08:21:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: "op_inflight": 0, 2024-04-24T08:21:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: "oplen_avg": { 2024-04-24T08:21:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 8531, 2024-04-24T08:21:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 8940 2024-04-24T08:21:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: "op": 8531, 2024-04-24T08:21:37.544 INFO:teuthology.orchestra.run.smithi028.stdout: "op_r": 4869, 2024-04-24T08:21:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "op_w": 3662, 2024-04-24T08:21:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "op_rmw": 0, 2024-04-24T08:21:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "op_pg": 0, 2024-04-24T08:21:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_stat": 0, 2024-04-24T08:21:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_create": 11, 2024-04-24T08:21:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_read": 34, 2024-04-24T08:21:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_write": 2290, 2024-04-24T08:21:37.545 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writefull": 51, 2024-04-24T08:21:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_writesame": 0, 2024-04-24T08:21:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_append": 0, 2024-04-24T08:21:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_zero": 0, 2024-04-24T08:21:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_truncate": 1, 2024-04-24T08:21:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_delete": 991, 2024-04-24T08:21:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_mapext": 0, 2024-04-24T08:21:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_sparse_read": 0, 2024-04-24T08:21:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_clonerange": 0, 2024-04-24T08:21:37.546 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_getxattr": 4835, 2024-04-24T08:21:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_setxattr": 22, 2024-04-24T08:21:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_cmpxattr": 0, 2024-04-24T08:21:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_rmxattr": 0, 2024-04-24T08:21:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_resetxattrs": 0, 2024-04-24T08:21:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_call": 0, 2024-04-24T08:21:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_watch": 0, 2024-04-24T08:21:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_notify": 0, 2024-04-24T08:21:37.547 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T08:21:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls": 0, 2024-04-24T08:21:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_pgls_filter": 0, 2024-04-24T08:21:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "osdop_other": 6, 2024-04-24T08:21:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_active": 0, 2024-04-24T08:21:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_send": 0, 2024-04-24T08:21:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_resend": 0, 2024-04-24T08:21:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "linger_ping": 0, 2024-04-24T08:21:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_active": 0, 2024-04-24T08:21:37.548 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_send": 0, 2024-04-24T08:21:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "poolop_resend": 0, 2024-04-24T08:21:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_active": 0, 2024-04-24T08:21:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_send": 0, 2024-04-24T08:21:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "poolstat_resend": 0, 2024-04-24T08:21:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_active": 0, 2024-04-24T08:21:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_send": 0, 2024-04-24T08:21:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "statfs_resend": 0, 2024-04-24T08:21:37.549 INFO:teuthology.orchestra.run.smithi028.stdout: "command_active": 0, 2024-04-24T08:21:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: "command_send": 0, 2024-04-24T08:21:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: "command_resend": 0, 2024-04-24T08:21:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: "map_epoch": 101, 2024-04-24T08:21:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: "map_full": 0, 2024-04-24T08:21:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: "map_inc": 16, 2024-04-24T08:21:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_sessions": 12, 2024-04-24T08:21:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_open": 12, 2024-04-24T08:21:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_session_close": 0, 2024-04-24T08:21:37.550 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_laggy": 0, 2024-04-24T08:21:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_wr": 493, 2024-04-24T08:21:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_rd": 0, 2024-04-24T08:21:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_del": 206 2024-04-24T08:21:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "oft": { 2024-04-24T08:21:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_objs": 1, 2024-04-24T08:21:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_kv_pairs": 0, 2024-04-24T08:21:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_updates": 0, 2024-04-24T08:21:37.551 INFO:teuthology.orchestra.run.smithi028.stdout: "omap_total_removes": 0 2024-04-24T08:21:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: "osd-slow-ops": { 2024-04-24T08:21:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: "slow_ops_count": 1 2024-04-24T08:21:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: "purge_queue": { 2024-04-24T08:21:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops": 0, 2024-04-24T08:21:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_ops_high_water": 10, 2024-04-24T08:21:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing": 0, 2024-04-24T08:21:37.552 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executing_high_water": 10, 2024-04-24T08:21:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed_ops": 469, 2024-04-24T08:21:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_executed": 469, 2024-04-24T08:21:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "pq_item_in_journal": 0 2024-04-24T08:21:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:21:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-24T08:21:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-24T08:21:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-24T08:21:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 23829, 2024-04-24T08:21:37.553 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 7452482, 2024-04-24T08:21:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 23829, 2024-04-24T08:21:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-24T08:21:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-24T08:21:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 23829, 2024-04-24T08:21:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 7452482, 2024-04-24T08:21:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-24T08:21:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.554 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_bytes": { 2024-04-24T08:21:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-24T08:21:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-24T08:21:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-24T08:21:37.555 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-24T08:21:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-24T08:21:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-24T08:21:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 8531, 2024-04-24T08:21:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 37951944, 2024-04-24T08:21:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 8531, 2024-04-24T08:21:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 37951944, 2024-04-24T08:21:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-24T08:21:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.556 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-objecter_ops": { 2024-04-24T08:21:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-24T08:21:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 1024, 2024-04-24T08:21:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-24T08:21:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 0, 2024-04-24T08:21:37.557 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 0, 2024-04-24T08:21:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 0, 2024-04-24T08:21:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 8531, 2024-04-24T08:21:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 8531, 2024-04-24T08:21:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 8531, 2024-04-24T08:21:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 8531, 2024-04-24T08:21:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-24T08:21:37.558 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle": { 2024-04-24T08:21:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-24T08:21:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-04-24T08:21:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-24T08:21:37.559 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 469, 2024-04-24T08:21:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 47369, 2024-04-24T08:21:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 469, 2024-04-24T08:21:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-24T08:21:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-24T08:21:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 18, 2024-04-24T08:21:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 47369, 2024-04-24T08:21:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-24T08:21:37.560 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-write_buf_throttle-0x5608a7e92500": { 2024-04-24T08:21:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-24T08:21:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 3758096384, 2024-04-24T08:21:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-24T08:21:37.561 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 11070, 2024-04-24T08:21:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 37398625, 2024-04-24T08:21:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 11070, 2024-04-24T08:21:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-24T08:21:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-24T08:21:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 2272, 2024-04-24T08:21:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 37398625, 2024-04-24T08:21:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-24T08:21:37.562 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:37.563 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:37.563 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:21:37.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:37 smithi028 ceph-mon[27786]: pgmap v351: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:21:38.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:37 smithi117 ceph-mon[32438]: pgmap v351: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:21:38.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:37 smithi194 ceph-mon[32836]: pgmap v351: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:21:38.497 INFO:tasks.check_counter:Daemon mds.g mds.exported=3 2024-04-24T08:21:38.497 INFO:tasks.check_counter:Daemon mds.g mds.imported=0 2024-04-24T08:21:38.497 INFO:tasks.check_counter:Daemon mds.g mds.root_rsnaps=6 2024-04-24T08:21:38.497 INFO:tasks.check_counter:Daemon mds.g mds_server.req_mksnap_latency.avgcount=11 2024-04-24T08:21:38.498 INFO:tasks.check_counter:Daemon mds.g mds_server.req_rmsnap_latency.avgcount=5 2024-04-24T08:21:38.498 INFO:tasks.check_counter:Daemon mds.g mds.dir_split=3 2024-04-24T08:21:38.498 DEBUG:tasks.check_counter:Getting stats from j 2024-04-24T08:21:38.498 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph daemon mds.j perf dump 2024-04-24T08:21:38.971 INFO:teuthology.orchestra.run.smithi028.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.a/config 2024-04-24T08:21:39.764 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:39 smithi194 ceph-mon[32836]: pgmap v352: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:21:40.016 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:39 smithi028 ceph-mon[27786]: pgmap v352: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:21:40.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:39 smithi117 ceph-mon[32438]: pgmap v352: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-04-24T08:21:41.321 INFO:teuthology.orchestra.run.smithi028.stdout:{ 2024-04-24T08:21:41.322 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:21:41.322 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 0, 2024-04-24T08:21:41.322 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 2, 2024-04-24T08:21:41.322 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 0, 2024-04-24T08:21:41.322 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 0, 2024-04-24T08:21:41.322 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 4, 2024-04-24T08:21:41.323 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 0, 2024-04-24T08:21:41.323 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.004157469, 2024-04-24T08:21:41.323 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.000000000, 2024-04-24T08:21:41.323 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.001082544, 2024-04-24T08:21:41.323 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:21:41.323 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:41.323 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:41.323 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:41.323 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:41.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:41.324 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:41.324 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:41.324 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:41.324 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:41.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:41.324 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T08:21:41.324 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T08:21:41.324 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:41.325 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:21:41.325 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 2, 2024-04-24T08:21:41.325 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 177, 2024-04-24T08:21:41.325 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 320, 2024-04-24T08:21:41.325 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 187456, 2024-04-24T08:21:41.325 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 5, 2024-04-24T08:21:41.325 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 3, 2024-04-24T08:21:41.325 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.036554555, 2024-04-24T08:21:41.325 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.026146224, 2024-04-24T08:21:41.326 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.004104740, 2024-04-24T08:21:41.326 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:21:41.326 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:41.326 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 174, 2024-04-24T08:21:41.326 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.037927783, 2024-04-24T08:21:41.326 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000217975 2024-04-24T08:21:41.326 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:41.326 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:41.326 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:41.327 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:41.327 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:41.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:41.327 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-24T08:21:41.327 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 187456 2024-04-24T08:21:41.327 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:41.327 INFO:teuthology.orchestra.run.smithi028.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:21:41.327 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_messages": 247, 2024-04-24T08:21:41.327 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages": 233, 2024-04-24T08:21:41.327 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_bytes": 1737168, 2024-04-24T08:21:41.328 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_bytes": 143072, 2024-04-24T08:21:41.328 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_created_connections": 2, 2024-04-24T08:21:41.328 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_active_connections": 1, 2024-04-24T08:21:41.328 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_total_time": 0.108837165, 2024-04-24T08:21:41.328 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_send_time": 0.038258829, 2024-04-24T08:21:41.328 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_recv_time": 0.083303915, 2024-04-24T08:21:41.328 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_running_fast_dispatch_time": 0.009849829, 2024-04-24T08:21:41.328 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:41.328 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 233, 2024-04-24T08:21:41.329 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.038356852, 2024-04-24T08:21:41.329 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000164621 2024-04-24T08:21:41.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:41.329 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:41.329 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:41.329 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:41.329 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:41.329 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:41.329 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_recv_encrypted_bytes": 1737168, 2024-04-24T08:21:41.329 INFO:teuthology.orchestra.run.smithi028.stdout: "msgr_send_encrypted_bytes": 143072 2024-04-24T08:21:41.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:41.330 INFO:teuthology.orchestra.run.smithi028.stdout: "cct": { 2024-04-24T08:21:41.330 INFO:teuthology.orchestra.run.smithi028.stdout: "total_workers": 0, 2024-04-24T08:21:41.330 INFO:teuthology.orchestra.run.smithi028.stdout: "unhealthy_workers": 0 2024-04-24T08:21:41.330 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:41.330 INFO:teuthology.orchestra.run.smithi028.stdout: "mempool": { 2024-04-24T08:21:41.330 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:21:41.330 INFO:teuthology.orchestra.run.smithi028.stdout: "bloom_filter_items": 0, 2024-04-24T08:21:41.331 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:21:41.331 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:21:41.331 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:21:41.331 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:21:41.331 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:21:41.331 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:21:41.331 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:21:41.331 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:21:41.331 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:21:41.332 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:21:41.332 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:21:41.332 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:21:41.332 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:21:41.332 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_extent_items": 0, 2024-04-24T08:21:41.332 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:21:41.332 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_blob_items": 0, 2024-04-24T08:21:41.332 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:21:41.332 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:21:41.332 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:21:41.333 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:21:41.333 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:21:41.333 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:21:41.333 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:21:41.333 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_txc_items": 0, 2024-04-24T08:21:41.333 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:21:41.333 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:21:41.333 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:21:41.334 INFO:teuthology.orchestra.run.smithi028.stdout: "bluestore_writing_items": 0, 2024-04-24T08:21:41.334 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_bytes": 0, 2024-04-24T08:21:41.334 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_items": 0, 2024-04-24T08:21:41.334 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:21:41.334 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:21:41.334 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:21:41.334 INFO:teuthology.orchestra.run.smithi028.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:21:41.334 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_bytes": 149745, 2024-04-24T08:21:41.334 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_anon_items": 31, 2024-04-24T08:21:41.335 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:21:41.335 INFO:teuthology.orchestra.run.smithi028.stdout: "buffer_meta_items": 0, 2024-04-24T08:21:41.335 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_bytes": 0, 2024-04-24T08:21:41.335 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_items": 0, 2024-04-24T08:21:41.335 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:21:41.335 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_mapbl_items": 0, 2024-04-24T08:21:41.335 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:21:41.335 INFO:teuthology.orchestra.run.smithi028.stdout: "osd_pglog_items": 0, 2024-04-24T08:21:41.335 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_bytes": 0, 2024-04-24T08:21:41.336 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_items": 0, 2024-04-24T08:21:41.336 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:21:41.336 INFO:teuthology.orchestra.run.smithi028.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:21:41.336 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_bytes": 0, 2024-04-24T08:21:41.336 INFO:teuthology.orchestra.run.smithi028.stdout: "pgmap_items": 0, 2024-04-24T08:21:41.336 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_bytes": 928, 2024-04-24T08:21:41.336 INFO:teuthology.orchestra.run.smithi028.stdout: "mds_co_items": 2, 2024-04-24T08:21:41.337 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_bytes": 0, 2024-04-24T08:21:41.337 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_1_items": 0, 2024-04-24T08:21:41.337 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_bytes": 0, 2024-04-24T08:21:41.337 INFO:teuthology.orchestra.run.smithi028.stdout: "unittest_2_items": 0 2024-04-24T08:21:41.337 INFO:teuthology.orchestra.run.smithi028.stdout: }, 2024-04-24T08:21:41.337 INFO:teuthology.orchestra.run.smithi028.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:21:41.337 INFO:teuthology.orchestra.run.smithi028.stdout: "val": 0, 2024-04-24T08:21:41.337 INFO:teuthology.orchestra.run.smithi028.stdout: "max": 104857600, 2024-04-24T08:21:41.337 INFO:teuthology.orchestra.run.smithi028.stdout: "get_started": 0, 2024-04-24T08:21:41.337 INFO:teuthology.orchestra.run.smithi028.stdout: "get": 245, 2024-04-24T08:21:41.338 INFO:teuthology.orchestra.run.smithi028.stdout: "get_sum": 1699862, 2024-04-24T08:21:41.338 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:41.338 INFO:teuthology.orchestra.run.smithi028.stdout: "get_or_fail_success": 245, 2024-04-24T08:21:41.338 INFO:teuthology.orchestra.run.smithi028.stdout: "take": 0, 2024-04-24T08:21:41.338 INFO:teuthology.orchestra.run.smithi028.stdout: "take_sum": 0, 2024-04-24T08:21:41.338 INFO:teuthology.orchestra.run.smithi028.stdout: "put": 245, 2024-04-24T08:21:41.338 INFO:teuthology.orchestra.run.smithi028.stdout: "put_sum": 1699862, 2024-04-24T08:21:41.338 INFO:teuthology.orchestra.run.smithi028.stdout: "wait": { 2024-04-24T08:21:41.339 INFO:teuthology.orchestra.run.smithi028.stdout: "avgcount": 0, 2024-04-24T08:21:41.339 INFO:teuthology.orchestra.run.smithi028.stdout: "sum": 0.000000000, 2024-04-24T08:21:41.339 INFO:teuthology.orchestra.run.smithi028.stdout: "avgtime": 0.000000000 2024-04-24T08:21:41.339 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:41.339 INFO:teuthology.orchestra.run.smithi028.stdout: } 2024-04-24T08:21:41.339 INFO:teuthology.orchestra.run.smithi028.stdout:} 2024-04-24T08:21:42.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:41 smithi028 ceph-mon[27786]: pgmap v353: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:21:42.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:41 smithi117 ceph-mon[32438]: pgmap v353: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:21:42.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:41 smithi194 ceph-mon[32836]: pgmap v353: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-04-24T08:21:42.227 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-04-24T08:21:42.228 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-04-24T08:21:42.228 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-04-24T08:21:42.228 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-04-24T08:21:42.228 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-04-24T08:21:42.228 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-04-24T08:21:42.228 DEBUG:tasks.check_counter:Getting stats from b 2024-04-24T08:21:42.229 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph daemon mds.b perf dump 2024-04-24T08:21:42.499 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:21:43.997 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:43 smithi117 ceph-mon[32438]: pgmap v354: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:21:44.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:43 smithi028 ceph-mon[27786]: pgmap v354: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:21:44.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:43 smithi194 ceph-mon[32836]: pgmap v354: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:21:44.442 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-04-24T08:21:44.442 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:21:44.442 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 67, 2024-04-24T08:21:44.442 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 109, 2024-04-24T08:21:44.443 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 18929, 2024-04-24T08:21:44.443 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 80827, 2024-04-24T08:21:44.443 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 15, 2024-04-24T08:21:44.443 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 4, 2024-04-24T08:21:44.443 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.022674993, 2024-04-24T08:21:44.443 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.004416070, 2024-04-24T08:21:44.443 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.013473230, 2024-04-24T08:21:44.443 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.000657641, 2024-04-24T08:21:44.443 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:44.444 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 103, 2024-04-24T08:21:44.444 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.027220805, 2024-04-24T08:21:44.444 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000264279 2024-04-24T08:21:44.444 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.444 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:44.444 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 88, 2024-04-24T08:21:44.444 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000019082, 2024-04-24T08:21:44.444 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000216 2024-04-24T08:21:44.445 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.445 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 10880, 2024-04-24T08:21:44.445 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 18208 2024-04-24T08:21:44.445 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.445 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:21:44.445 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 1279, 2024-04-24T08:21:44.445 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 1104, 2024-04-24T08:21:44.445 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 2040694, 2024-04-24T08:21:44.445 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 565052, 2024-04-24T08:21:44.446 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 7, 2024-04-24T08:21:44.446 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 5, 2024-04-24T08:21:44.446 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.134820904, 2024-04-24T08:21:44.446 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.045867662, 2024-04-24T08:21:44.446 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.105328231, 2024-04-24T08:21:44.446 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.012243167, 2024-04-24T08:21:44.446 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:44.446 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1107, 2024-04-24T08:21:44.447 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.107338350, 2024-04-24T08:21:44.447 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000096963 2024-04-24T08:21:44.447 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.447 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:44.447 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 1683, 2024-04-24T08:21:44.447 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000449606, 2024-04-24T08:21:44.447 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000267 2024-04-24T08:21:44.447 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.447 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 1784736, 2024-04-24T08:21:44.448 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 158592 2024-04-24T08:21:44.448 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.448 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:21:44.448 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 59, 2024-04-24T08:21:44.448 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 228, 2024-04-24T08:21:44.448 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 13787, 2024-04-24T08:21:44.448 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 860502, 2024-04-24T08:21:44.448 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 41, 2024-04-24T08:21:44.448 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 7, 2024-04-24T08:21:44.449 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.027529016, 2024-04-24T08:21:44.449 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.008042339, 2024-04-24T08:21:44.449 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.016357398, 2024-04-24T08:21:44.449 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.000541408, 2024-04-24T08:21:44.449 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:44.449 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 226, 2024-04-24T08:21:44.449 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.041715621, 2024-04-24T08:21:44.449 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000184582 2024-04-24T08:21:44.450 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.450 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:44.450 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 46, 2024-04-24T08:21:44.450 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000005796, 2024-04-24T08:21:44.450 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000126 2024-04-24T08:21:44.450 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.450 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 5184, 2024-04-24T08:21:44.450 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 473296 2024-04-24T08:21:44.450 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.451 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-04-24T08:21:44.451 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 1, 2024-04-24T08:21:44.451 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-04-24T08:21:44.451 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.451 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-MDSRank": { 2024-04-24T08:21:44.451 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-04-24T08:21:44.451 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-04-24T08:21:44.451 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 73, 2024-04-24T08:21:44.451 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.002103695, 2024-04-24T08:21:44.452 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000028817 2024-04-24T08:21:44.452 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:44.452 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.452 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-PurgeQueue": { 2024-04-24T08:21:44.452 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-04-24T08:21:44.452 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-04-24T08:21:44.452 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2, 2024-04-24T08:21:44.452 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000007731, 2024-04-24T08:21:44.452 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000003865 2024-04-24T08:21:44.453 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:44.453 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.453 INFO:teuthology.orchestra.run.smithi117.stdout: "mds": { 2024-04-24T08:21:44.453 INFO:teuthology.orchestra.run.smithi117.stdout: "request": 25, 2024-04-24T08:21:44.453 INFO:teuthology.orchestra.run.smithi117.stdout: "reply": 11, 2024-04-24T08:21:44.453 INFO:teuthology.orchestra.run.smithi117.stdout: "reply_latency": { 2024-04-24T08:21:44.453 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 11, 2024-04-24T08:21:44.453 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.003030438, 2024-04-24T08:21:44.453 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000275494 2024-04-24T08:21:44.454 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.454 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_reply": 0, 2024-04-24T08:21:44.454 INFO:teuthology.orchestra.run.smithi117.stdout: "forward": 14, 2024-04-24T08:21:44.454 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_complete": 0, 2024-04-24T08:21:44.454 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_keys": 0, 2024-04-24T08:21:44.454 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_commit": 11, 2024-04-24T08:21:44.454 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_split": 0, 2024-04-24T08:21:44.454 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_merge": 0, 2024-04-24T08:21:44.455 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes": 17, 2024-04-24T08:21:44.455 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_top": 0, 2024-04-24T08:21:44.455 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_bottom": 17, 2024-04-24T08:21:44.455 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pin_tail": 0, 2024-04-24T08:21:44.455 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pinned": 16, 2024-04-24T08:21:44.455 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_expired": 0, 2024-04-24T08:21:44.455 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_with_caps": 3, 2024-04-24T08:21:44.455 INFO:teuthology.orchestra.run.smithi117.stdout: "caps": 4, 2024-04-24T08:21:44.455 INFO:teuthology.orchestra.run.smithi117.stdout: "subtrees": 2, 2024-04-24T08:21:44.456 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse": 30, 2024-04-24T08:21:44.456 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_hit": 18, 2024-04-24T08:21:44.456 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_forward": 8, 2024-04-24T08:21:44.456 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_discover": 2, 2024-04-24T08:21:44.456 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_dir_fetch": 0, 2024-04-24T08:21:44.456 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_remote_ino": 0, 2024-04-24T08:21:44.456 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_lock": 0, 2024-04-24T08:21:44.456 INFO:teuthology.orchestra.run.smithi117.stdout: "load_cent": 0, 2024-04-24T08:21:44.456 INFO:teuthology.orchestra.run.smithi117.stdout: "q": 0, 2024-04-24T08:21:44.457 INFO:teuthology.orchestra.run.smithi117.stdout: "exported": 0, 2024-04-24T08:21:44.457 INFO:teuthology.orchestra.run.smithi117.stdout: "exported_inodes": 0, 2024-04-24T08:21:44.457 INFO:teuthology.orchestra.run.smithi117.stdout: "imported": 0, 2024-04-24T08:21:44.457 INFO:teuthology.orchestra.run.smithi117.stdout: "imported_inodes": 0, 2024-04-24T08:21:44.457 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_dir_fetch": 0, 2024-04-24T08:21:44.457 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_backtrace_fetch": 0, 2024-04-24T08:21:44.457 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_peer_discover": 0, 2024-04-24T08:21:44.457 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rfiles": 325, 2024-04-24T08:21:44.457 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rbytes": 13424230511, 2024-04-24T08:21:44.458 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rsnaps": 6, 2024-04-24T08:21:44.458 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_fetch": 0, 2024-04-24T08:21:44.458 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_set_tag": 0, 2024-04-24T08:21:44.458 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T08:21:44.458 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_inotable_repaired": 0, 2024-04-24T08:21:44.458 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_inodes": 0, 2024-04-24T08:21:44.458 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T08:21:44.458 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dirfrag_rstats": 0, 2024-04-24T08:21:44.458 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_file_inodes": 0, 2024-04-24T08:21:44.459 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_inode_file_caps": 0, 2024-04-24T08:21:44.459 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_revoke": 1, 2024-04-24T08:21:44.459 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_grant": 2, 2024-04-24T08:21:44.459 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T08:21:44.459 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T08:21:44.459 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T08:21:44.459 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps": 1, 2024-04-24T08:21:44.459 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps_dirty": 0, 2024-04-24T08:21:44.459 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_cap_release": 0, 2024-04-24T08:21:44.460 INFO:teuthology.orchestra.run.smithi117.stdout: "process_request_cap_release": 1 2024-04-24T08:21:44.460 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.460 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_cache": { 2024-04-24T08:21:44.460 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update": 0, 2024-04-24T08:21:44.460 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update_receipt": 3, 2024-04-24T08:21:44.460 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_try_discover": 2, 2024-04-24T08:21:44.460 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_send_discover": 4, 2024-04-24T08:21:44.460 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_handle_discover": 0, 2024-04-24T08:21:44.460 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays": 0, 2024-04-24T08:21:44.461 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_delayed": 0, 2024-04-24T08:21:44.461 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_enqueuing": 0, 2024-04-24T08:21:44.461 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_created": 0, 2024-04-24T08:21:44.461 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_enqueued": 0, 2024-04-24T08:21:44.461 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_reintegrated": 0, 2024-04-24T08:21:44.461 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_migrated": 0, 2024-04-24T08:21:44.461 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_processing": 0, 2024-04-24T08:21:44.461 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_enqueued": 0, 2024-04-24T08:21:44.462 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_prioritized": 0, 2024-04-24T08:21:44.462 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_started": 0, 2024-04-24T08:21:44.462 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_completed": 0, 2024-04-24T08:21:44.462 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_path": 0, 2024-04-24T08:21:44.462 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_inode": 0, 2024-04-24T08:21:44.462 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T08:21:44.462 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_exportdir": 0, 2024-04-24T08:21:44.462 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_flush": 0, 2024-04-24T08:21:44.462 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragmentdir": 0, 2024-04-24T08:21:44.463 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragstats": 0, 2024-04-24T08:21:44.463 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_inodestats": 0 2024-04-24T08:21:44.463 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.463 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_log": { 2024-04-24T08:21:44.463 INFO:teuthology.orchestra.run.smithi117.stdout: "evlrg": 0, 2024-04-24T08:21:44.463 INFO:teuthology.orchestra.run.smithi117.stdout: "evadd": 10, 2024-04-24T08:21:44.463 INFO:teuthology.orchestra.run.smithi117.stdout: "evex": 0, 2024-04-24T08:21:44.463 INFO:teuthology.orchestra.run.smithi117.stdout: "evtrm": 0, 2024-04-24T08:21:44.463 INFO:teuthology.orchestra.run.smithi117.stdout: "ev": 10, 2024-04-24T08:21:44.464 INFO:teuthology.orchestra.run.smithi117.stdout: "evexg": 0, 2024-04-24T08:21:44.464 INFO:teuthology.orchestra.run.smithi117.stdout: "evexd": 0, 2024-04-24T08:21:44.464 INFO:teuthology.orchestra.run.smithi117.stdout: "segadd": 2, 2024-04-24T08:21:44.464 INFO:teuthology.orchestra.run.smithi117.stdout: "segex": 0, 2024-04-24T08:21:44.464 INFO:teuthology.orchestra.run.smithi117.stdout: "segtrm": 0, 2024-04-24T08:21:44.464 INFO:teuthology.orchestra.run.smithi117.stdout: "seg": 2, 2024-04-24T08:21:44.464 INFO:teuthology.orchestra.run.smithi117.stdout: "segmjr": 2, 2024-04-24T08:21:44.464 INFO:teuthology.orchestra.run.smithi117.stdout: "segexg": 0, 2024-04-24T08:21:44.464 INFO:teuthology.orchestra.run.smithi117.stdout: "segexd": 0, 2024-04-24T08:21:44.465 INFO:teuthology.orchestra.run.smithi117.stdout: "expos": 4194304, 2024-04-24T08:21:44.465 INFO:teuthology.orchestra.run.smithi117.stdout: "wrpos": 4197189, 2024-04-24T08:21:44.465 INFO:teuthology.orchestra.run.smithi117.stdout: "rdpos": 0, 2024-04-24T08:21:44.465 INFO:teuthology.orchestra.run.smithi117.stdout: "jlat": { 2024-04-24T08:21:44.465 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 9, 2024-04-24T08:21:44.465 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.016535996, 2024-04-24T08:21:44.465 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.001837332 2024-04-24T08:21:44.465 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.466 INFO:teuthology.orchestra.run.smithi117.stdout: "replayed": 0 2024-04-24T08:21:44.466 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.466 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_mem": { 2024-04-24T08:21:44.466 INFO:teuthology.orchestra.run.smithi117.stdout: "ino": 20, 2024-04-24T08:21:44.466 INFO:teuthology.orchestra.run.smithi117.stdout: "ino+": 20, 2024-04-24T08:21:44.466 INFO:teuthology.orchestra.run.smithi117.stdout: "ino-": 0, 2024-04-24T08:21:44.466 INFO:teuthology.orchestra.run.smithi117.stdout: "dir": 19, 2024-04-24T08:21:44.466 INFO:teuthology.orchestra.run.smithi117.stdout: "dir+": 20, 2024-04-24T08:21:44.467 INFO:teuthology.orchestra.run.smithi117.stdout: "dir-": 1, 2024-04-24T08:21:44.467 INFO:teuthology.orchestra.run.smithi117.stdout: "dn": 17, 2024-04-24T08:21:44.467 INFO:teuthology.orchestra.run.smithi117.stdout: "dn+": 17, 2024-04-24T08:21:44.467 INFO:teuthology.orchestra.run.smithi117.stdout: "dn-": 0, 2024-04-24T08:21:44.467 INFO:teuthology.orchestra.run.smithi117.stdout: "cap": 4, 2024-04-24T08:21:44.467 INFO:teuthology.orchestra.run.smithi117.stdout: "cap+": 9, 2024-04-24T08:21:44.467 INFO:teuthology.orchestra.run.smithi117.stdout: "cap-": 5, 2024-04-24T08:21:44.467 INFO:teuthology.orchestra.run.smithi117.stdout: "rss": 51944, 2024-04-24T08:21:44.467 INFO:teuthology.orchestra.run.smithi117.stdout: "heap": 223608 2024-04-24T08:21:44.468 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.468 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_server": { 2024-04-24T08:21:44.468 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_client_request": 26, 2024-04-24T08:21:44.468 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_server_request": 0, 2024-04-24T08:21:44.468 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_request": 25, 2024-04-24T08:21:44.468 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_session": 30, 2024-04-24T08:21:44.468 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_peer_request": 0, 2024-04-24T08:21:44.468 INFO:teuthology.orchestra.run.smithi117.stdout: "req_create_latency": { 2024-04-24T08:21:44.468 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.469 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.469 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.469 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.469 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getattr_latency": { 2024-04-24T08:21:44.469 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 6, 2024-04-24T08:21:44.469 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.001938509, 2024-04-24T08:21:44.469 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000323084 2024-04-24T08:21:44.469 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.469 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getfilelock_latency": { 2024-04-24T08:21:44.470 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.470 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.470 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.470 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.470 INFO:teuthology.orchestra.run.smithi117.stdout: "req_link_latency": { 2024-04-24T08:21:44.470 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.470 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.470 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.470 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.471 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookup_latency": { 2024-04-24T08:21:44.471 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 3, 2024-04-24T08:21:44.471 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000846732, 2024-04-24T08:21:44.471 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000282244 2024-04-24T08:21:44.471 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.471 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookuphash_latency": { 2024-04-24T08:21:44.471 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.471 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.471 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.472 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.472 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupino_latency": { 2024-04-24T08:21:44.472 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2, 2024-04-24T08:21:44.472 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000245197, 2024-04-24T08:21:44.472 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000122598 2024-04-24T08:21:44.472 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.472 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupname_latency": { 2024-04-24T08:21:44.472 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.472 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.473 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.473 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.473 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupparent_latency": { 2024-04-24T08:21:44.473 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.473 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.473 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.473 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.473 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupsnap_latency": { 2024-04-24T08:21:44.473 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.474 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.474 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.474 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.474 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lssnap_latency": { 2024-04-24T08:21:44.474 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.474 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.474 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.474 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.475 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mkdir_latency": { 2024-04-24T08:21:44.475 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.475 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.475 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.475 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.476 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mknod_latency": { 2024-04-24T08:21:44.476 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.476 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.476 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.476 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.476 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mksnap_latency": { 2024-04-24T08:21:44.476 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.476 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.476 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.477 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.477 INFO:teuthology.orchestra.run.smithi117.stdout: "req_open_latency": { 2024-04-24T08:21:44.477 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.477 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.477 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.477 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.477 INFO:teuthology.orchestra.run.smithi117.stdout: "req_readdir_latency": { 2024-04-24T08:21:44.477 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.478 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.478 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.478 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.478 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rename_latency": { 2024-04-24T08:21:44.478 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.478 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.478 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.478 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.478 INFO:teuthology.orchestra.run.smithi117.stdout: "req_renamesnap_latency": { 2024-04-24T08:21:44.479 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.479 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.479 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.479 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.479 INFO:teuthology.orchestra.run.smithi117.stdout: "req_snapdiff_latency": { 2024-04-24T08:21:44.479 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.479 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.479 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.479 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.480 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmdir_latency": { 2024-04-24T08:21:44.480 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.480 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.480 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.480 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.480 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmsnap_latency": { 2024-04-24T08:21:44.480 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.480 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.480 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.481 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.481 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmxattr_latency": { 2024-04-24T08:21:44.481 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.481 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.481 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.481 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.481 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setattr_latency": { 2024-04-24T08:21:44.481 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.481 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.482 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.482 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.482 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setdirlayout_latency": { 2024-04-24T08:21:44.482 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.482 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.482 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.482 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.482 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setfilelock_latency": { 2024-04-24T08:21:44.482 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.483 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.483 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.483 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.483 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setlayout_latency": { 2024-04-24T08:21:44.483 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.483 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.483 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.483 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.483 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setxattr_latency": { 2024-04-24T08:21:44.484 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.484 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.484 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.484 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.484 INFO:teuthology.orchestra.run.smithi117.stdout: "req_symlink_latency": { 2024-04-24T08:21:44.484 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.484 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.484 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.484 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.484 INFO:teuthology.orchestra.run.smithi117.stdout: "req_unlink_latency": { 2024-04-24T08:21:44.485 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.485 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.485 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.485 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.485 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_revoke_eviction": 0, 2024-04-24T08:21:44.485 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_acquisition_throttle": 0, 2024-04-24T08:21:44.485 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getvxattr_latency": { 2024-04-24T08:21:44.485 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.485 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.485 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.486 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:44.486 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.486 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_sessions": { 2024-04-24T08:21:44.486 INFO:teuthology.orchestra.run.smithi117.stdout: "session_count": 2, 2024-04-24T08:21:44.486 INFO:teuthology.orchestra.run.smithi117.stdout: "session_add": 5, 2024-04-24T08:21:44.486 INFO:teuthology.orchestra.run.smithi117.stdout: "session_remove": 3, 2024-04-24T08:21:44.486 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_open": 2, 2024-04-24T08:21:44.486 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_stale": 0, 2024-04-24T08:21:44.486 INFO:teuthology.orchestra.run.smithi117.stdout: "total_load": 8, 2024-04-24T08:21:44.487 INFO:teuthology.orchestra.run.smithi117.stdout: "average_load": 2, 2024-04-24T08:21:44.487 INFO:teuthology.orchestra.run.smithi117.stdout: "avg_session_uptime": 660, 2024-04-24T08:21:44.487 INFO:teuthology.orchestra.run.smithi117.stdout: "mdthresh_evicted": 0 2024-04-24T08:21:44.487 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.487 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-04-24T08:21:44.487 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:21:44.487 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-04-24T08:21:44.487 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:21:44.488 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:21:44.488 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:21:44.488 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:21:44.488 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:21:44.488 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:21:44.488 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:21:44.488 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:21:44.488 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:21:44.488 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:21:44.489 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:21:44.489 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:21:44.489 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:21:44.489 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-04-24T08:21:44.489 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:21:44.489 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-04-24T08:21:44.489 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:21:44.489 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:21:44.489 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:21:44.490 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:21:44.490 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:21:44.490 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:21:44.490 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:21:44.490 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-04-24T08:21:44.490 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:21:44.490 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:21:44.490 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:21:44.490 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-04-24T08:21:44.491 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-04-24T08:21:44.491 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-04-24T08:21:44.491 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:21:44.491 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:21:44.491 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:21:44.491 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:21:44.491 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 267184, 2024-04-24T08:21:44.491 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 117, 2024-04-24T08:21:44.491 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 88, 2024-04-24T08:21:44.492 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 1, 2024-04-24T08:21:44.492 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-04-24T08:21:44.492 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-04-24T08:21:44.492 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:21:44.492 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-04-24T08:21:44.492 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:21:44.492 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-04-24T08:21:44.492 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 8320, 2024-04-24T08:21:44.492 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 173, 2024-04-24T08:21:44.493 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:21:44.493 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:21:44.493 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-04-24T08:21:44.493 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-04-24T08:21:44.493 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 85586, 2024-04-24T08:21:44.493 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 436, 2024-04-24T08:21:44.493 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-04-24T08:21:44.493 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-04-24T08:21:44.493 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-04-24T08:21:44.494 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-04-24T08:21:44.494 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.494 INFO:teuthology.orchestra.run.smithi117.stdout: "objecter": { 2024-04-24T08:21:44.494 INFO:teuthology.orchestra.run.smithi117.stdout: "op_active": 0, 2024-04-24T08:21:44.494 INFO:teuthology.orchestra.run.smithi117.stdout: "op_laggy": 0, 2024-04-24T08:21:44.494 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send": 53, 2024-04-24T08:21:44.494 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send_bytes": 14234, 2024-04-24T08:21:44.495 INFO:teuthology.orchestra.run.smithi117.stdout: "op_resend": 28, 2024-04-24T08:21:44.495 INFO:teuthology.orchestra.run.smithi117.stdout: "op_reply": 53, 2024-04-24T08:21:44.495 INFO:teuthology.orchestra.run.smithi117.stdout: "op_latency": { 2024-04-24T08:21:44.495 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 53, 2024-04-24T08:21:44.495 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.307994041, 2024-04-24T08:21:44.495 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.005811208 2024-04-24T08:21:44.495 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.495 INFO:teuthology.orchestra.run.smithi117.stdout: "op_inflight": 0, 2024-04-24T08:21:44.495 INFO:teuthology.orchestra.run.smithi117.stdout: "oplen_avg": { 2024-04-24T08:21:44.496 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 53, 2024-04-24T08:21:44.496 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 81 2024-04-24T08:21:44.496 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.496 INFO:teuthology.orchestra.run.smithi117.stdout: "op": 53, 2024-04-24T08:21:44.496 INFO:teuthology.orchestra.run.smithi117.stdout: "op_r": 0, 2024-04-24T08:21:44.496 INFO:teuthology.orchestra.run.smithi117.stdout: "op_w": 53, 2024-04-24T08:21:44.496 INFO:teuthology.orchestra.run.smithi117.stdout: "op_rmw": 0, 2024-04-24T08:21:44.496 INFO:teuthology.orchestra.run.smithi117.stdout: "op_pg": 0, 2024-04-24T08:21:44.496 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_stat": 0, 2024-04-24T08:21:44.497 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_create": 10, 2024-04-24T08:21:44.497 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_read": 0, 2024-04-24T08:21:44.497 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_write": 9, 2024-04-24T08:21:44.497 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writefull": 9, 2024-04-24T08:21:44.497 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writesame": 0, 2024-04-24T08:21:44.497 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_append": 0, 2024-04-24T08:21:44.497 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_zero": 0, 2024-04-24T08:21:44.497 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_truncate": 0, 2024-04-24T08:21:44.497 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_delete": 6, 2024-04-24T08:21:44.497 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_mapext": 0, 2024-04-24T08:21:44.498 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_sparse_read": 0, 2024-04-24T08:21:44.498 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_clonerange": 0, 2024-04-24T08:21:44.498 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_getxattr": 0, 2024-04-24T08:21:44.498 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_setxattr": 20, 2024-04-24T08:21:44.498 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_cmpxattr": 0, 2024-04-24T08:21:44.498 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_rmxattr": 0, 2024-04-24T08:21:44.498 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_resetxattrs": 0, 2024-04-24T08:21:44.498 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_call": 0, 2024-04-24T08:21:44.499 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_watch": 0, 2024-04-24T08:21:44.499 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_notify": 0, 2024-04-24T08:21:44.499 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T08:21:44.499 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls": 0, 2024-04-24T08:21:44.499 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls_filter": 0, 2024-04-24T08:21:44.499 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_other": 0, 2024-04-24T08:21:44.499 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_active": 0, 2024-04-24T08:21:44.499 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_send": 0, 2024-04-24T08:21:44.500 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_resend": 0, 2024-04-24T08:21:44.500 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_ping": 0, 2024-04-24T08:21:44.500 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_active": 0, 2024-04-24T08:21:44.500 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_send": 0, 2024-04-24T08:21:44.500 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_resend": 0, 2024-04-24T08:21:44.500 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_active": 0, 2024-04-24T08:21:44.500 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_send": 0, 2024-04-24T08:21:44.500 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_resend": 0, 2024-04-24T08:21:44.500 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_active": 0, 2024-04-24T08:21:44.501 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_send": 0, 2024-04-24T08:21:44.501 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_resend": 0, 2024-04-24T08:21:44.501 INFO:teuthology.orchestra.run.smithi117.stdout: "command_active": 0, 2024-04-24T08:21:44.501 INFO:teuthology.orchestra.run.smithi117.stdout: "command_send": 0, 2024-04-24T08:21:44.501 INFO:teuthology.orchestra.run.smithi117.stdout: "command_resend": 0, 2024-04-24T08:21:44.501 INFO:teuthology.orchestra.run.smithi117.stdout: "map_epoch": 101, 2024-04-24T08:21:44.501 INFO:teuthology.orchestra.run.smithi117.stdout: "map_full": 0, 2024-04-24T08:21:44.501 INFO:teuthology.orchestra.run.smithi117.stdout: "map_inc": 15, 2024-04-24T08:21:44.501 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_sessions": 9, 2024-04-24T08:21:44.502 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_open": 9, 2024-04-24T08:21:44.502 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_close": 0, 2024-04-24T08:21:44.502 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_laggy": 0, 2024-04-24T08:21:44.502 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_wr": 26, 2024-04-24T08:21:44.502 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_rd": 0, 2024-04-24T08:21:44.502 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_del": 1 2024-04-24T08:21:44.502 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.502 INFO:teuthology.orchestra.run.smithi117.stdout: "oft": { 2024-04-24T08:21:44.503 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_objs": 1, 2024-04-24T08:21:44.503 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_kv_pairs": 0, 2024-04-24T08:21:44.503 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_updates": 0, 2024-04-24T08:21:44.503 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_removes": 0 2024-04-24T08:21:44.503 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.503 INFO:teuthology.orchestra.run.smithi117.stdout: "osd-slow-ops": { 2024-04-24T08:21:44.503 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_ops_count": 0 2024-04-24T08:21:44.503 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.503 INFO:teuthology.orchestra.run.smithi117.stdout: "purge_queue": { 2024-04-24T08:21:44.504 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops": 0, 2024-04-24T08:21:44.504 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T08:21:44.504 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing": 0, 2024-04-24T08:21:44.504 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_high_water": 0, 2024-04-24T08:21:44.504 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed_ops": 0, 2024-04-24T08:21:44.504 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed": 0, 2024-04-24T08:21:44.504 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_item_in_journal": 0 2024-04-24T08:21:44.504 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.505 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:21:44.505 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-04-24T08:21:44.505 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-04-24T08:21:44.505 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-04-24T08:21:44.505 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 1401, 2024-04-24T08:21:44.505 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 1927144, 2024-04-24T08:21:44.505 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:44.505 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 1401, 2024-04-24T08:21:44.505 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-04-24T08:21:44.505 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-04-24T08:21:44.506 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 1401, 2024-04-24T08:21:44.506 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 1927144, 2024-04-24T08:21:44.506 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-04-24T08:21:44.506 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.506 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.506 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.506 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:44.506 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_bytes": { 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-04-24T08:21:44.507 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 53, 2024-04-24T08:21:44.508 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 14234, 2024-04-24T08:21:44.508 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 53, 2024-04-24T08:21:44.508 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 14234, 2024-04-24T08:21:44.508 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-04-24T08:21:44.508 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.508 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.508 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.508 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:44.509 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.509 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_ops": { 2024-04-24T08:21:44.509 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-04-24T08:21:44.509 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 1024, 2024-04-24T08:21:44.509 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-04-24T08:21:44.509 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-04-24T08:21:44.509 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-04-24T08:21:44.509 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:44.509 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-04-24T08:21:44.510 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 53, 2024-04-24T08:21:44.510 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 53, 2024-04-24T08:21:44.510 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 53, 2024-04-24T08:21:44.510 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 53, 2024-04-24T08:21:44.510 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-04-24T08:21:44.510 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.510 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.510 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.511 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:44.511 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.511 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle": { 2024-04-24T08:21:44.511 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-04-24T08:21:44.511 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-04-24T08:21:44.511 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-04-24T08:21:44.511 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-04-24T08:21:44.511 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-04-24T08:21:44.511 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:44.512 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-04-24T08:21:44.512 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-04-24T08:21:44.512 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-04-24T08:21:44.512 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 0, 2024-04-24T08:21:44.512 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 0, 2024-04-24T08:21:44.512 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-04-24T08:21:44.512 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.512 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.512 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.513 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:44.513 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:44.513 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle-0x561bc9050500": { 2024-04-24T08:21:44.513 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-04-24T08:21:44.513 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-04-24T08:21:44.513 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-04-24T08:21:44.513 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 10, 2024-04-24T08:21:44.513 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 2885, 2024-04-24T08:21:44.513 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:44.514 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 10, 2024-04-24T08:21:44.514 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-04-24T08:21:44.514 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-04-24T08:21:44.514 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 9, 2024-04-24T08:21:44.514 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 2885, 2024-04-24T08:21:44.514 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-04-24T08:21:44.514 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:44.514 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:44.514 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:44.515 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:44.515 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:44.515 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-04-24T08:21:45.178 INFO:tasks.check_counter:Daemon mds.b mds.exported=0 2024-04-24T08:21:45.179 INFO:tasks.check_counter:Daemon mds.b mds.imported=0 2024-04-24T08:21:45.179 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-04-24T08:21:45.179 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=0 2024-04-24T08:21:45.179 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=0 2024-04-24T08:21:45.179 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=0 2024-04-24T08:21:45.179 DEBUG:tasks.check_counter:Getting stats from e 2024-04-24T08:21:45.179 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph daemon mds.e perf dump 2024-04-24T08:21:45.417 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:21:45.838 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:45 smithi117 ceph-mon[32438]: pgmap v355: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:21:46.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:45 smithi028 ceph-mon[27786]: pgmap v355: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:21:46.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:45 smithi194 ceph-mon[32836]: pgmap v355: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:21:47.242 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-04-24T08:21:47.242 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:21:47.242 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 0, 2024-04-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 2, 2024-04-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 0, 2024-04-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 0, 2024-04-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 4, 2024-04-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 0, 2024-04-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.000940233, 2024-04-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.000000000, 2024-04-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.000108105, 2024-04-24T08:21:47.243 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:47.244 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:47.245 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T08:21:47.245 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T08:21:47.245 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:47.245 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:21:47.245 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 6, 2024-04-24T08:21:47.245 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 180, 2024-04-24T08:21:47.245 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 3344, 2024-04-24T08:21:47.245 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 266496, 2024-04-24T08:21:47.245 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 6, 2024-04-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 3, 2024-04-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.010961645, 2024-04-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.007077388, 2024-04-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.001770738, 2024-04-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 178, 2024-04-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.021303101, 2024-04-24T08:21:47.246 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000119680 2024-04-24T08:21:47.247 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:47.247 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:47.247 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:47.247 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:47.247 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:47.247 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:47.247 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-24T08:21:47.247 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 266496 2024-04-24T08:21:47.247 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:47.247 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:21:47.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 243, 2024-04-24T08:21:47.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 231, 2024-04-24T08:21:47.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 1734144, 2024-04-24T08:21:47.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 142624, 2024-04-24T08:21:47.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 2, 2024-04-24T08:21:47.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 1, 2024-04-24T08:21:47.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.047450449, 2024-04-24T08:21:47.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.014039606, 2024-04-24T08:21:47.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.037527317, 2024-04-24T08:21:47.248 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.006031167, 2024-04-24T08:21:47.249 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:47.249 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 230, 2024-04-24T08:21:47.249 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.012109058, 2024-04-24T08:21:47.249 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000052648 2024-04-24T08:21:47.249 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:47.249 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:47.249 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:47.249 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:47.249 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:47.249 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 1734144, 2024-04-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 142624 2024-04-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-04-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 0, 2024-04-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-04-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-04-24T08:21:47.250 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:21:47.251 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-04-24T08:21:47.251 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:21:47.251 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:21:47.251 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:21:47.251 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:21:47.251 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:21:47.251 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:21:47.251 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:21:47.251 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:21:47.252 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:21:47.252 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:21:47.252 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:21:47.252 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:21:47.252 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:21:47.252 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-04-24T08:21:47.252 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:21:47.252 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-04-24T08:21:47.252 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:21:47.252 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:21:47.253 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:21:47.253 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:21:47.253 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:21:47.253 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:21:47.253 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:21:47.253 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-04-24T08:21:47.253 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:21:47.253 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:21:47.253 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:21:47.253 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-04-24T08:21:47.254 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-04-24T08:21:47.254 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-04-24T08:21:47.254 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:21:47.254 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:21:47.254 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:21:47.254 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:21:47.254 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 148933, 2024-04-24T08:21:47.254 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 30, 2024-04-24T08:21:47.254 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:21:47.255 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 0, 2024-04-24T08:21:47.255 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-04-24T08:21:47.255 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-04-24T08:21:47.255 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:21:47.255 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-04-24T08:21:47.255 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:21:47.255 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-04-24T08:21:47.255 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 0, 2024-04-24T08:21:47.255 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 0, 2024-04-24T08:21:47.256 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:21:47.256 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:21:47.256 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-04-24T08:21:47.256 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-04-24T08:21:47.256 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 928, 2024-04-24T08:21:47.256 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 2, 2024-04-24T08:21:47.256 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-04-24T08:21:47.256 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-04-24T08:21:47.256 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-04-24T08:21:47.256 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-04-24T08:21:47.257 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:47.257 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:21:47.257 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-04-24T08:21:47.257 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-04-24T08:21:47.257 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-04-24T08:21:47.257 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 245, 2024-04-24T08:21:47.257 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 1699862, 2024-04-24T08:21:47.257 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:47.257 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 245, 2024-04-24T08:21:47.258 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-04-24T08:21:47.258 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-04-24T08:21:47.258 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 245, 2024-04-24T08:21:47.258 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 1699862, 2024-04-24T08:21:47.258 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-04-24T08:21:47.258 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:47.258 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:47.258 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:47.258 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:47.259 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:47.259 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-04-24T08:21:47.878 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:47 smithi117 ceph-mon[32438]: pgmap v356: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:21:47.979 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.e 2024-04-24T08:21:47.980 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.e 2024-04-24T08:21:47.980 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-04-24T08:21:47.980 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-04-24T08:21:47.980 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-04-24T08:21:47.980 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.e 2024-04-24T08:21:47.980 DEBUG:tasks.check_counter:Getting stats from h 2024-04-24T08:21:47.980 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph daemon mds.h perf dump 2024-04-24T08:21:48.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:47 smithi028 ceph-mon[27786]: pgmap v356: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:21:48.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:47 smithi194 ceph-mon[32836]: pgmap v356: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:21:48.220 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:21:49.947 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:49 smithi117 ceph-mon[32438]: pgmap v357: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:21:49.948 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-04-24T08:21:49.948 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:21:49.948 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 719, 2024-04-24T08:21:49.948 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 721, 2024-04-24T08:21:49.948 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 249302, 2024-04-24T08:21:49.948 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 228706, 2024-04-24T08:21:49.948 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 6, 2024-04-24T08:21:49.948 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 2, 2024-04-24T08:21:49.949 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.066391050, 2024-04-24T08:21:49.949 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.026255632, 2024-04-24T08:21:49.949 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.015318864, 2024-04-24T08:21:49.949 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.017667793, 2024-04-24T08:21:49.949 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:49.949 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 719, 2024-04-24T08:21:49.949 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.028314959, 2024-04-24T08:21:49.949 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000039381 2024-04-24T08:21:49.949 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.949 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:49.950 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.950 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.950 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.950 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.950 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T08:21:49.950 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T08:21:49.950 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.950 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:21:49.951 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 892, 2024-04-24T08:21:49.951 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 876, 2024-04-24T08:21:49.951 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 1925734, 2024-04-24T08:21:49.951 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 343537, 2024-04-24T08:21:49.951 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 4, 2024-04-24T08:21:49.951 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 3, 2024-04-24T08:21:49.951 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.108961162, 2024-04-24T08:21:49.951 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.030811778, 2024-04-24T08:21:49.951 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.054216320, 2024-04-24T08:21:49.952 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.029559462, 2024-04-24T08:21:49.952 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:49.952 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 876, 2024-04-24T08:21:49.952 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.048972753, 2024-04-24T08:21:49.952 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000055904 2024-04-24T08:21:49.952 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.952 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:49.952 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.953 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.953 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.953 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.953 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 1772032, 2024-04-24T08:21:49.953 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 145264 2024-04-24T08:21:49.953 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.953 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:21:49.953 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 706, 2024-04-24T08:21:49.953 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 879, 2024-04-24T08:21:49.954 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 236864, 2024-04-24T08:21:49.954 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 694560, 2024-04-24T08:21:49.954 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 6, 2024-04-24T08:21:49.954 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 4, 2024-04-24T08:21:49.954 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.095984498, 2024-04-24T08:21:49.954 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.031490403, 2024-04-24T08:21:49.954 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.024206135, 2024-04-24T08:21:49.954 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.030217177, 2024-04-24T08:21:49.955 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:49.955 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 876, 2024-04-24T08:21:49.955 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.060025062, 2024-04-24T08:21:49.955 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000068521 2024-04-24T08:21:49.955 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.955 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:49.955 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.955 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.955 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.956 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.956 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-24T08:21:49.956 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 470688 2024-04-24T08:21:49.956 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.956 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-04-24T08:21:49.956 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 1, 2024-04-24T08:21:49.956 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-04-24T08:21:49.956 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.956 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-MDSRank": { 2024-04-24T08:21:49.956 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-04-24T08:21:49.957 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-04-24T08:21:49.957 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 3463, 2024-04-24T08:21:49.957 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.104244744, 2024-04-24T08:21:49.957 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000030102 2024-04-24T08:21:49.957 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:49.957 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.957 INFO:teuthology.orchestra.run.smithi117.stdout: "finisher-PurgeQueue": { 2024-04-24T08:21:49.957 INFO:teuthology.orchestra.run.smithi117.stdout: "queue_len": 0, 2024-04-24T08:21:49.957 INFO:teuthology.orchestra.run.smithi117.stdout: "complete_latency": { 2024-04-24T08:21:49.958 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.958 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.958 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.958 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:49.958 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.958 INFO:teuthology.orchestra.run.smithi117.stdout: "mds": { 2024-04-24T08:21:49.958 INFO:teuthology.orchestra.run.smithi117.stdout: "request": 0, 2024-04-24T08:21:49.958 INFO:teuthology.orchestra.run.smithi117.stdout: "reply": 0, 2024-04-24T08:21:49.959 INFO:teuthology.orchestra.run.smithi117.stdout: "reply_latency": { 2024-04-24T08:21:49.959 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.959 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.959 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.959 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.959 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_reply": 0, 2024-04-24T08:21:49.959 INFO:teuthology.orchestra.run.smithi117.stdout: "forward": 0, 2024-04-24T08:21:49.959 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_complete": 0, 2024-04-24T08:21:49.960 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_fetch_keys": 0, 2024-04-24T08:21:49.960 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_commit": 0, 2024-04-24T08:21:49.960 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_split": 0, 2024-04-24T08:21:49.960 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_merge": 0, 2024-04-24T08:21:49.960 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes": 22, 2024-04-24T08:21:49.960 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_top": 0, 2024-04-24T08:21:49.960 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_bottom": 22, 2024-04-24T08:21:49.960 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pin_tail": 0, 2024-04-24T08:21:49.960 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_pinned": 22, 2024-04-24T08:21:49.961 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_expired": 0, 2024-04-24T08:21:49.961 INFO:teuthology.orchestra.run.smithi117.stdout: "inodes_with_caps": 0, 2024-04-24T08:21:49.961 INFO:teuthology.orchestra.run.smithi117.stdout: "caps": 0, 2024-04-24T08:21:49.961 INFO:teuthology.orchestra.run.smithi117.stdout: "subtrees": 5, 2024-04-24T08:21:49.961 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse": 0, 2024-04-24T08:21:49.961 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_hit": 0, 2024-04-24T08:21:49.961 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_forward": 0, 2024-04-24T08:21:49.961 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_discover": 0, 2024-04-24T08:21:49.962 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_dir_fetch": 0, 2024-04-24T08:21:49.962 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_remote_ino": 0, 2024-04-24T08:21:49.962 INFO:teuthology.orchestra.run.smithi117.stdout: "traverse_lock": 0, 2024-04-24T08:21:49.962 INFO:teuthology.orchestra.run.smithi117.stdout: "load_cent": 0, 2024-04-24T08:21:49.962 INFO:teuthology.orchestra.run.smithi117.stdout: "q": 0, 2024-04-24T08:21:49.962 INFO:teuthology.orchestra.run.smithi117.stdout: "exported": 0, 2024-04-24T08:21:49.962 INFO:teuthology.orchestra.run.smithi117.stdout: "exported_inodes": 0, 2024-04-24T08:21:49.962 INFO:teuthology.orchestra.run.smithi117.stdout: "imported": 0, 2024-04-24T08:21:49.963 INFO:teuthology.orchestra.run.smithi117.stdout: "imported_inodes": 0, 2024-04-24T08:21:49.963 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_dir_fetch": 0, 2024-04-24T08:21:49.963 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_backtrace_fetch": 0, 2024-04-24T08:21:49.963 INFO:teuthology.orchestra.run.smithi117.stdout: "openino_peer_discover": 0, 2024-04-24T08:21:49.963 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rfiles": 0, 2024-04-24T08:21:49.963 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rbytes": 0, 2024-04-24T08:21:49.963 INFO:teuthology.orchestra.run.smithi117.stdout: "root_rsnaps": 0, 2024-04-24T08:21:49.964 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_fetch": 0, 2024-04-24T08:21:49.964 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_set_tag": 0, 2024-04-24T08:21:49.964 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T08:21:49.964 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_inotable_repaired": 0, 2024-04-24T08:21:49.964 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_inodes": 0, 2024-04-24T08:21:49.964 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T08:21:49.964 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_dirfrag_rstats": 0, 2024-04-24T08:21:49.964 INFO:teuthology.orchestra.run.smithi117.stdout: "scrub_file_inodes": 0, 2024-04-24T08:21:49.964 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_inode_file_caps": 0, 2024-04-24T08:21:49.965 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_revoke": 0, 2024-04-24T08:21:49.965 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_grant": 0, 2024-04-24T08:21:49.965 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T08:21:49.965 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T08:21:49.965 INFO:teuthology.orchestra.run.smithi117.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T08:21:49.965 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps": 0, 2024-04-24T08:21:49.965 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_caps_dirty": 0, 2024-04-24T08:21:49.966 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_cap_release": 0, 2024-04-24T08:21:49.966 INFO:teuthology.orchestra.run.smithi117.stdout: "process_request_cap_release": 0 2024-04-24T08:21:49.966 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.966 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_cache": { 2024-04-24T08:21:49.966 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update": 0, 2024-04-24T08:21:49.966 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_update_receipt": 0, 2024-04-24T08:21:49.966 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_try_discover": 0, 2024-04-24T08:21:49.966 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_send_discover": 0, 2024-04-24T08:21:49.967 INFO:teuthology.orchestra.run.smithi117.stdout: "dir_handle_discover": 0, 2024-04-24T08:21:49.967 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays": 0, 2024-04-24T08:21:49.967 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_delayed": 0, 2024-04-24T08:21:49.967 INFO:teuthology.orchestra.run.smithi117.stdout: "num_strays_enqueuing": 0, 2024-04-24T08:21:49.967 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_created": 0, 2024-04-24T08:21:49.967 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_enqueued": 0, 2024-04-24T08:21:49.967 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_reintegrated": 0, 2024-04-24T08:21:49.967 INFO:teuthology.orchestra.run.smithi117.stdout: "strays_migrated": 0, 2024-04-24T08:21:49.967 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_processing": 0, 2024-04-24T08:21:49.968 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_enqueued": 0, 2024-04-24T08:21:49.968 INFO:teuthology.orchestra.run.smithi117.stdout: "num_recovering_prioritized": 0, 2024-04-24T08:21:49.968 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_started": 0, 2024-04-24T08:21:49.968 INFO:teuthology.orchestra.run.smithi117.stdout: "recovery_completed": 0, 2024-04-24T08:21:49.968 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_path": 0, 2024-04-24T08:21:49.968 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_quiesce_inode": 0, 2024-04-24T08:21:49.968 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T08:21:49.968 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_exportdir": 0, 2024-04-24T08:21:49.968 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_flush": 0, 2024-04-24T08:21:49.969 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragmentdir": 0, 2024-04-24T08:21:49.969 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_fragstats": 0, 2024-04-24T08:21:49.969 INFO:teuthology.orchestra.run.smithi117.stdout: "ireq_inodestats": 0 2024-04-24T08:21:49.969 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.969 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_log": { 2024-04-24T08:21:49.969 INFO:teuthology.orchestra.run.smithi117.stdout: "evlrg": 0, 2024-04-24T08:21:49.969 INFO:teuthology.orchestra.run.smithi117.stdout: "evadd": 0, 2024-04-24T08:21:49.969 INFO:teuthology.orchestra.run.smithi117.stdout: "evex": 0, 2024-04-24T08:21:49.969 INFO:teuthology.orchestra.run.smithi117.stdout: "evtrm": 0, 2024-04-24T08:21:49.970 INFO:teuthology.orchestra.run.smithi117.stdout: "ev": 26, 2024-04-24T08:21:49.970 INFO:teuthology.orchestra.run.smithi117.stdout: "evexg": 0, 2024-04-24T08:21:49.970 INFO:teuthology.orchestra.run.smithi117.stdout: "evexd": 0, 2024-04-24T08:21:49.970 INFO:teuthology.orchestra.run.smithi117.stdout: "segadd": 0, 2024-04-24T08:21:49.970 INFO:teuthology.orchestra.run.smithi117.stdout: "segex": 0, 2024-04-24T08:21:49.970 INFO:teuthology.orchestra.run.smithi117.stdout: "segtrm": 0, 2024-04-24T08:21:49.970 INFO:teuthology.orchestra.run.smithi117.stdout: "seg": 2, 2024-04-24T08:21:49.970 INFO:teuthology.orchestra.run.smithi117.stdout: "segmjr": 2, 2024-04-24T08:21:49.970 INFO:teuthology.orchestra.run.smithi117.stdout: "segexg": 0, 2024-04-24T08:21:49.971 INFO:teuthology.orchestra.run.smithi117.stdout: "segexd": 0, 2024-04-24T08:21:49.971 INFO:teuthology.orchestra.run.smithi117.stdout: "expos": 4194304, 2024-04-24T08:21:49.971 INFO:teuthology.orchestra.run.smithi117.stdout: "wrpos": 4254781, 2024-04-24T08:21:49.971 INFO:teuthology.orchestra.run.smithi117.stdout: "rdpos": 4254207, 2024-04-24T08:21:49.971 INFO:teuthology.orchestra.run.smithi117.stdout: "jlat": { 2024-04-24T08:21:49.971 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.971 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.971 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.972 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.972 INFO:teuthology.orchestra.run.smithi117.stdout: "replayed": 26 2024-04-24T08:21:49.972 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.972 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_mem": { 2024-04-24T08:21:49.972 INFO:teuthology.orchestra.run.smithi117.stdout: "ino": 21, 2024-04-24T08:21:49.972 INFO:teuthology.orchestra.run.smithi117.stdout: "ino+": 21, 2024-04-24T08:21:49.972 INFO:teuthology.orchestra.run.smithi117.stdout: "ino-": 0, 2024-04-24T08:21:49.972 INFO:teuthology.orchestra.run.smithi117.stdout: "dir": 16, 2024-04-24T08:21:49.972 INFO:teuthology.orchestra.run.smithi117.stdout: "dir+": 16, 2024-04-24T08:21:49.972 INFO:teuthology.orchestra.run.smithi117.stdout: "dir-": 0, 2024-04-24T08:21:49.973 INFO:teuthology.orchestra.run.smithi117.stdout: "dn": 22, 2024-04-24T08:21:49.973 INFO:teuthology.orchestra.run.smithi117.stdout: "dn+": 22, 2024-04-24T08:21:49.973 INFO:teuthology.orchestra.run.smithi117.stdout: "dn-": 0, 2024-04-24T08:21:49.973 INFO:teuthology.orchestra.run.smithi117.stdout: "cap": 0, 2024-04-24T08:21:49.973 INFO:teuthology.orchestra.run.smithi117.stdout: "cap+": 0, 2024-04-24T08:21:49.973 INFO:teuthology.orchestra.run.smithi117.stdout: "cap-": 0, 2024-04-24T08:21:49.973 INFO:teuthology.orchestra.run.smithi117.stdout: "rss": 52044, 2024-04-24T08:21:49.973 INFO:teuthology.orchestra.run.smithi117.stdout: "heap": 215416 2024-04-24T08:21:49.974 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.974 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_server": { 2024-04-24T08:21:49.974 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_client_request": 0, 2024-04-24T08:21:49.974 INFO:teuthology.orchestra.run.smithi117.stdout: "dispatch_server_request": 0, 2024-04-24T08:21:49.974 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_request": 0, 2024-04-24T08:21:49.974 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_client_session": 0, 2024-04-24T08:21:49.974 INFO:teuthology.orchestra.run.smithi117.stdout: "handle_peer_request": 0, 2024-04-24T08:21:49.974 INFO:teuthology.orchestra.run.smithi117.stdout: "req_create_latency": { 2024-04-24T08:21:49.975 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.975 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.975 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.975 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.975 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getattr_latency": { 2024-04-24T08:21:49.975 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.975 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.975 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.976 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.976 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getfilelock_latency": { 2024-04-24T08:21:49.976 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.976 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.976 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.976 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.976 INFO:teuthology.orchestra.run.smithi117.stdout: "req_link_latency": { 2024-04-24T08:21:49.976 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.977 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.977 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.977 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.977 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookup_latency": { 2024-04-24T08:21:49.977 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.977 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.977 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.977 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.978 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookuphash_latency": { 2024-04-24T08:21:49.978 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.978 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.978 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.978 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.978 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupino_latency": { 2024-04-24T08:21:49.978 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.978 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.979 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.979 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.979 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupname_latency": { 2024-04-24T08:21:49.979 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.979 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.979 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.979 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.979 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupparent_latency": { 2024-04-24T08:21:49.980 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.980 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.980 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.980 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.980 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lookupsnap_latency": { 2024-04-24T08:21:49.980 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.980 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.980 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.981 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.981 INFO:teuthology.orchestra.run.smithi117.stdout: "req_lssnap_latency": { 2024-04-24T08:21:49.981 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.981 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.981 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.981 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.981 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mkdir_latency": { 2024-04-24T08:21:49.981 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.982 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.982 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.982 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.983 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mknod_latency": { 2024-04-24T08:21:49.983 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.983 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.983 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.983 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.983 INFO:teuthology.orchestra.run.smithi117.stdout: "req_mksnap_latency": { 2024-04-24T08:21:49.983 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.983 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.984 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.984 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.984 INFO:teuthology.orchestra.run.smithi117.stdout: "req_open_latency": { 2024-04-24T08:21:49.984 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.984 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.984 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.984 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.984 INFO:teuthology.orchestra.run.smithi117.stdout: "req_readdir_latency": { 2024-04-24T08:21:49.984 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.985 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.985 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.985 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.985 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rename_latency": { 2024-04-24T08:21:49.985 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.985 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.985 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.985 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.985 INFO:teuthology.orchestra.run.smithi117.stdout: "req_renamesnap_latency": { 2024-04-24T08:21:49.986 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.986 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.986 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.986 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.986 INFO:teuthology.orchestra.run.smithi117.stdout: "req_snapdiff_latency": { 2024-04-24T08:21:49.986 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.986 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.986 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.987 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.987 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmdir_latency": { 2024-04-24T08:21:49.987 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.987 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.987 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.987 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.987 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmsnap_latency": { 2024-04-24T08:21:49.987 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.987 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.988 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.988 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.988 INFO:teuthology.orchestra.run.smithi117.stdout: "req_rmxattr_latency": { 2024-04-24T08:21:49.988 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.988 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.988 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.988 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.988 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setattr_latency": { 2024-04-24T08:21:49.988 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.989 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.989 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.989 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.989 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setdirlayout_latency": { 2024-04-24T08:21:49.989 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.989 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.989 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.989 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.989 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setfilelock_latency": { 2024-04-24T08:21:49.989 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.990 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.990 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.990 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.990 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setlayout_latency": { 2024-04-24T08:21:49.990 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.990 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.990 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.990 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.991 INFO:teuthology.orchestra.run.smithi117.stdout: "req_setxattr_latency": { 2024-04-24T08:21:49.991 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.991 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.991 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.991 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.991 INFO:teuthology.orchestra.run.smithi117.stdout: "req_symlink_latency": { 2024-04-24T08:21:49.991 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.991 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.991 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.992 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.992 INFO:teuthology.orchestra.run.smithi117.stdout: "req_unlink_latency": { 2024-04-24T08:21:49.992 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.992 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.992 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.992 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.992 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_revoke_eviction": 0, 2024-04-24T08:21:49.992 INFO:teuthology.orchestra.run.smithi117.stdout: "cap_acquisition_throttle": 0, 2024-04-24T08:21:49.992 INFO:teuthology.orchestra.run.smithi117.stdout: "req_getvxattr_latency": { 2024-04-24T08:21:49.992 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:49.993 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:49.993 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:49.993 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:49.993 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.993 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_sessions": { 2024-04-24T08:21:49.993 INFO:teuthology.orchestra.run.smithi117.stdout: "session_count": 2, 2024-04-24T08:21:49.993 INFO:teuthology.orchestra.run.smithi117.stdout: "session_add": 5, 2024-04-24T08:21:49.993 INFO:teuthology.orchestra.run.smithi117.stdout: "session_remove": 3, 2024-04-24T08:21:49.993 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_open": 2, 2024-04-24T08:21:49.994 INFO:teuthology.orchestra.run.smithi117.stdout: "sessions_stale": 0, 2024-04-24T08:21:49.994 INFO:teuthology.orchestra.run.smithi117.stdout: "total_load": 0, 2024-04-24T08:21:49.994 INFO:teuthology.orchestra.run.smithi117.stdout: "average_load": 0, 2024-04-24T08:21:49.994 INFO:teuthology.orchestra.run.smithi117.stdout: "avg_session_uptime": 5145, 2024-04-24T08:21:49.994 INFO:teuthology.orchestra.run.smithi117.stdout: "mdthresh_evicted": 0 2024-04-24T08:21:49.994 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:49.994 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-04-24T08:21:49.994 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:21:49.995 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-04-24T08:21:49.995 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:21:49.995 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:21:49.995 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:21:49.995 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:21:49.995 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:21:49.995 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:21:49.995 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:21:49.995 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:21:49.996 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:21:49.996 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:21:49.996 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:21:49.996 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:21:49.996 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:21:49.996 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-04-24T08:21:49.996 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:21:49.996 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-04-24T08:21:49.997 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:21:49.997 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:21:49.997 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:21:49.997 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:21:49.997 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:21:49.997 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:21:49.997 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:21:49.997 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-04-24T08:21:49.997 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:21:49.997 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:21:49.998 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:21:49.998 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-04-24T08:21:49.998 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-04-24T08:21:49.998 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-04-24T08:21:49.998 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:21:49.998 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:21:49.998 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:21:49.998 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:21:49.998 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 190000, 2024-04-24T08:21:49.999 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 68, 2024-04-24T08:21:49.999 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:21:49.999 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 0, 2024-04-24T08:21:49.999 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-04-24T08:21:49.999 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-04-24T08:21:49.999 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:21:49.999 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-04-24T08:21:49.999 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:21:50.000 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-04-24T08:21:50.000 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 8320, 2024-04-24T08:21:50.000 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 173, 2024-04-24T08:21:50.000 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:21:50.000 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:21:50.000 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-04-24T08:21:50.000 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-04-24T08:21:50.000 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 93168, 2024-04-24T08:21:50.001 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 1011, 2024-04-24T08:21:50.001 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-04-24T08:21:50.001 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-04-24T08:21:50.001 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-04-24T08:21:50.001 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-04-24T08:21:50.001 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:50.001 INFO:teuthology.orchestra.run.smithi117.stdout: "objecter": { 2024-04-24T08:21:50.001 INFO:teuthology.orchestra.run.smithi117.stdout: "op_active": 0, 2024-04-24T08:21:50.002 INFO:teuthology.orchestra.run.smithi117.stdout: "op_laggy": 0, 2024-04-24T08:21:50.002 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send": 2046, 2024-04-24T08:21:50.002 INFO:teuthology.orchestra.run.smithi117.stdout: "op_send_bytes": 21, 2024-04-24T08:21:50.002 INFO:teuthology.orchestra.run.smithi117.stdout: "op_resend": 3, 2024-04-24T08:21:50.002 INFO:teuthology.orchestra.run.smithi117.stdout: "op_reply": 2046, 2024-04-24T08:21:50.002 INFO:teuthology.orchestra.run.smithi117.stdout: "op_latency": { 2024-04-24T08:21:50.002 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2046, 2024-04-24T08:21:50.002 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 130.768396959, 2024-04-24T08:21:50.003 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.063914172 2024-04-24T08:21:50.003 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:50.003 INFO:teuthology.orchestra.run.smithi117.stdout: "op_inflight": 0, 2024-04-24T08:21:50.003 INFO:teuthology.orchestra.run.smithi117.stdout: "oplen_avg": { 2024-04-24T08:21:50.003 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 2046, 2024-04-24T08:21:50.003 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 2047 2024-04-24T08:21:50.003 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:50.003 INFO:teuthology.orchestra.run.smithi117.stdout: "op": 2046, 2024-04-24T08:21:50.004 INFO:teuthology.orchestra.run.smithi117.stdout: "op_r": 2046, 2024-04-24T08:21:50.004 INFO:teuthology.orchestra.run.smithi117.stdout: "op_w": 0, 2024-04-24T08:21:50.004 INFO:teuthology.orchestra.run.smithi117.stdout: "op_rmw": 0, 2024-04-24T08:21:50.004 INFO:teuthology.orchestra.run.smithi117.stdout: "op_pg": 0, 2024-04-24T08:21:50.004 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_stat": 1325, 2024-04-24T08:21:50.004 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_create": 0, 2024-04-24T08:21:50.004 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_read": 719, 2024-04-24T08:21:50.004 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_write": 0, 2024-04-24T08:21:50.004 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writefull": 0, 2024-04-24T08:21:50.005 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_writesame": 0, 2024-04-24T08:21:50.005 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_append": 0, 2024-04-24T08:21:50.005 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_zero": 0, 2024-04-24T08:21:50.005 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_truncate": 0, 2024-04-24T08:21:50.005 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_delete": 0, 2024-04-24T08:21:50.005 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_mapext": 0, 2024-04-24T08:21:50.005 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_sparse_read": 0, 2024-04-24T08:21:50.005 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_clonerange": 0, 2024-04-24T08:21:50.006 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_getxattr": 1, 2024-04-24T08:21:50.006 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_setxattr": 0, 2024-04-24T08:21:50.006 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_cmpxattr": 0, 2024-04-24T08:21:50.006 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_rmxattr": 0, 2024-04-24T08:21:50.006 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_resetxattrs": 0, 2024-04-24T08:21:50.006 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_call": 0, 2024-04-24T08:21:50.006 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_watch": 0, 2024-04-24T08:21:50.007 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_notify": 0, 2024-04-24T08:21:50.007 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T08:21:50.007 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls": 0, 2024-04-24T08:21:50.007 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_pgls_filter": 0, 2024-04-24T08:21:50.007 INFO:teuthology.orchestra.run.smithi117.stdout: "osdop_other": 0, 2024-04-24T08:21:50.007 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_active": 0, 2024-04-24T08:21:50.007 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_send": 0, 2024-04-24T08:21:50.007 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_resend": 0, 2024-04-24T08:21:50.007 INFO:teuthology.orchestra.run.smithi117.stdout: "linger_ping": 0, 2024-04-24T08:21:50.008 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_active": 0, 2024-04-24T08:21:50.008 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_send": 0, 2024-04-24T08:21:50.008 INFO:teuthology.orchestra.run.smithi117.stdout: "poolop_resend": 0, 2024-04-24T08:21:50.008 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_active": 0, 2024-04-24T08:21:50.008 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_send": 0, 2024-04-24T08:21:50.008 INFO:teuthology.orchestra.run.smithi117.stdout: "poolstat_resend": 0, 2024-04-24T08:21:50.008 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_active": 0, 2024-04-24T08:21:50.009 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_send": 0, 2024-04-24T08:21:50.009 INFO:teuthology.orchestra.run.smithi117.stdout: "statfs_resend": 0, 2024-04-24T08:21:50.009 INFO:teuthology.orchestra.run.smithi117.stdout: "command_active": 0, 2024-04-24T08:21:50.009 INFO:teuthology.orchestra.run.smithi117.stdout: "command_send": 0, 2024-04-24T08:21:50.009 INFO:teuthology.orchestra.run.smithi117.stdout: "command_resend": 0, 2024-04-24T08:21:50.009 INFO:teuthology.orchestra.run.smithi117.stdout: "map_epoch": 101, 2024-04-24T08:21:50.009 INFO:teuthology.orchestra.run.smithi117.stdout: "map_full": 0, 2024-04-24T08:21:50.009 INFO:teuthology.orchestra.run.smithi117.stdout: "map_inc": 15, 2024-04-24T08:21:50.009 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_sessions": 5, 2024-04-24T08:21:50.010 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_open": 5, 2024-04-24T08:21:50.010 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_session_close": 0, 2024-04-24T08:21:50.010 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_laggy": 0, 2024-04-24T08:21:50.010 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_wr": 0, 2024-04-24T08:21:50.010 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_rd": 2, 2024-04-24T08:21:50.010 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_del": 0 2024-04-24T08:21:50.010 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:50.010 INFO:teuthology.orchestra.run.smithi117.stdout: "oft": { 2024-04-24T08:21:50.011 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_objs": 0, 2024-04-24T08:21:50.011 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_kv_pairs": 0, 2024-04-24T08:21:50.011 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_updates": 0, 2024-04-24T08:21:50.011 INFO:teuthology.orchestra.run.smithi117.stdout: "omap_total_removes": 0 2024-04-24T08:21:50.011 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:50.011 INFO:teuthology.orchestra.run.smithi117.stdout: "osd-slow-ops": { 2024-04-24T08:21:50.011 INFO:teuthology.orchestra.run.smithi117.stdout: "slow_ops_count": 0 2024-04-24T08:21:50.011 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:50.011 INFO:teuthology.orchestra.run.smithi117.stdout: "purge_queue": { 2024-04-24T08:21:50.012 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops": 0, 2024-04-24T08:21:50.012 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T08:21:50.012 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing": 0, 2024-04-24T08:21:50.012 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executing_high_water": 0, 2024-04-24T08:21:50.012 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed_ops": 0, 2024-04-24T08:21:50.012 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_executed": 0, 2024-04-24T08:21:50.012 INFO:teuthology.orchestra.run.smithi117.stdout: "pq_item_in_journal": 0 2024-04-24T08:21:50.012 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:50.013 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:21:50.013 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-04-24T08:21:50.013 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-04-24T08:21:50.013 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-04-24T08:21:50.013 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 2313, 2024-04-24T08:21:50.013 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 2187167, 2024-04-24T08:21:50.013 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:50.013 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 2313, 2024-04-24T08:21:50.013 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-04-24T08:21:50.014 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-04-24T08:21:50.014 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 2313, 2024-04-24T08:21:50.014 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 2187167, 2024-04-24T08:21:50.014 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-04-24T08:21:50.014 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:50.014 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:50.014 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:50.014 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:50.015 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:50.015 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_bytes": { 2024-04-24T08:21:50.015 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-04-24T08:21:50.015 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-04-24T08:21:50.015 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-04-24T08:21:50.015 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-04-24T08:21:50.015 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-04-24T08:21:50.015 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:50.016 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-04-24T08:21:50.016 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 2046, 2024-04-24T08:21:50.016 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 60482, 2024-04-24T08:21:50.016 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 2046, 2024-04-24T08:21:50.016 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 60482, 2024-04-24T08:21:50.016 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-04-24T08:21:50.016 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:50.016 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:50.017 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:50.017 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:50.017 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:50.017 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-objecter_ops": { 2024-04-24T08:21:50.017 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-04-24T08:21:50.017 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 1024, 2024-04-24T08:21:50.017 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-04-24T08:21:50.017 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-04-24T08:21:50.017 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-04-24T08:21:50.018 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:50.018 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-04-24T08:21:50.018 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 2046, 2024-04-24T08:21:50.018 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 2046, 2024-04-24T08:21:50.018 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 2046, 2024-04-24T08:21:50.018 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 2046, 2024-04-24T08:21:50.018 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-04-24T08:21:50.018 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:50.019 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:50.019 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:50.019 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:50.019 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:50.019 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle": { 2024-04-24T08:21:50.019 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-04-24T08:21:50.019 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-04-24T08:21:50.019 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-04-24T08:21:50.020 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-04-24T08:21:50.020 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-04-24T08:21:50.020 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:50.020 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-04-24T08:21:50.020 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-04-24T08:21:50.020 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-04-24T08:21:50.020 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 0, 2024-04-24T08:21:50.020 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 0, 2024-04-24T08:21:50.021 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-04-24T08:21:50.021 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:50.021 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:50.021 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:50.021 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:50.021 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:50.021 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-write_buf_throttle-0x558b656066e0": { 2024-04-24T08:21:50.022 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-04-24T08:21:50.022 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 3758096384, 2024-04-24T08:21:50.022 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-04-24T08:21:50.022 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 0, 2024-04-24T08:21:50.022 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 0, 2024-04-24T08:21:50.022 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:50.022 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 0, 2024-04-24T08:21:50.022 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-04-24T08:21:50.023 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-04-24T08:21:50.023 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 0, 2024-04-24T08:21:50.023 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 0, 2024-04-24T08:21:50.023 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-04-24T08:21:50.023 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:50.023 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:50.023 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:50.023 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:50.024 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:50.024 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-04-24T08:21:50.029 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:49 smithi194 ceph-mon[32836]: pgmap v357: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:21:50.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:49 smithi028 ceph-mon[27786]: pgmap v357: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:21:50.764 INFO:tasks.check_counter:Daemon mds.h mds.exported=0 2024-04-24T08:21:50.765 INFO:tasks.check_counter:Daemon mds.h mds.imported=0 2024-04-24T08:21:50.765 INFO:tasks.check_counter:Daemon mds.h mds.root_rsnaps=0 2024-04-24T08:21:50.765 INFO:tasks.check_counter:Daemon mds.h mds_server.req_mksnap_latency.avgcount=0 2024-04-24T08:21:50.765 INFO:tasks.check_counter:Daemon mds.h mds_server.req_rmsnap_latency.avgcount=0 2024-04-24T08:21:50.765 INFO:tasks.check_counter:Daemon mds.h mds.dir_split=0 2024-04-24T08:21:50.765 DEBUG:tasks.check_counter:Getting stats from k 2024-04-24T08:21:50.765 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph daemon mds.k perf dump 2024-04-24T08:21:51.001 INFO:teuthology.orchestra.run.smithi117.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.b/config 2024-04-24T08:21:52.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:51 smithi028 ceph-mon[27786]: pgmap v358: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:21:52.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:51 smithi117 ceph-mon[32438]: pgmap v358: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:21:52.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:51 smithi194 ceph-mon[32836]: pgmap v358: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:21:52.758 INFO:teuthology.orchestra.run.smithi117.stdout:{ 2024-04-24T08:21:52.758 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:21:52.759 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 0, 2024-04-24T08:21:52.759 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 2, 2024-04-24T08:21:52.759 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 0, 2024-04-24T08:21:52.759 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 0, 2024-04-24T08:21:52.759 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 4, 2024-04-24T08:21:52.759 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 0, 2024-04-24T08:21:52.759 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.000973544, 2024-04-24T08:21:52.759 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.000000000, 2024-04-24T08:21:52.760 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.000206052, 2024-04-24T08:21:52.760 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:21:52.760 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:52.760 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:52.760 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:52.760 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:52.760 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:52.761 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:52.761 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:52.761 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:52.761 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:52.761 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:52.761 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T08:21:52.761 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T08:21:52.761 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:52.761 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:21:52.762 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 247, 2024-04-24T08:21:52.762 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 235, 2024-04-24T08:21:52.762 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 1735232, 2024-04-24T08:21:52.762 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 145184, 2024-04-24T08:21:52.762 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 2, 2024-04-24T08:21:52.762 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 1, 2024-04-24T08:21:52.762 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.049049025, 2024-04-24T08:21:52.762 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.013522745, 2024-04-24T08:21:52.763 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.039960506, 2024-04-24T08:21:52.763 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.006079918, 2024-04-24T08:21:52.763 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:52.763 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 234, 2024-04-24T08:21:52.763 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.012325284, 2024-04-24T08:21:52.763 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000052672 2024-04-24T08:21:52.763 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:52.763 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:52.763 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:52.764 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:52.764 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:52.764 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:52.764 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 1735232, 2024-04-24T08:21:52.764 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 145184 2024-04-24T08:21:52.764 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:52.764 INFO:teuthology.orchestra.run.smithi117.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:21:52.764 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_messages": 6, 2024-04-24T08:21:52.765 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages": 182, 2024-04-24T08:21:52.765 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_bytes": 3344, 2024-04-24T08:21:52.765 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_bytes": 188432, 2024-04-24T08:21:52.765 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_created_connections": 5, 2024-04-24T08:21:52.765 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_active_connections": 3, 2024-04-24T08:21:52.765 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_total_time": 0.011435823, 2024-04-24T08:21:52.765 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_send_time": 0.007594089, 2024-04-24T08:21:52.765 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_recv_time": 0.001692608, 2024-04-24T08:21:52.765 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:21:52.766 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:52.766 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 180, 2024-04-24T08:21:52.766 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.019244179, 2024-04-24T08:21:52.766 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000106912 2024-04-24T08:21:52.766 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:52.766 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:52.766 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:52.767 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:52.767 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:52.767 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:52.767 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-24T08:21:52.767 INFO:teuthology.orchestra.run.smithi117.stdout: "msgr_send_encrypted_bytes": 188432 2024-04-24T08:21:52.767 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:52.767 INFO:teuthology.orchestra.run.smithi117.stdout: "cct": { 2024-04-24T08:21:52.767 INFO:teuthology.orchestra.run.smithi117.stdout: "total_workers": 0, 2024-04-24T08:21:52.767 INFO:teuthology.orchestra.run.smithi117.stdout: "unhealthy_workers": 0 2024-04-24T08:21:52.768 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:52.768 INFO:teuthology.orchestra.run.smithi117.stdout: "mempool": { 2024-04-24T08:21:52.768 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:21:52.768 INFO:teuthology.orchestra.run.smithi117.stdout: "bloom_filter_items": 0, 2024-04-24T08:21:52.768 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:21:52.768 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:21:52.768 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:21:52.768 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:21:52.768 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:21:52.769 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:21:52.769 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:21:52.769 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:21:52.769 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:21:52.769 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:21:52.769 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:21:52.769 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:21:52.769 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:21:52.769 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_extent_items": 0, 2024-04-24T08:21:52.770 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:21:52.770 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_blob_items": 0, 2024-04-24T08:21:52.770 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:21:52.770 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:21:52.770 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:21:52.770 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:21:52.770 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:21:52.770 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:21:52.770 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:21:52.771 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_txc_items": 0, 2024-04-24T08:21:52.771 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:21:52.771 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:21:52.771 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:21:52.771 INFO:teuthology.orchestra.run.smithi117.stdout: "bluestore_writing_items": 0, 2024-04-24T08:21:52.771 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_bytes": 0, 2024-04-24T08:21:52.771 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_items": 0, 2024-04-24T08:21:52.771 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:21:52.772 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:21:52.772 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:21:52.772 INFO:teuthology.orchestra.run.smithi117.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:21:52.772 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_bytes": 148692, 2024-04-24T08:21:52.772 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_anon_items": 30, 2024-04-24T08:21:52.772 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:21:52.772 INFO:teuthology.orchestra.run.smithi117.stdout: "buffer_meta_items": 0, 2024-04-24T08:21:52.772 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_bytes": 0, 2024-04-24T08:21:52.772 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_items": 0, 2024-04-24T08:21:52.773 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:21:52.773 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_mapbl_items": 0, 2024-04-24T08:21:52.773 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:21:52.773 INFO:teuthology.orchestra.run.smithi117.stdout: "osd_pglog_items": 0, 2024-04-24T08:21:52.773 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_bytes": 0, 2024-04-24T08:21:52.773 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_items": 0, 2024-04-24T08:21:52.773 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:21:52.773 INFO:teuthology.orchestra.run.smithi117.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:21:52.774 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_bytes": 0, 2024-04-24T08:21:52.774 INFO:teuthology.orchestra.run.smithi117.stdout: "pgmap_items": 0, 2024-04-24T08:21:52.774 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_bytes": 928, 2024-04-24T08:21:52.774 INFO:teuthology.orchestra.run.smithi117.stdout: "mds_co_items": 2, 2024-04-24T08:21:52.774 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_bytes": 0, 2024-04-24T08:21:52.774 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_1_items": 0, 2024-04-24T08:21:52.774 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_bytes": 0, 2024-04-24T08:21:52.775 INFO:teuthology.orchestra.run.smithi117.stdout: "unittest_2_items": 0 2024-04-24T08:21:52.775 INFO:teuthology.orchestra.run.smithi117.stdout: }, 2024-04-24T08:21:52.775 INFO:teuthology.orchestra.run.smithi117.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:21:52.775 INFO:teuthology.orchestra.run.smithi117.stdout: "val": 0, 2024-04-24T08:21:52.775 INFO:teuthology.orchestra.run.smithi117.stdout: "max": 104857600, 2024-04-24T08:21:52.775 INFO:teuthology.orchestra.run.smithi117.stdout: "get_started": 0, 2024-04-24T08:21:52.775 INFO:teuthology.orchestra.run.smithi117.stdout: "get": 249, 2024-04-24T08:21:52.775 INFO:teuthology.orchestra.run.smithi117.stdout: "get_sum": 1700382, 2024-04-24T08:21:52.775 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:52.776 INFO:teuthology.orchestra.run.smithi117.stdout: "get_or_fail_success": 249, 2024-04-24T08:21:52.776 INFO:teuthology.orchestra.run.smithi117.stdout: "take": 0, 2024-04-24T08:21:52.776 INFO:teuthology.orchestra.run.smithi117.stdout: "take_sum": 0, 2024-04-24T08:21:52.776 INFO:teuthology.orchestra.run.smithi117.stdout: "put": 249, 2024-04-24T08:21:52.776 INFO:teuthology.orchestra.run.smithi117.stdout: "put_sum": 1700382, 2024-04-24T08:21:52.776 INFO:teuthology.orchestra.run.smithi117.stdout: "wait": { 2024-04-24T08:21:52.776 INFO:teuthology.orchestra.run.smithi117.stdout: "avgcount": 0, 2024-04-24T08:21:52.776 INFO:teuthology.orchestra.run.smithi117.stdout: "sum": 0.000000000, 2024-04-24T08:21:52.776 INFO:teuthology.orchestra.run.smithi117.stdout: "avgtime": 0.000000000 2024-04-24T08:21:52.777 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:52.777 INFO:teuthology.orchestra.run.smithi117.stdout: } 2024-04-24T08:21:52.777 INFO:teuthology.orchestra.run.smithi117.stdout:} 2024-04-24T08:21:53.464 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-04-24T08:21:53.464 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-04-24T08:21:53.465 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-04-24T08:21:53.465 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-04-24T08:21:53.465 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-04-24T08:21:53.465 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-04-24T08:21:53.465 DEBUG:tasks.check_counter:Getting stats from c 2024-04-24T08:21:53.465 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph daemon mds.c perf dump 2024-04-24T08:21:53.757 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:21:53.879 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:53 smithi194 ceph-mon[32836]: pgmap v359: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:21:54.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:53 smithi028 ceph-mon[27786]: pgmap v359: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:21:54.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:53 smithi117 ceph-mon[32438]: pgmap v359: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:21:55.523 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-24T08:21:55.524 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:21:55.524 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 1498, 2024-04-24T08:21:55.524 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 1499, 2024-04-24T08:21:55.524 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 12681122, 2024-04-24T08:21:55.524 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 476056, 2024-04-24T08:21:55.524 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 7, 2024-04-24T08:21:55.524 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 3, 2024-04-24T08:21:55.524 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.157419541, 2024-04-24T08:21:55.525 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.046107790, 2024-04-24T08:21:55.525 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.049073326, 2024-04-24T08:21:55.525 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.048352966, 2024-04-24T08:21:55.525 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:55.525 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1497, 2024-04-24T08:21:55.525 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.059106186, 2024-04-24T08:21:55.525 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000039483 2024-04-24T08:21:55.525 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.525 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:55.525 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.526 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.526 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.526 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.526 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T08:21:55.526 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T08:21:55.526 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.526 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:21:55.526 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 949, 2024-04-24T08:21:55.527 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 934, 2024-04-24T08:21:55.527 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 18732436, 2024-04-24T08:21:55.527 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 363198, 2024-04-24T08:21:55.527 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 5, 2024-04-24T08:21:55.527 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 4, 2024-04-24T08:21:55.527 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.119839973, 2024-04-24T08:21:55.527 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.034297871, 2024-04-24T08:21:55.527 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.065649464, 2024-04-24T08:21:55.528 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.027211067, 2024-04-24T08:21:55.528 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:55.528 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 933, 2024-04-24T08:21:55.528 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.049608828, 2024-04-24T08:21:55.528 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000053171 2024-04-24T08:21:55.528 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.528 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:55.528 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.528 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.529 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.529 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.529 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 1769712, 2024-04-24T08:21:55.529 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 146256 2024-04-24T08:21:55.529 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.529 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:21:55.529 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 398, 2024-04-24T08:21:55.529 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 571, 2024-04-24T08:21:55.529 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 8489591, 2024-04-24T08:21:55.530 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 599377, 2024-04-24T08:21:55.530 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 7, 2024-04-24T08:21:55.530 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 5, 2024-04-24T08:21:55.530 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.052200379, 2024-04-24T08:21:55.530 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.019356264, 2024-04-24T08:21:55.530 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.015761993, 2024-04-24T08:21:55.530 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.012158290, 2024-04-24T08:21:55.530 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:55.531 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 569, 2024-04-24T08:21:55.531 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.071950904, 2024-04-24T08:21:55.531 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000126451 2024-04-24T08:21:55.531 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.531 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:55.531 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.531 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.531 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.531 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.532 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-04-24T08:21:55.532 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 474720 2024-04-24T08:21:55.532 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.532 INFO:teuthology.orchestra.run.smithi194.stdout: "cct": { 2024-04-24T08:21:55.532 INFO:teuthology.orchestra.run.smithi194.stdout: "total_workers": 1, 2024-04-24T08:21:55.532 INFO:teuthology.orchestra.run.smithi194.stdout: "unhealthy_workers": 0 2024-04-24T08:21:55.532 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.532 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-MDSRank": { 2024-04-24T08:21:55.533 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-04-24T08:21:55.533 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-04-24T08:21:55.533 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 4301, 2024-04-24T08:21:55.533 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.116184384, 2024-04-24T08:21:55.533 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000027013 2024-04-24T08:21:55.533 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:55.533 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.533 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-PurgeQueue": { 2024-04-24T08:21:55.533 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-04-24T08:21:55.534 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-04-24T08:21:55.534 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.534 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.534 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.534 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:55.534 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.534 INFO:teuthology.orchestra.run.smithi194.stdout: "mds": { 2024-04-24T08:21:55.535 INFO:teuthology.orchestra.run.smithi194.stdout: "request": 0, 2024-04-24T08:21:55.535 INFO:teuthology.orchestra.run.smithi194.stdout: "reply": 0, 2024-04-24T08:21:55.535 INFO:teuthology.orchestra.run.smithi194.stdout: "reply_latency": { 2024-04-24T08:21:55.535 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.535 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.535 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.535 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.536 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_reply": 0, 2024-04-24T08:21:55.536 INFO:teuthology.orchestra.run.smithi194.stdout: "forward": 0, 2024-04-24T08:21:55.536 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_complete": 0, 2024-04-24T08:21:55.536 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_keys": 0, 2024-04-24T08:21:55.536 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_commit": 0, 2024-04-24T08:21:55.536 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_split": 0, 2024-04-24T08:21:55.536 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_merge": 0, 2024-04-24T08:21:55.536 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes": 1597, 2024-04-24T08:21:55.537 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_top": 0, 2024-04-24T08:21:55.537 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_bottom": 1597, 2024-04-24T08:21:55.537 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pin_tail": 0, 2024-04-24T08:21:55.537 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pinned": 1597, 2024-04-24T08:21:55.537 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_expired": 0, 2024-04-24T08:21:55.537 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_with_caps": 0, 2024-04-24T08:21:55.537 INFO:teuthology.orchestra.run.smithi194.stdout: "caps": 0, 2024-04-24T08:21:55.537 INFO:teuthology.orchestra.run.smithi194.stdout: "subtrees": 5, 2024-04-24T08:21:55.537 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse": 0, 2024-04-24T08:21:55.537 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_hit": 0, 2024-04-24T08:21:55.538 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_forward": 0, 2024-04-24T08:21:55.538 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_discover": 0, 2024-04-24T08:21:55.538 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_dir_fetch": 0, 2024-04-24T08:21:55.538 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_remote_ino": 0, 2024-04-24T08:21:55.538 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_lock": 0, 2024-04-24T08:21:55.538 INFO:teuthology.orchestra.run.smithi194.stdout: "load_cent": 0, 2024-04-24T08:21:55.538 INFO:teuthology.orchestra.run.smithi194.stdout: "q": 0, 2024-04-24T08:21:55.538 INFO:teuthology.orchestra.run.smithi194.stdout: "exported": 0, 2024-04-24T08:21:55.539 INFO:teuthology.orchestra.run.smithi194.stdout: "exported_inodes": 0, 2024-04-24T08:21:55.539 INFO:teuthology.orchestra.run.smithi194.stdout: "imported": 0, 2024-04-24T08:21:55.539 INFO:teuthology.orchestra.run.smithi194.stdout: "imported_inodes": 0, 2024-04-24T08:21:55.539 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_dir_fetch": 0, 2024-04-24T08:21:55.539 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_backtrace_fetch": 0, 2024-04-24T08:21:55.539 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_peer_discover": 0, 2024-04-24T08:21:55.539 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rfiles": 325, 2024-04-24T08:21:55.539 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rbytes": 13424230511, 2024-04-24T08:21:55.540 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rsnaps": 6, 2024-04-24T08:21:55.540 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_fetch": 0, 2024-04-24T08:21:55.540 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_set_tag": 0, 2024-04-24T08:21:55.540 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T08:21:55.540 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_inotable_repaired": 0, 2024-04-24T08:21:55.540 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_inodes": 0, 2024-04-24T08:21:55.540 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T08:21:55.540 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dirfrag_rstats": 0, 2024-04-24T08:21:55.541 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_file_inodes": 0, 2024-04-24T08:21:55.541 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_inode_file_caps": 0, 2024-04-24T08:21:55.541 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_revoke": 0, 2024-04-24T08:21:55.541 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_grant": 0, 2024-04-24T08:21:55.541 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T08:21:55.541 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T08:21:55.541 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T08:21:55.541 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps": 0, 2024-04-24T08:21:55.541 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps_dirty": 0, 2024-04-24T08:21:55.542 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_cap_release": 0, 2024-04-24T08:21:55.542 INFO:teuthology.orchestra.run.smithi194.stdout: "process_request_cap_release": 0 2024-04-24T08:21:55.542 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.542 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_cache": { 2024-04-24T08:21:55.542 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update": 0, 2024-04-24T08:21:55.542 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update_receipt": 0, 2024-04-24T08:21:55.542 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_try_discover": 0, 2024-04-24T08:21:55.542 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_send_discover": 0, 2024-04-24T08:21:55.542 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_handle_discover": 0, 2024-04-24T08:21:55.543 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays": 0, 2024-04-24T08:21:55.543 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_delayed": 0, 2024-04-24T08:21:55.543 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_enqueuing": 0, 2024-04-24T08:21:55.543 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_created": 0, 2024-04-24T08:21:55.543 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_enqueued": 0, 2024-04-24T08:21:55.543 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_reintegrated": 0, 2024-04-24T08:21:55.543 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_migrated": 0, 2024-04-24T08:21:55.543 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_processing": 0, 2024-04-24T08:21:55.543 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_enqueued": 0, 2024-04-24T08:21:55.544 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_prioritized": 0, 2024-04-24T08:21:55.544 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_started": 0, 2024-04-24T08:21:55.544 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_completed": 0, 2024-04-24T08:21:55.544 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_path": 0, 2024-04-24T08:21:55.544 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_inode": 0, 2024-04-24T08:21:55.544 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T08:21:55.544 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_exportdir": 0, 2024-04-24T08:21:55.544 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_flush": 0, 2024-04-24T08:21:55.544 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragmentdir": 0, 2024-04-24T08:21:55.545 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragstats": 0, 2024-04-24T08:21:55.545 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_inodestats": 0 2024-04-24T08:21:55.545 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.545 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_log": { 2024-04-24T08:21:55.545 INFO:teuthology.orchestra.run.smithi194.stdout: "evlrg": 0, 2024-04-24T08:21:55.545 INFO:teuthology.orchestra.run.smithi194.stdout: "evadd": 0, 2024-04-24T08:21:55.545 INFO:teuthology.orchestra.run.smithi194.stdout: "evex": 0, 2024-04-24T08:21:55.545 INFO:teuthology.orchestra.run.smithi194.stdout: "evtrm": 0, 2024-04-24T08:21:55.545 INFO:teuthology.orchestra.run.smithi194.stdout: "ev": 11070, 2024-04-24T08:21:55.546 INFO:teuthology.orchestra.run.smithi194.stdout: "evexg": 0, 2024-04-24T08:21:55.546 INFO:teuthology.orchestra.run.smithi194.stdout: "evexd": 0, 2024-04-24T08:21:55.546 INFO:teuthology.orchestra.run.smithi194.stdout: "segadd": 0, 2024-04-24T08:21:55.546 INFO:teuthology.orchestra.run.smithi194.stdout: "segex": 0, 2024-04-24T08:21:55.546 INFO:teuthology.orchestra.run.smithi194.stdout: "segtrm": 0, 2024-04-24T08:21:55.546 INFO:teuthology.orchestra.run.smithi194.stdout: "seg": 16, 2024-04-24T08:21:55.546 INFO:teuthology.orchestra.run.smithi194.stdout: "segmjr": 2, 2024-04-24T08:21:55.546 INFO:teuthology.orchestra.run.smithi194.stdout: "segexg": 0, 2024-04-24T08:21:55.546 INFO:teuthology.orchestra.run.smithi194.stdout: "segexd": 0, 2024-04-24T08:21:55.547 INFO:teuthology.orchestra.run.smithi194.stdout: "expos": 4194304, 2024-04-24T08:21:55.547 INFO:teuthology.orchestra.run.smithi194.stdout: "wrpos": 41592929, 2024-04-24T08:21:55.547 INFO:teuthology.orchestra.run.smithi194.stdout: "rdpos": 41592817, 2024-04-24T08:21:55.547 INFO:teuthology.orchestra.run.smithi194.stdout: "jlat": { 2024-04-24T08:21:55.547 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.547 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.547 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.547 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.548 INFO:teuthology.orchestra.run.smithi194.stdout: "replayed": 11070 2024-04-24T08:21:55.548 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.548 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_mem": { 2024-04-24T08:21:55.548 INFO:teuthology.orchestra.run.smithi194.stdout: "ino": 937, 2024-04-24T08:21:55.548 INFO:teuthology.orchestra.run.smithi194.stdout: "ino+": 1411, 2024-04-24T08:21:55.548 INFO:teuthology.orchestra.run.smithi194.stdout: "ino-": 474, 2024-04-24T08:21:55.548 INFO:teuthology.orchestra.run.smithi194.stdout: "dir": 114, 2024-04-24T08:21:55.548 INFO:teuthology.orchestra.run.smithi194.stdout: "dir+": 134, 2024-04-24T08:21:55.548 INFO:teuthology.orchestra.run.smithi194.stdout: "dir-": 20, 2024-04-24T08:21:55.548 INFO:teuthology.orchestra.run.smithi194.stdout: "dn": 1597, 2024-04-24T08:21:55.549 INFO:teuthology.orchestra.run.smithi194.stdout: "dn+": 1654, 2024-04-24T08:21:55.549 INFO:teuthology.orchestra.run.smithi194.stdout: "dn-": 57, 2024-04-24T08:21:55.549 INFO:teuthology.orchestra.run.smithi194.stdout: "cap": 0, 2024-04-24T08:21:55.549 INFO:teuthology.orchestra.run.smithi194.stdout: "cap+": 0, 2024-04-24T08:21:55.549 INFO:teuthology.orchestra.run.smithi194.stdout: "cap-": 0, 2024-04-24T08:21:55.549 INFO:teuthology.orchestra.run.smithi194.stdout: "rss": 57456, 2024-04-24T08:21:55.549 INFO:teuthology.orchestra.run.smithi194.stdout: "heap": 215416 2024-04-24T08:21:55.549 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.550 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_server": { 2024-04-24T08:21:55.550 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_client_request": 0, 2024-04-24T08:21:55.550 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_server_request": 0, 2024-04-24T08:21:55.550 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_request": 0, 2024-04-24T08:21:55.550 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_session": 0, 2024-04-24T08:21:55.550 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_peer_request": 0, 2024-04-24T08:21:55.550 INFO:teuthology.orchestra.run.smithi194.stdout: "req_create_latency": { 2024-04-24T08:21:55.550 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.550 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.551 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.551 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.551 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getattr_latency": { 2024-04-24T08:21:55.551 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.551 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.551 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.551 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.551 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getfilelock_latency": { 2024-04-24T08:21:55.551 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.552 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.552 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.552 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.552 INFO:teuthology.orchestra.run.smithi194.stdout: "req_link_latency": { 2024-04-24T08:21:55.552 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.552 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.552 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.552 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.552 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookup_latency": { 2024-04-24T08:21:55.552 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.553 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.553 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.553 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.553 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookuphash_latency": { 2024-04-24T08:21:55.553 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.553 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.553 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.553 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.553 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupino_latency": { 2024-04-24T08:21:55.554 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.554 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.554 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.554 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.554 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupname_latency": { 2024-04-24T08:21:55.554 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.554 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.554 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.554 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.554 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupparent_latency": { 2024-04-24T08:21:55.555 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.555 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.555 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.555 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.555 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupsnap_latency": { 2024-04-24T08:21:55.555 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.555 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.555 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.555 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.556 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lssnap_latency": { 2024-04-24T08:21:55.556 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.556 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.556 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.556 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.557 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mkdir_latency": { 2024-04-24T08:21:55.557 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.557 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.557 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.557 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.557 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mknod_latency": { 2024-04-24T08:21:55.557 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.557 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.558 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.558 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.558 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mksnap_latency": { 2024-04-24T08:21:55.558 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.558 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.558 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.558 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.558 INFO:teuthology.orchestra.run.smithi194.stdout: "req_open_latency": { 2024-04-24T08:21:55.558 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.559 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.559 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.559 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.559 INFO:teuthology.orchestra.run.smithi194.stdout: "req_readdir_latency": { 2024-04-24T08:21:55.559 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.559 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.559 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.559 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.559 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rename_latency": { 2024-04-24T08:21:55.560 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.560 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.560 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.560 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.560 INFO:teuthology.orchestra.run.smithi194.stdout: "req_renamesnap_latency": { 2024-04-24T08:21:55.560 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.560 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.560 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.560 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.561 INFO:teuthology.orchestra.run.smithi194.stdout: "req_snapdiff_latency": { 2024-04-24T08:21:55.561 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.561 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.561 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.561 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.561 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmdir_latency": { 2024-04-24T08:21:55.561 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.561 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.561 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.562 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.562 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmsnap_latency": { 2024-04-24T08:21:55.562 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.562 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.562 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.562 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.562 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmxattr_latency": { 2024-04-24T08:21:55.562 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.563 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.563 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.563 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.563 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setattr_latency": { 2024-04-24T08:21:55.563 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.563 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.563 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.563 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.563 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setdirlayout_latency": { 2024-04-24T08:21:55.563 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.564 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.564 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.564 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.564 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setfilelock_latency": { 2024-04-24T08:21:55.564 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.564 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.564 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.564 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.565 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setlayout_latency": { 2024-04-24T08:21:55.565 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.565 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.565 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.565 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.565 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setxattr_latency": { 2024-04-24T08:21:55.565 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.565 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.565 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.566 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.566 INFO:teuthology.orchestra.run.smithi194.stdout: "req_symlink_latency": { 2024-04-24T08:21:55.566 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.566 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.566 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.566 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.566 INFO:teuthology.orchestra.run.smithi194.stdout: "req_unlink_latency": { 2024-04-24T08:21:55.566 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.567 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.567 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.567 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.567 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_revoke_eviction": 0, 2024-04-24T08:21:55.567 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_acquisition_throttle": 0, 2024-04-24T08:21:55.567 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getvxattr_latency": { 2024-04-24T08:21:55.567 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.567 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.568 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.568 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:55.568 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.568 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_sessions": { 2024-04-24T08:21:55.568 INFO:teuthology.orchestra.run.smithi194.stdout: "session_count": 2, 2024-04-24T08:21:55.568 INFO:teuthology.orchestra.run.smithi194.stdout: "session_add": 5, 2024-04-24T08:21:55.568 INFO:teuthology.orchestra.run.smithi194.stdout: "session_remove": 3, 2024-04-24T08:21:55.568 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_open": 4, 2024-04-24T08:21:55.568 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_stale": 0, 2024-04-24T08:21:55.568 INFO:teuthology.orchestra.run.smithi194.stdout: "total_load": 0, 2024-04-24T08:21:55.569 INFO:teuthology.orchestra.run.smithi194.stdout: "average_load": 0, 2024-04-24T08:21:55.569 INFO:teuthology.orchestra.run.smithi194.stdout: "avg_session_uptime": 3667, 2024-04-24T08:21:55.569 INFO:teuthology.orchestra.run.smithi194.stdout: "mdthresh_evicted": 0 2024-04-24T08:21:55.569 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.569 INFO:teuthology.orchestra.run.smithi194.stdout: "mempool": { 2024-04-24T08:21:55.569 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:21:55.569 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_items": 0, 2024-04-24T08:21:55.569 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:21:55.570 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:21:55.570 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:21:55.570 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:21:55.570 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:21:55.570 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:21:55.570 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:21:55.570 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:21:55.570 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:21:55.570 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:21:55.571 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:21:55.571 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:21:55.571 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:21:55.571 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_items": 0, 2024-04-24T08:21:55.571 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:21:55.571 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_items": 0, 2024-04-24T08:21:55.571 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:21:55.571 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:21:55.572 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:21:55.572 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:21:55.572 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:21:55.572 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:21:55.572 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:21:55.572 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_items": 0, 2024-04-24T08:21:55.572 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:21:55.572 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:21:55.572 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:21:55.572 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_items": 0, 2024-04-24T08:21:55.573 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_bytes": 0, 2024-04-24T08:21:55.573 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_items": 0, 2024-04-24T08:21:55.573 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:21:55.573 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:21:55.573 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:21:55.573 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:21:55.573 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_bytes": 253030, 2024-04-24T08:21:55.573 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_items": 1090, 2024-04-24T08:21:55.573 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:21:55.574 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_items": 0, 2024-04-24T08:21:55.574 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_bytes": 0, 2024-04-24T08:21:55.574 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_items": 0, 2024-04-24T08:21:55.574 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:21:55.574 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_items": 0, 2024-04-24T08:21:55.574 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:21:55.574 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_items": 0, 2024-04-24T08:21:55.574 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_bytes": 8320, 2024-04-24T08:21:55.574 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_items": 173, 2024-04-24T08:21:55.575 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:21:55.575 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:21:55.575 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_bytes": 0, 2024-04-24T08:21:55.575 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_items": 0, 2024-04-24T08:21:55.575 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_bytes": 3559345, 2024-04-24T08:21:55.575 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_items": 38665, 2024-04-24T08:21:55.575 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_bytes": 0, 2024-04-24T08:21:55.575 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_items": 0, 2024-04-24T08:21:55.575 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_bytes": 0, 2024-04-24T08:21:55.575 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_items": 0 2024-04-24T08:21:55.576 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.576 INFO:teuthology.orchestra.run.smithi194.stdout: "objecter": { 2024-04-24T08:21:55.576 INFO:teuthology.orchestra.run.smithi194.stdout: "op_active": 0, 2024-04-24T08:21:55.576 INFO:teuthology.orchestra.run.smithi194.stdout: "op_laggy": 0, 2024-04-24T08:21:55.576 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send": 2571, 2024-04-24T08:21:55.576 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send_bytes": 26, 2024-04-24T08:21:55.576 INFO:teuthology.orchestra.run.smithi194.stdout: "op_resend": 4, 2024-04-24T08:21:55.576 INFO:teuthology.orchestra.run.smithi194.stdout: "op_reply": 2571, 2024-04-24T08:21:55.576 INFO:teuthology.orchestra.run.smithi194.stdout: "op_latency": { 2024-04-24T08:21:55.577 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2571, 2024-04-24T08:21:55.577 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 80.644313715, 2024-04-24T08:21:55.577 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.031366905 2024-04-24T08:21:55.577 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.577 INFO:teuthology.orchestra.run.smithi194.stdout: "op_inflight": 0, 2024-04-24T08:21:55.577 INFO:teuthology.orchestra.run.smithi194.stdout: "oplen_avg": { 2024-04-24T08:21:55.577 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2571, 2024-04-24T08:21:55.577 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 2572 2024-04-24T08:21:55.577 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.577 INFO:teuthology.orchestra.run.smithi194.stdout: "op": 2571, 2024-04-24T08:21:55.578 INFO:teuthology.orchestra.run.smithi194.stdout: "op_r": 2571, 2024-04-24T08:21:55.578 INFO:teuthology.orchestra.run.smithi194.stdout: "op_w": 0, 2024-04-24T08:21:55.578 INFO:teuthology.orchestra.run.smithi194.stdout: "op_rmw": 0, 2024-04-24T08:21:55.578 INFO:teuthology.orchestra.run.smithi194.stdout: "op_pg": 0, 2024-04-24T08:21:55.578 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_stat": 1542, 2024-04-24T08:21:55.578 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_create": 0, 2024-04-24T08:21:55.578 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_read": 1026, 2024-04-24T08:21:55.578 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_write": 0, 2024-04-24T08:21:55.579 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writefull": 0, 2024-04-24T08:21:55.579 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writesame": 0, 2024-04-24T08:21:55.579 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_append": 0, 2024-04-24T08:21:55.579 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_zero": 0, 2024-04-24T08:21:55.579 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_truncate": 0, 2024-04-24T08:21:55.579 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_delete": 0, 2024-04-24T08:21:55.579 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_mapext": 0, 2024-04-24T08:21:55.579 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_sparse_read": 0, 2024-04-24T08:21:55.579 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_clonerange": 0, 2024-04-24T08:21:55.579 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_getxattr": 2, 2024-04-24T08:21:55.580 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_setxattr": 0, 2024-04-24T08:21:55.580 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_cmpxattr": 0, 2024-04-24T08:21:55.580 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_rmxattr": 0, 2024-04-24T08:21:55.580 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_resetxattrs": 0, 2024-04-24T08:21:55.580 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_call": 0, 2024-04-24T08:21:55.580 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_watch": 0, 2024-04-24T08:21:55.580 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_notify": 0, 2024-04-24T08:21:55.580 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T08:21:55.581 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls": 0, 2024-04-24T08:21:55.581 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls_filter": 0, 2024-04-24T08:21:55.581 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_other": 0, 2024-04-24T08:21:55.581 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_active": 0, 2024-04-24T08:21:55.581 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_send": 0, 2024-04-24T08:21:55.581 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_resend": 0, 2024-04-24T08:21:55.581 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_ping": 0, 2024-04-24T08:21:55.581 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_active": 0, 2024-04-24T08:21:55.582 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_send": 0, 2024-04-24T08:21:55.582 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_resend": 0, 2024-04-24T08:21:55.582 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_active": 0, 2024-04-24T08:21:55.582 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_send": 0, 2024-04-24T08:21:55.582 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_resend": 0, 2024-04-24T08:21:55.582 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_active": 0, 2024-04-24T08:21:55.582 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_send": 0, 2024-04-24T08:21:55.582 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_resend": 0, 2024-04-24T08:21:55.583 INFO:teuthology.orchestra.run.smithi194.stdout: "command_active": 0, 2024-04-24T08:21:55.583 INFO:teuthology.orchestra.run.smithi194.stdout: "command_send": 0, 2024-04-24T08:21:55.583 INFO:teuthology.orchestra.run.smithi194.stdout: "command_resend": 0, 2024-04-24T08:21:55.583 INFO:teuthology.orchestra.run.smithi194.stdout: "map_epoch": 101, 2024-04-24T08:21:55.583 INFO:teuthology.orchestra.run.smithi194.stdout: "map_full": 0, 2024-04-24T08:21:55.583 INFO:teuthology.orchestra.run.smithi194.stdout: "map_inc": 15, 2024-04-24T08:21:55.583 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_sessions": 8, 2024-04-24T08:21:55.583 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_open": 8, 2024-04-24T08:21:55.583 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_close": 0, 2024-04-24T08:21:55.583 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_laggy": 0, 2024-04-24T08:21:55.584 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_wr": 0, 2024-04-24T08:21:55.584 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_rd": 2, 2024-04-24T08:21:55.584 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_del": 0 2024-04-24T08:21:55.584 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.584 INFO:teuthology.orchestra.run.smithi194.stdout: "oft": { 2024-04-24T08:21:55.584 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_objs": 0, 2024-04-24T08:21:55.584 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_kv_pairs": 0, 2024-04-24T08:21:55.584 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_updates": 0, 2024-04-24T08:21:55.584 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_removes": 0 2024-04-24T08:21:55.585 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.585 INFO:teuthology.orchestra.run.smithi194.stdout: "osd-slow-ops": { 2024-04-24T08:21:55.585 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_ops_count": 0 2024-04-24T08:21:55.585 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.585 INFO:teuthology.orchestra.run.smithi194.stdout: "purge_queue": { 2024-04-24T08:21:55.585 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops": 0, 2024-04-24T08:21:55.585 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T08:21:55.585 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing": 0, 2024-04-24T08:21:55.585 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_high_water": 0, 2024-04-24T08:21:55.586 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed_ops": 0, 2024-04-24T08:21:55.586 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed": 0, 2024-04-24T08:21:55.586 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_item_in_journal": 0 2024-04-24T08:21:55.586 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.586 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:21:55.586 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-04-24T08:21:55.586 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-04-24T08:21:55.586 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-04-24T08:21:55.586 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 2841, 2024-04-24T08:21:55.587 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 39630746, 2024-04-24T08:21:55.587 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:55.587 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 2841, 2024-04-24T08:21:55.587 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-04-24T08:21:55.587 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-04-24T08:21:55.587 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 2841, 2024-04-24T08:21:55.587 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 39630746, 2024-04-24T08:21:55.587 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-04-24T08:21:55.587 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.588 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.588 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.588 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:55.588 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.588 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_bytes": { 2024-04-24T08:21:55.588 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-04-24T08:21:55.588 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-04-24T08:21:55.588 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-04-24T08:21:55.588 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-04-24T08:21:55.589 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-04-24T08:21:55.589 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:55.589 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-04-24T08:21:55.589 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 2571, 2024-04-24T08:21:55.589 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 37413107, 2024-04-24T08:21:55.589 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 2571, 2024-04-24T08:21:55.589 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 37413107, 2024-04-24T08:21:55.589 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-04-24T08:21:55.589 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.590 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.590 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.590 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:55.590 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.590 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_ops": { 2024-04-24T08:21:55.590 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-04-24T08:21:55.590 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 1024, 2024-04-24T08:21:55.590 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-04-24T08:21:55.591 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-04-24T08:21:55.591 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-04-24T08:21:55.591 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:55.591 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-04-24T08:21:55.591 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 2571, 2024-04-24T08:21:55.591 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 2571, 2024-04-24T08:21:55.591 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 2571, 2024-04-24T08:21:55.591 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 2571, 2024-04-24T08:21:55.591 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-04-24T08:21:55.592 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.592 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.592 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.592 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:55.592 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.592 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle": { 2024-04-24T08:21:55.592 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-04-24T08:21:55.592 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-04-24T08:21:55.592 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-04-24T08:21:55.593 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-04-24T08:21:55.593 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-04-24T08:21:55.593 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:55.593 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-04-24T08:21:55.593 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-04-24T08:21:55.593 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-04-24T08:21:55.593 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 0, 2024-04-24T08:21:55.593 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 0, 2024-04-24T08:21:55.594 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-04-24T08:21:55.594 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.594 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.594 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.594 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:55.594 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:55.594 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle-0x55a5186d86e0": { 2024-04-24T08:21:55.594 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-04-24T08:21:55.594 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-04-24T08:21:55.595 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-04-24T08:21:55.595 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-04-24T08:21:55.595 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-04-24T08:21:55.595 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:55.595 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-04-24T08:21:55.595 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-04-24T08:21:55.595 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-04-24T08:21:55.595 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 0, 2024-04-24T08:21:55.596 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 0, 2024-04-24T08:21:55.596 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-04-24T08:21:55.596 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:55.596 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:55.596 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:55.596 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:55.596 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:55.596 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-24T08:21:55.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:55 smithi194 ceph-mon[32836]: pgmap v360: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:21:56.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:55 smithi028 ceph-mon[27786]: pgmap v360: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:21:56.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:55 smithi117 ceph-mon[32438]: pgmap v360: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-04-24T08:21:56.325 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-04-24T08:21:56.325 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-04-24T08:21:56.325 INFO:tasks.check_counter:Daemon mds.c mds.root_rsnaps=6 2024-04-24T08:21:56.325 INFO:tasks.check_counter:Daemon mds.c mds_server.req_mksnap_latency.avgcount=0 2024-04-24T08:21:56.325 INFO:tasks.check_counter:Daemon mds.c mds_server.req_rmsnap_latency.avgcount=0 2024-04-24T08:21:56.326 INFO:tasks.check_counter:Daemon mds.c mds.dir_split=0 2024-04-24T08:21:56.326 DEBUG:tasks.check_counter:Getting stats from f 2024-04-24T08:21:56.326 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph daemon mds.f perf dump 2024-04-24T08:21:56.574 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:21:57.924 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:57 smithi194 ceph-mon[32836]: pgmap v361: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:21:58.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:57 smithi028 ceph-mon[27786]: pgmap v361: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:21:58.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:57 smithi117 ceph-mon[32438]: pgmap v361: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-04-24T08:21:58.171 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-24T08:21:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:21:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 2404, 2024-04-24T08:21:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 2406, 2024-04-24T08:21:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 682448, 2024-04-24T08:21:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 764479, 2024-04-24T08:21:58.172 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 6, 2024-04-24T08:21:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 2, 2024-04-24T08:21:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.203863368, 2024-04-24T08:21:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.064528848, 2024-04-24T08:21:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.051589310, 2024-04-24T08:21:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.072713919, 2024-04-24T08:21:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2404, 2024-04-24T08:21:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.094760152, 2024-04-24T08:21:58.173 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000039417 2024-04-24T08:21:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 0, 2024-04-24T08:21:58.174 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 0 2024-04-24T08:21:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:21:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 274, 2024-04-24T08:21:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 257, 2024-04-24T08:21:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 1773676, 2024-04-24T08:21:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 148522, 2024-04-24T08:21:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 3, 2024-04-24T08:21:58.175 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 2, 2024-04-24T08:21:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.051678185, 2024-04-24T08:21:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.014853294, 2024-04-24T08:21:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.042946281, 2024-04-24T08:21:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.005845063, 2024-04-24T08:21:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 257, 2024-04-24T08:21:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.014134361, 2024-04-24T08:21:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000054997 2024-04-24T08:21:58.176 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 1773120, 2024-04-24T08:21:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 147824 2024-04-24T08:21:58.177 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:21:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 3, 2024-04-24T08:21:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 179, 2024-04-24T08:21:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 588, 2024-04-24T08:21:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 474766, 2024-04-24T08:21:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 6, 2024-04-24T08:21:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 4, 2024-04-24T08:21:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.009897402, 2024-04-24T08:21:58.178 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.006176723, 2024-04-24T08:21:58.179 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.001792497, 2024-04-24T08:21:58.179 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.000108817, 2024-04-24T08:21:58.179 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:21:58.179 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 176, 2024-04-24T08:21:58.179 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.031238506, 2024-04-24T08:21:58.179 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000177491 2024-04-24T08:21:58.179 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.179 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:21:58.180 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.180 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.180 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.180 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-24T08:21:58.180 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 474448 2024-04-24T08:21:58.180 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.180 INFO:teuthology.orchestra.run.smithi194.stdout: "cct": { 2024-04-24T08:21:58.180 INFO:teuthology.orchestra.run.smithi194.stdout: "total_workers": 1, 2024-04-24T08:21:58.180 INFO:teuthology.orchestra.run.smithi194.stdout: "unhealthy_workers": 0 2024-04-24T08:21:58.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.181 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-MDSRank": { 2024-04-24T08:21:58.181 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-04-24T08:21:58.181 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-04-24T08:21:58.181 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 3991, 2024-04-24T08:21:58.181 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.110757887, 2024-04-24T08:21:58.181 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000027751 2024-04-24T08:21:58.181 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:58.181 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.182 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-PurgeQueue": { 2024-04-24T08:21:58.182 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-04-24T08:21:58.182 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-04-24T08:21:58.182 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.182 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.182 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.182 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:58.182 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.182 INFO:teuthology.orchestra.run.smithi194.stdout: "mds": { 2024-04-24T08:21:58.183 INFO:teuthology.orchestra.run.smithi194.stdout: "request": 0, 2024-04-24T08:21:58.183 INFO:teuthology.orchestra.run.smithi194.stdout: "reply": 0, 2024-04-24T08:21:58.183 INFO:teuthology.orchestra.run.smithi194.stdout: "reply_latency": { 2024-04-24T08:21:58.183 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.183 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.183 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.183 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.183 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_reply": 0, 2024-04-24T08:21:58.183 INFO:teuthology.orchestra.run.smithi194.stdout: "forward": 0, 2024-04-24T08:21:58.184 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_complete": 0, 2024-04-24T08:21:58.184 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_keys": 0, 2024-04-24T08:21:58.184 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_commit": 0, 2024-04-24T08:21:58.184 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_split": 0, 2024-04-24T08:21:58.184 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_merge": 0, 2024-04-24T08:21:58.184 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes": 0, 2024-04-24T08:21:58.184 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_top": 0, 2024-04-24T08:21:58.184 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_bottom": 0, 2024-04-24T08:21:58.184 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pin_tail": 0, 2024-04-24T08:21:58.185 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pinned": 0, 2024-04-24T08:21:58.185 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_expired": 0, 2024-04-24T08:21:58.185 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_with_caps": 0, 2024-04-24T08:21:58.185 INFO:teuthology.orchestra.run.smithi194.stdout: "caps": 0, 2024-04-24T08:21:58.185 INFO:teuthology.orchestra.run.smithi194.stdout: "subtrees": 1, 2024-04-24T08:21:58.185 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse": 0, 2024-04-24T08:21:58.185 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_hit": 0, 2024-04-24T08:21:58.185 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_forward": 0, 2024-04-24T08:21:58.185 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_discover": 0, 2024-04-24T08:21:58.185 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_dir_fetch": 0, 2024-04-24T08:21:58.186 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_remote_ino": 0, 2024-04-24T08:21:58.186 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_lock": 0, 2024-04-24T08:21:58.186 INFO:teuthology.orchestra.run.smithi194.stdout: "load_cent": 0, 2024-04-24T08:21:58.186 INFO:teuthology.orchestra.run.smithi194.stdout: "q": 0, 2024-04-24T08:21:58.186 INFO:teuthology.orchestra.run.smithi194.stdout: "exported": 0, 2024-04-24T08:21:58.186 INFO:teuthology.orchestra.run.smithi194.stdout: "exported_inodes": 0, 2024-04-24T08:21:58.186 INFO:teuthology.orchestra.run.smithi194.stdout: "imported": 0, 2024-04-24T08:21:58.186 INFO:teuthology.orchestra.run.smithi194.stdout: "imported_inodes": 0, 2024-04-24T08:21:58.187 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_dir_fetch": 0, 2024-04-24T08:21:58.187 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_backtrace_fetch": 0, 2024-04-24T08:21:58.187 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_peer_discover": 0, 2024-04-24T08:21:58.187 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rfiles": 0, 2024-04-24T08:21:58.187 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rbytes": 0, 2024-04-24T08:21:58.187 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rsnaps": 0, 2024-04-24T08:21:58.187 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_fetch": 0, 2024-04-24T08:21:58.187 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_set_tag": 0, 2024-04-24T08:21:58.187 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T08:21:58.187 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_inotable_repaired": 0, 2024-04-24T08:21:58.188 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_inodes": 0, 2024-04-24T08:21:58.188 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T08:21:58.188 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dirfrag_rstats": 0, 2024-04-24T08:21:58.188 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_file_inodes": 0, 2024-04-24T08:21:58.188 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_inode_file_caps": 0, 2024-04-24T08:21:58.188 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_revoke": 0, 2024-04-24T08:21:58.188 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_grant": 0, 2024-04-24T08:21:58.188 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T08:21:58.188 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T08:21:58.189 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T08:21:58.189 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps": 0, 2024-04-24T08:21:58.189 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps_dirty": 0, 2024-04-24T08:21:58.189 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_cap_release": 0, 2024-04-24T08:21:58.189 INFO:teuthology.orchestra.run.smithi194.stdout: "process_request_cap_release": 0 2024-04-24T08:21:58.189 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.189 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_cache": { 2024-04-24T08:21:58.189 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update": 0, 2024-04-24T08:21:58.189 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update_receipt": 0, 2024-04-24T08:21:58.189 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_try_discover": 0, 2024-04-24T08:21:58.190 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_send_discover": 0, 2024-04-24T08:21:58.190 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_handle_discover": 0, 2024-04-24T08:21:58.190 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays": 0, 2024-04-24T08:21:58.190 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_delayed": 0, 2024-04-24T08:21:58.190 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_enqueuing": 0, 2024-04-24T08:21:58.190 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_created": 0, 2024-04-24T08:21:58.190 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_enqueued": 0, 2024-04-24T08:21:58.190 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_reintegrated": 0, 2024-04-24T08:21:58.191 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_migrated": 0, 2024-04-24T08:21:58.191 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_processing": 0, 2024-04-24T08:21:58.191 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_enqueued": 0, 2024-04-24T08:21:58.191 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_prioritized": 0, 2024-04-24T08:21:58.191 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_started": 0, 2024-04-24T08:21:58.191 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_completed": 0, 2024-04-24T08:21:58.191 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_path": 0, 2024-04-24T08:21:58.191 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_inode": 0, 2024-04-24T08:21:58.191 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T08:21:58.192 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_exportdir": 0, 2024-04-24T08:21:58.192 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_flush": 0, 2024-04-24T08:21:58.192 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragmentdir": 0, 2024-04-24T08:21:58.192 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragstats": 0, 2024-04-24T08:21:58.192 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_inodestats": 0 2024-04-24T08:21:58.192 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.192 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_log": { 2024-04-24T08:21:58.192 INFO:teuthology.orchestra.run.smithi194.stdout: "evlrg": 0, 2024-04-24T08:21:58.192 INFO:teuthology.orchestra.run.smithi194.stdout: "evadd": 0, 2024-04-24T08:21:58.193 INFO:teuthology.orchestra.run.smithi194.stdout: "evex": 0, 2024-04-24T08:21:58.193 INFO:teuthology.orchestra.run.smithi194.stdout: "evtrm": 0, 2024-04-24T08:21:58.193 INFO:teuthology.orchestra.run.smithi194.stdout: "ev": 10, 2024-04-24T08:21:58.193 INFO:teuthology.orchestra.run.smithi194.stdout: "evexg": 0, 2024-04-24T08:21:58.193 INFO:teuthology.orchestra.run.smithi194.stdout: "evexd": 0, 2024-04-24T08:21:58.193 INFO:teuthology.orchestra.run.smithi194.stdout: "segadd": 0, 2024-04-24T08:21:58.193 INFO:teuthology.orchestra.run.smithi194.stdout: "segex": 0, 2024-04-24T08:21:58.193 INFO:teuthology.orchestra.run.smithi194.stdout: "segtrm": 0, 2024-04-24T08:21:58.193 INFO:teuthology.orchestra.run.smithi194.stdout: "seg": 2, 2024-04-24T08:21:58.194 INFO:teuthology.orchestra.run.smithi194.stdout: "segmjr": 2, 2024-04-24T08:21:58.194 INFO:teuthology.orchestra.run.smithi194.stdout: "segexg": 0, 2024-04-24T08:21:58.194 INFO:teuthology.orchestra.run.smithi194.stdout: "segexd": 0, 2024-04-24T08:21:58.194 INFO:teuthology.orchestra.run.smithi194.stdout: "expos": 4194304, 2024-04-24T08:21:58.194 INFO:teuthology.orchestra.run.smithi194.stdout: "wrpos": 4197189, 2024-04-24T08:21:58.194 INFO:teuthology.orchestra.run.smithi194.stdout: "rdpos": 4197048, 2024-04-24T08:21:58.194 INFO:teuthology.orchestra.run.smithi194.stdout: "jlat": { 2024-04-24T08:21:58.194 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.194 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.195 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.195 INFO:teuthology.orchestra.run.smithi194.stdout: "replayed": 10 2024-04-24T08:21:58.195 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.195 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_mem": { 2024-04-24T08:21:58.195 INFO:teuthology.orchestra.run.smithi194.stdout: "ino": 3, 2024-04-24T08:21:58.195 INFO:teuthology.orchestra.run.smithi194.stdout: "ino+": 3, 2024-04-24T08:21:58.195 INFO:teuthology.orchestra.run.smithi194.stdout: "ino-": 0, 2024-04-24T08:21:58.195 INFO:teuthology.orchestra.run.smithi194.stdout: "dir": 1, 2024-04-24T08:21:58.195 INFO:teuthology.orchestra.run.smithi194.stdout: "dir+": 1, 2024-04-24T08:21:58.196 INFO:teuthology.orchestra.run.smithi194.stdout: "dir-": 0, 2024-04-24T08:21:58.196 INFO:teuthology.orchestra.run.smithi194.stdout: "dn": 0, 2024-04-24T08:21:58.196 INFO:teuthology.orchestra.run.smithi194.stdout: "dn+": 0, 2024-04-24T08:21:58.196 INFO:teuthology.orchestra.run.smithi194.stdout: "dn-": 0, 2024-04-24T08:21:58.196 INFO:teuthology.orchestra.run.smithi194.stdout: "cap": 0, 2024-04-24T08:21:58.196 INFO:teuthology.orchestra.run.smithi194.stdout: "cap+": 0, 2024-04-24T08:21:58.196 INFO:teuthology.orchestra.run.smithi194.stdout: "cap-": 0, 2024-04-24T08:21:58.196 INFO:teuthology.orchestra.run.smithi194.stdout: "rss": 53060, 2024-04-24T08:21:58.197 INFO:teuthology.orchestra.run.smithi194.stdout: "heap": 215416 2024-04-24T08:21:58.197 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.197 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_server": { 2024-04-24T08:21:58.197 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_client_request": 0, 2024-04-24T08:21:58.197 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_server_request": 0, 2024-04-24T08:21:58.197 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_request": 0, 2024-04-24T08:21:58.197 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_session": 0, 2024-04-24T08:21:58.197 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_peer_request": 0, 2024-04-24T08:21:58.197 INFO:teuthology.orchestra.run.smithi194.stdout: "req_create_latency": { 2024-04-24T08:21:58.197 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.198 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.198 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.198 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getattr_latency": { 2024-04-24T08:21:58.198 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.198 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.198 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.198 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.198 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getfilelock_latency": { 2024-04-24T08:21:58.199 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.199 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.199 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.199 INFO:teuthology.orchestra.run.smithi194.stdout: "req_link_latency": { 2024-04-24T08:21:58.199 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.199 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.199 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.199 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.200 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookup_latency": { 2024-04-24T08:21:58.200 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.200 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.200 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.200 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookuphash_latency": { 2024-04-24T08:21:58.200 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.200 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.200 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.200 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupino_latency": { 2024-04-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupname_latency": { 2024-04-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.201 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.202 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.202 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupparent_latency": { 2024-04-24T08:21:58.202 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.202 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.202 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.202 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.202 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupsnap_latency": { 2024-04-24T08:21:58.203 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.203 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.203 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.203 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lssnap_latency": { 2024-04-24T08:21:58.203 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.203 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.203 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.203 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.204 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mkdir_latency": { 2024-04-24T08:21:58.204 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.204 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.204 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.205 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mknod_latency": { 2024-04-24T08:21:58.205 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.205 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.205 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.205 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.205 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mksnap_latency": { 2024-04-24T08:21:58.206 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.206 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.206 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.206 INFO:teuthology.orchestra.run.smithi194.stdout: "req_open_latency": { 2024-04-24T08:21:58.206 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.206 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.206 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.206 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.207 INFO:teuthology.orchestra.run.smithi194.stdout: "req_readdir_latency": { 2024-04-24T08:21:58.207 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.207 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.207 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.207 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.207 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rename_latency": { 2024-04-24T08:21:58.207 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.207 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.207 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.208 INFO:teuthology.orchestra.run.smithi194.stdout: "req_renamesnap_latency": { 2024-04-24T08:21:58.208 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.208 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.208 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.208 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.208 INFO:teuthology.orchestra.run.smithi194.stdout: "req_snapdiff_latency": { 2024-04-24T08:21:58.208 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.208 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmdir_latency": { 2024-04-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmsnap_latency": { 2024-04-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.209 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.210 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.210 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmxattr_latency": { 2024-04-24T08:21:58.210 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.210 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.210 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.210 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.210 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setattr_latency": { 2024-04-24T08:21:58.211 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.211 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.211 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.211 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setdirlayout_latency": { 2024-04-24T08:21:58.211 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.211 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.211 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.211 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.211 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setfilelock_latency": { 2024-04-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setlayout_latency": { 2024-04-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.212 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.213 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setxattr_latency": { 2024-04-24T08:21:58.213 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.213 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.213 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.213 INFO:teuthology.orchestra.run.smithi194.stdout: "req_symlink_latency": { 2024-04-24T08:21:58.213 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.213 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.213 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.213 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.214 INFO:teuthology.orchestra.run.smithi194.stdout: "req_unlink_latency": { 2024-04-24T08:21:58.214 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.214 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.214 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.214 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.214 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_revoke_eviction": 0, 2024-04-24T08:21:58.214 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_acquisition_throttle": 0, 2024-04-24T08:21:58.214 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getvxattr_latency": { 2024-04-24T08:21:58.215 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.215 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.215 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.215 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:58.215 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.215 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_sessions": { 2024-04-24T08:21:58.215 INFO:teuthology.orchestra.run.smithi194.stdout: "session_count": 2, 2024-04-24T08:21:58.215 INFO:teuthology.orchestra.run.smithi194.stdout: "session_add": 5, 2024-04-24T08:21:58.215 INFO:teuthology.orchestra.run.smithi194.stdout: "session_remove": 3, 2024-04-24T08:21:58.215 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_open": 4, 2024-04-24T08:21:58.216 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_stale": 0, 2024-04-24T08:21:58.216 INFO:teuthology.orchestra.run.smithi194.stdout: "total_load": 0, 2024-04-24T08:21:58.216 INFO:teuthology.orchestra.run.smithi194.stdout: "average_load": 0, 2024-04-24T08:21:58.216 INFO:teuthology.orchestra.run.smithi194.stdout: "avg_session_uptime": 3669, 2024-04-24T08:21:58.216 INFO:teuthology.orchestra.run.smithi194.stdout: "mdthresh_evicted": 0 2024-04-24T08:21:58.216 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.216 INFO:teuthology.orchestra.run.smithi194.stdout: "mempool": { 2024-04-24T08:21:58.216 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:21:58.216 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_items": 0, 2024-04-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:21:58.217 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_items": 0, 2024-04-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_items": 0, 2024-04-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:21:58.218 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_items": 0, 2024-04-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:21:58.219 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_items": 0, 2024-04-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_bytes": 0, 2024-04-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_items": 0, 2024-04-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_bytes": 181331, 2024-04-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_items": 48, 2024-04-24T08:21:58.220 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_items": 0, 2024-04-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_bytes": 0, 2024-04-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_items": 0, 2024-04-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_items": 0, 2024-04-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_items": 0, 2024-04-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_bytes": 8320, 2024-04-24T08:21:58.221 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_items": 173, 2024-04-24T08:21:58.222 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:21:58.222 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:21:58.222 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_bytes": 0, 2024-04-24T08:21:58.222 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_items": 0, 2024-04-24T08:21:58.222 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_bytes": 8512, 2024-04-24T08:21:58.222 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_items": 13, 2024-04-24T08:21:58.222 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_bytes": 0, 2024-04-24T08:21:58.222 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_items": 0, 2024-04-24T08:21:58.222 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_bytes": 0, 2024-04-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_items": 0 2024-04-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi194.stdout: "objecter": { 2024-04-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi194.stdout: "op_active": 0, 2024-04-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi194.stdout: "op_laggy": 0, 2024-04-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send": 2407, 2024-04-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send_bytes": 21, 2024-04-24T08:21:58.223 INFO:teuthology.orchestra.run.smithi194.stdout: "op_resend": 3, 2024-04-24T08:21:58.224 INFO:teuthology.orchestra.run.smithi194.stdout: "op_reply": 2407, 2024-04-24T08:21:58.224 INFO:teuthology.orchestra.run.smithi194.stdout: "op_latency": { 2024-04-24T08:21:58.224 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2407, 2024-04-24T08:21:58.224 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 29.221395217, 2024-04-24T08:21:58.224 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.012140172 2024-04-24T08:21:58.224 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.224 INFO:teuthology.orchestra.run.smithi194.stdout: "op_inflight": 0, 2024-04-24T08:21:58.224 INFO:teuthology.orchestra.run.smithi194.stdout: "oplen_avg": { 2024-04-24T08:21:58.225 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2407, 2024-04-24T08:21:58.225 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 2408 2024-04-24T08:21:58.225 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.225 INFO:teuthology.orchestra.run.smithi194.stdout: "op": 2407, 2024-04-24T08:21:58.225 INFO:teuthology.orchestra.run.smithi194.stdout: "op_r": 2407, 2024-04-24T08:21:58.225 INFO:teuthology.orchestra.run.smithi194.stdout: "op_w": 0, 2024-04-24T08:21:58.225 INFO:teuthology.orchestra.run.smithi194.stdout: "op_rmw": 0, 2024-04-24T08:21:58.225 INFO:teuthology.orchestra.run.smithi194.stdout: "op_pg": 0, 2024-04-24T08:21:58.225 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_stat": 1569, 2024-04-24T08:21:58.225 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_create": 0, 2024-04-24T08:21:58.226 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_read": 836, 2024-04-24T08:21:58.226 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_write": 0, 2024-04-24T08:21:58.226 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writefull": 0, 2024-04-24T08:21:58.226 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writesame": 0, 2024-04-24T08:21:58.226 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_append": 0, 2024-04-24T08:21:58.226 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_zero": 0, 2024-04-24T08:21:58.226 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_truncate": 0, 2024-04-24T08:21:58.226 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_delete": 0, 2024-04-24T08:21:58.226 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_mapext": 0, 2024-04-24T08:21:58.227 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_sparse_read": 0, 2024-04-24T08:21:58.227 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_clonerange": 0, 2024-04-24T08:21:58.227 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_getxattr": 1, 2024-04-24T08:21:58.227 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_setxattr": 0, 2024-04-24T08:21:58.227 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_cmpxattr": 0, 2024-04-24T08:21:58.227 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_rmxattr": 0, 2024-04-24T08:21:58.227 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_resetxattrs": 0, 2024-04-24T08:21:58.227 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_call": 0, 2024-04-24T08:21:58.227 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_watch": 0, 2024-04-24T08:21:58.227 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_notify": 0, 2024-04-24T08:21:58.228 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T08:21:58.228 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls": 0, 2024-04-24T08:21:58.228 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls_filter": 0, 2024-04-24T08:21:58.228 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_other": 0, 2024-04-24T08:21:58.228 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_active": 0, 2024-04-24T08:21:58.228 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_send": 0, 2024-04-24T08:21:58.228 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_resend": 0, 2024-04-24T08:21:58.228 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_ping": 0, 2024-04-24T08:21:58.229 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_active": 0, 2024-04-24T08:21:58.229 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_send": 0, 2024-04-24T08:21:58.229 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_resend": 0, 2024-04-24T08:21:58.229 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_active": 0, 2024-04-24T08:21:58.229 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_send": 0, 2024-04-24T08:21:58.229 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_resend": 0, 2024-04-24T08:21:58.229 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_active": 0, 2024-04-24T08:21:58.229 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_send": 0, 2024-04-24T08:21:58.229 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_resend": 0, 2024-04-24T08:21:58.230 INFO:teuthology.orchestra.run.smithi194.stdout: "command_active": 0, 2024-04-24T08:21:58.230 INFO:teuthology.orchestra.run.smithi194.stdout: "command_send": 0, 2024-04-24T08:21:58.230 INFO:teuthology.orchestra.run.smithi194.stdout: "command_resend": 0, 2024-04-24T08:21:58.230 INFO:teuthology.orchestra.run.smithi194.stdout: "map_epoch": 101, 2024-04-24T08:21:58.230 INFO:teuthology.orchestra.run.smithi194.stdout: "map_full": 0, 2024-04-24T08:21:58.230 INFO:teuthology.orchestra.run.smithi194.stdout: "map_inc": 15, 2024-04-24T08:21:58.230 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_sessions": 4, 2024-04-24T08:21:58.230 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_open": 4, 2024-04-24T08:21:58.231 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_close": 0, 2024-04-24T08:21:58.231 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_laggy": 0, 2024-04-24T08:21:58.231 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_wr": 0, 2024-04-24T08:21:58.231 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_rd": 2, 2024-04-24T08:21:58.231 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_del": 0 2024-04-24T08:21:58.231 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.231 INFO:teuthology.orchestra.run.smithi194.stdout: "oft": { 2024-04-24T08:21:58.231 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_objs": 0, 2024-04-24T08:21:58.231 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_kv_pairs": 0, 2024-04-24T08:21:58.231 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_updates": 0, 2024-04-24T08:21:58.232 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_removes": 0 2024-04-24T08:21:58.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.232 INFO:teuthology.orchestra.run.smithi194.stdout: "osd-slow-ops": { 2024-04-24T08:21:58.232 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_ops_count": 0 2024-04-24T08:21:58.232 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.232 INFO:teuthology.orchestra.run.smithi194.stdout: "purge_queue": { 2024-04-24T08:21:58.232 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops": 0, 2024-04-24T08:21:58.232 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T08:21:58.232 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing": 0, 2024-04-24T08:21:58.233 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_high_water": 0, 2024-04-24T08:21:58.233 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed_ops": 0, 2024-04-24T08:21:58.233 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed": 0, 2024-04-24T08:21:58.233 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_item_in_journal": 0 2024-04-24T08:21:58.233 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.233 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:21:58.233 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-04-24T08:21:58.233 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-04-24T08:21:58.233 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-04-24T08:21:58.233 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 2677, 2024-04-24T08:21:58.234 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 2199011, 2024-04-24T08:21:58.234 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:58.234 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 2677, 2024-04-24T08:21:58.234 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-04-24T08:21:58.234 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-04-24T08:21:58.234 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 2677, 2024-04-24T08:21:58.234 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 2199011, 2024-04-24T08:21:58.234 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-04-24T08:21:58.235 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.235 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.235 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.235 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:58.235 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.235 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_bytes": { 2024-04-24T08:21:58.235 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-04-24T08:21:58.235 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-04-24T08:21:58.235 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-04-24T08:21:58.236 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-04-24T08:21:58.236 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-04-24T08:21:58.236 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:58.236 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-04-24T08:21:58.236 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 2407, 2024-04-24T08:21:58.236 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 2890, 2024-04-24T08:21:58.236 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 2407, 2024-04-24T08:21:58.236 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 2890, 2024-04-24T08:21:58.236 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-04-24T08:21:58.237 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.237 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.237 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.237 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:58.237 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.237 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_ops": { 2024-04-24T08:21:58.237 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-04-24T08:21:58.237 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 1024, 2024-04-24T08:21:58.237 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-04-24T08:21:58.237 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-04-24T08:21:58.238 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-04-24T08:21:58.238 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:58.238 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-04-24T08:21:58.238 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 2407, 2024-04-24T08:21:58.238 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 2407, 2024-04-24T08:21:58.238 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 2407, 2024-04-24T08:21:58.238 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 2407, 2024-04-24T08:21:58.238 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-04-24T08:21:58.238 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.239 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.239 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.239 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:58.239 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.239 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle": { 2024-04-24T08:21:58.239 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-04-24T08:21:58.239 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-04-24T08:21:58.239 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-04-24T08:21:58.239 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-04-24T08:21:58.240 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-04-24T08:21:58.240 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:58.240 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-04-24T08:21:58.240 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-04-24T08:21:58.240 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-04-24T08:21:58.240 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 0, 2024-04-24T08:21:58.240 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 0, 2024-04-24T08:21:58.240 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-04-24T08:21:58.240 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.240 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.241 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.241 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:58.241 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:21:58.241 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle-0x55e09cd5a6e0": { 2024-04-24T08:21:58.241 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-04-24T08:21:58.241 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-04-24T08:21:58.241 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-04-24T08:21:58.241 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-04-24T08:21:58.242 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-04-24T08:21:58.242 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-04-24T08:21:58.242 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-04-24T08:21:58.242 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-04-24T08:21:58.242 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-04-24T08:21:58.242 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 0, 2024-04-24T08:21:58.242 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 0, 2024-04-24T08:21:58.242 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-04-24T08:21:58.243 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:21:58.243 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:21:58.243 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:21:58.243 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:58.243 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:21:58.243 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-24T08:21:58.978 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-04-24T08:21:58.978 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-04-24T08:21:58.979 INFO:tasks.check_counter:Daemon mds.f mds.root_rsnaps=0 2024-04-24T08:21:58.979 INFO:tasks.check_counter:Daemon mds.f mds_server.req_mksnap_latency.avgcount=0 2024-04-24T08:21:58.979 INFO:tasks.check_counter:Daemon mds.f mds_server.req_rmsnap_latency.avgcount=0 2024-04-24T08:21:58.979 INFO:tasks.check_counter:Daemon mds.f mds.dir_split=0 2024-04-24T08:21:58.979 DEBUG:tasks.check_counter:Getting stats from i 2024-04-24T08:21:58.979 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph daemon mds.i perf dump 2024-04-24T08:21:59.217 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:21:59.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:21:59 smithi194 ceph-mon[32836]: pgmap v362: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:22:00.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:21:59 smithi028 ceph-mon[27786]: pgmap v362: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:22:00.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:21:59 smithi117 ceph-mon[32438]: pgmap v362: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-04-24T08:22:00.872 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-24T08:22:00.872 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:22:00.872 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 387, 2024-04-24T08:22:00.872 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 510, 2024-04-24T08:22:00.872 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 1799510, 2024-04-24T08:22:00.873 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 682508, 2024-04-24T08:22:00.873 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 18, 2024-04-24T08:22:00.873 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 3, 2024-04-24T08:22:00.873 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.074298197, 2024-04-24T08:22:00.873 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.023630818, 2024-04-24T08:22:00.873 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.051324695, 2024-04-24T08:22:00.873 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.007665538, 2024-04-24T08:22:00.873 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:22:00.873 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 509, 2024-04-24T08:22:00.874 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.058309139, 2024-04-24T08:22:00.874 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000114556 2024-04-24T08:22:00.874 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.874 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:22:00.874 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 21, 2024-04-24T08:22:00.874 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000002003, 2024-04-24T08:22:00.874 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000095 2024-04-24T08:22:00.874 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.875 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 1785120, 2024-04-24T08:22:00.875 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 509312 2024-04-24T08:22:00.875 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.875 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:22:00.875 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 109, 2024-04-24T08:22:00.875 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 175, 2024-04-24T08:22:00.875 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 30616, 2024-04-24T08:22:00.875 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 508970, 2024-04-24T08:22:00.875 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 39, 2024-04-24T08:22:00.876 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 8, 2024-04-24T08:22:00.876 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.028719068, 2024-04-24T08:22:00.876 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.006355014, 2024-04-24T08:22:00.876 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.020729666, 2024-04-24T08:22:00.876 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.000880432, 2024-04-24T08:22:00.876 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:22:00.876 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 173, 2024-04-24T08:22:00.876 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.012372604, 2024-04-24T08:22:00.877 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000071517 2024-04-24T08:22:00.877 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.877 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:22:00.877 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 128, 2024-04-24T08:22:00.877 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000026898, 2024-04-24T08:22:00.877 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000210 2024-04-24T08:22:00.877 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.877 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 18272, 2024-04-24T08:22:00.877 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 161136 2024-04-24T08:22:00.878 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.878 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:22:00.878 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 1328, 2024-04-24T08:22:00.878 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 1030, 2024-04-24T08:22:00.878 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 363234, 2024-04-24T08:22:00.878 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 446622, 2024-04-24T08:22:00.878 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 7, 2024-04-24T08:22:00.879 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 5, 2024-04-24T08:22:00.879 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.100569900, 2024-04-24T08:22:00.879 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.037486872, 2024-04-24T08:22:00.879 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.073859693, 2024-04-24T08:22:00.879 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.006392092, 2024-04-24T08:22:00.879 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:22:00.879 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 1028, 2024-04-24T08:22:00.879 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.068316319, 2024-04-24T08:22:00.880 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000066455 2024-04-24T08:22:00.880 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.880 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:22:00.880 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2143, 2024-04-24T08:22:00.880 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000564600, 2024-04-24T08:22:00.880 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000263 2024-04-24T08:22:00.880 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.880 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 1728, 2024-04-24T08:22:00.881 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 6160 2024-04-24T08:22:00.881 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.881 INFO:teuthology.orchestra.run.smithi194.stdout: "cct": { 2024-04-24T08:22:00.881 INFO:teuthology.orchestra.run.smithi194.stdout: "total_workers": 1, 2024-04-24T08:22:00.881 INFO:teuthology.orchestra.run.smithi194.stdout: "unhealthy_workers": 0 2024-04-24T08:22:00.881 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.881 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-MDSRank": { 2024-04-24T08:22:00.881 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-04-24T08:22:00.881 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-04-24T08:22:00.882 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 151, 2024-04-24T08:22:00.882 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.008126597, 2024-04-24T08:22:00.882 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000053818 2024-04-24T08:22:00.882 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:22:00.882 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.882 INFO:teuthology.orchestra.run.smithi194.stdout: "finisher-PurgeQueue": { 2024-04-24T08:22:00.882 INFO:teuthology.orchestra.run.smithi194.stdout: "queue_len": 0, 2024-04-24T08:22:00.882 INFO:teuthology.orchestra.run.smithi194.stdout: "complete_latency": { 2024-04-24T08:22:00.883 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2, 2024-04-24T08:22:00.883 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000011385, 2024-04-24T08:22:00.883 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000005692 2024-04-24T08:22:00.883 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:22:00.883 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.883 INFO:teuthology.orchestra.run.smithi194.stdout: "mds": { 2024-04-24T08:22:00.883 INFO:teuthology.orchestra.run.smithi194.stdout: "request": 23, 2024-04-24T08:22:00.883 INFO:teuthology.orchestra.run.smithi194.stdout: "reply": 8, 2024-04-24T08:22:00.884 INFO:teuthology.orchestra.run.smithi194.stdout: "reply_latency": { 2024-04-24T08:22:00.884 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 8, 2024-04-24T08:22:00.884 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.006260425, 2024-04-24T08:22:00.884 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000782553 2024-04-24T08:22:00.884 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.884 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_reply": 0, 2024-04-24T08:22:00.884 INFO:teuthology.orchestra.run.smithi194.stdout: "forward": 15, 2024-04-24T08:22:00.884 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_complete": 0, 2024-04-24T08:22:00.884 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_fetch_keys": 0, 2024-04-24T08:22:00.885 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_commit": 11, 2024-04-24T08:22:00.885 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_split": 0, 2024-04-24T08:22:00.885 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_merge": 0, 2024-04-24T08:22:00.885 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes": 32, 2024-04-24T08:22:00.885 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_top": 0, 2024-04-24T08:22:00.885 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_bottom": 32, 2024-04-24T08:22:00.885 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pin_tail": 0, 2024-04-24T08:22:00.885 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_pinned": 32, 2024-04-24T08:22:00.886 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_expired": 1, 2024-04-24T08:22:00.886 INFO:teuthology.orchestra.run.smithi194.stdout: "inodes_with_caps": 8, 2024-04-24T08:22:00.886 INFO:teuthology.orchestra.run.smithi194.stdout: "caps": 8, 2024-04-24T08:22:00.886 INFO:teuthology.orchestra.run.smithi194.stdout: "subtrees": 5, 2024-04-24T08:22:00.886 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse": 34, 2024-04-24T08:22:00.886 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_hit": 10, 2024-04-24T08:22:00.886 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_forward": 13, 2024-04-24T08:22:00.886 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_discover": 7, 2024-04-24T08:22:00.887 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_dir_fetch": 0, 2024-04-24T08:22:00.887 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_remote_ino": 0, 2024-04-24T08:22:00.887 INFO:teuthology.orchestra.run.smithi194.stdout: "traverse_lock": 0, 2024-04-24T08:22:00.887 INFO:teuthology.orchestra.run.smithi194.stdout: "load_cent": 0, 2024-04-24T08:22:00.887 INFO:teuthology.orchestra.run.smithi194.stdout: "q": 0, 2024-04-24T08:22:00.887 INFO:teuthology.orchestra.run.smithi194.stdout: "exported": 0, 2024-04-24T08:22:00.887 INFO:teuthology.orchestra.run.smithi194.stdout: "exported_inodes": 0, 2024-04-24T08:22:00.887 INFO:teuthology.orchestra.run.smithi194.stdout: "imported": 3, 2024-04-24T08:22:00.888 INFO:teuthology.orchestra.run.smithi194.stdout: "imported_inodes": 9, 2024-04-24T08:22:00.888 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_dir_fetch": 0, 2024-04-24T08:22:00.888 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_backtrace_fetch": 60, 2024-04-24T08:22:00.888 INFO:teuthology.orchestra.run.smithi194.stdout: "openino_peer_discover": 0, 2024-04-24T08:22:00.888 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rfiles": 325, 2024-04-24T08:22:00.888 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rbytes": 13424230511, 2024-04-24T08:22:00.888 INFO:teuthology.orchestra.run.smithi194.stdout: "root_rsnaps": 6, 2024-04-24T08:22:00.888 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_fetch": 60, 2024-04-24T08:22:00.888 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_set_tag": 0, 2024-04-24T08:22:00.889 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_backtrace_repaired": 0, 2024-04-24T08:22:00.889 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_inotable_repaired": 0, 2024-04-24T08:22:00.889 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_inodes": 12, 2024-04-24T08:22:00.889 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dir_base_inodes": 0, 2024-04-24T08:22:00.889 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_dirfrag_rstats": 12, 2024-04-24T08:22:00.889 INFO:teuthology.orchestra.run.smithi194.stdout: "scrub_file_inodes": 48, 2024-04-24T08:22:00.889 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_inode_file_caps": 0, 2024-04-24T08:22:00.889 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_revoke": 0, 2024-04-24T08:22:00.890 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_grant": 1, 2024-04-24T08:22:00.890 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_trunc": 0, 2024-04-24T08:22:00.890 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-04-24T08:22:00.890 INFO:teuthology.orchestra.run.smithi194.stdout: "ceph_cap_op_flush_ack": 0, 2024-04-24T08:22:00.890 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps": 8, 2024-04-24T08:22:00.890 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_caps_dirty": 1, 2024-04-24T08:22:00.890 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_cap_release": 0, 2024-04-24T08:22:00.890 INFO:teuthology.orchestra.run.smithi194.stdout: "process_request_cap_release": 3 2024-04-24T08:22:00.891 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.891 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_cache": { 2024-04-24T08:22:00.891 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update": 0, 2024-04-24T08:22:00.891 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_update_receipt": 0, 2024-04-24T08:22:00.891 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_try_discover": 0, 2024-04-24T08:22:00.891 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_send_discover": 6, 2024-04-24T08:22:00.891 INFO:teuthology.orchestra.run.smithi194.stdout: "dir_handle_discover": 0, 2024-04-24T08:22:00.891 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays": 0, 2024-04-24T08:22:00.892 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_delayed": 0, 2024-04-24T08:22:00.892 INFO:teuthology.orchestra.run.smithi194.stdout: "num_strays_enqueuing": 0, 2024-04-24T08:22:00.892 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_created": 0, 2024-04-24T08:22:00.892 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_enqueued": 0, 2024-04-24T08:22:00.892 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_reintegrated": 0, 2024-04-24T08:22:00.892 INFO:teuthology.orchestra.run.smithi194.stdout: "strays_migrated": 0, 2024-04-24T08:22:00.892 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_processing": 0, 2024-04-24T08:22:00.892 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_enqueued": 0, 2024-04-24T08:22:00.892 INFO:teuthology.orchestra.run.smithi194.stdout: "num_recovering_prioritized": 0, 2024-04-24T08:22:00.893 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_started": 0, 2024-04-24T08:22:00.893 INFO:teuthology.orchestra.run.smithi194.stdout: "recovery_completed": 0, 2024-04-24T08:22:00.893 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_path": 0, 2024-04-24T08:22:00.893 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_quiesce_inode": 0, 2024-04-24T08:22:00.893 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_enqueue_scrub": 0, 2024-04-24T08:22:00.893 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_exportdir": 0, 2024-04-24T08:22:00.893 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_flush": 0, 2024-04-24T08:22:00.893 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragmentdir": 0, 2024-04-24T08:22:00.893 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_fragstats": 0, 2024-04-24T08:22:00.894 INFO:teuthology.orchestra.run.smithi194.stdout: "ireq_inodestats": 0 2024-04-24T08:22:00.894 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.894 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_log": { 2024-04-24T08:22:00.894 INFO:teuthology.orchestra.run.smithi194.stdout: "evlrg": 0, 2024-04-24T08:22:00.894 INFO:teuthology.orchestra.run.smithi194.stdout: "evadd": 26, 2024-04-24T08:22:00.894 INFO:teuthology.orchestra.run.smithi194.stdout: "evex": 0, 2024-04-24T08:22:00.894 INFO:teuthology.orchestra.run.smithi194.stdout: "evtrm": 0, 2024-04-24T08:22:00.894 INFO:teuthology.orchestra.run.smithi194.stdout: "ev": 26, 2024-04-24T08:22:00.895 INFO:teuthology.orchestra.run.smithi194.stdout: "evexg": 0, 2024-04-24T08:22:00.895 INFO:teuthology.orchestra.run.smithi194.stdout: "evexd": 0, 2024-04-24T08:22:00.895 INFO:teuthology.orchestra.run.smithi194.stdout: "segadd": 2, 2024-04-24T08:22:00.895 INFO:teuthology.orchestra.run.smithi194.stdout: "segex": 0, 2024-04-24T08:22:00.895 INFO:teuthology.orchestra.run.smithi194.stdout: "segtrm": 0, 2024-04-24T08:22:00.895 INFO:teuthology.orchestra.run.smithi194.stdout: "seg": 2, 2024-04-24T08:22:00.895 INFO:teuthology.orchestra.run.smithi194.stdout: "segmjr": 2, 2024-04-24T08:22:00.895 INFO:teuthology.orchestra.run.smithi194.stdout: "segexg": 0, 2024-04-24T08:22:00.896 INFO:teuthology.orchestra.run.smithi194.stdout: "segexd": 0, 2024-04-24T08:22:00.896 INFO:teuthology.orchestra.run.smithi194.stdout: "expos": 4194304, 2024-04-24T08:22:00.896 INFO:teuthology.orchestra.run.smithi194.stdout: "wrpos": 4254781, 2024-04-24T08:22:00.896 INFO:teuthology.orchestra.run.smithi194.stdout: "rdpos": 0, 2024-04-24T08:22:00.896 INFO:teuthology.orchestra.run.smithi194.stdout: "jlat": { 2024-04-24T08:22:00.896 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 18, 2024-04-24T08:22:00.896 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.036827743, 2024-04-24T08:22:00.896 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.002045985 2024-04-24T08:22:00.896 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.897 INFO:teuthology.orchestra.run.smithi194.stdout: "replayed": 0 2024-04-24T08:22:00.897 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.897 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_mem": { 2024-04-24T08:22:00.897 INFO:teuthology.orchestra.run.smithi194.stdout: "ino": 31, 2024-04-24T08:22:00.897 INFO:teuthology.orchestra.run.smithi194.stdout: "ino+": 31, 2024-04-24T08:22:00.897 INFO:teuthology.orchestra.run.smithi194.stdout: "ino-": 0, 2024-04-24T08:22:00.897 INFO:teuthology.orchestra.run.smithi194.stdout: "dir": 26, 2024-04-24T08:22:00.897 INFO:teuthology.orchestra.run.smithi194.stdout: "dir+": 27, 2024-04-24T08:22:00.898 INFO:teuthology.orchestra.run.smithi194.stdout: "dir-": 1, 2024-04-24T08:22:00.898 INFO:teuthology.orchestra.run.smithi194.stdout: "dn": 32, 2024-04-24T08:22:00.898 INFO:teuthology.orchestra.run.smithi194.stdout: "dn+": 33, 2024-04-24T08:22:00.898 INFO:teuthology.orchestra.run.smithi194.stdout: "dn-": 1, 2024-04-24T08:22:00.898 INFO:teuthology.orchestra.run.smithi194.stdout: "cap": 8, 2024-04-24T08:22:00.898 INFO:teuthology.orchestra.run.smithi194.stdout: "cap+": 11, 2024-04-24T08:22:00.898 INFO:teuthology.orchestra.run.smithi194.stdout: "cap-": 3, 2024-04-24T08:22:00.898 INFO:teuthology.orchestra.run.smithi194.stdout: "rss": 53220, 2024-04-24T08:22:00.899 INFO:teuthology.orchestra.run.smithi194.stdout: "heap": 223608 2024-04-24T08:22:00.899 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.899 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_server": { 2024-04-24T08:22:00.899 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_client_request": 28, 2024-04-24T08:22:00.899 INFO:teuthology.orchestra.run.smithi194.stdout: "dispatch_server_request": 0, 2024-04-24T08:22:00.899 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_request": 23, 2024-04-24T08:22:00.899 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_client_session": 30, 2024-04-24T08:22:00.899 INFO:teuthology.orchestra.run.smithi194.stdout: "handle_peer_request": 1, 2024-04-24T08:22:00.899 INFO:teuthology.orchestra.run.smithi194.stdout: "req_create_latency": { 2024-04-24T08:22:00.900 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.900 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.900 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.900 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.900 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getattr_latency": { 2024-04-24T08:22:00.900 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2, 2024-04-24T08:22:00.900 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.003085124, 2024-04-24T08:22:00.900 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.001542562 2024-04-24T08:22:00.901 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.901 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getfilelock_latency": { 2024-04-24T08:22:00.901 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.901 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.901 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.901 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.901 INFO:teuthology.orchestra.run.smithi194.stdout: "req_link_latency": { 2024-04-24T08:22:00.901 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.901 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.902 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.902 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.902 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookup_latency": { 2024-04-24T08:22:00.902 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 2, 2024-04-24T08:22:00.902 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.001578690, 2024-04-24T08:22:00.902 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000789345 2024-04-24T08:22:00.903 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.903 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookuphash_latency": { 2024-04-24T08:22:00.903 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.903 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.903 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.903 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.903 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupino_latency": { 2024-04-24T08:22:00.903 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.903 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.904 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.904 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.904 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupname_latency": { 2024-04-24T08:22:00.904 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.904 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.904 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.904 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.904 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupparent_latency": { 2024-04-24T08:22:00.904 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.905 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.905 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.905 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.905 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lookupsnap_latency": { 2024-04-24T08:22:00.905 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.905 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.905 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.905 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.905 INFO:teuthology.orchestra.run.smithi194.stdout: "req_lssnap_latency": { 2024-04-24T08:22:00.906 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.906 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.906 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.906 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.906 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mkdir_latency": { 2024-04-24T08:22:00.907 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.907 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.907 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.907 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.907 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mknod_latency": { 2024-04-24T08:22:00.907 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.907 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.907 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.908 INFO:teuthology.orchestra.run.smithi194.stdout: "req_mksnap_latency": { 2024-04-24T08:22:00.908 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.908 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.908 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.908 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.908 INFO:teuthology.orchestra.run.smithi194.stdout: "req_open_latency": { 2024-04-24T08:22:00.908 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.908 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.909 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.909 INFO:teuthology.orchestra.run.smithi194.stdout: "req_readdir_latency": { 2024-04-24T08:22:00.909 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 4, 2024-04-24T08:22:00.909 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.001596611, 2024-04-24T08:22:00.909 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000399152 2024-04-24T08:22:00.909 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.909 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rename_latency": { 2024-04-24T08:22:00.909 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.910 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.910 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.910 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.910 INFO:teuthology.orchestra.run.smithi194.stdout: "req_renamesnap_latency": { 2024-04-24T08:22:00.910 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.910 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.910 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.910 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.910 INFO:teuthology.orchestra.run.smithi194.stdout: "req_snapdiff_latency": { 2024-04-24T08:22:00.910 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.911 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.911 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.911 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmdir_latency": { 2024-04-24T08:22:00.911 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.911 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.911 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.911 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.911 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmsnap_latency": { 2024-04-24T08:22:00.912 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.912 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.912 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.912 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.912 INFO:teuthology.orchestra.run.smithi194.stdout: "req_rmxattr_latency": { 2024-04-24T08:22:00.912 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.912 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.912 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.912 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.913 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setattr_latency": { 2024-04-24T08:22:00.913 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.913 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.913 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.913 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.913 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setdirlayout_latency": { 2024-04-24T08:22:00.913 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.913 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.914 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.914 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.914 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setfilelock_latency": { 2024-04-24T08:22:00.914 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.914 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.914 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.914 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.914 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setlayout_latency": { 2024-04-24T08:22:00.914 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.915 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.915 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.915 INFO:teuthology.orchestra.run.smithi194.stdout: "req_setxattr_latency": { 2024-04-24T08:22:00.915 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.915 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.915 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.915 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.915 INFO:teuthology.orchestra.run.smithi194.stdout: "req_symlink_latency": { 2024-04-24T08:22:00.916 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.916 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.916 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.916 INFO:teuthology.orchestra.run.smithi194.stdout: "req_unlink_latency": { 2024-04-24T08:22:00.916 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.916 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.916 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.916 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.917 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_revoke_eviction": 0, 2024-04-24T08:22:00.917 INFO:teuthology.orchestra.run.smithi194.stdout: "cap_acquisition_throttle": 0, 2024-04-24T08:22:00.917 INFO:teuthology.orchestra.run.smithi194.stdout: "req_getvxattr_latency": { 2024-04-24T08:22:00.917 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.917 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.917 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.917 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:22:00.917 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.917 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_sessions": { 2024-04-24T08:22:00.917 INFO:teuthology.orchestra.run.smithi194.stdout: "session_count": 2, 2024-04-24T08:22:00.918 INFO:teuthology.orchestra.run.smithi194.stdout: "session_add": 5, 2024-04-24T08:22:00.918 INFO:teuthology.orchestra.run.smithi194.stdout: "session_remove": 3, 2024-04-24T08:22:00.918 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_open": 2, 2024-04-24T08:22:00.918 INFO:teuthology.orchestra.run.smithi194.stdout: "sessions_stale": 0, 2024-04-24T08:22:00.918 INFO:teuthology.orchestra.run.smithi194.stdout: "total_load": 4, 2024-04-24T08:22:00.918 INFO:teuthology.orchestra.run.smithi194.stdout: "average_load": 2, 2024-04-24T08:22:00.918 INFO:teuthology.orchestra.run.smithi194.stdout: "avg_session_uptime": 658, 2024-04-24T08:22:00.918 INFO:teuthology.orchestra.run.smithi194.stdout: "mdthresh_evicted": 0 2024-04-24T08:22:00.918 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.919 INFO:teuthology.orchestra.run.smithi194.stdout: "mempool": { 2024-04-24T08:22:00.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:22:00.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_items": 0, 2024-04-24T08:22:00.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:22:00.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:22:00.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:22:00.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:22:00.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:22:00.919 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:22:00.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:22:00.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:22:00.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:22:00.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:22:00.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:22:00.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:22:00.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:22:00.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_items": 0, 2024-04-24T08:22:00.920 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:22:00.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_items": 0, 2024-04-24T08:22:00.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:22:00.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:22:00.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:22:00.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:22:00.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:22:00.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:22:00.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:22:00.921 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_items": 0, 2024-04-24T08:22:00.922 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:22:00.922 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:22:00.922 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:22:00.922 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_items": 0, 2024-04-24T08:22:00.922 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_bytes": 0, 2024-04-24T08:22:00.922 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_items": 0, 2024-04-24T08:22:00.922 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:22:00.923 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:22:00.923 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:22:00.923 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:22:00.923 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_bytes": 284679, 2024-04-24T08:22:00.923 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_items": 137, 2024-04-24T08:22:00.923 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:22:00.923 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_items": 0, 2024-04-24T08:22:00.923 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_bytes": 0, 2024-04-24T08:22:00.923 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_items": 0, 2024-04-24T08:22:00.924 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:22:00.924 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_items": 0, 2024-04-24T08:22:00.924 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:22:00.924 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_items": 0, 2024-04-24T08:22:00.924 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_bytes": 8320, 2024-04-24T08:22:00.924 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_items": 173, 2024-04-24T08:22:00.924 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:22:00.924 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:22:00.924 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_bytes": 0, 2024-04-24T08:22:00.925 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_items": 0, 2024-04-24T08:22:00.925 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_bytes": 139680, 2024-04-24T08:22:00.925 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_items": 1327, 2024-04-24T08:22:00.925 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_bytes": 0, 2024-04-24T08:22:00.925 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_items": 0, 2024-04-24T08:22:00.925 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_bytes": 0, 2024-04-24T08:22:00.925 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_items": 0 2024-04-24T08:22:00.925 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.925 INFO:teuthology.orchestra.run.smithi194.stdout: "objecter": { 2024-04-24T08:22:00.926 INFO:teuthology.orchestra.run.smithi194.stdout: "op_active": 0, 2024-04-24T08:22:00.926 INFO:teuthology.orchestra.run.smithi194.stdout: "op_laggy": 0, 2024-04-24T08:22:00.926 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send": 124, 2024-04-24T08:22:00.926 INFO:teuthology.orchestra.run.smithi194.stdout: "op_send_bytes": 73305, 2024-04-24T08:22:00.926 INFO:teuthology.orchestra.run.smithi194.stdout: "op_resend": 28, 2024-04-24T08:22:00.926 INFO:teuthology.orchestra.run.smithi194.stdout: "op_reply": 124, 2024-04-24T08:22:00.926 INFO:teuthology.orchestra.run.smithi194.stdout: "op_latency": { 2024-04-24T08:22:00.926 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 124, 2024-04-24T08:22:00.926 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.757984534, 2024-04-24T08:22:00.927 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.006112778 2024-04-24T08:22:00.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.927 INFO:teuthology.orchestra.run.smithi194.stdout: "op_inflight": 0, 2024-04-24T08:22:00.927 INFO:teuthology.orchestra.run.smithi194.stdout: "oplen_avg": { 2024-04-24T08:22:00.927 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 124, 2024-04-24T08:22:00.927 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 151 2024-04-24T08:22:00.927 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.927 INFO:teuthology.orchestra.run.smithi194.stdout: "op": 124, 2024-04-24T08:22:00.927 INFO:teuthology.orchestra.run.smithi194.stdout: "op_r": 60, 2024-04-24T08:22:00.927 INFO:teuthology.orchestra.run.smithi194.stdout: "op_w": 64, 2024-04-24T08:22:00.928 INFO:teuthology.orchestra.run.smithi194.stdout: "op_rmw": 0, 2024-04-24T08:22:00.928 INFO:teuthology.orchestra.run.smithi194.stdout: "op_pg": 0, 2024-04-24T08:22:00.928 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_stat": 0, 2024-04-24T08:22:00.928 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_create": 10, 2024-04-24T08:22:00.928 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_read": 0, 2024-04-24T08:22:00.928 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_write": 18, 2024-04-24T08:22:00.928 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writefull": 12, 2024-04-24T08:22:00.928 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_writesame": 0, 2024-04-24T08:22:00.929 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_append": 0, 2024-04-24T08:22:00.929 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_zero": 0, 2024-04-24T08:22:00.929 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_truncate": 0, 2024-04-24T08:22:00.929 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_delete": 6, 2024-04-24T08:22:00.929 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_mapext": 0, 2024-04-24T08:22:00.929 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_sparse_read": 0, 2024-04-24T08:22:00.929 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_clonerange": 0, 2024-04-24T08:22:00.929 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_getxattr": 60, 2024-04-24T08:22:00.929 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_setxattr": 20, 2024-04-24T08:22:00.930 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_cmpxattr": 0, 2024-04-24T08:22:00.930 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_rmxattr": 0, 2024-04-24T08:22:00.930 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_resetxattrs": 0, 2024-04-24T08:22:00.930 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_call": 0, 2024-04-24T08:22:00.930 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_watch": 0, 2024-04-24T08:22:00.930 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_notify": 0, 2024-04-24T08:22:00.930 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_src_cmpxattr": 0, 2024-04-24T08:22:00.930 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls": 0, 2024-04-24T08:22:00.930 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_pgls_filter": 0, 2024-04-24T08:22:00.931 INFO:teuthology.orchestra.run.smithi194.stdout: "osdop_other": 0, 2024-04-24T08:22:00.931 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_active": 0, 2024-04-24T08:22:00.931 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_send": 0, 2024-04-24T08:22:00.931 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_resend": 0, 2024-04-24T08:22:00.931 INFO:teuthology.orchestra.run.smithi194.stdout: "linger_ping": 0, 2024-04-24T08:22:00.931 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_active": 0, 2024-04-24T08:22:00.931 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_send": 0, 2024-04-24T08:22:00.931 INFO:teuthology.orchestra.run.smithi194.stdout: "poolop_resend": 0, 2024-04-24T08:22:00.931 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_active": 0, 2024-04-24T08:22:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_send": 0, 2024-04-24T08:22:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "poolstat_resend": 0, 2024-04-24T08:22:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_active": 0, 2024-04-24T08:22:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_send": 0, 2024-04-24T08:22:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "statfs_resend": 0, 2024-04-24T08:22:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "command_active": 0, 2024-04-24T08:22:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "command_send": 0, 2024-04-24T08:22:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "command_resend": 0, 2024-04-24T08:22:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "map_epoch": 101, 2024-04-24T08:22:00.932 INFO:teuthology.orchestra.run.smithi194.stdout: "map_full": 0, 2024-04-24T08:22:00.933 INFO:teuthology.orchestra.run.smithi194.stdout: "map_inc": 15, 2024-04-24T08:22:00.933 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_sessions": 10, 2024-04-24T08:22:00.933 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_open": 10, 2024-04-24T08:22:00.933 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_session_close": 0, 2024-04-24T08:22:00.933 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_laggy": 0, 2024-04-24T08:22:00.933 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_wr": 22, 2024-04-24T08:22:00.933 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_rd": 0, 2024-04-24T08:22:00.933 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_del": 3 2024-04-24T08:22:00.934 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.934 INFO:teuthology.orchestra.run.smithi194.stdout: "oft": { 2024-04-24T08:22:00.934 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_objs": 1, 2024-04-24T08:22:00.934 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_kv_pairs": 0, 2024-04-24T08:22:00.934 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_updates": 0, 2024-04-24T08:22:00.934 INFO:teuthology.orchestra.run.smithi194.stdout: "omap_total_removes": 0 2024-04-24T08:22:00.934 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.934 INFO:teuthology.orchestra.run.smithi194.stdout: "osd-slow-ops": { 2024-04-24T08:22:00.934 INFO:teuthology.orchestra.run.smithi194.stdout: "slow_ops_count": 0 2024-04-24T08:22:00.935 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.935 INFO:teuthology.orchestra.run.smithi194.stdout: "purge_queue": { 2024-04-24T08:22:00.935 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops": 0, 2024-04-24T08:22:00.935 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_ops_high_water": 0, 2024-04-24T08:22:00.935 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing": 0, 2024-04-24T08:22:00.935 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executing_high_water": 0, 2024-04-24T08:22:00.935 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed_ops": 0, 2024-04-24T08:22:00.935 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_executed": 0, 2024-04-24T08:22:00.935 INFO:teuthology.orchestra.run.smithi194.stdout: "pq_item_in_journal": 0 2024-04-24T08:22:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:22:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-04-24T08:22:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-04-24T08:22:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-04-24T08:22:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 1820, 2024-04-24T08:22:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 2007907, 2024-04-24T08:22:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-04-24T08:22:00.936 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 1820, 2024-04-24T08:22:00.937 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-04-24T08:22:00.937 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-04-24T08:22:00.937 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 1820, 2024-04-24T08:22:00.937 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 2007907, 2024-04-24T08:22:00.937 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-04-24T08:22:00.937 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.937 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.937 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.937 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:22:00.937 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.938 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_bytes": { 2024-04-24T08:22:00.938 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-04-24T08:22:00.938 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-04-24T08:22:00.938 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-04-24T08:22:00.938 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-04-24T08:22:00.938 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-04-24T08:22:00.938 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-04-24T08:22:00.938 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-04-24T08:22:00.939 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 124, 2024-04-24T08:22:00.939 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 73305, 2024-04-24T08:22:00.939 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 124, 2024-04-24T08:22:00.939 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 73305, 2024-04-24T08:22:00.939 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-04-24T08:22:00.939 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.939 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.939 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.939 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:22:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-objecter_ops": { 2024-04-24T08:22:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-04-24T08:22:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 1024, 2024-04-24T08:22:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-04-24T08:22:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-04-24T08:22:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-04-24T08:22:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-04-24T08:22:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-04-24T08:22:00.940 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 124, 2024-04-24T08:22:00.941 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 124, 2024-04-24T08:22:00.941 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 124, 2024-04-24T08:22:00.941 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 124, 2024-04-24T08:22:00.941 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-04-24T08:22:00.941 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.941 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.941 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.941 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:22:00.941 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.942 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle": { 2024-04-24T08:22:00.942 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-04-24T08:22:00.942 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-04-24T08:22:00.942 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-04-24T08:22:00.942 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 0, 2024-04-24T08:22:00.942 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 0, 2024-04-24T08:22:00.942 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-04-24T08:22:00.942 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 0, 2024-04-24T08:22:00.942 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-04-24T08:22:00.943 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-04-24T08:22:00.943 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 0, 2024-04-24T08:22:00.943 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 0, 2024-04-24T08:22:00.943 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-04-24T08:22:00.943 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.943 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.943 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.943 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:22:00.944 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:00.944 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-write_buf_throttle-0x560311a0a500": { 2024-04-24T08:22:00.944 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-04-24T08:22:00.944 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 3758096384, 2024-04-24T08:22:00.944 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-04-24T08:22:00.944 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 26, 2024-04-24T08:22:00.944 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 60477, 2024-04-24T08:22:00.944 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-04-24T08:22:00.944 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 26, 2024-04-24T08:22:00.945 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-04-24T08:22:00.945 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-04-24T08:22:00.945 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 18, 2024-04-24T08:22:00.945 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 60477, 2024-04-24T08:22:00.945 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-04-24T08:22:00.945 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:00.945 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:00.945 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:00.945 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:22:00.946 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:22:00.946 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-24T08:22:01.693 INFO:tasks.check_counter:Daemon mds.i mds.exported=0 2024-04-24T08:22:01.693 INFO:tasks.check_counter:Daemon mds.i mds.imported=3 2024-04-24T08:22:01.693 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-04-24T08:22:01.693 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-04-24T08:22:01.693 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-04-24T08:22:01.693 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-04-24T08:22:01.693 DEBUG:tasks.check_counter:Getting stats from l 2024-04-24T08:22:01.694 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0 shell --fsid 473aaf58-0210-11ef-bc93-c7b262605968 -- ceph daemon mds.l perf dump 2024-04-24T08:22:01.936 INFO:teuthology.orchestra.run.smithi194.stderr:Inferring config /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/mon.c/config 2024-04-24T08:22:01.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:01 smithi194 ceph-mon[32836]: pgmap v363: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:22:02.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:01 smithi028 ceph-mon[27786]: pgmap v363: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:22:02.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:01 smithi117 ceph-mon[32438]: pgmap v363: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-04-24T08:22:03.624 INFO:teuthology.orchestra.run.smithi194.stdout:{ 2024-04-24T08:22:03.625 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-0": { 2024-04-24T08:22:03.625 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 4, 2024-04-24T08:22:03.625 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 4, 2024-04-24T08:22:03.625 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 3024, 2024-04-24T08:22:03.625 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 448, 2024-04-24T08:22:03.625 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 4, 2024-04-24T08:22:03.625 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 0, 2024-04-24T08:22:03.625 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.001260864, 2024-04-24T08:22:03.626 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.000030681, 2024-04-24T08:22:03.626 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.000509395, 2024-04-24T08:22:03.626 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:22:03.626 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:22:03.626 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 3, 2024-04-24T08:22:03.626 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000651979, 2024-04-24T08:22:03.626 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000217326 2024-04-24T08:22:03.626 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:03.627 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:22:03.627 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:03.627 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:03.627 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:03.627 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:03.627 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-04-24T08:22:03.627 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 448 2024-04-24T08:22:03.627 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:03.627 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-1": { 2024-04-24T08:22:03.627 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 250, 2024-04-24T08:22:03.628 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 238, 2024-04-24T08:22:03.628 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 1736048, 2024-04-24T08:22:03.628 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 147104, 2024-04-24T08:22:03.628 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 2, 2024-04-24T08:22:03.628 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 1, 2024-04-24T08:22:03.628 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.048469273, 2024-04-24T08:22:03.628 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.014205922, 2024-04-24T08:22:03.628 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.039399680, 2024-04-24T08:22:03.628 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.005954176, 2024-04-24T08:22:03.629 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:22:03.629 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 237, 2024-04-24T08:22:03.629 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.014141548, 2024-04-24T08:22:03.629 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000059668 2024-04-24T08:22:03.629 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:03.629 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:22:03.629 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:03.629 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:03.629 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:03.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:03.630 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 1736048, 2024-04-24T08:22:03.630 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 147104 2024-04-24T08:22:03.630 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:03.630 INFO:teuthology.orchestra.run.smithi194.stdout: "AsyncMessenger::Worker-2": { 2024-04-24T08:22:03.630 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_messages": 2, 2024-04-24T08:22:03.630 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages": 184, 2024-04-24T08:22:03.630 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_bytes": 320, 2024-04-24T08:22:03.630 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_bytes": 267104, 2024-04-24T08:22:03.631 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_created_connections": 6, 2024-04-24T08:22:03.631 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_active_connections": 3, 2024-04-24T08:22:03.631 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_total_time": 0.011687181, 2024-04-24T08:22:03.631 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_send_time": 0.007896331, 2024-04-24T08:22:03.631 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_recv_time": 0.001501692, 2024-04-24T08:22:03.631 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-04-24T08:22:03.631 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_messages_queue_lat": { 2024-04-24T08:22:03.631 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 181, 2024-04-24T08:22:03.631 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.035587910, 2024-04-24T08:22:03.632 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000196618 2024-04-24T08:22:03.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:03.632 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_handle_ack_lat": { 2024-04-24T08:22:03.632 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:03.632 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:03.632 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:03.632 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:03.632 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_recv_encrypted_bytes": 320, 2024-04-24T08:22:03.632 INFO:teuthology.orchestra.run.smithi194.stdout: "msgr_send_encrypted_bytes": 267104 2024-04-24T08:22:03.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:03.633 INFO:teuthology.orchestra.run.smithi194.stdout: "cct": { 2024-04-24T08:22:03.633 INFO:teuthology.orchestra.run.smithi194.stdout: "total_workers": 0, 2024-04-24T08:22:03.633 INFO:teuthology.orchestra.run.smithi194.stdout: "unhealthy_workers": 0 2024-04-24T08:22:03.633 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:03.633 INFO:teuthology.orchestra.run.smithi194.stdout: "mempool": { 2024-04-24T08:22:03.633 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_bytes": 0, 2024-04-24T08:22:03.633 INFO:teuthology.orchestra.run.smithi194.stdout: "bloom_filter_items": 0, 2024-04-24T08:22:03.633 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_bytes": 0, 2024-04-24T08:22:03.634 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_alloc_items": 0, 2024-04-24T08:22:03.634 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_bytes": 0, 2024-04-24T08:22:03.634 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_data_items": 0, 2024-04-24T08:22:03.634 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_bytes": 0, 2024-04-24T08:22:03.634 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_onode_items": 0, 2024-04-24T08:22:03.634 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_bytes": 0, 2024-04-24T08:22:03.634 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_meta_items": 0, 2024-04-24T08:22:03.634 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_bytes": 0, 2024-04-24T08:22:03.635 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_other_items": 0, 2024-04-24T08:22:03.635 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_bytes": 0, 2024-04-24T08:22:03.635 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_cache_buffer_items": 0, 2024-04-24T08:22:03.635 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_bytes": 0, 2024-04-24T08:22:03.635 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_extent_items": 0, 2024-04-24T08:22:03.635 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_bytes": 0, 2024-04-24T08:22:03.635 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_blob_items": 0, 2024-04-24T08:22:03.635 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_bytes": 0, 2024-04-24T08:22:03.636 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_shared_blob_items": 0, 2024-04-24T08:22:03.636 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_bytes": 0, 2024-04-24T08:22:03.636 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_inline_bl_items": 0, 2024-04-24T08:22:03.636 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_bytes": 0, 2024-04-24T08:22:03.636 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_fsck_items": 0, 2024-04-24T08:22:03.636 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_bytes": 0, 2024-04-24T08:22:03.636 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_txc_items": 0, 2024-04-24T08:22:03.636 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_bytes": 0, 2024-04-24T08:22:03.636 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_deferred_items": 0, 2024-04-24T08:22:03.637 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_bytes": 0, 2024-04-24T08:22:03.637 INFO:teuthology.orchestra.run.smithi194.stdout: "bluestore_writing_items": 0, 2024-04-24T08:22:03.637 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_bytes": 0, 2024-04-24T08:22:03.637 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_items": 0, 2024-04-24T08:22:03.637 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_bytes": 0, 2024-04-24T08:22:03.637 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_reader_items": 0, 2024-04-24T08:22:03.637 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_bytes": 0, 2024-04-24T08:22:03.637 INFO:teuthology.orchestra.run.smithi194.stdout: "bluefs_file_writer_items": 0, 2024-04-24T08:22:03.637 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_bytes": 148692, 2024-04-24T08:22:03.638 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_anon_items": 30, 2024-04-24T08:22:03.638 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_bytes": 0, 2024-04-24T08:22:03.638 INFO:teuthology.orchestra.run.smithi194.stdout: "buffer_meta_items": 0, 2024-04-24T08:22:03.638 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_bytes": 0, 2024-04-24T08:22:03.638 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_items": 0, 2024-04-24T08:22:03.638 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_bytes": 0, 2024-04-24T08:22:03.638 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_mapbl_items": 0, 2024-04-24T08:22:03.638 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_bytes": 0, 2024-04-24T08:22:03.639 INFO:teuthology.orchestra.run.smithi194.stdout: "osd_pglog_items": 0, 2024-04-24T08:22:03.639 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_bytes": 0, 2024-04-24T08:22:03.639 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_items": 0, 2024-04-24T08:22:03.639 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_bytes": 0, 2024-04-24T08:22:03.639 INFO:teuthology.orchestra.run.smithi194.stdout: "osdmap_mapping_items": 0, 2024-04-24T08:22:03.639 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_bytes": 0, 2024-04-24T08:22:03.639 INFO:teuthology.orchestra.run.smithi194.stdout: "pgmap_items": 0, 2024-04-24T08:22:03.639 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_bytes": 928, 2024-04-24T08:22:03.639 INFO:teuthology.orchestra.run.smithi194.stdout: "mds_co_items": 2, 2024-04-24T08:22:03.640 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_bytes": 0, 2024-04-24T08:22:03.640 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_1_items": 0, 2024-04-24T08:22:03.640 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_bytes": 0, 2024-04-24T08:22:03.640 INFO:teuthology.orchestra.run.smithi194.stdout: "unittest_2_items": 0 2024-04-24T08:22:03.640 INFO:teuthology.orchestra.run.smithi194.stdout: }, 2024-04-24T08:22:03.640 INFO:teuthology.orchestra.run.smithi194.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-04-24T08:22:03.640 INFO:teuthology.orchestra.run.smithi194.stdout: "val": 0, 2024-04-24T08:22:03.640 INFO:teuthology.orchestra.run.smithi194.stdout: "max": 104857600, 2024-04-24T08:22:03.640 INFO:teuthology.orchestra.run.smithi194.stdout: "get_started": 0, 2024-04-24T08:22:03.640 INFO:teuthology.orchestra.run.smithi194.stdout: "get": 252, 2024-04-24T08:22:03.641 INFO:teuthology.orchestra.run.smithi194.stdout: "get_sum": 1700772, 2024-04-24T08:22:03.641 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_fail": 0, 2024-04-24T08:22:03.641 INFO:teuthology.orchestra.run.smithi194.stdout: "get_or_fail_success": 252, 2024-04-24T08:22:03.641 INFO:teuthology.orchestra.run.smithi194.stdout: "take": 0, 2024-04-24T08:22:03.641 INFO:teuthology.orchestra.run.smithi194.stdout: "take_sum": 0, 2024-04-24T08:22:03.641 INFO:teuthology.orchestra.run.smithi194.stdout: "put": 252, 2024-04-24T08:22:03.641 INFO:teuthology.orchestra.run.smithi194.stdout: "put_sum": 1700772, 2024-04-24T08:22:03.641 INFO:teuthology.orchestra.run.smithi194.stdout: "wait": { 2024-04-24T08:22:03.641 INFO:teuthology.orchestra.run.smithi194.stdout: "avgcount": 0, 2024-04-24T08:22:03.642 INFO:teuthology.orchestra.run.smithi194.stdout: "sum": 0.000000000, 2024-04-24T08:22:03.642 INFO:teuthology.orchestra.run.smithi194.stdout: "avgtime": 0.000000000 2024-04-24T08:22:03.642 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:22:03.642 INFO:teuthology.orchestra.run.smithi194.stdout: } 2024-04-24T08:22:03.642 INFO:teuthology.orchestra.run.smithi194.stdout:} 2024-04-24T08:22:03.940 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:03 smithi194 ceph-mon[32836]: pgmap v364: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:22:04.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:03 smithi028 ceph-mon[27786]: pgmap v364: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:22:04.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:03 smithi117 ceph-mon[32438]: pgmap v364: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 20 op/s 2024-04-24T08:22:04.342 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-04-24T08:22:04.342 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-04-24T08:22:04.343 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-04-24T08:22:04.343 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-04-24T08:22:04.343 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-04-24T08:22:04.343 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-04-24T08:22:04.343 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-04-24T08:22:04.354 INFO:tasks.kclient:Unmounting kernel clients... 2024-04-24T08:22:04.355 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T08:22:04.355 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:22:04.431 DEBUG:teuthology.orchestra.run.smithi028:> set -ex 2024-04-24T08:22:04.431 DEBUG:teuthology.orchestra.run.smithi028:> dd if=/proc/self/mounts of=/dev/stdout 2024-04-24T08:22:04.490 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:22:04.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.711+0000 7f6c84800640 1 -- 172.21.15.28:0/3122805655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c7c104450 msgr2=0x7f6c7c104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:04.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.711+0000 7f6c84800640 1 --2- 172.21.15.28:0/3122805655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c7c104450 0x7f6c7c104830 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f6c70009900 tx=0x7f6c7002a410 comp rx=0 tx=0).stop 2024-04-24T08:22:04.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.711+0000 7f6c84800640 1 -- 172.21.15.28:0/3122805655 shutdown_connections 2024-04-24T08:22:04.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.711+0000 7f6c84800640 1 --2- 172.21.15.28:0/3122805655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c7c109520 0x7f6c7c1110f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:04.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.711+0000 7f6c84800640 1 --2- 172.21.15.28:0/3122805655 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c7c104e00 0x7f6c7c108e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:04.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.711+0000 7f6c84800640 1 --2- 172.21.15.28:0/3122805655 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c7c104450 0x7f6c7c104830 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:04.713 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.711+0000 7f6c84800640 1 -- 172.21.15.28:0/3122805655 >> 172.21.15.28:0/3122805655 conn(0x7f6c7c0ffcb0 msgr2=0x7f6c7c1020d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:04.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.712+0000 7f6c84800640 1 -- 172.21.15.28:0/3122805655 shutdown_connections 2024-04-24T08:22:04.714 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.712+0000 7f6c84800640 1 -- 172.21.15.28:0/3122805655 wait complete. 2024-04-24T08:22:04.715 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.712+0000 7f6c84800640 1 Processor -- start 2024-04-24T08:22:04.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.713+0000 7f6c84800640 1 -- start start 2024-04-24T08:22:04.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.713+0000 7f6c84800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c7c104450 0x7f6c7c1a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:04.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.713+0000 7f6c84800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c7c109520 0x7f6c7c1a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:04.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.713+0000 7f6c84800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c7c19a960 0x7f6c7c19ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:04.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.713+0000 7f6c84800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c114620 con 0x7f6c7c19a960 2024-04-24T08:22:04.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.713+0000 7f6c84800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c1144a0 con 0x7f6c7c104450 2024-04-24T08:22:04.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.713+0000 7f6c84800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f6c7c1147a0 con 0x7f6c7c109520 2024-04-24T08:22:04.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.713+0000 7f6c81e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c7c104450 0x7f6c7c1a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:04.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.713+0000 7f6c82800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c7c19a960 0x7f6c7c19ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.713+0000 7f6c81e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c7c104450 0x7f6c7c1a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:50900/0 (socket says 172.21.15.28:50900) 2024-04-24T08:22:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.713+0000 7f6c81400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c7c109520 0x7f6c7c1a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.713+0000 7f6c81e00640 1 -- 172.21.15.28:0/3789593411 learned_addr learned my addr 172.21.15.28:0/3789593411 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:22:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.714+0000 7f6c81400640 1 -- 172.21.15.28:0/3789593411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c7c104450 msgr2=0x7f6c7c1a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.714+0000 7f6c81400640 1 --2- 172.21.15.28:0/3789593411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c7c104450 0x7f6c7c1a07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.714+0000 7f6c81400640 1 -- 172.21.15.28:0/3789593411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c7c19a960 msgr2=0x7f6c7c19ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.714+0000 7f6c81400640 1 --2- 172.21.15.28:0/3789593411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c7c19a960 0x7f6c7c19ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.714+0000 7f6c81400640 1 -- 172.21.15.28:0/3789593411 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c7c19b480 con 0x7f6c7c109520 2024-04-24T08:22:04.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.714+0000 7f6c81400640 1 --2- 172.21.15.28:0/3789593411 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c7c109520 0x7f6c7c1a0cf0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00e9c0 tx=0x7f6c6c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.714+0000 7f6c76a00640 1 -- 172.21.15.28:0/3789593411 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c00ccc0 con 0x7f6c7c109520 2024-04-24T08:22:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.714+0000 7f6c84800640 1 -- 172.21.15.28:0/3789593411 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c7c19b770 con 0x7f6c7c109520 2024-04-24T08:22:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.715+0000 7f6c76a00640 1 -- 172.21.15.28:0/3789593411 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c6c005020 con 0x7f6c7c109520 2024-04-24T08:22:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.715+0000 7f6c76a00640 1 -- 172.21.15.28:0/3789593411 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c6c004a00 con 0x7f6c7c109520 2024-04-24T08:22:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.715+0000 7f6c84800640 1 -- 172.21.15.28:0/3789593411 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c7c1a79c0 con 0x7f6c7c109520 2024-04-24T08:22:04.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.716+0000 7f6c84800640 1 -- 172.21.15.28:0/3789593411 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c7c068e20 con 0x7f6c7c109520 2024-04-24T08:22:04.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.719+0000 7f6c76a00640 1 -- 172.21.15.28:0/3789593411 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f6c6c0051d0 con 0x7f6c7c109520 2024-04-24T08:22:04.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.720+0000 7f6c76a00640 1 --2- 172.21.15.28:0/3789593411 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6c640a3da0 0x7f6c640a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:04.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.720+0000 7f6c76a00640 1 -- 172.21.15.28:0/3789593411 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8600+0+0 (secure 0 0 0) 0x7f6c6c0d0760 con 0x7f6c7c109520 2024-04-24T08:22:04.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.720+0000 7f6c81e00640 1 --2- 172.21.15.28:0/3789593411 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6c640a3da0 0x7f6c640a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:04.721 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.721+0000 7f6c81e00640 1 --2- 172.21.15.28:0/3789593411 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6c640a3da0 0x7f6c640a6260 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6c700096c0 tx=0x7f6c700046c0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:04.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.724+0000 7f6c76a00640 1 -- 172.21.15.28:0/3789593411 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f6c6c09cd50 con 0x7f6c7c109520 2024-04-24T08:22:04.935 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.935+0000 7f6c84800640 1 -- 172.21.15.28:0/3789593411 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6c7c104e00 con 0x7f6c7c109520 2024-04-24T08:22:04.938 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.937+0000 7f6c76a00640 1 -- 172.21.15.28:0/3789593411 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) ==== 74+0+23192 (secure 0 0 0) 0x7f6c6c09c650 con 0x7f6c7c109520 2024-04-24T08:22:04.938 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:22:04.939 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":101,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:21:29.242317+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":100,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1819419617":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6801/361125225":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/3758227512":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/874707228":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/2660639233":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/361125225":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:22:04.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.940+0000 7f6c84800640 1 -- 172.21.15.28:0/3789593411 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6c640a3da0 msgr2=0x7f6c640a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:04.941 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.940+0000 7f6c84800640 1 --2- 172.21.15.28:0/3789593411 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6c640a3da0 0x7f6c640a6260 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f6c700096c0 tx=0x7f6c700046c0 comp rx=0 tx=0).stop 2024-04-24T08:22:04.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.940+0000 7f6c84800640 1 -- 172.21.15.28:0/3789593411 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c7c109520 msgr2=0x7f6c7c1a0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:04.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.940+0000 7f6c84800640 1 --2- 172.21.15.28:0/3789593411 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c7c109520 0x7f6c7c1a0cf0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c00e9c0 tx=0x7f6c6c00ee80 comp rx=0 tx=0).stop 2024-04-24T08:22:04.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.941+0000 7f6c84800640 1 -- 172.21.15.28:0/3789593411 shutdown_connections 2024-04-24T08:22:04.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.941+0000 7f6c84800640 1 --2- 172.21.15.28:0/3789593411 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f6c640a3da0 0x7f6c640a6260 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:04.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.941+0000 7f6c84800640 1 --2- 172.21.15.28:0/3789593411 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f6c7c19a960 0x7f6c7c19ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:04.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.941+0000 7f6c84800640 1 --2- 172.21.15.28:0/3789593411 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f6c7c109520 0x7f6c7c1a0cf0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:04.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.941+0000 7f6c84800640 1 --2- 172.21.15.28:0/3789593411 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c7c104450 0x7f6c7c1a07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:04.942 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.941+0000 7f6c84800640 1 -- 172.21.15.28:0/3789593411 >> 172.21.15.28:0/3789593411 conn(0x7f6c7c0ffcb0 msgr2=0x7f6c7c102000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:04.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.941+0000 7f6c84800640 1 -- 172.21.15.28:0/3789593411 shutdown_connections 2024-04-24T08:22:04.943 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:04.941+0000 7f6c84800640 1 -- 172.21.15.28:0/3789593411 wait complete. 2024-04-24T08:22:04.963 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-04-24T08:02:34.086371+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-04-24T08:07:49.819536+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-04-24T08:07:50.493486+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 100, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-04-24T08:22:04.963 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-04-24T08:22:05.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.187+0000 7f3faa800640 1 -- 172.21.15.28:0/4224551349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa40743b0 msgr2=0x7f3fa4074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:05.188 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.187+0000 7f3faa800640 1 --2- 172.21.15.28:0/4224551349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa40743b0 0x7f3fa4074790 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f3f98009900 tx=0x7f3f9802a480 comp rx=0 tx=0).stop 2024-04-24T08:22:05.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.187+0000 7f3faa800640 1 -- 172.21.15.28:0/4224551349 shutdown_connections 2024-04-24T08:22:05.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.187+0000 7f3faa800640 1 --2- 172.21.15.28:0/4224551349 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa410e770 0x7f3fa4110b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.187+0000 7f3faa800640 1 --2- 172.21.15.28:0/4224551349 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fa4074cd0 0x7f3fa410e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.187+0000 7f3faa800640 1 --2- 172.21.15.28:0/4224551349 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa40743b0 0x7f3fa4074790 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.187+0000 7f3faa800640 1 -- 172.21.15.28:0/4224551349 >> 172.21.15.28:0/4224551349 conn(0x7f3fa40fd9a0 msgr2=0x7f3fa40ffdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:05.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.187+0000 7f3faa800640 1 -- 172.21.15.28:0/4224551349 shutdown_connections 2024-04-24T08:22:05.189 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.187+0000 7f3faa800640 1 -- 172.21.15.28:0/4224551349 wait complete. 2024-04-24T08:22:05.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.188+0000 7f3faa800640 1 Processor -- start 2024-04-24T08:22:05.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.188+0000 7f3faa800640 1 -- start start 2024-04-24T08:22:05.191 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.189+0000 7f3faa800640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fa40743b0 0x7f3fa41a0680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:05.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.189+0000 7f3faa800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa4074cd0 0x7f3fa41a0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:05.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.189+0000 7f3faa800640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa410e770 0x7f3fa419a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:05.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.189+0000 7f3faa800640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa41144c0 con 0x7f3fa410e770 2024-04-24T08:22:05.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.189+0000 7f3faa800640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa4114340 con 0x7f3fa4074cd0 2024-04-24T08:22:05.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.189+0000 7f3faa800640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f3fa4114640 con 0x7f3fa40743b0 2024-04-24T08:22:05.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.189+0000 7f3fa9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa410e770 0x7f3fa419a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:05.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.189+0000 7f3fa9400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fa40743b0 0x7f3fa41a0680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:05.192 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.189+0000 7f3fa9400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fa40743b0 0x7f3fa41a0680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60130/0 (socket says 172.21.15.28:60130) 2024-04-24T08:22:05.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.189+0000 7f3fa9400640 1 -- 172.21.15.28:0/2547829142 learned_addr learned my addr 172.21.15.28:0/2547829142 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:22:05.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.189+0000 7f3fa9e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa410e770 0x7f3fa419a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55382/0 (socket says 172.21.15.28:55382) 2024-04-24T08:22:05.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.189+0000 7f3fa8a00640 1 --2- 172.21.15.28:0/2547829142 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa4074cd0 0x7f3fa41a0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:05.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.190+0000 7f3fa9400640 1 -- 172.21.15.28:0/2547829142 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa4074cd0 msgr2=0x7f3fa41a0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:05.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.190+0000 7f3fa9400640 1 --2- 172.21.15.28:0/2547829142 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa4074cd0 0x7f3fa41a0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.190+0000 7f3fa9400640 1 -- 172.21.15.28:0/2547829142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa410e770 msgr2=0x7f3fa419a840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:05.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.190+0000 7f3fa9400640 1 --2- 172.21.15.28:0/2547829142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa410e770 0x7f3fa419a840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.190+0000 7f3fa9400640 1 -- 172.21.15.28:0/2547829142 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3fa419b010 con 0x7f3fa40743b0 2024-04-24T08:22:05.193 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.190+0000 7f3fa9400640 1 --2- 172.21.15.28:0/2547829142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fa40743b0 0x7f3fa41a0680 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3f98004770 tx=0x7f3f98003c10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:05.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.190+0000 7f3fa2000640 1 -- 172.21.15.28:0/2547829142 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f9803f070 con 0x7f3fa40743b0 2024-04-24T08:22:05.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.190+0000 7f3faa800640 1 -- 172.21.15.28:0/2547829142 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3fa419b2a0 con 0x7f3fa40743b0 2024-04-24T08:22:05.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.190+0000 7f3fa2000640 1 -- 172.21.15.28:0/2547829142 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f98031ca0 con 0x7f3fa40743b0 2024-04-24T08:22:05.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.190+0000 7f3fa2000640 1 -- 172.21.15.28:0/2547829142 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f98035030 con 0x7f3fa40743b0 2024-04-24T08:22:05.194 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.190+0000 7f3faa800640 1 -- 172.21.15.28:0/2547829142 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3fa41a74d0 con 0x7f3fa40743b0 2024-04-24T08:22:05.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.192+0000 7f3faa800640 1 -- 172.21.15.28:0/2547829142 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3fa4075a60 con 0x7f3fa40743b0 2024-04-24T08:22:05.195 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.194+0000 7f3fa2000640 1 -- 172.21.15.28:0/2547829142 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f3f98042070 con 0x7f3fa40743b0 2024-04-24T08:22:05.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.195+0000 7f3fa2000640 1 --2- 172.21.15.28:0/2547829142 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3f8c0a3d20 0x7f3f8c0a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:05.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.195+0000 7f3fa2000640 1 -- 172.21.15.28:0/2547829142 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8600+0+0 (secure 0 0 0) 0x7f3f980f0ef0 con 0x7f3fa40743b0 2024-04-24T08:22:05.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.195+0000 7f3fa8a00640 1 --2- 172.21.15.28:0/2547829142 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3f8c0a3d20 0x7f3f8c0a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:05.196 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.196+0000 7f3fa8a00640 1 --2- 172.21.15.28:0/2547829142 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3f8c0a3d20 0x7f3f8c0a61e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3f900095f0 tx=0x7f3f90005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:05.199 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.199+0000 7f3fa2000640 1 -- 172.21.15.28:0/2547829142 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f3f980bd4e0 con 0x7f3fa40743b0 2024-04-24T08:22:05.406 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.406+0000 7f3faa800640 1 -- 172.21.15.28:0/2547829142 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3fa41098f0 con 0x7f3fa40743b0 2024-04-24T08:22:05.407 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.407+0000 7f3fa2000640 1 -- 172.21.15.28:0/2547829142 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v101) ==== 93+0+10 (secure 0 0 0) 0x7f3f980bcde0 con 0x7f3fa40743b0 2024-04-24T08:22:05.407 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 1 2024-04-24T08:22:05.410 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.409+0000 7f3faa800640 1 -- 172.21.15.28:0/2547829142 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3f8c0a3d20 msgr2=0x7f3f8c0a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:05.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.409+0000 7f3faa800640 1 --2- 172.21.15.28:0/2547829142 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3f8c0a3d20 0x7f3f8c0a61e0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f3f900095f0 tx=0x7f3f90005f90 comp rx=0 tx=0).stop 2024-04-24T08:22:05.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.409+0000 7f3faa800640 1 -- 172.21.15.28:0/2547829142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fa40743b0 msgr2=0x7f3fa41a0680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:05.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.409+0000 7f3faa800640 1 --2- 172.21.15.28:0/2547829142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fa40743b0 0x7f3fa41a0680 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f3f98004770 tx=0x7f3f98003c10 comp rx=0 tx=0).stop 2024-04-24T08:22:05.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.409+0000 7f3faa800640 1 -- 172.21.15.28:0/2547829142 shutdown_connections 2024-04-24T08:22:05.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.409+0000 7f3faa800640 1 --2- 172.21.15.28:0/2547829142 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f3f8c0a3d20 0x7f3f8c0a61e0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.409+0000 7f3faa800640 1 --2- 172.21.15.28:0/2547829142 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f3fa410e770 0x7f3fa419a840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.409+0000 7f3faa800640 1 --2- 172.21.15.28:0/2547829142 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3fa4074cd0 0x7f3fa41a0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.409+0000 7f3faa800640 1 --2- 172.21.15.28:0/2547829142 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f3fa40743b0 0x7f3fa41a0680 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.411 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.409+0000 7f3faa800640 1 -- 172.21.15.28:0/2547829142 >> 172.21.15.28:0/2547829142 conn(0x7f3fa40fd9a0 msgr2=0x7f3fa40ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:05.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.410+0000 7f3faa800640 1 -- 172.21.15.28:0/2547829142 shutdown_connections 2024-04-24T08:22:05.412 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.410+0000 7f3faa800640 1 -- 172.21.15.28:0/2547829142 wait complete. 2024-04-24T08:22:05.432 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-04-24T08:22:05.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.650+0000 7fe599400640 1 -- 172.21.15.28:0/532537162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5940684b0 msgr2=0x7fe594104d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:05.655 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.650+0000 7fe599400640 1 --2- 172.21.15.28:0/532537162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5940684b0 0x7fe594104d70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7fe584009900 tx=0x7fe58402a3f0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.650+0000 7fe599400640 1 -- 172.21.15.28:0/532537162 shutdown_connections 2024-04-24T08:22:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.650+0000 7fe599400640 1 --2- 172.21.15.28:0/532537162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe59410b010 0x7fe59410d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.650+0000 7fe599400640 1 --2- 172.21.15.28:0/532537162 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe594100f40 0x7fe5941052b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.656 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.650+0000 7fe599400640 1 --2- 172.21.15.28:0/532537162 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe5940684b0 0x7fe594104d70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.650+0000 7fe599400640 1 -- 172.21.15.28:0/532537162 >> 172.21.15.28:0/532537162 conn(0x7fe594075d00 msgr2=0x7fe594076110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.650+0000 7fe599400640 1 -- 172.21.15.28:0/532537162 shutdown_connections 2024-04-24T08:22:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.650+0000 7fe599400640 1 -- 172.21.15.28:0/532537162 wait complete. 2024-04-24T08:22:05.657 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.651+0000 7fe599400640 1 Processor -- start 2024-04-24T08:22:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.651+0000 7fe599400640 1 -- start start 2024-04-24T08:22:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.652+0000 7fe599400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe5940684b0 0x7fe594109db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.652+0000 7fe599400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe594100f40 0x7fe59410a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:05.658 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.652+0000 7fe599400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59410b010 0x7fe59419f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.652+0000 7fe599400640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe594112320 con 0x7fe59410b010 2024-04-24T08:22:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.652+0000 7fe599400640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe5941121a0 con 0x7fe594100f40 2024-04-24T08:22:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.652+0000 7fe599400640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe5941124a0 con 0x7fe5940684b0 2024-04-24T08:22:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.652+0000 7fe593e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe5940684b0 0x7fe594109db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.652+0000 7fe598a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59410b010 0x7fe59419f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.652+0000 7fe593e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe5940684b0 0x7fe594109db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:60146/0 (socket says 172.21.15.28:60146) 2024-04-24T08:22:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.652+0000 7fe593e00640 1 -- 172.21.15.28:0/778737625 learned_addr learned my addr 172.21.15.28:0/778737625 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:22:05.659 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.652+0000 7fe58be00640 1 --2- 172.21.15.28:0/778737625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe594100f40 0x7fe59410a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:05.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.652+0000 7fe593e00640 1 -- 172.21.15.28:0/778737625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe594100f40 msgr2=0x7fe59410a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:05.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.652+0000 7fe593e00640 1 --2- 172.21.15.28:0/778737625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe594100f40 0x7fe59410a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.652+0000 7fe593e00640 1 -- 172.21.15.28:0/778737625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59410b010 msgr2=0x7fe59419f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:05.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.652+0000 7fe593e00640 1 --2- 172.21.15.28:0/778737625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59410b010 0x7fe59419f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.652+0000 7fe593e00640 1 -- 172.21.15.28:0/778737625 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe59419f9d0 con 0x7fe5940684b0 2024-04-24T08:22:05.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.653+0000 7fe593e00640 1 --2- 172.21.15.28:0/778737625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe5940684b0 0x7fe594109db0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe58402a910 tx=0x7fe584003b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:05.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.653+0000 7fe591600640 1 -- 172.21.15.28:0/778737625 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe58403f070 con 0x7fe5940684b0 2024-04-24T08:22:05.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.653+0000 7fe599400640 1 -- 172.21.15.28:0/778737625 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe59419fc60 con 0x7fe5940684b0 2024-04-24T08:22:05.660 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.653+0000 7fe591600640 1 -- 172.21.15.28:0/778737625 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe584031c90 con 0x7fe5940684b0 2024-04-24T08:22:05.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.653+0000 7fe591600640 1 -- 172.21.15.28:0/778737625 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe584035030 con 0x7fe5940684b0 2024-04-24T08:22:05.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.653+0000 7fe599400640 1 -- 172.21.15.28:0/778737625 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5941a7550 con 0x7fe5940684b0 2024-04-24T08:22:05.661 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.654+0000 7fe599400640 1 -- 172.21.15.28:0/778737625 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe594069190 con 0x7fe5940684b0 2024-04-24T08:22:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.657+0000 7fe591600640 1 -- 172.21.15.28:0/778737625 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe584042070 con 0x7fe5940684b0 2024-04-24T08:22:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.658+0000 7fe591600640 1 --2- 172.21.15.28:0/778737625 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe5780a3d20 0x7fe5780a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.658+0000 7fe591600640 1 -- 172.21.15.28:0/778737625 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8600+0+0 (secure 0 0 0) 0x7fe5840f15f0 con 0x7fe5940684b0 2024-04-24T08:22:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.658+0000 7fe58be00640 1 --2- 172.21.15.28:0/778737625 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe5780a3d20 0x7fe5780a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.659+0000 7fe58be00640 1 --2- 172.21.15.28:0/778737625 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe5780a3d20 0x7fe5780a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe59410ae30 tx=0x7fe57c005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:05.662 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.660+0000 7fe591600640 1 -- 172.21.15.28:0/778737625 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe5840bdbe0 con 0x7fe5940684b0 2024-04-24T08:22:05.868 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.868+0000 7fe599400640 1 -- 172.21.15.28:0/778737625 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fe5941a7720 con 0x7fe5940684b0 2024-04-24T08:22:05.869 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.868+0000 7fe591600640 1 -- 172.21.15.28:0/778737625 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) ==== 104+0+11 (secure 0 0 0) 0x7fe5840bd4e0 con 0x7fe5940684b0 2024-04-24T08:22:05.869 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 32 2024-04-24T08:22:05.872 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.872+0000 7fe599400640 1 -- 172.21.15.28:0/778737625 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe5780a3d20 msgr2=0x7fe5780a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:05.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.872+0000 7fe599400640 1 --2- 172.21.15.28:0/778737625 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe5780a3d20 0x7fe5780a61e0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fe59410ae30 tx=0x7fe57c005f90 comp rx=0 tx=0).stop 2024-04-24T08:22:05.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.872+0000 7fe599400640 1 -- 172.21.15.28:0/778737625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe5940684b0 msgr2=0x7fe594109db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:05.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.872+0000 7fe599400640 1 --2- 172.21.15.28:0/778737625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe5940684b0 0x7fe594109db0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7fe58402a910 tx=0x7fe584003b20 comp rx=0 tx=0).stop 2024-04-24T08:22:05.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.872+0000 7fe599400640 1 -- 172.21.15.28:0/778737625 shutdown_connections 2024-04-24T08:22:05.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.872+0000 7fe599400640 1 --2- 172.21.15.28:0/778737625 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe5780a3d20 0x7fe5780a61e0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.872+0000 7fe599400640 1 --2- 172.21.15.28:0/778737625 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe59410b010 0x7fe59419f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.873 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.872+0000 7fe599400640 1 --2- 172.21.15.28:0/778737625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe594100f40 0x7fe59410a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.872+0000 7fe599400640 1 --2- 172.21.15.28:0/778737625 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe5940684b0 0x7fe594109db0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:05.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.872+0000 7fe599400640 1 -- 172.21.15.28:0/778737625 >> 172.21.15.28:0/778737625 conn(0x7fe594075d00 msgr2=0x7fe5940ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:05.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.872+0000 7fe599400640 1 -- 172.21.15.28:0/778737625 shutdown_connections 2024-04-24T08:22:05.874 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:05.872+0000 7fe599400640 1 -- 172.21.15.28:0/778737625 wait complete. 2024-04-24T08:22:05.893 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-04-24T08:22:06.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:05 smithi028 ceph-mon[27786]: pgmap v365: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-24T08:22:06.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:05 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3789593411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:22:06.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:05 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/2547829142' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:22:06.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.107+0000 7ff3d7e00640 1 -- 172.21.15.28:0/2831840095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d810d500 msgr2=0x7ff3d810d8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.108 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.107+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/2831840095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d810d500 0x7ff3d810d8e0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8009930 tx=0x7ff3c802a420 comp rx=0 tx=0).stop 2024-04-24T08:22:06.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.108+0000 7ff3d7e00640 1 -- 172.21.15.28:0/2831840095 shutdown_connections 2024-04-24T08:22:06.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.108+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/2831840095 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3d8108f40 0x7ff3d81153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.108+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/2831840095 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3d8108580 0x7ff3d8108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.108+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/2831840095 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d810d500 0x7ff3d810d8e0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.108+0000 7ff3d7e00640 1 -- 172.21.15.28:0/2831840095 >> 172.21.15.28:0/2831840095 conn(0x7ff3d8077e20 msgr2=0x7ff3d8106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:06.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.108+0000 7ff3d7e00640 1 -- 172.21.15.28:0/2831840095 shutdown_connections 2024-04-24T08:22:06.109 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.108+0000 7ff3d7e00640 1 -- 172.21.15.28:0/2831840095 wait complete. 2024-04-24T08:22:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.109+0000 7ff3d7e00640 1 Processor -- start 2024-04-24T08:22:06.111 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.109+0000 7ff3d7e00640 1 -- start start 2024-04-24T08:22:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.109+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3d8108580 0x7ff3d819ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.109+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3d8108f40 0x7ff3d819f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.109+0000 7ff3d7e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d810d500 0x7ff3d81a3600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.109+0000 7ff3d7e00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d81188c0 con 0x7ff3d810d500 2024-04-24T08:22:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.109+0000 7ff3d7e00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d8118740 con 0x7ff3d8108f40 2024-04-24T08:22:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.109+0000 7ff3d7e00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff3d8118a40 con 0x7ff3d8108580 2024-04-24T08:22:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.110+0000 7ff3d6000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3d8108f40 0x7ff3d819f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:06.112 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.110+0000 7ff3d6a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3d8108580 0x7ff3d819ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.110+0000 7ff3d6000640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3d8108f40 0x7ff3d819f280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:50936/0 (socket says 172.21.15.28:50936) 2024-04-24T08:22:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.110+0000 7ff3d6000640 1 -- 172.21.15.28:0/687247683 learned_addr learned my addr 172.21.15.28:0/687247683 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:22:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.110+0000 7ff3d7400640 1 --2- 172.21.15.28:0/687247683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d810d500 0x7ff3d81a3600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.110+0000 7ff3d6000640 1 -- 172.21.15.28:0/687247683 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3d8108580 msgr2=0x7ff3d819ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.110+0000 7ff3d6000640 1 --2- 172.21.15.28:0/687247683 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3d8108580 0x7ff3d819ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.110+0000 7ff3d6000640 1 -- 172.21.15.28:0/687247683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d810d500 msgr2=0x7ff3d81a3600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.110+0000 7ff3d6000640 1 --2- 172.21.15.28:0/687247683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d810d500 0x7ff3d81a3600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.113 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.110+0000 7ff3d6000640 1 -- 172.21.15.28:0/687247683 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3d81a3d30 con 0x7ff3d8108f40 2024-04-24T08:22:06.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.110+0000 7ff3d6000640 1 --2- 172.21.15.28:0/687247683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3d8108f40 0x7ff3d819f280 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000e9f0 tx=0x7ff3c000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:06.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.111+0000 7ff3cf400640 1 -- 172.21.15.28:0/687247683 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c000cd80 con 0x7ff3d8108f40 2024-04-24T08:22:06.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.111+0000 7ff3cf400640 1 -- 172.21.15.28:0/687247683 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff3c0005500 con 0x7ff3d8108f40 2024-04-24T08:22:06.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.111+0000 7ff3cf400640 1 -- 172.21.15.28:0/687247683 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff3c0010690 con 0x7ff3d8108f40 2024-04-24T08:22:06.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.111+0000 7ff3d7e00640 1 -- 172.21.15.28:0/687247683 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3d81a4020 con 0x7ff3d8108f40 2024-04-24T08:22:06.114 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.111+0000 7ff3d7e00640 1 -- 172.21.15.28:0/687247683 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3d81ab8b0 con 0x7ff3d8108f40 2024-04-24T08:22:06.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.112+0000 7ff3d7e00640 1 -- 172.21.15.28:0/687247683 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3d8069190 con 0x7ff3d8108f40 2024-04-24T08:22:06.115 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.114+0000 7ff3cf400640 1 -- 172.21.15.28:0/687247683 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff3c00046c0 con 0x7ff3d8108f40 2024-04-24T08:22:06.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.115+0000 7ff3cf400640 1 --2- 172.21.15.28:0/687247683 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3bc0a3da0 0x7ff3bc0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.116+0000 7ff3cf400640 1 -- 172.21.15.28:0/687247683 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8600+0+0 (secure 0 0 0) 0x7ff3c00d0c80 con 0x7ff3d8108f40 2024-04-24T08:22:06.116 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.116+0000 7ff3d6a00640 1 --2- 172.21.15.28:0/687247683 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3bc0a3da0 0x7ff3bc0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:06.117 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.117+0000 7ff3d6a00640 1 --2- 172.21.15.28:0/687247683 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3bc0a3da0 0x7ff3bc0a6260 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8004770 tx=0x7ff3c80058d0 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:06.118 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.118+0000 7ff3cf400640 1 -- 172.21.15.28:0/687247683 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff3c009d270 con 0x7ff3d8108f40 2024-04-24T08:22:06.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:05 smithi117 ceph-mon[32438]: pgmap v365: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-24T08:22:06.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:05 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3789593411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:22:06.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:05 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/2547829142' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:22:06.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:05 smithi194 ceph-mon[32836]: pgmap v365: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 22 KiB/s wr, 21 op/s 2024-04-24T08:22:06.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:05 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3789593411' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:22:06.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:05 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/2547829142' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-04-24T08:22:06.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.328+0000 7ff3d7e00640 1 -- 172.21.15.28:0/687247683 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ff3d81abb10 con 0x7ff3d8108f40 2024-04-24T08:22:06.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.329+0000 7ff3cf400640 1 -- 172.21.15.28:0/687247683 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) ==== 100+0+11 (secure 0 0 0) 0x7ff3c009cb70 con 0x7ff3d8108f40 2024-04-24T08:22:06.329 INFO:teuthology.orchestra.run.smithi028.stdout:pg_num: 64 2024-04-24T08:22:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.331+0000 7ff3d7e00640 1 -- 172.21.15.28:0/687247683 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3bc0a3da0 msgr2=0x7ff3bc0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.331+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/687247683 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3bc0a3da0 0x7ff3bc0a6260 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff3c8004770 tx=0x7ff3c80058d0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.331+0000 7ff3d7e00640 1 -- 172.21.15.28:0/687247683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3d8108f40 msgr2=0x7ff3d819f280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.331+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/687247683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3d8108f40 0x7ff3d819f280 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7ff3c000e9f0 tx=0x7ff3c000edf0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.332+0000 7ff3d7e00640 1 -- 172.21.15.28:0/687247683 shutdown_connections 2024-04-24T08:22:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.332+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/687247683 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3bc0a3da0 0x7ff3bc0a6260 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.332+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/687247683 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3d810d500 0x7ff3d81a3600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.332+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/687247683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3d8108f40 0x7ff3d819f280 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.332+0000 7ff3d7e00640 1 --2- 172.21.15.28:0/687247683 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3d8108580 0x7ff3d819ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.332+0000 7ff3d7e00640 1 -- 172.21.15.28:0/687247683 >> 172.21.15.28:0/687247683 conn(0x7ff3d8077e20 msgr2=0x7ff3d8078340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.332+0000 7ff3d7e00640 1 -- 172.21.15.28:0/687247683 shutdown_connections 2024-04-24T08:22:06.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.332+0000 7ff3d7e00640 1 -- 172.21.15.28:0/687247683 wait complete. 2024-04-24T08:22:06.353 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-04-24T08:22:06.567 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.566+0000 7f9068c00640 1 -- 172.21.15.28:0/1612794261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9064104320 msgr2=0x7f9064104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.566+0000 7f9068c00640 1 --2- 172.21.15.28:0/1612794261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9064104320 0x7f9064104700 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f9054009900 tx=0x7f905402a400 comp rx=0 tx=0).stop 2024-04-24T08:22:06.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.566+0000 7f9068c00640 1 -- 172.21.15.28:0/1612794261 shutdown_connections 2024-04-24T08:22:06.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.566+0000 7f9068c00640 1 --2- 172.21.15.28:0/1612794261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90641093f0 0x7f9064110fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.566+0000 7f9068c00640 1 --2- 172.21.15.28:0/1612794261 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f9064104cd0 0x7f9064108d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.567+0000 7f9068c00640 1 --2- 172.21.15.28:0/1612794261 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9064104320 0x7f9064104700 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.567+0000 7f9068c00640 1 -- 172.21.15.28:0/1612794261 >> 172.21.15.28:0/1612794261 conn(0x7f90640ffb40 msgr2=0x7f9064101f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:06.568 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.567+0000 7f9068c00640 1 -- 172.21.15.28:0/1612794261 shutdown_connections 2024-04-24T08:22:06.569 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.567+0000 7f9068c00640 1 -- 172.21.15.28:0/1612794261 wait complete. 2024-04-24T08:22:06.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.568+0000 7f9068c00640 1 Processor -- start 2024-04-24T08:22:06.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.568+0000 7f9068c00640 1 -- start start 2024-04-24T08:22:06.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.568+0000 7f9068c00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9064104320 0x7f90641a06b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.571 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.568+0000 7f9068c00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90641093f0 0x7f90641a0bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.568+0000 7f9068c00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f906419a8b0 0x7f906419ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.568+0000 7f9068c00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f90641144f0 con 0x7f9064104320 2024-04-24T08:22:06.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.568+0000 7f9068c00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f9064114370 con 0x7f90641093f0 2024-04-24T08:22:06.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.568+0000 7f9068c00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f9064114670 con 0x7f906419a8b0 2024-04-24T08:22:06.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.568+0000 7f9063400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9064104320 0x7f90641a06b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:06.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.568+0000 7f9063400640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9064104320 0x7f90641a06b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.28:3300/0 says I am v2:172.21.15.28:55422/0 (socket says 172.21.15.28:55422) 2024-04-24T08:22:06.572 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.568+0000 7f9063400640 1 -- 172.21.15.28:0/3090613161 learned_addr learned my addr 172.21.15.28:0/3090613161 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:22:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.569+0000 7f9062a00640 1 --2- 172.21.15.28:0/3090613161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90641093f0 0x7f90641a0bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.569+0000 7f9063e00640 1 --2- 172.21.15.28:0/3090613161 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f906419a8b0 0x7f906419ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.569+0000 7f9063400640 1 -- 172.21.15.28:0/3090613161 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f906419a8b0 msgr2=0x7f906419ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.569+0000 7f9063400640 1 --2- 172.21.15.28:0/3090613161 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f906419a8b0 0x7f906419ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.569+0000 7f9063400640 1 -- 172.21.15.28:0/3090613161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90641093f0 msgr2=0x7f90641a0bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.569+0000 7f9063400640 1 --2- 172.21.15.28:0/3090613161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90641093f0 0x7f90641a0bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.569+0000 7f9063400640 1 -- 172.21.15.28:0/3090613161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f906419b3d0 con 0x7f9064104320 2024-04-24T08:22:06.573 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.569+0000 7f9063e00640 1 --2- 172.21.15.28:0/3090613161 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f906419a8b0 0x7f906419ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:22:06.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.569+0000 7f9063400640 1 --2- 172.21.15.28:0/3090613161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9064104320 0x7f90641a06b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9054004770 tx=0x7f9054003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:06.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.569+0000 7f905be00640 1 -- 172.21.15.28:0/3090613161 <== mon.0 v2:172.21.15.28:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f905403f070 con 0x7f9064104320 2024-04-24T08:22:06.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.570+0000 7f905be00640 1 -- 172.21.15.28:0/3090613161 <== mon.0 v2:172.21.15.28:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9054031ca0 con 0x7f9064104320 2024-04-24T08:22:06.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.570+0000 7f9068c00640 1 -- 172.21.15.28:0/3090613161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f906419b660 con 0x7f9064104320 2024-04-24T08:22:06.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.570+0000 7f905be00640 1 -- 172.21.15.28:0/3090613161 <== mon.0 v2:172.21.15.28:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9054035030 con 0x7f9064104320 2024-04-24T08:22:06.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.570+0000 7f9068c00640 1 -- 172.21.15.28:0/3090613161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f90641a7800 con 0x7f9064104320 2024-04-24T08:22:06.574 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.571+0000 7f9068c00640 1 -- 172.21.15.28:0/3090613161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9064069190 con 0x7f9064104320 2024-04-24T08:22:06.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.573+0000 7f905be00640 1 -- 172.21.15.28:0/3090613161 <== mon.0 v2:172.21.15.28:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f9054042070 con 0x7f9064104320 2024-04-24T08:22:06.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.574+0000 7f905be00640 1 --2- 172.21.15.28:0/3090613161 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f90480a3d20 0x7f90480a61e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:06.575 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.574+0000 7f905be00640 1 -- 172.21.15.28:0/3090613161 <== mon.0 v2:172.21.15.28:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8600+0+0 (secure 0 0 0) 0x7f90540f0f00 con 0x7f9064104320 2024-04-24T08:22:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.575+0000 7f9062a00640 1 --2- 172.21.15.28:0/3090613161 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f90480a3d20 0x7f90480a61e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:06.576 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.575+0000 7f9062a00640 1 --2- 172.21.15.28:0/3090613161 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f90480a3d20 0x7f90480a61e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f906410aa50 tx=0x7f9050005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:06.580 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.579+0000 7f905be00640 1 -- 172.21.15.28:0/3090613161 <== mon.0 v2:172.21.15.28:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f90540bd5b0 con 0x7f9064104320 2024-04-24T08:22:06.803 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.803+0000 7f9068c00640 1 -- 172.21.15.28:0/3090613161 --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f906419bd90 con 0x7f9064104320 2024-04-24T08:22:06.804 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.804+0000 7f905be00640 1 -- 172.21.15.28:0/3090613161 <== mon.0 v2:172.21.15.28:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v101) ==== 81+0+941 (secure 0 0 0) 0x7f90540bceb0 con 0x7f9064104320 2024-04-24T08:22:06.805 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/1819419617 2024-04-25T08:10:20.116072+0000 2024-04-24T08:22:06.805 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/361125225 2024-04-25T08:10:20.116072+0000 2024-04-24T08:22:06.805 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3758227512 2024-04-25T08:10:20.116072+0000 2024-04-24T08:22:06.805 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/874707228 2024-04-25T08:10:20.116072+0000 2024-04-24T08:22:06.805 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/3308597107 2024-04-25T07:59:49.915756+0000 2024-04-24T08:22:06.806 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/409391969 2024-04-25T07:59:49.915756+0000 2024-04-24T08:22:06.806 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2660639233 2024-04-25T08:10:20.116072+0000 2024-04-24T08:22:06.806 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/3308597107 2024-04-25T07:59:49.915756+0000 2024-04-24T08:22:06.806 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/3378984747 2024-04-25T07:59:49.915756+0000 2024-04-24T08:22:06.806 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/361125225 2024-04-25T08:10:20.116072+0000 2024-04-24T08:22:06.806 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/4131458386 2024-04-25T07:59:15.381957+0000 2024-04-24T08:22:06.806 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/2417007332 2024-04-25T07:59:15.381957+0000 2024-04-24T08:22:06.806 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/234090859 2024-04-25T07:59:15.381957+0000 2024-04-24T08:22:06.807 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6800/2371619813 2024-04-25T07:59:15.381957+0000 2024-04-24T08:22:06.807 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:0/4081129566 2024-04-25T07:59:49.915756+0000 2024-04-24T08:22:06.807 INFO:teuthology.orchestra.run.smithi028.stdout:172.21.15.28:6801/2371619813 2024-04-25T07:59:15.381957+0000 2024-04-24T08:22:06.807 INFO:teuthology.orchestra.run.smithi028.stderr:listed 16 entries 2024-04-24T08:22:06.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.807+0000 7f9068c00640 1 -- 172.21.15.28:0/3090613161 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f90480a3d20 msgr2=0x7f90480a61e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.807+0000 7f9068c00640 1 --2- 172.21.15.28:0/3090613161 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f90480a3d20 0x7f90480a61e0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f906410aa50 tx=0x7f9050005f90 comp rx=0 tx=0).stop 2024-04-24T08:22:06.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.807+0000 7f9068c00640 1 -- 172.21.15.28:0/3090613161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9064104320 msgr2=0x7f90641a06b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:06.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.807+0000 7f9068c00640 1 --2- 172.21.15.28:0/3090613161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9064104320 0x7f90641a06b0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f9054004770 tx=0x7f9054003c10 comp rx=0 tx=0).stop 2024-04-24T08:22:06.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.807+0000 7f9068c00640 1 -- 172.21.15.28:0/3090613161 shutdown_connections 2024-04-24T08:22:06.808 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.807+0000 7f9068c00640 1 --2- 172.21.15.28:0/3090613161 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f90480a3d20 0x7f90480a61e0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.807+0000 7f9068c00640 1 --2- 172.21.15.28:0/3090613161 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f906419a8b0 0x7f906419ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.807+0000 7f9068c00640 1 --2- 172.21.15.28:0/3090613161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90641093f0 0x7f90641a0bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.807+0000 7f9068c00640 1 --2- 172.21.15.28:0/3090613161 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f9064104320 0x7f90641a06b0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.807+0000 7f9068c00640 1 -- 172.21.15.28:0/3090613161 >> 172.21.15.28:0/3090613161 conn(0x7f90640ffb40 msgr2=0x7f9064101e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.808+0000 7f9068c00640 1 -- 172.21.15.28:0/3090613161 shutdown_connections 2024-04-24T08:22:06.809 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:06.808+0000 7f9068c00640 1 -- 172.21.15.28:0/3090613161 wait complete. 2024-04-24T08:22:06.829 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-04-24T08:22:06.829 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:22:06.829 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-04-24T08:22:07.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:06 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/778737625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:22:07.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:06 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/687247683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:22:07.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:06 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/778737625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:22:07.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:06 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/687247683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:22:07.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:06 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/778737625' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-04-24T08:22:07.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:06 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/687247683' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-04-24T08:22:08.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:07 smithi117 ceph-mon[32438]: pgmap v366: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:22:08.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:07 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3090613161' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T08:22:08.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:07 smithi194 ceph-mon[32836]: pgmap v366: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:22:08.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:07 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3090613161' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T08:22:08.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:07 smithi028 ceph-mon[27786]: pgmap v366: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 22 op/s 2024-04-24T08:22:08.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:07 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3090613161' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-04-24T08:22:08.391 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi028.front.sepia.ceph.com 2024-04-24T08:22:08.392 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:22:08.392 DEBUG:teuthology.orchestra.run.smithi028:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-04-24T08:22:08.484 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:22:08.484 DEBUG:teuthology.orchestra.run.smithi028:> ip netns list 2024-04-24T08:22:08.581 INFO:teuthology.orchestra.run.smithi028.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-04-24T08:22:08.582 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:22:08.582 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-04-24T08:22:08.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-04-24T08:22:08.643 DEBUG:teuthology.orchestra.run.smithi028:> sudo ip link delete ceph-brx 2024-04-24T08:22:08.745 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-04-24T08:22:08.756 INFO:tasks.cephadm:Teardown begin 2024-04-24T08:22:08.756 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-04-24T08:22:08.756 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:22:09.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.022+0000 7fab0d200640 1 -- 172.21.15.28:0/1875733267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab08110bd0 msgr2=0x7fab0810e880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:09.023 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.022+0000 7fab0d200640 1 --2- 172.21.15.28:0/1875733267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab08110bd0 0x7fab0810e880 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7faaf8009860 tx=0x7faaf802a340 comp rx=0 tx=0).stop 2024-04-24T08:22:09.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.024+0000 7fab0d200640 1 -- 172.21.15.28:0/1875733267 shutdown_connections 2024-04-24T08:22:09.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.024+0000 7fab0d200640 1 --2- 172.21.15.28:0/1875733267 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab0810ef60 0x7fab0811d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:09.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.024+0000 7fab0d200640 1 --2- 172.21.15.28:0/1875733267 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab08110bd0 0x7fab0810e880 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:09.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.024+0000 7fab0d200640 1 --2- 172.21.15.28:0/1875733267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab08110760 0x7fab0810e340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:09.025 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.024+0000 7fab0d200640 1 -- 172.21.15.28:0/1875733267 >> 172.21.15.28:0/1875733267 conn(0x7fab0806d120 msgr2=0x7fab0806d530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:09.026 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.025+0000 7fab0d200640 1 -- 172.21.15.28:0/1875733267 shutdown_connections 2024-04-24T08:22:09.027 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.027+0000 7fab0d200640 1 -- 172.21.15.28:0/1875733267 wait complete. 2024-04-24T08:22:09.029 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.028+0000 7fab0d200640 1 Processor -- start 2024-04-24T08:22:09.031 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.031+0000 7fab0d200640 1 -- start start 2024-04-24T08:22:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.032+0000 7fab0d200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab08110760 0x7fab081a7760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.032+0000 7fab0d200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab081a7ca0 0x7fab081ac020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.032+0000 7fab0d200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab081ac560 0x7fab081aca10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.032+0000 7fab0d200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fab081211a0 con 0x7fab081ac560 2024-04-24T08:22:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.032+0000 7fab0d200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fab08121020 con 0x7fab08110760 2024-04-24T08:22:09.032 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.032+0000 7fab0d200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fab08121320 con 0x7fab081a7ca0 2024-04-24T08:22:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.032+0000 7fab05e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab08110760 0x7fab081a7760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.032+0000 7fab05e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab08110760 0x7fab081a7760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:50974/0 (socket says 172.21.15.28:50974) 2024-04-24T08:22:09.033 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.032+0000 7fab05e00640 1 -- 172.21.15.28:0/15943058 learned_addr learned my addr 172.21.15.28:0/15943058 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:22:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.032+0000 7fab06800640 1 --2- 172.21.15.28:0/15943058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab081ac560 0x7fab081aca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.033+0000 7fab05e00640 1 -- 172.21.15.28:0/15943058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab081a7ca0 msgr2=0x7fab081ac020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.033+0000 7fab05e00640 1 --2- 172.21.15.28:0/15943058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab081a7ca0 0x7fab081ac020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.033+0000 7fab05e00640 1 -- 172.21.15.28:0/15943058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab081ac560 msgr2=0x7fab081aca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.033+0000 7fab05e00640 1 --2- 172.21.15.28:0/15943058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab081ac560 0x7fab081aca10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:09.034 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.033+0000 7fab05e00640 1 -- 172.21.15.28:0/15943058 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab081b42d0 con 0x7fab08110760 2024-04-24T08:22:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.034+0000 7fab05e00640 1 --2- 172.21.15.28:0/15943058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab08110760 0x7fab081a7760 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7faaf400e9f0 tx=0x7faaf400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:09.035 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.035+0000 7faafea00640 1 -- 172.21.15.28:0/15943058 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf400ccc0 con 0x7fab08110760 2024-04-24T08:22:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.035+0000 7faafea00640 1 -- 172.21.15.28:0/15943058 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faaf4005500 con 0x7fab08110760 2024-04-24T08:22:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.035+0000 7fab0d200640 1 -- 172.21.15.28:0/15943058 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab081b4500 con 0x7fab08110760 2024-04-24T08:22:09.036 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.035+0000 7faafea00640 1 -- 172.21.15.28:0/15943058 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faaf4010690 con 0x7fab08110760 2024-04-24T08:22:09.037 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.035+0000 7fab0d200640 1 -- 172.21.15.28:0/15943058 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab081b4a20 con 0x7fab08110760 2024-04-24T08:22:09.039 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.038+0000 7fab0d200640 1 -- 172.21.15.28:0/15943058 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab0806e3b0 con 0x7fab08110760 2024-04-24T08:22:09.041 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.041+0000 7faafea00640 1 -- 172.21.15.28:0/15943058 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7faaf40046c0 con 0x7fab08110760 2024-04-24T08:22:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.043+0000 7faafea00640 1 --2- 172.21.15.28:0/15943058 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7faaec0a3e70 0x7faaec0a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.043+0000 7faafea00640 1 -- 172.21.15.28:0/15943058 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8644+0+0 (secure 0 0 0) 0x7faaf40d0b00 con 0x7fab08110760 2024-04-24T08:22:09.044 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.044+0000 7fab05400640 1 --2- 172.21.15.28:0/15943058 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7faaec0a3e70 0x7faaec0a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:09.045 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.045+0000 7fab05400640 1 --2- 172.21.15.28:0/15943058 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7faaec0a3e70 0x7faaec0a6330 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faaf8009780 tx=0x7faaf8030460 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:09.052 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.052+0000 7faafea00640 1 -- 172.21.15.28:0/15943058 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7faaf409d120 con 0x7fab08110760 2024-04-24T08:22:09.538 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.537+0000 7fab0d200640 1 -- 172.21.15.28:0/15943058 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fab0810ef60 con 0x7fab08110760 2024-04-24T08:22:09.539 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.539+0000 7faafea00640 1 -- 172.21.15.28:0/15943058 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11871 (secure 0 0 0) 0x7faaf409ca20 con 0x7fab08110760 2024-04-24T08:22:09.542 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:22:09.543 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:14:38.663141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:22:09.543 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-24T08:22:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.543+0000 7fab0d200640 1 -- 172.21.15.28:0/15943058 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7faaec0a3e70 msgr2=0x7faaec0a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.544+0000 7fab0d200640 1 --2- 172.21.15.28:0/15943058 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7faaec0a3e70 0x7faaec0a6330 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7faaf8009780 tx=0x7faaf8030460 comp rx=0 tx=0).stop 2024-04-24T08:22:09.544 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.544+0000 7fab0d200640 1 -- 172.21.15.28:0/15943058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab08110760 msgr2=0x7fab081a7760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.544+0000 7fab0d200640 1 --2- 172.21.15.28:0/15943058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab08110760 0x7fab081a7760 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7faaf400e9f0 tx=0x7faaf400eeb0 comp rx=0 tx=0).stop 2024-04-24T08:22:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.544+0000 7fab0d200640 1 -- 172.21.15.28:0/15943058 shutdown_connections 2024-04-24T08:22:09.545 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.544+0000 7fab0d200640 1 --2- 172.21.15.28:0/15943058 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7faaec0a3e70 0x7faaec0a6330 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.544+0000 7fab0d200640 1 --2- 172.21.15.28:0/15943058 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fab081ac560 0x7fab081aca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.544+0000 7fab0d200640 1 --2- 172.21.15.28:0/15943058 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fab081a7ca0 0x7fab081ac020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.544+0000 7fab0d200640 1 --2- 172.21.15.28:0/15943058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab08110760 0x7fab081a7760 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.544+0000 7fab0d200640 1 -- 172.21.15.28:0/15943058 >> 172.21.15.28:0/15943058 conn(0x7fab0806d120 msgr2=0x7fab0811a2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.545+0000 7fab0d200640 1 -- 172.21.15.28:0/15943058 shutdown_connections 2024-04-24T08:22:09.546 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.545+0000 7fab0d200640 1 -- 172.21.15.28:0/15943058 wait complete. 2024-04-24T08:22:09.574 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-04-24T08:22:09.880 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.880+0000 7fa5cfc00640 1 -- 172.21.15.28:0/1561755684 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5c8102240 msgr2=0x7fa5c8102620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:09.881 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.880+0000 7fa5cfc00640 1 --2- 172.21.15.28:0/1561755684 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5c8102240 0x7fa5c8102620 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc009900 tx=0x7fa5bc02a420 comp rx=0 tx=0).stop 2024-04-24T08:22:09.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.881+0000 7fa5cfc00640 1 -- 172.21.15.28:0/1561755684 shutdown_connections 2024-04-24T08:22:09.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.881+0000 7fa5cfc00640 1 --2- 172.21.15.28:0/1561755684 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c8107310 0x7fa5c810eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:09.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.881+0000 7fa5cfc00640 1 --2- 172.21.15.28:0/1561755684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5c8102bf0 0x7fa5c8106c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:09.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.881+0000 7fa5cfc00640 1 --2- 172.21.15.28:0/1561755684 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5c8102240 0x7fa5c8102620 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:09.882 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.881+0000 7fa5cfc00640 1 -- 172.21.15.28:0/1561755684 >> 172.21.15.28:0/1561755684 conn(0x7fa5c80793c0 msgr2=0x7fa5c80797d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:09.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.882+0000 7fa5cfc00640 1 -- 172.21.15.28:0/1561755684 shutdown_connections 2024-04-24T08:22:09.883 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.882+0000 7fa5cfc00640 1 -- 172.21.15.28:0/1561755684 wait complete. 2024-04-24T08:22:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.885+0000 7fa5cfc00640 1 Processor -- start 2024-04-24T08:22:09.886 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.885+0000 7fa5cfc00640 1 -- start start 2024-04-24T08:22:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.886+0000 7fa5cfc00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5c8102240 0x7fa5c80737e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.886+0000 7fa5cfc00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5c8107310 0x7fa5c806d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.886+0000 7fa5cfc00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c806dd90 0x7fa5c806e240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.886+0000 7fa5cfc00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c806c250 con 0x7fa5c806dd90 2024-04-24T08:22:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.886+0000 7fa5cfc00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c806c0d0 con 0x7fa5c8102240 2024-04-24T08:22:09.887 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.886+0000 7fa5cfc00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fa5c806c3d0 con 0x7fa5c8107310 2024-04-24T08:22:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.886+0000 7fa5cd200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5c8102240 0x7fa5c80737e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.886+0000 7fa5cd200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5c8102240 0x7fa5c80737e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49428/0 (socket says 172.21.15.28:49428) 2024-04-24T08:22:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.886+0000 7fa5cd200640 1 -- 172.21.15.28:0/569324910 learned_addr learned my addr 172.21.15.28:0/569324910 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:22:09.888 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.886+0000 7fa5c7e00640 1 --2- 172.21.15.28:0/569324910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5c8107310 0x7fa5c806d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.886+0000 7fa5cdc00640 1 --2- 172.21.15.28:0/569324910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c806dd90 0x7fa5c806e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.887+0000 7fa5c7e00640 1 -- 172.21.15.28:0/569324910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5c8102240 msgr2=0x7fa5c80737e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.887+0000 7fa5c7e00640 1 --2- 172.21.15.28:0/569324910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5c8102240 0x7fa5c80737e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.887+0000 7fa5c7e00640 1 -- 172.21.15.28:0/569324910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c806dd90 msgr2=0x7fa5c806e240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.887+0000 7fa5c7e00640 1 --2- 172.21.15.28:0/569324910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c806dd90 0x7fa5c806e240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.887+0000 7fa5c7e00640 1 -- 172.21.15.28:0/569324910 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa5c81b3f90 con 0x7fa5c8107310 2024-04-24T08:22:09.889 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.887+0000 7fa5cdc00640 1 --2- 172.21.15.28:0/569324910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c806dd90 0x7fa5c806e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-04-24T08:22:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.888+0000 7fa5c7e00640 1 --2- 172.21.15.28:0/569324910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5c8107310 0x7fa5c806d850 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800b3c0 tx=0x7fa5b800b880 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.889+0000 7fa5c5600640 1 -- 172.21.15.28:0/569324910 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b8011830 con 0x7fa5c8107310 2024-04-24T08:22:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.889+0000 7fa5c5600640 1 -- 172.21.15.28:0/569324910 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa5b8016070 con 0x7fa5c8107310 2024-04-24T08:22:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.889+0000 7fa5cfc00640 1 -- 172.21.15.28:0/569324910 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa5c81b4160 con 0x7fa5c8107310 2024-04-24T08:22:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.890+0000 7fa5cfc00640 1 -- 172.21.15.28:0/569324910 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa5c81b46d0 con 0x7fa5c8107310 2024-04-24T08:22:09.891 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.890+0000 7fa5c5600640 1 -- 172.21.15.28:0/569324910 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa5b800f2e0 con 0x7fa5c8107310 2024-04-24T08:22:09.892 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.891+0000 7fa5cfc00640 1 -- 172.21.15.28:0/569324910 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa5ac005290 con 0x7fa5c8107310 2024-04-24T08:22:09.895 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.895+0000 7fa5c5600640 1 -- 172.21.15.28:0/569324910 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fa5b8010380 con 0x7fa5c8107310 2024-04-24T08:22:09.902 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.901+0000 7fa5c5600640 1 --2- 172.21.15.28:0/569324910 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa5b00a3da0 0x7fa5b00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.902+0000 7fa5c5600640 1 -- 172.21.15.28:0/569324910 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8644+0+0 (secure 0 0 0) 0x7fa5b80d0280 con 0x7fa5c8107310 2024-04-24T08:22:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.902+0000 7fa5c5600640 1 -- 172.21.15.28:0/569324910 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fa5b809c9b0 con 0x7fa5c8107310 2024-04-24T08:22:09.904 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.902+0000 7fa5cd200640 1 --2- 172.21.15.28:0/569324910 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa5b00a3da0 0x7fa5b00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:09.905 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:09.905+0000 7fa5cd200640 1 --2- 172.21.15.28:0/569324910 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa5b00a3da0 0x7fa5b00a6260 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc031550 tx=0x7fa5bc005850 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:10.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:09 smithi028 ceph-mon[27786]: pgmap v367: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2024-04-24T08:22:10.039 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:09 smithi028 ceph-mon[27786]: osdmap e102: 12 total, 12 up, 12 in 2024-04-24T08:22:10.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:09 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/15943058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:22:10.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:09 smithi117 ceph-mon[32438]: pgmap v367: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2024-04-24T08:22:10.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:09 smithi117 ceph-mon[32438]: osdmap e102: 12 total, 12 up, 12 in 2024-04-24T08:22:10.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:09 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/15943058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:22:10.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:09 smithi194 ceph-mon[32836]: pgmap v367: 97 pgs: 97 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 44 KiB/s wr, 23 op/s 2024-04-24T08:22:10.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:09 smithi194 ceph-mon[32836]: osdmap e102: 12 total, 12 up, 12 in 2024-04-24T08:22:10.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:09 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/15943058' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:22:10.376 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.375+0000 7fa5cfc00640 1 -- 172.21.15.28:0/569324910 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa5ac005580 con 0x7fa5c8107310 2024-04-24T08:22:10.377 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.377+0000 7fa5c5600640 1 -- 172.21.15.28:0/569324910 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 26 v26) ==== 94+0+11871 (secure 0 0 0) 0x7fa5b8096e20 con 0x7fa5c8107310 2024-04-24T08:22:10.378 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:22:10.378 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":26,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6835/1751430859","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6834","nonce":1751430859},{"type":"v1","addr":"172.21.15.28:6835","nonce":1751430859}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14466,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6841/666043045","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6840","nonce":666043045},{"type":"v1","addr":"172.21.15.28:6841","nonce":666043045}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24434,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6833/4248331494","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":4248331494},{"type":"v1","addr":"172.21.15.117:6833","nonce":4248331494}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24466,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.194:6833/2281745957","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6832","nonce":2281745957},{"type":"v1","addr":"172.21.15.194:6833","nonce":2281745957}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24470,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.28:6837/711950790","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6836","nonce":711950790},{"type":"v1","addr":"172.21.15.28:6837","nonce":711950790}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24482,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2458524548","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2458524548},{"type":"v1","addr":"172.21.15.117:6835","nonce":2458524548}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[{"mdsmap":{"epoch":26,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-04-24T08:07:51.585580+0000","modified":"2024-04-24T08:14:38.663141+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24574,"mds_1":24572,"mds_2":24556},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"h","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.117:6837/862115546","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6836","nonce":862115546},{"type":"v1","addr":"172.21.15.117:6837","nonce":862115546}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24523":{"gid":24523,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6835/356867408","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6834","nonce":356867408},{"type":"v1","addr":"172.21.15.194:6835","nonce":356867408}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24535":{"gid":24535,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.194:6837/3046177244","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6836","nonce":3046177244},{"type":"v1","addr":"172.21.15.194:6837","nonce":3046177244}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24556":{"gid":24556,"name":"i","rank":2,"incarnation":21,"state":"up:active","state_seq":7,"addr":"172.21.15.194:6839/2785495353","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6838","nonce":2785495353},{"type":"v1","addr":"172.21.15.194:6839","nonce":2785495353}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24572":{"gid":24572,"name":"b","rank":1,"incarnation":19,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6839/3606021351","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6838","nonce":3606021351},{"type":"v1","addr":"172.21.15.117:6839","nonce":3606021351}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24574":{"gid":24574,"name":"g","rank":0,"incarnation":14,"state":"up:active","state_seq":5,"addr":"172.21.15.28:6839/1247092633","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6838","nonce":1247092633},{"type":"v1","addr":"172.21.15.28:6839","nonce":1247092633}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24574,"qdb_cluster":[24556,24572,24574]},"id":1}]} 2024-04-24T08:22:10.378 INFO:teuthology.orchestra.run.smithi028.stderr:dumped fsmap epoch 26 2024-04-24T08:22:10.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.380+0000 7fa5cfc00640 1 -- 172.21.15.28:0/569324910 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa5b00a3da0 msgr2=0x7fa5b00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:10.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.380+0000 7fa5cfc00640 1 --2- 172.21.15.28:0/569324910 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa5b00a3da0 0x7fa5b00a6260 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fa5bc031550 tx=0x7fa5bc005850 comp rx=0 tx=0).stop 2024-04-24T08:22:10.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.380+0000 7fa5cfc00640 1 -- 172.21.15.28:0/569324910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5c8107310 msgr2=0x7fa5c806d850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:10.381 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.380+0000 7fa5cfc00640 1 --2- 172.21.15.28:0/569324910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5c8107310 0x7fa5c806d850 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fa5b800b3c0 tx=0x7fa5b800b880 comp rx=0 tx=0).stop 2024-04-24T08:22:10.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.381+0000 7fa5cfc00640 1 -- 172.21.15.28:0/569324910 shutdown_connections 2024-04-24T08:22:10.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.381+0000 7fa5cfc00640 1 --2- 172.21.15.28:0/569324910 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fa5b00a3da0 0x7fa5b00a6260 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:10.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.381+0000 7fa5cfc00640 1 --2- 172.21.15.28:0/569324910 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fa5c806dd90 0x7fa5c806e240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:10.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.381+0000 7fa5cfc00640 1 --2- 172.21.15.28:0/569324910 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fa5c8107310 0x7fa5c806d850 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:10.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.381+0000 7fa5cfc00640 1 --2- 172.21.15.28:0/569324910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5c8102240 0x7fa5c80737e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:10.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.381+0000 7fa5cfc00640 1 -- 172.21.15.28:0/569324910 >> 172.21.15.28:0/569324910 conn(0x7fa5c80793c0 msgr2=0x7fa5c8100150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:10.382 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.381+0000 7fa5cfc00640 1 -- 172.21.15.28:0/569324910 shutdown_connections 2024-04-24T08:22:10.383 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.381+0000 7fa5cfc00640 1 -- 172.21.15.28:0/569324910 wait complete. 2024-04-24T08:22:10.412 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-04-24T08:22:10.716 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.716+0000 7f04a3200640 1 -- 172.21.15.28:0/2127788244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a4069db0 msgr2=0x7f04a4106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.716+0000 7f04a3200640 1 --2- 172.21.15.28:0/2127788244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a4069db0 0x7f04a4106a70 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f0490009860 tx=0x7f049002a360 comp rx=0 tx=0).stop 2024-04-24T08:22:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.716+0000 7f04a3200640 1 -- 172.21.15.28:0/2127788244 shutdown_connections 2024-04-24T08:22:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.716+0000 7f04a3200640 1 --2- 172.21.15.28:0/2127788244 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a4069db0 0x7f04a4106a70 secure :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f0490009860 tx=0x7f049002a360 comp rx=0 tx=0).stop 2024-04-24T08:22:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.716+0000 7f04a3200640 1 --2- 172.21.15.28:0/2127788244 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04a41071a0 0x7f04a410ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.716+0000 7f04a3200640 1 --2- 172.21.15.28:0/2127788244 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04a4069400 0x7f04a40697e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:10.717 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.716+0000 7f04a3200640 1 -- 172.21.15.28:0/2127788244 >> 172.21.15.28:0/2127788244 conn(0x7f04a4076840 msgr2=0x7f04a4076c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.718+0000 7f04a3200640 1 -- 172.21.15.28:0/2127788244 shutdown_connections 2024-04-24T08:22:10.718 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.718+0000 7f04a3200640 1 -- 172.21.15.28:0/2127788244 wait complete. 2024-04-24T08:22:10.719 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.719+0000 7f04a3200640 1 Processor -- start 2024-04-24T08:22:10.720 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.719+0000 7f04a3200640 1 -- start start 2024-04-24T08:22:10.724 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.720+0000 7f04a3200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a4069400 0x7f04a419e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.720+0000 7f04a3200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04a41071a0 0x7f04a419ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.720+0000 7f04a3200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04a4198770 0x7f04a4198bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.720+0000 7f04a3200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f04a41121e0 con 0x7f04a4069400 2024-04-24T08:22:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.720+0000 7f04a3200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f04a4112060 con 0x7f04a4198770 2024-04-24T08:22:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.720+0000 7f04a3200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f04a4112360 con 0x7f04a41071a0 2024-04-24T08:22:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.720+0000 7f04a1400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04a41071a0 0x7f04a419ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.720+0000 7f04a1e00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a4069400 0x7f04a419e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:10.725 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.720+0000 7f04a1400640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04a41071a0 0x7f04a419ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.194:3300/0 says I am v2:172.21.15.28:33024/0 (socket says 172.21.15.28:33024) 2024-04-24T08:22:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.720+0000 7f04a1400640 1 -- 172.21.15.28:0/722293549 learned_addr learned my addr 172.21.15.28:0/722293549 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:22:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.720+0000 7f04a2800640 1 --2- 172.21.15.28:0/722293549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04a4198770 0x7f04a4198bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.721+0000 7f04a1400640 1 -- 172.21.15.28:0/722293549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04a4198770 msgr2=0x7f04a4198bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.721+0000 7f04a1400640 1 --2- 172.21.15.28:0/722293549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04a4198770 0x7f04a4198bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.721+0000 7f04a1400640 1 -- 172.21.15.28:0/722293549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a4069400 msgr2=0x7f04a419e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.721+0000 7f04a1400640 1 --2- 172.21.15.28:0/722293549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a4069400 0x7f04a419e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:10.726 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.721+0000 7f04a1400640 1 -- 172.21.15.28:0/722293549 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f04a4199370 con 0x7f04a41071a0 2024-04-24T08:22:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.721+0000 7f04a1e00640 1 --2- 172.21.15.28:0/722293549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a4069400 0x7f04a419e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-04-24T08:22:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.721+0000 7f04a2800640 1 --2- 172.21.15.28:0/722293549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04a4198770 0x7f04a4198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:22:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.721+0000 7f04a1400640 1 --2- 172.21.15.28:0/722293549 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04a41071a0 0x7f04a419ebf0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0490035040 tx=0x7f0490032830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.722+0000 7f049aa00640 1 -- 172.21.15.28:0/722293549 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490004040 con 0x7f04a41071a0 2024-04-24T08:22:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.722+0000 7f049aa00640 1 -- 172.21.15.28:0/722293549 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0490048040 con 0x7f04a41071a0 2024-04-24T08:22:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.722+0000 7f04a3200640 1 -- 172.21.15.28:0/722293549 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f04a4199600 con 0x7f04a41071a0 2024-04-24T08:22:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.722+0000 7f049aa00640 1 -- 172.21.15.28:0/722293549 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0490005550 con 0x7f04a41071a0 2024-04-24T08:22:10.727 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.722+0000 7f04a3200640 1 -- 172.21.15.28:0/722293549 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f04a406bd00 con 0x7f04a41071a0 2024-04-24T08:22:10.728 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.725+0000 7f04a3200640 1 -- 172.21.15.28:0/722293549 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f04a4103550 con 0x7f04a41071a0 2024-04-24T08:22:10.733 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.732+0000 7f049aa00640 1 -- 172.21.15.28:0/722293549 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f0490004300 con 0x7f04a41071a0 2024-04-24T08:22:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.736+0000 7f049aa00640 1 --2- 172.21.15.28:0/722293549 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f04880c5db0 0x7f04880c8270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:10.736 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.736+0000 7f049aa00640 1 -- 172.21.15.28:0/722293549 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8644+0+0 (secure 0 0 0) 0x7f04900efa40 con 0x7f04a41071a0 2024-04-24T08:22:10.737 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.736+0000 7f04a1e00640 1 --2- 172.21.15.28:0/722293549 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f04880c5db0 0x7f04880c8270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.738+0000 7f04a1e00640 1 --2- 172.21.15.28:0/722293549 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f04880c5db0 0x7f04880c8270 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0494005fd0 tx=0x7f049400b390 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:10.738 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:10.738+0000 7f049aa00640 1 -- 172.21.15.28:0/722293549 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f04900bc310 con 0x7f04a41071a0 2024-04-24T08:22:11.121 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.120+0000 7f04a3200640 1 -- 172.21.15.28:0/722293549 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f04a410a370 con 0x7f04a41071a0 2024-04-24T08:22:11.123 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.123+0000 7f049aa00640 1 -- 172.21.15.28:0/722293549 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v102) ==== 74+0+23259 (secure 0 0 0) 0x7f0490045af0 con 0x7f04a41071a0 2024-04-24T08:22:11.127 INFO:teuthology.orchestra.run.smithi028.stdout: 2024-04-24T08:22:11.127 INFO:teuthology.orchestra.run.smithi028.stdout:{"epoch":102,"fsid":"473aaf58-0210-11ef-bc93-c7b262605968","created":"2024-04-24T07:58:42.318512+0000","modified":"2024-04-24T08:22:08.796042+0000","last_up_change":"2024-04-24T08:06:54.741161+0000","last_in_change":"2024-04-24T08:06:34.014803+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-04-24T08:02:34.086371+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-04-24T08:07:49.819536+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-04-24T08:07:50.493486+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"102","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":102,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9400ba83-bcfd-4b69-9a1e-4a3cd9795b6a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6802","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6803","nonce":1414246664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6804","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6805","nonce":1414246664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6808","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6809","nonce":1414246664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6806","nonce":1414246664},{"type":"v1","addr":"172.21.15.28:6807","nonce":1414246664}]},"public_addr":"172.21.15.28:6803/1414246664","cluster_addr":"172.21.15.28:6805/1414246664","heartbeat_back_addr":"172.21.15.28:6809/1414246664","heartbeat_front_addr":"172.21.15.28:6807/1414246664","state":["exists","up"]},{"osd":1,"uuid":"a1b6b23d-4deb-4210-af96-64b7217b06aa","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6801","nonce":3269221814}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6803","nonce":3269221814}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6807","nonce":3269221814}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":3269221814},{"type":"v1","addr":"172.21.15.117:6805","nonce":3269221814}]},"public_addr":"172.21.15.117:6801/3269221814","cluster_addr":"172.21.15.117:6803/3269221814","heartbeat_back_addr":"172.21.15.117:6807/3269221814","heartbeat_front_addr":"172.21.15.117:6805/3269221814","state":["exists","up"]},{"osd":2,"uuid":"223ab66d-7bb6-4225-b292-009846c746d4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6800","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6801","nonce":65787773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6802","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6803","nonce":65787773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6806","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6807","nonce":65787773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6804","nonce":65787773},{"type":"v1","addr":"172.21.15.194:6805","nonce":65787773}]},"public_addr":"172.21.15.194:6801/65787773","cluster_addr":"172.21.15.194:6803/65787773","heartbeat_back_addr":"172.21.15.194:6807/65787773","heartbeat_front_addr":"172.21.15.194:6805/65787773","state":["exists","up"]},{"osd":3,"uuid":"32e5f179-3598-4a9b-93c4-48728d942883","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6810","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6811","nonce":3962695585}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6812","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6813","nonce":3962695585}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6816","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6817","nonce":3962695585}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6814","nonce":3962695585},{"type":"v1","addr":"172.21.15.28:6815","nonce":3962695585}]},"public_addr":"172.21.15.28:6811/3962695585","cluster_addr":"172.21.15.28:6813/3962695585","heartbeat_back_addr":"172.21.15.28:6817/3962695585","heartbeat_front_addr":"172.21.15.28:6815/3962695585","state":["exists","up"]},{"osd":4,"uuid":"c07ffaec-979d-4a1d-b2fd-0d92b3fd303a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6809","nonce":1649446681}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6811","nonce":1649446681}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6815","nonce":1649446681}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":1649446681},{"type":"v1","addr":"172.21.15.117:6813","nonce":1649446681}]},"public_addr":"172.21.15.117:6809/1649446681","cluster_addr":"172.21.15.117:6811/1649446681","heartbeat_back_addr":"172.21.15.117:6815/1649446681","heartbeat_front_addr":"172.21.15.117:6813/1649446681","state":["exists","up"]},{"osd":5,"uuid":"fcbda2ee-300e-498a-bb5c-c1d112d03f05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6808","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6809","nonce":1399608260}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6810","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6811","nonce":1399608260}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6814","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6815","nonce":1399608260}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6812","nonce":1399608260},{"type":"v1","addr":"172.21.15.194:6813","nonce":1399608260}]},"public_addr":"172.21.15.194:6809/1399608260","cluster_addr":"172.21.15.194:6811/1399608260","heartbeat_back_addr":"172.21.15.194:6815/1399608260","heartbeat_front_addr":"172.21.15.194:6813/1399608260","state":["exists","up"]},{"osd":6,"uuid":"6c606db5-1d3f-4f2e-bb7e-0ce5d4e31ba4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6818","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6819","nonce":1356466279}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6820","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6821","nonce":1356466279}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6824","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6825","nonce":1356466279}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6822","nonce":1356466279},{"type":"v1","addr":"172.21.15.28:6823","nonce":1356466279}]},"public_addr":"172.21.15.28:6819/1356466279","cluster_addr":"172.21.15.28:6821/1356466279","heartbeat_back_addr":"172.21.15.28:6825/1356466279","heartbeat_front_addr":"172.21.15.28:6823/1356466279","state":["exists","up"]},{"osd":7,"uuid":"657dc8f6-2eae-4b01-b57c-9bd7fbc5c541","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6817","nonce":1472212690}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6819","nonce":1472212690}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6823","nonce":1472212690}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":1472212690},{"type":"v1","addr":"172.21.15.117:6821","nonce":1472212690}]},"public_addr":"172.21.15.117:6817/1472212690","cluster_addr":"172.21.15.117:6819/1472212690","heartbeat_back_addr":"172.21.15.117:6823/1472212690","heartbeat_front_addr":"172.21.15.117:6821/1472212690","state":["exists","up"]},{"osd":8,"uuid":"7bd54baf-23e4-4f9d-8c03-79bc64308158","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6816","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6817","nonce":301057688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6818","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6819","nonce":301057688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6822","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6823","nonce":301057688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6820","nonce":301057688},{"type":"v1","addr":"172.21.15.194:6821","nonce":301057688}]},"public_addr":"172.21.15.194:6817/301057688","cluster_addr":"172.21.15.194:6819/301057688","heartbeat_back_addr":"172.21.15.194:6823/301057688","heartbeat_front_addr":"172.21.15.194:6821/301057688","state":["exists","up"]},{"osd":9,"uuid":"c9268da7-6a4a-4f31-b4ff-99e934adc71e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6826","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6827","nonce":2537306688}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6828","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6829","nonce":2537306688}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6832","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6833","nonce":2537306688}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.28:6830","nonce":2537306688},{"type":"v1","addr":"172.21.15.28:6831","nonce":2537306688}]},"public_addr":"172.21.15.28:6827/2537306688","cluster_addr":"172.21.15.28:6829/2537306688","heartbeat_back_addr":"172.21.15.28:6833/2537306688","heartbeat_front_addr":"172.21.15.28:6831/2537306688","state":["exists","up"]},{"osd":10,"uuid":"107dfd4b-d52f-4fdd-925e-d1fcac3cffe7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6825","nonce":2345072654}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6826","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6827","nonce":2345072654}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6831","nonce":2345072654}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":2345072654},{"type":"v1","addr":"172.21.15.117:6829","nonce":2345072654}]},"public_addr":"172.21.15.117:6825/2345072654","cluster_addr":"172.21.15.117:6827/2345072654","heartbeat_back_addr":"172.21.15.117:6831/2345072654","heartbeat_front_addr":"172.21.15.117:6829/2345072654","state":["exists","up"]},{"osd":11,"uuid":"22674865-b8f4-445d-8152-ba0a903586b4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6824","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6825","nonce":275366773}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6826","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6827","nonce":275366773}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6830","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6831","nonce":275366773}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.194:6828","nonce":275366773},{"type":"v1","addr":"172.21.15.194:6829","nonce":275366773}]},"public_addr":"172.21.15.194:6825/275366773","cluster_addr":"172.21.15.194:6827/275366773","heartbeat_back_addr":"172.21.15.194:6831/275366773","heartbeat_front_addr":"172.21.15.194:6829/275366773","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:01:42.212371+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:07.585497+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:02:31.549078+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:05.686319+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:31.481412+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:03:56.599301+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:31.328935+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:04:58.014813+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:23.618881+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:05:59.062605+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:26.111104+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-04-24T08:06:52.241834+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.6","mappings":[{"from":8,"to":11}]},{"pgid":"2.c","mappings":[{"from":2,"to":4}]},{"pgid":"2.13","mappings":[{"from":2,"to":11}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.e","mappings":[{"from":4,"to":8}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.1b","mappings":[{"from":4,"to":7}]},{"pgid":"3.21","mappings":[{"from":5,"to":9}]},{"pgid":"3.22","mappings":[{"from":3,"to":9}]},{"pgid":"3.23","mappings":[{"from":4,"to":8}]},{"pgid":"3.28","mappings":[{"from":3,"to":6}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.28:0/1819419617":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6801/361125225":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/3758227512":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/874707228":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6801/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/409391969":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/2660639233":"2024-04-25T08:10:20.116072+0000","172.21.15.28:6800/3308597107":"2024-04-25T07:59:49.915756+0000","172.21.15.28:0/3378984747":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6800/361125225":"2024-04-25T08:10:20.116072+0000","172.21.15.28:0/4131458386":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/2417007332":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/234090859":"2024-04-25T07:59:15.381957+0000","172.21.15.28:6800/2371619813":"2024-04-25T07:59:15.381957+0000","172.21.15.28:0/4081129566":"2024-04-25T07:59:49.915756+0000","172.21.15.28:6801/2371619813":"2024-04-25T07:59:15.381957+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-04-24T08:22:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.126+0000 7f04a3200640 1 -- 172.21.15.28:0/722293549 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f04880c5db0 msgr2=0x7f04880c8270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.126+0000 7f04a3200640 1 --2- 172.21.15.28:0/722293549 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f04880c5db0 0x7f04880c8270 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f0494005fd0 tx=0x7f049400b390 comp rx=0 tx=0).stop 2024-04-24T08:22:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.126+0000 7f04a3200640 1 -- 172.21.15.28:0/722293549 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04a41071a0 msgr2=0x7f04a419ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.126+0000 7f04a3200640 1 --2- 172.21.15.28:0/722293549 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04a41071a0 0x7f04a419ebf0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f0490035040 tx=0x7f0490032830 comp rx=0 tx=0).stop 2024-04-24T08:22:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.126+0000 7f04a3200640 1 -- 172.21.15.28:0/722293549 shutdown_connections 2024-04-24T08:22:11.128 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.126+0000 7f04a3200640 1 --2- 172.21.15.28:0/722293549 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f04880c5db0 0x7f04880c8270 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:11.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.126+0000 7f04a3200640 1 --2- 172.21.15.28:0/722293549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f04a4198770 0x7f04a4198bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:11.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.126+0000 7f04a3200640 1 --2- 172.21.15.28:0/722293549 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f04a41071a0 0x7f04a419ebf0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:11.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.126+0000 7f04a3200640 1 --2- 172.21.15.28:0/722293549 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f04a4069400 0x7f04a419e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:11.129 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.126+0000 7f04a3200640 1 -- 172.21.15.28:0/722293549 >> 172.21.15.28:0/722293549 conn(0x7f04a4076840 msgr2=0x7f04a40ff200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:11.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:10 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/569324910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:22:11.130 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.128+0000 7f04a3200640 1 -- 172.21.15.28:0/722293549 shutdown_connections 2024-04-24T08:22:11.131 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.131+0000 7f04a3200640 1 -- 172.21.15.28:0/722293549 wait complete. 2024-04-24T08:22:11.158 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:10 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/569324910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:22:11.162 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-04-24T08:22:11.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:10 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/569324910' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-04-24T08:22:11.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.445+0000 7fe6f4a00640 1 -- 172.21.15.28:0/1804078312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f00743b0 msgr2=0x7fe6f0074790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:11.446 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.445+0000 7fe6f4a00640 1 --2- 172.21.15.28:0/1804078312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f00743b0 0x7fe6f0074790 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fe6e0009900 tx=0x7fe6e002a480 comp rx=0 tx=0).stop 2024-04-24T08:22:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.446+0000 7fe6f4a00640 1 -- 172.21.15.28:0/1804078312 shutdown_connections 2024-04-24T08:22:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.446+0000 7fe6f4a00640 1 --2- 172.21.15.28:0/1804078312 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6f010e760 0x7fe6f0110b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.446+0000 7fe6f4a00640 1 --2- 172.21.15.28:0/1804078312 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6f0074cd0 0x7fe6f010e220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.446+0000 7fe6f4a00640 1 --2- 172.21.15.28:0/1804078312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f00743b0 0x7fe6f0074790 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:11.447 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.446+0000 7fe6f4a00640 1 -- 172.21.15.28:0/1804078312 >> 172.21.15.28:0/1804078312 conn(0x7fe6f00fd9d0 msgr2=0x7fe6f00ffdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:11.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.446+0000 7fe6f4a00640 1 -- 172.21.15.28:0/1804078312 shutdown_connections 2024-04-24T08:22:11.448 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.446+0000 7fe6f4a00640 1 -- 172.21.15.28:0/1804078312 wait complete. 2024-04-24T08:22:11.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.447+0000 7fe6f4a00640 1 Processor -- start 2024-04-24T08:22:11.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.447+0000 7fe6f4a00640 1 -- start start 2024-04-24T08:22:11.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.448+0000 7fe6f4a00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f00743b0 0x7fe6f0101e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:11.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.448+0000 7fe6f4a00640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6f0074cd0 0x7fe6f01023c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:11.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.448+0000 7fe6f4a00640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6f010e760 0x7fe6f0102900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:11.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.448+0000 7fe6f4a00640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f0114520 con 0x7fe6f010e760 2024-04-24T08:22:11.449 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.448+0000 7fe6f4a00640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f01143a0 con 0x7fe6f00743b0 2024-04-24T08:22:11.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.448+0000 7fe6f4a00640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fe6f01146a0 con 0x7fe6f0074cd0 2024-04-24T08:22:11.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.448+0000 7fe6ef400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f00743b0 0x7fe6f0101e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:11.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.448+0000 7fe6ef400640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f00743b0 0x7fe6f0101e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49482/0 (socket says 172.21.15.28:49482) 2024-04-24T08:22:11.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.448+0000 7fe6ef400640 1 -- 172.21.15.28:0/1934256073 learned_addr learned my addr 172.21.15.28:0/1934256073 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:22:11.450 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.448+0000 7fe6eea00640 1 --2- 172.21.15.28:0/1934256073 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6f0074cd0 0x7fe6f01023c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:11.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.449+0000 7fe6eea00640 1 -- 172.21.15.28:0/1934256073 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f00743b0 msgr2=0x7fe6f0101e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:11.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.449+0000 7fe6eea00640 1 --2- 172.21.15.28:0/1934256073 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f00743b0 0x7fe6f0101e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:11.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.449+0000 7fe6eea00640 1 -- 172.21.15.28:0/1934256073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6f010e760 msgr2=0x7fe6f0102900 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-04-24T08:22:11.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.451+0000 7fe6eea00640 1 --2- 172.21.15.28:0/1934256073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6f010e760 0x7fe6f0102900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:11.451 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.451+0000 7fe6eea00640 1 -- 172.21.15.28:0/1934256073 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6f01aaf90 con 0x7fe6f0074cd0 2024-04-24T08:22:11.452 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.452+0000 7fe6eea00640 1 --2- 172.21.15.28:0/1934256073 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6f0074cd0 0x7fe6f01023c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00b3f0 tx=0x7fe6dc00b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.453+0000 7fe6e7e00640 1 -- 172.21.15.28:0/1934256073 <== mon.1 v2:172.21.15.194:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc015020 con 0x7fe6f0074cd0 2024-04-24T08:22:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.453+0000 7fe6e7e00640 1 -- 172.21.15.28:0/1934256073 <== mon.1 v2:172.21.15.194:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe6dc011d10 con 0x7fe6f0074cd0 2024-04-24T08:22:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.453+0000 7fe6e7e00640 1 -- 172.21.15.28:0/1934256073 <== mon.1 v2:172.21.15.194:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe6dc00f2e0 con 0x7fe6f0074cd0 2024-04-24T08:22:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.453+0000 7fe6f4a00640 1 -- 172.21.15.28:0/1934256073 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6f01ab280 con 0x7fe6f0074cd0 2024-04-24T08:22:11.455 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.454+0000 7fe6f4a00640 1 -- 172.21.15.28:0/1934256073 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6f01ab780 con 0x7fe6f0074cd0 2024-04-24T08:22:11.458 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.457+0000 7fe6f4a00640 1 -- 172.21.15.28:0/1934256073 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe6d0005290 con 0x7fe6f0074cd0 2024-04-24T08:22:11.464 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.463+0000 7fe6e7e00640 1 -- 172.21.15.28:0/1934256073 <== mon.1 v2:172.21.15.194:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fe6dc018020 con 0x7fe6f0074cd0 2024-04-24T08:22:11.469 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.468+0000 7fe6e7e00640 1 --2- 172.21.15.28:0/1934256073 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe6d40a3f20 0x7fe6d40a63e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:11.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.469+0000 7fe6e7e00640 1 -- 172.21.15.28:0/1934256073 <== mon.1 v2:172.21.15.194:3300/0 5 ==== osd_map(102..102 src has 1..102) ==== 8644+0+0 (secure 0 0 0) 0x7fe6dc0d0700 con 0x7fe6f0074cd0 2024-04-24T08:22:11.470 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.470+0000 7fe6e7e00640 1 -- 172.21.15.28:0/1934256073 <== mon.1 v2:172.21.15.194:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fe6dc045030 con 0x7fe6f0074cd0 2024-04-24T08:22:11.471 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.470+0000 7fe6ef400640 1 --2- 172.21.15.28:0/1934256073 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe6d40a3f20 0x7fe6d40a63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:11.473 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.473+0000 7fe6ef400640 1 --2- 172.21.15.28:0/1934256073 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe6d40a3f20 0x7fe6d40a63e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe6e00096c0 tx=0x7fe6e002a980 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:11.922 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.922+0000 7fe6f4a00640 1 -- 172.21.15.28:0/1934256073 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7fe6d0005850 con 0x7fe6f0074cd0 2024-04-24T08:22:11.962 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.961+0000 7fe6e7e00640 1 -- 172.21.15.28:0/1934256073 <== mon.1 v2:172.21.15.194:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v27) ==== 161+0+0 (secure 0 0 0) 0x7fe6dc09ce50 con 0x7fe6f0074cd0 2024-04-24T08:22:11.965 INFO:teuthology.orchestra.run.smithi028.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-04-24T08:22:11.967 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.967+0000 7fe6f4a00640 1 -- 172.21.15.28:0/1934256073 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe6d40a3f20 msgr2=0x7fe6d40a63e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:11.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.967+0000 7fe6f4a00640 1 --2- 172.21.15.28:0/1934256073 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe6d40a3f20 0x7fe6d40a63e0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe6e00096c0 tx=0x7fe6e002a980 comp rx=0 tx=0).stop 2024-04-24T08:22:11.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.967+0000 7fe6f4a00640 1 -- 172.21.15.28:0/1934256073 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6f0074cd0 msgr2=0x7fe6f01023c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:11.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.967+0000 7fe6f4a00640 1 --2- 172.21.15.28:0/1934256073 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6f0074cd0 0x7fe6f01023c0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7fe6dc00b3f0 tx=0x7fe6dc00b8b0 comp rx=0 tx=0).stop 2024-04-24T08:22:11.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.967+0000 7fe6f4a00640 1 -- 172.21.15.28:0/1934256073 shutdown_connections 2024-04-24T08:22:11.969 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.967+0000 7fe6f4a00640 1 --2- 172.21.15.28:0/1934256073 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fe6d40a3f20 0x7fe6d40a63e0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:11.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.967+0000 7fe6f4a00640 1 --2- 172.21.15.28:0/1934256073 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fe6f010e760 0x7fe6f0102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:11.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.967+0000 7fe6f4a00640 1 --2- 172.21.15.28:0/1934256073 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fe6f0074cd0 0x7fe6f01023c0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:11.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.967+0000 7fe6f4a00640 1 --2- 172.21.15.28:0/1934256073 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f00743b0 0x7fe6f0101e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:11.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.967+0000 7fe6f4a00640 1 -- 172.21.15.28:0/1934256073 >> 172.21.15.28:0/1934256073 conn(0x7fe6f00fd9d0 msgr2=0x7fe6f00fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:11.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.969+0000 7fe6f4a00640 1 -- 172.21.15.28:0/1934256073 shutdown_connections 2024-04-24T08:22:11.970 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:11.969+0000 7fe6f4a00640 1 -- 172.21.15.28:0/1934256073 wait complete. 2024-04-24T08:22:12.012 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-04-24T08:22:12.024 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:11 smithi117 ceph-mon[32438]: pgmap v369: 97 pgs: 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 79 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-24T08:22:12.024 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:11 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/722293549' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:22:12.112 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:11 smithi028 ceph-mon[27786]: pgmap v369: 97 pgs: 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 79 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-24T08:22:12.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:11 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/722293549' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:22:12.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:11 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a[27782]: 2024-04-24T08:22:11.925+0000 7f5b3fe00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:22:12.190 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:11 smithi194 ceph-mon[32836]: pgmap v369: 97 pgs: 10 active+clean+snaptrim_wait, 8 active+clean+snaptrim, 79 active+clean; 15 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 16 KiB/s rd, 54 KiB/s wr, 27 op/s 2024-04-24T08:22:12.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:11 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/722293549' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-04-24T08:22:12.191 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:22:12 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-c[63839]: ignoring --setuser ceph since I am not root 2024-04-24T08:22:12.191 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:22:12 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-c[63839]: ignoring --setgroup ceph since I am not root 2024-04-24T08:22:12.191 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:22:12 smithi194 ceph-mds[63843]: -- 172.21.15.194:0/175003342 <== mon.1 v2:172.21.15.194:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x559215f2ba40 con 0x559215eff000 2024-04-24T08:22:12.192 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:22:12 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-c[63839]: starting mds.c at 2024-04-24T08:22:12.192 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:22:12 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-f[63533]: ignoring --setuser ceph since I am not root 2024-04-24T08:22:12.192 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:22:12 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-f[63533]: ignoring --setgroup ceph since I am not root 2024-04-24T08:22:12.192 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:22:12 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-f[63533]: starting mds.f at 2024-04-24T08:22:12.192 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:22:12 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-i[64160]: ignoring --setuser ceph since I am not root 2024-04-24T08:22:12.193 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:22:12 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-i[64160]: ignoring --setgroup ceph since I am not root 2024-04-24T08:22:12.193 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:22:12 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-i[64160]: starting mds.i at 2024-04-24T08:22:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.326+0000 7f16c7200640 1 -- 172.21.15.28:0/914933178 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16c8108580 msgr2=0x7f16c8108a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.326+0000 7f16c7200640 1 --2- 172.21.15.28:0/914933178 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16c8108580 0x7f16c8108a00 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f16b0009900 tx=0x7f16b002a3f0 comp rx=0 tx=0).stop 2024-04-24T08:22:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.326+0000 7f16c7200640 1 -- 172.21.15.28:0/914933178 shutdown_connections 2024-04-24T08:22:12.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.326+0000 7f16c7200640 1 --2- 172.21.15.28:0/914933178 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c8108f40 0x7f16c81153b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.326+0000 7f16c7200640 1 --2- 172.21.15.28:0/914933178 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16c8108580 0x7f16c8108a00 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.326+0000 7f16c7200640 1 --2- 172.21.15.28:0/914933178 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16c810d500 0x7f16c810d8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.326+0000 7f16c7200640 1 -- 172.21.15.28:0/914933178 >> 172.21.15.28:0/914933178 conn(0x7f16c8077e20 msgr2=0x7f16c8106e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.328+0000 7f16c7200640 1 -- 172.21.15.28:0/914933178 shutdown_connections 2024-04-24T08:22:12.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.328+0000 7f16c7200640 1 -- 172.21.15.28:0/914933178 wait complete. 2024-04-24T08:22:12.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.329+0000 7f16c7200640 1 Processor -- start 2024-04-24T08:22:12.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.329+0000 7f16c7200640 1 -- start start 2024-04-24T08:22:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.330+0000 7f16c7200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c8108580 0x7f16c81a4950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.330+0000 7f16c7200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16c8108f40 0x7f16c81a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.330+0000 7f16c7200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16c810d500 0x7f16c819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.330+0000 7f16c7200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7f16c81188b0 con 0x7f16c8108580 2024-04-24T08:22:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.330+0000 7f16bfe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16c8108f40 0x7f16c81a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.330+0000 7f16c7200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7f16c8118730 con 0x7f16c8108f40 2024-04-24T08:22:12.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.330+0000 7f16bfe00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16c8108f40 0x7f16c81a4e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49520/0 (socket says 172.21.15.28:49520) 2024-04-24T08:22:12.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.330+0000 7f16bfe00640 1 -- 172.21.15.28:0/1857340029 learned_addr learned my addr 172.21.15.28:0/1857340029 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:22:12.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.330+0000 7f16c6800640 1 --2- 172.21.15.28:0/1857340029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16c810d500 0x7f16c819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:12.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.330+0000 7f16c7200640 1 -- 172.21.15.28:0/1857340029 --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7f16c8118a30 con 0x7f16c810d500 2024-04-24T08:22:12.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.331+0000 7f16bfe00640 1 -- 172.21.15.28:0/1857340029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16c810d500 msgr2=0x7f16c819eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:12.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.331+0000 7f16c5e00640 1 --2- 172.21.15.28:0/1857340029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c8108580 0x7f16c81a4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:12.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.331+0000 7f16bfe00640 1 --2- 172.21.15.28:0/1857340029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16c810d500 0x7f16c819eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:12.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.331+0000 7f16bfe00640 1 -- 172.21.15.28:0/1857340029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c8108580 msgr2=0x7f16c81a4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:12.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.331+0000 7f16bfe00640 1 --2- 172.21.15.28:0/1857340029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c8108580 0x7f16c81a4950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:12.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.331+0000 7f16bfe00640 1 -- 172.21.15.28:0/1857340029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f16c819f2d0 con 0x7f16c8108f40 2024-04-24T08:22:12.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.332+0000 7f16bfe00640 1 --2- 172.21.15.28:0/1857340029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16c8108f40 0x7f16c81a4e90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f16b0006010 tx=0x7f16b0003c10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:12.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.333+0000 7f16bea00640 1 -- 172.21.15.28:0/1857340029 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b003f070 con 0x7f16c8108f40 2024-04-24T08:22:12.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.333+0000 7f16c7200640 1 -- 172.21.15.28:0/1857340029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f16c819f560 con 0x7f16c8108f40 2024-04-24T08:22:12.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.333+0000 7f16bea00640 1 -- 172.21.15.28:0/1857340029 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f16b0031c30 con 0x7f16c8108f40 2024-04-24T08:22:12.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.333+0000 7f16bea00640 1 -- 172.21.15.28:0/1857340029 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f16b0035030 con 0x7f16c8108f40 2024-04-24T08:22:12.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.333+0000 7f16c7200640 1 -- 172.21.15.28:0/1857340029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f16c81ab810 con 0x7f16c8108f40 2024-04-24T08:22:12.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.336+0000 7f16c7200640 1 -- 172.21.15.28:0/1857340029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f16c8069190 con 0x7f16c8108f40 2024-04-24T08:22:12.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.339+0000 7f16bea00640 1 -- 172.21.15.28:0/1857340029 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7f16b0042070 con 0x7f16c8108f40 2024-04-24T08:22:12.342 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.342+0000 7f16bea00640 1 --2- 172.21.15.28:0/1857340029 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f16ac0a3da0 0x7f16ac0a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:12.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.342+0000 7f16bea00640 1 -- 172.21.15.28:0/1857340029 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8858+0+0 (secure 0 0 0) 0x7f16b00f14c0 con 0x7f16c8108f40 2024-04-24T08:22:12.343 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.343+0000 7f16c5e00640 1 --2- 172.21.15.28:0/1857340029 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f16ac0a3da0 0x7f16ac0a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:12.344 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.344+0000 7f16c5e00640 1 --2- 172.21.15.28:0/1857340029 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f16ac0a3da0 0x7f16ac0a6260 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f16c806b340 tx=0x7f16b8005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:12.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.346+0000 7f16bea00640 1 -- 172.21.15.28:0/1857340029 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7f16b00bda10 con 0x7f16c8108f40 2024-04-24T08:22:12.378 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:22:12 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-b[62891]: ignoring --setuser ceph since I am not root 2024-04-24T08:22:12.378 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:22:12 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-b[62891]: ignoring --setgroup ceph since I am not root 2024-04-24T08:22:12.378 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:22:12 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-b[62891]: starting mds.b at 2024-04-24T08:22:12.379 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:22:12 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-h[62567]: ignoring --setuser ceph since I am not root 2024-04-24T08:22:12.379 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:22:12 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-h[62567]: ignoring --setgroup ceph since I am not root 2024-04-24T08:22:12.379 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:22:12 smithi117 ceph-mds[62571]: -- 172.21.15.117:0/3787851270 <== mon.2 v2:172.21.15.117:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55a266017a40 con 0x55a265feac00 2024-04-24T08:22:12.379 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:22:12 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-h[62567]: starting mds.h at 2024-04-24T08:22:12.540 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 24 08:22:12 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-g[66667]: ignoring --setuser ceph since I am not root 2024-04-24T08:22:12.540 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 24 08:22:12 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-g[66667]: ignoring --setgroup ceph since I am not root 2024-04-24T08:22:12.540 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 24 08:22:12 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-g[66667]: starting mds.g at 2024-04-24T08:22:12.842 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.841+0000 7f16c7200640 1 -- 172.21.15.28:0/1857340029 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f16c8110a50 con 0x7f16c8108f40 2024-04-24T08:22:12.968 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.968+0000 7f16bea00640 1 -- 172.21.15.28:0/1857340029 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v28) ==== 104+0+0 (secure 0 0 0) 0x7f16b00bd310 con 0x7f16c8108f40 2024-04-24T08:22:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.970+0000 7f16c7200640 1 -- 172.21.15.28:0/1857340029 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f16ac0a3da0 msgr2=0x7f16ac0a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:12.971 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.970+0000 7f16c7200640 1 --2- 172.21.15.28:0/1857340029 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f16ac0a3da0 0x7f16ac0a6260 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f16c806b340 tx=0x7f16b8005f90 comp rx=0 tx=0).stop 2024-04-24T08:22:12.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.970+0000 7f16c7200640 1 -- 172.21.15.28:0/1857340029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16c8108f40 msgr2=0x7f16c81a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:12.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.970+0000 7f16c7200640 1 --2- 172.21.15.28:0/1857340029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16c8108f40 0x7f16c81a4e90 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f16b0006010 tx=0x7f16b0003c10 comp rx=0 tx=0).stop 2024-04-24T08:22:12.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.971+0000 7f16c7200640 1 -- 172.21.15.28:0/1857340029 shutdown_connections 2024-04-24T08:22:12.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.971+0000 7f16c7200640 1 --2- 172.21.15.28:0/1857340029 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7f16ac0a3da0 0x7f16ac0a6260 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:12.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.971+0000 7f16c7200640 1 --2- 172.21.15.28:0/1857340029 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7f16c810d500 0x7f16c819eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:12.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.971+0000 7f16c7200640 1 --2- 172.21.15.28:0/1857340029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16c8108f40 0x7f16c81a4e90 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:12.972 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.971+0000 7f16c7200640 1 --2- 172.21.15.28:0/1857340029 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7f16c8108580 0x7f16c81a4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.971+0000 7f16c7200640 1 -- 172.21.15.28:0/1857340029 >> 172.21.15.28:0/1857340029 conn(0x7f16c8077e20 msgr2=0x7f16c8078380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.971+0000 7f16c7200640 1 -- 172.21.15.28:0/1857340029 shutdown_connections 2024-04-24T08:22:12.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:12.971+0000 7f16c7200640 1 -- 172.21.15.28:0/1857340029 wait complete. 2024-04-24T08:22:12.992 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-04-24T08:22:13.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:12 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1934256073' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-24T08:22:13.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:12 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-24T08:22:13.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:12 smithi028 ceph-mon[27786]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-24T08:22:13.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:12 smithi028 ceph-mon[27786]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:22:13.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:12 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-24T08:22:13.040 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:12 smithi028 ceph-mon[27786]: osdmap e103: 12 total, 12 up, 12 in 2024-04-24T08:22:13.041 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:12 smithi028 ceph-mon[27786]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-24T08:22:13.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:12 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1934256073' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-24T08:22:13.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:12 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-24T08:22:13.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:12 smithi117 ceph-mon[32438]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-24T08:22:13.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:12 smithi117 ceph-mon[32438]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:22:13.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:12 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-24T08:22:13.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:12 smithi117 ceph-mon[32438]: osdmap e103: 12 total, 12 up, 12 in 2024-04-24T08:22:13.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:12 smithi117 ceph-mon[32438]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-24T08:22:13.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:12 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1934256073' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-24T08:22:13.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:12 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-04-24T08:22:13.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:12 smithi194 ceph-mon[32836]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-04-24T08:22:13.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:12 smithi194 ceph-mon[32836]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-04-24T08:22:13.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:12 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-04-24T08:22:13.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:12 smithi194 ceph-mon[32836]: osdmap e103: 12 total, 12 up, 12 in 2024-04-24T08:22:13.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:12 smithi194 ceph-mon[32836]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-04-24T08:22:13.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.283+0000 7fc4bb200640 1 -- 172.21.15.28:0/3818301057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b410e710 msgr2=0x7fc4b4110b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:13.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.283+0000 7fc4bb200640 1 --2- 172.21.15.28:0/3818301057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b410e710 0x7fc4b4110b00 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fc4a4009860 tx=0x7fc4a402a340 comp rx=0 tx=0).stop 2024-04-24T08:22:13.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.285+0000 7fc4bb200640 1 -- 172.21.15.28:0/3818301057 shutdown_connections 2024-04-24T08:22:13.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.285+0000 7fc4bb200640 1 --2- 172.21.15.28:0/3818301057 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b410e710 0x7fc4b4110b00 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:13.287 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.286+0000 7fc4bb200640 1 --2- 172.21.15.28:0/3818301057 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4b4101510 0x7fc4b410e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:13.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.286+0000 7fc4bb200640 1 --2- 172.21.15.28:0/3818301057 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4b4100b60 0x7fc4b4100f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:13.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.286+0000 7fc4bb200640 1 -- 172.21.15.28:0/3818301057 >> 172.21.15.28:0/3818301057 conn(0x7fc4b4077ae0 msgr2=0x7fc4b40fee70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:13.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.286+0000 7fc4bb200640 1 -- 172.21.15.28:0/3818301057 shutdown_connections 2024-04-24T08:22:13.288 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.286+0000 7fc4bb200640 1 -- 172.21.15.28:0/3818301057 wait complete. 2024-04-24T08:22:13.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.288+0000 7fc4bb200640 1 Processor -- start 2024-04-24T08:22:13.289 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.289+0000 7fc4bb200640 1 -- start start 2024-04-24T08:22:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.290+0000 7fc4bb200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4b4100b60 0x7fc4b410c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.290+0000 7fc4bb200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b4101510 0x7fc4b410cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.290+0000 7fc4bb200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4b410e710 0x7fc4b41aab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:13.291 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.290+0000 7fc4bb200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b4103b30 con 0x7fc4b4101510 2024-04-24T08:22:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.290+0000 7fc4bb200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b41039b0 con 0x7fc4b410e710 2024-04-24T08:22:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.290+0000 7fc4bb200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7fc4b4103cb0 con 0x7fc4b4100b60 2024-04-24T08:22:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.290+0000 7fc4ba800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4b410e710 0x7fc4b41aab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.290+0000 7fc4ba800640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4b410e710 0x7fc4b41aab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49542/0 (socket says 172.21.15.28:49542) 2024-04-24T08:22:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.290+0000 7fc4ba800640 1 -- 172.21.15.28:0/215741347 learned_addr learned my addr 172.21.15.28:0/215741347 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:22:13.292 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.291+0000 7fc4b9e00640 1 --2- 172.21.15.28:0/215741347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4b4100b60 0x7fc4b410c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.291+0000 7fc4b9400640 1 --2- 172.21.15.28:0/215741347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b4101510 0x7fc4b410cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.291+0000 7fc4ba800640 1 -- 172.21.15.28:0/215741347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4b4100b60 msgr2=0x7fc4b410c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.291+0000 7fc4ba800640 1 --2- 172.21.15.28:0/215741347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4b4100b60 0x7fc4b410c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.291+0000 7fc4ba800640 1 -- 172.21.15.28:0/215741347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b4101510 msgr2=0x7fc4b410cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.291+0000 7fc4ba800640 1 --2- 172.21.15.28:0/215741347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b4101510 0x7fc4b410cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:13.293 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.291+0000 7fc4ba800640 1 -- 172.21.15.28:0/215741347 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4b41ab070 con 0x7fc4b410e710 2024-04-24T08:22:13.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.291+0000 7fc4b9400640 1 --2- 172.21.15.28:0/215741347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b4101510 0x7fc4b410cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-04-24T08:22:13.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.292+0000 7fc4ba800640 1 --2- 172.21.15.28:0/215741347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4b410e710 0x7fc4b41aab30 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc4a402a930 tx=0x7fc4a4033d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:13.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.292+0000 7fc4aaa00640 1 -- 172.21.15.28:0/215741347 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a4037040 con 0x7fc4b410e710 2024-04-24T08:22:13.294 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.293+0000 7fc4aaa00640 1 -- 172.21.15.28:0/215741347 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4a4030550 con 0x7fc4b410e710 2024-04-24T08:22:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.293+0000 7fc4aaa00640 1 -- 172.21.15.28:0/215741347 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4a40054e0 con 0x7fc4b410e710 2024-04-24T08:22:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.293+0000 7fc4bb200640 1 -- 172.21.15.28:0/215741347 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4b41ab240 con 0x7fc4b410e710 2024-04-24T08:22:13.295 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.293+0000 7fc4bb200640 1 -- 172.21.15.28:0/215741347 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4b41ab6a0 con 0x7fc4b410e710 2024-04-24T08:22:13.297 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.294+0000 7fc4bb200640 1 -- 172.21.15.28:0/215741347 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc4b4068d40 con 0x7fc4b410e710 2024-04-24T08:22:13.302 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.302+0000 7fc4aaa00640 1 -- 172.21.15.28:0/215741347 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7fc4a4042020 con 0x7fc4b410e710 2024-04-24T08:22:13.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.304+0000 7fc4aaa00640 1 --2- 172.21.15.28:0/215741347 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc4a00a3da0 0x7fc4a00a6260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:13.305 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.304+0000 7fc4aaa00640 1 -- 172.21.15.28:0/215741347 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(103..103 src has 1..103) ==== 8858+0+0 (secure 0 0 0) 0x7fc4a40f01a0 con 0x7fc4b410e710 2024-04-24T08:22:13.306 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.305+0000 7fc4b9e00640 1 --2- 172.21.15.28:0/215741347 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc4a00a3da0 0x7fc4a00a6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:13.307 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.307+0000 7fc4b9e00640 1 --2- 172.21.15.28:0/215741347 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc4a00a3da0 0x7fc4a00a6260 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0005fd0 tx=0x7fc4b0009260 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:13.315 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.313+0000 7fc4aaa00640 1 -- 172.21.15.28:0/215741347 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7fc4a40bc800 con 0x7fc4b410e710 2024-04-24T08:22:13.712 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.712+0000 7fc4bb200640 1 -- 172.21.15.28:0/215741347 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7fc4b41ab9c0 con 0x7fc4b410e710 2024-04-24T08:22:13.973 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.972+0000 7fc4aaa00640 1 -- 172.21.15.28:0/215741347 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v104) ==== 181+0+0 (secure 0 0 0) 0x7fc4a40bc100 con 0x7fc4b410e710 2024-04-24T08:22:13.974 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_metadata' removed 2024-04-24T08:22:13.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.980+0000 7fc4bb200640 1 -- 172.21.15.28:0/215741347 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc4a00a3da0 msgr2=0x7fc4a00a6260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:13.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.980+0000 7fc4bb200640 1 --2- 172.21.15.28:0/215741347 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc4a00a3da0 0x7fc4a00a6260 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc4b0005fd0 tx=0x7fc4b0009260 comp rx=0 tx=0).stop 2024-04-24T08:22:13.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.980+0000 7fc4bb200640 1 -- 172.21.15.28:0/215741347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4b410e710 msgr2=0x7fc4b41aab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:13.980 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.980+0000 7fc4bb200640 1 --2- 172.21.15.28:0/215741347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4b410e710 0x7fc4b41aab30 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fc4a402a930 tx=0x7fc4a4033d10 comp rx=0 tx=0).stop 2024-04-24T08:22:13.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.980+0000 7fc4bb200640 1 -- 172.21.15.28:0/215741347 shutdown_connections 2024-04-24T08:22:13.981 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.980+0000 7fc4bb200640 1 --2- 172.21.15.28:0/215741347 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7fc4a00a3da0 0x7fc4a00a6260 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:13.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.980+0000 7fc4bb200640 1 --2- 172.21.15.28:0/215741347 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7fc4b4101510 0x7fc4b410cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:13.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.980+0000 7fc4bb200640 1 --2- 172.21.15.28:0/215741347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4b410e710 0x7fc4b41aab30 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:13.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.980+0000 7fc4bb200640 1 --2- 172.21.15.28:0/215741347 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7fc4b4100b60 0x7fc4b410c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:13.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.980+0000 7fc4bb200640 1 -- 172.21.15.28:0/215741347 >> 172.21.15.28:0/215741347 conn(0x7fc4b4077ae0 msgr2=0x7fc4b4110920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:13.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.981+0000 7fc4bb200640 1 -- 172.21.15.28:0/215741347 shutdown_connections 2024-04-24T08:22:13.982 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:13.981+0000 7fc4bb200640 1 -- 172.21.15.28:0/215741347 wait complete. 2024-04-24T08:22:14.009 DEBUG:teuthology.orchestra.run.smithi028:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-04-24T08:22:14.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: pgmap v371: 97 pgs: 22 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 61 active+clean; 14 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 34 KiB/s wr, 9 op/s 2024-04-24T08:22:14.113 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/1857340029' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:22:14.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:22:14.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-24T08:22:14.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-24T08:22:14.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: Cluster is now healthy 2024-04-24T08:22:14.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-24T08:22:14.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.194:6834/2230469040,v1:172.21.15.194:6837/2230469040] up:boot 2024-04-24T08:22:14.114 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.194:6836/3695567519,v1:172.21.15.194:6839/3695567519] up:boot 2024-04-24T08:22:14.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.194:6835/3450216331,v1:172.21.15.194:6838/3450216331] up:boot 2024-04-24T08:22:14.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.117:6836/935711163,v1:172.21.15.117:6837/935711163] up:boot 2024-04-24T08:22:14.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.117:6838/1253999030,v1:172.21.15.117:6839/1253999030] up:boot 2024-04-24T08:22:14.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: mds.? [v2:172.21.15.28:6838/3965688391,v1:172.21.15.28:6839/3965688391] up:boot 2024-04-24T08:22:14.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: fsmap 12 up:standby 2024-04-24T08:22:14.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:22:14.115 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:22:14.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T08:22:14.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:22:14.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:22:14.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:22:14.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/215741347' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:22:14.116 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:13 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:22:14.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: pgmap v371: 97 pgs: 22 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 61 active+clean; 14 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 34 KiB/s wr, 9 op/s 2024-04-24T08:22:14.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/1857340029' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:22:14.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:22:14.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-24T08:22:14.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-24T08:22:14.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: Cluster is now healthy 2024-04-24T08:22:14.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-24T08:22:14.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.194:6834/2230469040,v1:172.21.15.194:6837/2230469040] up:boot 2024-04-24T08:22:14.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.194:6836/3695567519,v1:172.21.15.194:6839/3695567519] up:boot 2024-04-24T08:22:14.129 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.194:6835/3450216331,v1:172.21.15.194:6838/3450216331] up:boot 2024-04-24T08:22:14.130 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.117:6836/935711163,v1:172.21.15.117:6837/935711163] up:boot 2024-04-24T08:22:14.130 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.117:6838/1253999030,v1:172.21.15.117:6839/1253999030] up:boot 2024-04-24T08:22:14.130 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: mds.? [v2:172.21.15.28:6838/3965688391,v1:172.21.15.28:6839/3965688391] up:boot 2024-04-24T08:22:14.130 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: fsmap 12 up:standby 2024-04-24T08:22:14.130 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:22:14.130 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:22:14.130 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T08:22:14.131 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:22:14.131 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:22:14.131 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:22:14.131 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/215741347' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:22:14.131 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:13 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:22:14.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: pgmap v371: 97 pgs: 22 active+clean+snaptrim_wait, 14 active+clean+snaptrim, 61 active+clean; 14 GiB data, 42 GiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 34 KiB/s wr, 9 op/s 2024-04-24T08:22:14.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/1857340029' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:22:14.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-04-24T08:22:14.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-04-24T08:22:14.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-04-24T08:22:14.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: Cluster is now healthy 2024-04-24T08:22:14.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-04-24T08:22:14.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.194:6834/2230469040,v1:172.21.15.194:6837/2230469040] up:boot 2024-04-24T08:22:14.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.194:6836/3695567519,v1:172.21.15.194:6839/3695567519] up:boot 2024-04-24T08:22:14.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.194:6835/3450216331,v1:172.21.15.194:6838/3450216331] up:boot 2024-04-24T08:22:14.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.117:6836/935711163,v1:172.21.15.117:6837/935711163] up:boot 2024-04-24T08:22:14.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.117:6838/1253999030,v1:172.21.15.117:6839/1253999030] up:boot 2024-04-24T08:22:14.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: mds.? [v2:172.21.15.28:6838/3965688391,v1:172.21.15.28:6839/3965688391] up:boot 2024-04-24T08:22:14.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: fsmap 12 up:standby 2024-04-24T08:22:14.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-04-24T08:22:14.192 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-04-24T08:22:14.193 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-04-24T08:22:14.193 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-04-24T08:22:14.193 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-04-24T08:22:14.193 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: from='mgr.25225 172.21.15.28:0/3650551322' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-04-24T08:22:14.193 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/215741347' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:22:14.193 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:13 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:22:14.325 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.325+0000 7ff3a9200640 1 -- 172.21.15.28:0/2770928375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3a4104a30 msgr2=0x7ff3a4104eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:14.326 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.325+0000 7ff3a9200640 1 --2- 172.21.15.28:0/2770928375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3a4104a30 0x7ff3a4104eb0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7ff39c009be0 tx=0x7ff39c02a6f0 comp rx=0 tx=0).stop 2024-04-24T08:22:14.327 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.327+0000 7ff3a9200640 1 -- 172.21.15.28:0/2770928375 shutdown_connections 2024-04-24T08:22:14.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.327+0000 7ff3a9200640 1 --2- 172.21.15.28:0/2770928375 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3a406ab40 0x7ff3a406afc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:14.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.327+0000 7ff3a9200640 1 --2- 172.21.15.28:0/2770928375 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3a4104a30 0x7ff3a4104eb0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:14.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.327+0000 7ff3a9200640 1 --2- 172.21.15.28:0/2770928375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3a4106fd0 0x7ff3a406a600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:14.328 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.327+0000 7ff3a9200640 1 -- 172.21.15.28:0/2770928375 >> 172.21.15.28:0/2770928375 conn(0x7ff3a4077ee0 msgr2=0x7ff3a40782f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:14.329 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.329+0000 7ff3a9200640 1 -- 172.21.15.28:0/2770928375 shutdown_connections 2024-04-24T08:22:14.330 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.329+0000 7ff3a9200640 1 -- 172.21.15.28:0/2770928375 wait complete. 2024-04-24T08:22:14.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.330+0000 7ff3a9200640 1 Processor -- start 2024-04-24T08:22:14.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.331+0000 7ff3a9200640 1 -- start start 2024-04-24T08:22:14.331 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.331+0000 7ff3a9200640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3a406ab40 0x7ff3a419ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:14.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.331+0000 7ff3a9200640 1 --2- >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3a4104a30 0x7ff3a419f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:14.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.331+0000 7ff3a9200640 1 --2- >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3a4106fd0 0x7ff3a41a3700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:14.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.331+0000 7ff3a9200640 1 -- --> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a4118920 con 0x7ff3a4106fd0 2024-04-24T08:22:14.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.331+0000 7ff3a9200640 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a41187a0 con 0x7ff3a406ab40 2024-04-24T08:22:14.332 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.331+0000 7ff3a9200640 1 -- --> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] -- mon_getmap magic: 0 -- 0x7ff3a4118aa0 con 0x7ff3a4104a30 2024-04-24T08:22:14.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.332+0000 7ff3a1e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3a406ab40 0x7ff3a419ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:14.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.332+0000 7ff3a1e00640 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3a406ab40 0x7ff3a419ee40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.28:49554/0 (socket says 172.21.15.28:49554) 2024-04-24T08:22:14.333 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.332+0000 7ff3a1e00640 1 -- 172.21.15.28:0/3229753110 learned_addr learned my addr 172.21.15.28:0/3229753110 (peer_addr_for_me v2:172.21.15.28:0/0) 2024-04-24T08:22:14.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.332+0000 7ff3a1400640 1 --2- 172.21.15.28:0/3229753110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3a4104a30 0x7ff3a419f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:14.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.332+0000 7ff3a2800640 1 --2- 172.21.15.28:0/3229753110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3a4106fd0 0x7ff3a41a3700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:14.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.332+0000 7ff3a1e00640 1 -- 172.21.15.28:0/3229753110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3a4104a30 msgr2=0x7ff3a419f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:14.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.332+0000 7ff3a1e00640 1 --2- 172.21.15.28:0/3229753110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3a4104a30 0x7ff3a419f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:14.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.332+0000 7ff3a1e00640 1 -- 172.21.15.28:0/3229753110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3a4106fd0 msgr2=0x7ff3a41a3700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:14.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.332+0000 7ff3a1e00640 1 --2- 172.21.15.28:0/3229753110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3a4106fd0 0x7ff3a41a3700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:14.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.332+0000 7ff3a1e00640 1 -- 172.21.15.28:0/3229753110 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff3a41a3e30 con 0x7ff3a406ab40 2024-04-24T08:22:14.334 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.333+0000 7ff3a1e00640 1 --2- 172.21.15.28:0/3229753110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3a406ab40 0x7ff3a419ee40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff39400d950 tx=0x7ff39400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:14.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.336+0000 7ff39aa00640 1 -- 172.21.15.28:0/3229753110 <== mon.2 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff394017070 con 0x7ff3a406ab40 2024-04-24T08:22:14.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.336+0000 7ff39aa00640 1 -- 172.21.15.28:0/3229753110 <== mon.2 v2:172.21.15.117:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff394009c30 con 0x7ff3a406ab40 2024-04-24T08:22:14.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.336+0000 7ff39aa00640 1 -- 172.21.15.28:0/3229753110 <== mon.2 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff39400f690 con 0x7ff3a406ab40 2024-04-24T08:22:14.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.336+0000 7ff3a9200640 1 -- 172.21.15.28:0/3229753110 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff3a41a4120 con 0x7ff3a406ab40 2024-04-24T08:22:14.337 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.336+0000 7ff3a9200640 1 -- 172.21.15.28:0/3229753110 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff3a41ab9b0 con 0x7ff3a406ab40 2024-04-24T08:22:14.340 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.339+0000 7ff3a9200640 1 -- 172.21.15.28:0/3229753110 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff3a4101510 con 0x7ff3a406ab40 2024-04-24T08:22:14.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.341+0000 7ff39aa00640 1 -- 172.21.15.28:0/3229753110 <== mon.2 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 27) ==== 138777+0+0 (secure 0 0 0) 0x7ff3940109d0 con 0x7ff3a406ab40 2024-04-24T08:22:14.345 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.343+0000 7ff39aa00640 1 --2- 172.21.15.28:0/3229753110 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3880a3e70 0x7ff3880a6330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-04-24T08:22:14.346 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.344+0000 7ff39aa00640 1 -- 172.21.15.28:0/3229753110 <== mon.2 v2:172.21.15.117:3300/0 5 ==== osd_map(104..104 src has 1..104) ==== 8333+0+0 (secure 0 0 0) 0x7ff3940d01b0 con 0x7ff3a406ab40 2024-04-24T08:22:14.347 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.346+0000 7ff3a1400640 1 --2- 172.21.15.28:0/3229753110 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3880a3e70 0x7ff3880a6330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-04-24T08:22:14.348 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.347+0000 7ff3a1400640 1 --2- 172.21.15.28:0/3229753110 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3880a3e70 0x7ff3880a6330 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4078700 tx=0x7ff390005f90 comp rx=0 tx=0).ready entity=mgr.25225 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-04-24T08:22:14.354 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.352+0000 7ff39aa00640 1 -- 172.21.15.28:0/3229753110 <== mon.2 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+188370 (secure 0 0 0) 0x7ff394009de0 con 0x7ff3a406ab40 2024-04-24T08:22:14.743 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.742+0000 7ff3a9200640 1 -- 172.21.15.28:0/3229753110 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7ff3a41abb80 con 0x7ff3a406ab40 2024-04-24T08:22:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:14.992+0000 7ff39aa00640 1 -- 172.21.15.28:0/3229753110 <== mon.2 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v105) ==== 169+0+0 (secure 0 0 0) 0x7ff39409c910 con 0x7ff3a406ab40 2024-04-24T08:22:14.994 INFO:teuthology.orchestra.run.smithi028.stderr:pool 'cephfs_data' removed 2024-04-24T08:22:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:15.000+0000 7ff3a9200640 1 -- 172.21.15.28:0/3229753110 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3880a3e70 msgr2=0x7ff3880a6330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:15.000+0000 7ff3a9200640 1 --2- 172.21.15.28:0/3229753110 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3880a3e70 0x7ff3880a6330 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4078700 tx=0x7ff390005f90 comp rx=0 tx=0).stop 2024-04-24T08:22:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:15.000+0000 7ff3a9200640 1 -- 172.21.15.28:0/3229753110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3a406ab40 msgr2=0x7ff3a419ee40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-04-24T08:22:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:15.000+0000 7ff3a9200640 1 --2- 172.21.15.28:0/3229753110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3a406ab40 0x7ff3a419ee40 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff39400d950 tx=0x7ff39400de10 comp rx=0 tx=0).stop 2024-04-24T08:22:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:15.000+0000 7ff3a9200640 1 -- 172.21.15.28:0/3229753110 shutdown_connections 2024-04-24T08:22:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:15.000+0000 7ff3a9200640 1 --2- 172.21.15.28:0/3229753110 >> [v2:172.21.15.28:6800/353036813,v1:172.21.15.28:6801/353036813] conn(0x7ff3880a3e70 0x7ff3880a6330 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:15.001 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:15.000+0000 7ff3a9200640 1 --2- 172.21.15.28:0/3229753110 >> [v2:172.21.15.28:3300/0,v1:172.21.15.28:6789/0] conn(0x7ff3a4106fd0 0x7ff3a41a3700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:15.000+0000 7ff3a9200640 1 --2- 172.21.15.28:0/3229753110 >> [v2:172.21.15.194:3300/0,v1:172.21.15.194:6789/0] conn(0x7ff3a4104a30 0x7ff3a419f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-04-24T08:22:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:15.000+0000 7ff3a9200640 1 --2- 172.21.15.28:0/3229753110 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3a406ab40 0x7ff3a419ee40 secure :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7ff39400d950 tx=0x7ff39400de10 comp rx=0 tx=0).stop 2024-04-24T08:22:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:15.000+0000 7ff3a9200640 1 -- 172.21.15.28:0/3229753110 >> 172.21.15.28:0/3229753110 conn(0x7ff3a4077ee0 msgr2=0x7ff3a40fe5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-04-24T08:22:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:15.000+0000 7ff3a9200640 1 -- 172.21.15.28:0/3229753110 shutdown_connections 2024-04-24T08:22:15.002 INFO:teuthology.orchestra.run.smithi028.stderr:2024-04-24T08:22:15.000+0000 7ff3a9200640 1 -- 172.21.15.28:0/3229753110 wait complete. 2024-04-24T08:22:15.029 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T08:22:15.146 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T08:22:15.213 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T08:22:15.248 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:14 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-24T08:22:15.248 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:14 smithi194 ceph-mon[32836]: osdmap e104: 12 total, 12 up, 12 in 2024-04-24T08:22:15.248 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:14 smithi194 ceph-mon[32836]: pgmap v373: 65 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 3 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:22:15.248 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:14 smithi194 ceph-mon[32836]: from='client.? 172.21.15.28:0/3229753110' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:22:15.248 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:14 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:22:15.254 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-04-24T08:22:15.254 DEBUG:teuthology.orchestra.run.smithi028:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-24T08:22:15.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:14 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-24T08:22:15.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:14 smithi028 ceph-mon[27786]: osdmap e104: 12 total, 12 up, 12 in 2024-04-24T08:22:15.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:14 smithi028 ceph-mon[27786]: pgmap v373: 65 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 3 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:22:15.290 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:14 smithi028 ceph-mon[27786]: from='client.? 172.21.15.28:0/3229753110' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:22:15.291 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:14 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:22:15.331 DEBUG:teuthology.orchestra.run.smithi117:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-24T08:22:15.359 DEBUG:teuthology.orchestra.run.smithi194:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-04-24T08:22:15.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:14 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-04-24T08:22:15.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:14 smithi117 ceph-mon[32438]: osdmap e104: 12 total, 12 up, 12 in 2024-04-24T08:22:15.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:14 smithi117 ceph-mon[32438]: pgmap v373: 65 pgs: 40 active+clean+snaptrim_wait, 22 active+clean+snaptrim, 3 active+clean; 14 GiB data, 41 GiB used, 1.0 TiB / 1.0 TiB avail 2024-04-24T08:22:15.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:14 smithi117 ceph-mon[32438]: from='client.? 172.21.15.28:0/3229753110' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:22:15.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:14 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-04-24T08:22:15.381 INFO:tasks.cephadm:Stopping all daemons... 2024-04-24T08:22:15.382 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-04-24T08:22:15.382 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mon.a 2024-04-24T08:22:15.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:15 smithi028 systemd[1]: Stopping Ceph mon.a for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:22:16.241 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:15 smithi028 ceph-mon[27786]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-24T08:22:16.241 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:15 smithi028 ceph-mon[27786]: osdmap e105: 12 total, 12 up, 12 in 2024-04-24T08:22:16.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:15 smithi117 ceph-mon[32438]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-24T08:22:16.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:15 smithi117 ceph-mon[32438]: osdmap e105: 12 total, 12 up, 12 in 2024-04-24T08:22:16.440 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:15 smithi194 ceph-mon[32836]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-04-24T08:22:16.441 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:15 smithi194 ceph-mon[32836]: osdmap e105: 12 total, 12 up, 12 in 2024-04-24T08:22:16.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:16 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a[27782]: 2024-04-24T08:22:16.238+0000 7f5b46c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:22:16.522 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:16 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a[27782]: 2024-04-24T08:22:16.238+0000 7f5b46c00700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-04-24T08:22:16.523 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:16 smithi028 podman[93534]: 2024-04-24 08:22:16.310353945 +0000 UTC m=+0.305094328 container died 29ad43385eba63f095616e23346567f09deee4aba577f58b5fe3980f1d535002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T08:22:16.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:16 smithi028 podman[93534]: 2024-04-24 08:22:16.743989601 +0000 UTC m=+0.738729968 container cleanup 29ad43385eba63f095616e23346567f09deee4aba577f58b5fe3980f1d535002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True) 2024-04-24T08:22:16.789 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:16 smithi028 bash[93534]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a 2024-04-24T08:22:17.289 INFO:journalctl@ceph.mon.a.smithi028.stdout:Apr 24 08:22:16 smithi028 podman[93546]: 2024-04-24 08:22:16.987140409 +0000 UTC m=+0.650633980 container remove 29ad43385eba63f095616e23346567f09deee4aba577f58b5fe3980f1d535002 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-24T08:22:17.533 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mon.a.service' 2024-04-24T08:22:17.718 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:22:17.718 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-04-24T08:22:17.719 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-04-24T08:22:17.719 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mon.b 2024-04-24T08:22:18.128 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:17 smithi117 systemd[1]: Stopping Ceph mon.b for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:22:20.258 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:19 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-b[32434]: 2024-04-24T08:22:19.992+0000 7f527c68b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:22:20.258 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:19 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-b[32434]: 2024-04-24T08:22:19.992+0000 7f527c68b700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-04-24T08:22:20.628 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:20 smithi117 podman[68264]: 2024-04-24 08:22:20.257811227 +0000 UTC m=+1.402548325 container died abc0e1bdf477d099e6c87e2d251156b31f37696cb0f0292bcab10324a32a57db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-b, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:22:26.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:26 smithi117 podman[68264]: 2024-04-24 08:22:26.125230628 +0000 UTC m=+7.269967743 container cleanup abc0e1bdf477d099e6c87e2d251156b31f37696cb0f0292bcab10324a32a57db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-b, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD) 2024-04-24T08:22:26.378 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:26 smithi117 bash[68264]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-b 2024-04-24T08:22:29.511 INFO:journalctl@ceph.mon.b.smithi117.stdout:Apr 24 08:22:29 smithi117 podman[68274]: 2024-04-24 08:22:29.260356977 +0000 UTC m=+9.000460656 container remove abc0e1bdf477d099e6c87e2d251156b31f37696cb0f0292bcab10324a32a57db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-b, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:22:29.516 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mon.b.service' 2024-04-24T08:22:29.613 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:22:29.614 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-04-24T08:22:29.614 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-04-24T08:22:29.614 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mon.c 2024-04-24T08:22:29.941 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:29 smithi194 systemd[1]: Stopping Ceph mon.c for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:22:32.644 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:32 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-c[32832]: 2024-04-24T08:22:32.375+0000 7fdd0f312700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:22:32.644 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:32 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-c[32832]: 2024-04-24T08:22:32.375+0000 7fdd0f312700 -1 mon.c@1(probing) e3 *** Got Signal Terminated *** 2024-04-24T08:22:32.644 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:32 smithi194 podman[69777]: 2024-04-24 08:22:32.558561526 +0000 UTC m=+1.447649090 container died 082e362d480d34294bfa9c791119d288ed5cb7179a08eff1434cd931be2a4226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-c, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux ) 2024-04-24T08:22:38.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:37 smithi194 podman[69777]: 2024-04-24 08:22:37.878317257 +0000 UTC m=+6.767404812 container cleanup 082e362d480d34294bfa9c791119d288ed5cb7179a08eff1434cd931be2a4226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-c, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-04-24T08:22:38.191 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:37 smithi194 bash[69777]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-c 2024-04-24T08:22:41.126 INFO:journalctl@ceph.mon.c.smithi194.stdout:Apr 24 08:22:40 smithi194 podman[69790]: 2024-04-24 08:22:40.835605767 +0000 UTC m=+8.277337251 container remove 082e362d480d34294bfa9c791119d288ed5cb7179a08eff1434cd931be2a4226 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mon-c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:22:41.143 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mon.c.service' 2024-04-24T08:22:41.228 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:22:41.229 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-04-24T08:22:41.229 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-04-24T08:22:41.229 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mgr.x 2024-04-24T08:22:45.040 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:22:44 smithi028 systemd[1]: Stopping Ceph mgr.x for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:22:55.790 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:22:55 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:22:55.387+0000 7f8c2f200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:22:55.791 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:22:55 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x[28010]: 2024-04-24T08:22:55.387+0000 7f8c2f200700 -1 mgr handle_mgr_signal *** Got signal Terminated *** 2024-04-24T08:22:55.791 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:22:55 smithi028 podman[93650]: 2024-04-24 08:22:55.478970438 +0000 UTC m=+8.886618645 container died 8e12d69a1c64607dc54909196c173b811b4ae98663f9b173799c1e4bcb47b688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-24T08:22:56.255 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:22:55 smithi028 podman[93650]: 2024-04-24 08:22:55.976367513 +0000 UTC m=+9.384015733 container cleanup 8e12d69a1c64607dc54909196c173b811b4ae98663f9b173799c1e4bcb47b688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-04-24T08:22:56.255 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:22:55 smithi028 bash[93650]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x 2024-04-24T08:22:56.539 INFO:journalctl@ceph.mgr.x.smithi028.stdout:Apr 24 08:22:56 smithi028 podman[93664]: 2024-04-24 08:22:56.253043354 +0000 UTC m=+0.744658485 container remove 8e12d69a1c64607dc54909196c173b811b4ae98663f9b173799c1e4bcb47b688 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:1f81fb397ae98da7563d451a78c61574c8f4e6e0, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-x, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:22:56.804 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mgr.x.service' 2024-04-24T08:22:56.916 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:22:56.917 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-04-24T08:22:56.917 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-04-24T08:22:56.917 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mgr.y 2024-04-24T08:22:57.378 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:22:56 smithi117 systemd[1]: Stopping Ceph mgr.y for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:22:58.878 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:22:58 smithi117 podman[68379]: 2024-04-24 08:22:58.385238518 +0000 UTC m=+1.287806877 container died 8cbd65784aa56f5a13684632900288a67fc5abd21568987653cdc7a4324ee809 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:23:02.378 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:23:02 smithi117 podman[68379]: 2024-04-24 08:23:02.098366587 +0000 UTC m=+5.000934932 container cleanup 8cbd65784aa56f5a13684632900288a67fc5abd21568987653cdc7a4324ee809 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:23:02.379 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:23:02 smithi117 bash[68379]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y 2024-04-24T08:23:03.779 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:23:03 smithi117 podman[68393]: 2024-04-24 08:23:03.559695288 +0000 UTC m=+5.063529325 container remove 8cbd65784aa56f5a13684632900288a67fc5abd21568987653cdc7a4324ee809 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-y, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-24T08:23:03.780 INFO:journalctl@ceph.mgr.y.smithi117.stdout:Apr 24 08:23:03 smithi117 systemd[1]: ceph-473aaf58-0210-11ef-bc93-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-04-24T08:23:03.802 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mgr.y.service' 2024-04-24T08:23:03.837 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:23:03.838 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-04-24T08:23:03.838 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-04-24T08:23:03.838 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mgr.z 2024-04-24T08:23:04.182 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:23:03 smithi194 systemd[1]: Stopping Ceph mgr.z for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:23:04.440 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:23:04 smithi194 podman[69912]: 2024-04-24 08:23:04.1815712 +0000 UTC m=+0.144779849 container died 91b29e229a6f88843c38093dc6591277f5511b6c89002aea74cb7cddb332fd51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True) 2024-04-24T08:23:04.861 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:23:04 smithi194 podman[69912]: 2024-04-24 08:23:04.58471491 +0000 UTC m=+0.547923555 container cleanup 91b29e229a6f88843c38093dc6591277f5511b6c89002aea74cb7cddb332fd51 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:23:04.862 INFO:journalctl@ceph.mgr.z.smithi194.stdout:Apr 24 08:23:04 smithi194 bash[69912]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mgr-z 2024-04-24T08:23:04.986 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mgr.z.service' 2024-04-24T08:23:05.023 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:23:05.023 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-04-24T08:23:05.023 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-04-24T08:23:05.023 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.0 2024-04-24T08:23:05.539 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:23:05 smithi028 systemd[1]: Stopping Ceph osd.0 for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:23:06.040 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:23:05 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-0[39966]: 2024-04-24T08:23:05.661+0000 7f08a9400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:23:06.040 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:23:05 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-0[39966]: 2024-04-24T08:23:05.661+0000 7f08a9400700 -1 osd.0 105 *** Got signal Terminated *** 2024-04-24T08:23:06.041 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:23:05 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-0[39966]: 2024-04-24T08:23:05.661+0000 7f08a9400700 -1 osd.0 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:23:12.039 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:23:11 smithi028 podman[93771]: 2024-04-24 08:23:11.747823662 +0000 UTC m=+6.262618486 container died 53d01449da7e066897555b5c7f72a2bd5f9c29cfb3b1a12acafd3a1bcd23dffa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-24T08:23:12.484 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:23:12 smithi028 podman[93771]: 2024-04-24 08:23:12.223531495 +0000 UTC m=+6.738326356 container cleanup 53d01449da7e066897555b5c7f72a2bd5f9c29cfb3b1a12acafd3a1bcd23dffa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-04-24T08:23:12.484 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:23:12 smithi028 bash[93771]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-0 2024-04-24T08:23:12.788 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:23:12 smithi028 podman[93791]: 2024-04-24 08:23:12.483125218 +0000 UTC m=+0.718080487 container remove 53d01449da7e066897555b5c7f72a2bd5f9c29cfb3b1a12acafd3a1bcd23dffa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-24T08:23:13.540 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:23:13 smithi028 podman[93881]: 2024-04-24 08:23:13.187361861 +0000 UTC m=+0.031689526 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:23:13.540 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:23:13 smithi028 podman[93881]: 2024-04-24 08:23:13.300890693 +0000 UTC m=+0.145218382 container create 75e8bae7b6b754b227293ee38cd4c9aeef7fdb01ba91406293ca595482e1ced2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:23:14.039 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:23:13 smithi028 podman[93881]: 2024-04-24 08:23:13.604574989 +0000 UTC m=+0.448902735 container init 75e8bae7b6b754b227293ee38cd4c9aeef7fdb01ba91406293ca595482e1ced2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux ) 2024-04-24T08:23:14.040 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:23:13 smithi028 podman[93881]: 2024-04-24 08:23:13.617201369 +0000 UTC m=+0.461529031 container start 75e8bae7b6b754b227293ee38cd4c9aeef7fdb01ba91406293ca595482e1ced2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-24T08:23:14.040 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:23:13 smithi028 podman[93881]: 2024-04-24 08:23:13.811087162 +0000 UTC m=+0.655414907 container attach 75e8bae7b6b754b227293ee38cd4c9aeef7fdb01ba91406293ca595482e1ced2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-24T08:23:15.039 INFO:journalctl@ceph.osd.0.smithi028.stdout:Apr 24 08:23:14 smithi028 podman[93881]: 2024-04-24 08:23:14.676729285 +0000 UTC m=+1.521057011 container died 75e8bae7b6b754b227293ee38cd4c9aeef7fdb01ba91406293ca595482e1ced2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-0-deactivate, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-24T08:23:15.478 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.0.service' 2024-04-24T08:23:15.602 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:23:15.603 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-04-24T08:23:15.603 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-04-24T08:23:15.603 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.1 2024-04-24T08:23:15.920 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:23:15 smithi117 systemd[1]: Stopping Ceph osd.1 for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:23:16.378 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:23:15 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-1[37090]: 2024-04-24T08:23:15.918+0000 7fd7d4cee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:23:16.378 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:23:15 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-1[37090]: 2024-04-24T08:23:15.918+0000 7fd7d4cee700 -1 osd.1 105 *** Got signal Terminated *** 2024-04-24T08:23:16.378 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:23:15 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-1[37090]: 2024-04-24T08:23:15.918+0000 7fd7d4cee700 -1 osd.1 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:23:21.970 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:23:21 smithi117 podman[68516]: 2024-04-24 08:23:21.686234151 +0000 UTC m=+5.889896228 container died b2878dac59329d67c2ea9e4c6ed082f72f9599930156e144aaa9e8a0d10bdc1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-1, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD) 2024-04-24T08:23:22.363 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:23:22 smithi117 podman[68516]: 2024-04-24 08:23:22.060856641 +0000 UTC m=+6.264518715 container cleanup b2878dac59329d67c2ea9e4c6ed082f72f9599930156e144aaa9e8a0d10bdc1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-1, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-04-24T08:23:22.363 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:23:22 smithi117 bash[68516]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-1 2024-04-24T08:23:22.628 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:23:22 smithi117 podman[68534]: 2024-04-24 08:23:22.36266996 +0000 UTC m=+0.674099374 container remove b2878dac59329d67c2ea9e4c6ed082f72f9599930156e144aaa9e8a0d10bdc1b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-1, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-24T08:23:22.926 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:23:22 smithi117 podman[68625]: 2024-04-24 08:23:22.561180737 +0000 UTC m=+0.017872529 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:23:22.926 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:23:22 smithi117 podman[68625]: 2024-04-24 08:23:22.68049328 +0000 UTC m=+0.137185112 container create 0c37f122fec5cb5b69e752955a7b8abc2becfe2d71ea4de426ef7b349e6a8557 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:23:22.926 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:23:22 smithi117 podman[68625]: 2024-04-24 08:23:22.925103501 +0000 UTC m=+0.381795300 container init 0c37f122fec5cb5b69e752955a7b8abc2becfe2d71ea4de426ef7b349e6a8557 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:23:23.378 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:23:22 smithi117 podman[68625]: 2024-04-24 08:23:22.92870423 +0000 UTC m=+0.385396028 container start 0c37f122fec5cb5b69e752955a7b8abc2becfe2d71ea4de426ef7b349e6a8557 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0) 2024-04-24T08:23:23.378 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:23:23 smithi117 podman[68625]: 2024-04-24 08:23:23.067002271 +0000 UTC m=+0.523694079 container attach 0c37f122fec5cb5b69e752955a7b8abc2becfe2d71ea4de426ef7b349e6a8557 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:23:23.713 INFO:journalctl@ceph.osd.1.smithi117.stdout:Apr 24 08:23:23 smithi117 podman[68625]: 2024-04-24 08:23:23.439278806 +0000 UTC m=+0.895970602 container died 0c37f122fec5cb5b69e752955a7b8abc2becfe2d71ea4de426ef7b349e6a8557 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-1-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS) 2024-04-24T08:23:24.009 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.1.service' 2024-04-24T08:23:24.047 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:23:24.047 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-04-24T08:23:24.047 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-04-24T08:23:24.047 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.2 2024-04-24T08:23:24.348 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:23:24 smithi194 systemd[1]: Stopping Ceph osd.2 for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:23:24.690 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:23:24 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-2[38007]: 2024-04-24T08:23:24.346+0000 7f743be08700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:23:24.691 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:23:24 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-2[38007]: 2024-04-24T08:23:24.346+0000 7f743be08700 -1 osd.2 105 *** Got signal Terminated *** 2024-04-24T08:23:24.691 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:23:24 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-2[38007]: 2024-04-24T08:23:24.346+0000 7f743be08700 -1 osd.2 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:23:30.331 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:23:30 smithi194 podman[70046]: 2024-04-24 08:23:30.046309208 +0000 UTC m=+5.807436841 container died 0ee6ffc09e0e11f3effdb9890d95e7d3d8abac38e550c376ac88d266c3542043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-04-24T08:23:30.690 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:23:30 smithi194 podman[70046]: 2024-04-24 08:23:30.429729809 +0000 UTC m=+6.190857441 container cleanup 0ee6ffc09e0e11f3effdb9890d95e7d3d8abac38e550c376ac88d266c3542043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS) 2024-04-24T08:23:30.691 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:23:30 smithi194 bash[70046]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-2 2024-04-24T08:23:31.092 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:23:30 smithi194 podman[70064]: 2024-04-24 08:23:30.764679663 +0000 UTC m=+0.714089684 container remove 0ee6ffc09e0e11f3effdb9890d95e7d3d8abac38e550c376ac88d266c3542043 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-24T08:23:31.377 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:23:31 smithi194 podman[70158]: 2024-04-24 08:23:30.994346958 +0000 UTC m=+0.018628583 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:23:31.377 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:23:31 smithi194 podman[70158]: 2024-04-24 08:23:31.119300369 +0000 UTC m=+0.143581980 container create 956a80ceca4a89a5c5ac90de4d9c2c2bec51931908c1cda16b681b56c0fc25f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-24T08:23:31.690 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:23:31 smithi194 podman[70158]: 2024-04-24 08:23:31.376884162 +0000 UTC m=+0.401165775 container init 956a80ceca4a89a5c5ac90de4d9c2c2bec51931908c1cda16b681b56c0fc25f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-2-deactivate, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:23:31.691 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:23:31 smithi194 podman[70158]: 2024-04-24 08:23:31.38024578 +0000 UTC m=+0.404527393 container start 956a80ceca4a89a5c5ac90de4d9c2c2bec51931908c1cda16b681b56c0fc25f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-2-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD) 2024-04-24T08:23:31.691 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:23:31 smithi194 podman[70158]: 2024-04-24 08:23:31.520021088 +0000 UTC m=+0.544302701 container attach 956a80ceca4a89a5c5ac90de4d9c2c2bec51931908c1cda16b681b56c0fc25f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-04-24T08:23:32.190 INFO:journalctl@ceph.osd.2.smithi194.stdout:Apr 24 08:23:31 smithi194 podman[70158]: 2024-04-24 08:23:31.885627083 +0000 UTC m=+0.909908695 container died 956a80ceca4a89a5c5ac90de4d9c2c2bec51931908c1cda16b681b56c0fc25f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-2-deactivate, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-04-24T08:23:32.555 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.2.service' 2024-04-24T08:23:32.595 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:23:32.595 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-04-24T08:23:32.595 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-04-24T08:23:32.595 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.3 2024-04-24T08:23:33.039 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 24 08:23:32 smithi028 systemd[1]: Stopping Ceph osd.3 for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:23:33.540 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 24 08:23:33 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-3[46879]: 2024-04-24T08:23:33.216+0000 7f4c5da00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:23:33.540 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 24 08:23:33 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-3[46879]: 2024-04-24T08:23:33.216+0000 7f4c5da00700 -1 osd.3 105 *** Got signal Terminated *** 2024-04-24T08:23:33.540 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 24 08:23:33 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-3[46879]: 2024-04-24T08:23:33.216+0000 7f4c5da00700 -1 osd.3 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:23:39.443 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 24 08:23:39 smithi028 podman[93998]: 2024-04-24 08:23:39.122708809 +0000 UTC m=+6.043421682 container died ff511362c1af93be125d5ed83fdabb630ee9c132d5ae8b37c376fdf0a7649d1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-04-24T08:23:39.789 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 24 08:23:39 smithi028 podman[93998]: 2024-04-24 08:23:39.586633392 +0000 UTC m=+6.507346279 container cleanup ff511362c1af93be125d5ed83fdabb630ee9c132d5ae8b37c376fdf0a7649d1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T08:23:39.790 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 24 08:23:39 smithi028 bash[93998]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-3 2024-04-24T08:23:40.289 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 24 08:23:39 smithi028 podman[94019]: 2024-04-24 08:23:39.879700332 +0000 UTC m=+0.736204647 container remove ff511362c1af93be125d5ed83fdabb630ee9c132d5ae8b37c376fdf0a7649d1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.build-date=20240424, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-24T08:23:41.144 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 24 08:23:40 smithi028 podman[94102]: 2024-04-24 08:23:40.70079101 +0000 UTC m=+0.037368253 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:23:41.144 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 24 08:23:40 smithi028 podman[94102]: 2024-04-24 08:23:40.839030735 +0000 UTC m=+0.175608011 container create dd3671e27f345335244e2ae13338581807c842c37e6d691572a1d569b66ba67b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-04-24T08:23:41.144 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 24 08:23:41 smithi028 podman[94102]: 2024-04-24 08:23:41.1425155 +0000 UTC m=+0.479092783 container init dd3671e27f345335244e2ae13338581807c842c37e6d691572a1d569b66ba67b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T08:23:41.539 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 24 08:23:41 smithi028 podman[94102]: 2024-04-24 08:23:41.154683831 +0000 UTC m=+0.491261093 container start dd3671e27f345335244e2ae13338581807c842c37e6d691572a1d569b66ba67b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-24T08:23:41.539 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 24 08:23:41 smithi028 podman[94102]: 2024-04-24 08:23:41.250237108 +0000 UTC m=+0.586814376 container attach dd3671e27f345335244e2ae13338581807c842c37e6d691572a1d569b66ba67b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-3-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:23:41.970 INFO:journalctl@ceph.osd.3.smithi028.stdout:Apr 24 08:23:41 smithi028 podman[94102]: 2024-04-24 08:23:41.620210247 +0000 UTC m=+0.956787525 container died dd3671e27f345335244e2ae13338581807c842c37e6d691572a1d569b66ba67b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-3-deactivate, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T08:23:42.401 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.3.service' 2024-04-24T08:23:42.504 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:23:42.505 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-04-24T08:23:42.505 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-04-24T08:23:42.505 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.4 2024-04-24T08:23:42.815 INFO:journalctl@ceph.osd.4.smithi117.stdout:Apr 24 08:23:42 smithi117 systemd[1]: Stopping Ceph osd.4 for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:23:43.128 INFO:journalctl@ceph.osd.4.smithi117.stdout:Apr 24 08:23:42 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-4[44079]: 2024-04-24T08:23:42.813+0000 7f555262b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:23:43.128 INFO:journalctl@ceph.osd.4.smithi117.stdout:Apr 24 08:23:42 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-4[44079]: 2024-04-24T08:23:42.813+0000 7f555262b700 -1 osd.4 105 *** Got signal Terminated *** 2024-04-24T08:23:43.128 INFO:journalctl@ceph.osd.4.smithi117.stdout:Apr 24 08:23:42 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-4[44079]: 2024-04-24T08:23:42.813+0000 7f555262b700 -1 osd.4 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:23:48.732 INFO:journalctl@ceph.osd.4.smithi117.stdout:Apr 24 08:23:48 smithi117 podman[68743]: 2024-04-24 08:23:48.441760464 +0000 UTC m=+5.739573890 container died 198a698228fb3b44400872a04e6b60a2337ea32b4bf24ea4db355ea250e501d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-4, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-04-24T08:23:49.024 INFO:journalctl@ceph.osd.4.smithi117.stdout:Apr 24 08:23:48 smithi117 podman[68743]: 2024-04-24 08:23:48.82228706 +0000 UTC m=+6.120100483 container cleanup 198a698228fb3b44400872a04e6b60a2337ea32b4bf24ea4db355ea250e501d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-24T08:23:49.024 INFO:journalctl@ceph.osd.4.smithi117.stdout:Apr 24 08:23:48 smithi117 bash[68743]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-4 2024-04-24T08:23:49.378 INFO:journalctl@ceph.osd.4.smithi117.stdout:Apr 24 08:23:49 smithi117 podman[68760]: 2024-04-24 08:23:49.023770492 +0000 UTC m=+0.577958521 container remove 198a698228fb3b44400872a04e6b60a2337ea32b4bf24ea4db355ea250e501d2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-04-24T08:23:49.711 INFO:journalctl@ceph.osd.4.smithi117.stdout:Apr 24 08:23:49 smithi117 podman[68851]: 2024-04-24 08:23:49.332426509 +0000 UTC m=+0.018251877 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:23:49.711 INFO:journalctl@ceph.osd.4.smithi117.stdout:Apr 24 08:23:49 smithi117 podman[68851]: 2024-04-24 08:23:49.449787643 +0000 UTC m=+0.135613038 container create 896dded8e38e11f09a2fe73afa3001fb46f6455fffcf1967ea0821d2e50520f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-4-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-24T08:23:49.711 INFO:journalctl@ceph.osd.4.smithi117.stdout:Apr 24 08:23:49 smithi117 podman[68851]: 2024-04-24 08:23:49.710418655 +0000 UTC m=+0.396244004 container init 896dded8e38e11f09a2fe73afa3001fb46f6455fffcf1967ea0821d2e50520f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True) 2024-04-24T08:23:50.008 INFO:journalctl@ceph.osd.4.smithi117.stdout:Apr 24 08:23:49 smithi117 podman[68851]: 2024-04-24 08:23:49.713656045 +0000 UTC m=+0.399481395 container start 896dded8e38e11f09a2fe73afa3001fb46f6455fffcf1967ea0821d2e50520f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240424, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-24T08:23:50.008 INFO:journalctl@ceph.osd.4.smithi117.stdout:Apr 24 08:23:49 smithi117 podman[68851]: 2024-04-24 08:23:49.80231157 +0000 UTC m=+0.488136920 container attach 896dded8e38e11f09a2fe73afa3001fb46f6455fffcf1967ea0821d2e50520f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-4-deactivate, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-04-24T08:23:50.378 INFO:journalctl@ceph.osd.4.smithi117.stdout:Apr 24 08:23:50 smithi117 podman[68851]: 2024-04-24 08:23:50.008213318 +0000 UTC m=+0.694038674 container died 896dded8e38e11f09a2fe73afa3001fb46f6455fffcf1967ea0821d2e50520f3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:23:50.697 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.4.service' 2024-04-24T08:23:50.737 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:23:50.737 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-04-24T08:23:50.738 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-04-24T08:23:50.738 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.5 2024-04-24T08:23:51.059 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 24 08:23:50 smithi194 systemd[1]: Stopping Ceph osd.5 for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:23:51.441 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 24 08:23:51 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-5[44967]: 2024-04-24T08:23:51.057+0000 7f638f29e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:23:51.441 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 24 08:23:51 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-5[44967]: 2024-04-24T08:23:51.057+0000 7f638f29e700 -1 osd.5 105 *** Got signal Terminated *** 2024-04-24T08:23:51.441 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 24 08:23:51 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-5[44967]: 2024-04-24T08:23:51.057+0000 7f638f29e700 -1 osd.5 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:23:57.190 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 24 08:23:56 smithi194 podman[70275]: 2024-04-24 08:23:56.917446387 +0000 UTC m=+5.982522590 container died 117b04849d0d0b7c615fa630b0c97768a5fce7afc5ed7c687c8d0ce8330e7f24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2) 2024-04-24T08:23:57.526 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 24 08:23:57 smithi194 podman[70275]: 2024-04-24 08:23:57.282792155 +0000 UTC m=+6.347868353 container cleanup 117b04849d0d0b7c615fa630b0c97768a5fce7afc5ed7c687c8d0ce8330e7f24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-5, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424) 2024-04-24T08:23:57.526 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 24 08:23:57 smithi194 bash[70275]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-5 2024-04-24T08:23:57.826 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 24 08:23:57 smithi194 podman[70292]: 2024-04-24 08:23:57.525992558 +0000 UTC m=+0.613035858 container remove 117b04849d0d0b7c615fa630b0c97768a5fce7afc5ed7c687c8d0ce8330e7f24 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-04-24T08:23:58.107 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 24 08:23:57 smithi194 podman[70383]: 2024-04-24 08:23:57.728484397 +0000 UTC m=+0.018298339 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:23:58.107 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 24 08:23:57 smithi194 podman[70383]: 2024-04-24 08:23:57.843501507 +0000 UTC m=+0.133315425 container create d2a6cc36b08fd9f07a154bfa8088d3c3627889454041061e5e74652691420cdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-04-24T08:23:58.402 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 24 08:23:58 smithi194 podman[70383]: 2024-04-24 08:23:58.107011831 +0000 UTC m=+0.396825757 container init d2a6cc36b08fd9f07a154bfa8088d3c3627889454041061e5e74652691420cdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-04-24T08:23:58.402 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 24 08:23:58 smithi194 podman[70383]: 2024-04-24 08:23:58.11047679 +0000 UTC m=+0.400290714 container start d2a6cc36b08fd9f07a154bfa8088d3c3627889454041061e5e74652691420cdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:23:58.402 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 24 08:23:58 smithi194 podman[70383]: 2024-04-24 08:23:58.231851565 +0000 UTC m=+0.521665487 container attach d2a6cc36b08fd9f07a154bfa8088d3c3627889454041061e5e74652691420cdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-24T08:23:58.685 INFO:journalctl@ceph.osd.5.smithi194.stdout:Apr 24 08:23:58 smithi194 podman[70383]: 2024-04-24 08:23:58.401923854 +0000 UTC m=+0.691737789 container died d2a6cc36b08fd9f07a154bfa8088d3c3627889454041061e5e74652691420cdf (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-04-24T08:23:59.022 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.5.service' 2024-04-24T08:23:59.062 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:23:59.062 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-04-24T08:23:59.063 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-04-24T08:23:59.063 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.6 2024-04-24T08:23:59.539 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 24 08:23:59 smithi028 systemd[1]: Stopping Ceph osd.6 for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:24:00.040 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 24 08:23:59 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-6[53385]: 2024-04-24T08:23:59.673+0000 7faf7ec00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:24:00.040 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 24 08:23:59 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-6[53385]: 2024-04-24T08:23:59.673+0000 7faf7ec00700 -1 osd.6 105 *** Got signal Terminated *** 2024-04-24T08:24:00.041 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 24 08:23:59 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-6[53385]: 2024-04-24T08:23:59.673+0000 7faf7ec00700 -1 osd.6 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:24:05.789 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 24 08:24:05 smithi028 podman[94231]: 2024-04-24 08:24:05.532683722 +0000 UTC m=+6.000624689 container died c5d12798c355a015c7e548f60e95dab3e13586677a9508f04ec1ea42f21242b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-6, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-24T08:24:06.218 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 24 08:24:06 smithi028 podman[94231]: 2024-04-24 08:24:06.007809276 +0000 UTC m=+6.475750236 container cleanup c5d12798c355a015c7e548f60e95dab3e13586677a9508f04ec1ea42f21242b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-6, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-04-24T08:24:06.219 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 24 08:24:06 smithi028 bash[94231]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-6 2024-04-24T08:24:06.539 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 24 08:24:06 smithi028 podman[94249]: 2024-04-24 08:24:06.217180252 +0000 UTC m=+0.666940051 container remove c5d12798c355a015c7e548f60e95dab3e13586677a9508f04ec1ea42f21242b6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-6, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-24T08:24:07.348 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 24 08:24:07 smithi028 podman[94337]: 2024-04-24 08:24:06.948675277 +0000 UTC m=+0.034878291 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:24:07.348 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 24 08:24:07 smithi028 podman[94337]: 2024-04-24 08:24:07.076505938 +0000 UTC m=+0.162709008 container create d1a6227a1de34136d40437ddf2194a278c8bada27925c134abb5bae40cb8de4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:24:07.348 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 24 08:24:07 smithi028 podman[94337]: 2024-04-24 08:24:07.346769401 +0000 UTC m=+0.432972441 container init d1a6227a1de34136d40437ddf2194a278c8bada27925c134abb5bae40cb8de4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=) 2024-04-24T08:24:07.789 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 24 08:24:07 smithi028 podman[94337]: 2024-04-24 08:24:07.35855983 +0000 UTC m=+0.444762866 container start d1a6227a1de34136d40437ddf2194a278c8bada27925c134abb5bae40cb8de4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-6-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux ) 2024-04-24T08:24:07.789 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 24 08:24:07 smithi028 podman[94337]: 2024-04-24 08:24:07.45381342 +0000 UTC m=+0.540016460 container attach d1a6227a1de34136d40437ddf2194a278c8bada27925c134abb5bae40cb8de4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-6-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:24:08.105 INFO:journalctl@ceph.osd.6.smithi028.stdout:Apr 24 08:24:07 smithi028 podman[94337]: 2024-04-24 08:24:07.824294498 +0000 UTC m=+0.910497587 container died d1a6227a1de34136d40437ddf2194a278c8bada27925c134abb5bae40cb8de4e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-6-deactivate, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5) 2024-04-24T08:24:08.440 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.6.service' 2024-04-24T08:24:08.567 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:24:08.567 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-04-24T08:24:08.567 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-04-24T08:24:08.567 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.7 2024-04-24T08:24:08.878 INFO:journalctl@ceph.osd.7.smithi117.stdout:Apr 24 08:24:08 smithi117 systemd[1]: Stopping Ceph osd.7 for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:24:09.132 INFO:journalctl@ceph.osd.7.smithi117.stdout:Apr 24 08:24:08 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-7[51090]: 2024-04-24T08:24:08.881+0000 7f1e7cbad700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:24:09.132 INFO:journalctl@ceph.osd.7.smithi117.stdout:Apr 24 08:24:08 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-7[51090]: 2024-04-24T08:24:08.881+0000 7f1e7cbad700 -1 osd.7 105 *** Got signal Terminated *** 2024-04-24T08:24:09.132 INFO:journalctl@ceph.osd.7.smithi117.stdout:Apr 24 08:24:08 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-7[51090]: 2024-04-24T08:24:08.881+0000 7f1e7cbad700 -1 osd.7 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:24:14.541 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:14 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:14.077+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.308617+0000 front 2024-04-24T08:23:08.308244+0000 (oldest deadline 2024-04-24T08:24:13.608353+0000) 2024-04-24T08:24:14.992 INFO:journalctl@ceph.osd.7.smithi117.stdout:Apr 24 08:24:14 smithi117 podman[68956]: 2024-04-24 08:24:14.723366861 +0000 UTC m=+5.964569889 container died e42eeee523499c7f8133481a4490b599c9fe907d5e8c1496a72c953d085c3953 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-7, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-04-24T08:24:15.367 INFO:journalctl@ceph.osd.7.smithi117.stdout:Apr 24 08:24:15 smithi117 podman[68956]: 2024-04-24 08:24:15.082220415 +0000 UTC m=+6.323423460 container cleanup e42eeee523499c7f8133481a4490b599c9fe907d5e8c1496a72c953d085c3953 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-7, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-24T08:24:15.368 INFO:journalctl@ceph.osd.7.smithi117.stdout:Apr 24 08:24:15 smithi117 bash[68956]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-7 2024-04-24T08:24:15.368 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:15 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:15.110+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:15.539 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:15 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:15.074+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.308617+0000 front 2024-04-24T08:23:08.308244+0000 (oldest deadline 2024-04-24T08:24:13.608353+0000) 2024-04-24T08:24:15.628 INFO:journalctl@ceph.osd.7.smithi117.stdout:Apr 24 08:24:15 smithi117 podman[68985]: 2024-04-24 08:24:15.367069341 +0000 UTC m=+0.647715916 container remove e42eeee523499c7f8133481a4490b599c9fe907d5e8c1496a72c953d085c3953 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-7, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-04-24T08:24:15.996 INFO:journalctl@ceph.osd.7.smithi117.stdout:Apr 24 08:24:15 smithi117 podman[69079]: 2024-04-24 08:24:15.565890682 +0000 UTC m=+0.017633383 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:24:15.996 INFO:journalctl@ceph.osd.7.smithi117.stdout:Apr 24 08:24:15 smithi117 podman[69079]: 2024-04-24 08:24:15.693437234 +0000 UTC m=+0.145179923 container create ca954b77fb4dbe16f2fe0d70c471293a50348e8ed49b8e4d1301d6b399230466 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-04-24T08:24:15.996 INFO:journalctl@ceph.osd.7.smithi117.stdout:Apr 24 08:24:15 smithi117 podman[69079]: 2024-04-24 08:24:15.887591328 +0000 UTC m=+0.339334016 container init ca954b77fb4dbe16f2fe0d70c471293a50348e8ed49b8e4d1301d6b399230466 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-7-deactivate, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True) 2024-04-24T08:24:15.996 INFO:journalctl@ceph.osd.7.smithi117.stdout:Apr 24 08:24:15 smithi117 podman[69079]: 2024-04-24 08:24:15.890602124 +0000 UTC m=+0.342344813 container start ca954b77fb4dbe16f2fe0d70c471293a50348e8ed49b8e4d1301d6b399230466 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-7-deactivate, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T08:24:15.996 INFO:journalctl@ceph.osd.7.smithi117.stdout:Apr 24 08:24:15 smithi117 podman[69079]: 2024-04-24 08:24:15.995752883 +0000 UTC m=+0.447495574 container attach ca954b77fb4dbe16f2fe0d70c471293a50348e8ed49b8e4d1301d6b399230466 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-7-deactivate, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:24:16.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:16 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:16.061+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:16.378 INFO:journalctl@ceph.osd.7.smithi117.stdout:Apr 24 08:24:16 smithi117 podman[69079]: 2024-04-24 08:24:16.199192518 +0000 UTC m=+0.650935199 container died ca954b77fb4dbe16f2fe0d70c471293a50348e8ed49b8e4d1301d6b399230466 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-7-deactivate, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True) 2024-04-24T08:24:16.539 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:16 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:16.103+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.308617+0000 front 2024-04-24T08:23:08.308244+0000 (oldest deadline 2024-04-24T08:24:13.608353+0000) 2024-04-24T08:24:16.852 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.7.service' 2024-04-24T08:24:16.889 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:24:16.889 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-04-24T08:24:16.890 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-04-24T08:24:16.890 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.8 2024-04-24T08:24:17.191 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 24 08:24:16 smithi194 systemd[1]: Stopping Ceph osd.8 for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:24:17.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:17 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:17.089+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:17.449 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 24 08:24:17 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-8[51672]: 2024-04-24T08:24:17.220+0000 7fce3e54e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:24:17.449 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 24 08:24:17 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-8[51672]: 2024-04-24T08:24:17.220+0000 7fce3e54e700 -1 osd.8 105 *** Got signal Terminated *** 2024-04-24T08:24:17.449 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 24 08:24:17 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-8[51672]: 2024-04-24T08:24:17.220+0000 7fce3e54e700 -1 osd.8 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:24:17.539 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:17 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:17.058+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.308617+0000 front 2024-04-24T08:23:08.308244+0000 (oldest deadline 2024-04-24T08:24:13.608353+0000) 2024-04-24T08:24:18.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:18 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:18.018+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.308617+0000 front 2024-04-24T08:23:08.308244+0000 (oldest deadline 2024-04-24T08:24:13.608353+0000) 2024-04-24T08:24:18.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:18 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:18.065+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:18.690 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:18 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:18.223+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:19.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:19 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:19.012+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.308617+0000 front 2024-04-24T08:23:08.308244+0000 (oldest deadline 2024-04-24T08:24:13.608353+0000) 2024-04-24T08:24:19.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:19 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:19.106+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:19.690 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:19 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:19.243+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:20.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:20 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:20.022+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.308617+0000 front 2024-04-24T08:23:08.308244+0000 (oldest deadline 2024-04-24T08:24:13.608353+0000) 2024-04-24T08:24:20.628 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:20 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:20.139+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:20.690 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:20 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:20.216+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:21.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:21 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:21.008+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.308617+0000 front 2024-04-24T08:23:08.308244+0000 (oldest deadline 2024-04-24T08:24:13.608353+0000) 2024-04-24T08:24:21.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:21 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:21.101+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:21.690 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:21 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:21.205+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:22.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:22 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:22.002+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.308617+0000 front 2024-04-24T08:23:08.308244+0000 (oldest deadline 2024-04-24T08:24:13.608353+0000) 2024-04-24T08:24:22.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:22 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:22.065+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:22.690 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:22 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:22.243+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:23.096 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 24 08:24:22 smithi194 podman[70489]: 2024-04-24 08:24:22.79909223 +0000 UTC m=+5.712295025 container died 7d0aa61c5f070f75cdd7223388b7a7cd7c5b5a15220416840fd78030070032c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-8, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-04-24T08:24:23.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:22 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:22.973+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.308617+0000 front 2024-04-24T08:23:08.308244+0000 (oldest deadline 2024-04-24T08:24:13.608353+0000) 2024-04-24T08:24:23.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:23 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:23.037+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:23.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:23 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:23.201+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:23.441 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 24 08:24:23 smithi194 podman[70489]: 2024-04-24 08:24:23.187076655 +0000 UTC m=+6.100279499 container cleanup 7d0aa61c5f070f75cdd7223388b7a7cd7c5b5a15220416840fd78030070032c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-8, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T08:24:23.442 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 24 08:24:23 smithi194 bash[70489]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-8 2024-04-24T08:24:23.803 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 24 08:24:23 smithi194 podman[70520]: 2024-04-24 08:24:23.48853096 +0000 UTC m=+0.686875355 container remove 7d0aa61c5f070f75cdd7223388b7a7cd7c5b5a15220416840fd78030070032c4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD) 2024-04-24T08:24:24.125 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 24 08:24:23 smithi194 podman[70613]: 2024-04-24 08:24:23.706142986 +0000 UTC m=+0.018150764 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:24:24.125 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 24 08:24:23 smithi194 podman[70613]: 2024-04-24 08:24:23.857204069 +0000 UTC m=+0.169211819 container create 86b2313e5882a2e9089ac2edaf1b00a89e45730c089390b6c21b9b09342dfdc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-04-24T08:24:24.126 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 24 08:24:24 smithi194 podman[70613]: 2024-04-24 08:24:24.124867484 +0000 UTC m=+0.436875232 container init 86b2313e5882a2e9089ac2edaf1b00a89e45730c089390b6c21b9b09342dfdc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-24T08:24:24.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:23 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:23.981+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.308617+0000 front 2024-04-24T08:23:08.308244+0000 (oldest deadline 2024-04-24T08:24:13.608353+0000) 2024-04-24T08:24:24.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:23 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:23.997+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:24.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:23 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:23.997+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.078688+0000 front 2024-04-24T08:23:18.078716+0000 (oldest deadline 2024-04-24T08:24:23.978817+0000) 2024-04-24T08:24:24.421 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:24 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:24.166+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:24.421 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 24 08:24:24 smithi194 podman[70613]: 2024-04-24 08:24:24.128080567 +0000 UTC m=+0.440088315 container start 86b2313e5882a2e9089ac2edaf1b00a89e45730c089390b6c21b9b09342dfdc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-8-deactivate, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-04-24T08:24:24.421 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 24 08:24:24 smithi194 podman[70613]: 2024-04-24 08:24:24.208953634 +0000 UTC m=+0.520961390 container attach 86b2313e5882a2e9089ac2edaf1b00a89e45730c089390b6c21b9b09342dfdc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424) 2024-04-24T08:24:24.690 INFO:journalctl@ceph.osd.8.smithi194.stdout:Apr 24 08:24:24 smithi194 podman[70613]: 2024-04-24 08:24:24.420521592 +0000 UTC m=+0.732529348 container died 86b2313e5882a2e9089ac2edaf1b00a89e45730c089390b6c21b9b09342dfdc0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-04-24T08:24:25.009 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.8.service' 2024-04-24T08:24:25.046 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:24:25.046 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-04-24T08:24:25.046 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-04-24T08:24:25.047 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.9 2024-04-24T08:24:25.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:24 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:24.989+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.308617+0000 front 2024-04-24T08:23:08.308244+0000 (oldest deadline 2024-04-24T08:24:13.608353+0000) 2024-04-24T08:24:25.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:24 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:24.989+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.911071+0000 front 2024-04-24T08:23:18.911161+0000 (oldest deadline 2024-04-24T08:24:24.712912+0000) 2024-04-24T08:24:25.290 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:25 smithi028 systemd[1]: Stopping Ceph osd.9 for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:24:25.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:25 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:25.003+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:25.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:25 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:25.003+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.078688+0000 front 2024-04-24T08:23:18.078716+0000 (oldest deadline 2024-04-24T08:24:23.978817+0000) 2024-04-24T08:24:25.440 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:25 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:25.181+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:26.030 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:25 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:25.642+0000 7f7eb9e00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:24:26.030 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:25 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:25.642+0000 7f7eb9e00700 -1 osd.9 105 *** Got signal Terminated *** 2024-04-24T08:24:26.030 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:25 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:25.642+0000 7f7eb9e00700 -1 osd.9 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:24:26.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:26 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:26.028+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.308617+0000 front 2024-04-24T08:23:08.308244+0000 (oldest deadline 2024-04-24T08:24:13.608353+0000) 2024-04-24T08:24:26.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:26 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:26.028+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.911071+0000 front 2024-04-24T08:23:18.911161+0000 (oldest deadline 2024-04-24T08:24:24.712912+0000) 2024-04-24T08:24:26.295 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:26 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:26.044+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:26.295 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:26 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:26.044+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.078688+0000 front 2024-04-24T08:23:18.078716+0000 (oldest deadline 2024-04-24T08:24:23.978817+0000) 2024-04-24T08:24:26.690 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:26 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:26.199+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:27.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:27 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:27.026+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.308617+0000 front 2024-04-24T08:23:08.308244+0000 (oldest deadline 2024-04-24T08:24:13.608353+0000) 2024-04-24T08:24:27.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:27 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:27.026+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.911071+0000 front 2024-04-24T08:23:18.911161+0000 (oldest deadline 2024-04-24T08:24:24.712912+0000) 2024-04-24T08:24:27.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:27 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:27.057+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:27.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:27 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:27.057+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.078688+0000 front 2024-04-24T08:23:18.078716+0000 (oldest deadline 2024-04-24T08:24:23.978817+0000) 2024-04-24T08:24:27.440 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:27 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:27.170+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:27.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:27 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:27.170+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:28.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:28 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:28.013+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.308617+0000 front 2024-04-24T08:23:08.308244+0000 (oldest deadline 2024-04-24T08:24:13.608353+0000) 2024-04-24T08:24:28.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:28 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:28.013+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.911071+0000 front 2024-04-24T08:23:18.911161+0000 (oldest deadline 2024-04-24T08:24:24.712912+0000) 2024-04-24T08:24:28.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:28 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:28.012+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:28.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:28 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:28.012+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.078688+0000 front 2024-04-24T08:23:18.078716+0000 (oldest deadline 2024-04-24T08:24:23.978817+0000) 2024-04-24T08:24:28.440 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:28 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:28.123+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:28.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:28 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:28.124+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:29.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:29 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:29.005+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.308617+0000 front 2024-04-24T08:23:08.308244+0000 (oldest deadline 2024-04-24T08:24:13.608353+0000) 2024-04-24T08:24:29.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:29 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:29.005+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.911071+0000 front 2024-04-24T08:23:18.911161+0000 (oldest deadline 2024-04-24T08:24:24.712912+0000) 2024-04-24T08:24:29.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:28 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:28.971+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:29.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:28 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:28.971+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.078688+0000 front 2024-04-24T08:23:18.078716+0000 (oldest deadline 2024-04-24T08:24:23.978817+0000) 2024-04-24T08:24:29.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:29 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:29.138+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:29.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:29 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:29.138+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:30.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:29 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:29.974+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.308617+0000 front 2024-04-24T08:23:08.308244+0000 (oldest deadline 2024-04-24T08:24:13.608353+0000) 2024-04-24T08:24:30.289 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:29 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9[60382]: 2024-04-24T08:24:29.974+0000 7f7eb5600700 -1 osd.9 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.911071+0000 front 2024-04-24T08:23:18.911161+0000 (oldest deadline 2024-04-24T08:24:24.712912+0000) 2024-04-24T08:24:30.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:29 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:29.921+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:30.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:29 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:29.921+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.078688+0000 front 2024-04-24T08:23:18.078716+0000 (oldest deadline 2024-04-24T08:24:23.978817+0000) 2024-04-24T08:24:30.440 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:30 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:30.098+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:30.440 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:30 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:30.098+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:31.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:30 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:30.880+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:31.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:30 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:30.880+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.078688+0000 front 2024-04-24T08:23:18.078716+0000 (oldest deadline 2024-04-24T08:24:23.978817+0000) 2024-04-24T08:24:31.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:30 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:30.880+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:27.280592+0000 front 2024-04-24T08:23:27.280673+0000 (oldest deadline 2024-04-24T08:24:30.780665+0000) 2024-04-24T08:24:31.440 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:31 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:31.108+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:31.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:31 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:31.108+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:31.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:31 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:31.108+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:26.584288+0000 front 2024-04-24T08:23:26.584327+0000 (oldest deadline 2024-04-24T08:24:30.684457+0000) 2024-04-24T08:24:31.981 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:31 smithi028 podman[94457]: 2024-04-24 08:24:31.719980711 +0000 UTC m=+6.234152666 container died 481df7831341f1b3974381f83cdb1e50b6158df53abfb7b8370a42f7e85d185f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, ceph=True) 2024-04-24T08:24:32.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:31 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:31.850+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:32.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:31 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:31.850+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.078688+0000 front 2024-04-24T08:23:18.078716+0000 (oldest deadline 2024-04-24T08:24:23.978817+0000) 2024-04-24T08:24:32.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:31 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:31.850+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:27.280592+0000 front 2024-04-24T08:23:27.280673+0000 (oldest deadline 2024-04-24T08:24:30.780665+0000) 2024-04-24T08:24:32.250 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:32 smithi028 podman[94457]: 2024-04-24 08:24:32.059986357 +0000 UTC m=+6.574158318 container cleanup 481df7831341f1b3974381f83cdb1e50b6158df53abfb7b8370a42f7e85d185f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:24:32.250 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:32 smithi028 bash[94457]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9 2024-04-24T08:24:32.440 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:32 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:32.085+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:32.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:32 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:32.085+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:32.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:32 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:32.085+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:26.584288+0000 front 2024-04-24T08:23:26.584327+0000 (oldest deadline 2024-04-24T08:24:30.684457+0000) 2024-04-24T08:24:32.539 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:32 smithi028 podman[94477]: 2024-04-24 08:24:32.249072691 +0000 UTC m=+0.508648791 container remove 481df7831341f1b3974381f83cdb1e50b6158df53abfb7b8370a42f7e85d185f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424) 2024-04-24T08:24:33.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:32 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:32.828+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:33.129 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:32 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:32.828+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.078688+0000 front 2024-04-24T08:23:18.078716+0000 (oldest deadline 2024-04-24T08:24:23.978817+0000) 2024-04-24T08:24:33.129 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:32 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:32.828+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:27.280592+0000 front 2024-04-24T08:23:27.280673+0000 (oldest deadline 2024-04-24T08:24:30.780665+0000) 2024-04-24T08:24:33.352 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:33 smithi028 podman[94567]: 2024-04-24 08:24:32.986363386 +0000 UTC m=+0.042794266 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:24:33.352 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:33 smithi028 podman[94567]: 2024-04-24 08:24:33.086913083 +0000 UTC m=+0.143343983 container create b7b59e47995e75f074aa0daf78bcfae26d115332daf056960095a2484ca31506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-24T08:24:33.440 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:33 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:33.116+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:33.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:33 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:33.116+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:33.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:33 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:33.117+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:26.584288+0000 front 2024-04-24T08:23:26.584327+0000 (oldest deadline 2024-04-24T08:24:30.684457+0000) 2024-04-24T08:24:33.789 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:33 smithi028 podman[94567]: 2024-04-24 08:24:33.349838642 +0000 UTC m=+0.406269599 container init b7b59e47995e75f074aa0daf78bcfae26d115332daf056960095a2484ca31506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:24:33.789 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:33 smithi028 podman[94567]: 2024-04-24 08:24:33.363221672 +0000 UTC m=+0.419652542 container start b7b59e47995e75f074aa0daf78bcfae26d115332daf056960095a2484ca31506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-04-24T08:24:33.789 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:33 smithi028 podman[94567]: 2024-04-24 08:24:33.481041488 +0000 UTC m=+0.537472435 container attach b7b59e47995e75f074aa0daf78bcfae26d115332daf056960095a2484ca31506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-24T08:24:34.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:33 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:33.784+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:34.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:33 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:33.784+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.078688+0000 front 2024-04-24T08:23:18.078716+0000 (oldest deadline 2024-04-24T08:24:23.978817+0000) 2024-04-24T08:24:34.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:33 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:33.784+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:27.280592+0000 front 2024-04-24T08:23:27.280673+0000 (oldest deadline 2024-04-24T08:24:30.780665+0000) 2024-04-24T08:24:34.129 INFO:journalctl@ceph.osd.9.smithi028.stdout:Apr 24 08:24:33 smithi028 podman[94567]: 2024-04-24 08:24:33.827297258 +0000 UTC m=+0.883728178 container died b7b59e47995e75f074aa0daf78bcfae26d115332daf056960095a2484ca31506 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-9-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-24T08:24:34.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:34 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:34.113+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:34.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:34 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:34.113+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:34.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:34 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:34.113+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:26.584288+0000 front 2024-04-24T08:23:26.584327+0000 (oldest deadline 2024-04-24T08:24:30.684457+0000) 2024-04-24T08:24:34.566 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.9.service' 2024-04-24T08:24:34.685 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:24:34.686 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-04-24T08:24:34.686 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-04-24T08:24:34.686 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.10 2024-04-24T08:24:35.018 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:34 smithi117 systemd[1]: Stopping Ceph osd.10 for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:24:35.018 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:34 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:34.806+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:35.018 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:34 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:34.806+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.078688+0000 front 2024-04-24T08:23:18.078716+0000 (oldest deadline 2024-04-24T08:24:23.978817+0000) 2024-04-24T08:24:35.018 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:34 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:34.806+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:27.280592+0000 front 2024-04-24T08:23:27.280673+0000 (oldest deadline 2024-04-24T08:24:30.780665+0000) 2024-04-24T08:24:35.019 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:35 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:35.016+0000 7ff43eb23700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:24:35.019 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:35 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:35.016+0000 7ff43eb23700 -1 osd.10 105 *** Got signal Terminated *** 2024-04-24T08:24:35.019 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:35 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:35.016+0000 7ff43eb23700 -1 osd.10 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:24:35.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:35 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:35.078+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:35.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:35 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:35.078+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:35.442 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:35 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:35.078+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:26.584288+0000 front 2024-04-24T08:23:26.584327+0000 (oldest deadline 2024-04-24T08:24:30.684457+0000) 2024-04-24T08:24:36.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:35 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:35.809+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:36.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:35 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:35.809+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.078688+0000 front 2024-04-24T08:23:18.078716+0000 (oldest deadline 2024-04-24T08:24:23.978817+0000) 2024-04-24T08:24:36.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:35 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:35.809+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:27.280592+0000 front 2024-04-24T08:23:27.280673+0000 (oldest deadline 2024-04-24T08:24:30.780665+0000) 2024-04-24T08:24:36.440 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:36 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:36.054+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:36.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:36 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:36.054+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:36.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:36 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:36.054+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:26.584288+0000 front 2024-04-24T08:23:26.584327+0000 (oldest deadline 2024-04-24T08:24:30.684457+0000) 2024-04-24T08:24:37.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:36 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:36.828+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:37.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:36 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:36.828+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.078688+0000 front 2024-04-24T08:23:18.078716+0000 (oldest deadline 2024-04-24T08:24:23.978817+0000) 2024-04-24T08:24:37.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:36 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:36.828+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:27.280592+0000 front 2024-04-24T08:23:27.280673+0000 (oldest deadline 2024-04-24T08:24:30.780665+0000) 2024-04-24T08:24:37.440 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:37 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:37.059+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:37.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:37 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:37.059+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:37.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:37 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:37.059+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:26.584288+0000 front 2024-04-24T08:23:26.584327+0000 (oldest deadline 2024-04-24T08:24:30.684457+0000) 2024-04-24T08:24:38.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:37 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:37.860+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:38.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:37 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:37.860+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.078688+0000 front 2024-04-24T08:23:18.078716+0000 (oldest deadline 2024-04-24T08:24:23.978817+0000) 2024-04-24T08:24:38.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:37 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:37.860+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:27.280592+0000 front 2024-04-24T08:23:27.280673+0000 (oldest deadline 2024-04-24T08:24:30.780665+0000) 2024-04-24T08:24:38.440 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:38 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:38.085+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:38.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:38 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:38.085+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:38.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:38 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:38.085+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:26.584288+0000 front 2024-04-24T08:23:26.584327+0000 (oldest deadline 2024-04-24T08:24:30.684457+0000) 2024-04-24T08:24:39.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:38 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:38.896+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:39.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:38 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:38.896+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.078688+0000 front 2024-04-24T08:23:18.078716+0000 (oldest deadline 2024-04-24T08:24:23.978817+0000) 2024-04-24T08:24:39.378 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:38 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:38.896+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:27.280592+0000 front 2024-04-24T08:23:27.280673+0000 (oldest deadline 2024-04-24T08:24:30.780665+0000) 2024-04-24T08:24:39.440 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:39 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:39.087+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:39.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:39 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:39.087+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:39.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:39 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:39.087+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:26.584288+0000 front 2024-04-24T08:23:26.584327+0000 (oldest deadline 2024-04-24T08:24:30.684457+0000) 2024-04-24T08:24:40.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:39 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:39.853+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:08.177097+0000 front 2024-04-24T08:23:08.176968+0000 (oldest deadline 2024-04-24T08:24:14.076969+0000) 2024-04-24T08:24:40.129 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:39 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:39.853+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:18.078688+0000 front 2024-04-24T08:23:18.078716+0000 (oldest deadline 2024-04-24T08:24:23.978817+0000) 2024-04-24T08:24:40.129 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:39 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:39.853+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:27.280592+0000 front 2024-04-24T08:23:27.280673+0000 (oldest deadline 2024-04-24T08:24:30.780665+0000) 2024-04-24T08:24:40.129 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:39 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10[57670]: 2024-04-24T08:24:39.853+0000 7ff43b117700 -1 osd.10 105 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-24T08:23:38.084483+0000 front 2024-04-24T08:23:38.084358+0000 (oldest deadline 2024-04-24T08:24:39.184201+0000) 2024-04-24T08:24:40.440 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:40 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:40.086+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:40.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:40 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:40.086+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:40.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:40 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:40.086+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:26.584288+0000 front 2024-04-24T08:23:26.584327+0000 (oldest deadline 2024-04-24T08:24:30.684457+0000) 2024-04-24T08:24:40.835 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:40 smithi117 podman[69184]: 2024-04-24 08:24:40.574556026 +0000 UTC m=+5.694510421 container died 189f0ab883c56d95c23e9fa1b7a4ef333962be110614a9f23d58fa391a654bd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:24:41.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:40 smithi117 podman[69184]: 2024-04-24 08:24:40.932987831 +0000 UTC m=+6.052942213 container cleanup 189f0ab883c56d95c23e9fa1b7a4ef333962be110614a9f23d58fa391a654bd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-04-24T08:24:41.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:40 smithi117 bash[69184]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10 2024-04-24T08:24:41.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:41 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:41.092+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:41.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:41 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:41.092+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:41.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:41 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:41.092+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:26.584288+0000 front 2024-04-24T08:23:26.584327+0000 (oldest deadline 2024-04-24T08:24:30.684457+0000) 2024-04-24T08:24:41.544 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:41 smithi117 podman[69201]: 2024-04-24 08:24:41.234723241 +0000 UTC m=+0.658361011 container remove 189f0ab883c56d95c23e9fa1b7a4ef333962be110614a9f23d58fa391a654bd1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-04-24T08:24:41.821 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:41 smithi117 podman[69290]: 2024-04-24 08:24:41.543728641 +0000 UTC m=+0.108265844 container create 00b629c4ff4ec087819c05712a954899ded0b41455d0101dba30123b3d81d7fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-04-24T08:24:41.821 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:41 smithi117 podman[69290]: 2024-04-24 08:24:41.453461407 +0000 UTC m=+0.017998575 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:24:41.821 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:41 smithi117 podman[69290]: 2024-04-24 08:24:41.746291412 +0000 UTC m=+0.310828575 container init 00b629c4ff4ec087819c05712a954899ded0b41455d0101dba30123b3d81d7fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T08:24:41.821 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:41 smithi117 podman[69290]: 2024-04-24 08:24:41.749491296 +0000 UTC m=+0.314028443 container start 00b629c4ff4ec087819c05712a954899ded0b41455d0101dba30123b3d81d7fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10-deactivate, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424) 2024-04-24T08:24:41.822 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:41 smithi117 podman[69290]: 2024-04-24 08:24:41.820842374 +0000 UTC m=+0.385379521 container attach 00b629c4ff4ec087819c05712a954899ded0b41455d0101dba30123b3d81d7fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10-deactivate, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-04-24T08:24:42.128 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:42 smithi117 podman[69290]: 2024-04-24 08:24:42.039961358 +0000 UTC m=+0.604498521 container died 00b629c4ff4ec087819c05712a954899ded0b41455d0101dba30123b3d81d7fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10-deactivate, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:24:42.440 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:42 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:42.133+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:42.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:42 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:42.133+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:42.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:42 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:42.133+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:26.584288+0000 front 2024-04-24T08:23:26.584327+0000 (oldest deadline 2024-04-24T08:24:30.684457+0000) 2024-04-24T08:24:42.575 INFO:journalctl@ceph.osd.10.smithi117.stdout:Apr 24 08:24:42 smithi117 podman[69316]: 2024-04-24 08:24:42.557443826 +0000 UTC m=+0.511225164 container remove 00b629c4ff4ec087819c05712a954899ded0b41455d0101dba30123b3d81d7fb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-10-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0) 2024-04-24T08:24:42.580 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.10.service' 2024-04-24T08:24:42.618 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:24:42.618 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-04-24T08:24:42.618 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-04-24T08:24:42.619 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.11 2024-04-24T08:24:42.933 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:42 smithi194 systemd[1]: Stopping Ceph osd.11 for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:24:43.190 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:42 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:42.932+0000 7f8da1439700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:24:43.191 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:42 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:42.932+0000 7f8da1439700 -1 osd.11 105 *** Got signal Terminated *** 2024-04-24T08:24:43.191 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:42 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:42.932+0000 7f8da1439700 -1 osd.11 105 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-04-24T08:24:43.191 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:43 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:43.154+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:43.191 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:43 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:43.154+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:43.191 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:43 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:43.154+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:26.584288+0000 front 2024-04-24T08:23:26.584327+0000 (oldest deadline 2024-04-24T08:24:30.684457+0000) 2024-04-24T08:24:43.191 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:43 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:43.154+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-24T08:23:36.386765+0000 front 2024-04-24T08:23:36.386650+0000 (oldest deadline 2024-04-24T08:24:42.286418+0000) 2024-04-24T08:24:44.440 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:44 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:44.138+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:44.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:44 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:44.138+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:44.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:44 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:44.138+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:26.584288+0000 front 2024-04-24T08:23:26.584327+0000 (oldest deadline 2024-04-24T08:24:30.684457+0000) 2024-04-24T08:24:44.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:44 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:44.138+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-24T08:23:36.386765+0000 front 2024-04-24T08:23:36.386650+0000 (oldest deadline 2024-04-24T08:24:42.286418+0000) 2024-04-24T08:24:45.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:45 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:45.112+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:45.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:45 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:45.112+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:45.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:45 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:45.112+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:26.584288+0000 front 2024-04-24T08:23:26.584327+0000 (oldest deadline 2024-04-24T08:24:30.684457+0000) 2024-04-24T08:24:45.442 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:45 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:45.112+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-24T08:23:36.386765+0000 front 2024-04-24T08:23:36.386650+0000 (oldest deadline 2024-04-24T08:24:42.286418+0000) 2024-04-24T08:24:46.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:46 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:46.068+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:46.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:46 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:46.068+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:46.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:46 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:46.068+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:26.584288+0000 front 2024-04-24T08:23:26.584327+0000 (oldest deadline 2024-04-24T08:24:30.684457+0000) 2024-04-24T08:24:46.441 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:46 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:46.068+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-24T08:23:36.386765+0000 front 2024-04-24T08:23:36.386650+0000 (oldest deadline 2024-04-24T08:24:42.286418+0000) 2024-04-24T08:24:47.342 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:47 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:47.023+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6806 osd.0 since back 2024-04-24T08:23:06.880965+0000 front 2024-04-24T08:23:06.880809+0000 (oldest deadline 2024-04-24T08:24:17.481189+0000) 2024-04-24T08:24:47.342 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:47 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:47.023+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.117:6804 osd.1 since back 2024-04-24T08:23:20.684008+0000 front 2024-04-24T08:23:20.683853+0000 (oldest deadline 2024-04-24T08:24:26.583826+0000) 2024-04-24T08:24:47.343 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:47 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:47.023+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.194:6804 osd.2 since back 2024-04-24T08:23:26.584288+0000 front 2024-04-24T08:23:26.584327+0000 (oldest deadline 2024-04-24T08:24:30.684457+0000) 2024-04-24T08:24:47.343 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:47 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11[58715]: 2024-04-24T08:24:47.023+0000 7f8d99a25700 -1 osd.11 105 heartbeat_check: no reply from 172.21.15.28:6814 osd.3 since back 2024-04-24T08:23:36.386765+0000 front 2024-04-24T08:23:36.386650+0000 (oldest deadline 2024-04-24T08:24:42.286418+0000) 2024-04-24T08:24:49.042 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:48 smithi194 podman[70717]: 2024-04-24 08:24:48.743023137 +0000 UTC m=+5.933201771 container died 0807c0b5e45b2ba120ed95642b3955bdbc0b0c7d944847f56ca1d1d37cf48a12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-04-24T08:24:49.427 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:49 smithi194 podman[70717]: 2024-04-24 08:24:49.132523162 +0000 UTC m=+6.322701806 container cleanup 0807c0b5e45b2ba120ed95642b3955bdbc0b0c7d944847f56ca1d1d37cf48a12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:24:49.427 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:49 smithi194 bash[70717]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11 2024-04-24T08:24:49.691 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:49 smithi194 podman[70735]: 2024-04-24 08:24:49.425765715 +0000 UTC m=+0.680193212 container remove 0807c0b5e45b2ba120ed95642b3955bdbc0b0c7d944847f56ca1d1d37cf48a12 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:24:50.013 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:49 smithi194 podman[70826]: 2024-04-24 08:24:49.655892651 +0000 UTC m=+0.017720239 image pull b98aea5aa06f9f14aacd874e6fe7f4a9ab599f4e4a4707fadae99d37abf77a02 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644 2024-04-24T08:24:50.014 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:49 smithi194 podman[70826]: 2024-04-24 08:24:49.784712342 +0000 UTC m=+0.146539915 container create ad5f7cc2495256d0e5bf64420efec2db8685bfbc5797bc25653630c097afef3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-24T08:24:50.014 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:50 smithi194 podman[70826]: 2024-04-24 08:24:50.013031982 +0000 UTC m=+0.374859564 container init ad5f7cc2495256d0e5bf64420efec2db8685bfbc5797bc25653630c097afef3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11-deactivate, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-24T08:24:50.311 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:50 smithi194 podman[70826]: 2024-04-24 08:24:50.016500091 +0000 UTC m=+0.378327665 container start ad5f7cc2495256d0e5bf64420efec2db8685bfbc5797bc25653630c097afef3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True) 2024-04-24T08:24:50.311 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:50 smithi194 podman[70826]: 2024-04-24 08:24:50.07907742 +0000 UTC m=+0.440904994 container attach ad5f7cc2495256d0e5bf64420efec2db8685bfbc5797bc25653630c097afef3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-04-24T08:24:50.590 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:50 smithi194 podman[70826]: 2024-04-24 08:24:50.310885549 +0000 UTC m=+0.672713136 container died ad5f7cc2495256d0e5bf64420efec2db8685bfbc5797bc25653630c097afef3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-24T08:24:50.840 INFO:journalctl@ceph.osd.11.smithi194.stdout:Apr 24 08:24:50 smithi194 podman[70853]: 2024-04-24 08:24:50.823818994 +0000 UTC m=+0.507869170 container remove ad5f7cc2495256d0e5bf64420efec2db8685bfbc5797bc25653630c097afef3a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-osd-11-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-24T08:24:50.844 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@osd.11.service' 2024-04-24T08:24:50.881 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:24:50.882 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-04-24T08:24:50.883 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-04-24T08:24:50.883 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.a 2024-04-24T08:24:51.289 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 24 08:24:51 smithi028 systemd[1]: Stopping Ceph mds.a for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:24:52.039 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 24 08:24:51 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-a[67040]: 2024-04-24T08:24:51.612+0000 7fed46600700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:24:52.039 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 24 08:24:51 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-a[67040]: 2024-04-24T08:24:51.612+0000 7fed46600700 -1 mds.a *** got signal Terminated *** 2024-04-24T08:24:55.101 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 24 08:24:54 smithi028 podman[94675]: 2024-04-24 08:24:54.793435103 +0000 UTC m=+3.459602122 container died e9d1c0734f937c94376e5d3cefdfe5c1c5653bc950dc0136be151867107ffbd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-a, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-04-24T08:24:55.464 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 24 08:24:55 smithi028 podman[94675]: 2024-04-24 08:24:55.24489255 +0000 UTC m=+3.911059568 container cleanup e9d1c0734f937c94376e5d3cefdfe5c1c5653bc950dc0136be151867107ffbd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-a, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-04-24T08:24:55.464 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 24 08:24:55 smithi028 bash[94675]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-a 2024-04-24T08:24:55.789 INFO:journalctl@ceph.mds.a.smithi028.stdout:Apr 24 08:24:55 smithi028 podman[94701]: 2024-04-24 08:24:55.462624216 +0000 UTC m=+0.656261037 container remove e9d1c0734f937c94376e5d3cefdfe5c1c5653bc950dc0136be151867107ffbd6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-a, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:24:56.030 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.a.service' 2024-04-24T08:24:56.156 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:24:56.156 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-04-24T08:24:56.156 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-04-24T08:24:56.157 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.d 2024-04-24T08:24:56.539 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 24 08:24:56 smithi028 systemd[1]: Stopping Ceph mds.d for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:24:57.039 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 24 08:24:56 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-d[66053]: 2024-04-24T08:24:56.746+0000 7f1b18c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:24:57.039 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 24 08:24:56 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-d[66053]: 2024-04-24T08:24:56.746+0000 7f1b18c00700 -1 mds.d *** got signal Terminated *** 2024-04-24T08:25:00.398 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 24 08:25:00 smithi028 podman[94806]: 2024-04-24 08:25:00.138467511 +0000 UTC m=+3.523264712 container died 9ef75adc945fc74cf88f4e78eb881427cfdced967a1536b2c81d62ca6d17b220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-d, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-24T08:25:00.742 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 24 08:25:00 smithi028 podman[94806]: 2024-04-24 08:25:00.506992434 +0000 UTC m=+3.891789675 container cleanup 9ef75adc945fc74cf88f4e78eb881427cfdced967a1536b2c81d62ca6d17b220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-d, ceph=True, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T08:25:00.743 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 24 08:25:00 smithi028 bash[94806]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-d 2024-04-24T08:25:01.039 INFO:journalctl@ceph.mds.d.smithi028.stdout:Apr 24 08:25:00 smithi028 podman[94820]: 2024-04-24 08:25:00.741332135 +0000 UTC m=+0.590186207 container remove 9ef75adc945fc74cf88f4e78eb881427cfdced967a1536b2c81d62ca6d17b220 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-d, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:25:01.260 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.d.service' 2024-04-24T08:25:01.376 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:25:01.376 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-04-24T08:25:01.376 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-04-24T08:25:01.377 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.g 2024-04-24T08:25:01.789 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 24 08:25:01 smithi028 systemd[1]: Stopping Ceph mds.g for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:25:02.289 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 24 08:25:02 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-g[66667]: 2024-04-24T08:25:02.016+0000 7fb81f200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:25:02.289 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 24 08:25:02 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-g[66667]: 2024-04-24T08:25:02.016+0000 7fb81f200700 -1 mds.g *** got signal Terminated *** 2024-04-24T08:25:04.526 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 24 08:25:04 smithi028 podman[94924]: 2024-04-24 08:25:04.230803218 +0000 UTC m=+2.329874733 container died 5dce281f50a2520588c44ff7ce7bc526fd6819abe3e26051a8790326d6019afa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-g, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-04-24T08:25:04.789 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 24 08:25:04 smithi028 podman[94924]: 2024-04-24 08:25:04.635774043 +0000 UTC m=+2.734845551 container cleanup 5dce281f50a2520588c44ff7ce7bc526fd6819abe3e26051a8790326d6019afa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-g, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.build-date=20240424, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-04-24T08:25:04.789 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 24 08:25:04 smithi028 bash[94924]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-g 2024-04-24T08:25:05.287 INFO:journalctl@ceph.mds.g.smithi028.stdout:Apr 24 08:25:04 smithi028 podman[94939]: 2024-04-24 08:25:04.828425174 +0000 UTC m=+0.579489038 container remove 5dce281f50a2520588c44ff7ce7bc526fd6819abe3e26051a8790326d6019afa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-g, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-24T08:25:05.329 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.g.service' 2024-04-24T08:25:05.444 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:25:05.444 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-04-24T08:25:05.444 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-04-24T08:25:05.444 DEBUG:teuthology.orchestra.run.smithi028:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.j 2024-04-24T08:25:06.039 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 24 08:25:05 smithi028 systemd[1]: Stopping Ceph mds.j for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:25:06.539 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 24 08:25:06 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-j[65694]: 2024-04-24T08:25:06.096+0000 7fbe3b000700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:25:06.539 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 24 08:25:06 smithi028 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-j[65694]: 2024-04-24T08:25:06.096+0000 7fbe3b000700 -1 mds.j *** got signal Terminated *** 2024-04-24T08:25:08.289 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 24 08:25:08 smithi028 podman[95045]: 2024-04-24 08:25:08.003795353 +0000 UTC m=+2.048553865 container died 98ab3eba529b945687d27c86e6572556c2144bc77bbdfe4ad1e72effc3908b95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0) 2024-04-24T08:25:08.708 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 24 08:25:08 smithi028 podman[95045]: 2024-04-24 08:25:08.406216758 +0000 UTC m=+2.450975286 container cleanup 98ab3eba529b945687d27c86e6572556c2144bc77bbdfe4ad1e72effc3908b95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-j, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-04-24T08:25:08.708 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 24 08:25:08 smithi028 bash[95045]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-j 2024-04-24T08:25:08.961 INFO:journalctl@ceph.mds.j.smithi028.stdout:Apr 24 08:25:08 smithi028 podman[95070]: 2024-04-24 08:25:08.707130562 +0000 UTC m=+0.685754702 container remove 98ab3eba529b945687d27c86e6572556c2144bc77bbdfe4ad1e72effc3908b95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-j, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-04-24T08:25:08.968 DEBUG:teuthology.orchestra.run.smithi028:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.j.service' 2024-04-24T08:25:09.085 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:25:09.085 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-04-24T08:25:09.085 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-04-24T08:25:09.086 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.b 2024-04-24T08:25:09.373 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:25:09 smithi117 systemd[1]: Stopping Ceph mds.b for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:25:09.628 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:25:09 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-b[62891]: 2024-04-24T08:25:09.372+0000 7fa94bb4a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:25:09.628 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:25:09 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-b[62891]: 2024-04-24T08:25:09.372+0000 7fa94bb4a700 -1 mds.b *** got signal Terminated *** 2024-04-24T08:25:12.378 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:25:12 smithi117 podman[69395]: 2024-04-24 08:25:12.114855729 +0000 UTC m=+2.841907046 container died cc15683a7e2d9f76fadbc1d73558683f5e35f3cfac110a7210dd2492359aef2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-b, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_CLEAN=True) 2024-04-24T08:25:12.765 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:25:12 smithi117 podman[69395]: 2024-04-24 08:25:12.488538909 +0000 UTC m=+3.215590232 container cleanup cc15683a7e2d9f76fadbc1d73558683f5e35f3cfac110a7210dd2492359aef2a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240424, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-04-24T08:25:12.765 INFO:journalctl@ceph.mds.b.smithi117.stdout:Apr 24 08:25:12 smithi117 bash[69395]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-b 2024-04-24T08:25:12.918 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.b.service' 2024-04-24T08:25:12.952 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:25:12.952 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-04-24T08:25:12.953 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-04-24T08:25:12.953 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.e 2024-04-24T08:25:13.378 INFO:journalctl@ceph.mds.e.smithi117.stdout:Apr 24 08:25:13 smithi117 systemd[1]: Stopping Ceph mds.e for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:25:13.378 INFO:journalctl@ceph.mds.e.smithi117.stdout:Apr 24 08:25:13 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-e[62258]: 2024-04-24T08:25:13.275+0000 7f07a6eae700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:25:13.379 INFO:journalctl@ceph.mds.e.smithi117.stdout:Apr 24 08:25:13 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-e[62258]: 2024-04-24T08:25:13.275+0000 7f07a6eae700 -1 mds.e *** got signal Terminated *** 2024-04-24T08:25:15.251 INFO:journalctl@ceph.mds.e.smithi117.stdout:Apr 24 08:25:14 smithi117 podman[69529]: 2024-04-24 08:25:14.950382867 +0000 UTC m=+1.785515532 container died 25e09601568a697f1dda8718d24a97f60ac361997a738a38f880a7fa2b48c0a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-e, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-04-24T08:25:15.576 INFO:journalctl@ceph.mds.e.smithi117.stdout:Apr 24 08:25:15 smithi117 podman[69529]: 2024-04-24 08:25:15.316973418 +0000 UTC m=+2.152106074 container cleanup 25e09601568a697f1dda8718d24a97f60ac361997a738a38f880a7fa2b48c0a5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-e, org.label-schema.vendor=CentOS, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:25:15.577 INFO:journalctl@ceph.mds.e.smithi117.stdout:Apr 24 08:25:15 smithi117 bash[69529]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-e 2024-04-24T08:25:15.736 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.e.service' 2024-04-24T08:25:15.769 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:25:15.769 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-04-24T08:25:15.770 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-04-24T08:25:15.770 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.h 2024-04-24T08:25:16.121 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:25:15 smithi117 systemd[1]: Stopping Ceph mds.h for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:25:16.378 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:25:16 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-h[62567]: 2024-04-24T08:25:16.120+0000 7f2bc6bb9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:25:16.378 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:25:16 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-h[62567]: 2024-04-24T08:25:16.120+0000 7f2bc6bb9700 -1 mds.h *** got signal Terminated *** 2024-04-24T08:25:20.378 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:25:20 smithi117 podman[69648]: 2024-04-24 08:25:20.126987503 +0000 UTC m=+4.144615663 container died 070fa8e2068bf39f9c159592fe0fa22919620cee3e670413a2513620284d1d8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-h, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-04-24T08:25:20.671 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:25:20 smithi117 podman[69648]: 2024-04-24 08:25:20.445333958 +0000 UTC m=+4.462962103 container cleanup 070fa8e2068bf39f9c159592fe0fa22919620cee3e670413a2513620284d1d8d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-h, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:25:20.672 INFO:journalctl@ceph.mds.h.smithi117.stdout:Apr 24 08:25:20 smithi117 bash[69648]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-h 2024-04-24T08:25:20.883 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.h.service' 2024-04-24T08:25:20.915 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:25:20.915 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-04-24T08:25:20.915 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-04-24T08:25:20.915 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.k 2024-04-24T08:25:21.233 INFO:journalctl@ceph.mds.k.smithi117.stdout:Apr 24 08:25:20 smithi117 systemd[1]: Stopping Ceph mds.k for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:25:21.628 INFO:journalctl@ceph.mds.k.smithi117.stdout:Apr 24 08:25:21 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-k[61948]: 2024-04-24T08:25:21.231+0000 7feecec31700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:25:21.628 INFO:journalctl@ceph.mds.k.smithi117.stdout:Apr 24 08:25:21 smithi117 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-k[61948]: 2024-04-24T08:25:21.231+0000 7feecec31700 -1 mds.k *** got signal Terminated *** 2024-04-24T08:25:24.442 INFO:journalctl@ceph.mds.k.smithi117.stdout:Apr 24 08:25:24 smithi117 podman[69768]: 2024-04-24 08:25:24.13356425 +0000 UTC m=+3.002440429 container died 8a8f543f281c41933ac1c8af908d432b83250f318ee1bdba92439c296b16c7ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T08:25:24.758 INFO:journalctl@ceph.mds.k.smithi117.stdout:Apr 24 08:25:24 smithi117 podman[69768]: 2024-04-24 08:25:24.515180906 +0000 UTC m=+3.384057087 container cleanup 8a8f543f281c41933ac1c8af908d432b83250f318ee1bdba92439c296b16c7ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-k, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-04-24T08:25:24.758 INFO:journalctl@ceph.mds.k.smithi117.stdout:Apr 24 08:25:24 smithi117 bash[69768]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-k 2024-04-24T08:25:24.880 DEBUG:teuthology.orchestra.run.smithi117:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.k.service' 2024-04-24T08:25:24.910 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:25:24.910 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-04-24T08:25:24.910 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-04-24T08:25:24.911 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.c 2024-04-24T08:25:25.190 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:25:24 smithi194 systemd[1]: Stopping Ceph mds.c for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:25:25.459 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:25:25 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-c[63839]: 2024-04-24T08:25:25.230+0000 7f7a08506700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:25:25.459 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:25:25 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-c[63839]: 2024-04-24T08:25:25.230+0000 7f7a08506700 -1 mds.c *** got signal Terminated *** 2024-04-24T08:25:28.382 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:25:28 smithi194 podman[70933]: 2024-04-24 08:25:28.08543805 +0000 UTC m=+2.980075897 container died d1d18f3a853e002baebe18e0f190d16a45c754c1529e0286930b2b1b036d04c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, ceph=True, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-04-24T08:25:28.690 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:25:28 smithi194 podman[70933]: 2024-04-24 08:25:28.480269328 +0000 UTC m=+3.374907175 container cleanup d1d18f3a853e002baebe18e0f190d16a45c754c1529e0286930b2b1b036d04c7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-c, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240424, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67) 2024-04-24T08:25:28.691 INFO:journalctl@ceph.mds.c.smithi194.stdout:Apr 24 08:25:28 smithi194 bash[70933]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-c 2024-04-24T08:25:28.855 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.c.service' 2024-04-24T08:25:28.888 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:25:28.888 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-04-24T08:25:28.889 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-04-24T08:25:28.889 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.f 2024-04-24T08:25:29.190 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:25:28 smithi194 systemd[1]: Stopping Ceph mds.f for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:25:29.690 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:25:29 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-f[63533]: 2024-04-24T08:25:29.220+0000 7f0bf1456700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:25:29.691 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:25:29 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-f[63533]: 2024-04-24T08:25:29.220+0000 7f0bf1456700 -1 mds.f *** got signal Terminated *** 2024-04-24T08:25:32.410 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:25:32 smithi194 podman[71067]: 2024-04-24 08:25:32.093911104 +0000 UTC m=+2.982907609 container died fc34c6ee65bff0444d5b77c1569b94e0d224fb2703ec46aee328222273775ea9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240424, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.vendor=CentOS) 2024-04-24T08:25:32.410 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:25:32 smithi194 podman[71067]: 2024-04-24 08:25:32.409767597 +0000 UTC m=+3.298764102 container cleanup fc34c6ee65bff0444d5b77c1569b94e0d224fb2703ec46aee328222273775ea9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-f, org.label-schema.build-date=20240424, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-main-apr17-a654945-1f81fb3, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-04-24T08:25:32.690 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:25:32 smithi194 bash[71067]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-f 2024-04-24T08:25:32.691 INFO:journalctl@ceph.mds.f.smithi194.stdout:Apr 24 08:25:32 smithi194 podman[71081]: 2024-04-24 08:25:32.609862269 +0000 UTC m=+0.510012805 container remove fc34c6ee65bff0444d5b77c1569b94e0d224fb2703ec46aee328222273775ea9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.schema-version=1.0, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240424) 2024-04-24T08:25:32.832 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.f.service' 2024-04-24T08:25:32.864 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:25:32.865 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-04-24T08:25:32.865 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-04-24T08:25:32.865 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.i 2024-04-24T08:25:33.187 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:25:32 smithi194 systemd[1]: Stopping Ceph mds.i for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:25:33.440 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:25:33 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-i[64160]: 2024-04-24T08:25:33.186+0000 7f0993d6e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:25:33.441 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:25:33 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-i[64160]: 2024-04-24T08:25:33.186+0000 7f0993d6e700 -1 mds.i *** got signal Terminated *** 2024-04-24T08:25:36.190 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:25:36 smithi194 podman[71190]: 2024-04-24 08:25:36.085117817 +0000 UTC m=+3.006180450 container died a5cb61bed1c11731badde0d125941a2f273e0b94b8b7daaacea463f87c0e6038 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-i, ceph=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-04-24T08:25:36.690 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:25:36 smithi194 podman[71190]: 2024-04-24 08:25:36.48688599 +0000 UTC m=+3.407948624 container cleanup a5cb61bed1c11731badde0d125941a2f273e0b94b8b7daaacea463f87c0e6038 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-i, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240424) 2024-04-24T08:25:36.691 INFO:journalctl@ceph.mds.i.smithi194.stdout:Apr 24 08:25:36 smithi194 bash[71190]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-i 2024-04-24T08:25:36.832 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.i.service' 2024-04-24T08:25:36.864 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:25:36.864 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-04-24T08:25:36.864 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-04-24T08:25:36.865 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.l 2024-04-24T08:25:37.174 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 24 08:25:36 smithi194 systemd[1]: Stopping Ceph mds.l for 473aaf58-0210-11ef-bc93-c7b262605968... 2024-04-24T08:25:37.440 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 24 08:25:37 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-l[63227]: 2024-04-24T08:25:37.172+0000 7fa367f6b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-04-24T08:25:37.441 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 24 08:25:37 smithi194 ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-l[63227]: 2024-04-24T08:25:37.172+0000 7fa367f6b700 -1 mds.l *** got signal Terminated *** 2024-04-24T08:25:41.441 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 24 08:25:41 smithi194 podman[71313]: 2024-04-24 08:25:41.185250809 +0000 UTC m=+4.105326041 container died 95974d00367b0e2b6faefea8308efc63834ee3dd1cdc790790031b074b5aa24a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-l, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240424, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=rishabh-main-apr17-a654945-1f81fb3) 2024-04-24T08:25:41.881 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 24 08:25:41 smithi194 podman[71313]: 2024-04-24 08:25:41.531895979 +0000 UTC m=+4.451971217 container cleanup 95974d00367b0e2b6faefea8308efc63834ee3dd1cdc790790031b074b5aa24a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-l, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=rishabh-main-apr17-a654945-1f81fb3, org.label-schema.build-date=20240424, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-04-24T08:25:41.882 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 24 08:25:41 smithi194 bash[71313]: ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-l 2024-04-24T08:25:41.882 INFO:journalctl@ceph.mds.l.smithi194.stdout:Apr 24 08:25:41 smithi194 podman[71327]: 2024-04-24 08:25:41.682562814 +0000 UTC m=+0.483997876 container remove 95974d00367b0e2b6faefea8308efc63834ee3dd1cdc790790031b074b5aa24a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:efd0727610781e3b0f3fd91053d49c16375fcec8c20c19c323264ec873568644, name=ceph-473aaf58-0210-11ef-bc93-c7b262605968-mds-l, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=rishabh-main-apr17-a654945-1f81fb3, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=e83daab126629de4b83b70f7819e84da953b5b67, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240424, GIT_BRANCH=HEAD) 2024-04-24T08:25:41.903 DEBUG:teuthology.orchestra.run.smithi194:> sudo pkill -f 'journalctl -f -n 0 -u ceph-473aaf58-0210-11ef-bc93-c7b262605968@mds.l.service' 2024-04-24T08:25:41.933 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-04-24T08:25:41.933 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-04-24T08:25:41.934 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 473aaf58-0210-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-24T08:25:42.267 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:25:47.449 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 473aaf58-0210-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-24T08:25:47.636 INFO:teuthology.orchestra.run.smithi117.stdout:Deleting cluster with fsid: 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:25:50.106 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 473aaf58-0210-11ef-bc93-c7b262605968 --force --keep-logs 2024-04-24T08:25:50.295 INFO:teuthology.orchestra.run.smithi194.stdout:Deleting cluster with fsid: 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:25:52.703 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T08:25:52.796 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T08:25:52.827 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-04-24T08:25:52.853 INFO:tasks.cephadm:Archiving crash dumps... 2024-04-24T08:25:52.855 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204/remote/smithi028/crash 2024-04-24T08:25:52.856 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/crash -- . 2024-04-24T08:25:52.958 INFO:teuthology.orchestra.run.smithi028.stderr:tar: /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-24T08:25:52.958 INFO:teuthology.orchestra.run.smithi028.stderr:tar: Error is not recoverable: exiting now 2024-04-24T08:25:52.963 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204/remote/smithi117/crash 2024-04-24T08:25:52.964 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/crash -- . 2024-04-24T08:25:53.181 INFO:teuthology.orchestra.run.smithi117.stderr:tar: /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-24T08:25:53.182 INFO:teuthology.orchestra.run.smithi117.stderr:tar: Error is not recoverable: exiting now 2024-04-24T08:25:53.183 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/crash to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204/remote/smithi194/crash 2024-04-24T08:25:53.184 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/crash -- . 2024-04-24T08:25:53.219 INFO:teuthology.orchestra.run.smithi194.stderr:tar: /var/lib/ceph/473aaf58-0210-11ef-bc93-c7b262605968/crash: Cannot open: No such file or directory 2024-04-24T08:25:53.220 INFO:teuthology.orchestra.run.smithi194.stderr:tar: Error is not recoverable: exiting now 2024-04-24T08:25:53.221 INFO:tasks.cephadm:Checking cluster log for badness... 2024-04-24T08:25:53.221 DEBUG:teuthology.orchestra.run.smithi028:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-04-24T08:25:53.383 INFO:tasks.cephadm:Compressing logs... 2024-04-24T08:25:53.384 DEBUG:teuthology.orchestra.run.smithi028:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T08:25:53.388 DEBUG:teuthology.orchestra.run.smithi117:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T08:25:53.390 DEBUG:teuthology.orchestra.run.smithi194:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-04-24T08:25:53.416 INFO:teuthology.orchestra.run.smithi194.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.log 2024-04-24T08:25:53.416 INFO:teuthology.orchestra.run.smithi194.stderr:: No such file or directory 2024-04-24T08:25:53.416 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.2.log 2024-04-24T08:25:53.417 INFO:teuthology.orchestra.run.smithi194.stderr:gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.f.log 2024-04-24T08:25:53.417 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-24T08:25:53.417 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.5.log 2024-04-24T08:25:53.417 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.c.log 2024-04-24T08:25:53.417 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mon.c.log 2024-04-24T08:25:53.417 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mgr.z.log 2024-04-24T08:25:53.422 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.i.log 2024-04-24T08:25:53.422 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-24T08:25:53.422 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.11.log 2024-04-24T08:25:53.422 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.8.log 2024-04-24T08:25:53.422 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-24T08:25:53.422 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.l.log 2024-04-24T08:25:53.422 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-24T08:25:53.424 INFO:teuthology.orchestra.run.smithi117.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.log 2024-04-24T08:25:53.424 INFO:teuthology.orchestra.run.smithi117.stderr:: No such file or directory 2024-04-24T08:25:53.424 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-24T08:25:53.424 INFO:teuthology.orchestra.run.smithi117.stderr:gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.7.log 2024-04-24T08:25:53.424 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.log: /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.b.log 2024-04-24T08:25:53.425 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.4.log 2024-04-24T08:25:53.425 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.1.log 2024-04-24T08:25:53.425 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mgr.y.log 2024-04-24T08:25:53.425 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.h.log 2024-04-24T08:25:53.425 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.10.log 2024-04-24T08:25:53.425 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.h.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-24T08:25:53.425 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.e.log 2024-04-24T08:25:53.425 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-24T08:25:53.426 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mon.b.log 2024-04-24T08:25:53.426 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.k.log 2024-04-24T08:25:53.426 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mon.b.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-24T08:25:53.484 INFO:teuthology.orchestra.run.smithi028.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-04-24T08:25:53.484 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.log 2024-04-24T08:25:53.485 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.audit.log 2024-04-24T08:25:53.486 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.g.log 2024-04-24T08:25:53.488 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.j.log 2024-04-24T08:25:53.489 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.3.log 2024-04-24T08:25:53.491 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.j.log: 87.0% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-24T08:25:53.491 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.0.log 2024-04-24T08:25:53.493 INFO:teuthology.orchestra.run.smithi028.stderr:gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mgr.x.log 2024-04-24T08:25:53.494 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.3.log: /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mon.a.log 2024-04-24T08:25:53.494 INFO:teuthology.orchestra.run.smithi028.stderr: 94.5% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.j.log.gz 2024-04-24T08:25:53.495 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.6.log 2024-04-24T08:25:53.496 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.cephadm.log 2024-04-24T08:25:53.498 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.a.log 2024-04-24T08:25:53.499 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-volume.log 2024-04-24T08:25:53.500 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.a.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.d.log 2024-04-24T08:25:53.501 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.9.log 2024-04-24T08:25:53.502 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-04-24T08:25:53.502 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.l.log: /var/log/ceph/cephadm.log: 83.5% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-24T08:25:53.504 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.9.log: 94.4% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.a.log.gz 2024-04-24T08:25:53.506 INFO:teuthology.orchestra.run.smithi028.stderr:/var/log/ceph/cephadm.log: 94.3% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.d.log.gz 2024-04-24T08:25:53.531 INFO:teuthology.orchestra.run.smithi028.stderr: 83.9% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-24T08:25:53.548 INFO:teuthology.orchestra.run.smithi028.stderr: 90.2% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-24T08:25:53.569 INFO:teuthology.orchestra.run.smithi117.stderr:/var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.k.log: /var/log/ceph/cephadm.log: 83.6% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.cephadm.log.gz 2024-04-24T08:25:53.585 INFO:teuthology.orchestra.run.smithi028.stderr: 93.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-24T08:25:53.601 INFO:teuthology.orchestra.run.smithi117.stderr: 92.1% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mgr.y.log.gz 2024-04-24T08:25:53.632 INFO:teuthology.orchestra.run.smithi117.stderr: 87.0% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-24T08:25:53.664 INFO:teuthology.orchestra.run.smithi117.stderr: 90.3% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.audit.log.gz 2024-04-24T08:25:53.667 INFO:teuthology.orchestra.run.smithi117.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-24T08:25:53.681 INFO:teuthology.orchestra.run.smithi194.stderr: 94.5% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.l.log.gz 2024-04-24T08:25:53.685 INFO:teuthology.orchestra.run.smithi117.stderr: 94.2% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.h.log.gz 2024-04-24T08:25:53.717 INFO:teuthology.orchestra.run.smithi117.stderr: 94.4% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.k.log.gz 2024-04-24T08:25:53.717 INFO:teuthology.orchestra.run.smithi117.stderr: 94.5% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.e.log.gz 2024-04-24T08:25:53.720 INFO:teuthology.orchestra.run.smithi194.stderr: 92.2% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mgr.z.log.gz 2024-04-24T08:25:53.723 INFO:teuthology.orchestra.run.smithi117.stderr: 91.0% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-24T08:25:53.759 INFO:teuthology.orchestra.run.smithi194.stderr: 92.8% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.i.log.gz 2024-04-24T08:25:53.804 INFO:teuthology.orchestra.run.smithi028.stderr: 90.9% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-24T08:25:53.805 INFO:teuthology.orchestra.run.smithi117.stderr: 92.7% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.b.log.gz 2024-04-24T08:25:53.832 INFO:teuthology.orchestra.run.smithi194.stderr: 94.5% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.f.log.gz 2024-04-24T08:25:53.834 INFO:teuthology.orchestra.run.smithi194.stderr: 91.0% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-volume.log.gz 2024-04-24T08:25:53.900 INFO:teuthology.orchestra.run.smithi194.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-04-24T08:25:53.901 INFO:teuthology.orchestra.run.smithi194.stderr: 90.3% 87.0% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.audit.log.gz -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph.log.gz 2024-04-24T08:25:53.901 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-04-24T08:25:55.060 INFO:teuthology.orchestra.run.smithi028.stderr: 90.3% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mgr.x.log.gz 2024-04-24T08:25:59.090 INFO:teuthology.orchestra.run.smithi194.stderr: 91.8% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mon.c.log.gz 2024-04-24T08:26:00.266 INFO:teuthology.orchestra.run.smithi194.stderr: 95.3% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.c.log.gz 2024-04-24T08:26:01.830 INFO:teuthology.orchestra.run.smithi117.stderr: 91.7% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mon.b.log.gz 2024-04-24T08:26:02.656 INFO:teuthology.orchestra.run.smithi028.stderr: 90.3% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mon.a.log.gz 2024-04-24T08:26:58.202 INFO:teuthology.orchestra.run.smithi028.stderr: 93.8% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-mds.g.log.gz 2024-04-24T08:29:16.815 INFO:teuthology.orchestra.run.smithi194.stderr: 92.2% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.11.log.gz 2024-04-24T08:29:16.866 INFO:teuthology.orchestra.run.smithi117.stderr: 92.2% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.4.log.gz 2024-04-24T08:31:46.020 INFO:teuthology.orchestra.run.smithi117.stderr: 92.2% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.10.log.gz 2024-04-24T08:36:26.594 INFO:teuthology.orchestra.run.smithi117.stderr: 92.1% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.7.log.gz 2024-04-24T08:37:05.926 INFO:teuthology.orchestra.run.smithi117.stderr: 92.0% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.1.log.gz 2024-04-24T08:37:05.928 INFO:teuthology.orchestra.run.smithi117.stderr: 2024-04-24T08:37:05.928 INFO:teuthology.orchestra.run.smithi117.stderr:real 11m12.524s 2024-04-24T08:37:05.928 INFO:teuthology.orchestra.run.smithi117.stderr:user 7m37.279s 2024-04-24T08:37:05.929 INFO:teuthology.orchestra.run.smithi117.stderr:sys 0m17.962s 2024-04-24T08:38:43.253 INFO:teuthology.orchestra.run.smithi028.stderr: 92.2% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.9.log.gz 2024-04-24T08:39:32.117 INFO:teuthology.orchestra.run.smithi194.stderr: 92.2% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.8.log.gz 2024-04-24T08:39:59.016 INFO:teuthology.orchestra.run.smithi028.stderr: 92.1% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.6.log.gz 2024-04-24T08:40:15.217 INFO:teuthology.orchestra.run.smithi028.stderr: 92.0% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.3.log.gz 2024-04-24T08:40:17.432 INFO:teuthology.orchestra.run.smithi028.stderr: 92.0% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.0.log.gz 2024-04-24T08:40:17.436 INFO:teuthology.orchestra.run.smithi028.stderr: 2024-04-24T08:40:17.437 INFO:teuthology.orchestra.run.smithi028.stderr:real 14m24.013s 2024-04-24T08:40:17.437 INFO:teuthology.orchestra.run.smithi028.stderr:user 20m51.295s 2024-04-24T08:40:17.437 INFO:teuthology.orchestra.run.smithi028.stderr:sys 2m33.931s 2024-04-24T08:41:06.470 INFO:teuthology.orchestra.run.smithi194.stderr: 92.2% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.5.log.gz 2024-04-24T08:41:35.012 INFO:teuthology.orchestra.run.smithi194.stderr: 92.1% -- replaced with /var/log/ceph/473aaf58-0210-11ef-bc93-c7b262605968/ceph-osd.2.log.gz 2024-04-24T08:41:35.033 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-04-24T08:41:35.033 INFO:teuthology.orchestra.run.smithi194.stderr:real 15m41.629s 2024-04-24T08:41:35.033 INFO:teuthology.orchestra.run.smithi194.stderr:user 7m46.424s 2024-04-24T08:41:35.034 INFO:teuthology.orchestra.run.smithi194.stderr:sys 0m19.738s 2024-04-24T08:41:35.034 INFO:tasks.cephadm:Archiving logs... 2024-04-24T08:41:35.035 DEBUG:teuthology.misc:Transferring archived files from smithi028:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204/remote/smithi028/log 2024-04-24T08:41:35.036 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T08:42:48.463 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204/remote/smithi117/log 2024-04-24T08:42:48.464 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T08:44:03.156 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204/remote/smithi194/log 2024-04-24T08:44:03.157 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/log/ceph -- . 2024-04-24T08:45:19.037 INFO:tasks.cephadm:Removing cluster... 2024-04-24T08:45:19.038 DEBUG:teuthology.orchestra.run.smithi028:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 473aaf58-0210-11ef-bc93-c7b262605968 --force 2024-04-24T08:45:19.574 INFO:teuthology.orchestra.run.smithi028.stdout:Deleting cluster with fsid: 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:45:21.506 DEBUG:teuthology.orchestra.run.smithi117:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 473aaf58-0210-11ef-bc93-c7b262605968 --force 2024-04-24T08:45:21.699 INFO:teuthology.orchestra.run.smithi117.stdout:Deleting cluster with fsid: 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:45:22.446 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 473aaf58-0210-11ef-bc93-c7b262605968 --force 2024-04-24T08:45:22.858 INFO:teuthology.orchestra.run.smithi194.stdout:Deleting cluster with fsid: 473aaf58-0210-11ef-bc93-c7b262605968 2024-04-24T08:45:23.705 INFO:tasks.cephadm:Removing cephadm ... 2024-04-24T08:45:23.706 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-24T08:45:23.777 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-24T08:45:23.879 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf /home/ubuntu/cephtest/cephadm 2024-04-24T08:45:23.896 INFO:tasks.cephadm:Teardown complete 2024-04-24T08:45:23.896 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-04-24T08:45:23.908 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-04-24T08:45:23.909 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T08:45:23.916 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T08:45:23.919 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-04-24T08:45:24.072 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-04-24T08:45:24.086 INFO:teuthology.task.clock:Checking final clock skew... 2024-04-24T08:45:24.087 DEBUG:teuthology.orchestra.run.smithi028:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T08:45:24.095 DEBUG:teuthology.orchestra.run.smithi117:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T08:45:24.097 DEBUG:teuthology.orchestra.run.smithi194:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-04-24T08:45:24.111 INFO:teuthology.orchestra.run.smithi117.stderr:bash: line 1: ntpq: command not found 2024-04-24T08:45:24.112 INFO:teuthology.orchestra.run.smithi194.stderr:bash: line 1: ntpq: command not found 2024-04-24T08:45:24.122 INFO:teuthology.orchestra.run.smithi117.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T08:45:24.122 INFO:teuthology.orchestra.run.smithi117.stdout:=============================================================================== 2024-04-24T08:45:24.122 INFO:teuthology.orchestra.run.smithi117.stdout:^- hv01.front.sepia.ceph.com 2 6 377 51 +1214us[+1214us] +/- 598ms 2024-04-24T08:45:24.122 INFO:teuthology.orchestra.run.smithi117.stdout:^* hv02.front.sepia.ceph.com 3 6 377 54 -414us[ -398us] +/- 64ms 2024-04-24T08:45:24.122 INFO:teuthology.orchestra.run.smithi117.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 49 +550us[ +550us] +/- 77ms 2024-04-24T08:45:24.123 INFO:teuthology.orchestra.run.smithi117.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:45:24.144 INFO:teuthology.orchestra.run.smithi028.stderr:bash: line 1: ntpq: command not found 2024-04-24T08:45:24.184 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T08:45:24.184 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-04-24T08:45:24.184 INFO:teuthology.orchestra.run.smithi194.stdout:^- hv01.front.sepia.ceph.com 2 6 377 59 +1233us[+1208us] +/- 598ms 2024-04-24T08:45:24.184 INFO:teuthology.orchestra.run.smithi194.stdout:^* hv02.front.sepia.ceph.com 3 6 377 55 -397us[ -422us] +/- 64ms 2024-04-24T08:45:24.184 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 59 +543us[ +518us] +/- 77ms 2024-04-24T08:45:24.184 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:45:24.265 INFO:teuthology.orchestra.run.smithi028.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-04-24T08:45:24.265 INFO:teuthology.orchestra.run.smithi028.stdout:=============================================================================== 2024-04-24T08:45:24.265 INFO:teuthology.orchestra.run.smithi028.stdout:^- hv01.front.sepia.ceph.com 2 7 377 119 +1300us[+1280us] +/- 597ms 2024-04-24T08:45:24.265 INFO:teuthology.orchestra.run.smithi028.stdout:^* hv02.front.sepia.ceph.com 3 6 377 55 -385us[ -402us] +/- 64ms 2024-04-24T08:45:24.265 INFO:teuthology.orchestra.run.smithi028.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 56 +528us[ +510us] +/- 77ms 2024-04-24T08:45:24.265 INFO:teuthology.orchestra.run.smithi028.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-04-24T08:45:24.268 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-04-24T08:45:24.278 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-04-24T08:45:24.279 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-04-24T08:45:24.344 DEBUG:teuthology.orchestra.run.smithi028:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T08:45:24.727 DEBUG:teuthology.orchestra.run.smithi117:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T08:45:24.813 DEBUG:teuthology.orchestra.run.smithi194:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-04-24T08:45:25.025 DEBUG:teuthology.orchestra.run.smithi028:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T08:45:25.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:45:25.123 DEBUG:teuthology.orchestra.run.smithi117:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T08:45:25.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:45:25.155 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-04-24T08:45:25.186 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:45:25.187 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-04-24T08:45:25.198 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-04-24T08:45:25.239 INFO:teuthology.task.internal:Duration was 3353.758292 seconds 2024-04-24T08:45:25.240 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-04-24T08:45:25.272 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-04-24T08:45:25.273 DEBUG:teuthology.orchestra.run.smithi028:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T08:45:25.281 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T08:45:25.283 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-04-24T08:45:25.340 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T08:45:25.350 INFO:teuthology.orchestra.run.smithi117.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T08:45:25.486 INFO:teuthology.orchestra.run.smithi028.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-04-24T08:45:25.902 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-04-24T08:45:25.902 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi028.front.sepia.ceph.com 2024-04-24T08:45:25.902 DEBUG:teuthology.orchestra.run.smithi028:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T08:45:25.975 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi117.front.sepia.ceph.com 2024-04-24T08:45:25.975 DEBUG:teuthology.orchestra.run.smithi117:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T08:45:26.001 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi194.front.sepia.ceph.com 2024-04-24T08:45:26.001 DEBUG:teuthology.orchestra.run.smithi194:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-04-24T08:45:26.039 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-04-24T08:45:26.039 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T08:45:26.047 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T08:45:26.050 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-04-24T08:45:26.137 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-04-24T08:45:26.137 DEBUG:teuthology.orchestra.run.smithi028:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T08:45:26.142 DEBUG:teuthology.orchestra.run.smithi117:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T08:45:26.145 DEBUG:teuthology.orchestra.run.smithi194:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-04-24T08:45:26.685 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-04-24T08:45:26.698 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-04-24T08:45:26.698 DEBUG:teuthology.orchestra.run.smithi028:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T08:45:26.786 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T08:45:26.812 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-04-24T08:45:26.853 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-04-24T08:45:26.866 DEBUG:teuthology.orchestra.run.smithi028:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T08:45:26.874 DEBUG:teuthology.orchestra.run.smithi117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T08:45:26.876 DEBUG:teuthology.orchestra.run.smithi194:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-04-24T08:45:26.923 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = core 2024-04-24T08:45:26.929 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = core 2024-04-24T08:45:27.022 INFO:teuthology.orchestra.run.smithi028.stdout:kernel.core_pattern = core 2024-04-24T08:45:27.086 DEBUG:teuthology.orchestra.run.smithi028:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T08:45:27.141 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:45:27.141 DEBUG:teuthology.orchestra.run.smithi117:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T08:45:27.155 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:45:27.156 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/archive/coredump 2024-04-24T08:45:27.170 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-04-24T08:45:27.170 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-04-24T08:45:27.183 INFO:teuthology.task.internal:Transferring archived files... 2024-04-24T08:45:27.185 DEBUG:teuthology.misc:Transferring archived files from smithi028:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204/remote/smithi028 2024-04-24T08:45:27.185 DEBUG:teuthology.orchestra.run.smithi028:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T08:45:27.301 DEBUG:teuthology.misc:Transferring archived files from smithi117:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204/remote/smithi117 2024-04-24T08:45:27.301 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T08:45:27.338 DEBUG:teuthology.misc:Transferring archived files from smithi194:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204/remote/smithi194 2024-04-24T08:45:27.338 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-04-24T08:45:27.374 INFO:teuthology.task.internal:Removing archive directory... 2024-04-24T08:45:27.375 DEBUG:teuthology.orchestra.run.smithi028:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T08:45:27.382 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T08:45:27.385 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf -- /home/ubuntu/cephtest/archive 2024-04-24T08:45:27.435 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-04-24T08:45:27.449 INFO:teuthology.task.internal:Not uploading archives. 2024-04-24T08:45:27.449 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-04-24T08:45:27.481 INFO:teuthology.task.internal:Tidying up after the test... 2024-04-24T08:45:27.482 DEBUG:teuthology.orchestra.run.smithi028:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T08:45:27.489 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T08:45:27.492 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-04-24T08:45:27.504 INFO:teuthology.orchestra.run.smithi117.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 08:45 /home/ubuntu/cephtest 2024-04-24T08:45:27.506 INFO:teuthology.orchestra.run.smithi194.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 08:45 /home/ubuntu/cephtest 2024-04-24T08:45:27.539 INFO:teuthology.orchestra.run.smithi028.stdout: 393243 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Apr 24 08:45 /home/ubuntu/cephtest 2024-04-24T08:45:27.543 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-04-24T08:45:27.555 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-04-24T08:45:27.615 INFO:teuthology.nuke:Checking targets against current locks 2024-04-24T08:45:27.642 DEBUG:teuthology.nuke:shortname: smithi028 2024-04-24T08:45:27.642 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T08:45:27.666 DEBUG:teuthology.nuke:shortname: smithi117 2024-04-24T08:45:27.667 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T08:45:27.672 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi028.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 07:41:23.360035', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJ23IGa87DVqE2toJsvUYoiEyF3yd2TwVJzcc7VwcWrA5K90KJYbl9LUcnJ2MWJyHTiNpXwFyTJCjg+OOU9F7tU='} 2024-04-24T08:45:27.689 DEBUG:teuthology.nuke:shortname: smithi194 2024-04-24T08:45:27.690 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-04-24T08:45:27.696 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 07:41:23.361045', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLdWI3E0mxIOK39/rI/CQ3u+lUiKGS+RjCJ1l3jgc0xvGxn8Fto+KTFYA1RzUMOA1faH9bpZXiBz4ztcy65WQhk='} 2024-04-24T08:45:27.715 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-04-24_07:32:23-fs-rishabh-main-apr17-a654945-testing-default-smithi/7671204', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-04-24 07:41:23.361851', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNxqTNGooqW5Con49ykJPPFp8+6bTVVnxDBRVkmJdgBxNiLAlD7MYelsNSzzGjmMfTh8wk8ql9yQ9MLteaMjq+I='} 2024-04-24T08:45:27.728 INFO:teuthology.orchestra.console.smithi028:Power off 2024-04-24T08:45:27.729 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-24T08:45:27.752 DEBUG:teuthology.orchestra.console.smithi028:power off output: Chassis Power Control: Down/Off 2024-04-24T08:45:27.752 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:45:27.771 INFO:teuthology.orchestra.console.smithi117:Power off 2024-04-24T08:45:27.771 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-24T08:45:27.789 INFO:teuthology.orchestra.console.smithi194:Power off 2024-04-24T08:45:27.790 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-04-24T08:45:27.799 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-04-24T08:45:27.799 DEBUG:teuthology.orchestra.console.smithi117:power off output: Chassis Power Control: Down/Off 2024-04-24T08:45:27.799 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:45:27.813 DEBUG:teuthology.orchestra.console.smithi194:power off output: Chassis Power Control: Down/Off 2024-04-24T08:45:27.814 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:45:27.823 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-04-24T08:45:27.838 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-04-24T08:45:31.800 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:45:31.825 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:45:31.839 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:45:31.913 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is on 2024-04-24T08:45:31.936 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-04-24T08:45:31.950 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-04-24T08:45:35.916 DEBUG:teuthology.orchestra.console.smithi028:pexpect command: ipmitool -H smithi028.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:45:35.938 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:45:35.951 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:45:36.027 DEBUG:teuthology.orchestra.console.smithi028:check power output: Chassis Power is off 2024-04-24T08:45:36.048 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is on 2024-04-24T08:45:36.062 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-04-24T08:45:36.128 INFO:teuthology.orchestra.console.smithi028:Power off completed 2024-04-24T08:45:40.053 DEBUG:teuthology.orchestra.console.smithi117:pexpect command: ipmitool -H smithi117.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:45:40.062 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-04-24T08:45:40.164 DEBUG:teuthology.orchestra.console.smithi117:check power output: Chassis Power is off 2024-04-24T08:45:40.173 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is off 2024-04-24T08:45:40.266 INFO:teuthology.orchestra.console.smithi117:Power off completed 2024-04-24T08:45:40.274 INFO:teuthology.orchestra.console.smithi194:Power off completed 2024-04-24T08:45:40.381 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 806ec36b21129f113ce356c007dd59694d340b37 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/secure wsync/no} objectstore-ec/bluestore-comp omap_limit/10000 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/random export-check n/3 replication/default} standby-replay tasks/{0-subvolume/{with-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/no 5-workunit/suites/ffsb}} duration: 3353.7582924365997 failure_reason: 'Command failed (workunit test suites/ffsb.sh) on smithi028 with status 1: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=1f81fb397ae98da7563d451a78c61574c8f4e6e0 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/ffsb.sh''' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=282c113091c842419bfbf9e31405648e status: fail success: false 2024-04-24T08:45:40.382 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-04-24T08:45:40.460 INFO:teuthology.run:FAIL